Which AI crawlers reach professional-practice websites: a log study

Of the named search and AI crawlers, which actually reach professional-practice websites, with what frequency and crawl depth, what share of their requests succeed, and how does this differ between hosting arrangements and site types?

• Participants: professional-practice websites (medical, dental, legal, financial advisory, engineering and similar) whose owners agree to contribute access logs for a defined period. Each participant receives an opaque site id; the mapping is held by Performance Data and never published.
• Inputs: web server access logs in common or combined format, or Cloudflare Logpush HTTP request logs as newline-delimited JSON, for a period of at least thirty consecutive days.
• Extraction: requests whose user agent contains a named crawler token (Googlebot, Bingbot, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, anthropic-ai, PerplexityBot, Google-Extended, Applebot, DuckDuckBot, CCBot and others listed in the script) are retained; every other request is discarded at parse time. Only the date, path, status and crawler name are kept.
• Aggregation: per site, per day, per crawler: request count, distinct paths, status code mix and robots.txt fetches. Site-level metadata recorded separately: category, hosting arrangement, whether served through a CDN, and the site’s own robots.txt and crawler settings at the time.
• Analysis: presence and frequency by crawler across sites; crawl depth as distinct paths per crawler; success rates; comparison by hosting arrangement and by whether the site had explicitly allowed the crawler.
• Verification where feasible: crawler identity is taken from the user agent as a stated limitation; where participants’ logs include source IP ranges, requests are cross-checked against published crawler IP ranges and the verified share is reported.

No findings are stated until the first annual collection is complete, reviewed against the method and signed. This page will then carry the findings with the collection period and participant count, and the tables above will be populated.

• Participants are self-selected sites that agreed to share logs and are not a random sample of professional practices.
• Crawler identity is inferred from user agent strings, which can be spoofed; where source IP verification is possible the verified share is reported, otherwise counts are upper bounds.
• Log formats and retention vary; sites with sampled or truncated logs may understate crawler activity.
• Presence in logs shows that a crawler fetched pages, not that the site was indexed or cited.
• Thirty days may miss crawlers that visit less frequently.

The parsing script is open: scripts/research/crawler_log_study.py in the site repository takes a site id and one or more log files, retains only requests from the named crawlers, and writes one row per site, day and crawler with the aggregates above. Participants can run it themselves and contribute only the aggregated output, which contains no visitor data.

• Which AI crawlers exist, what they respect and how to log them
• Crawl and render readiness
• AI crawl readiness checklist
• Cloudflare AI crawler settings across 1,000 small-business domains