Cloudflare AI crawler settings across 1,000 small-business domains

What proportion of small-business websites served through Cloudflare are unreachable to the search crawlers that answer engines rely on, how much of that is a deliberate robots.txt decision versus an edge default, and is the proportion rising or falling quarter to quarter?

• Sample: 1,000 domains of US small businesses in professional and technical categories, drawn from public business registers by a documented random procedure, with the selection rule and the exclusion list published with the results. Domains are never named individually.
• Cloudflare detection: a domain is classified as served through Cloudflare when response headers carry a Cloudflare server identifier or a cf-ray header on the robots.txt or home page fetch.
• robots.txt observation: the served robots.txt is fetched and parsed under RFC 9309 rules for each named crawler (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, Googlebot as control) and classified as allowed, disallowed for all paths, or no applicable rule. A marker is recorded where the file appears to be a managed robots.txt.
• Edge observation: the home page is fetched once per crawler with that crawler’s documented user agent string and the status code and any challenge page are recorded. One request per crawler per domain, a delay between domains, no retries, from a disclosed egress address.
• Classification: each domain-crawler pair is recorded as reachable, robots-disallowed, edge-blocked (403, 429, 503 or challenge), or unreachable for other reasons (DNS, TLS, timeout), with a timestamp.
• Analysis: shares by crawler, by Cloudflare versus non-Cloudflare domains, and by quarter; the control crawler (Googlebot) separates AI-specific blocks from general bot protection.

No findings are stated until the first quarterly collection is complete, reviewed against the method and signed. This page will then carry the findings with their collection date, and the tables above will be populated. Earlier collections remain visible as the series grows.

• The sample is small businesses in selected categories and does not represent all websites.
• A fetch from one egress address on one day is a snapshot; bot-protection behavior can vary by source network, by time and by request pattern, and a challenge page may reflect reputation of the egress address rather than a crawler-specific rule.
• User agent strings are the documented ones; a site may treat the real crawler differently because it also verifies the source IP range.
• robots.txt parsing follows RFC 9309; sites with non-standard files may be classified conservatively.
• Cloudflare detection by headers can miss domains behind other proxies in front of Cloudflare.
• The study measures reachability, not citation; a reachable site is not necessarily cited.

The collection script is open: scripts/research/cloudflare_ai_crawler_settings.py in the site repository takes a domain list and writes one observation row per domain and crawler with a timestamp. It makes one request per crawler per domain with a delay, no retries and a short timeout, and records exactly what was observed. The domain selection rule and the egress address are published with each quarter’s results so that the collection can be repeated by anyone.

• Cloudflare’s default AI crawler block and how it silently removes you from answers
• Crawl and render readiness
• Does Cloudflare block AI crawlers by default?
• Which AI crawlers reach professional-practice websites: a log study