Insights · Field note · Growth and AI visibility
Cloudflare’s default AI crawler block and how it silently removes you from answers
A sensible default for publishers who want to withhold content from training is a silent exit from citations for organizations that want to be found.
In July 2024 Cloudflare introduced a one-click setting to block AI bots and crawlers, and in July 2025 it made blocking the default for new domains added to its network. A managed robots.txt can add disallow rules as well. The settings are reasonable for a publisher protecting content from training. For a professional firm that wants answer engines to cite it, they remove the firm from those engines without anyone noticing, because the site still works in every browser.
In detail
What the evidence shows.
What Cloudflare changed
Cloudflare documented a one-click option in July 2024 that blocks known AI bots, scrapers and crawlers at the edge. In July 2025 it announced that new domains would have AI crawlers blocked by default, with site owners choosing whether to allow them. Cloudflare also offers a managed robots.txt that can insert directives. These are settings in the zone dashboard, not in the site’s own files.
Why nobody notices
The block applies to crawlers, not browsers. Staff, clients and the marketing agency all see a working site. Rankings in Google may be unaffected because Googlebot is not in the blocked set. The only symptom is absence: the firm does not appear in ChatGPT, Claude or Perplexity answers, and absence looks like a content problem rather than a settings problem.
How to tell
Fetch the site as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot and look for 403 responses or challenge pages. Read the zone’s bot settings and the served robots.txt, which may differ from the file in the repository if a managed version is injected. Check the CDN logs for the named crawlers.
Making the decision deliberately
The right setting depends on what the organization wants. A firm that depends on being cited should allow the search crawlers explicitly and decide about training crawlers separately. This site allows the named crawlers in robots.txt and verified the zone settings, which is a settings change at the dashboard level rather than code. See Cloudflare Pages versus managed WordPress hosting.
By David Selakovic · Published 2026-10-16 · Revised 2026-10-16 · Reviewed every 180 days