Insights · Question
Does Cloudflare block AI crawlers by default?
For domains added to Cloudflare since July 2025, yes by default: Cloudflare announced that new domains would have AI crawlers blocked unless the site owner chose to allow them. Older zones are not blocked unless someone enabled the one-click AI bot block Cloudflare introduced in July 2024, or a managed robots.txt that adds disallow rules. The settings live in the zone dashboard, not in the site’s files, and the site works normally in every browser, so the only symptom is absence from AI answers. Check the zone settings, fetch the site as each named crawler and read the logs.
In more detail
The reasoning.
Cloudflare’s position is reasonable for publishers who want to withhold content from model training. For a professional firm that wants answer engines to cite it, the default is a silent exit from those engines, and the organization usually learns of it only when it asks why competitors appear in answers and it does not.
The fix is a settings change, not code: allow the named search crawlers explicitly, decide about training crawlers separately, confirm no managed robots.txt directive disallows them, and confirm bot-protection rules do not challenge verified crawlers. Then verify by fetching as each crawler and by reviewing the crawler lines in the logs.
This site runs on Cloudflare with the crawlers allowed and the settings verified. See Cloudflare’s default AI crawler block and how it silently removes you from answers and should we block AI crawlers.
By David Selakovic · Published 2026-10-16 · Revised 2026-10-16 · Reviewed every 90 days