Insights · Question
Should we block AI crawlers?
It depends on what you want, and the decision is two decisions. Search crawlers such as OAI-SearchBot, ClaudeBot and PerplexityBot fetch pages so that engines can cite them; blocking them removes the organization from those engines’ answers. Training-only controls such as GPTBot and Google-Extended govern whether content may be used to train models; blocking them withholds content from training without affecting search or citation. An organization that depends on being found should allow the search crawlers and decide about training deliberately. Most organizations that block everything did not make a decision; they inherited a default.
In more detail
The reasoning.
The case for blocking training crawlers is real for publishers whose content is the product. For a professional firm, the content exists to make the firm’s expertise findable, and withholding it from training has little upside while blocking search crawlers has a large downside.
Whatever is decided, it should be explicit and verified. robots.txt should name the agents allowed and disallowed; the CDN and bot-protection settings should agree; and the logs should show the allowed crawlers arriving. A decision that exists only in a dashboard default is not a decision.
See which AI crawlers exist, what they respect and how to log them, and the glossary entries for GPTBot and Google-Extended.
By David Selakovic · Published 2026-10-16 · Revised 2026-10-16 · Reviewed every 90 days