Insights · Field note · Growth and AI visibility
Which AI crawlers exist, what they respect and how to log them
The named agents behind the major engines, what each is for, which robots.txt token governs it, and the only way to know whether it arrives.
The engines that cite pages send named crawlers: Googlebot for Google Search and its AI features, with Google-Extended as a training-only control; OAI-SearchBot for ChatGPT search and GPTBot for OpenAI training; ClaudeBot for Anthropic; PerplexityBot for Perplexity; Bingbot for Bing and Copilot. Each identifies itself by user agent and documents that it respects robots.txt. Whether one actually reaches a site is not a matter of policy; it is a matter of the server logs.
In detail
What the evidence shows.
The agents and what they are for
Google documents Googlebot as the crawler for Search, which also feeds AI Overviews and AI Mode, and Google-Extended as a robots token that governs use of content for Gemini training without affecting Search. OpenAI documents GPTBot (training), OAI-SearchBot (search results in ChatGPT) and ChatGPT-User (fetches a user initiates). Anthropic’s ClaudeBot, Perplexity’s PerplexityBot and Microsoft’s Bingbot follow the same pattern: a declared user agent and a published commitment to robots.txt.
Search versus training
The distinction matters for the decision. Blocking a training crawler withholds content from model training; blocking a search crawler removes the site from that engine’s cited sources. An organization that wants to be cited in ChatGPT must allow OAI-SearchBot whatever it decides about GPTBot. Many sites block both without having made that decision.
Where blocks come from
robots.txt rules written years ago. Managed robots.txt features at the hosting or CDN layer. Bot-protection settings that challenge or block unknown agents. Firewall rules. Cloudflare’s default block on new zones. Each is invisible from a browser and visible only by fetching as the crawler and reading the logs.
Logging them
Filter server or CDN logs by user agent for each named crawler. Record arrivals, frequency, status codes and the pages fetched. A crawler that never appears is blocked somewhere; a crawler that receives 403 or challenge pages is blocked at the edge; a crawler that fetches only the home page is finding nothing to follow. This log review is the last step of the AI crawl readiness checklist and the only proof that access exists.
By David Selakovic · Published 2026-10-16 · Revised 2026-10-16 · Reviewed every 180 days