Glossary · C
ClaudeBot
ClaudeBot is the web crawler Anthropic uses to collect content for its Claude models, identified by its user agent and governed by robots.txt.
Extended definition
In more detail.
Like other AI crawlers, ClaudeBot can be allowed or disallowed per site. Blocking it removes the site’s content from what Claude can draw on; allowing it makes the content available for retrieval and citation. Some hosting platforms and managed robots.txt features disallow it without the site owner’s knowledge. The decision should be explicit and verified in the server logs.
In our work
Where it appears.
ClaudeBot is one of the named crawlers in every crawl and render readiness audit, and this site allows it explicitly in robots.txt.
By David Selakovic · Published 2026-10-16 · Revised 2026-10-16 · Reviewed every 180 days