Insights · Checklist
AI crawl readiness checklist
An engine cannot cite a page its crawler could not fetch or could not read. This checklist confirms access, rendering and indexing for every named crawler, and ends with the log review that proves they arrive. Run it before any content program begins and again after every hosting, security or platform change.
How to use it
Before the first step.
- Assign a technical owner with access to hosting, CDN or security settings and server logs.
- List the crawlers that matter: Googlebot, Bingbot, GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and any sector-specific ones.
- Keep the evidence from each step; it becomes the baseline for the next run.
Steps
Each one owned and evidenced.
| # | Step | Owner | Evidence |
|---|---|---|---|
| 01 | List every crawler the organization wants to allow and decide explicitly about training-only crawlers. | Marketing lead with security | Crawler decision record |
| 02 | Fetch the home page and three deep pages as each named crawler and compare the response to a browser view. | Technical owner | Fetch transcripts per crawler |
| 03 | Confirm every word of main content is present in the HTML response with scripts disabled. | Technical owner | Text diff, scripts on versus off |
| 04 | Review robots.txt for disallow rules affecting named crawlers, including managed or host-injected rules. | Technical owner | robots.txt copy with annotations |
| 05 | Review meta robots and X-Robots-Tag headers on templates for noindex or nosnippet directives. | Technical owner | Header and meta export per template |
| 06 | Review CDN, firewall and bot-protection settings for blocks, challenges or rate limits on named crawlers. | Technical owner with security | Settings screenshots, dated |
| 07 | Confirm sitemaps are valid, split by section, list only indexable pages and carry honest modification dates. | Technical owner | Sitemap validation report |
| 08 | Confirm canonical tags, redirects and hreflang agree and that no redirect chains exceed one hop. | Technical owner | Crawl report of canonicals and redirects |
| 09 | Check response times and error rates for crawler requests against a budget. | Technical owner | Log summary by crawler |
| 10 | Review server or CDN logs for each named crawler: arrivals, frequency, status codes, pages fetched. | Technical owner | Log review table |
| 11 | Submit sitemaps and confirm indexing status in Search Console and Bing Webmaster Tools. | Marketing lead | Coverage reports |
| 12 | Schedule the next run and define the triggers (platform, hosting, security or template changes). | Marketing lead | Calendar entry and trigger list |
By David Selakovic · Published 2026-10-16 · Revised 2026-10-16 · Reviewed every 180 days