Glossary · C
Crawl budget
Crawl budget is the number of pages a crawler will fetch from a site in a given period, determined by the site’s capacity to serve requests and the crawler’s assessment of what is worth fetching.
Extended definition
In more detail.
Large or slow sites can exhaust their budget on duplicates, redirects and low-value pages, leaving important pages unfetched or stale. Fast responses, clean canonicals, accurate sitemaps and few errors raise the effective budget. AI crawlers have budgets of their own, often smaller, which is why a site that wants to be cited should make every fetch count: finished HTML, no redirect chains, no content hidden behind scripts.
In our work
Where it appears.
Sitemaps split by section with honest modification dates and a clean redirect map are part of llms.txt and machine-readable indexes.
By David Selakovic · Published 2026-10-16 · Revised 2026-10-16 · Reviewed every 180 days