Crawl budget.
Crawl budget is the number of requests a search engine allocates to fetching from a site within a given period.
A crawler does not fetch a whole site on every visit; it arrives with a limited budget and decides for itself where to spend it. For small sites this is usually not a problem; on sites of thousands of pages, where the budget goes determines visibility directly.
The things that waste the budget are well known: redirect chains, addresses that return errors, pages generating endless filter combinations, and the same content repeated at different addresses. Each consumes a request that could have gone to real content.
How the budget is distributed is read from server logs, and the most valuable thing to read is this: which pages does the crawler visit, and how often? Frequently fetched pages are the ones the engine believes represent the site — and that list usually does not match the site’s own list of priorities.
The reason for the divergence is usually the internal link structure: pages that receive many links are crawled often, pages that receive none from the body are crawled rarely or never. A sitemap is a discovery channel but carries no priority signal; priority is read from the link structure.
The same logic applies to AI crawlers, and there the effect is harsher: a page that is never fetched does not exist for an answer engine, however published it may be.
A budget problem is rarely real on small sites; on a site of a few hundred pages the crawler gets to everything anyway. The problem starts at thousands of pages, especially in systems that generate filter and search combinations.
The most effective fix is to block the addresses that consume the budget at the source: close the parameters generating endless combinations to crawling, reduce redirect chains to a single hop, and clear the addresses returning errors. All three make more difference than editing the sitemap.
- Redirect chains — every hop consumes a request; reduce them to one.
- Addresses returning errors — every page crawled and not found is wasted budget.
- Endless combinations — filter and sort parameters generate unlimited addresses.
- Repeated content — the same page being reachable at different addresses.
- Weak internal linking — pages that receive no links are crawled rarely or never.
The fifth item is the most overlooked, and it was measured on this site: fourteen pages were found that received no link from the body. A sitemap provides discovery but carries no priority signal — priority is read from the link structure.