Crawl budget: what it is, how to check it and how to optimize it
Crawl budget is the number of URLs on a site that a search crawler can and wants to crawl in a given time. Google combines two parts: the crawl capacity limit (how much load your server can take) and crawl demand (how much Google needs to crawl your pages). The concept is practical: the larger the site and the slower the server, the more noticeable it is that the crawler does not reach everything; sites with a few hundred pages usually have nothing to worry about.
How Google defines crawl budget
Google Search Central documentation describes crawl budget as the set of URLs that Google can and wants to crawl. It combines two things:
- Crawl capacity limit — how many parallel connections Googlebot can keep open, and with what delay, without overloading your server. It goes up when the site responds quickly and down when it slows, returns 5xx errors or sends rate-limiting signals such as HTTP 429.
- Crawl demand — how much Google wants to crawl the site at all. It depends on the site's size, update frequency, page quality and relevance compared with other sites.
The key consequence: even if your server could handle more, Google will not crawl more than demand calls for. A faster server helps, but low-value and duplicate URLs reduce demand more than speed can make up for.
What It Depends On
- Response speed and stability. If the server is slow or returns errors, the bot reduces its pace to avoid harming the site.
- Demand for crawling. Popular and frequently updated pages are crawled more often.
- Quality of structure. Junk addresses waste budget instead of useful pages.
Who needs to worry about crawl budget
Google's crawl budget guide is written for three kinds of sites:
- large sites with 1 million+ unique pages whose content changes moderately often (about once a week);
- medium or larger sites with 10,000+ unique pages whose content changes very rapidly (daily);
- sites with a large share of URLs in Search Console marked "Discovered – currently not indexed".
If new pages on your site are crawled the same day they are published, Google says you do not need the guide. For a typical service site or a blog with a few hundred pages, slow indexing is rarely about crawl budget; quality, duplicates or missing internal links are the usual cause — see orphan pages.
Where the Budget Leaks
| Source of Leak | What to Do |
|---|---|
| Endless filters and sorts | Block parameters from crawling, keep canonical addresses |
| Internal site search | Block results pages |
| Redirect chains | Reduce to one step — see redirect check |
| Broken links | Fix — how to find |
| Duplicates and technical copies | Consolidate with canonical addresses |
What Really Speeds Up Crawling
- Fast server response. The bot adjusts its pace to the load: the faster the site responds, the more pages it can crawl.
- Clear structure. A page three clicks from the homepage is found faster than one hidden deep.
- Up-to-date sitemap and a link to it in robots.txt.
- Regular updates. A site that changes is crawled more often.
The Crawl Stats report in Search Console
You will find it under Settings → Crawl stats. It shows how Googlebot crawled the site over the last 90 days: total crawl requests, total download size, average response time and host status — whether there were problems fetching robots.txt, resolving DNS or connecting to the server. Below the charts, requests are broken down by response code, file type, purpose (discovery of new URLs or refresh of known ones) and Googlebot type.
| What to look at | Warning sign |
|---|---|
| Average response time | Rising while crawl requests fall — the server is limiting crawling |
| Host status | Errors fetching robots.txt or connecting to the server |
| By response | A noticeable share of 3xx, 404 and 5xx instead of 200 |
| By purpose | Almost no discovery although you publish new pages regularly |
| By file type | Resources (scripts, images, JSON) take most of the requests |
Server log analysis: what only logs show
Crawl Stats gives a summary and samples; server logs show every request — exactly which URLs Googlebot fetched, how often and what it got back. A couple of commands for an nginx or Apache log in the standard format:
# response codes Googlebot received
grep -i "googlebot" access.log | awk '{print $9}' | sort | uniq -c | sort -rn
# most frequently crawled URLs
grep -i "googlebot" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -50User agents are easy to fake, so verify the requests really come from Google: a reverse DNS lookup of the IP should return a hostname in googlebot.com or google.com, and a forward lookup of that hostname should return the same IP. More in server log analysis for SEO.
How to Understand That the Budget Is Being Wasted
- Check the crawl statistics in Search Console: how many pages per day and with what response codes.
- Compare the number of crawled addresses with the number of useful pages on the site.
- Check what share of the crawl goes to redirects and errors.
- If useful pages are crawled once a month, while junk pages are crawled daily — focus on structure, not links.
How to optimize crawl budget step by step
- Consolidate duplicates with canonical URLs or redirects for parameter, www/non-www and trailing-slash variants — see canonical URL.
- Block in robots.txt what should never be crawled: endless filters, sort orders, internal search. Do not use noindex for this — Google still requests the page. To avoid blocking too much, see checking robots.txt.
- Return 404 or 410 for removed pages and get rid of soft 404s — placeholder pages that return 200. According to Google, soft 404s keep being crawled and waste budget.
- Remove long redirect chains and internal links to redirecting URLs.
- Keep your sitemap current: only canonical URLs that return 200, with
<lastmod>for updated pages. Check it with the sitemap analyzer. - Support 304 Not Modified for conditional requests, so Google can reuse its copy of unchanged pages instead of downloading them again.
- Speed up the server and watch 5xx and 429 responses — they lower the crawl capacity limit directly.
- Compare Crawl Stats a few weeks after the changes: the share of 200 responses and of discovery should grow.
Crawl budget myths
- "noindex saves crawl budget." No: Google requests the page, sees noindex and only then drops it. A robots.txt block saves crawling — but then the crawler never sees the noindex. The difference is explained in noindex, nofollow and robots.txt.
- "Crawl-delay in robots.txt will slow Googlebot down." Google does not support that rule; it adjusts its rate based on how your server responds.
- "More crawling means higher rankings." Google has said that an increased crawl rate does not by itself lead to better positions. Crawling is a precondition for indexing, not a ranking factor.
- "Small sites must optimize crawl budget." A site with a few hundred pages gets crawled in full anyway.
- "AI bots use up Google's crawl budget." Crawl budget is about Google's crawlers. But every bot adds server load, and a slow server lowers Googlebot's capacity limit. If third-party bots load your site noticeably, manage them in robots.txt — see how to block AI bots in robots.txt.