Crawl Budget: What It Is and When It Matters
How many pages a bot is ready to crawl and what it depends on, where the budget leaks on large sites, how to see it in Search Console, and why small sites don't need to worry.
Crawl budget is the number of pages on a site that a search bot is willing to crawl in a given time. The concept is practical: the larger the site and the slower the server, the more noticeable it is that the bot does not reach everything.
What It Depends On
- Response speed and stability. If the server is slow or returns errors, the bot reduces its pace to avoid harming the site.
- Demand for crawling. Popular and frequently updated pages are crawled more often.
- Quality of structure. Junk addresses waste budget instead of useful pages.
Where the Budget Leaks
| Source of Leak | What to Do |
|---|---|
| Endless filters and sorts | Block parameters from crawling, keep canonical addresses |
| Internal site search | Block results pages |
| Redirect chains | Reduce to one step — see redirect check |
| Broken links | Fix — how to find |
| Duplicates and technical copies | Consolidate with canonical addresses |
For a small site, this is not a problem
As long as there are a few hundred pages, the bot crawls everything without optimization. Crawl budget becomes a real issue on large catalogs, marketplaces, and sites with address generation through filters.What Really Speeds Up Crawling
- Fast server response. The bot adjusts its pace to the load: the faster the site responds, the more pages it can crawl.
- Clear structure. A page three clicks from the homepage is found faster than one hidden deep.
- Up-to-date sitemap and a link to it in robots.txt.
- Regular updates. A site that changes is crawled more often.
How to Understand That the Budget Is Being Wasted
- Check the crawl statistics in Search Console: how many pages per day and with what response codes.
- Compare the number of crawled addresses with the number of useful pages on the site.
- Check what share of the crawl goes to redirects and errors.
- If useful pages are crawled once a month, while junk pages are crawled daily — focus on structure, not links.
FAQ
When does crawl budget become a problem?
On large sites with address generation through filters. As long as there are a few hundred pages, the bot crawls everything without optimization.
What consumes the budget most often?
Endless filters and sorts, internal search pages, redirect chains, broken links, and technical duplicates.
How to see the problem?
In the crawl statistics of Search Console: how many pages are crawled per day and what share goes to redirects and errors.
Was this article helpful?
Thank you! We will consider this in future updates.
Try it on your project
Everything described in the article is available in the dashboard — the registration bonus is already in your balance.
Open dashboard