Crawl Budget: What Actually Works
How many URLs a search engine can and wants to crawl on a site in a given period, determined by the server's capacity and how much demand there is for the content.
Samir Haddad
The key figures
- Who it matters for
- Google says mainly large sites or sites with rapidly changing content
- Crawl capacity
- how much crawling the server can handle without problems
- Crawl demand
- how much Google wants to crawl based on popularity and freshness
- Crawl Stats report
- Search Console shows requests, response times and file types
Why this is worth getting right
For large sites, wasted crawling on duplicates, parameters and low-value URLs can delay the discovery of important new or updated pages.
Do this, not that
Do
- Remove or block crawl traps such as endless filters
- Keep server response times fast
- Return correct status codes for removed pages
- Keep sitemaps limited to canonical, indexable URLs
- Review the Crawl Stats report regularly
Don’t
- Worrying about crawl budget on small sites
- Infinite calendar or filter URLs
- Soft 404 pages returning 200
- Long redirect chains
When to bring in help
Our advice Bring in help when a large site has many discovered but not indexed pages, or when crawl activity drops sharply.
Where this comes from
- Google Search Central — Large site owner's guide to managing your crawl budget
- Google Search Central — Crawl Stats report
The figures and practices above come from the sources listed.
Working on something like this?
We take on SEO Services work for teams who want it done once, properly. Tell us what you are building and we will tell you honestly whether we are the right studio for it. Start a project.
Where to go next
Spotted something wrong? Report an error on this page. We correct on the page and say what changed.
