SEO GLOSSARY
What is crawl budget?
Crawl budget is the set of URLs on a site that Google can and wants to crawl in a given time. It matters for large or fast-changing sites, because pages Google does not crawl cannot be indexed, so wasted crawls on duplicates and errors delay new and updated pages.
- Free for 7 days
- Every tool unlocked
- No card needed
Updated 23 October 2026
What is crawl budget in SEO, and how does Google set it?
Google sets crawl budget from two things: crawl capacity limit (how much crawling your server can handle) and crawl demand (how much Google wants to crawl your URLs).
Crawl capacity limit rises when your site responds quickly and steadily, and falls when it slows down or returns server errors (5xx) or rate-limit signals (429). Crawl demand depends on your site's size, update frequency, page quality and relevance; Google names perceived inventory, popularity and staleness as the factors you can influence.
Google treats each hostname as a separate site, so www.example.com and shop.example.com have separate crawl budgets. And crawling is not indexing: every crawled page is still evaluated before it can enter the index.
Does my site need to worry about crawl budget?
Most sites do not. Google says that if your pages are usually crawled the same day they are published, keeping your sitemap current and checking the Page Indexing report is enough.
Google stresses these are rough estimates, not exact thresholds. Big ecommerce catalogues with faceted filters are the classic case.
| Site profile (Google's rough guide) | Care about crawl budget? |
|---|---|
| 1 million+ unique pages, content changes about weekly | Yes |
| 10,000+ unique pages, content changes daily | Yes |
| Many URLs marked Discovered, currently not indexed | Yes |
| A typical small business or blog | Rarely: fix quality and links first |
How to check and optimize crawl budget, with an example
Read Search Console's Crawl Stats report for requests, response codes and host problems, then remove the URLs wasting crawls.
Example: a store with filter URLs like ?color=red&size=9 can generate thousands of near-identical pages. The technical SEO audit crawls your pages in 30 seconds to 2.5 minutes and lists issues by severity, and the Googlebot simulator shows the canonical, links and status Google reads on any URL.
- Consolidate duplicate content so Google crawls unique pages, not unique URLs.
- Block truly unneeded URLs (sorted or filtered duplicates) in robots.txt; do not rely on noindex, which Google still requests.
- Return 404 or 410 for permanently removed pages and eliminate soft 404s.
- Keep sitemaps up to date, with lastmod for changed pages.
- Avoid long redirect chains and speed up server responses.
QUESTIONS PEOPLE ASK
Frequently asked questions
Still unsure? Email support@digimetrics.ai and a person will reply.
How do I increase crawl budget?
Google names two ways: add server resources if your site cannot handle more crawling, and improve content quality, popularity and uniqueness, which raises crawl demand.
Can I set my crawl budget?
No. Google sets it automatically. You influence it by cutting wasted URLs, keeping your server fast and healthy, and publishing pages worth recrawling.
Where is crawl budget shown in Google Search Console?
Search Console has no single crawl budget number. The Crawl Stats report, under Property settings, shows total requests, response codes and host availability over time.
Does noindex save crawl budget?
No. Google still requests the page, then drops it when it sees noindex. Use robots.txt for URLs you never want crawled.
Find the URLs wasting Google's time.
Crawl your site and get every issue ranked by severity. Free for 7 days, no card.