If Googlebot is spending its time on the wrong URLs, the fix is almost never “increase crawl budget”. It is to stop generating URLs that were never worth crawling.
Crawl budget gets discussed as though it were a resource you can request more of. It is not. It is an output, the result of how much Google trusts a host and how much useful, distinct content it believes that host produces. Treating it as an input leads teams to optimise the wrong end of the system.
What the logs usually show
On most large South African e-commerce sites we audit, between 40% and 70% of Googlebot requests land on URLs nobody would ever want ranked: filter permutations, session-tagged variants, paginated tails twelve pages deep, and internal search results that were never meant to be crawlable.
None of that is a budget problem. It is an architecture problem that presents as a budget problem.
The diagnostic sequence
Before touching robots.txt, work through this order. Each step removes noise that would otherwise distort the next.
- Segment log requests by template, not by URL. You are looking for the template generating disproportionate hits relative to the revenue it produces.
- Separate genuine Googlebot from spoofed agents by reverse DNS. Roughly one in eight “Googlebot” hits in a raw log is not.
- Compare crawl frequency against last-modified dates. Templates crawled far more often than they change signal an instability Google is trying to resolve.
- Only then look at what is blocked, canonicalised or noindexed, and check that those directives are actually being honoured.
Why robots.txt is usually the wrong first move
Blocking a parameter in robots.txt stops the crawl but not the discovery. The URLs remain in Google’s frontier, still consuming scheduling attention, and now with no way for Google to see the canonical tag that would have resolved them. You have hidden the symptom and removed the cure.
The question is not “how do we get more crawl budget”. It is “which of these URLs should never have existed”.
What actually works
Stop linking to combinations you do not want indexed. Move faceted navigation behind an allowlist of the four or five filter combinations with genuine search demand, and render those server-side. Return 404 for the rest rather than 200 with a canonical. A canonical is a suggestion; a status code is not.
On a large faceted catalogue this one change can take the crawlable URL set from hundreds of thousands of combinations down to a few thousand, without a single robots.txt directive being added. The crawl that was being spent on noise goes back to the templates that earn revenue.
When crawl budget genuinely is the constraint
It occasionally is, on sites above a few million genuinely distinct, genuinely valuable URLs, or on hosts with server response times slow enough that Google throttles itself to avoid causing harm. In the second case the fix is infrastructure, not SEO. Check your time-to-first-byte distribution at the 95th percentile before concluding anything else.