Google Says Speed Wins Crawl Budget

Google has recently reaffirmed that the one million page crawl budget threshold remains unchanged, even after five years. However, the tech giant highlighted that the efficiency of your website’s database plays a far greater role in crawl performance than simply the number of pages your site has.

While it’s commonly believed that sites with fewer than a million pages typically don’t need to worry about crawl budget, Google clarified that this threshold is more of a guideline than a strict rule. The figure is described as “probably” one million pages, meaning it’s not set in stone and can vary based on other site factors.

Crucially, Google stressed that slow-performing databases can impact crawlability, regardless of how many pages a website contains. In other words, even smaller sites may experience crawl issues if their backend infrastructure is sluggish or unresponsive.

In terms of SEO, this means webmasters and developers should pay close attention to database performance. Sites that load slowly due to database bottlenecks could see reduced crawling activity, which in turn may affect indexing and visibility in search results.

Rather than focusing solely on page count, it’s now more important than ever to ensure your website’s infrastructure is optimised. This includes fast database queries, efficient server responses, and smooth page delivery.

Ultimately, Google’s message is clear: technical performance is key. Even if your site is well below the million-page mark, poor database speed could still hinder your search presence.

Keeping databases lean and fast is now just as critical as good content and link-building in the broader SEO strategy.

If you’ve been solely monitoring how many pages your site has, it may be time to shift your attention to the performance side of things.

This reminder from Google is a useful prompt to audit your site’s backend and make improvements where necessary—ensuring better crawl efficiency and search performance moving forward.

The Million-Page Rule Stays The Same

In a recent episode of the Search Off the Record podcast, Gary Illyes reinforced Google’s long-standing stance on crawl budget thresholds. This came after his co-host, Martin Splitt, raised a question about the current limits.

Illyes responded by saying, “I would say 1 million is okay probably.” That small word — “probably” — is worth noting. While Google often refers to the one million page mark as a general rule of thumb, it’s not a fixed limit. The inclusion of the new database efficiency factor means that even websites with far fewer pages could encounter crawl issues if their systems aren’t well optimised.

What’s particularly interesting is that this threshold has remained unchanged since 2020. In that time, the web has evolved dramatically, with more websites relying on JavaScript, dynamic content, and increasingly complex structures. Despite these changes, Google’s crawl budget guidance has stayed the same.

This highlights the growing importance of backend efficiency. Rather than focusing purely on how many pages a site has, webmasters should ensure their infrastructure is capable of handling Google’s crawling processes effectively.

Your Database Speed Is What Matters

The key takeaway from Gary Illyes’ comments is this: slow database performance can be a bigger obstacle to effective crawling than the sheer number of pages on a website.

Illyes pointed out, “If you are making expensive database calls, that’s going to cost the server a lot.” This means that the speed at which your backend operates plays a crucial role in how well Googlebot can access your content.

To illustrate, a website with 500,000 pages that rely on slow, resource-heavy database queries may experience more crawling issues than a site with 2 million fast-loading static pages.

So what does this mean for website owners and SEO professionals? Rather than focusing solely on page counts, it’s essential to review how well your database is performing. Websites that use dynamic features, serve real-time content, or involve complex backend operations need to pay close attention to loading speeds and server efficiency.

Improving performance at the infrastructure level could make a significant difference to how frequently and thoroughly your site is crawled by search engines.

The Real Resource Hog: Indexing, Not Crawling

Gary Illyes expressed a view that goes against what many in the SEO community often assume.

He stated, “It’s not crawling that is eating up the resources, it’s indexing and potentially serving or what you are doing with the data when you are processing that data.”

This insight shifts the focus. If the act of crawling doesn’t demand much in terms of server resources, then restricting Googlebot may not be as effective as some believe. Instead, the emphasis should be on ensuring that your content is straightforward for Google to process once it has already been crawled.

In other words, optimising how your site handles data post-crawl—especially during indexing and serving—might be more impactful than simply managing crawl behaviour.

Why The Threshold Remains Stable

Google has been working towards lowering its overall crawling activity, but according to Gary Illyes, achieving that goal isn’t as straightforward as it might seem.

He pointed out the difficulty by saying, “You saved seven bytes from each request that you make and then this new product will add back eight.”

This highlights the ongoing balance between making systems more efficient and introducing new features. As improvements are made in one area, additional demands may offset those gains. It helps to explain why the crawl budget threshold has remained relatively unchanged. Despite advancements in Google’s infrastructure, the underlying principles around crawl budget and when it becomes a concern have largely stayed the same.

What You Should Do Now

For websites with fewer than 1 million pages, there’s no immediate need to worry about crawl budget. You should carry on with your existing SEO approach, ensuring your site continues to offer high-quality content and a smooth user experience.

For larger websites, improving the efficiency of your database should now become a top priority. Pay close attention to the following areas: how long your queries take to execute, whether your caching systems are effective, and how quickly your dynamic content is generated.

Across all websites, it may be time to shift your focus away from limiting crawling and instead concentrate on making indexing easier. Since crawling itself isn’t the main strain on Google’s resources, helping Google to better process and understand your content should yield greater benefits.

Some key technical areas worth reviewing include your database’s performance, how fast your server responds, how well your content is delivered, and whether your caching is properly configured. These factors all play a role in ensuring your site remains accessible, efficient, and competitive in search.