Search Console Crawl Stats: a practical guide to diagnosing crawl budget

The Crawl Stats report is one of the few places in Search Console where you see Googlebot through the lens of your own server: how many requests it made, what it got back, how much it downloaded, and how long it took. It's a report for advanced users, and Google itself warns that if your site has fewer than a thousand pages, you probably don't need to worry about this level of detail. But if you run a large portal, an e-commerce site, or a site struggling with slow recrawl, this is where crawl budget becomes measurable.

Below I walk through the path I follow on a real site to diagnose crawl waste, chart by chart.

Prerequisites and where to click

Crawl Stats only exists for root-level properties: a Domain property (example.com) or a root URL-prefix property (https://example.com). If your property is https://example.com/blog/, forget it, the report won't show up.

The path isn't in the obvious sidebar menu. You open Search Console, click Settings (the gear icon, Property settings), and then Crawl stats. That's the first stumbling block I've seen people run into: looking under the indexing menu and not finding it.

Before interpreting anything, three facts about the data that change how you read it:

• These are actual requested URLs, not canonical ones. Unlike other reports, there's no consolidation by canonical here. If Googlebot requested page1, which redirects to page2 and then page3, you'll see three separate requests, and the redirect response falls under the Other file types file type.
• Only the selected domain counts. A resource hosted elsewhere (an image at google.com/img.png embedded in your page) won't show up. Subdomains under the parent domain do show up, though: under example.com you'll see en.example.com, de.example.com, and so on.
• Coverage is partial. Google acknowledges as a known issue that not every request is counted, so there will be a gap between your server logs and the numbers here. Don't treat this as an accounting source, treat it as a trend.

The three numbers at the top

The header shows Total crawl requests, Total download size, and Average response time. This is where the crossover with server-side Core Web Vitals begins.

Average response time is the aggregate TTFB that Googlebot experienced. If it rose alongside total requests, that's a classic sign of a server under crawl pressure. Total download size matters because every byte Googlebot downloads from a useless resource (a thumbnail, duplicated CSS, a giant XML file) is a byte it didn't spend recrawling content that actually changes.

Useful detail: if Google has already cached a resource used across several pages, it only downloads it once. So a high download figure usually comes from unique pages/resources, not from shared assets.

Host status: the first place I look

Host status summarizes whether Google found availability problems over the last 90 days. Green is what you want. If it's red, there was at least one significant issue in the past week, and it's worth investigating whether it's recurring.

It's split into three categories, each with a chart and a red dotted line marking the problem threshold (for example, DNS failure above 5% of requests on a given day):

• robots.txt fetch: failure rate on requests for the file.
• DNS resolution: when your DNS didn't respond or didn't recognize the host.
• Server connectivity: when the server didn't respond or didn't complete the response.

The behavior of robots.txt deserves attention because it's where I see the most crawl budget evaporate without anyone noticing. Google's rule:

• 200 with a file (valid, invalid, or empty) = success. A syntax error doesn't invalidate the response, Google just ignores the broken rule.
• 403/404/410 = also success (it means "no robots.txt exists," crawl everything).
• 429/5XX = failure. And here's where the problem lies.

When robots.txt returns a 5XX, Google doesn't just fall back to the last valid version right away. For the first 12 hours it stops crawling the site and only keeps requesting the file. From 12 hours to 30 days, it uses the last good robots.txt it has on hand. After 30 days, if the homepage responds, it acts as if there were no robots.txt; if the homepage is also down, it stops crawling. In other words: a robots.txt that returns an intermittent 500 can freeze your crawling for half a day without producing a visible error anywhere else.

Crawl responses: where the waste lives

The Crawl responses table groups status codes by percentage of total requests (not by URL). Under normal circumstances, the overwhelming majority should be 200.

What I hunt for here:

• 301/302/308/307: redirects are "good," but remember that every hop in the chain counts as a request. A chain A → B → C triples the crawl cost of a single destination. Shortening redirect chains is the cheapest crawl budget fix there is.
• 404: not every 404 needs to be fixed, sometimes it's the correct behavior. But a high volume of 404s coming from broken internal links is budget thrown away.
• 5XX (Server error): these trigger availability alerts and should be fixed. If Google is over-crawling and knocking your server over, you can request a lower crawl rate, but first confirm it isn't an actual availability problem.
• robots.txt unreachable: different from a 404. If it shows up here, reread the section above, it's the worst type of response.

Clicking any row opens up example URLs. Google notes that the examples aren't exhaustive, they're a representative sample and may be weighted by day, so don't conclude that a URL wasn't crawled just because it doesn't appear in the list.

File type and Googlebot type: understanding what consumes crawl

The File type table shows the percentage of responses by format (HTML, image, JS, CSS, PDF, JSON, XML, feed, and Other file types, which is where redirects fall). The percentage is of responses, not bytes.

If response time is poor, this is where I find the culprit: is Google requesting lots of small images that should be blocked? Is it fetching resources hosted on a slow third party? Clicking on a type shows you average response time per day and number of requests per day, letting you match slowness spikes for a specific type with overall slowness spikes.

The Googlebot type table reveals who made the requests:

• Smartphone / Desktop: page crawling.
• Page resource load: secondary fetch of images and CSS to render before indexing. If an image is loaded as a page resource, it counts here, not as Image.
• AdsBot: crawls roughly every ~2 weeks; a spike here is usually a new Dynamic Search Ads target.
• StoreBot: product shopping crawler.

This cross-reference is what closes the diagnosis. A spike in requests without a corresponding rise in new traffic is often AdsBot or a page resource, not content. If the spike is AdsBot right after someone launched a campaign, it's not an SEO problem.

Crawl purpose and what to do with the diagnosis

The last piece is Crawl purpose: Discovery (a URL never crawled before) versus Refresh (recrawl of a known page). If you have pages that change fast and aren't being recrawled enough, make sure they're in a sitemap; for slow-changing pages, request a one-off recrawl. If you've uploaded a lot of new content or a sitemap and don't see an increase in discovery crawls, something is blocking discovery.

The thread tying it all together: every request Googlebot spends on a chained redirect, a broken-link 404, a useless resource, or a failed robots.txt attempt is a request it didn't spend discovering and refreshing the content that generates traffic. Crawl Stats is the only report that quantifies this waste before and after a fix. Make the change (shorten redirects, stabilize robots.txt, block a useless resource), wait a few days, and compare the 200 percentage and average response time. It's measurable engineering, not guesswork.