Crawl Budget, Server Log Analysis and Verifying Googlebot

Version note: The code in this post was built and run with .NET 10 (ASP.NET Core 10 W3C (World Wide Web Consortium) logging, System.Net.IPNetwork); crawl budget and Googlebot verification details are taken from Google's crawling documentation as of November 2026.

Guess First

Bookstore's catalog has a few thousand books. A developer opens the server logs and sees two things: most requests with a "Googlebot" user agent go to ?genre=...&sort=...&page=... combinations, and a "Googlebot" from an unknown IP (Internet Protocol) is pulling every book page at dozens of requests per second. The team wants to open a sprint because "our crawl budget is being wasted".

Two questions. Is crawl budget really a problem for a site with a few thousand pages? And is that fast "Googlebot" in the log really Google; how do you tell?

In the earlier posts of the series we saw ways to tell Google what we want. In this post we look the other way: we read from server logs what Google, and others pretending to be Google, actually do on the site.

Whose Problem Is Crawl Budget?

Google's crawl budget guide says up front who it's written for: sites with more than a million unique pages that change about once a week, sites with more than ten thousand pages that change daily, and sites where a large part of the URLs (Uniform Resource Locators) stay in the "Discovered - currently not indexed" state in Search Console. Google adds that these numbers are rough estimates, not exact thresholds. The answer to the first opening question: Bookstore, with a few thousand pages, isn't in this group. In Google's words, a site whose pages get crawled the day they're published doesn't need to read this guide.

The concepts are still useful, because the things that make crawling worse are the same on small sites. Crawl budget is the intersection of two things. The crawl capacity limit (also called hostload) is how many connections Googlebot can keep open at the same time without overloading the server: it rises as the server responds quickly and consistently, and drops as it slows down or returns 5xx and 429. Crawl demand is Google's interest in the URLs: it's determined by the site's size, update frequency, page quality and popularity. Even with capacity available, Google doesn't crawl if demand is low; even with high demand, crawling is throttled if the server is slow.

What Spends Crawling and What Saves It?

The guide's recommendations read like a summary of the earlier posts of the series:

• Consolidate duplicate content with canonicals and redirects; let crawling go to unique pages.
• Block unimportant URLs (infinite scroll copies, different orders of the same list) with robots.txt.
• Return 404 or 410 for permanently removed pages; blocked URLs stay in the crawl queue longer.
• Clean up soft 404s; they get crawled again and again.
• Keep the sitemap up to date and give an accurate lastmod.
• Avoid long redirect chains.
• Serve pages fast and support the 304 (Not Modified) response, so unchanged pages aren't downloaded again.

What doesn't work is written down plainly too. noindex doesn't reduce crawling; Google still has to request the page. Temporarily closing a section with robots.txt to "shift the budget elsewhere" doesn't work; unless Google is already hitting the capacity limit, it doesn't give the freed budget to other pages. crawl-delay in robots.txt isn't supported by Google either.

Server Logs: The W3C Format and Producing It in ASP.NET Core

Search Console gives a summary from Google's point of view; a server log shows every request. IIS (Internet Information Services) writes its logs in the W3C extended log format. ASP.NET Core's built-in W3C logger produces the same format; for an application running Kestrel directly or behind Nginx, you can use the same tools as for IIS logs:

using Microsoft.AspNetCore.HttpLogging;

WebApplicationBuilder builder = WebApplication.CreateBuilder(args);
// W3C extended log format: the same layout IIS writes
builder.Services.AddW3CLogging(o =>
{
    o.LoggingFields = W3CLoggingFields.Date | W3CLoggingFields.Time | W3CLoggingFields.ClientIpAddress
        | W3CLoggingFields.Method | W3CLoggingFields.UriStem | W3CLoggingFields.UriQuery
        | W3CLoggingFields.ProtocolStatus | W3CLoggingFields.TimeTaken | W3CLoggingFields.UserAgent;
    o.LogDirectory = Path.Combine(builder.Environment.ContentRootPath, "logs");
    o.FileName = "access";
    o.FlushInterval = TimeSpan.FromSeconds(1);
});

WebApplication app = builder.Build();
app.UseW3CLogging();

I sent 70 requests to this Bookstore demo application with curl: the published user agent strings of Googlebot, bingbot and GPTBot plus an ordinary browser; book pages, nonexistent slugs, old URLs and 36 different filter combinations. All of these requests came from my machine; so every "bot" in the log is actually fake. The start of the log file:

#Version: 1.0
#Start-Date: 2026-11-08 09:56:40
#Fields: date time c-ip cs-method cs-uri-stem cs-uri-query sc-status time-taken cs(User-Agent)
2026-11-08 09:56:39 ::1 GET /robots.txt - 200 2.4844 Mozilla/5.0+(Linux;+Android+6.0.1;+Nexus+5X+Build/MMB29P)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/156.0.0.0+Mobile+Safari/537.36+(compatible;+Googlebot/2.1;++http://www.google.com/bot.html)
2026-11-08 09:56:39 ::1 GET /robots.txt - 200 0.1036 Mozilla/5.0+(compatible;+bingbot/2.0;++http://www.bing.com/bingbot.htm)

The format is simple: the #Fields: line gives the column order, values are separated by spaces, and spaces inside the user agent are encoded as +. In IIS, which fields are written is chosen in the site settings; since the order can change, the parser must read the column names from #Fields:.

A Console Tool That Extracts Bot Traffic

public static class W3CLog
{
    // reads IIS / ASP.NET Core W3C logs; the "#Fields:" line tells us the column order
    public static IEnumerable<LogEntry> Read(string file)
    {
        string[] fields = [];
        foreach (string line in File.ReadLines(file))
        {
            if (line.StartsWith("#Fields:", StringComparison.Ordinal))
            {
                fields = line["#Fields:".Length..].Split(' ', StringSplitOptions.RemoveEmptyEntries);
                continue;
            }
            if (line.StartsWith('#') || line.Length == 0)
                continue;

            string[] v = line.Split(' ');
            string Get(string name) => v[Array.IndexOf(fields, name)];
            string query = Get("cs-uri-query");
            yield return new LogEntry(
                DateTime.Parse($"{Get("date")} {Get("time")}"),
                Get("c-ip"),
                Get("cs-method"),
                Get("cs-uri-stem"),
                query == "-" ? null : query,
                int.Parse(Get("sc-status")),
                Get("cs(User-Agent)").Replace('+', ' '));
        }
    }

    public static string BotName(string userAgent) => userAgent switch
    {
        _ when userAgent.Contains("Googlebot", StringComparison.OrdinalIgnoreCase) => "Googlebot",
        _ when userAgent.Contains("bingbot", StringComparison.OrdinalIgnoreCase) => "bingbot",
        _ when userAgent.Contains("GPTBot", StringComparison.OrdinalIgnoreCase) => "GPTBot",
        _ => "other",
    };
}

File.ReadLines reads the file line by line; a daily IIS log of several GB can be processed without loading it into memory. The program prints the number of requests and the status code breakdown per bot, Googlebot's share of requests with a query string, and the paths it requests most:

$ dotnet run -- ../Bookstore.CrawlSite/logs/access20261108.0000.txt
70 requests in access20261108.0000.txt

requests per bot (by user agent) and status codes
  Googlebot    45   200x40 301x2 404x3
  other        13   200x13
  bingbot       6   200x4 301x2
  GPTBot        6   200x6

Googlebot: 36 of 45 requests (80%) carry a query string
Googlebot top paths
    36  /books
     1  /robots.txt
     1  /books/dune
     1  /books/foundation
     1  /books/neuromancer

On a real site, this table would say: 80 percent of Googlebot's requests go to filter and sort variations of the same list page, and only three requests fall on the book pages themselves. That's the typical picture of faceted navigation eating up crawling. The fix is in the earlier posts of the series: dropping parameters that don't change content, such as sort order, from the canonical, taking filter links that produce endless combinations out of crawling, and blocking those parameters in robots.txt. The 404s are nonexistent slugs (their links should be fixed, or old slugs tied to 301s), the 301s are old URLs (do internal links still point to the old address?). But before deciding anything, you need to check one thing: is this Googlebot really Googlebot?

Verifying Googlebot

A user agent is what a client says about itself; anyone can write "Googlebot" with curl, as I did above. Scrapers and security scanners do this a lot, because many sites are reluctant to block Googlebot. Google gives two verification methods. Manual verification: look at the IP's reverse DNS (Domain Name System) record; the domain must end in googlebot.com, google.com or googleusercontent.com; then resolve that host name with forward DNS, and it must come back to the same IP. With the example IP from Google's documentation:

$ host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
$ host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

The forward check is essential: the owner of an IP block can write the reverse DNS record however they like; someone can set the PTR (pointer) record of their own IP to crawl-1-2-3-4.googlebot.com, but they can't control the DNS of googlebot.com. The second method is comparing with the IP range JSON (JavaScript Object Notation) files Google publishes: common-crawlers.json for common crawlers such as Googlebot, special-crawlers.json for special crawlers such as AdsBot, user-triggered-fetchers.json and user-triggered-fetchers-google.json for user-triggered fetchers; all under developers.google.com/static/crawling/ipranges/. The C# counterpart of both methods:

public sealed class GooglebotVerifier(HttpClient http)
{
    private static readonly string[] GoogleDomains = [".googlebot.com", ".google.com", ".googleusercontent.com"];
    private const string RangesUrl = "https://developers.google.com/static/crawling/ipranges/common-crawlers.json";

    // method 1: reverse DNS -> domain check -> forward DNS must return the same IP
    public static async Task<(bool Ok, string Detail)> VerifyByDnsAsync(IPAddress ip)
    {
        string host;
        try
        {
            host = (await Dns.GetHostEntryAsync(ip)).HostName;
        }
        catch (SocketException ex)
        {
            return (false, $"no reverse DNS ({ex.SocketErrorCode})");
        }
        if (!GoogleDomains.Any(d => host.EndsWith(d, StringComparison.OrdinalIgnoreCase)))
            return (false, $"reverse DNS {host} is not a Google domain");

        IPAddress[] forward = await Dns.GetHostAddressesAsync(host);
        return forward.Contains(ip)
            ? (true, $"{host} -> {string.Join(", ", forward.Select(a => a.ToString()))}")
            : (false, $"{host} does not resolve back to {ip}");
    }

    // method 2: compare with the IP ranges Google publishes (no DNS round trip per request)
    public async Task<IReadOnlyList<IPNetwork>> LoadRangesAsync(CancellationToken ct)
    {
        RangeFile file = (await http.GetFromJsonAsync<RangeFile>(RangesUrl, ct))!;
        return file.Prefixes
            .Select(p => IPNetwork.Parse(p.Ipv4Prefix ?? p.Ipv6Prefix!))
            .ToList();
    }

    public static bool InRanges(IPAddress ip, IReadOnlyList<IPNetwork> ranges) => ranges.Any(r => r.Contains(ip));

    private sealed record RangeFile(DateTime CreationTime, List<Prefix> Prefixes);
    private sealed record Prefix(string? Ipv4Prefix, string? Ipv6Prefix);
}

System.Net.IPNetwork is the CIDR (Classless Inter-Domain Routing) type that came with .NET 8; Contains tells you whether an IP is inside the block, with no extra package needed. The output for the "Googlebots" in the log and three test IPs (Google's example IP, loopback, and an address from the TEST-NET-3 block reserved for documentation):

$ dotnet run -- ../Bookstore.CrawlSite/logs/access20261108.0000.txt 66.249.66.1,127.0.0.1,203.0.113.7
# same tables as above; then:
who claims to be Googlebot?
  ::1             FAKE  reverse DNS localhost is not a Google domain

317 prefixes in common-crawlers.json
  66.249.66.1     dns=VERIFIED ranges=True  crawl-66-249-66-1.googlebot.com -> 66.249.66.1
  127.0.0.1       dns=FAKE     ranges=False reverse DNS localhost is not a Google domain
  203.0.113.7     dns=FAKE     ranges=False no reverse DNS (HostNotFound)

The answer to the second opening question goes through this check. All 45 "Googlebot" requests in the log come from a single fake source; on a real site, this result would show that the one crawling the filter combinations is a scraper, not Google, and that the crawl budget sprint would solve the wrong problem. The two methods have different uses: DNS verification is right for the odd suspicious IP, but means two DNS queries per request; the IP range check suits bulk log analysis and real-time decisions better (such as the rate limiter granting Googlebot a wider limit). Downloading the JSON files once a day and keeping them in memory is enough; the file has a creationTime field.

The Crawl Stats Report

The Google-side counterpart of the log is the Crawl Stats report in the settings section of Search Console. Total crawl requests, total download size and average response time as time series, with breakdowns below by response code, file type, crawl purpose (discovery of new URLs or refresh of known ones) and Googlebot type. A host status section also shows robots.txt fetching, DNS resolution and server connectivity. Reading the report side by side with your log analysis means comparing only verified Googlebot requests: if the "Googlebot" count in the log is far higher than in the report, fake bots make up the difference.

From mertozen.com's Point of View

mertozen.com is a site far below Google's crawl budget thresholds; Googlebot failing to crawl pages in time isn't an expected problem. Still, two points are worth watching in the logs. First, the combinations of category, series, tag, sort and page parameters in the blog list; since the site's robots.txt has no Disallow at all, all of these combinations are crawlable, and their indexing is already under control thanks to their canonicals, but they do get crawled. Second, the /en/ move on 7 November: the old ?lang= URLs now return 301, and Google will crawl them again for a while. Requests containing lang= decreasing over time in the log is the most direct sign that Google has processed the new structure.

Since the site runs on IIS, its logs are already in W3C format; the tool above works without any change. The location and retention period of Plesk's log files are hosting settings; for analysis, the logs need to be pulled from the server regularly, and since they contain personal data (IP addresses), the retention policy must be followed.

Production Notes

Make sure logs are written with the IP the application sees, not the reverse proxy's. Behind Nginx or a CDN (Content Delivery Network), the c-ip column shows the proxy's IP; no request can pass verification. Setting up forwarded headers correctly is covered in the Web Servers and Hosting series post "ASP.NET Core Behind a Reverse Proxy: Forwarded Headers, KnownIPNetworks and the Real Client IP".

Do the verification in log analysis, not synchronously on every request. Two DNS queries in the request path mean latency and load on the DNS server; keep the IP range list in memory.

Be careful with the decision to block fake Googlebots. Returning 403 to a client whose user agent is Googlebot and that fails verification makes sense; but stay on the safe side so you don't block the real Googlebot when the verification infrastructure (DNS, JSON download) breaks: if verification can't be done, don't block.

Remember that timestamps in the logs are UTC (Coordinated Universal Time); the W3C logger and IIS write UTC by default. When comparing with Search Console's reports, the time zone difference can shift daily totals.

Common Pitfalls

Spending time on crawl budget optimization on a small site. Google's own guide is for large sites; at a few thousand pages, the real problems are content quality and indexability.

Trusting the user agent. Not every request saying "Googlebot" is Google; analysis and decisions should be made with verified IPs.

Looking only at reverse DNS. The PTR record can be fake; check that forward DNS comes back to the same IP.

Trying to reduce crawling with noindex. Google still requests the page; the way to reduce crawling is not producing the URL at all, or robots.txt.

Blocking permanently removed pages with robots.txt. Blocked URLs stay in the crawl queue longer; return 404 or 410.

A Small Experiment

★☆☆ Count the requests containing "Googlebot" in one day's access log of your own site, deduplicate their IPs, and verify the first five by hand with host.

★★☆ Add a "verified Googlebot only" filter to the analysis tool: for requests verified with the IP range list, print the status breakdown and query string share separately. Compare the shares of fake and real Googlebots.

★★★ Turn GooglebotVerifier into an ASP.NET Core service: have it refresh the IP ranges once a day with a BackgroundService, open a separate rate limiter partition for verified Googlebot, and log requests whose user agent is Googlebot but can't be verified with a separate counter. Use it to complete the rate limiter exercise from the sixth post of the series.

Crawling and log analysis are the tools for watching daily how Google sees the site. The moment they're needed most is a site migration where every URL changes. In the last post of the series we'll set up URL mapping, the 301 plan, Search Console's Change of Address tool and a script that verifies every old URL after the migration, through mertozen.com's /en/ move.