A robots.txt Guide: RFC 9309, Allow/Disallow Rules and What Not to Block

Version note: The code in this post was built and run with .NET 10 (ASP.NET Core 10) and Angular 22.2; the rules are explained based on RFC (Request for Comments) 9309 and Google's robots.txt documentation as of November 2026.

Guess First

Here is Bookstore's robots.txt file. A teammate wanted to "hide the account pages from Googlebot as well" and added a second group at the bottom:

User-agent: *
Disallow: /search
Allow: /search/help
Disallow: /*sort=
Disallow: /*.pdf$

User-agent: Googlebot
Disallow: /account

Three questions. Can Googlebot crawl /search?q=dune? Can Bingbot crawl /Search?q=dune? Can Bingbot crawl /files/catalog.pdf?v=2? We'll see the answers halfway through the post, with the output of a real matcher; write your guesses down.

In the first post of the series we designed our URLs (Uniform Resource Locators). Now we move on to the file that decides which of those URLs get crawled. robots.txt lived as a de facto standard for more than twenty years; in September 2022 it became an IETF (Internet Engineering Task Force) standard under the name REP (Robots Exclusion Protocol) with RFC 9309. Three of its authors are from Google, and Google's robots.txt documentation fills the gaps the RFC leaves with Google's own behavior. We'll read the two together.

Location, Scope and Format

According to RFC 9309, the rules live in a file named /robots.txt, all lowercase, at the root of the site. Google's documentation clarifies the scope: the file only applies to the protocol, host and port it's served from. https://bookstore.example/robots.txt doesn't cover https://api.bookstore.example or http://bookstore.example; every subdomain needs its own file. Crawlers never read a robots.txt placed in a subfolder.

The format is plain text encoded as UTF-8 (Unicode Transformation Format, 8-bit). Lines take the form field: value, and anything after # is a comment. Google recognizes four fields: user-agent, allow, disallow and sitemap. Other fields such as crawl-delay don't exist as far as Google is concerned; Google adjusts its crawl rate itself based on the server's response time and error rate. Field names are case-insensitive, while path values are case-sensitive and must start with /.

The size limit appears in the RFC as "at least 500 kibibytes must be parsed"; Google enforces this limit and ignores anything after 500 KiB. A hand-written file never gets close to that limit, but an auto-generated robots.txt that prints a separate line per product can. If your line count is growing that large, you're using the wrong tool: use path patterns instead of blocking URLs one by one.

Groups and User-Agent Matching

A group starts with one or more User-agent lines and consists of the rules below them. A crawler matches its own product token (Googlebot, Bingbot) against the groups case-insensitively. The RFC 9309 rule is clear: if several groups name the same crawler, their rules are combined; if none matches, the * group applies. The critical point is the flip side of that second sentence: if there is a group carrying the crawler's own name, the * group is completely out of the picture, and the two groups are not combined. Google's documentation says the same: each crawler follows only the single group that matches it most specifically.

That's exactly what happens in the opening file. The moment the teammate opened a separate group for Googlebot, Googlebot stopped seeing Disallow: /search, Disallow: /*sort= and Disallow: /*.pdf$; it now sees only Disallow: /account. A line written as "an extra restriction" removed every previous restriction for Googlebot.

Rule Precedence: The Most Specific Match

Once the group is chosen, among the rules that match the path, the "most specific" one in the RFC's words wins, meaning the rule with the most octets. If an Allow and a Disallow are equally long, Allow is used. Order doesn't matter; a rule at the top of the file doesn't override one at the bottom. There are two special characters: * matches zero or more of any character, and $ marks the end of the URL. Matching is done on the path and query string and is case-sensitive.

To check the guesses, I implemented these RFC 9309 rules in a small C# class. It doesn't cover every corner of a real parser (such as percent-encoding normalization, and it counts length in characters rather than octets), but the group selection and precedence logic are the same:

public bool IsAllowed(string productToken, string path)
{
    string token = productToken.ToLowerInvariant();
    // RFC 9309: all groups naming this token are combined; "*" only if none does
    var own = _groups.Where(g => g.Agents.Contains(token)).ToList();
    var groups = own.Count > 0 ? own : _groups.Where(g => g.Agents.Contains("*")).ToList();
    List<RobotsRule> rules = groups.SelectMany(g => g.Rules).ToList();

    RobotsRule? best = null;
    foreach (RobotsRule rule in rules.Where(r => Matches(r.Path, path)))
    {
        // the longest pattern wins; on a tie, allow wins
        if (best is null || rule.Path.Length > best.Path.Length
            || (rule.Path.Length == best.Path.Length && rule.Allow))
            best = rule;
    }
    return best?.Allow ?? true;
}

private static bool Matches(string pattern, string path)
{
    string regex = "^" + Regex.Escape(pattern).Replace(@"\*", ".*");
    if (regex.EndsWith(@"\$")) regex = regex[..^2] + "$";
    return Regex.IsMatch(path, regex);
}

The parsing side is simple: consecutive User-agent lines join the same group, a User-agent that follows a rule line starts a new group, and an empty Disallow: blocks nothing. Running it with the opening file:

$ dotnet run --no-launch-profile -- check
Googlebot  /search?q=dune                   allowed
Googlebot  /account/orders                  blocked
Bingbot    /search?q=dune                   blocked
Bingbot    /search/help                     allowed
Bingbot    /books?genre=scifi&sort=price    blocked
Bingbot    /books?genre=scifi               allowed
Bingbot    /Search?q=dune                   allowed
Bingbot    /files/catalog.pdf               blocked
Bingbot    /files/catalog.pdf?v=2           allowed

All three answers are the opposite of most people's first guess. Googlebot can enter /search because it has its own group. Bingbot can enter /Search because path matching is case-sensitive and the /search rule doesn't cover /Search; the lowercase normalization from the first post pays off here too. /files/catalog.pdf?v=2 isn't blocked, because $ marks the end of the URL and the query string is part of the URL. The /search/help example is the textbook case of the precedence rule: Allow: /search/help (12 characters) is more specific than Disallow: /search (7 characters).

Don't write your own parser for a real system; the C++ robots.txt parser Google itself uses is open source (the google/robotstxt repository), and the robots.txt report in Search Console shows when Google last fetched your file and what it saw while parsing it. The point of this class isn't to memorize rules but to make the effect of a change testable before it goes live.

Status Codes: The robots.txt Response Itself

A crawler's behavior is determined not only by the file's content but also by the HTTP (HyperText Transfer Protocol) status code it gets when requesting the file. RFC 9309 and Google's documentation together:

The practical meaning of the table: a 404 for robots.txt is harmless, a 500 is not. If your application generates robots.txt from a database query and the endpoint returns 500 when the database goes down, Google stops crawling the site for that period. Generate robots.txt statically or without dependencies. A robots.txt request being redirected to a login page (an authentication middleware catching everything) is another classic accident; if the redirect target is an HTML (HyperText Markup Language) login page, the crawler finds no valid rules there.

Caching: the RFC says crawlers generally shouldn't use a cached copy for more than 24 hours; Google also caches for about 24 hours in general and states that it may lengthen or shorten that period based on max-age in the Cache-Control header. This has a concrete consequence on the Angular side, as we'll see shortly.

Environment-Specific robots.txt in ASP.NET Core

Bookstore's staging environment runs the same code as production. If you put robots.txt in as a static file, staging publishes production's rules too. Generating it from an endpoint makes it easy to vary by environment:

app.MapGet("/robots.txt", (IWebHostEnvironment env, HttpContext context) =>
{
    string body = env.IsProduction()
        ? """
          User-agent: *
          Disallow: /search
          Disallow: /account/
          Disallow: /*?*sort=

          Sitemap: https://bookstore.example/sitemap.xml
          """
        : """
          # non-production environment: do not crawl anything
          User-agent: *
          Disallow: /
          """;
    context.Response.Headers.CacheControl = "public, max-age=3600";
    return Results.Text(body + "\n", "text/plain; charset=utf-8");
});

$ ASPNETCORE_ENVIRONMENT=Production dotnet run --no-launch-profile --urls http://localhost:20103
$ curl -si http://localhost:20103/robots.txt
HTTP/1.1 200 OK
Content-Length: 120
Content-Type: text/plain; charset=utf-8
Cache-Control: public, max-age=3600

User-agent: *
Disallow: /search
Disallow: /account/
Disallow: /*?*sort=

Sitemap: https://bookstore.example/sitemap.xml

$ ASPNETCORE_ENVIRONMENT=Staging dotnet run --no-launch-profile --urls http://localhost:20103
$ curl -si http://localhost:20103/robots.txt
HTTP/1.1 200 OK
Content-Length: 78
Content-Type: text/plain; charset=utf-8
Cache-Control: public, max-age=3600

# non-production environment: do not crawl anything
User-agent: *
Disallow: /

The endpoint deliberately depends on no service, database or configuration read; there's no way it falls into the 5xx row of the table above. The Sitemap line must be an absolute URL, can appear more than once, can point to another host and doesn't belong to any group; it doesn't matter where in the file you put it. The Disallow: / on staging is important but not enough; why it isn't enough is in the pitfalls section.

A Static robots.txt in Angular and a Hidden Cache Header

In an Angular CLI (Command Line Interface) project, every file placed in the public/ folder is copied to the root of the build output. Write public/robots.txt, run ng build, and the file comes out as dist/bookstore-ng/browser/robots.txt, served by express.static in the SSR (Server-Side Rendering) server. The response I got on the first try:

$ curl -si http://localhost:20110/robots.txt
HTTP/1.1 200 OK
Cache-Control: public, max-age=31536000
Content-Type: text/plain; charset=utf-8

User-agent: *
Disallow: /search
Disallow: /account

Sitemap: https://bookstore.example/sitemap.xml

max-age=31536000 means one year. The server.ts generated by ng new --ssr applies maxAge: '1y', which is right for hashed bundle files, to every static file, and robots.txt is one of them. Google says it can adjust its cache lifetime based on max-age, and once a CDN (Content Delivery Network) sits in between, a one-year header can genuinely mean serving a stale file for a year. The fix uses setHeaders:

app.use(
  express.static(browserDistFolder, {
    maxAge: '1y',
    index: false,
    redirect: false,
    setHeaders: (res, path) => {
      // robots.txt and sitemap.xml must not be cached for a year
      if (path.endsWith('robots.txt') || path.endsWith('.xml')) {
        res.setHeader('Cache-Control', 'public, max-age=3600');
      }
    },
  }),
);

$ curl -sI http://localhost:20110/robots.txt | grep -i cache-control
Cache-Control: public, max-age=3600

The downside of a static file is environment separation: if the same build goes to both staging and production, both publish the same robots.txt. Either swap the file during deploy or generate robots.txt with a route on the Express side. The build and deploy flow of SPA (Single-Page Application) projects is in "Angular Build and Deploy: ng build, Environment Files, Budgets and SPA Routing" in the Angular series.

robots.txt Blocks Crawling, Not Indexing

This is the most common misunderstanding. Disallow stops Googlebot from requesting the URL; it doesn't stop the URL from entering the index. A blocked URL that gets descriptive links from other sites can appear in results without its content ever being read, with a note saying no description is available instead of a snippet. On top of that, a noindex you put on a blocked page is never seen, because the crawler never requests the page. If you want a page out of the index, you have to leave it open to crawling and give it noindex; we cover that together with X-Robots-Tag in the next post.

The same logic applies to rendering. When Google renders a page, it requests CSS (Cascading Style Sheets), JavaScript and images separately; if those are blocked by robots.txt, the rendered page is incomplete. A line like Disallow: /assets/ or Disallow: /*.js$ means the content of an SPA becomes completely invisible.

mertozen.com's robots.txt

This site's robots.txt is also generated by an ASP.NET Core endpoint, and in the site's source code it has this structure: first a User-agent: * group with Allow: /, then separate groups for GPTBot, PerplexityBot, ClaudeBot, Google-Extended and CCBot, each again containing only Allow: /, and finally a Sitemap line with an absolute URL. There is no Disallow at all.

Under RFC 9309, the effect of this file today is the same as a file containing only the Sitemap line: nobody is blocked from anything, and Allow: / repeats the permission an empty file already grants. The separate groups do nothing today, but they're a trap for tomorrow: if someone adds Disallow: /araclar/ to the * group one day, the five bots with their own groups won't see that rule; the exact same thing as the Googlebot example in the opening. The groups do carry an intent: the site's GEO (Generative Engine Optimization) notes say it deliberately wants its content to be read by AI (Artificial Intelligence) systems. A less fragile way to keep that intent is to give a bot its own group only when it needs rules different from *. The fact that Google-Extended is not a crawler but a separate usage-control token, and the split between training, search and user-initiated fetches among AI bots, are the topic of "Understanding and Managing AI Bots" in the GEO series. The decision to change anything is the site owner's; here I'm only showing how to read the rules.

Production Notes

Test robots.txt on every deployed change. Putting a matcher like the one above into your CI (Continuous Integration) pipeline and checking a "these URLs must stay crawlable" list (home page, book pages, CSS and JavaScript bundles) on every build prevents a single wrong Disallow from shutting the whole site.

After release, check the robots.txt report in Search Console: it shows when Google last fetched the file, which status code it got and any parsing warnings. A change takes effect not instantly but with a delay as long as the cache lifetime.

Save the file as UTF-8 without a BOM (Byte Order Mark). Google ignores a BOM at the start of the file and invalid lines, but not every crawler's parser is that forgiving; an invisible BOM at the start of the first line can make the first User-agent line unrecognizable to a stricter parser. Check the encoding your editor chooses for new files.

Don't use robots.txt to slow crawling down. Crawl-delay isn't supported by Google; for a temporary overload Google recommends returning 503 or 429, which we'll see in the sixth post of the series.

Common Pitfalls

"Hiding" staging with robots.txt. RFC 9309 says it explicitly: these rules are not a form of access authorization. robots.txt is a public file that even advertises what you're hiding; for a bot that ignores the rules it means nothing. The real protection for staging is authentication, an IP (Internet Protocol) restriction or a VPN (Virtual Private Network). Disallow: / is only the belt next to those braces.

Adding Disallow to remove a page from the index. If the page is already indexed, Disallow doesn't remove it; it also prevents the noindex from being seen.

Blocking CSS and JavaScript. The rendered page stays empty, and mobile friendliness and content evaluation break.

Opening a separate group for a bot and assuming the * rules still apply to it. Groups aren't combined; repeat every rule that bot needs in its own group.

Forgetting case and $. Disallow: /Search doesn't block /search; Disallow: /*.pdf$ doesn't block a PDF with ?v=2 appended.

A Small Experiment

★☆☆ Fix the opening file by copying the rules of the * group into the Googlebot group as well, and run the matcher again. /search?q=dune should now come out as blocked for Googlebot.

★★☆ Rewrite the Disallow: /*.pdf$ rule so it also covers versioned PDFs, and compare the results for /files/catalog.pdf?v=2 and /files/catalog.pdfx. Which pattern tells them apart correctly?

★★★ Write an integration test for the robots.txt endpoint: start the application in the Production and Staging environments with WebApplicationFactory, read the returned file with RobotsTxt.Parse, and assert the rule "book pages and bundles are crawlable in Production, nothing is crawlable in Staging". Using WebApplicationFactory is covered in the ASP.NET Core Web API (Application Programming Interface) series post "Integration Testing an ASP.NET Core Web API: End-to-End Tests with WebApplicationFactory".

robots.txt stops the crawler at the door; but if you want a page you let in to stay out of the results, you need another mechanism. In the next post we'll look at the robots meta tag and the X-Robots-Tag header, keeping PDFs out of the index, and why robots.txt and noindex don't work together.