noindex and X-Robots-Tag: Correctly Keeping Pages and PDFs out of the Index

Version note: The code in this post was built and run with .NET 10 (ASP.NET Core 10) and Angular 22.2; the rules are explained based on Google's robots meta tag documentation as of November 2026.

Guess First

Bookstore published its 2024 catalog as a PDF at /uploads/catalog-2024.pdf. The prices are out of date, but the PDF still shows up in Google and customers keep trying to order at the old prices. The team does two things: it adds Disallow: /uploads/ to robots.txt and submits a removal request for the URL (Uniform Resource Locator) in Search Console's Removals tool. The next day the PDF is gone from the results; the team closes the ticket.

What's the state of the PDF six months later? And if the same team keeps the account page of its Angular application out of the index by adding a noindex meta tag with JavaScript in the component's constructor, does Google see that tag?

In the previous post of the series we saw that robots.txt blocks crawling but not indexing. This post is about the tool on the indexing side: the robots meta tag and its HTTP (HyperText Transfer Protocol) header counterpart, X-Robots-Tag.

Two Ways, Same Rules

You can state how a page should appear in the index, per page, in two places. For HTML (HyperText Markup Language) pages, a meta tag inside <head>:

<meta name="robots" content="noindex, follow">
<meta name="googlebot" content="max-snippet:120">

Or an HTTP header on the response itself:

X-Robots-Tag: noindex
X-Robots-Tag: googlebot: max-snippet:120

Every rule in the meta tag can also be sent as a header. The header has two advantages: it's the only option for non-HTML files (PDFs, images, videos), and it can be applied centrally on the server without touching the page content. Google accepts both multiple X-Robots-Tag lines and a comma-separated list in the header, and you can target a single crawler by prefixing a rule with its name (googlebot: nofollow). As the meta tag's name value, Google recognizes only robots, googlebot and googlebot-news.

About the meta tag's placement, a note was added to the documentation in March 2026; the behavior didn't change, it just got written down: Google honors a robots meta tag even when it finds it outside <head>, for example inside <body>. That may sound like good news but it's really a warning: a <meta name="robots" content="noindex"> that a CMS (Content Management System) editor pastes into a post body can drop the page from the index. If you render user content as HTML, add meta tag sanitizing to your list.

The Rules and What They Do

A summary of the rules Google supports:

A few details. all and index, follow are the default behavior; writing them has no effect. noarchive and nocache are now ignored by Google Search. When rules conflict, the more restrictive one wins: with nosnippet and max-snippet:50 together, no snippet appears at all. If you give different crawlers different rules, the sum of the negative rules applies to Googlebot: a generic nofollow plus a Googlebot-specific noindex means noindex, nofollow for Googlebot. For unavailable_after, the date can be given in RFC (Request for Comments) 822, RFC 850 or ISO (International Organization for Standardization) 8601 format; after that date Google crawls the page less often.

The fact that nosnippet also covers AI Overviews and AI Mode makes it the page-level way to opt out of generative AI features. We discuss the cost of that from a GEO (Generative Engine Optimization) point of view, and the alternatives, in the GEO series; here, just note the mechanism.

An X-Robots-Tag Middleware in ASP.NET Core

Bookstore has three rules: PDFs under /uploads must not be indexed; on-site search results must not be indexed but the links in them should be followed; and the Black Friday campaign page must drop out of results after 1 December, with its snippet limited to 120 characters for Googlebot. Gathering the rules in one middleware is easier to audit than adding headers endpoint by endpoint:

namespace Bookstore.Noindex;

public sealed class RobotsHeaderMiddleware(RequestDelegate next)
{
    public Task InvokeAsync(HttpContext context)
    {
        // headers must be added before the first byte of the body is written
        context.Response.OnStarting(() =>
        {
            foreach (string rule in RulesFor(context.Request.Path))
                context.Response.Headers.Append("X-Robots-Tag", rule);
            return Task.CompletedTask;
        });
        return next(context);
    }

    private static IEnumerable<string> RulesFor(PathString path)
    {
        if (path.StartsWithSegments("/uploads") && path.Value!.EndsWith(".pdf", StringComparison.OrdinalIgnoreCase))
            yield return "noindex";
        if (path.StartsWithSegments("/search"))
            yield return "noindex, follow";
        if (path.StartsWithSegments("/campaigns/black-friday"))
        {
            yield return "unavailable_after: 2026-12-01T00:00:00+03:00";
            yield return "googlebot: max-snippet:120";
        }
    }
}

WebApplicationBuilder builder = WebApplication.CreateBuilder(args);
WebApplication app = builder.Build();

app.UseMiddleware<RobotsHeaderMiddleware>(); // before static files: covers PDFs too
app.UseStaticFiles();

app.MapGet("/search", (string? q) => Results.Content($"<h1>Results for {q}</h1>", "text/html"));
app.MapGet("/campaigns/black-friday", () => Results.Content("<h1>Black Friday</h1>", "text/html"));
app.MapGet("/books/{slug}", (string slug) => Results.Content($"<h1>{slug}</h1>", "text/html"));
app.Run();

I use OnStarting to add the header right before the response starts being written: that way both the PDFs served by the static file middleware and the HTML produced by endpoints go through the same rule. The real responses:

$ curl -s -D - -o /dev/null http://localhost:20104/uploads/catalog-2024.pdf
HTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noindex
$ curl -s -D - -o /dev/null http://localhost:20104/search?q=dune
HTTP/1.1 200 OK
Content-Type: text/html
X-Robots-Tag: noindex, follow
$ curl -s -D - -o /dev/null http://localhost:20104/campaigns/black-friday
HTTP/1.1 200 OK
Content-Type: text/html
X-Robots-Tag: unavailable_after: 2026-12-01T00:00:00+03:00
X-Robots-Tag: googlebot: max-snippet:120
$ curl -s -D - -o /dev/null http://localhost:20104/books/dune
HTTP/1.1 200 OK
Content-Type: text/html

The book page has no header because we want it indexed; there's no need to write a rule for the default behavior. The two separate header lines on the campaign page are the readable way to give two different targets two separate rules.

The Ordering Bug: Static Files Swallow the Header

When I moved the same middleware after UseStaticFiles(), the responses changed like this:

$ curl -s -D - -o /dev/null http://localhost:20104/uploads/catalog-2024.pdf
HTTP/1.1 200 OK
Content-Type: application/pdf
$ curl -s -D - -o /dev/null http://localhost:20104/search?q=dune
HTTP/1.1 200 OK
Content-Type: text/html
X-Robots-Tag: noindex, follow

The search page is still correct, but the PDF's header is gone. When the static file middleware finds the file, it writes the response itself and short-circuits the pipeline there; the middleware behind it never runs. A bug like this is hard to notice without a test: the pages work, only a header is missing. The logic of middleware ordering is in "The ASP.NET Core Middleware Pipeline: Ordering, Use, Run, Map and Your Own Middleware" in the ASP.NET Core Web API (Application Programming Interface) series. Alternatively, for static files only, you could add the header in the StaticFileOptions.OnPrepareResponse callback; but splitting the rules across two places invites updating one and forgetting the other.

robots.txt and noindex Don't Work Together

That was the opening team's first mistake. The moment Disallow: /uploads/ lands in robots.txt, Googlebot stops requesting the PDF, and it can't see the header of a response it never requested. Google's documentation states it clearly: robots meta tags and X-Robots-Tag are only discovered when the URL is crawled; rules on a URL whose crawling is blocked are ignored. So noindex never takes effect for a blocked URL, and as long as the URL gets links from other sites it can stay in the index as a result without content.

The right order is: first give it noindex and leave the URL open to crawling. Let Google crawl the URL again, see the noindex and drop it from results. Only after that, if you want, block crawling with robots.txt. There's also a crawl budget side to this: Google notes that noindex pages still have to be requested, so noindex doesn't reduce crawling. We cover crawl budget in the eleventh post of the series.

The team's second mistake was the Removals tool. According to the Search Console help page, a successful temporary removal lasts only about six months, and it doesn't delete the URL from the index; it hides it in results. The permanent solution lives on the page itself: removing the content and returning 404 or 410, restricting access with a password, or giving it noindex. So the answer to the opening question is "six months later the PDF most likely comes back": the removal has expired, and because of robots.txt, even a noindex couldn't be seen. The Removals tool buys time in emergencies (such as personal data published by mistake); it isn't the solution itself.

The right flow for Bookstore: if the old catalog is definitely no longer needed, delete the file and return 404 or 410 (Google handles the two the same way; details in the sixth post of the series). If the file stays for customers' archive, give it noindex with X-Robots-Tag and remove the /uploads/ block from robots.txt.

noindex with JavaScript: The Right Place in Angular

In December 2025 Google added two notes to its JavaScript SEO (Search Engine Optimization) documentation. First, adding or changing a robots meta tag with JavaScript is possible, but when Google sees noindex it may skip rendering and JavaScript execution; so if you want the page indexed, the original HTML must not contain noindex. Second, Google doesn't define how a noindex added with JavaScript is handled. So removing noindex with JavaScript is unreliable, and adding it isn't guaranteed either.

In Bookstore's Angular application, the account page runs with RenderMode.Client; the server sends only an empty shell HTML. The component adds noindex in its constructor:

@Component({
  selector: 'app-account',
  template: `<h1>My account</h1><p>Orders and wishlist.</p>`,
})
export class Account {
  constructor() {
    // runs only in the browser: this route is RenderMode.Client
    inject(SeoService).update({ title: 'My account', description: '', path: '/account', noindex: true });
  }
}

The tag isn't in the HTML the server sends; it only appears once the browser runs the JavaScript. That's exactly the zone where Google doesn't define its behavior. The solution is to move the rule into the response the server sends. Angular's server routes can set headers regardless of render mode:

{ path: 'account', renderMode: RenderMode.Client, headers: { 'X-Robots-Tag': 'noindex' } },

$ curl -s -D - -o acc.html http://localhost:20110/account | grep -iE "^HTTP|^x-robots|^content-type"
HTTP/1.1 200 OK
content-type: text/html;charset=UTF-8
x-robots-tag: noindex
$ grep -o '<meta name="robots"[^>]*>\|<title>[^<]*</title>' acc.html
<title>BookstoreNg</title>

There's no robots meta tag in the HTML and the title is still the application default; but the header is there in the first response. On the search page, which runs with SSR (Server-Side Rendering), the tag added by SeoService arrives inside the first HTML because it was rendered on the server:

$ curl -s "http://localhost:20110/search?q=dun" | grep -o '<meta name="robots"[^>]*>'
<meta name="robots" content="noindex">

The rule: everything that affects the indexing decision (noindex, canonical, status code) must be in the first HTML or in the HTTP response. We'll set up render modes and server routes in detail in the ninth post of the series.

The robots Meta Tag on mertozen.com

In this site's source code, the robots meta tag is printed from a single partial view. The default value is index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1. The first two rules repeat the default behavior, so they have no effect; the other three explicitly leave the snippet length and image/video previews unlimited. It's a choice consistent with the site's preference for being cited in AI features.

The 404 page and the unexpected-error page override this value with noindex, follow: the page shouldn't be indexed, but the home page and blog links on it should be followed. Since the 404 page already returns a real 404 status code (the site uses UseStatusCodePagesWithReExecute("/404")), Google doesn't process its content; noindex is a second safety net here. The 503 page returned when the CMS is temporarily unavailable also carries its own noindex. The on-site search in the blog list (?q=) gets no noindex; instead its canonical points to the list address with the search term dropped, which is also a valid choice; we discuss the difference between canonical and noindex in the fifth post of the series.

Production Notes

Test your X-Robots-Tag rules. Headers are invisible; if a page accidentally gets noindex, the site keeps working, only traffic slowly drops. In an integration test, assert that critical URLs (home page, book pages) carry no X-Robots-Tag, and that PDFs and search pages do.

Watch out for staging's noindex leaking into production. A rule that gives every page noindex based on the environment takes production out of the index on a deploy where the environment name is set wrong. This scenario is another reason to protect staging with authentication rather than noindex.

Search Console's URL Inspection tool shows the robots rules Google sees for a URL and its index status; the "Excluded by 'noindex' tag" row in the "Pages" report lists the noindex URLs. If a URL you shouldn't see is in that list, the problem is usually in a template. An introduction to Search Console is in the SEO Fundamentals series.

Check that CDN (Content Delivery Network) and reverse proxy layers preserve the headers. Some layers may drop unknown headers or serve a stale cached response without the header. Run your tests against the real address in front of the application, not against Kestrel.

Common Pitfalls

Putting noindex on a page blocked by robots.txt. The two cancel each other out; the noindex is never seen.

Treating the Removals tool as a permanent solution. A removal lasts about six months; if the page itself has no 404, 410, password or noindex, the URL comes back.

Putting noindex in the first HTML of a page you want indexed and removing it with JavaScript. When Google sees noindex, it may skip rendering; the JavaScript may never run.

Looking for a meta tag for a PDF. A PDF has no <head>; X-Robots-Tag is the only way.

Registering the middleware after UseStaticFiles(). Static files never see the rules.

Making a page noindex and pointing its canonical to another page. The two signals conflict: one says "don't index this page", the other says "this page is a copy of that one". Decide which one you want; details in the canonical post.

A Small Experiment

★☆☆ Check the robots meta tag and X-Robots-Tag header of three URLs on your own site (a page, a PDF, a search result) with curl -s -D - -o /dev/null. Does anything you don't want indexed come back without a header?

★★☆ Add a rule to RobotsHeaderMiddleware that looks at the response status code: give 4xx and 5xx responses a noindex header. Discuss how necessary this is for the 404 page, given that Google doesn't process 4xx content anyway.

★★★ Write a test class with WebApplicationFactory: keep a list of URLs and the expected X-Robots-Tag values (or their absence) for each as a table, and have the test request every URL and compare the headers. Then move the middleware after UseStaticFiles() and watch the test break on the PDF row.

Keeping things out of the index is one side; announcing the URLs you do want indexed to Google is the other. In the next post we'll look at the protocol limits of XML (Extensible Markup Language) sitemaps, when Google takes lastmod seriously, generating a streaming sitemap from EF (Entity Framework) Core in ASP.NET Core, and IndexNow, which announces changes to Bing and Yandex instantly, with code.