noindex and X-Robots-Tag: Correctly Keeping Pages and PDFs out of the Index
The way to keep a page or PDF out of the index is noindex, not robots.txt: a noindex on a blocked URL is never seen, and the Removals tool only hides a URL for about six months. We build an X-Robots-Tag middleware in ASP.NET Core and look at the static file ordering bug, the risk of noindex added with JavaScript in Angular, and server route headers, with real outputs.
8 Nov 2026
·
11 min
·
4
