Mert Özen Further With Every Line

Blog

Mert Özen's blog posts on software development, backend, frontend, and DevOps.

ALL All 933 posts
BE Backend 204 posts
FE Frontend 124 posts
DO DevOps 105 posts
DB Database 44 posts
AI Artificial Intelligence 61 posts
SEC Security 67 posts
DSN Software Design 88 posts
ARC Software Architecture 108 posts
QA Testing & Quality 17 posts
PM Project Management 39 posts
CS Computer Science Fundamentals 37 posts
SEO SEO & GEO 26 posts
DEV Developer Tools 13 posts
Series
1 posts found #x-robots-tag
Newest Oldest Popular
noindex and X-Robots-Tag: Correctly Keeping Pages and PDFs out of the Index
SEO & GEO PART 03

noindex and X-Robots-Tag: Correctly Keeping Pages and PDFs out of the Index

The way to keep a page or PDF out of the index is noindex, not robots.txt: a noindex on a blocked URL is never seen, and the Removals tool only hides a URL for about six months. We build an X-Robots-Tag middleware in ASP.NET Core and look at the static file ordering bug, the risk of noindex added with JavaScript in Angular, and server route headers, with real outputs.

8 Nov 2026 11 min 4
Blog Map