Blog
Mert Özen's blog posts on software development, backend, frontend, and DevOps.
A robots.txt Guide: RFC 9309, Allow/Disallow Rules and What Not to Block
robots.txt controls crawling, not indexing: a crawler follows the one group carrying its name, the longest matching rule wins, and a robots.txt returning 5xx stops crawling. We test RFC 9309 rules in C#, generate robots.txt per environment in ASP.NET Core, see Angular SSR giving robots.txt a one-year cache header, and read mertozen.com's file.
Understanding and Managing AI Bots: Training vs Search vs User-Initiated Fetches
AI bots come in three classes: training crawlers such as GPTBot and ClaudeBot, search bots such as OAI-SearchBot and Claude-SearchBot, and user-initiated fetches such as ChatGPT-User; to opt out of training but stay citable, you block only the first class. We verify robots.txt scenarios against RFC 9309 rules and cover Google's snippet and Search Console controls, mertozen.com's robots.txt and separating AI bots in logs.