Mert Özen Further With Every Line

Blog

Mert Özen's blog posts on software development, backend, frontend, and DevOps.

ALL All 933 posts
BE Backend 204 posts
FE Frontend 124 posts
DO DevOps 105 posts
DB Database 44 posts
AI Artificial Intelligence 61 posts
SEC Security 67 posts
DSN Software Design 88 posts
ARC Software Architecture 108 posts
QA Testing & Quality 17 posts
PM Project Management 39 posts
CS Computer Science Fundamentals 37 posts
SEO SEO & GEO 26 posts
DEV Developer Tools 13 posts
Series
1 posts found #ai-crawlers
Newest Oldest Popular
Understanding and Managing AI Bots: Training vs Search vs User-Initiated Fetches
SEO & GEO PART 03

Understanding and Managing AI Bots: Training vs Search vs User-Initiated Fetches

AI bots come in three classes: training crawlers such as GPTBot and ClaudeBot, search bots such as OAI-SearchBot and Claude-SearchBot, and user-initiated fetches such as ChatGPT-User; to opt out of training but stay citable, you block only the first class. We verify robots.txt scenarios against RFC 9309 rules and cover Google's snippet and Search Console controls, mertozen.com's robots.txt and separating AI bots in logs.

8 Nov 2026 10 min 5
Blog Map