Blog
Mert Özen's blog posts on software development, backend, frontend, and DevOps.
On-Call and Runbooks: Rotations, Escalation and Writing Actionable Runbooks
Sustainable on-call is built with a primary and a secondary, a page budget and a runbook of copyable commands for every alert. We cover a rotation adapted to the three-person Bookstore team, Alertmanager routing verified with amtool, a runbook kept in the repo, the mistakes we found by trying it on a real outage, and automation that moves its first two steps into the alert text.
Alerting: Symptom-Based Alerts, Alertmanager, Grafana Alerting, Runbooks and Alert Fatigue
Alerts go on symptoms the user feels, and each one has a runbook; causes are tickets, not pages. For Bookstore we write burn-rate, latency, retry, missing-telemetry and Watchdog alerts in Prometheus, route them with Alertmanager, watch grouping, silences and inhibition during a real failure, and measure why the error-rate alert goes quiet when the API is down.