Blog
Mert Özen's blog posts on software development, backend, frontend, and DevOps.
SLO-Based Alerting: Burn Rate, Multiwindow Alerts and Ending Alert Fatigue
An SLO-based alert measures how fast the error budget is spent (burn rate) over a long and a short window together: the long window stays quiet on short spikes, the short one closes the alert soon after the outage. For Bookstore's checkout we turn the Workbook's 14.4/6/1 thresholds into Prometheus rules, test them with promtool, measure when they fire and stop in a live outage, and compare them with what Sloth generates.
SLOs and Error Budgets: Setting Targets, the Math of Nines and an Error Budget Policy
An SLO is a target derived from today's measurements and user expectations and stricter than the SLA; the room it leaves is the error budget, and a policy written in advance says what happens when it runs out. For Bookstore's checkout we turn a 99.5 percent availability and an 800 ms latency SLO into a document, Prometheus rules and OpenSLO, convert nines into minutes, and measure how a new bucket and a short window misreport the budget.
Alerting: Symptom-Based Alerts, Alertmanager, Grafana Alerting, Runbooks and Alert Fatigue
Alerts go on symptoms the user feels, and each one has a runbook; causes are tickets, not pages. For Bookstore we write burn-rate, latency, retry, missing-telemetry and Watchdog alerts in Prometheus, route them with Alertmanager, watch grouping, silences and inhibition during a real failure, and measure why the error-rate alert goes quiet when the API is down.