Blog
Mert Özen's blog posts on software development, backend, frontend, and DevOps.
What Is SRE? Its Relation to DevOps, Toil, the Error Budget Idea and SRE in a Small Team
SRE means running operations with software engineering and tying reliability to a measurable target; 100 percent is the wrong target, and the gap, the error budget, decides between speed and reliability. We cover what toil is, how SRE relates to DevOps and, with an inventory table, which practices the Bookstore team without a separate SRE team should adopt.
Choosing SLIs: From User Journeys to Measurable Signals
A good SLI comes from the user's journey and is the ratio of good events to valid events; 4xx answers and health checks stay out of the denominator. For Bookstore's checkout we write availability and latency SLIs from ASP.NET Core's http.server.request.duration histogram and measure how mixing all endpoints and averaging hide errors, and why the application metric sees nothing while the API is down.
SLOs and Error Budgets: Setting Targets, the Math of Nines and an Error Budget Policy
An SLO is a target derived from today's measurements and user expectations and stricter than the SLA; the room it leaves is the error budget, and a policy written in advance says what happens when it runs out. For Bookstore's checkout we turn a 99.5 percent availability and an 800 ms latency SLO into a document, Prometheus rules and OpenSLO, convert nines into minutes, and measure how a new bucket and a short window misreport the budget.
SLO-Based Alerting: Burn Rate, Multiwindow Alerts and Ending Alert Fatigue
An SLO-based alert measures how fast the error budget is spent (burn rate) over a long and a short window together: the long window stays quiet on short spikes, the short one closes the alert soon after the outage. For Bookstore's checkout we turn the Workbook's 14.4/6/1 thresholds into Prometheus rules, test them with promtool, measure when they fire and stop in a live outage, and compare them with what Sloth generates.
On-Call and Runbooks: Rotations, Escalation and Writing Actionable Runbooks
Sustainable on-call is built with a primary and a secondary, a page budget and a runbook of copyable commands for every alert. We cover a rotation adapted to the three-person Bookstore team, Alertmanager routing verified with amtool, a runbook kept in the repo, the mistakes we found by trying it on a real outage, and automation that moves its first two steps into the alert text.
Incident Management: Severity, Incident Commander, Communication and Mitigation First
Incident management means running an outage from a single decision point, the incident commander, and stopping the customer impact before looking for the cause; severity definitions, roles and levers are prepared in advance. We add a payment timeout and a kill switch that need no restart to Bookstore, play a tabletop exercise in the lab where the payment provider first slows down and then fails, and measure the effect of every decision.
The Postmortem Process: Review Meetings, Action Item Tracking and the Limits of Incident Metrics
The postmortem process means writing down in advance which incidents get a postmortem, running the draft through a review led by someone other than the author, and tracking action items with an owner, a date and a ticket; averages such as MTTR are mostly noise with few samples. We write the postmortem of Bookstore's tabletop, build a .NET tool that checks action items, and show that a 38 percent MTTM improvement appears 40 percent of the time even when nothing changes.
Real Incidents 1: Globally Propagated Config — Cloudflare 2025, Google Cloud 2025 and CrowdStrike 2024
In the major outages of Cloudflare, Google Cloud and CrowdStrike the trigger was not code but a configuration or data change that spread worldwide within seconds; the lesson is to validate your own configuration like input and roll it out in stages. We summarize the three cases from their official postmortems, reproduce the mistake with Bookstore's switches.json (500s on an invalid value, a crash loop on a broken file) and fix it with a check tool and a last-known-good reader.
Real Incidents 2: Automation, Hidden Dependencies and Recovery Paths — AWS 2025, Facebook 2021, GitLab 2017, Knight Capital 2012
In the AWS, Facebook, GitLab and Knight Capital cases the outage grew because of automation's own failures, hidden dependencies and recovery paths that depended on what broke; the lesson is to rehearse the recovery path too. We tell the four cases from official sources, run a restore drill on Bookstore's SQL Server whose first run comes out red, prepare a dependency map and break-glass access, and write an alert that catches mixed versions.
Game Days and Chaos Engineering: Rehearsing Incidents Before They Happen
A game day is a controlled rehearsal of an outage with a steady state, a hypothesis, a small blast radius and an abort condition written in advance; chaos engineering turns it into a measurable experiment. We inject faults into Bookstore's payment client with Polly's chaos strategies, measure two hypotheses failing (the page comes 46 minutes late, the runbook misses the cause) and feed the results into the runbook and a retry, cutting checkout errors from 9.3 to 0.9 percent.