Blog
Mert Özen's blog posts on software development, backend, frontend, and DevOps.
Game Days and Chaos Engineering: Rehearsing Incidents Before They Happen
A game day is a controlled rehearsal of an outage with a steady state, a hypothesis, a small blast radius and an abort condition written in advance; chaos engineering turns it into a measurable experiment. We inject faults into Bookstore's payment client with Polly's chaos strategies, measure two hypotheses failing (the page comes 46 minutes late, the runbook misses the cause) and feed the results into the runbook and a retry, cutting checkout errors from 9.3 to 0.9 percent.
Real Incidents 2: Automation, Hidden Dependencies and Recovery Paths — AWS 2025, Facebook 2021, GitLab 2017, Knight Capital 2012
In the AWS, Facebook, GitLab and Knight Capital cases the outage grew because of automation's own failures, hidden dependencies and recovery paths that depended on what broke; the lesson is to rehearse the recovery path too. We tell the four cases from official sources, run a restore drill on Bookstore's SQL Server whose first run comes out red, prepare a dependency map and break-glass access, and write an alert that catches mixed versions.
Real Incidents 1: Globally Propagated Config — Cloudflare 2025, Google Cloud 2025 and CrowdStrike 2024
In the major outages of Cloudflare, Google Cloud and CrowdStrike the trigger was not code but a configuration or data change that spread worldwide within seconds; the lesson is to validate your own configuration like input and roll it out in stages. We summarize the three cases from their official postmortems, reproduce the mistake with Bookstore's switches.json (500s on an invalid value, a crash loop on a broken file) and fix it with a check tool and a last-known-good reader.
The Postmortem Process: Review Meetings, Action Item Tracking and the Limits of Incident Metrics
The postmortem process means writing down in advance which incidents get a postmortem, running the draft through a review led by someone other than the author, and tracking action items with an owner, a date and a ticket; averages such as MTTR are mostly noise with few samples. We write the postmortem of Bookstore's tabletop, build a .NET tool that checks action items, and show that a 38 percent MTTM improvement appears 40 percent of the time even when nothing changes.
Incident Management: Severity, Incident Commander, Communication and Mitigation First
Incident management means running an outage from a single decision point, the incident commander, and stopping the customer impact before looking for the cause; severity definitions, roles and levers are prepared in advance. We add a payment timeout and a kill switch that need no restart to Bookstore, play a tabletop exercise in the lab where the payment provider first slows down and then fails, and measure the effect of every decision.
On-Call and Runbooks: Rotations, Escalation and Writing Actionable Runbooks
Sustainable on-call is built with a primary and a secondary, a page budget and a runbook of copyable commands for every alert. We cover a rotation adapted to the three-person Bookstore team, Alertmanager routing verified with amtool, a runbook kept in the repo, the mistakes we found by trying it on a real outage, and automation that moves its first two steps into the alert text.
SLO-Based Alerting: Burn Rate, Multiwindow Alerts and Ending Alert Fatigue
An SLO-based alert measures how fast the error budget is spent (burn rate) over a long and a short window together: the long window stays quiet on short spikes, the short one closes the alert soon after the outage. For Bookstore's checkout we turn the Workbook's 14.4/6/1 thresholds into Prometheus rules, test them with promtool, measure when they fire and stop in a live outage, and compare them with what Sloth generates.
SLOs and Error Budgets: Setting Targets, the Math of Nines and an Error Budget Policy
An SLO is a target derived from today's measurements and user expectations and stricter than the SLA; the room it leaves is the error budget, and a policy written in advance says what happens when it runs out. For Bookstore's checkout we turn a 99.5 percent availability and an 800 ms latency SLO into a document, Prometheus rules and OpenSLO, convert nines into minutes, and measure how a new bucket and a short window misreport the budget.
Choosing SLIs: From User Journeys to Measurable Signals
A good SLI comes from the user's journey and is the ratio of good events to valid events; 4xx answers and health checks stay out of the denominator. For Bookstore's checkout we write availability and latency SLIs from ASP.NET Core's http.server.request.duration histogram and measure how mixing all endpoints and averaging hide errors, and why the application metric sees nothing while the API is down.
What Is SRE? Its Relation to DevOps, Toil, the Error Budget Idea and SRE in a Small Team
SRE means running operations with software engineering and tying reliability to a measurable target; 100 percent is the wrong target, and the gap, the error budget, decides between speed and reliability. We cover what toil is, how SRE relates to DevOps and, with an inventory table, which practices the Bookstore team without a separate SRE team should adopt.
Querying Application Insights and Log Analytics with KQL: where, summarize, bin, join, percentiles and Alert Queries
When you query Application Insights with KQL you count with sum(itemCount), not count(): with 25 percent sampling, 300 checkouts shrink to 82 rows and none of the 7 failed ones remains. We collect Bookstore's telemetry in the Application Insights schema without an Azure account and try where, summarize, percentile, bin, make-series, join, lookup, parse/extract and alert queries on real data.
What's New in Observability from .NET 8 to .NET 11: IMeterFactory, Gauge, New Built-in Metrics and Built-in Tracing in ASP.NET Core
In .NET 11 ASP.NET Core writes the OpenTelemetry attributes of the HTTP server span itself and MemoryCache publishes dotnet.cache.* metrics, with no instrumentation package. We map the releases from .NET 8's IMeterFactory and .NET 9's Gauge to .NET 10's Identity metrics and schema URLs, run the .NET 11 items on the GA SDK and give tips for reading older code.