The Postmortem Process: Review Meetings, Action Item Tracking and the Limits of Incident Metrics

Version note: The examples in this post were tried with .NET 10 (two small tools run as file-based apps). The data in the incident metrics calculation is exercise data, not a real team's record.

Guess First

The team behind Bookstore, the online bookstore used as the shared example across this site's series, looks at its incident numbers at the end of every quarter. The second quarter had 7 incidents with a mean time to mitigate of 61 minutes; the third quarter had 8, and the mean dropped to 38 minutes. You are about to write "MTTM improved by 38 percent" on a management slide. Is the improvement real, or could the difference between two quarters appear by chance? Give your guess as a percentage: if the team had changed nothing, in how many pairs of quarters would a difference this large show up? Second question: the postmortem of last month's payment provider incident has 4 action items. Who knows today how many of them are still open, and from where? The answers are in the second half of the post, with a real calculation.

Which Incidents Need a Postmortem?

A postmortem is the document that writes down, after an incident, what happened, why it happened and what will be done so it does not happen again. The "Root Cause Analysis: 5 Whys, Fishbone and Blameless Postmortem" post of the Debugging and Problem Solving series covered what goes inside the document: blameless language, the timeline, cause analysis with 5 Whys and fishbone, and a template. This post is about the process around the document: which incident gets a postmortem, how the draft goes through review, how action items are tracked, and what incident metrics can and cannot show.

Writing a postmortem for every incident wears the team out and turns the documents into a formality; writing none means living through the same outage a third time. So the criteria must be written down in advance. The "Postmortem Culture" chapter of Google's SRE book lists typical triggers: user-visible downtime beyond a threshold, any data loss, on-call intervention such as a rollback, resolution time beyond a threshold, and a person rather than monitoring finding the problem. The same chapter adds that any stakeholder may ask for a postmortem of an event. Bookstore's criteria are in docs/postmortems/README.md:

## When a postmortem is required
- SEV1 or SEV2.
- More than 20 % of a 30-day error budget spent by one incident (`docs/slo/error-budget-policy.md`).
- Any data loss, any wrong charge, or a security-relevant finding.
- A customer or support found it before an alert did.
- Production was changed by hand (rollback, lever in `docs/incidents/levers.md`, manual data fix).
- Anyone asks for one, including for a near miss.

The last two items are the ones most often skipped. "A customer found it before we did" shows the blind spot of monitoring; "production was changed by hand" makes visible how often and why levers such as the switches.json from the previous post are used. The criterion from the error budget policy in the third post of the series is here too: an incident that alone spends more than 20 percent of the 30-day budget gets a postmortem whatever its severity.

From Draft to Publication

A postmortem needs its own timeline too; otherwise "we will write it this week" takes two months and people forget the details. Bookstore's flow:

## Flow and deadlines
| Step | Who | When |
|---|---|---|
| Draft (timeline from the incident document, impact in numbers, contributing factors, action items) | IC picks an author, usually the ops lead | within 3 working days |
| Review meeting (45 minutes) | author, IC, people who were in the incident, owner of each affected service; facilitator is not the author | within 5 working days |
| Publish (merge to `main`, post the link in #engineering) | author | after the review |
| Action items tracked | each owner; `postmortem-lint` in CI lists overdue ones | until done |

The author starts from the timeline in the incident document; that is why the scribe's notes in the previous post's tabletop were one-line facts with UTC times. The tabletop's postmortem is in the repository as docs/postmortems/2026-11-14-payment-provider-tabletop.md. Its impact and contributing factors sections:

## Impact
- 22:09:47–22:27:47 UTC (18 minutes). Customer-visible: checkout.
- 1,598 orders cancelled; each cancelled order also deleted the customer's cart.
- 1,380 checkouts answered 503 by the kill switch (carts kept), 456 of them while the provider was already healthy.
- 5 minutes of checkouts at about 4 s (latency SLO), 0 errors in that phase.

## Contributing factors
- The payment timeout was fixed at startup; there was no lever until this incident's preparation added one.
- A failed payment cancels the order after the cart has been deleted, so a provider problem costs customers their carts.
- With the kill switch on, Bookstore.Api makes no calls to the provider, so our metrics cannot show its recovery.
- The page does not tell whether a mitigation works: 503s from the kill switch count as failures like 502s.

Note two details. Impact is written in numbers: not "some customers were affected" but "1,598 orders cancelled, each one a cart". And the contributing factors contain systems, not people: instead of "Emre turned the kill switch off late", "with the kill switch on, our metrics cannot show that the provider has recovered". Both describe the same event, but only the second can become an action item. The "Where we got lucky" section is deliberate as well: writing down what ended well by chance rather than by design points at the candidate for the next incident.

Running the Review Meeting

The draft is usually written by the incident's ops lead, who knows the timeline best. The point of the review meeting is not to approve the draft but to test it: is the timeline right, is the impact given in numbers, is the cause deep enough, do the action items really prevent or shrink this incident? Google's SRE book looks at similar criteria in review: is the key data collected, is the impact assessment complete, is the root cause examined deeply enough, are the action plan and priorities appropriate, have stakeholders been told the outcome.

A few rules make the meeting work. The facilitator should not be the author; authors start defending their own text. No new debugging happens in the meeting; missing information becomes an action item. Questions are asked as "why did this make sense with what you knew at the time?" rather than "why did you do that?"; the first produces information, the second defense. At Bookstore the agenda is 45 minutes and written in docs/postmortems/README.md:

## Review meeting agenda
1. (5 min) The facilitator reads the impact and the timeline summary. No new debugging in the meeting.
2. (15 min) Timeline questions: when did we know, what did we believe, why did that make sense at the time?
3. (15 min) Contributing factors and action items: does each item make this incident less likely, find it sooner
   or make it smaller? One owner, a date, a ticket.
4. (5 min) What went well and should be kept.
5. (5 min) Decide: severity confirmed, postmortem approved or a second round.

The "what went well" item matters more than it looks. The kill switch keeping the carts or the runbook showing the cause within a minute in the tabletop are things the next incident needs too; if they are not written down, the next "improvement" may break them without anyone noticing.

Action Items: Owner, Date, Class

Action items are the lasting trace of a postmortem. The "Postmortem Culture" chapter of Google's SRE Workbook describes bad action items: vague verbs such as "improve" or "make better", items without an owner or a priority, lists made only of mitigations. A good action item has one owner, a tracking ticket, a priority and a verifiable end state. The same chapter quotes Ben Treynor Sloss: "a postmortem without subsequent action is indistinguishable from no postmortem." (Google SRE Workbook, "Postmortem Culture: Learning from Failure", 2018). An action item nobody tracks turns the postmortem into a memoir.

Bookstore puts action items into four classes: prevent (the same trigger can no longer cause this outcome), detect (we notice sooner), mitigate (if it happens again, the impact is smaller or shorter) and process (on-call, communication, review). Writing the class is a balance check: a list with only mitigate and process items accepts that the same incident will happen again. The Workbook also warns against lists made only of mitigations.

Once the rules are written down, checking them is easy to hand over to a tool. tools/postmortem-lint/postmortem-lint.cs is a small program that runs with .NET 10's file-based apps, dotnet run file.cs without a project file: it reads the "Action items" table of every file under docs/postmortems/, reports an error for an item without a class, an owner, a date or a ticket, and lists open items that are past their date. The core of the code:

string[] classes = ["prevent", "detect", "mitigate", "process"];
string[] notAnOwner = ["", "team", "everyone", "tbd", "?"];
...
foreach (string[] c in rows)
{
    // Columns: | ID | Action | Class | Owner | Due | Ticket | Status |
    total++;
    string id = c[0];
    if (!classes.Contains(c[2].ToLowerInvariant()))
    {
        Report(file, id, $"class '{c[2]}' is not one of {string.Join("/", classes)}");
    }
    if (notAnOwner.Contains(c[3].ToLowerInvariant()))
    {
        Report(file, id, "needs one named owner");
    }
    if (!Regex.IsMatch(c[5], @"^[A-Z]+-\d+$"))
    {
        Report(file, id, $"needs a ticket (got '{c[5]}')");
    }
    bool isOpen = !c[6].Equals("done", StringComparison.OrdinalIgnoreCase);
    if (!DateOnly.TryParseExact(c[4], "yyyy-MM-dd", CultureInfo.InvariantCulture, DateTimeStyles.None, out DateOnly due))
    {
        Report(file, id, $"due date '{c[4]}' is not yyyy-MM-dd");
    }
    else if (isOpen && due < today)
    {
        overdue++;
        Console.WriteLine($"OVERDUE {Path.GetFileName(file)} {id}: {c[1]} (owner {c[3]}, due {due:yyyy-MM-dd}, {today.DayNumber - due.DayNumber} days late)");
    }
    ...
}

The first draft before the review had the typical mistakes: the verb "improve", "fix" instead of a class, "team" as the owner, "soon" as the date. On the first draft the lint said:

dotnet run tools/postmortem-lint/postmortem-lint.cs -- docs/postmortems 2026-11-17
ERROR 2026-11-14-payment-provider-tabletop.md PM-1: class 'fix' is not one of prevent/detect/mitigate/process
ERROR 2026-11-14-payment-provider-tabletop.md PM-2: needs one named owner
ERROR 2026-11-14-payment-provider-tabletop.md PM-2: needs a ticket (got '')
ERROR 2026-11-14-payment-provider-tabletop.md PM-2: due date 'soon' is not yyyy-MM-dd
ERROR 2026-11-14-payment-provider-tabletop.md PM-3: needs a ticket (got '')
3 action items, 3 open, 0 overdue, 5 problems

Because the exit code is 1, the same command works as a CI pipeline step and keeps a postmortem with incomplete action items out of main. The final version after the review has four items, one of each class; running the same command on two different days also shows the follow-up part:

## Action items
| ID | Action | Class | Owner | Due | Ticket | Status |
|---|---|---|---|---|---|---|
| PM-1 | Keep the cart when a payment fails: delete it only after the payment succeeded | mitigate | Ayse | 2026-11-27 | BOOK-412 | open |
| PM-2 | Synthetic payment probe that also runs while checkout is paused; ticket alert "provider healthy, checkout paused for 5 minutes" | detect | Emre | 2026-11-24 | BOOK-413 | open |
| PM-3 | Runbook: when to raise the timeout and when to use the kill switch; check the provider every 5 minutes while paused | process | Emre | 2026-11-18 | BOOK-414 | done |
| PM-4 | Validate switches.json in the pipeline before it is shipped (keys, types, ranges) | prevent | Selin | 2026-12-01 | BOOK-415 | open |

dotnet run tools/postmortem-lint/postmortem-lint.cs -- docs/postmortems 2026-11-20
4 action items, 3 open, 0 overdue, 0 problems

dotnet run tools/postmortem-lint/postmortem-lint.cs -- docs/postmortems 2026-11-30
OVERDUE 2026-11-14-payment-provider-tabletop.md PM-1: Keep the cart when a payment fails: delete it only after the payment succeeded (owner Ayse, due 2026-11-27, 3 days late)
OVERDUE 2026-11-14-payment-provider-tabletop.md PM-2: Synthetic payment probe that also runs while checkout is paused; ticket alert "provider healthy, checkout paused for 5 minutes" (owner Emre, due 2026-11-24, 6 days late)
4 action items, 3 open, 2 overdue, 0 problems

The second run answers the second question of Guess First: nobody has to remember the open action items, the tool says it every day. Bookstore adds this output to the weekly on-call handoff note; the "Still open" section of the template in the fifth post of the series exists for exactly this. An overdue item is either done or gets a new date in a review on purpose; it does not quietly go stale.

Learning from Near Misses

A near miss, an event that almost became an incident but did not, is the cheapest source of a postmortem. Nobody was affected, so there is no guilt and no pressure, but the same weakness is there. A deploy that started going to the wrong environment and stopped at the pipeline's approval step is a near miss; so was the command that could attach to the wrong process, which we found while trying the runbook in the fifth post of the series. These do not need a full postmortem; a half-page note ("what happened, why it did not end badly, what could happen next time") and an action item if needed are enough. The "anyone asks for one, including for a near miss" line in the README makes this explicitly allowed.

Incident Metrics and Their Limits

Turning incidents into numbers is a natural wish: MTTD, MTTM and MTTR, the mean times to detect, to mitigate and to resolve. The problem is that incident durations are very skewed: most incidents last a few tens of minutes, and now and then one lasts hours and decides the mean on its own. With few samples (7–8 incidents a quarter) the change of that mean from quarter to quarter is mostly noise. Štěpán Davidovič's report "Incident Metrics in SRE", written for Google (O'Reilly, 2021), shows this with a Monte Carlo simulation: large improvements and deteriorations in MTTR can appear while nothing changes.

We tried it ourselves with Bookstore's sample data. tools/incident-metrics/sample-history.csv is a made-up but realistic list of incidents over two quarters (exercise data, not a real team's record). incident-metrics.cs prints for each quarter the median time to detect; the mean, median and maximum time to mitigate; the number of incidents customers noticed first; and the budget spent (the sum of the percentages of a 30-day budget each incident spent). Then it tests the difference with a bootstrap, drawing samples from the observed data again and again to measure how often a difference appears by chance:

// Bootstrap: draw both quarters again from the pooled incidents (as if nothing had changed) and count how often the
// difference between the two means is at least as large as the one we observed.
string[] quarters = [.. incidents.Select(i => i.Quarter).Distinct().Order()];
double[] a = [.. incidents.Where(i => i.Quarter == quarters[0]).Select(i => i.MinutesToMitigate)];
double[] b = [.. incidents.Where(i => i.Quarter == quarters[1]).Select(i => i.MinutesToMitigate)];
double[] pooled = [.. a, .. b];
double observed = Math.Abs(a.Average() - b.Average());
Random random = new(2026);
const int Runs = 100_000;
int asLarge = 0;
for (int run = 0; run < Runs; run++)
{
    double meanA = Enumerable.Range(0, a.Length).Average(_ => pooled[random.Next(pooled.Length)]);
    double meanB = Enumerable.Range(0, b.Length).Average(_ => pooled[random.Next(pooled.Length)]);
    if (Math.Abs(meanA - meanB) >= observed)
    {
        asLarge++;
    }
}

dotnet run tools/incident-metrics/incident-metrics.cs -- tools/incident-metrics/sample-history.csv
quarter  n  detect(median)  mitigate(mean / median / max)  by customer  budget spent
2026-Q2  7         7 min        61.1 / 35 / 240 min        2 of 7        92%
2026-Q3  8         6 min        37.6 / 38 / 61 min        2 of 8        66%
MTTM 2026-Q2 → 2026-Q3: 61.1 → 37.6 min (-38%)
A difference this large appears by chance in 39.6% of 100,000 resampled quarter pairs.

The mean dropped by 38 percent, but the median rose from 35 to 38 minutes: the duration of a typical incident did not change, the third quarter simply lacks a single 240-minute SEV1. The bootstrap's answer is clear too: if both quarters came from the same distribution, a difference this large would appear in 39.6 percent of 100,000 draws. That answers the question of Guess First: "MTTM improved by 38 percent" cannot be defended with this data. It gets worse when it becomes a target: by the law named after the economist Charles Goodhart, a measure stops being a good measure once it becomes a target. A team with an MTTR target learns to close incidents early, or not to declare them at all.

This does not mean ignoring numbers. Some signals mislead less: the number of incidents by severity, the share of incidents customers noticed before we did (the "by customer" column; it shows the blind spot of monitoring), the error budget spent per incident (it measures user impact directly) and the share of action items closed on time (it shows whether learning happens). For outages caused by deploys there are DORA's change failure rate and its recovery metric, renamed failed deployment recovery time in 2023; the "Measuring and Speeding Up the Pipeline: DORA Metrics, Caching and Parallelism" post of the CI/CD Fundamentals series covered them. They need the same warning: look at the trend over a long period, not at one quarter.

Sharing the Postmortem

A postmortem is worth more the more people read it. Google's SRE book describes the ways used for this: the postmortem of the month in a newsletter, teams' postmortem reading clubs and the "Wheel of Misfortune" exercise, where an old postmortem is replayed with roles. In a small team like Bookstore's the equivalent is simple: the link of every published postmortem goes to the team channel, and once a month the last 15 minutes of the sprint review go to discussing a postmortem, if needed one published by another company. The eighth and ninth posts of the series are of that second kind.

The litmus test of the culture is the phrase "human error". If a postmortem's root cause ends with "human error", the analysis is not finished: the system that made the mistake possible, did not reveal it and amplified its impact is still in place. Google's SRE Workbook also stresses that trying to change human behavior is less reliable than changing automation and processes. That is why the first draft's item "be more careful when changing switches.json" became "validate the file in the pipeline" in the review.

In Production

Give the postmortem a deadline and name its owner when the incident closes. The last line of the tabletop's incident document was "Postmortem owner: Emre, draft by 2026-11-17"; without that line the postmortem would most likely never have been written.

Let action items live in the team's normal issue tracker. The table in the postmortem is a summary; the real tracking is in the ticket. The lint requiring a ticket ID enforces that link.

Read postmortems with security and privacy in mind too. A widely shared document must not contain customer data, internal IP addresses or secrets.

Common Pitfalls

Using the postmortem as a punishment. If a team that gets a postmortem counts as having done bad work, teams learn not to declare incidents. The criteria must be written and objective.

An action item list with only mitigations. A postmortem whose every item is "add to the runbook" accepts that the same incident will happen again. Look for at least one prevent or detect item.

Deciding on one quarter's MTTR. With few samples and a skewed distribution, the mean is the shadow of a single long incident.

The author running the review. The person running a review of their own text turns objections into a defense; the facilitator should be someone else.

A Small Experiment

★☆☆ Put the action items of your last incident (or of the tabletop in the sixth post of the series) into the four classes. If there is no prevent or detect item, write one.

★★☆ Add two rules to postmortem-lint.cs: reject items that start with a vague verb such as "Improve" or "Better", and warn when a postmortem has no prevent or detect item at all. Test it with a fake postmortem file.

★★★ Add a bootstrap for the median to incident-metrics.cs and compute how often the median difference between the two quarters appears by chance. Then add one more incident to each quarter of the sample data and see how much the result moves.

Our own postmortems teach us our own system; other companies' postmortems teach us outages we have not had yet. In the next post we read the outages of Cloudflare in November 2025, Google Cloud in June 2025 and CrowdStrike in July 2024 from the companies' own postmortems, and try their common pattern, globally propagated configuration, on Bookstore's switches.json.