Root Cause Analysis: 5 Whys, Fishbone and Blameless Postmortems

Version note: The code in this post was built and run with .NET 10; the analyzer example uses Microsoft.CodeAnalysis.BannedApiAnalyzers 5.6.0 and the tests were run with xUnit v3 (xunit.v3 4.0.2).

Guess First

The order total bug stayed in production for 10 days: v1.9.0, which contained the faulty commit, was deployed on Friday 11 September; accounting opened the ticket on Monday 21 September and the fix went out the same evening. In those days the Bookstore takes about 350 orders a day, some of the lines are discounted, and prices mostly end in .00, .45, .50, .90 and .95.

What percentage of orders do you think were charged the wrong amount: 1%, 5%, 20%? And one more: if your answer to "what is the root cause of this bug?" is "the developer doing the refactoring forgot an argument", by the end of this post you will see why we find that answer insufficient.

Symptom, Proximate Cause and Root Cause

The first ten posts of the series were about finding a bug. This post is about the question to ask after finding it: why was this bug possible, and how does it become impossible next time? Let's separate three concepts first, because most RCA (Root Cause Analysis) discussions go nowhere precisely because they mix these three up.

The symptom is what's visible from outside: in the reconciliation report some orders were charged 1 kuruş less than the invoice. The proximate cause is the technical event that directly produces the symptom: the Math.Round(x, 2) call in PriceCalculator rounds midpoint values with MidpointRounding.ToEven; in post 03 of this series we found with git bisect run that it came with the "Extract PriceCalculator" refactoring. The root cause is the set of conditions that allowed the proximate cause to appear and reach production unnoticed. It can rarely be stated in a single sentence.

Why does the distinction matter? Because each level has a different fix. Fixing the symptom means explaining the differences to accounting. Fixing the proximate cause means putting the missing argument back; we did that with one line in post 02 of the series. Fixing the root cause means preventing the next bug of the same class too.

5 Whys: Its Strength and Its Limits

5 Whys is a simple technique that comes from the Toyota Production System and was popularised by Taiichi Ohno: you start from the symptom and keep asking "why?" until the answer rests on a process or a design decision. The number five isn't a rule; it's a reminder not to stop at the first answer. In our case the chain looks like this:

1. Why was the customer charged 1 kuruş less? Because checkout rounded the 11.205 TL line price to 11.20, while the invoice service rounded it to 11.21.

2. Why did checkout round to 11.20? Because PriceCalculator uses the Math.Round(decimal, int) overload, and that overload rounds midpoint values to the nearest even number.

3. Why is that overload used? Because in the "Extract PriceCalculator" refactoring, the MidpointRounding.AwayFromZero argument was dropped while moving the code out of OrderService. Both overloads compile, so the compiler said nothing.

4. Why wasn't the change caught before the merge? Because none of the tests contained a midpoint value, and the PR (pull request) was approved quickly with the description "pure refactoring, no behaviour change".

5. Why did the system's correctness depend on one argument being written correctly? Because in the Bookstore money is a bare decimal; the rounding rule lives in no type and no rule, and is written by hand, separately, in checkout and in the invoice service.

The fifth answer points to a design decision, not to a person; we stopped at the right place. But look at the limits of the chain too. At step four the chain actually branched: tests, review and the compiler were three separate reasons, and we had to pick one. 5 Whys produces a single linear chain, while real incidents usually consist of several factors feeding each other. The second limit is that the answer depends on who asks: someone else on the same team could have ended the chain at a person after step three by saying "because the developer was careless". A chain that ends at a person ends with an action that doesn't work, like "be more careful".

Seeing Multiple Factors with a Fishbone

The fishbone diagram is a tool Kaoru Ishikawa developed for quality control, also called the Ishikawa diagram after him. The symptom goes at the head of the fish; possible causes are grouped into categories and laid out along the bones. The classic manufacturing categories (people, machines, methods, materials, measurement, environment) don't fit software well, so I use code, design, tests, review, monitoring and knowledge:

The diagram shows what 5 Whys couldn't fit into a single chain: this bug has no single root. A dangerous default at the code level, an ownerless rule at the design level, missing data in the tests, the "refactoring is harmless" assumption in review, a 10-day blind spot in monitoring, and the team not knowing the default of Math.Round. Had any one of these been different, the bug would either never have happened or been caught within hours. A good RCA spreads its actions across each of these bones.

A practical note: don't fill in the fishbone on your own. Filling it in during the incident meeting with the team, starting with a session where everyone writes down the factors on their mind while the categories are still empty, surfaces bones a single person can't see. Someone from accounting saying "we look at the reconciliation report once a week" turned out to be the real source of the monitoring bone.

Measuring the Impact

The first question of a postmortem is "how bad was it?" I'm not using real order data in this post; I estimate the impact by replaying 10 days of traffic with a seeded order generator. In a real incident you'd do the same calculation with a query that recalculates the lines in the orders table under both rules:

// Replays the 10 affected days with a seeded order generator: how many orders differ?
Random random = new(4127);
decimal[] discounts = [0.00m, 0.00m, 0.00m, 0.10m, 0.15m];
decimal[] endings = [0.00m, 0.45m, 0.50m, 0.90m, 0.95m];
int orders = 10 * 350, affected = 0;
decimal undercharged = 0;

for (int i = 0; i < orders; i++)
{
    decimal buggy = 0, correct = 0;
    int lineCount = random.Next(1, 5);
    for (int l = 0; l < lineCount; l++)
    {
        decimal price = random.Next(20, 400) + endings[random.Next(endings.Length)];
        decimal raw = price * (1 - discounts[random.Next(discounts.Length)]);
        int quantity = random.Next(1, 4);
        buggy += Math.Round(raw, 2, MidpointRounding.ToEven) * quantity;
        correct += Math.Round(raw, 2, MidpointRounding.AwayFromZero) * quantity;
    }
    if (buggy != correct)
    {
        affected++;
        undercharged += correct - buggy;
    }
}

Console.WriteLine($"orders = {orders}, affected = {affected} ({affected * 100.0 / orders:0.0}%), undercharged = {undercharged:0.00} TL");

$ dotnet run
orders = 3500, affected = 634 (18.1%), undercharged = 14.39 TL

The answer to the opening question: 634 of 3,500 orders, that is 18.1%. In the first post 5% of prices showed a difference; at the order level the rate is much higher, because an order has several lines, price endings aren't random, and one affected line is enough to affect the order. The total difference, though, is only 14.39 TL. There's an asymmetry too: for a positive amount AwayFromZero always rounds a midpoint up, while ToEven rounds it up sometimes and down sometimes. So the amount checkout charges can never be more than the invoice, only less. Because the difference was always in the customer's favour, no customer complained; the signal only came from accounting's weekly reconciliation. A loss of 14 TL is small, but needing a corrective invoice or an explanation for each of 634 orders means days of work for the accounting team.

Levels of Fix: From Hotfix to Making It Impossible

Thinking about the fix at three levels makes the "what will we do?" part of the RCA concrete.

Level one is the hotfix: putting the missing argument back. We did it in post 02 together with the failing test, and it was deployed the same evening as v1.10.1. A hotfix's job is to stop the bleeding; on its own it doesn't prevent the next similar bug.

Level two is the regression test: tests that catch this bug and its variations of the same class. Testing the ticket's order verbatim is a good start, but a single example only protects that example. So we also add a test that compares the whole price range, at several discount rates, against an independent reference calculation:

using Bookstore.Ordering;
using Xunit;

namespace Bookstore.Ordering.Tests;

public class OrderTotalRegressionTests
{
    [Fact]
    public void Order_10482_matches_the_invoice()
    {
        List<OrderLine> lines =
        [
            new("978-605-0000-01-1", Money.Of(12.45m), 1, 0.10m),
            new("978-605-0000-02-8", Money.Of(89.90m), 1, 0.00m),
            new("978-605-0000-03-5", Money.Of(45.50m), 2, 0.15m),
        ];

        Assert.Equal(Money.Of(178.47m), PriceCalculator.OrderTotal(lines));
    }

    [Theory]
    [InlineData(0.05)]
    [InlineData(0.10)]
    [InlineData(0.15)]
    [InlineData(0.25)]
    public void Every_price_rounds_midpoint_away_from_zero(decimal rate)
    {
        for (decimal price = 0.01m; price <= 999.99m; price += 0.01m)
        {
            decimal raw = price * (1 - rate);
            decimal expected = decimal.Floor(raw * 100 + 0.5m) / 100;   // reference: school rounding for positive amounts

            Assert.Equal(expected, Money.Of(price).ApplyDiscount(rate).Amount);
        }
    }
}

The way to see a test really catch the bug is to bring the bug back on purpose. I temporarily switched the rounding in the Money type below to ToEven and ran the tests:

$ dotnet test | grep -E "^failed|Expected|Actual|total:|failed:"
failed Bookstore.Ordering.Tests.OrderTotalRegressionTests.Every_price_rounds_midpoint_away_from_zero(rate: 0.15) (9ms)
  Expected: 0.09
  Actual:   0.08
failed Bookstore.Ordering.Tests.OrderTotalRegressionTests.Every_price_rounds_midpoint_away_from_zero(rate: 0.05) (0ms)
  Expected: 0.29
  Actual:   0.28
failed Bookstore.Ordering.Tests.OrderTotalRegressionTests.Every_price_rounds_midpoint_away_from_zero(rate: 0.1) (0ms)
  Expected: 0.05
  Actual:   0.04
failed Bookstore.Ordering.Tests.OrderTotalRegressionTests.Every_price_rounds_midpoint_away_from_zero(rate: 0.25) (0ms)
  Expected: 0.05
  Actual:   0.04
failed Bookstore.Ordering.Tests.OrderTotalRegressionTests.Order_10482_matches_the_invoice (2ms)
  Expected: 178.47
  Actual:   178.46
  total: 5
  failed: 5

All five tests are red, each with a different example. When I reverted the change, they all turned green. This small experiment is the test of the test itself.

Level three is the change that makes this class of bug impossible. The root cause was "the rounding rule has no owner"; so we give the rule an owner. Money is no longer a bare decimal but a value object with a single rounding policy:

namespace Bookstore.Ordering;

public readonly record struct Money
{
    public decimal Amount { get; }

    private Money(decimal amount) => Amount = amount;

    // The single rounding policy of the Bookstore: 2 decimals, midpoint away from zero
    public static Money Of(decimal amount) =>
        new(Math.Round(amount, 2, MidpointRounding.AwayFromZero));

    public Money ApplyDiscount(decimal rate) => Of(Amount * (1 - rate));

    public static Money operator +(Money left, Money right) => new(left.Amount + right.Amount);
    public static Money operator *(Money money, int quantity) => new(money.Amount * quantity);

    public override string ToString() => Amount.ToString("0.00", System.Globalization.CultureInfo.InvariantCulture);
}

Money's constructor is private; the only way to create a Money from an unrounded amount from outside is Of, and Of always rounds by the same rule. PriceCalculator no longer knows anything about rounding:

namespace Bookstore.Ordering;

public sealed record OrderLine(string Isbn, Money UnitPrice, int Quantity, decimal DiscountRate);

public static class PriceCalculator
{
    public static Money DiscountedUnitPrice(OrderLine line) => line.UnitPrice.ApplyDiscount(line.DiscountRate);

    public static Money LineTotal(OrderLine line) => DiscountedUnitPrice(line) * line.Quantity;

    public static Money OrderTotal(IEnumerable<OrderLine> lines) =>
        lines.Aggregate(Money.Of(0), (total, line) => total + LineTotal(line));
}

A type makes the right path easy but doesn't close the wrong one: tomorrow someone can write Math.Round(x, 2) again in another calculation. To close that at compile time we use Microsoft's BannedApiAnalyzers package. The package produces the RS0030 diagnostic on every line that uses an API (Application Programming Interface) listed in the BannedSymbols.txt file; we make it an error instead of a warning:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <WarningsAsErrors>RS0030</WarningsAsErrors>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.CodeAnalysis.BannedApiAnalyzers" Version="5.6.0" PrivateAssets="all" />
    <AdditionalFiles Include="BannedSymbols.txt" />
  </ItemGroup>
</Project>

M:System.Math.Round(System.Decimal);Use Money: money is rounded only inside Money
M:System.Math.Round(System.Decimal,System.Int32);Use Money: money is rounded only inside Money

The two banned overloads are the ones that silently leave the rounding rule to the default. The three-parameter overload that takes a MidpointRounding is allowed; someone who writes the rule explicitly is making a deliberate choice. Now if someone writes a new calculation for the shipping fee and calls Math.Round out of old habit:

$ dotnet build
ShippingFeeCalculator.cs(5,52): error RS0030: The symbol 'Math.Round(decimal, int)' is banned in this project: Use Money: money is rounded only inside Money (https://github.com/dotnet/roslyn/blob/main/src/RoslynAnalyzers/Microsoft.CodeAnalysis.BannedApiAnalyzers/BannedApiAnalyzers.Help.md) [Bookstore.Ordering.csproj]

The build breaks and the message says what to do. Had the "Extract PriceCalculator" commit been written with this rule in place, it wouldn't have built on the developer's own machine, before any PR was opened. The final state with the tests and the type:

$ dotnet test
./tests/Bookstore.Ordering.Tests/bin/Debug/net10.0/Bookstore.Ordering.Tests.dll (net10.0|arm64) passed (718ms)
Test run summary: Passed!
  total: 5
  failed: 0
  succeeded: 5
  skipped: 0

The last step is not to stop at one service. The invoice service applies the same rule with separate code; as long as the rule lives in two places, it can diverge in two places. Turning Money into a shared package used by both services gives the rule a single source.

A Blameless Postmortem: The Order Total Case

A postmortem is the written record of an incident: what happened, what the impact was, how it was resolved, what its root causes were and what will be done so it doesn't happen again. The "Postmortem Culture: Learning from Failure" chapter of Google's SRE (Site Reliability Engineering) book describes a blameless postmortem as a record that lays out the contributing causes without blaming any person or team, assuming everyone acted in good faith with the information they had at the time. The goal is fixing the system, not punishment. The same chapter also requires postmortems to be reviewed by senior engineers and the results to be shared; a postmortem nobody reads might as well not exist.

Being blameless doesn't mean hiding who did what. The timeline says "the refactoring commit was merged"; it doesn't say "X forgot". The difference is subtle but decisive: when people think they will be blamed they withhold information, and a postmortem built on withheld information yields no root cause. Below is the postmortem for this case, in the template I use.

Summary

Between 11 and 21 September 2026, checkout rounded discounted lines whose raw price fell exactly between two kuruş values differently from the invoice. For the affected orders the amount charged was 1 or a few kuruş less than the invoice. The problem was noticed on 21 September through accounting's reconciliation ticket and was resolved the same day with v1.10.1.

Impact

An estimated 634 orders (18.1% of 10 days of traffic), 14.39 TL undercharged in total. No visible error on the customer side; on the accounting side the invoice and payment records of 634 orders need reconciling.

Timeline

Root cause and contributing factors

The proximate cause is the MidpointRounding.AwayFromZero argument being dropped during the refactoring. The root cause is that the rounding rule for money amounts wasn't defined in a single place: the rule was written by hand in two services, and neither the compiler, the tests nor the types protected it. Contributing factors: no midpoint values in the test data, refactoring PRs not being reviewed separately for behaviour changes, reconciliation differences being monitored weekly rather than daily, and the default of Math.Round not being widely known in the team.

What went well, what went badly

Went well: the ticket had the order number and both amounts, so reproducing took less than an hour. A debugging journal was kept, and the rejected hypotheses were on record. Bisect produced a result within minutes thanks to a deterministic check program. Went badly: the bug went unnoticed for 10 days. The first reaction was to suspect the payment provider, which cost half an hour.

Action items

• A Money value object for money amounts with a single rounding policy; owner Ordering team, 2 October
• Move Money into a package shared with the invoice service; owners Ordering and Invoicing teams, 16 October
• Ban the default rounding overloads with BannedApiAnalyzers; owner Platform team, 28 September
• A regression test that scans the whole price range; owner Ordering team, 25 September (done)
• A job that compares charged and invoiced totals daily and raises an alert on any difference; owner Platform team, 9 October
• Send the list of the 634 affected orders to accounting; owner Ordering team, 22 September (done)

Every action has an owner and a date; an action without an owner is forgotten the moment it is written. Look at how the actions are spread, too: five of the six bones in the fishbone get at least one action. The action for the "knowledge" bone is this postmortem itself: a postmortem that is shared and read within the team makes sure everyone learns the default of Math.Round.

In Production

Decide in advance when you'll write a postmortem. The SRE book suggests triggers such as user-visible downtime beyond a set threshold, any data loss, manual intervention such as a rollback, resolution taking longer than a set time, and incidents that monitoring missed and people noticed. Our case fits the last trigger exactly: accounting found the bug, not monitoring. The incident management process (on-call, severity levels, communication) is covered in the upcoming SRE and Production Incidents series.

Write the postmortem within a few days of the incident, before the details fade. The debugging journal and the ticket history are the raw material of the timeline. A postmortem shouldn't contain customer data, the names of order owners or secrets; the list of affected orders goes into a separate record with restricted access, not into the postmortem.

Track the actions. A postmortem's value appears when its actions are closed. Seeing open actions in the same backlog as other work during sprint planning breaks the "we wrote a postmortem and forgot about it" cycle.

Common Pitfalls

The most common pitfall is ending the root cause at a person. "The developer forgot the argument" is a true sentence, but it isn't a root cause. People make mistakes; the system shouldn't allow a single human error to affect 634 orders over 10 days. That's the answer to the second opening question.

The second pitfall is fixing the proximate cause and stopping. Once the hotfix is deployed the pressure goes away and the RCA gets postponed. When the next similar bug arrives, you write the same postmortem again.

The third pitfall is treating 5 Whys like an exam with a single right answer. Two people can arrive at the same incident with two different chains; that's not a problem, it's a sign to switch to the fishbone.

The fourth pitfall is actions that can't be measured. "Reviews will be more careful" can never be closed, because nobody can say when it's done. "A test comparing behaviour is mandatory for refactoring PRs" or "RS0030 enabled as an error" can be closed.

The fifth pitfall is not calculating the impact. An incident closed as "1 kuruş, negligible" leaves behind 634 orders' worth of accounting work and a difference that has to be explained at the next audit.

A Small Experiment

★☆☆ Write the 5 Whys chain for a bug you've had recently. At which step did the chain tend to end at a person? At that step change the question to "why did the system allow this?" and keep going.

★★☆ Set up BannedApiAnalyzers in your own project and ban DateTime.Now (P:System.DateTime.Now). In how many places does the build break? Estimate the cost of moving each one to TimeProvider.

★★★ Add a test to Money for negative amounts (refunds). Where does AwayFromZero round -11.205, and should the invoice service apply the same rule to refunds? Write the decision like a postmortem action item: with an owner and a date.

We've covered the method, the tools and the postmortem. In the last two posts of the series we apply them to real-world cases; the next post has five bug hunts from the backend: a comparison that breaks under Turkish culture, a report shifted by 3 hours, socket exhaustion, a key that changes inside a Dictionary, and an async void that crashes the process.