Distributed Transactions and Two-Phase Commit: Why They Are Avoided in Microservices

Version note: The code in this post was built and run with C# 14 and .NET 10.

Guess First

Bookstore's checkout has to do two things together: decrement the book's stock in the Inventory database and take money from the customer's wallet in the Payment database. If one happens without the other, you get either a customer who paid but has no book reserved, or a free book. The classic solution is two-phase commit (2PC): a coordinator asks both databases "are you ready to commit?", and if both say "yes", it says "commit".

Both databases say "yes" and lock the stock row and the wallet row. At that exact moment the machine running the coordinator loses power. What should Inventory and Payment do now? If they commit, what if the coordinator had decided to abort? If they abort, what if the other one committed? When are the locks released?

The Atomic Commit Problem

In a single database, the transaction log provides atomicity: if the commit record reached the disk, the transaction happened; if not, it did not. For a transaction spread over several nodes that is not enough; if each node decides on its own, one may commit while another aborts. The atomic commit problem requires all participants to reach the same outcome (all commit or all abort).

This problem resembles the consensus of the previous post but differs from it. In consensus a majority decision is enough and any proposed value may be chosen. In atomic commit every participant has a veto: if even one says "no" (a constraint violation, a full disk, a deadlock), the outcome must be abort. And commit is possible only if everyone said "yes". That makes atomic commit more fragile than consensus, which can make progress with a majority: a single unreachable participant or coordinator can stall the whole transaction.

The Two Phases of 2PC

The protocol, described by Jim Gray in his 1978 "Notes on Data Base Operating Systems", has two phases. In the first, the coordinator sends prepare to every participant. The participant brings the transaction to a point where it can commit: it durably records the changes and the locks, checks constraints, and then votes YES or NO. A YES vote is a promise: "if you say commit, I will commit, even if I crash and restart". After that promise the participant may not abort on its own.

If all votes are YES, the coordinator first writes its decision (COMMIT) to its own log; that record is the commit point. Then, in the second phase, it sends the decision to all participants and waits for acknowledgements. A single NO or a timeout makes the decision ABORT. Participants release their locks as they receive the decision. For a participant that has not received it, the coordinator keeps resending the decision, reading it from its log after a restart if necessary.

2PC in .NET: System.Transactions

Most of us use TransactionScope with a single database connection, but the System.Transactions underneath is a real 2PC coordinator. For a resource to take part in a transaction, it only needs to implement IEnlistmentNotification: Prepare is the first phase, Commit or Rollback the second. A participant that prints every call:

using System.Transactions;

namespace Bookstore.TwoPhase;

// A participant that votes in the 2PC run by System.Transactions
public sealed class LoggingResource(string name, bool voteYes = true) : IEnlistmentNotification
{
    public void Prepare(PreparingEnlistment preparingEnlistment)
    {
        Console.WriteLine($"  {name}: prepare -> {(voteYes ? "YES" : "NO")}");
        if (voteYes) preparingEnlistment.Prepared();
        else preparingEnlistment.ForceRollback();
    }

    public void Commit(Enlistment enlistment)
    {
        Console.WriteLine($"  {name}: commit");
        enlistment.Done();
    }

    public void Rollback(Enlistment enlistment)
    {
        Console.WriteLine($"  {name}: rollback");
        enlistment.Done();
    }

    public void InDoubt(Enlistment enlistment)
    {
        Console.WriteLine($"  {name}: in doubt");
        enlistment.Done();
    }
}

Two checkouts with two volatile participants (resources that keep their state in memory and need no recovery after a crash), then an attempt with two durable participants (resources that do need recovery after a crash, like a real database), and finally a small coordinator of our own:

using System.Transactions;
using Bookstore.TwoPhase;

static void Checkout(bool paymentVotesYes)
{
    try
    {
        using TransactionScope scope = new();
        Transaction.Current!.EnlistVolatile(new LoggingResource("Inventory"), EnlistmentOptions.None);
        Transaction.Current!.EnlistVolatile(new LoggingResource("Payment", paymentVotesYes), EnlistmentOptions.None);
        Console.WriteLine("  scope.Complete()");
        scope.Complete();
    } // Dispose runs phase 1 and phase 2
    catch (TransactionAbortedException ex)
    {
        Console.WriteLine($"  {ex.GetType().Name}: {ex.Message}");
    }
}

Console.WriteLine("--- both vote yes ---");
Checkout(paymentVotesYes: true);
Console.WriteLine("--- payment votes no ---");
Checkout(paymentVotesYes: false);

Console.WriteLine("--- two durable resource managers ---");
try
{
    using TransactionScope scope = new();
    Transaction.Current!.EnlistDurable(Guid.NewGuid(), new LoggingResource("InventoryDb"), EnlistmentOptions.None);
    Transaction.Current!.EnlistDurable(Guid.NewGuid(), new LoggingResource("PaymentDb"), EnlistmentOptions.None);
    scope.Complete();
}
catch (Exception ex)
{
    Console.WriteLine($"  {ex.GetType().Name}: {ex.Message}");
}

Console.WriteLine("--- hand-written 2PC, coordinator crash ---");
Participant inventory = new("Inventory", "row lock on Stock(978-1449373320)");
Participant payment = new("Payment", "row lock on Wallet(customer-7)");
new Coordinator([inventory, payment]).Run(crashBeforeDecision: true);
Console.WriteLine($"  {inventory.TryTerminate([payment])}");
Console.WriteLine($"  {payment.TryTerminate([inventory])}");

--- both vote yes ---
  scope.Complete()
  Inventory: prepare -> YES
  Payment: prepare -> YES
  Inventory: commit
  Payment: commit
--- payment votes no ---
  scope.Complete()
  Inventory: prepare -> YES
  Payment: prepare -> NO
  Inventory: rollback
  TransactionAbortedException: The transaction has aborted.
--- two durable resource managers ---
  PlatformNotSupportedException: This platform does not support distributed transactions.

The first two scenarios show the protocol as it is. Complete only says "I want to commit"; the real work starts in Dispose: two prepares first, then two commits. When Payment says NO, Inventory gets a rollback and a TransactionAbortedException is thrown; Payment, which voted NO, is not sent a separate rollback, because its vote already meant abort.

The third scenario shows the platform limitation. As long as there are only volatile participants or a single durable one, .NET's built-in lightweight transaction manager handles the work. The moment a second durable participant joins, the transaction has to be escalated to a distributed transaction, which requires the operating system's transaction coordinator. Since .NET 7 this is supported only on Windows, with MSDTC (Microsoft Distributed Transaction Coordinator) and with TransactionManager.ImplicitDistributedTransactions = true set; on macOS and Linux you get the PlatformNotSupportedException above. In real life this happens when you open two SqlConnections to two different databases inside the same TransactionScope. The Microservices series showed the same error with two real SQL (Structured Query Language) Server databases in "No Transactions Across Microservices: Embracing Eventual Consistency by Design".

When the Coordinator Crashes: Blocking

Back to the opening question. Our small coordinator crashes after collecting all the YES votes, before writing or sending the decision. After a while the participants time out and ask the other participants what to do:

namespace Bookstore.TwoPhase;

public enum State { Working, Prepared, Committed, Aborted }

public sealed class Participant(string name, string lockedRow)
{
    public string Name { get; } = name;
    public State State { get; private set; } = State.Working;
    public string? HeldLock { get; private set; } = lockedRow;

    // Once it votes YES the participant has promised to commit if told to: it may not abort alone
    public bool Prepare() { State = State.Prepared; return true; }
    public void Commit() { State = State.Committed; HeldLock = null; }
    public void Abort() { State = State.Aborted; HeldLock = null; }

    // Timeout in the prepared state: ask the others what the decision was
    public string TryTerminate(IEnumerable<Participant> peers)
    {
        if (State != State.Prepared) return $"{Name}: already {State}";
        if (peers.Any(p => p.State == State.Committed)) { Commit(); return $"{Name}: learned COMMIT from a peer"; }
        if (peers.Any(p => p.State is State.Aborted or State.Working)) { Abort(); return $"{Name}: safe to ABORT"; }
        return $"{Name}: BLOCKED, still holding {HeldLock}";
    }
}

public sealed class Coordinator(IReadOnlyList<Participant> participants)
{
    public void Run(bool crashBeforeDecision)
    {
        bool allYes = participants.All(p => p.Prepare());
        Console.WriteLine($"  phase 1: votes = {(allYes ? "all YES" : "some NO")}");
        if (crashBeforeDecision)
        {
            Console.WriteLine("  coordinator crashes before writing/sending the decision");
            return;
        }
        foreach (Participant p in participants)
        {
            if (allYes) p.Commit(); else p.Abort();
        }
        Console.WriteLine($"  phase 2: {(allYes ? "COMMIT" : "ABORT")} sent");
    }
}

--- hand-written 2PC, coordinator crash ---
  phase 1: votes = all YES
  coordinator crashes before writing/sending the decision
  Inventory: BLOCKED, still holding row lock on Stock(978-1449373320)
  Payment: BLOCKED, still holding row lock on Wallet(customer-7)

Both participants are stuck. The reason lies in what a YES vote means: Inventory knows Payment also voted YES, but it does not know whether the coordinator wrote COMMIT to its log. If the coordinator crashed after the commit point (the decision is in the log but the message never left), aborting breaks atomicity; if it crashed before, committing may contradict the abort decision the coordinator will make when it restarts. The only correct answer is to wait. Had one participant voted NO or received the decision, the other could have learned it from that participant (cooperative termination); when everyone voted YES and is waiting, that does not help either.

The locks stay held during this wait. Every query that wants to touch the stock row waits; if the coordinator does not come back for hours, that part of the system stops for hours. SQL Server marks transactions in this state as "in-doubt" and, if needed, waits for an administrator to commit or abort them manually through MSDTC. If the manual decision contradicts the coordinator's real decision, atomicity is broken. This property of 2PC is called blocking, and it is the protocol's biggest weakness.

2PC vs Consensus, and Fault-Tolerant 2PC

The root of the problem is that the decision lives on a single node, the coordinator. The fix comes from the previous post: replicate the decision with consensus. Paxos Commit, from the 2006 paper "Consensus on Transaction Commit" by Jim Gray and Leslie Lamport, records each participant's vote on several acceptors with Paxos; even if one coordinator crashes, a majority knows the decision. Google Spanner applies the same idea: every participant is already a Paxos group, 2PC runs between the leaders of those groups, and because each group's state is replicated with Paxos, a single machine crashing does not block the transaction.

Three-phase commit (3PC), proposed by Dale Skeen in 1981, tries to remove blocking by adding a phase in between, but it can only do so under a synchronous network assumption with bounded message delays. As we saw in the second post, real networks do not keep that assumption, which is why 3PC never caught on in practice.

XA and Heterogeneous Distributed Transactions

To let different products take part in the same 2PC, the X/Open group published the XA (eXtended Architecture) standard in 1991: the interface between a transaction manager and resource managers (prepare, commit, rollback, recover). JTA (Jakarta Transactions) in Java, MySQL's XA START commands and PostgreSQL's PREPARE TRANSACTION command belong to this family. On Windows, MSDTC speaks both its own protocol and XA.

Good on paper, troublesome in practice. The transaction manager is a single point of failure and is usually not replicated itself. All participants run at the speed of the weakest link; if one participant's prepare takes long, everyone holds locks. Recovering in-doubt transactions requires operational know-how. And most importantly, a large part of modern infrastructure does not support XA: RabbitMQ and Kafka cannot be XA participants; Kafka's own transactions cover only what happens inside Kafka.

Why Microservices Avoid It

In a microservice architecture each service owns its data (database per service). Putting 2PC between services punches a hole in that ownership: the Order service's transaction holds locks in the Payment service's database, and one service's deployment or slowness stalls the other's transactions. Availability drops multiplicatively too: the probability that three participants, each 99.9 percent available, are all available at the same time is about 99.7 percent, and lower still once you add the coordinator. On top of that comes the platform limitation: in .NET there are no distributed transactions in Linux containers, and brokers do not speak XA.

That is why microservices design two things instead of atomicity. For consistency between a service's own database write and the event it publishes, the transactional outbox: the event is written to a table in the same local transaction as the business data, and a separate process moves it to the broker. For a flow spanning several services, the saga: every step is its own local transaction, and a failed step undoes the earlier steps with compensating actions. Unlike 2PC, a saga provides no isolation: intermediate states are visible to others, and designing for that is the domain's job. The details are in the Messaging and Event-Driven Architecture series, in "Transactional Outbox and Inbox: Understanding and Designing Around the Dual Write Problem" and "The Saga Pattern: Choreography or Orchestration?".

Common Pitfalls

Opening two different connections inside a TransactionScope. Even with the same connection string, if the second is opened before the first is closed, the transaction may escalate. On Linux that is an exception; on Windows an unexpected MSDTC dependency.

Making a remote HTTP (Hypertext Transfer Protocol) call inside the transaction. TransactionScope does not cover the HTTP call; the payment provider takes the money, then the local transaction rolls back and the two are inconsistent. Move the side effect after the transaction with an outbox.

Forgetting prepared transactions. In PostgreSQL a transaction prepared with PREPARE TRANSACTION and never committed or rolled back blocks its locks and vacuum indefinitely. If you use 2PC, an alert that watches in-doubt transactions is a must.

Mistaking 2PC for "exactly-once". 2PC makes a transaction atomic; it does not stop a client that timed out from starting the same transaction again. The need for idempotency from the first post applies here too.

Treating sagas as a free version of 2PC. A saga removes blocking but takes isolation away with it. A saga without compensating actions, or one that does not put irreversible steps last, produces half-finished operations.

A Small Experiment

★☆☆ Remove the scope.Complete() line in Checkout. Which calls reach the participants? Is prepare called?

★★☆ Add a step to Coordinator.Run that writes the decision to a file, and set up the scenario in which the coordinator crashes after that step. How does a restarted coordinator recover the participants by reading the file? If it crashed before writing the file, what should the decision be (presumed abort)?

★★★ Simulate a simplified version of Paxos Commit: assume the coordinator writes its decision to three separate "decision nodes" and that the decision is final once a majority has it. When a decision node and the coordinator crash at the same time, can the participants learn the decision? When does blocking come back?

In 2PC the participants waited while holding locks. In the next post we look at locks in the distributed world: distributed locks, leases, and how a process that believes it still holds the lock overwrites someone else's data because of a GC (garbage collector) pause. The fix is called a fencing token.