No Transactions Across Microservices: Embracing Eventual Consistency by Design

Version note: The code in this post was built and run with C# 14 and .NET 10.

Guess First

Inventory and Ordering are now separate services at Bookstore, each with its own database. Checkout has to do two things together: reserve a copy in Inventory and create the order in Ordering. Someone on the team proposes the solution they know from the monolith days: "Let's put both inside a TransactionScope; if one fails, both are rolled back." In the Dapper series post "Transactions with Dapper: BeginTransaction, Commit, Rollback and TransactionScope" we used this class with a single database.

The code runs in a Linux container or on a Mac. The two connections go to two separate databases. What do you think happens: are both committed together, does an exception arrive, or is the first operation done and the second lost? What would change on Windows? And most importantly: even if it worked, should we want it?

Why Don't We Use Distributed Transactions?

Let's try first. Two separate databases (one per service), a single TransactionScope:

using System.Transactions;
using Microsoft.Data.SqlClient;

// Connection info from an environment variable; two databases of two separate services
string server = Environment.GetEnvironmentVariable("BOOKSTORE_SQL")!;
string inventoryDb = server + ";Database=ms_05_inventory";
string orderingDb = server + ";Database=ms_05_ordering";

try
{
    // The "let's put both in one transaction" attempt
    using TransactionScope scope = new(TransactionScopeAsyncFlowOption.Enabled);

    await using (SqlConnection inventory = new(inventoryDb))
    {
        await inventory.OpenAsync();
        await new SqlCommand(
            "UPDATE dbo.StockItems SET Reserved = Reserved + 1 WHERE Isbn = '978-0441013593'", inventory)
            .ExecuteNonQueryAsync();
    }

    await using (SqlConnection ordering = new(orderingDb))
    {
        await ordering.OpenAsync(); // second resource: the transaction wants to be promoted to distributed
        await new SqlCommand(
            "INSERT dbo.Orders (Id, Status) VALUES (NEWID(), 'Placed')", ordering).ExecuteNonQueryAsync();
    }

    scope.Complete();
    Console.WriteLine("Committed both");
}
catch (Exception ex)
{
    Console.WriteLine($"{ex.GetType().Name}: {ex.Message}");
}

PlatformNotSupportedException: This platform does not support distributed transactions.

When the second connection opened, the transaction had two resources and .NET tried to promote it to a distributed transaction. Since .NET 7 this is supported only on Windows; even there it needs MSDTC (Microsoft Distributed Transaction Coordinator), and the application must explicitly set TransactionManager.ImplicitDistributedTransactions to true. On Linux and macOS the answer is a clear exception. Looking at both tables after the exception, we see the first UPDATE was rolled back too; no inconsistent half-done work remains:

SELECT Reserved FROM ms_05_inventory.dbo.StockItems WHERE Isbn = '978-0441013593';
SELECT COUNT(*) AS Orders FROM ms_05_ordering.dbo.Orders;

Reserved   
-----------
          0

(1 rows affected)
Orders     
-----------
          0

(1 rows affected)

So, on Windows with MSDTC, should we have used it? In a microservices context the answer is usually no, and platform support isn't the reason. The protocol behind distributed transactions is 2PC (two-phase commit): a coordinator first asks every participant "are you ready to commit?", and if all say "yes", it says "commit" in the second phase. This simple protocol has three costs in microservices.

The first is lock duration. From the moment participants say "ready" until the coordinator's decision arrives, they keep the affected rows locked. If the coordinator or the network has a problem in between, Inventory's stock row stays locked because of another service's problem. The second is availability: the operation can only complete if all participants are up at the same time; the chain multiplication we saw in earlier posts applies here too. The third, and the decisive one, is the participants: Bookstore's checkout involves the payment provider's HTTP (Hypertext Transfer Protocol) API (Application Programming Interface), a message broker and an email provider. None of them will join a 2PC run by your coordinator. That is the main idea of Pat Helland's 2007 paper "Life beyond Distributed Transactions: an Apostate's Opinion": systems that scale cannot rely on atomic transactions that span beyond a single machine, and applications have to be designed accordingly.

In the Messaging and Event-Driven Architecture series post "Transactional Outbox and Inbox: Understanding and Designing Around the Dual Write Problem" we also discussed why 2PC is not the answer from the message broker's point of view; the protocol's theory is the subject of the planned Distributed Systems Fundamentals series. This post's question is more architectural: without transactions, how do we design for consistency?

The Boundary Test: A Rule That Needs Consistency Lives in One Service

The first answer is drawing the boundary correctly. Every rule that must be consistent immediately (every invariant) should live inside one service, ideally inside one aggregate. In the DDD (Domain-Driven Design) series post "Aggregate Design: Consistency Boundaries, Small Aggregates, and Referencing by Id" we defined the aggregate exactly this way: the smallest boundary that must stay consistent within one transaction. A service boundary is wider than an aggregate boundary; so if a rule fits in one aggregate, it fits in one service.

Examples from Bookstore. "Stock never goes negative" lives inside Inventory's StockItem aggregate: a copy is reserved only if it is free. "An order's total is the sum of its lines" and "a coupon is used once per order" live inside Ordering's Order aggregate. None of these requires locking two services at once.

What about "at most 100 copies of a campaign book are sold"? There are two options here, and the decision defines the boundary. If the rule is about stock (100 copies are set aside in the warehouse for the campaign), it lives in Inventory and becomes part of the reservation. If it is about pricing (the first 100 orders get the discount), it lives in Ordering as a campaign counter. The wrong answer is putting half the rule in one and half in the other and trying to keep both consistent at once. If you see a rule that really needs two services' data immediately, that's a design signal: the boundary may be in the wrong place. The last criterion in Microsoft's boundary guide says the same: maintaining consistency sometimes means grouping the related functionality into a single service.

A Decision-Level Look at Sagas

For a workflow that spans several services, the remaining tool is the saga. The idea comes from "Sagas", the paper Hector Garcia-Molina and Kenneth Salem presented at SIGMOD (Special Interest Group on Management of Data) in 1987: split a long operation into a series of local transactions, each committed on its own; if a step fails, make up for the completed steps with compensating actions. Bookstore's checkout is a saga: Ordering creates the order, Inventory reserves stock, Payment takes the money. If payment fails, Inventory releases the stock it reserved and Ordering cancels the order.

Let's build the flow with choreography on a small in-memory bus: each service does its own work and publishes an event, and the next step happens in the service listening to that event:

namespace Bookstore.Contracts;

// Integration events between services (each published at the end of its own service's local transaction)
public sealed record OrderPlaced(Guid OrderId, string Isbn, int Quantity, decimal Total);
public sealed record StockReserved(Guid OrderId);
public sealed record OutOfStock(Guid OrderId);
public sealed record PaymentCaptured(Guid OrderId);
public sealed record PaymentFailed(Guid OrderId, string Reason);

using Bookstore.Contracts;

namespace Bookstore;

// The simplest in-memory bus standing in for a broker: hands each published event to its subscribers in turn
public sealed class Bus
{
    private readonly Dictionary<Type, List<Action<object>>> _handlers = [];
    private readonly Queue<object> _pending = new();

    public void Subscribe<T>(Action<T> handler)
    {
        if (!_handlers.TryGetValue(typeof(T), out List<Action<object>>? list))
            _handlers[typeof(T)] = list = [];
        list.Add(e => handler((T)e));
    }

    public void Publish(object e) => _pending.Enqueue(e);

    public void Drain()
    {
        while (_pending.TryDequeue(out object? e))
        {
            Console.WriteLine($"  -> {e}");
            foreach (Action<object> h in _handlers.GetValueOrDefault(e.GetType(), [])) h(e);
        }
    }
}

public sealed class Ordering(Bus bus)
{
    public Dictionary<Guid, string> Status { get; } = [];

    public void Place(Guid orderId, string isbn, int quantity, decimal total)
    {
        Status[orderId] = "Placed"; // this is where the user is told "we've received your order"
        bus.Publish(new OrderPlaced(orderId, isbn, quantity, total));
    }

    public void Wire()
    {
        bus.Subscribe<PaymentCaptured>(e => Status[e.OrderId] = "Paid");
        bus.Subscribe<OutOfStock>(e => Status[e.OrderId] = "Cancelled(OutOfStock)");
        bus.Subscribe<PaymentFailed>(e => Status[e.OrderId] = "Cancelled(PaymentFailed)");
    }
}

public sealed class Inventory(Bus bus, int onHand)
{
    private readonly Dictionary<Guid, int> _reservations = [];
    public int Available => onHand - _reservations.Values.Sum();

    public void Wire()
    {
        bus.Subscribe<OrderPlaced>(e =>
        {
            if (Available < e.Quantity) { bus.Publish(new OutOfStock(e.OrderId)); return; }
            _reservations[e.OrderId] = e.Quantity;
            bus.Publish(new StockReserved(e.OrderId));
        });
        // Compensating action: release the reserved stock if payment failed
        bus.Subscribe<PaymentFailed>(e => _reservations.Remove(e.OrderId));
    }
}

public sealed class Payments(Bus bus, Func<Guid, bool> providerAccepts)
{
    public void Wire() =>
        bus.Subscribe<StockReserved>(e => bus.Publish(providerAccepts(e.OrderId)
            ? new PaymentCaptured(e.OrderId)
            : new PaymentFailed(e.OrderId, "card declined")));
}

using System.Globalization;
using Bookstore;

CultureInfo.CurrentCulture = CultureInfo.InvariantCulture;
Guid ada = Guid.Parse("0199f0c4-5c1e-7000-8000-00000000000a");
Guid alan = Guid.Parse("0199f0c4-5c1e-7000-8000-00000000000b");

Bus bus = new();
Ordering ordering = new(bus);
Inventory inventory = new(bus, onHand: 3);
Payments payments = new(bus, providerAccepts: orderId => orderId != alan);
ordering.Wire(); inventory.Wire(); payments.Wire();

Console.WriteLine("Ada orders 2 x Dune");
ordering.Place(ada, "978-0441013593", 2, 25.00m);
bus.Drain();
Console.WriteLine("Alan orders 1 x Dune (card will be declined)");
ordering.Place(alan, "978-0441013593", 1, 12.50m);
bus.Drain();

foreach ((Guid id, string status) in ordering.Status)
    Console.WriteLine($"{(id == ada ? "Ada " : "Alan")}: {status}");
Console.WriteLine($"Dune available in Inventory: {inventory.Available}");

Ada orders 2 x Dune
  -> OrderPlaced { OrderId = 0199f0c4-5c1e-7000-8000-00000000000a, Isbn = 978-0441013593, Quantity = 2, Total = 25.00 }
  -> StockReserved { OrderId = 0199f0c4-5c1e-7000-8000-00000000000a }
  -> PaymentCaptured { OrderId = 0199f0c4-5c1e-7000-8000-00000000000a }
Alan orders 1 x Dune (card will be declined)
  -> OrderPlaced { OrderId = 0199f0c4-5c1e-7000-8000-00000000000b, Isbn = 978-0441013593, Quantity = 1, Total = 12.50 }
  -> StockReserved { OrderId = 0199f0c4-5c1e-7000-8000-00000000000b }
  -> PaymentFailed { OrderId = 0199f0c4-5c1e-7000-8000-00000000000b, Reason = card declined }
Ada : Paid
Alan: Cancelled(PaymentFailed)
Dune available in Inventory: 1

In Alan's order the payment was declined; when Inventory heard PaymentFailed it released the copy it had reserved, and Ordering cancelled the order with the reason PaymentFailed. One copy is left in stock, because only Ada's 2 copies are reserved. At no point were two services in the same transaction; each changed its own data in its own local transaction.

Three decisions need care. Compensation is not a rollback: what is undone is a business operation, it leaves a trace and sometimes can't be undone at all (like a sent email). That's why the order of steps matters; steps that can't be undone go as late as possible. Second, choreography or orchestration: in this three-step flow, coordination through events reads well enough, but as steps and branches grow, an orchestrator (a state machine) that shows the whole flow in one place becomes easier to follow. Third, isolation: a saga's intermediate states are visible to other operations; Alan's copy appears reserved for a few seconds, and in the meantime another customer may be told "out of stock". The details of these decisions, the pivot step concept and saga anomalies are in the Messaging series post "The Saga Pattern: Choreography or Orchestration?", and implementing one in .NET with a persisted state machine is in "Implementing a Saga in .NET: Persisted State Machines, Timeouts, and Compensation".

Outbox: If the Event Is Lost, the Saga Stops

The example above has a gap: Ordering.Place writes the order status and then publishes the event. In reality these are two writes to two separate systems: the database and the broker. If the database write succeeds and the broker write fails, the order stays "Placed" and the saga never moves on; the other way round, stock is reserved for an order that doesn't exist in the database. This dual write problem is the first decision in every microservices design that drives a saga through events. The answer is the transactional outbox: the event is written to an outbox table in the same local transaction as the business data, and a separate process carries it from there to the broker. The IntegrationEventLogEF project in dotnet/eShop is a concrete .NET example of the idea. How to build it, why the relay can produce duplicates and how an inbox handles them are in the Messaging series outbox post; what we need to know in this series is that wherever a saga is mentioned, the outbox is on the table as a decision too.

Intermediate States Are Part of the Domain

The most important part of accepting eventual consistency isn't technical but linguistic. The system no longer flips instantly between "the order happened" and "the order didn't happen"; in between there are states that last seconds, sometimes minutes. These states need a name, a meaning and a behavior the business has agreed to. That's why Bookstore's OrderStatus carries the values Placed, PaymentPending, Paid, Shipped, Delivered and Cancelled. The goal we set in the Introduction to Architecture series was also a description of an intermediate state: if the payment provider doesn't answer within 5 seconds, the order becomes PaymentPending and no error is returned to the user.

dotnet/eShop spells out the same approach even more explicitly: the OrderStatus enum in its Ordering domain has the values Submitted, AwaitingValidation, StockConfirmed, Paid, Shipped and Cancelled. Because stock validation and payment happen in separate services, the order carries every step between "submitted" and "confirmed" as a state.

These states have to be discussed with the domain expert, because each contains a business decision. Does the "we've received your order" email go out at Placed or at Paid? How long does an order stuck in PaymentPending wait, and what happens then? How long is the stock reservation held while payment is pending? The answers show up in code as timeouts, message texts and compensations.

Showing Eventual Consistency to the User

Intermediate states must reach the interface too. The checkout endpoint can no longer say "order complete", because it isn't; what it can say is "I've received your order and I'm processing it". HTTP has a status code for this: 202 Accepted says the request has been accepted for processing but processing hasn't finished. Putting the address where the status can be followed into the response is common practice:

using System.Collections.Concurrent;
using System.Net.Http.Json;

WebApplicationBuilder builder = WebApplication.CreateBuilder();
builder.Logging.ClearProviders();
builder.WebHost.UseUrls("http://127.0.0.1:5088");
WebApplication app = builder.Build();

ConcurrentDictionary<Guid, string> orders = new();

// Checkout: save the order, publish the event, don't wait for the outcome
app.MapPost("/orders", () =>
{
    Guid id = Guid.CreateVersion7();
    orders[id] = "Placed";
    _ = Task.Delay(300).ContinueWith(_ => orders[id] = "Paid"); // stock + payment complete in the background
    return Results.Accepted($"/orders/{id}", new OrderStatusDto(id, orders[id]));
});
app.MapGet("/orders/{id:guid}", (Guid id) =>
    orders.TryGetValue(id, out string? status) ? Results.Ok(new OrderStatusDto(id, status)) : Results.NotFound());
await app.StartAsync();

using HttpClient client = new() { BaseAddress = new Uri("http://127.0.0.1:5088") };
HttpResponseMessage placed = await client.PostAsync("/orders", content: null);
Console.WriteLine($"POST /orders -> {(int)placed.StatusCode} {placed.StatusCode}, Location: {placed.Headers.Location}");
Console.WriteLine($"  body: {await placed.Content.ReadAsStringAsync()}");

// The Angular side polls this address a few times (or waits for a SignalR notification)
for (int attempt = 1; attempt <= 3; attempt++)
{
    OrderStatusDto? now = await client.GetFromJsonAsync<OrderStatusDto>(placed.Headers.Location);
    Console.WriteLine($"GET #{attempt}: {now!.Status}");
    if (now.Status != "Placed") break;
    await Task.Delay(250);
}
await app.StopAsync();

public sealed record OrderStatusDto(Guid Id, string Status);

POST /orders -> 202 Accepted, Location: /orders/01a1b86b-a6a6-77ce-a3f6-7c273a825323
  body: {"id":"01a1b86b-a6a6-77ce-a3f6-7c273a825323","status":"Placed"}
GET #1: Placed
GET #2: Placed
GET #3: Paid

The Task.Delay(300).ContinueWith(...) in the example only imitates the saga running in the background. What matters is the contract: POST returns 202 and a Location, and the client reads the status from that address. On the Angular side this means the post-checkout screen has two stages. In the first, the message "Your order has been received, waiting for payment confirmation" and the order number appear; even if the page is closed, the order isn't lost. In the second, when the status moves to Paid the message becomes "Your order is confirmed", and if it moves to Cancelled the reason is explained. Polling, asking again every few seconds up to a limit, may be enough to learn the status; for a livelier experience the server can push the status itself with SignalR or with SSE (Server-Sent Events), which we saw in the ASP.NET Core Web API series post "What's New in .NET 10: AddValidation, OpenAPI 3.1 and Server-Sent Events in ASP.NET Core".

The interface should avoid two things. Hiding the intermediate state and spinning a spinner until payment completes shows eventual consistency to the user as a timeout. Showing the intermediate state as "successful" breaks trust when the order is cancelled later. An intermediate screen that tells the user the truth beats both extremes.

Common Pitfalls

Using distributed transactions because they "work" on Windows. A transaction that works between two SQL (Structured Query Language) Server instances with MSDTC is useless as soon as the first HTTP API or broker gets involved, and it ties locks to the fate of the network.

Splitting an invariant across two services. Calling two services synchronously for a rule that must be immediately consistent and saying "if both succeed, we're fine" is a protocol-less imitation of 2PC. The rule should live in one place.

Forgetting compensation. Writing the saga's happy path and leaving the failure path for later means stock reserved forever after the first declined card.

Events without an outbox. Writing to the database and then publishing to the broker guarantees that one day an event is lost and a saga silently stops.

Hiding the intermediate state from the user. Showing an "order complete" screen and later sending a cancellation email makes the customer pay the price of eventual consistency.

When Not to Use It

Eventual consistency isn't acceptable for every rule. If a rule really needs immediate consistency and it cuts across two services' data, consider merging the two parts into one service instead of patching it with a saga. That's why campaign pricing and order creation are in the same service at Bookstore. If there's one team and the system is small, a single database transaction inside a monolith or modular monolith reduces all of a saga's complexity to one line; you only need this post's techniques if you have a strong reason to split the services.

A Small Experiment

★☆☆ In the saga example, set Inventory's starting stock to 2 and add a third order. Run the OutOfStock path: what is Ordering's final status, and is Payment ever called?

★★☆ Remove the compensation handler in Inventory and run it again. What happens to the stock count? In a real system, who would notice this bug, and when? Suggest a metric or alert that would make it easier to notice.

★★★ Rewrite the same checkout flow with orchestration: a CheckoutSaga class listens to OrderPlaced, sends commands to Inventory and Payment, and moves on or starts compensation based on the results. In which design is it easier to add the rule "if payment hasn't completed in 5 seconds, PaymentPending"?

In this post we built consistency between services with events. In the next post we take a step back and look at communication itself: the difference between sync request/response and asynchronous events, where REST (Representational State Transfer), gRPC and messages fit, temporal coupling, and turning the checkout flow from a synchronous chain into an event-driven one.