What Is a Distributed System and Why Is It Hard? Partial Failure and the Fallacies of Distributed Computing

Version note: The code in this post was built and run with C# 14 and .NET 10.

Guess First

Bookstore's OrderService takes payments by sending an HTTP (Hypertext Transfer Protocol) request to an external payment provider. To keep users from staring at a frozen checkout page, a teammate lowered HttpClient.Timeout to 1 second and wrote a small loop that tries once more on timeout. The provider is slow today: it charges the card, then takes 2 seconds to answer.

for (int attempt = 1; attempt <= 2; attempt++)
{
    try
    {
        HttpResponseMessage response = await client.PostAsJsonAsync("/charges", order);
        Console.WriteLine($"attempt {attempt}: {(int)response.StatusCode}");
        break;
    }
    catch (TaskCanceledException ex) when (ex.InnerException is TimeoutException)
    {
        Console.WriteLine($"attempt {attempt}: timeout ({ex.GetType().Name} -> {ex.InnerException.GetType().Name})");
    }
}

What does the user see on screen? More importantly, how many times is 349.90 charged to the customer's card? Think about it, then look at the real run below.

What Is a Distributed System, and Why Distribute?

The definition in Martin Kleppmann's Cambridge lecture notes is plain enough: a distributed system is several computers (nodes) that try to get a job done together by exchanging messages over a network. The definition Leslie Lamport gave in a 1987 e-mail, quoted ever since, describes the painful side: if a computer you did not even know existed can crash and make your own computer unusable, you are in a distributed system.

By that definition you are already writing one. An ASP.NET Core API (Application Programming Interface), a database server, a Redis cache and a payment provider: four separate processes with a network between them. No microservices required; a single monolith talking to its database is already in this class. The reasons to distribute are familiar:

• Scale: a workload that exceeds one machine's CPU (Central Processing Unit), memory or disk.
• Fault tolerance: another machine takes over when one crashes; the main reason for replication.
• Latency: answering from a copy that is geographically close to the user (a user in Istanbul, a server in Frankfurt).
• Work that is distributed by nature: the payment provider, the shipping company and the e-mail service run on someone else's computers.

The price is what this series is about. Inside one process a method call either returns or throws; there is no third option. The moment you go over the network, the third option appears: nothing happens. No answer arrives, and you cannot tell what did not happen.

Partial Failure: Neither Fully Working nor Fully Down

A single computer usually either works or it does not. When hardware fails, the operating system panics and the process dies; a system that stops completely is preferred over one that limps along. A distributed system does not get that luxury: one node can crash while another keeps running, part of the network can drop packets while the rest is fine, a server can be alive but stuck in a garbage collector (GC) pause that lasts seconds. This is called partial failure, and it is exactly what makes distributed systems hard.

The painful part of partial failure is nondeterminism: send the same request twice and one succeeds while the other times out. Worse, for the request that timed out you cannot tell these four situations apart:

The request may have been lost in the network (1), the server may have crashed before processing it (2), it may have processed it and crashed before replying (3), or the response may have been lost or arrived later than you were willing to wait (4). In the first two no money was taken, in the last two it was. The client has the same information in all four: a TaskCanceledException.

A Timeout Is Not an Answer

Let us build the opening scenario in one console application. In the same process we start a fake payment provider with a minimal API; it charges the card by writing to ledger, waits 2 seconds and then answers. On the client side, a 1-second timeout and two attempts:

using System.Collections.Concurrent;
using System.Net.Http.Json;

namespace Bookstore.Payments;

public record ChargeRequest(string OrderId, decimal Amount);
public record ChargeResult(string ChargeId, string OrderId, decimal Amount);

public static class Program
{
    public static async Task Main()
    {
        ConcurrentBag<ChargeResult> ledger = [];
        ConcurrentDictionary<string, ChargeResult> processed = new();

        WebApplicationBuilder builder = WebApplication.CreateSlimBuilder();
        builder.Logging.ClearProviders();
        builder.WebHost.UseUrls("http://127.0.0.1:5077");
        WebApplication app = builder.Build();

        // Payment provider: charges the card, then answers slowly (2 s)
        app.MapPost("/charges", async (ChargeRequest request) =>
        {
            ChargeResult result = new($"ch_{ledger.Count + 1}", request.OrderId, request.Amount);
            ledger.Add(result);
            await Task.Delay(TimeSpan.FromSeconds(2));
            return Results.Ok(result);
        });

        // Same provider, but it remembers the Idempotency-Key
        app.MapPost("/v2/charges", async (HttpRequest http, ChargeRequest request) =>
        {
            string key = http.Headers["Idempotency-Key"].ToString();
            ChargeResult result = processed.GetOrAdd(key, _ =>
            {
                ChargeResult created = new($"ch_{ledger.Count + 1}", request.OrderId, request.Amount);
                ledger.Add(created);
                return created;
            });
            await Task.Delay(TimeSpan.FromSeconds(2));
            return Results.Ok(result);
        });

        await app.StartAsync();

        Uri provider = new("http://127.0.0.1:5077");
        using HttpClient client = new() { BaseAddress = provider, Timeout = TimeSpan.FromSeconds(1) };
        using HttpClient patient = new() { BaseAddress = provider, Timeout = Timeout.InfiniteTimeSpan };
        ChargeRequest order = new("order-1042", 349.90m);

The loop from the opening runs right below this setup. After the loop we wait a little so the provider can finish its half-done work, then count the ledger:

await Task.Delay(TimeSpan.FromSeconds(2.5));
Console.WriteLine($"charges for order-1042: {ledger.Count(c => c.OrderId == order.OrderId)}");

The real output:

--- naive retry ---
attempt 1: timeout (TaskCanceledException -> TimeoutException)
attempt 2: timeout (TaskCanceledException -> TimeoutException)
charges for order-1042: 2

Both attempts timed out and the customer paid twice. The user sees an error, most likely clicks "Pay" again, and the number goes to three. Nobody wrote buggy code here: HttpClient behaved correctly (since .NET 5, an expired Timeout throws a TaskCanceledException whose inner exception is a TimeoutException), and so did the provider. The mistake is reading a timeout as "the operation did not happen". The only thing a timeout says is that no answer arrived within the time you allowed.

So what if we remove the retry? Then in cases (1) and (2) the customer believes the order was placed while it was never paid, or sees an error although the payment was actually taken. Not retrying is wrong, and so is retrying blindly. The right question is: what does it take to resend the same request safely?

Idempotency Keys: Making Uncertainty Retryable

An operation is idempotent if doing it once or five times has the same result. "Set the order status to Paid" is idempotent; "charge 349.90 to the card" is not. The common way to make a non-idempotent operation idempotent is to have the client generate a unique key per logical operation and send the same key with every attempt, while the server stores the result under that key and returns it again on a repeat. Payment APIs such as Stripe do this with an Idempotency-Key header; a standard for the header is an Internet-Draft at the IETF (Internet Engineering Task Force).

ledger.Clear();
Console.WriteLine("--- retry with idempotency key ---");
string idempotencyKey = Guid.CreateVersion7().ToString();
for (int attempt = 1; attempt <= 3; attempt++)
{
    using HttpRequestMessage message = new(HttpMethod.Post, "/v2/charges") { Content = JsonContent.Create(order) };
    message.Headers.Add("Idempotency-Key", idempotencyKey);
    using CancellationTokenSource cts = new(attempt < 3 ? TimeSpan.FromSeconds(1) : TimeSpan.FromSeconds(5));
    try
    {
        HttpResponseMessage response = await patient.SendAsync(message, cts.Token);
        ChargeResult? charge = await response.Content.ReadFromJsonAsync<ChargeResult>();
        Console.WriteLine($"attempt {attempt}: {(int)response.StatusCode} {charge?.ChargeId}");
        break;
    }
    catch (TaskCanceledException)
    {
        Console.WriteLine($"attempt {attempt}: timeout");
    }
}
Console.WriteLine($"charges for order-1042: {ledger.Count(c => c.OrderId == order.OrderId)}");

On the provider side the /v2/charges endpoint keeps the key in a ConcurrentDictionary. The output:

--- retry with idempotency key ---
attempt 1: timeout
attempt 2: timeout
attempt 3: 200 ch_1

Three attempts, one charge, and on the third attempt the client gets back the ch_1 created by the first one. The ConcurrentDictionary only demonstrates the idea; in reality the key is stored together with the result in the same database transaction with a unique constraint, a different payload under the same key is rejected, and a second request for a key that is still being processed is made to wait. The details are in the Messaging and Event-Driven Architecture series, in "Idempotent Consumers: What Happens When the Same Message Arrives Twice?". The lesson to take from this post is more basic: uncertainty does not go away; you design to tolerate it.

An RPC Is Not a Local Call

RPC (Remote Procedure Call) has promised the same thing since the 1980s: call a remote function as if it were local. gRPC, WCF (Windows Communication Foundation), typed HttpClients and libraries such as Refit build this illusion very well; await paymentClient.ChargeAsync(order) looks exactly like await repository.SaveAsync(order). Yet they behave fundamentally differently:

A local call either returns or throws; a remote call has a third result, "unknown". A local call takes nanoseconds to microseconds and is predictable; a remote call takes anywhere from milliseconds to minutes. You pass a local call a reference; everything sent to a remote call is serialized and copied, and a DateTime's Kind or a decimal's precision can change on the way. Whether repeating a local call is safe is decided by your code; whether a remote call gets repeated is the network's decision, because the retry may come from an HTTP handler, a load balancer or a service mesh rather than your code.

That is why a good client library does not hide that it is remote: its method signatures take a CancellationToken, it separates "transient" from "permanent" exceptions, and it forces you to configure the timeout explicitly. In the C# Async series, "C# CancellationToken: Cancellation, Timeouts and Linked Tokens" showed how a timeout travels through the cancellation mechanism; what this post adds is that cancelling does not undo the remote work.

The Fallacies of Distributed Computing in .NET

In the 1990s at Sun Microsystems, L Peter Deutsch and colleagues collected the assumptions programmers new to networks silently make; James Gosling added the eighth item. The list has held for thirty years because every item still maps to a production incident today:

The first two items were the subject of this post. We will come back to the others throughout the series: the next post turns message loss into a system model, and the unreliability of clocks is the topic of the third.

Common Pitfalls

Treating a timeout as a failure. A timeout means "I do not know". After a timeout on a payment, a reservation or a stock decrement, the correct reaction is to query the outcome (GET /charges?orderId=... at the provider) or to retry with the same idempotency key.

Stacking retries layer by layer. Retries on HttpClient through Microsoft.Extensions.Http.Resilience, another retry at the service level with Polly, yet another at the API gateway: one user request can turn into 27 attempts. Retry in one layer only, with jittered backoff, and only for idempotent operations. The Microservices series covered these layers in "Microservices in Production: Distributed Tracing, Resilience and Independent Deployment".

Generating a new idempotency key on every attempt. The key represents the logical operation, not the attempt. If you call Guid.NewGuid() inside the loop, the key is useless.

Believing that a cancelled CancellationToken stops the remote work. The token only ends your waiting; the request to the server is already on its way.

"We do not have network problems, everything is in the same data center." Even between two machines in the same rack a switch reboots, a NIC (Network Interface Card) driver hangs, a GC pause lasts seconds. The assumption that the network is reliable usually breaks with the first serious traffic.

When Not to Distribute

If you can avoid distributing, do. A single strong server and a single database carry the load of most business applications for years and remove most of the problems in this post (except the database connection). Before moving a piece of work into a second process, ask: what does this boundary buy me in scale, fault tolerance or independent deployment, and what uncertainty does it add to every call in return? The Microservices series discusses this decision architecturally in "The Cost of Microservices: When Are They Unnecessary and When Are They Inevitable?"; this series covers the theory you will face once the decision is made.

A Small Experiment

★☆☆ Lower the provider's delay in the example from 2 seconds to 900 milliseconds and run it again. How many charges are there now? With a delay of 1000 milliseconds, is the result the same on every run?

★★☆ Send a different amount (Amount = 1.00m) to /v2/charges with the same key. What does the provider return? Why is this behavior dangerous, and how would you change the endpoint?

★★★ Add a GET /charges/{orderId} endpoint to the provider and replace the retry with "on timeout, query first; retry only if there is no charge". If the provider is still processing the first request between the two calls, when does this strategy make the wrong decision?

In this post we experienced uncertainty through one HTTP call. The next post defines the same uncertainty formally: synchronous and asynchronous network models, crash-stop, crash-recovery and Byzantine failures, the Two Generals problem and why exactly-once delivery is impossible.