Microservices in Production: Distributed Tracing, Resilience and Independent Deployment

Version note: The code in this post was built and run with C# 14 and .NET 10.

Guess First

The first morning of the back-to-school campaign. Support says "checkout takes four seconds and some orders fail". The checkout request goes from the Web BFF (Backend for Frontend) to Ordering, from Ordering to Inventory and Payments, and Payments calls the payment provider. Five teams look at their own logs and all give the same answer: "No errors on our side, downstream is just slow."

The payment provider's response time has gone from 300 ms to 3 seconds. Ordering, the BFF and Payments each retry a failed call three times. How many requests do you think the payment provider sees for each checkout attempt by a user? And instead of putting five teams' logs side by side, what would it have taken to find which service is slow?

Following One Request Through Five Services

In a monolith, a request's journey lived in one process, one log file and one stack trace. With microservices the same journey is spread across five processes, five logs and five teams' dashboards. Distributed tracing puts that journey back together. In OpenTelemetry (OTel) terms: a trace is the whole trail a request leaves through the system; a span is a single operation within that trail (an HTTP (Hypertext Transfer Protocol) call, a database query); every span carries a trace id, its own span id and a parent span id. Context propagation is carrying those identifiers between services. Over HTTP the carrier is the traceparent header from the W3C (World Wide Web Consortium) Trace Context Recommendation: a version, a 32-hex-character trace id, a 16-character parent id and flags.

In .NET this mechanism is built on System.Diagnostics.Activity, and most of it comes out of the box: ASP.NET Core opens a server span for every incoming request, and HttpClient opens a client span for every outgoing call and writes traceparent into the header. The OpenTelemetry SDK (Software Development Kit) is the piece that collects these spans and sends them to a backend (Jaeger, Zipkin, Grafana Tempo, Azure Monitor or the Aspire dashboard). In the example below, instead of an exporter I used a small ActivityListener that prints each span when it ends; the BFF opens a checkout span and calls Ordering, and Ordering calls Payments:

// A small listener imitating what an OpenTelemetry exporter does: prints every span that ends
using ActivityListener listener = new()
{
    ShouldListenTo = source => source.Name is "Bookstore.WebBff" or "Microsoft.AspNetCore" or "System.Net.Http",
    Sample = (ref ActivityCreationOptions<ActivityContext> _) => ActivitySamplingResult.AllDataAndRecorded,
    ActivityStopped = a => Console.WriteLine(
        $"span {a.SpanId} parent {(a.ParentSpanId == default ? "(root)          " : a.ParentSpanId)} {a.Kind,-8} {a.GetTagItem("service") ?? a.GetTagItem("url.full") ?? a.DisplayName}"),
};
ActivitySource.AddActivityListener(listener);

WebApplication ordering = Service("http://localhost:5601", app =>
    app.MapPost("/api/orders", async (HttpRequest request, IHttpClientFactory http) =>
    {
        Activity.Current?.SetTag("service", "ordering");
        Console.WriteLine($"ordering sees traceparent: {request.Headers.TraceParent}");
        await http.CreateClient().PostAsync("http://localhost:5602/api/payments", null);
        return Results.Ok();
    }));

using HttpClient bff = new();
using (Activity? checkout = new ActivitySource("Bookstore.WebBff").StartActivity("checkout"))
{
    Console.WriteLine($"trace id: {Activity.Current?.TraceId}");
    await bff.PostAsync("http://localhost:5601/api/orders", null);
}

trace id: cd70b8894c875bf7e4e6d1a7615845e1
ordering sees traceparent: 00-cd70b8894c875bf7e4e6d1a7615845e1-b79988a62487d768-01
payments sees traceparent: 00-cd70b8894c875bf7e4e6d1a7615845e1-42f8ea4d84425e41-01
span 7eb30198bfd38144 parent 42f8ea4d84425e41 Server   payments
span 42f8ea4d84425e41 parent 012e3be395663d75 Client   http://localhost:5602/api/payments
span 012e3be395663d75 parent b79988a62487d768 Server   ordering
span b79988a62487d768 parent be3a07970f6f68ea Client   http://localhost:5601/api/orders
span be3a07970f6f68ea parent (root)           Internal checkout

Read the output from the bottom up. The root span is checkout in the BFF. Under it is the client span for the BFF's call to Ordering (b799…), and the parent id in the traceparent that Ordering sees is exactly that. Ordering's server span (012e…) is a child of that client span, and Ordering's call to Payments (42f8…) is a child of Ordering. All five spans carry the same trace id (cd70…), and I didn't write a single line of code for it in any service; SetTag("service", ...) is there only to make the output readable, in reality that information is OpenTelemetry's service.name resource attribute. A tracing backend draws these five lines as a timeline, and the Guess First question is answered at a glance: 3.5 of the four seconds are in Payments' span to the payment provider.

In asynchronous flows, that is when events go through a broker, the trace context is carried in message headers; we covered that, and the observability of event-driven flows, in the Messaging and Event-Driven Architecture series post "The Cost of Event-Driven Architecture: Observability, Debugging, and When Not to Use It". Setting up the OpenTelemetry SDK, sampling and choosing a backend are topics for the upcoming Observability series in the DevOps category.

Logs, Metrics and Health Checks

A trace answers "where"; the answer to "why" is usually in the logs. For logs to be useful with microservices, two conditions apply. First, logs are collected in one place (log aggregation); connecting to five servers over SSH (Secure Shell) to read files doesn't work during an incident. Second, every log line carries the trace id, so when you find the slow span in the tracing backend you can jump to that span's logs with one click. ASP.NET Core's ILogger can add the trace id and span id to the log scope itself while an Activity is active; that's why the structured logging from the ASP.NET Core Web API (Application Programming Interface) series post "Logging in ASP.NET Core: ILogger, Log Levels, Structured Logging and Serilog" is a requirement with microservices, not a preference. Instead of inventing a separate header for a correlation id, use the W3C trace id; a separate id means a second propagation mechanism and a header someone forgets.

Metrics answer "how bad is it right now?": request count per service, error rate and duration distribution (p50, p95, p99). The four golden signals (latency, traffic, errors, saturation) from Google's SRE (Site Reliability Engineering) book are a good starting list. Alerts are set on p95/p99 and error rate, not on average duration; targets like "checkout p95 < 800 ms", which we wrote for Bookstore in the Introduction to Architecture series post "Quality Attribute Scenarios: Turning 'Make It Fast' into a Measurable Requirement", become measurable right here.

The health check API decides how the platform sees the service. Liveness answers "should I restart the process?", readiness answers "should I send traffic to this instance?" We wrote both as separate endpoints in the ASP.NET Core Web API series post "Health Checks in ASP.NET Core Web API: Liveness, Readiness and Database Checks". The rule specific to microservices: don't put other services into the readiness check. If Ordering's readiness depends on Payments, a problem in Payments takes every Ordering instance out of traffic and a partial failure becomes a full outage; Ordering's own database, on the other hand, is a legitimate part of readiness.

Cascading Failure: When One Service Slows Down

A service crashing is often less dangerous than a service slowing down. A crashed service returns errors immediately and callers notice. A slow service holds on to its callers' resources: while Payments answers in 3 seconds, every checkout request in Ordering holds an open connection, a request in memory and a waiting task for 3 seconds. Under campaign traffic (120 orders per minute) the waiting requests pile up, Ordering's connection pool fills, Ordering slows down, and the same thing repeats in the BFF. In the end even pages that have nothing to do with Payments, like catalog search, slow down because they share the same BFF instance's resources. Michael Nygard's Release It! (2nd edition, 2018) describes this chain as cascading failure and lists the defences as stability patterns.

The first defence is a timeout. Every network call needs an upper limit, and that limit must be smaller than the caller's own budget: if the BFF has 2 seconds for checkout, there's no point in Ordering waiting 5 seconds for Payments. The second defence is a bulkhead: separate resource pools for different downstreams, so requests waiting on Payments don't use up the connections for catalog calls. The third is the circuit breaker.

What Timeouts, Retries and Circuit Breakers Mean at the Service Level

Retry is the right tool for transient failures; but if it isn't thought about at the service level, it feeds the fire. In the Guess First chain, if the BFF, Ordering and Payments each make a call four times including the first attempt, the payment provider sees 4 × 4 × 4 = 64 requests for a single checkout attempt. Multiplying traffic by 64 while the provider is already slow makes it unable to recover. The rule is simple: retry happens in a single layer of the chain, preferably the one that understands the failure best and knows whether the request is idempotent. Non-idempotent calls such as POST are only retried with an idempotency key; we covered that in the Solution Patterns series post "Idempotency Keys: Not Creating the Same Order Twice on Double Clicks and Retries".

A circuit breaker, as Martin Fowler described it on his bliki in 2014 with credit to Nygard, stops sending requests downstream entirely for a while once the error rate crosses a threshold and fails immediately. The gain goes both ways: the caller doesn't spend resources waiting, and the downstream gets breathing room to recover. What to show while the circuit is open is a product decision; for Bookstore that decision was made in the Introduction to Architecture series: if the payment provider doesn't answer within 5 seconds the order becomes PaymentPending, no error is returned to the user, and the payment completes in the background.

.NET has a ready-made package for these patterns: AddStandardResilienceHandler() from Microsoft.Extensions.Http.Resilience. Its defaults according to Microsoft Learn, from the outermost inwards: a rate limiter (1,000 concurrent requests), a 30-second total timeout, three retries (exponential backoff, jitter, 2-second base delay), a circuit breaker (10% failure ratio over at least 100 requests in a 30-second window, staying open for 5 seconds) and a 10-second attempt timeout. These defaults are general purpose; 30 seconds is far too long for a flow the user is waiting on, like checkout, and adding the same handler to every layer takes you back to the 64 above. Derive the values from the service's own budget. The Polly resilience pipeline code is the topic of the upcoming "External Service Integration and Resilience" series in the Backend category, and the place of these patterns in system design is the topic of the reliability post in the upcoming System Design series.

Independent Deployment: One Pipeline per Service

In the first post of this series we put independent deployability at the heart of a microservice; in production its concrete form is a separate build and deploy pipeline for every service. A change in Catalog triggers only Catalog's pipeline, produces only Catalog's artifact (its container image) and deploys only Catalog. A release plan containing a step like "deploy all services the same night, in order" is, by the definition in the Introduction to Architecture series post "Understanding the Monolith: Single-Process, Modular, and Distributed Monoliths", a symptom of a distributed monolith.

The precondition for independent deployment is backward compatibility: the new version has to work with old consumers and old data. It has three legs. For APIs, the expand-contract from this series' "Service Contracts and Versioning" post. For the database, the schema version of the same idea: instead of renaming a column, add the new column, let both versions work together for a while, remove the old one later; during a rolling deploy, old and new instances write to the same database at the same time. For behaviour, feature flags: deploying the code and turning the feature on are separated, and the feature can be rolled back by switching it off. The Solution Patterns series post "Designing Feature Flags in Code: Toggle Types, Microsoft.FeatureManagement and Flag Debt" explains this separation. Deployment strategies like canary and blue-green are in the upcoming Deployment Strategies series.

Where Contract Tests Fit in CI

In this series' "Service Contracts and Versioning" post we wrote consumer-driven contract tests with Pact and said that a Pact Broker keeps track of which version was verified against which. In the CI (continuous integration) pipeline these tests take the place of an integration test environment. The flow roughly looks like this:

• The consumer (Notification) pipeline runs the consumer test and publishes the pact to the broker with its version number and branch.
• If the pact changed, the broker fires a webhook on the contract_requiring_verification_published event and runs a verification-only job of the provider (Catalog).
• On every build, the provider pipeline verifies the relevant pacts from the broker against its own code and writes the result to the broker.
• Before deploying, both sides ask pact-broker can-i-deploy --pacticipant Catalog --version 2.4.0 --to-environment production; if the answer is no, the deployment stops.
• After a deployment, pact-broker record-deployment tells the broker which version is in which environment; the next can-i-deploy question relies on that information.

The value of this arrangement is that it leaves no need for a shared and always slightly broken "integration" environment. Bringing up five services with matching versions in one environment and running end-to-end tests is slow and fragile, and it's a habit that indirectly pushes you towards releasing all services together. End-to-end tests don't have to disappear entirely; they can stay for a few critical flows (checkout) in a production-like environment, but contracts, not they, make the deployment decision.

Common Pitfalls

Adding tracing later

Deciding to set up tracing after the first incident means having no data at the moment you need it most. OpenTelemetry, structured logging and health checks belong in a new service's skeleton from day one; Aspire's service defaults give you exactly that skeleton.

Retries in every layer

The 64 requests above. Decide at the system level where retries happen and turn them off in the other layers.

Calls without timeouts

HttpClient's default timeout is 100 seconds. In a flow the user is waiting on, that means every resource being locked for 100 seconds when a downstream slows down.

Tying readiness to downstreams

Every Ordering instance dropping out of traffic because of a problem in Payments. Readiness checks only the service's own dependencies (its own database, its own cache).

Trusting a shared integration environment

The sentence "it worked in the test environment" assumes the other service versions in that environment are the same as in production. They never are.

A Small Experiment

★☆☆ Add await Task.Delay(500) to the example's Payments endpoint and change the listener to also print each span's Duration. Which spans take longer than 500 ms, and why more than one?

★★☆ Add AddStandardResilienceHandler() to Ordering's HttpClient, make Payments return 503 for the first two calls, and look at the listener output. How many client spans do you see, and are they all in the same trace?

★★★ Write a resilience budget for Bookstore's checkout flow: the BFF's total time, each layer's timeout, which layer does retries and which call needs an idempotency key. What does the user see when the payment provider goes to 3 seconds?

We've talked about running services too. What's left is the question we've put off since the start of the series: Bookstore already has a working monolith; how do you get from there to all of this without stopping the system? In the last post of the series we'll look at the strangler fig, branch by abstraction and the distributed monolith trap.