Security in Microservices: Edge Authentication, Token Propagation and Service-to-Service Trust

Version note: The code in this post was built and run with C# 14 and .NET 10; token validation used Microsoft.AspNetCore.Authentication.JwtBearer 10.0.14.

Guess First

Ada clicks "refund my order" on Bookstore's website. The request goes to the Web BFF (Backend for Frontend), the BFF calls Ordering, and Ordering calls Payments to pay the money back. The team has two proposals. The first: "Let's carry the user's access token unchanged along the chain so every service sees who's asking." The second: "Ordering goes to Payments with its own service identity; Payments trusts Ordering anyway. Ordering has already authenticated the user."

Payments' refund endpoint, written according to the second proposal, only checks whether the caller has the payments.refund permission. In her request Ada sends Grace's order number instead of her own, and a bug in Ordering passes that number to Payments without checking whether it belongs to Ada. What does Payments do? Does the first proposal solve this problem, or create another one?

Authentication at the Edge, Authorization Inside

In a monolith there was a single security boundary: authentication happened when the request entered the application, and after that everything happened in the same process, with the same ClaimsPrincipal. With microservices every service is a separate network endpoint and every call crosses a boundary. The split in the "Security" chapter of Sam Newman's Building Microservices (2nd edition, 2021) is widely accepted today: coarse-grained authentication at the edge (gateway or BFF), fine-grained authorization inside the services. OWASP's (Open Worldwide Application Security Project) Microservices Security Cheat Sheet says the same thing: a check at the gateway turns unauthorized requests away at the door but doesn't prove that an operation downstream is authorized; every service controls access to its own operations, internal calls included.

The reason is simple. The answer to "may Ada get this order refunded?" depends on who owns the order, its status and the refund window, and Ordering and Payments own that information. The gateway knowing it would mean falling into the god gateway trap from the earlier posts. The edge's job is to authenticate the identity and carry it inside reliably; the job of every service inside is to ask, with that identity, "is this operation open to this person?" The IDOR (Insecure Direct Object Reference) hole, as covered in the OWASP series post "OWASP A01: Broken Access Control (1) — IDOR, Admin Endpoints and Authorization in ASP.NET Core", shows up in microservices exactly when that question isn't asked at a service boundary.

Two questions remain: in what form is the user's identity carried inside, and how do you know that a request reaching a service really comes from a Bookstore service?

Carrying the User's Token Unchanged

The first proposal is to attach the BFF's access token to every call in the chain. It's easy to implement but has a problem: that token was issued for the BFF. The token's aud (audience) claim says which recipient the token was produced for; if the BFF's token is also accepted by Payments, anyone who gets hold of it (from a log line, a leaked service) can go straight to Payments with it. The JWT (JSON Web Token) access token profile in RFC (Request for Comments) 9068 also requires the resource server to validate the audience. That's why Payments' JWT bearer setup accepts only its own name:

// Demo only: in reality an STS signs tokens with an asymmetric key and the service knows only the public key
SymmetricSecurityKey signingKey = new(Convert.FromHexString("8f3c1a9e5b7d2f4061c8e3a7b9d5f1e2a4c6b8d0e2f4a6c8b0d2e4f6a8c0b2d4"));
const string Issuer = "https://login.bookstore.example";

WebApplicationBuilder builder = WebApplication.CreateBuilder();
builder.Logging.ClearProviders();
builder.Services.AddAuthentication().AddJwtBearer(options =>
{
    options.MapInboundClaims = false;
    options.TokenValidationParameters = new TokenValidationParameters
    {
        ValidIssuer = Issuer,
        ValidAudience = "payments", // Payments accepts only tokens issued for itself
        IssuerSigningKey = signingKey,
    };
    options.Events = new() { OnAuthenticationFailed = ctx => { Console.WriteLine($"  payments log: {ctx.Exception.GetType().Name}"); return Task.CompletedTask; } };
});

When Ada's token issued for the BFF (aud: bookstore-web-bff) reaches Payments unchanged:

  payments log: SecurityTokenInvalidAudienceException
forwarded user token: 401  | Bearer error="invalid_token", error_description="The audience '(null)' is invalid"

Payments rejected the token. In the description in the WWW-Authenticate header the audience value shows up as '(null)', because in this version the exception doesn't carry the wrong audience; the real reason is in the service's own log. Two conclusions follow. If you want to carry the user's token, either you keep its audience wide enough to cover all services (which means every service sees a token that can go to every other service; one service leaking means all of them leaking), or you get a new, narrow token at every boundary. The standard name for the second is token exchange. Carrying the user's token has another problem too: the token doesn't say who the service in the middle of the chain is; Payments can't tell whether the request was sent by Ordering or by something else that got hold of the token.

Token Exchange: On the User's Behalf, with Its Own Identity

RFC 8693 (OAuth 2.0 Token Exchange, January 2020) standardizes a service taking the token it holds to the authorization server, that is an STS (Security Token Service), and getting a new token for another recipient. Ordering sends Ada's token as subject_token and its own identity as client authentication, and asks for audience=payments. The RFC separates two semantics. With impersonation, whoever holds the new token is indistinguishable from the user. With delegation, the token carries both identities: sub is still Ada, and the act (actor) claim says that Ordering is acting on her behalf. Between services you should prefer delegation, because Payments can answer both "on whose behalf?" and "by whom?" and write both to its log.

In the example I represented the token after the exchange like this: sub = ada, aud = payments, scope = payments.refund, act = { "sub": "ordering" }. The token is short-lived (a few minutes) and valid only at Payments. In a leak, the damage is limited to a single service and a few minutes. The price is a call to the STS at every boundary crossing and the STS becoming a critical component; in practice exchanged tokens are cached until they expire.

Service-to-Service Trust: Client Credentials and mTLS

There are calls with no user too: a nightly job pulling a stock report from Inventory, Shipping asking Ordering for an address while processing the OrderPlaced event. Here the service speaks on its own behalf, and that's what OAuth 2.0's (RFC 6749) client credentials grant is for: the service goes to the STS with its own client identity and secret (or a certificate) and gets a token on its own behalf for a specific audience and scope.

The second way is mTLS. In an ordinary TLS (Transport Layer Security) connection only the server shows a certificate; with mTLS (mutual TLS) the client shows one too, and both sides recognize each other cryptographically. RFC 8705 defines OAuth clients authenticating with mTLS and binding the token to the client certificate (certificate-bound tokens), so a stolen token can't be used without the certificate's private key. The hard part of mTLS is the certificates themselves: issuing, distributing, keeping them short-lived and rotating them, revoking them. That's why mTLS should be done by a platform, not by hand; the service meshes mentioned in the previous post do it automatically, and SPIFFE (Secure Production Identity Framework for Everyone) is the open standard for giving workloads short-lived, platform-independent identities.

A realistic start for Bookstore: a short-lived token on every call between services (delegation through exchange when there's a user, client credentials when there isn't) and TLS provided by the platform. mTLS comes when compliance requires it or when a mesh is already in place.

Confused Deputy: Lending Out Authority

Back to the second proposal from Guess First. In 1988 Norm Hardy described a problem under the name "The Confused Deputy": a program with broad authority of its own (the deputy) uses that authority while serving the request of someone less privileged, and so does something the requester couldn't actually do. In microservices its most common form is doing work on a user's behalf with a powerful service identity. Two versions of Payments:

// Naive version: checks only the scope, never asks on whose behalf it was called
payments.MapPost("/naive/refunds/{orderId}", (string orderId, ClaimsPrincipal caller) =>
    caller.HasScope("payments.refund")
        ? Results.Ok($"refunded {orderId} (caller: {caller.FindFirstValue("sub")})")
        : Results.Forbid())
    .RequireAuthorization();

// Fixed version: a user context is required and the order must belong to that user
payments.MapPost("/refunds/{orderId}", (string orderId, ClaimsPrincipal caller) =>
{
    if (!caller.HasScope("payments.refund"))
        return Results.Forbid();
    string? user = caller.FindFirstValue("sub");
    string? actor = caller.FindFirstValue("act") is { } act
        ? JsonDocument.Parse(act).RootElement.GetProperty("sub").GetString()
        : null;
    if (actor is null || orderOwners.GetValueOrDefault(orderId) != user)
        return Results.Forbid();
    return Results.Ok($"refunded {orderId} for {user} (actor: {actor})");
}).RequireAuthorization();

Ada asks for a refund of Grace's order G-2002; Ordering goes to Payments first with its own client credentials token, then with the delegation token from the token exchange:

// 2) Ordering, with its own client credentials token, refunds Grace's order at Ada's request
string orderingOwnToken = Issue("ordering", "payments", "payments.refund");
await Call("naive, service token, G-2002", "/naive/refunds/G-2002", orderingOwnToken);
await Call("fixed, service token, G-2002", "/refunds/G-2002", orderingOwnToken);
// 3) After token exchange: sub = ada, act.sub = ordering, aud = payments
string exchanged = Issue("ada", "payments", "payments.refund", actor: "ordering");
await Call("fixed, exchanged token, G-2002", "/refunds/G-2002", exchanged);
await Call("fixed, exchanged token, A-1001", "/refunds/A-1001", exchanged);

naive, service token, G-2002: 200 "refunded G-2002 (caller: ordering)"
fixed, service token, G-2002: 403
fixed, exchanged token, G-2002: 403
fixed, exchanged token, A-1001: 200 "refunded A-1001 for ada (actor: ordering)"

In the naive version Grace's money was refunded at Ada's request, and Payments' log only says "caller: ordering"; who asked isn't even recorded. One missing check in Ordering lent all of Payments' authority to Ada. The fixed version requires two things: if the operation is done on behalf of a user, the token must carry that user (sub) and the service acting on their behalf (act), and the order must belong to that user. Same Ordering, same bug, but this time Payments checks with its own data and returns 403. This is defense in depth made concrete: the check in Ordering is the first line, the check in Payments is the second, and the second doesn't assume the first works.

In the example I kept order ownership in a small dictionary inside Payments; in reality Payments writes the customer identity to its own database when it creates the payment record and compares against it at refund time. The principle from this series' "Data Ownership and Database per Service" post turns into a security decision here: Payments bases its authorization decision not on another service saying "trust me" but on its own data.

Zero Trust: The Inside of the Network Isn't Trusted Either

Classic network security thinks like a castle: outside is dangerous, inside the firewall is safe. Special Publication 800-207, published in 2020 by NIST (National Institute of Standards and Technology), defines the zero trust architecture that drops this assumption: access to a resource is granted per request, based on identity and current context, not on where in the network the request comes from. The practical meaning for microservices is this: a request to Inventory coming from inside the cluster isn't reason enough to trust it. When a service is compromised, the attacker is inside the cluster too.

In Bookstore this means every service authenticates every request (internal calls included), network policies limit which service may talk to which (Notification has no need to reach Payments), and tokens are short-lived with narrow audiences. Zero trust isn't a product; it's the sum of the decisions above.

Secret Management

As the number of services grows, so does the number of secrets: each service's client secret, database connection, broker password, the payment provider's API (Application Programming Interface) key. The monolith habit of "everything in one appsettings.Production.json" becomes both a security and an operations problem here. We covered the basic rules in the OWASP series post "OWASP A04: Cryptographic Failures (2) — Password Storage, Data Protection and Secret Management"; the ones specific to microservices are these. Each service sees only its own secrets; Notification can't read Payments' provider key. Secrets live in a central secret store (Azure Key Vault, HashiCorp Vault or your cloud's equivalent) and services reach it with their own workload identities, which removes the "secret needed to get to the secrets" problem as well. Where possible, short-lived identities replace secrets: workload identity or a certificate instead of a client secret. Also remember that Kubernetes Secrets are only base64-encoded by default, not encrypted; encryption at rest in etcd is turned on separately.

Common Pitfalls

Trusting the internal network

Saying "this endpoint isn't exposed, it doesn't need authentication" means opening all internal endpoints to the first compromised service. Internal endpoints require tokens too.

One token with a broad audience

A token valid at every service looks convenient, but a token ending up in one service's log becomes the key to the whole system. Keep the audience narrow and pay the price of the exchange.

Doing user work with a powerful service identity

The confused deputy example above. If an operation is done on a user's behalf, the user context must be carried to the last service, and the last service must check authorization with its own data.

Doing authorization only at the gateway

The gateway says "Ada is signed in"; it doesn't say "Ada may get this order refunded". Fine-grained authorization stays in the service.

Sharing long-lived secrets

Several services using the same client secret makes it impossible to tell in the logs which service did what, and a leak forces rotating all of them at once.

A Small Experiment

★☆☆ Issue the example's BFF token with both bookstore-web-bff and payments as aud (multiple audiences) and send it to Payments again. Is it accepted? Write the risk of this design in one sentence.

★★☆ Add a rule to the fixed endpoint that the actor in the act claim may only be ordering. What should happen if Notification exchanges a user's token and tries to request a refund? Verify with output.

★★★ Draw a trust map for Bookstore: which service may call which, which calls are on a user's behalf (delegation), which are on the service's own behalf (client credentials), and which audience and scope each token carries? Is there an arrow from Notification to Payments on the map, and should there be?

Services now find each other and talk to each other securely. But when an order slows down in production while passing through five services, how will we see which service it's stuck in, and what do we do so the others don't go down with a service that falls over? In the next post we'll look at running microservices in production.