Authentication and Service-to-Service Trust: Keycloak, BFF Sessions, Token Propagation and Client Credentials

Version note: The code in this post was built and run with .NET 10, Aspire 13.6 (Aspire.Hosting.Keycloak and Aspire.Keycloak.Authentication 13.6.5-preview), Keycloak 26.6, YARP 2.3 and Grpc.Net.ClientFactory 2.85.

Guess First

Ada signs in through the BFF, and the BFF keeps the tokens it gets from Keycloak inside the session cookie. How many cookies reach the browser? Second question: a script on the internal network skips the BFF and calls Ordering directly with a token from Keycloak whose signature is valid; but the token was issued for Inventory. What does Ordering say? The answers come from real runs.

The Keycloak Realm: Imported Locally, Secrets Outside the Repository

The AppHost has had a Keycloak container since 02, but it was empty; in 06 we made it start stopped with WithExplicitStart(). Now we define the bookstore realm of Bookstore, the online bookstore used as the shared example across this site's series: two realm roles (customer and store-staff), two test users (Ada is a customer, Grace is also store staff) and two clients. bookstore-bff is a confidential client that uses the authorization code flow with PKCE for user sign-in. bookstore-bff-service is a service account for the BFF's calls that are not made on behalf of a user. The realm is imported on first start from a JSON file in the AppHost's Realms folder:

{
  "clientId": "bookstore-bff",
  "publicClient": false,
  "secret": "${BOOKSTORE_BFF_CLIENT_SECRET}",
  "standardFlowEnabled": true,
  "directAccessGrantsEnabled": false,
  "redirectUris": [ "http://localhost:5100/signin-oidc" ],
  "attributes": {
    "pkce.code.challenge.method": "S256",
    "post.logout.redirect.uris": "http://localhost:5100/signout-callback-oidc"
  },
  "protocolMappers": [
    {
      "name": "bookstore-api audience",
      "protocol": "openid-connect",
      "protocolMapper": "oidc-audience-mapper",
      "config": { "included.custom.audience": "bookstore-api", "access.token.claim": "true", "id.token.claim": "false" }
    },
    {
      "name": "realm roles as roles",
      "protocol": "openid-connect",
      "protocolMapper": "oidc-usermodel-realm-role-mapper",
      "config": { "claim.name": "roles", "multivalued": "true", "jsonType.label": "String", "access.token.claim": "true", "id.token.claim": "true" }
    }
  ]
}

The two mappers do two jobs. The audience mapper adds the bookstore-api audience to access tokens issued for the BFF; the services will accept only tokens with that audience. The role mapper produces a flat roles claim instead of the nested realm_access.roles structure Keycloak uses for roles; ASP.NET Core's RequireRole can read it directly.

The file holds no secrets. Keycloak fills values such as ${BOOKSTORE_BFF_CLIENT_SECRET} from its environment during the import, and the AppHost provides that environment:

// Generated once, kept in the AppHost's user secrets, never in the repository. The realm file refers to them
// as ${...} placeholders, which Keycloak fills from its environment during the import.
IResourceBuilder<ParameterResource> bffClientSecret = AddGeneratedSecret(builder, "bff-client-secret");
IResourceBuilder<ParameterResource> bffServiceSecret = AddGeneratedSecret(builder, "bff-service-secret");
IResourceBuilder<ParameterResource> demoPassword = AddGeneratedSecret(builder, "keycloak-demo-password");

// Local identity provider with the "bookstore" realm imported from Realms/. Production uses a managed Keycloak
// (or another OIDC provider) with its own realm management; the import is for development only.
IResourceBuilder<KeycloakResource> keycloak = builder.AddKeycloak("keycloak", 8180)
    .WithContainerName("msapp-keycloak")
    .WithDataVolume("msapp-keycloak-data")
    .WithLifetime(ContainerLifetime.Persistent)
    .WithRealmImport("./Realms")
    .WithEnvironment("BOOKSTORE_BFF_CLIENT_SECRET", bffClientSecret)
    .WithEnvironment("BOOKSTORE_BFF_SERVICE_SECRET", bffServiceSecret)
    .WithEnvironment("BOOKSTORE_DEMO_PASSWORD", demoPassword);

AddGeneratedSecret moves the code that generated the webhook secret in 08 into a small helper method at the end of the AppHost. The test users' password is shown in the dashboard as the keycloak-demo-password parameter. The Keycloak log confirmed the import: Realm 'bookstore' imported. The import runs only when the realm does not exist (Strategy: IGNORE_EXISTING); when you change the realm file you have to delete the msapp-keycloak-data volume. Aspire's Keycloak packages, both the AppHost side and the Aspire.Keycloak.Authentication package the services use, were still in preview on the day this post was written. In production the realm should be managed with a separate tool under version control, not imported when the container starts.

The BFF: Sign-in with OIDC, a Session in a Cookie

The structure from the BFF Pattern for SPAs post of the Authentication and Authorization series is set up here: Angular never sees a token. Sign-in happens in the BFF with OIDC, the tokens stay in the BFF's session, and only an HttpOnly cookie goes to the browser:

services.AddAuthentication(options =>
    {
        options.DefaultScheme = CookieAuthenticationDefaults.AuthenticationScheme;
        // An API call without a session gets 401; only /bff/login sends the browser to Keycloak.
        options.DefaultChallengeScheme = CookieAuthenticationDefaults.AuthenticationScheme;
    })
    .AddCookie(options =>
    {
        options.Cookie.Name = "bookstore.session";
        options.Cookie.HttpOnly = true;
        options.Cookie.SameSite = SameSiteMode.Strict;
        options.Cookie.SecurePolicy = isProduction ? CookieSecurePolicy.Always : CookieSecurePolicy.SameAsRequest;
        options.Events.OnRedirectToLogin = context =>
        {
            context.Response.StatusCode = StatusCodes.Status401Unauthorized;
            return Task.CompletedTask;
        };
    })
    .AddKeycloakOpenIdConnect("keycloak", realm: "bookstore", OpenIdConnectDefaults.AuthenticationScheme, options =>
    {
        options.ClientId = "bookstore-bff";
        options.ClientSecret = configuration["Keycloak:ClientSecret"];
        options.ResponseType = "code";
        options.SaveTokens = true;
        options.MapInboundClaims = false;
        options.TokenValidationParameters.NameClaimType = "name";
        options.TokenValidationParameters.RoleClaimType = "roles";
        options.RequireHttpsMetadata = isProduction;
    });

AddKeycloakOpenIdConnect finds Keycloak's address through service discovery (the keycloak resource and the realm name). The BFF has three endpoints: /bff/login starts the OIDC challenge, /bff/logout closes both the cookie and the Keycloak session (in our run /bff/user returned 401 again after logout), and /bff/user gives Angular the user's name, email and roles, not the token. We tried the sign-in with a small script that fills in Keycloak's login form the way a browser does:

login ada: HTTP 200, ended at /
cookies on localhost:5100 side: ['bookstore.session', 'bookstore.sessionC1', 'bookstore.sessionC2']
GET /bff/user -> HTTP 200
{"name":"Ada Lovelace","email":"ada@example.com","roles":["customer"]}
POST /api/orders -> HTTP 400
{"title":"Missing X-CSRF header"}
POST /api/orders -> HTTP 202 Location: /api/orders/01a1da9e-9876-7c65-b35a-f00965282c23

The first answer to Guess First: three. With SaveTokens = true the access, refresh and ID tokens sit encrypted inside the cookie, which exceeds the size limit of a single cookie, so ASP.NET Core splits it into chunks. That means a few extra KB of headers on every request. In production it is better to keep the session on the server (in Redis, for example) and carry only a key in the cookie; we come back to that below.

The first POST getting 400 is on purpose. A cookie is attached to every request the browser makes, including one coming from another site's form. The BFF requires an X-CSRF: 1 header on every non-GET request under /api. A form or an image tag on another site cannot add that header; Angular's HttpClient adds it easily (11). Keeping CSRF protection as one more layer, even with a SameSite=Strict cookie, is a good habit. Requests without a session behave correctly too: /bff/user and POST /api/orders return 401, while the book page is public and returns 200.

Token Propagation: The User's Token Goes to Ordering

The BFF's proxy routes carry the user's access token to the service behind them. In YARP the order routes require a session with "AuthorizationPolicy": "default"; a request transform adds the token:

// Token propagation: the signed-in user's access token goes to the service as a Bearer token.
context.AddRequestTransform(async transform =>
{
    if (transform.HttpContext.User.Identity?.IsAuthenticated == true &&
        await transform.HttpContext.GetTokenAsync("access_token") is { } accessToken)
    {
        transform.ProxyRequest.Headers.Authorization = new("Bearer", accessToken);
    }
});

Ordering validates the token itself and now takes the buyer from the token, not from the request body. This is the change the comments have promised since 04:

// Claims as Keycloak issues them (MapInboundClaims = false keeps their original names).
private static Buyer BuyerFrom(ClaimsPrincipal user) => new(
    user.FindFirstValue("sub") ?? throw new InvalidOperationException("The access token has no subject."),
    user.FindFirstValue("name") ?? "",
    (user.FindFirstValue("email") ?? "").ToLowerInvariant());

Buyer got the identity provider's stable user id (sub), and a migration added a Buyer_Id column to the Orders table. The "my orders" list now works with the sub from the token instead of an email parameter, and someone else's order id gets the same 404 as an id that does not exist. That is a contract change, and oasdiff reported it with three warnings: the customerName and customerEmail fields left the POST, the customerEmail parameter left the GET. No client uses this API yet, so we accepted the change. Validation and authorization on the Ordering side:

// Every order endpoint needs a customer's access token for the bookstore-api audience. The BFF forwards it.
builder.Services.AddAuthentication()
    .AddKeycloakJwtBearer("keycloak", realm: "bookstore", options =>
    {
        options.Audience = "bookstore-api";
        options.MapInboundClaims = false;
        options.TokenValidationParameters.RoleClaimType = "roles";
        options.RequireHttpsMetadata = builder.Environment.IsProduction();
    });
builder.Services.AddAuthorizationBuilder()
    .AddPolicy("customer", policy => policy.RequireRole("customer"));

We authorize with roles. Defining client scopes per API in Keycloak and building scope-based policies would also work; doing that without breaking the realm's default scopes is a job of its own and is covered in detail in the Protecting the API post of the Authentication and Authorization series. In Bookstore the role answers "who" and the audience answers "for which API".

Every Service Validates Its Own Token

The rule from the security post of the Microservices series: don't trust the internal network; every service validates the incoming token itself. The second question from Guess First tests exactly that. We got a client credentials token with the bookstore-bff-service client (audience inventory-api) and sent it to Ordering, skipping the BFF:

{'iss': 'http://localhost:8180/realms/bookstore', 'aud': ['inventory-api', 'account'], 'azp': 'bookstore-bff-service', 'scope': 'profile email'}

GET /api/v1/orders (no token)
HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer

GET /api/v1/orders (token issued for Inventory)
HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer error="invalid_token", error_description="The audience '(null)' is invalid"

The signature is valid and the issuer is right, but the audience is wrong: the token is rejected. The (null) in the error message comes from IdentityModel, the library that validates the token, hiding values that may count as personal data by default. Catalog's admin endpoints require the store-staff role. We sent the same price change through the BFF as Ada and as Grace:

login ada: HTTP 200, ended at /
PUT /api/catalog/admin/books/9780141439549/price -> HTTP 403
login grace: HTTP 200, ended at /
PUT /api/catalog/admin/books/9780141439549/price -> HTTP 204

Here the difference between 401 and 403 shows: Ada proved who she is but is not allowed to do this. Grace's change reached Ordering's copy through the outbox from 06 (PriceVersion 2).

Client Credentials for Inventory

The book page's stock query is not a user action; someone who is not signed in sees the book page too. The BFF makes this call with its own identity: it gets a token for bookstore-bff-service with OAuth 2.0 client credentials, keeps it in memory until 30 seconds before it expires, and adds it to the gRPC call. Inventory accepts only tokens with the inventory-api audience. Carrying the user's token to Inventory would break two things: an anonymous user has no token, and the user's token should have no access to Inventory.

builder.Services.AddGrpcClient<InventoryService.InventoryServiceClient>(options => options.Address = new Uri("http://_grpc.inventory"))
    // The stock query is the BFF's own call, not the user's: it carries the BFF's client credentials token.
    .AddCallCredentials(async (_, metadata, services) =>
        metadata.Add("Authorization", $"Bearer {await services.GetRequiredService<ServiceTokenProvider>().GetAsync(default)}"))
    // gRPC refuses to send call credentials over a channel without TLS. Locally the channel is plain HTTP/2;
    // in production TLS (or the service mesh) protects it and this setting goes away.
    .ConfigureChannel(channel => channel.UnsafeUseInsecureChannelCallCredentials = !builder.Environment.IsProduction());

Without the last line the book page returned 500 in 22 ms and the log stated the reason plainly: Call credential configured for gRPC client 'InventoryServiceClient' requires TLS, and the client isn't configured to use TLS. The error is an InvalidOperationException, not an RpcException, so the "show the page even without stock" logic from 09 did not catch it. One more cost of the cleartext HTTP/2 decision from 05. Calling Inventory without a token through the probe tool from 05 gives: Unauthenticated: Bad gRPC response. HTTP status code: 401.

Identity That Travels with Events

There is no HTTP request behind the saga and the consumers; it is not a user that sends ReserveStock to Inventory, it is Ordering. So messages carry no tokens. When "who started this" is needed, it becomes a field of the message:

// CustomerId is the identity provider's subject: consumers learn who acted from the message, never from a token.
public sealed record OrderPlaced(
    Guid OrderId, string CustomerId, string CustomerEmail, IReadOnlyList<OrderPlacedLine> Lines, decimal Total, string Currency,
    DateTimeOffset PlacedAtUtc);

OrderConfirmed and OrderCancelled got the same field (adding a field is not breaking, by the rule from 06), and the saga state carries it. In 11 Notifications will use it to send a SignalR message to the right user. After Ada's order the Notifications log said:

[notifications]       Order 01a1da9e-9876-7c65-b35a-f00965282c23 placed by 80b5e1c1-1bd5-4d68-bf17-e697950727c5 (ada@example.com): 199.90 TRY, 1 line(s)

The consumers' own identity is their RabbitMQ connection's identity: every service should connect to the broker with its own user and be able to read only its own queues. Locally they all use the same user; in production each service should get its own RabbitMQ user with permissions only on its own queues.

In Production

Token refresh is missing. In Keycloak the access token is valid for 5 minutes (accessTokenLifespan: 300); the BFF's cookie session lasts longer. After five minutes the BFF carries an expired token and Ordering returns 401. The BFF needs to renew the access token with the refresh token; instead of writing that by hand, a library such as Duende.AccessTokenManagement.OpenIdConnect can do it. The details are in the BFF for SPAs post of the Authentication and Authorization series.

Keep the session on the server. Giving cookie authentication's SessionStore an ITicketStore backed by Redis shrinks the three-part cookie to a small key and really deletes the session on the server at logout.

HTTPS and cookie settings. In production the cookie should be Secure and carry the __Host- name prefix, which forces it to be set only over HTTPS, only for that exact host and with Path=/. Locally we use plain HTTP, so we turn these settings on only in production.

Rotate the secrets. The client secrets live in Aspire parameters; in production they should come from a secret store and be changeable in Keycloak and the BFF together without downtime. Client authentication with a signed JWT instead of a client secret is another option.

Common Pitfalls

Trusting the internal network and turning authentication off in the services. A service that says "it is behind the BFF anyway" accepts any internal process that can skip the BFF, or a compromised pod. That is why the audience check above runs in every service.

Putting the access token into an event payload. Tokens are short-lived; a message can wait in a queue for hours and in a DLQ for days, and anyone who reads it could call APIs as the user. The message carries who acted, not a token.

Giving the token to Angular. A token in the browser can be stolen through XSS. The point of the BFF is to keep the token on the server.

Assuming claim names. By default ASP.NET Core turns claims such as sub into long XML names; MapInboundClaims = false keeps the names Keycloak issues. The role claim's name should be set explicitly with RoleClaimType too, or RequireRole finds no roles.

Sending the ID token to an API. The ID token is for the client; its audience is the BFF itself. APIs expect access tokens.

A Small Experiment

★☆☆ Sign in as Ada, wait more than five minutes and call GET /api/orders again. Which error shows up in Ordering's log? Why do you get 401 while the cookie is still valid?

★★☆ Add a field specific to the signed-in user to Catalog's read endpoints (for example, "is it in their favourites"). When the answer depends on the user, what should the output cache from 03 do? What do you need to add to the cache key?

★★★ Write the ITicketStore with Redis and shrink the BFF's cookie to a single, small key. What happens when you send the old cookie again after logout; what happened with the chunked cookie?

The services now know who is calling, each one validates its own token, and identity travels with events. Still ahead in this series are the Angular interface, distributed traces, tests, deployment and the look back: in the next post the Angular 22 application moves behind the BFF and we show the customer, live, how an order goes from "waiting for confirmation" to "confirmed".