gRPC in Production: Interceptors, Authentication, Load Balancing, Retries and Health Checks

Version note: The code in this post was built and run with .NET 10, Grpc.AspNetCore 2.85, Grpc.Net.ClientFactory 2.85, Grpc.AspNetCore.HealthChecks 2.85, Microsoft.AspNetCore.Authentication.JwtBearer 10.0 and Microsoft.Extensions.Http.Resilience 10.11.

Guess First

You've deployed Inventory to Kubernetes as three pods and put a standard ClusterIP Service in front of them. Ordering makes hundreds of GetStock calls a second. You look at the three pods' CPU graphs in Grafana. What do you see: three equal lines, or one pod on fire while the other two sit idle?

Second question: you've given Ordering's gRPC client a retry policy, "retry 3 times on Unavailable". A teammate also added AddStandardResilienceHandler() to the same client, which retries 3 times by default as well. When the proxy in front of Inventory starts returning 503, how many HTTP requests does a single GetStock call turn into by the time it reaches Inventory?

Interceptors: Putting Code Around Every Call

An interceptor is gRPC's middleware. ASP.NET Core middleware also runs for gRPC requests, but it sees the HTTP request; an interceptor sees the deserialized message, the method name and the ServerCallContext. We add two server interceptors to Inventory. The first logs each call's result and duration and adds a trailer telling which instance answered:

public sealed class ServerLoggingInterceptor(ILogger<ServerLoggingInterceptor> logger, IConfiguration configuration) : Interceptor
{
    private readonly string _instance = configuration["Inventory:InstanceName"] ?? "inventory";

    public override async Task<TResponse> UnaryServerHandler<TRequest, TResponse>(
        TRequest request, ServerCallContext context, UnaryServerMethod<TRequest, TResponse> continuation)
    {
        long started = Stopwatch.GetTimestamp();
        StatusCode status = StatusCode.OK;
        try
        {
            context.ResponseTrailers.Add("x-served-by", _instance);
            return await continuation(request, context);
        }
        catch (RpcException ex)
        {
            status = ex.StatusCode;
            throw;
        }
        finally
        {
            logger.LogInformation("{Method} {Status} in {Elapsed:0.0} ms, peer {Peer}, attempt {Attempt}",
                context.Method, status, Stopwatch.GetElapsedTime(started).TotalMilliseconds, context.Peer,
                context.RequestHeaders.GetValue("grpc-previous-rpc-attempts") ?? "0");
        }
    }
}

The second turns domain exceptions into statuses. The service code throws ReservationNotFoundException and knows nothing about gRPC; the translation happens in one place:

public sealed class ExceptionMappingInterceptor(ILogger<ExceptionMappingInterceptor> logger) : Interceptor
{
    public override async Task<TResponse> UnaryServerHandler<TRequest, TResponse>(
        TRequest request, ServerCallContext context, UnaryServerMethod<TRequest, TResponse> continuation)
    {
        try
        {
            return await continuation(request, context);
        }
        catch (RpcException)
        {
            throw;
        }
        catch (ReservationNotFoundException ex)
        {
            throw new RpcException(new Status(StatusCode.NotFound, ex.Message));
        }
        catch (Exception ex) when (ex is not OperationCanceledException)
        {
            // details go to the log, the client only gets a generic message
            logger.LogError(ex, "Unhandled exception in {Method}", context.Method);
            throw new RpcException(new Status(StatusCode.Internal, "Internal error. Correlation: " + context.GetHttpContext().TraceIdentifier));
        }
    }
}

builder.Services.AddGrpc(options =>
{
    // registration order = execution order: logging is outermost
    options.Interceptors.Add<ServerLoggingInterceptor>();
    options.Interceptors.Add<ExceptionMappingInterceptor>();
});

Order matters: logging is outside, so it sees the status produced by exception mapping. These two interceptors only override UnaryServerHandler; streaming calls have separate methods such as ServerStreamingServerHandler, and call types you don't override never pass through the interceptor. The client side follows the same model. Our client interceptor adds a correlation id to every call:

public sealed class CorrelationInterceptor : Interceptor
{
    public override AsyncUnaryCall<TResponse> AsyncUnaryCall<TRequest, TResponse>(
        TRequest request, ClientInterceptorContext<TRequest, TResponse> context,
        AsyncUnaryCallContinuation<TRequest, TResponse> continuation)
    {
        // add a correlation id to every call; the calling code doesn't need to know
        Metadata headers = context.Options.Headers ?? [];
        headers.Add("x-correlation-id", Guid.NewGuid().ToString("N")[..8]);
        var options = context.Options.WithHeaders(headers);

        return continuation(request, new ClientInterceptorContext<TRequest, TResponse>(context.Method, context.Host, options));
    }
}

Authentication: JWT Bearer and Call Credentials

In gRPC, authentication is done with ASP.NET Core's own infrastructure; it's the same setup we built with JWT in the ASP.NET Core Web API series. Reads are open, and methods that change stock require the inventory.write scope:

builder.Services.AddAuthentication().AddJwtBearer(options =>
{
    options.TokenValidationParameters = new TokenValidationParameters
    {
        ValidIssuer = "bookstore-identity",
        ValidAudience = "bookstore-inventory",
        IssuerSigningKey = new SymmetricSecurityKey(Encoding.UTF8.GetBytes(builder.Configuration["Jwt:SigningKey"]!))
    };
});
builder.Services.AddAuthorizationBuilder()
    .AddPolicy("inventory.write", policy => policy.RequireClaim("scope", "inventory.write"));

[Authorize(Policy = "inventory.write")]
public override async Task<ReserveStockResponse> ReserveStock(ReserveStockRequest request, ServerCallContext context)

The symmetric key is here to keep the experiment self-contained. In production an identity provider issues the token and Inventory validates it against that provider's public keys via Authority. On the client side, the way to attach the token to every call is call credentials:

services
    .AddGrpcClient<InventoryService.InventoryServiceClient>(o => o.Address = new Uri("http://localhost:5101"))
    .AddInterceptor(() => new CorrelationInterceptor())
    .AddCallCredentials((context, metadata) =>
    {
        if (token is not null)
        {
            metadata.Add("Authorization", $"Bearer {token}");
        }
        return Task.CompletedTask;
    })
    .ConfigureChannel(o => o.UnsafeUseInsecureChannelCallCredentials = allowInsecure);

I tried four cases in turn, and in the last step tried to release a reservation that doesn't exist:

no token                       -> Unauthenticated Bad gRPC response. HTTP status code: 401
token without scope            -> PermissionDenied Bad gRPC response. HTTP status code: 403
write token, insecure channel  -> InvalidOperationException: Call credential configured for gRPC client 'InventoryServiceClient' requires TLS, and the client isn't configured to use TLS
write token                    -> OK r-67f3a33f
release unknown reservation    -> NotFound Reservation r-missing not found.

The first two lines are interesting: the responses carry no grpc-status; the authorization middleware cuts the request off with HTTP 401 and 403 before it reaches gRPC. The client turns these into Unauthenticated and PermissionDenied using the mapping table from the previous post. The third line is a safety measure: by default, call credentials don't work on a channel without TLS, because the token would travel in plain text. UnsafeUseInsecureChannelCallCredentials is only for local experiments or environments where TLS terminates at a proxy in front. The last line is the exception mapping interceptor at work.

For service-to-service communication, mTLS (mutual TLS) is an alternative: the client presents a certificate too and the server validates it; certificate management replaces token management. In .NET the client certificate goes on the handler via ConfigurePrimaryHttpMessageHandler, and the server side is set up with Kestrel's ClientCertificateMode setting. In environments with a service mesh, the mesh's sidecar usually takes this job on.

Load Balancing: Why HTTP/2 Sticks to One Pod

Back to the first opening question. In HTTP/2, a channel opens one TCP connection and multiplexes all calls over it. An L4 (transport layer) load balancer only distributes connections; once a connection is assigned to a backend, every call inside it goes there. Kubernetes' ClusterIP Service also distributes at connection level by default.

To see this, I started Inventory as three instances (inv-1, inv-2, inv-3) and put a small test proxy in front of them that hands each new TCP connection to the next backend. I made 30 GetStock calls over one channel and counted the x-served-by trailers in the responses; I ran the client twice:

inv-1=30
inv-2=30

In each run every call went to a single instance. The proxy saw two connections and gave them to two different backends; that's all. The answer to the opening question: one pod burns, the others sit idle. On top of that, adding new pods doesn't help existing clients, because their connections are already established.

There are two solutions. The first is client-side load balancing: the client knows all the backend addresses, opens a connection to each and distributes the calls. A resolver supplies the addresses:

public static GrpcChannel CreateStaticBalancedChannel()
{
    var services = new ServiceCollection();
    services.AddSingleton<ResolverFactory>(new StaticResolverFactory(_ =>
    [
        new BalancerAddress("localhost", 5201),
        new BalancerAddress("localhost", 5202),
        new BalancerAddress("localhost", 5203)
    ]));

    return GrpcChannel.ForAddress("static:///inventory", new GrpcChannelOptions
    {
        Credentials = ChannelCredentials.Insecure,
        ServiceConfig = new ServiceConfig { LoadBalancingConfigs = { new RoundRobinConfig() } },
        ServiceProvider = services.BuildServiceProvider()
    });
}

inv-1=10, inv-2=10, inv-3=10

Since the address no longer starts with http://, you have to state whether TLS is used via Credentials. If you don't specify a load balancer, the default is pick_first: it sends everything to the first address it can connect to, which means a single instance again.

In Kubernetes, DNS supplies the address list. A headless Service (clusterIP: None) answers DNS queries with the pods' IP addresses instead of the Service's single virtual IP, and the dns:/// resolver spreads calls across them:

public static GrpcChannel CreateDnsBalancedChannel() =>
    GrpcChannel.ForAddress("dns:///localhost:5201", new GrpcChannelOptions
    {
        Credentials = ChannelCredentials.Insecure,
        ServiceConfig = new ServiceConfig { LoadBalancingConfigs = { new RoundRobinConfig() } }
    });

Locally localhost resolves to two addresses (127.0.0.1 and ::1); both lead to the same instance, but the peer field in the server log shows the calls being spread across two separate connections in turn:

peer ipv4:127.0.0.1:56220, attempt 0
peer ipv4:127.0.0.1:56220, attempt 0
peer ipv6:[::1]:56221, attempt 0
peer ipv4:127.0.0.1:56220, attempt 0
peer ipv6:[::1]:56221, attempt 0
peer ipv4:127.0.0.1:56220, attempt 0

One thing to watch: the DNS resolver caches addresses and by default re-queries only when a connection drops. To see new pods in a deployment that scales, register DnsResolverFactory with a refreshInterval.

The second solution is an L7 (application layer) proxy: Envoy, Linkerd or, on the .NET side, YARP understands HTTP/2 and distributes the calls from a single connection to separate backends. The client stays simple; in exchange every call gets an extra network hop and the proxy brings operational overhead. We cover this route in detail in the API Gateway and YARP series. Whichever you choose, once a streaming call is established all its messages go to the same backend; distribution happens per call.

Retries and Hedging

The gRPC client has its own retry mechanism, defined per method in the channel's service config:

public static ServiceConfig CreateRetryServiceConfig() => new()
{
    MethodConfigs =
    {
        new MethodConfig
        {
            // reads only: running it again is harmless
            Names = { new MethodName { Service = "bookstore.inventory.v1.InventoryService", Method = "GetStock" } },
            RetryPolicy = new RetryPolicy
            {
                MaxAttempts = 3,
                InitialBackoff = TimeSpan.FromMilliseconds(200),
                MaxBackoff = TimeSpan.FromSeconds(2),
                BackoffMultiplier = 2,
                RetryableStatusCodes = { StatusCode.Unavailable }
            }
        }
    }
};

Applying it to all methods with MethodName.Default is easy but dangerous: ReserveStock isn't idempotent, and if the first attempt succeeded on the server but the response was lost on the way, a retry opens a second reservation. Limit retries to methods that are harmless to run again. I started Inventory so that it fails the first two GetStock calls with Unavailable and made a single call. One line on the client, three on the server:

GetStock OK, available=12

/bookstore.inventory.v1.InventoryService/GetStock Unavailable in 3.7 ms, peer ipv6:[::1]:56346, attempt 0
/bookstore.inventory.v1.InventoryService/GetStock Unavailable in 0.1 ms, peer ipv6:[::1]:56346, attempt 1
/bookstore.inventory.v1.InventoryService/GetStock OK in 5.2 ms, peer ipv6:[::1]:56346, attempt 2

The attempt column is the grpc-previous-rpc-attempts header the client adds automatically. Retries have limits, too: the moment the server sends response headers (for example when the first message arrives in a server streaming call), the call is "committed" and is no longer retried. The deadline also covers all attempts. Hedging, on the other hand, sends several copies of the same call without waiting and uses the first successful response; it lowers latency but multiplies load. Retry and hedging can't both be defined for the same method.

What about AddStandardResilienceHandler? The handler from Microsoft.Extensions.Http.Resilience works at the HttpClient level and can be added to gRPC clients from Grpc.Net.ClientFactory 2.64 onward (earlier versions threw an exception at runtime). But it sees HTTP, not the gRPC status. First I tried the resilience handler alone, twice. When Inventory returned an Unavailable status:

Unavailable: Simulated outage.

/bookstore.inventory.v1.InventoryService/GetStock Unavailable in 4.2 ms, peer ipv6:[::1]:56352, attempt 0

No retry at all: the HTTP status is 200, a successful request in the handler's eyes. When I tried again with a test middleware imitating a proxy in front of Inventory returning 503, the handler kicked in and the server saw four HTTP requests. Then the second question's scenario: both a gRPC retry policy and the resilience handler on the same client:

Unavailable: Bad gRPC response. HTTP status code: 503
4
Unavailable: Bad gRPC response. HTTP status code: 503
12

A single GetStock reached Inventory as 12 HTTP requests: the gRPC layer makes 3 attempts and the handler sends 4 requests per attempt. Multiply that by the retries of Ordering's own caller and you send a wave of load to a service that's already struggling. In the System Design series we talked about retries multiplying across layers in general; this is its gRPC form. The rule is simple: retry in one layer only. If you want decisions based on gRPC statuses, that layer is gRPC's own retry policy. The in-depth treatment of other patterns such as circuit breakers and timeouts is the subject of the External Service Integration and Resilience series.

Health Checks, Reflection and Limits

Load balancers and Kubernetes need to ask an instance about its health before sending traffic to it. gRPC has a standard protocol for this (grpc.health.v1.Health). The Grpc.AspNetCore.HealthChecks package connects it to ASP.NET Core health checks, and Kubernetes' built-in gRPC probe speaks the same protocol:

builder.Services.AddGrpcReflection();
builder.Services.AddGrpcHealthChecks()
    .AddCheck("stock-store", () => Microsoft.Extensions.Diagnostics.HealthChecks.HealthCheckResult.Healthy());

app.MapGrpcHealthChecksService();

if (app.Environment.IsDevelopment())
{
    app.MapGrpcReflectionService();
}

Reflection makes the server's own contract queryable. Tools such as grpcurl or Postman can list and call methods without the .proto file. I queried both the health check and reflection with the clients from the Grpc.HealthCheck and Grpc.Reflection packages:

health: Serving
service: bookstore.inventory.v1.InventoryService
service: grpc.health.v1.Health
service: grpc.reflection.v1alpha.ServerReflection
service: grpc.reflection.v1.ServerReflection

When I started the same app in the Production environment, the health check still returned Serving, while reflection returned Unimplemented: Service is unimplemented. Turning reflection on in production means describing your whole API surface to anyone who doesn't know your contract; we limited it to Development.

Finally, message size. By default the server rejects incoming messages larger than 4 MB. When I sent a 5 MB book_id:

ResourceExhausted: Received message exceeds the maximum configured message size.

This error never reached the interceptors; the message is rejected before it's deserialized. You can change the limit per service with AddGrpc(o => o.MaxReceiveMessageSize = ...), but before raising it, consider whether splitting the message (streaming or paging) is the better fix. For the concurrent stream limit we saw in the previous post there's SocketsHttpHandler.EnableMultipleHttp2Connections. GrpcChannel already turns it on in the handler it creates itself; when you supply your own handler you have to set it to true explicitly.

In Production

Test load balancing before you deploy. In a three-pod staging environment, send load from a single client and look at requests per pod. A setup that balances at connection level shows itself on day one.

With client-side load balancing (LB), keeping addresses current is your responsibility. The DNS cache determines when new pods start getting traffic. If new pods sit idle after a scale-out, look at the resolver's refresh setting first.

Make the health check actually mean something. A check that always returns Healthy tells the load balancer nothing. Check the conditions under which you shouldn't receive traffic, such as the database connection or a critical dependency. But don't add every downstream either, or one dependency's problem takes every instance out at once.

Don't do heavy work in interceptors. They run on every call; synchronous I/O, serializing large objects or logging the message (it may contain personal data) adds both latency and risk.

Common Pitfalls

gRPC behind a Kubernetes ClusterIP. All traffic goes to whichever pod the connection first landed on. Use a headless Service with client-side LB, or an L7 proxy (or a service mesh).

Using dns:/// without specifying a load balancer. The default is pick_first; even if several addresses resolve, everything goes over one connection.

Spreading retries across layers. The gRPC retry policy, the resilience handler and the caller's own retries multiply each other. We produced 12 requests from a single call.

Putting non-idempotent methods into retries via MethodName.Default. Calls such as reservations, payments and order creation should either stay out of retries or be protected with an idempotency key.

Reflection and EnableDetailedErrors in production. Both are development conveniences; in production they hand an attacker a map.

A Small Experiment

★☆☆ Add a ServerStreamingServerHandler override to ServerLoggingInterceptor and log how long WatchStock calls stay open.

★★☆ Stop one of the three instances in the static resolver and repeat the 30 calls. Observe how round robin takes the closed address out of rotation and what happens when the instance comes back.

★★★ Protect ReserveStock with an idempotency key (if the same order_id arrives a second time, return the first reservation) and then define a retry policy for this method too. Use the test middleware to drop the response on the way and show that no duplicate reservation is created.

Inventory is now ready for production, but it's open only to clients that speak gRPC. In the next post we open the same service to Angular in the browser with gRPC-Web, and to clients expecting REST (Representational State Transfer) with JSON transcoding.