Your First gRPC Service and Client in ASP.NET Core: Kestrel, Deadlines and RpcException

Version note: The code in this post was built and run with .NET 10, Grpc.AspNetCore 2.85, Grpc.Net.ClientFactory 2.85 and Grpc.StatusProto 2.85.

Guess First

A customer clicks "Complete Order". The checkout side calls Ordering with a one-second deadline, and Ordering calls Inventory to reserve stock. Inventory's database is slow at that moment and the reservation takes three seconds. When the second is up, checkout says "timed out" and gives up.

Here's the question: what happens to the work in Inventory two seconds later? Does it finish and reserve the stock, or does it give up too? The answer depends on how Ordering registered its client, and at the end of the post we'll look at the real logs of both cases side by side.

Bringing Up Inventory: AddGrpc and MapGrpcService

We now turn the inventory.proto from the previous post into a real service. The solution has three projects: Bookstore.Inventory (gRPC server), Bookstore.Ordering (both a server and Inventory's client) and a small console tool for experiments, Bookstore.StockCheck. Ordering now has a contract of its own:

syntax = "proto3";

package bookstore.ordering.v1;

option csharp_namespace = "Bookstore.Ordering.V1";

import "bookstore/inventory/v1/inventory.proto";

service OrderingService {
  rpc PlaceOrder (PlaceOrderRequest) returns (PlaceOrderResponse);
}

message PlaceOrderRequest {
  string customer_id = 1;
  repeated bookstore.inventory.v1.ReservationLine lines = 2;
}

message PlaceOrderResponse {
  string order_id = 1;
  string reservation_id = 2;
}

Since the folder now holds the ordering contract too, Inventory compiles only the files it cares about:

<ItemGroup>
    <PackageReference Include="Grpc.AspNetCore" Version="2.85.0" />
    <PackageReference Include="Grpc.StatusProto" Version="2.85.0" />
</ItemGroup>

<ItemGroup>
    <Protobuf Include="..\..\protos\bookstore\common\**\*.proto;..\..\protos\bookstore\inventory\**\*.proto" ProtoRoot="..\..\protos" GrpcServices="Server" />
    <Compile Include="..\..\protos\DecimalValue.cs" Link="Protos\DecimalValue.cs" />
</ItemGroup>

We moved DecimalValue's partial conversion file into the protos/ folder and linked it into both projects; since each project generates the messages itself, the partial class has to be compiled in each project too. Program.cs is short:

using Bookstore.Inventory.Services;
using Bookstore.Inventory.Stock;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddGrpc();
builder.Services.Configure<InventoryOptions>(builder.Configuration.GetSection("Inventory"));
builder.Services.AddSingleton<InMemoryStockStore>();

var app = builder.Build();

app.MapGrpcService<InventoryGrpcService>();

app.Run();

AddGrpc registers the gRPC infrastructure with DI, and MapGrpcService opens an endpoint for each method in the contract, such as /bookstore.inventory.v1.InventoryService/GetStock. You don't choose the route; it comes from the package, service and method names. The service class derives from the generated abstract base and takes its dependencies through a primary constructor:

public sealed class InventoryGrpcService(InMemoryStockStore store, ILogger<InventoryGrpcService> logger)
    : InventoryService.InventoryServiceBase
{
    public override async Task<GetStockResponse> GetStock(GetStockRequest request, ServerCallContext context)
    {
        Guid bookId = ParseBookId(request.BookId, "book_id");
        string? correlationId = context.RequestHeaders.GetValue("x-correlation-id");
        logger.LogInformation("GetStock {BookId}, correlation {CorrelationId}", bookId, correlationId);

        StockItem item = await store.FindAsync(bookId, context.CancellationToken)
            ?? throw new RpcException(new Grpc.Core.Status(StatusCode.NotFound, $"Book {bookId} has no stock record."));

        context.ResponseTrailers.Add("x-stock-source", "memory");

        return new GetStockResponse
        {
            BookId = item.BookId.ToString(),
            Available = item.Available,
            ReservedQuantity = item.Reserved,
            Status = item.Available switch
            {
                0 => StockStatus.OutOfStock,
                < 3 => StockStatus.Low,
                _ => StockStatus.InStock
            },
            UnitPrice = new Money { CurrencyCode = "TRY", Amount = DecimalValue.FromDecimal(item.UnitPrice) },
            UpdatedAt = Timestamp.FromDateTimeOffset(item.UpdatedAt)
        };
    }
}

By default the service class is created anew for each call (when I put a counter into the logs of two PlaceOrder calls I saw two separate instances); it can take Singleton, Scoped or Transient dependencies. Here the store is a singleton because it keeps stock in memory; in real life a scoped DbContext would sit in its place. I spell out Grpc.Core.Status in full because it clashes with Google.Rpc.Status, which we'll use shortly.

Kestrel and HTTP/2: TLS, ALPN and h2c

gRPC needs version 2 of HTTP, i.e. HTTP/2. Kestrel can speak both HTTP/1.1 and HTTP/2 on one endpoint, but which one is used is decided during the TLS handshake via ALPN. In production the endpoint should use TLS and this negotiation just works. If you use unencrypted HTTP/2 (h2c) locally, or where TLS terminates at a proxy in front, you have to say explicitly that the endpoint speaks only HTTP/2:

"Kestrel": {
  "Endpoints": {
    "Grpc": {
      "Url": "http://localhost:5101",
      "Protocols": "Http2"
    }
  }
}

I tried what happens with Http1AndHttp2. Without TLS there's no negotiation, Kestrel treats the connection as HTTP/1.1 and closes the client's HTTP/2 connection:

Internal: Error starting gRPC call. HttpRequestException: The HTTP/2 server closed the connection. HTTP/2 error code 'HTTP_1_1_REQUIRED' (0xd). (HttpProtocolError) HttpProtocolException: The HTTP/2 server closed the connection. HTTP/2 error code 'HTTP_1_1_REQUIRED' (0xd). (HttpProtocolError)

If you see this error, the first place to look is the endpoint's Protocols setting. If you want to serve both REST (Representational State Transfer) and gRPC from the same app, there are two ways: a single TLS endpoint (ALPN separates them) or two ports, one Http1 and one Http2. We covered the connection-level details of HTTP/2 through Kestrel in the HTTP/2 post of the Networking and How HTTP Works series.

If you come across AppContext.SetSwitch("System.Net.Http.SocketsHttpHandler.Http2UnencryptedSupport", true) in an old project, it was needed in .NET Core 3.x for the client to speak h2c. Since .NET 5, a GrpcChannel with an http:// address uses h2c without extra configuration.

The Client: Channels Are Expensive, Clients Are Cheap

There are two concepts on the client side. GrpcChannel represents the long-lived connection to the server (an HttpClient and TCP connections underneath). The generated InventoryServiceClient is a thin wrapper on top of the channel; creating one is almost free. Hence the rule: create the channel once and share it; create clients as often as you like.

// a channel is expensive: create it once, use it for the app's lifetime
using GrpcChannel inventoryChannel = GrpcChannel.ForAddress("http://localhost:5101");
var inventory = new InventoryService.InventoryServiceClient(inventoryChannel);

To measure the difference I made the same GetStock call 200 times, first with a shared channel, then with a new channel every time:

for (int i = 0; i < 200; i++)
{
    // wrong: a new channel and a new TCP connection per call
    using GrpcChannel channel = GrpcChannel.ForAddress("http://localhost:5101");
    await new InventoryService.InventoryServiceClient(channel).GetStockAsync(request);
}

reused channel:   30 ms
channel per call: 93 ms
reused channel:   31 ms
channel per call: 95 ms
reused channel:   32 ms
channel per call: 94 ms

Release build, same machine, no TLS: a threefold difference. On a real network every new connection adds a TLS handshake and network latency, so the gap grows, and on top of that every closed connection lingers in TIME_WAIT in the operating system for a while.

In an ASP.NET Core app we use the client factory rather than managing the channel by hand. Ordering's Program.cs:

builder.Services.AddGrpc();
builder.Services
    .AddGrpcClient<InventoryService.InventoryServiceClient>(options =>
    {
        options.Address = new Uri(builder.Configuration["Services:Inventory"]!);
    })
    .EnableCallContextPropagation();

AddGrpcClient is built on IHttpClientFactory: the client is injected as transient, while the underlying HttpMessageHandler is pooled and recycled by the factory. Asking for InventoryService.InventoryServiceClient in the service's constructor is enough. We'll come back to EnableCallContextPropagation at the end of the post; the answer to the opening question is there.

Status Codes and RpcException

In gRPC every call ends with a status. OK if it succeeded, otherwise an error code and an optional description. On the server you throw an RpcException to return an error; the client catches the same type. The ones we use most in Bookstore:

• NotFound: the requested record doesn't exist. A book with no stock record.
• InvalidArgument: the request is wrong regardless of the system's state. A book_id that isn't a GUID.
• FailedPrecondition: the request is valid but the system isn't in a state that allows it right now. Reserving two of the last copy. Retrying without the state changing is pointless.
• Unavailable: a transient problem, the server can't be reached. The right candidate for a retry.
• DeadlineExceeded: the call didn't finish in the given time. The work on the server may or may not have completed.

If an exception other than RpcException escapes on the server, the client gets an Unknown status with the message "Exception was thrown by handler."; the real message isn't sent so it can't leak sensitive information. You can turn it on in development with EnableDetailedErrors; it should stay off in production.

So where's the HTTP status? When I asked about an unregistered book directly with curl, the response looked like this:

< HTTP/2 200
< content-type: application/grpc
< date: Fri, 13 Nov 2026 21:13:55 GMT
< server: Kestrel
< content-length: 0
< grpc-status: 5
< grpc-message: Book 00000000-0000-0000-0000-000000000000 has no stock record.

The HTTP status is 200. The real result is in the grpc-status header: 5, i.e. NotFound. Normally the status is carried in trailers that come after the body; since there's no body here, the server returned a "trailers-only" response and put the status straight into the headers. That's why, if you watch gRPC traffic with a monitoring tool that looks at HTTP status, every error looks like a success.

There's a reverse direction too. If a proxy between client and server (a load balancer, an API gateway) returns an HTTP error without grpc-status, the gRPC client converts it to a status using a mapping table: 401 → Unauthenticated, 403 → PermissionDenied, 404 → Unimplemented, 429/502/503/504 → Unavailable, everything else Unknown. So if you see Unimplemented on the client, the cause is usually a wrong route or wrong port, not a missing method on the server.

The Rich Error Model and Metadata

A status code and a one-line message say "what happened" but not "which field, and why". For that there's Google's google.rpc.Status model: a code, a message and detail messages packed with Any. In .NET, the Grpc.StatusProto package turns this model into an RpcException. For an invalid id:

private static Guid ParseBookId(string value, string field)
{
    if (Guid.TryParse(value, out Guid bookId))
    {
        return bookId;
    }

    throw new Google.Rpc.Status
    {
        Code = (int)Code.InvalidArgument,
        Message = "Invalid request.",
        Details =
        {
            Any.Pack(new BadRequest
            {
                FieldViolations =
                {
                    new BadRequest.Types.FieldViolation { Field = field, Description = $"'{value}' is not a valid UUID." }
                }
            })
        }
    }.ToRpcException();
}

When a reservation runs short of stock we likewise return FailedPrecondition with a PreconditionFailure detail. On the client, GetRpcStatus() gives you the model and GetDetail<T>() the detail you want:

catch (RpcException ex)
{
    Console.WriteLine($"{ex.StatusCode}: {ex.Status.Detail}");

    Google.Rpc.Status? rich = ex.GetRpcStatus();
    if (rich?.GetDetail<BadRequest>() is { } badRequest)
    {
        foreach (BadRequest.Types.FieldViolation violation in badRequest.FieldViolations)
        {
            Console.WriteLine($"  field {violation.Field}: {violation.Description}");
        }
    }

    if (rich?.GetDetail<PreconditionFailure>() is { } precondition)
    {
        foreach (PreconditionFailure.Types.Violation violation in precondition.Violations)
        {
            Console.WriteLine($"  {violation.Type} {violation.Subject}: {violation.Description}");
        }
    }
}

The real output of three scenarios:

== notfound
NotFound: Book 00000000-0000-0000-0000-000000000000 has no stock record.
== invalid
InvalidArgument: Invalid request.
  field book_id: '42' is not a valid UUID.
== reserve-too-many
FailedPrecondition: Insufficient stock.
  STOCK books/8b1d4e6a-2c3f-4a5b-9e7d-1f0a2b3c4d5e: Requested 2, available 1.

The details travel in binary in the grpc-status-details-bin trailer. Trailers are handled like headers, so their total size is limited to 8 KB in most environments; don't stuff hundreds of field violations into one error.

For context beyond the details there's metadata. You can send headers with the request and trailers at the end of the response. In the GetStock above, the server read the x-correlation-id header and added the x-stock-source trailer. To reach trailers on the client you have to keep hold of the call itself:

using (AsyncUnaryCall<GetStockResponse> call = inventory.GetStockAsync(
    new GetStockRequest { BookId = KnownBook },
    headers: new Metadata { { "x-correlation-id", "chk-1001" } },
    deadline: DateTime.UtcNow.AddSeconds(2)))
{
    GetStockResponse stock = await call.ResponseAsync;
    Console.WriteLine($"available={stock.Available} status={stock.Status} price={stock.UnitPrice.Amount.ToDecimal()} {stock.UnitPrice.CurrencyCode}");
    Console.WriteLine($"trailer x-stock-source={call.GetTrailers().GetValue("x-stock-source")}");
}

available=12 status=InStock price=249.9 TRY
trailer x-stock-source=memory

The server log shows the header arrived as well: GetStock 3f2c9a1e-7b4d-4c8e-9a51-2d6f0e8b7c13, correlation chk-1001. Because HTTP/2 headers are lowercase, Metadata keys are converted to lowercase; if you carry a binary value, the key must end in -bin.

Deadlines: The End of Waiting Forever

gRPC calls have no default deadline. Without one, the client waits until the server answers; if the server hangs, so does the client, and so does whoever called the client. A deadline is not a duration but a point in time (UTC); the library sends it as the remaining time in the grpc-timeout header and both sides track it independently.

To try it, I started Inventory so that every store call waits 3 seconds (Inventory__SimulatedLatencyMs=3000) and called it with a 500 ms deadline:

case "deadline":
    var watch = Stopwatch.StartNew();
    try
    {
        await inventory.GetStockAsync(new GetStockRequest { BookId = KnownBook }, deadline: DateTime.UtcNow.AddMilliseconds(500));
    }
    finally
    {
        Console.WriteLine($"elapsed: {watch.ElapsedMilliseconds} ms");
    }
    break;

elapsed: 534 ms
DeadlineExceeded:
elapsed: 3064 ms

The first two lines are the call with a deadline: the client gave up after half a second. The third line is the same call without a deadline; it waited the full three seconds. On the server, context.CancellationToken was cancelled the moment the deadline passed, and the Task.Delay in the store was cut short:

info: Grpc.AspNetCore.Server.ServerCallHandler[28]
      Service method 'GetStock' canceled.
      System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at Bookstore.Inventory.Stock.InMemoryStockStore.FindAsync(Guid bookId, CancellationToken cancellationToken)

That cut-off didn't happen by itself. Even though the HTTP request is aborted, the service method keeps running until you finish it; what stops the work is having passed the token to the store, and from there down to the database query. The mechanism is the same as in the CancellationToken post of the ASP.NET Core Web API series: every await you don't pass the token to won't hear the cancellation.

Carrying the Deadline Down the Chain

Now the opening scenario. Checkout calls Ordering's PlaceOrder with a one-second deadline, and Ordering calls Inventory's ReserveStock:

public override async Task<PlaceOrderResponse> PlaceOrder(PlaceOrderRequest request, ServerCallContext context)
{
    string orderId = $"o-{Guid.NewGuid():N}"[..10];
    logger.LogInformation("PlaceOrder {OrderId}, deadline {Deadline:HH:mm:ss.fff}", orderId, context.Deadline);

    // EnableCallContextPropagation carries the deadline and the CancellationToken
    ReserveStockResponse reservation = await inventory.ReserveStockAsync(
        new ReserveStockRequest { OrderId = orderId, Lines = { request.Lines } });

    return new PlaceOrderResponse { OrderId = orderId, ReservationId = reservation.ReservationId };
}

We pass neither a deadline nor a token to the inner call. With EnableCallContextPropagation on, the logs:

info: Bookstore.Ordering.Services.OrderingGrpcService[0]
      PlaceOrder o-b5e94785, deadline 21:14:30.691

info: Bookstore.Inventory.Services.InventoryGrpcService[0]
      ReserveStock for o-b5e94785, deadline 21:14:30.716
warn: Bookstore.Inventory.Services.InventoryGrpcService[0]
      ReserveStock for o-b5e94785 cancelled by the caller

The deadline Ordering received was passed on to Inventory (the 25 ms in between is a small drift caused by sending the deadline as remaining time). When the second was up, Inventory dropped the work and no stock was reserved. Ordering's inner call ended with Cancelled, the client got DeadlineExceeded, total time 1028 ms.

I ran the same code with the EnableCallContextPropagation line removed:

info: Bookstore.Ordering.Services.OrderingGrpcService[0]
      PlaceOrder o-2b254506, deadline 21:14:49.499
info: System.Net.Http.HttpClient.InventoryServiceClient.ClientHandler[101]
      Received HTTP response headers after 3018.1468ms - 200

info: Bookstore.Inventory.Services.InventoryGrpcService[0]
      ReserveStock for o-2b254506, deadline 23:59:59.999

The client again got DeadlineExceeded after one second. But the deadline Inventory saw was DateTime.MaxValue; it finished the work in three seconds and reserved stock for an order the customer had long given up on. That's the answer to the opening question: without propagation, the lower part of the chain never learns that the upper part gave up; it both wastes resources and produces side effects.

Propagation only works for calls made from inside a gRPC call; it reads the context from the ServerCallContext on the HttpContext. If Ordering uses the same client from a minimal API endpoint or a background service, propagation throws an error; in that case either enable the SuppressContextNotFoundErrors option or pass the deadline and CancellationToken by hand. If you give the inner call a shorter deadline, the smaller of the two is used.

In Production

Put a deadline on every call and pick its value from measurements. Giving 30 seconds to a call whose p99 latency is 80 ms is nearly the same as having no deadline. Build chains of internal service calls with propagation and let the outermost deadline set the budget.

Monitor gRPC statuses as metrics. Since the HTTP status is always 200, dashboards that look at HTTP status won't show errors. Record the gRPC status code as a separate dimension in traces and metrics, and set alerts on the rates of Unavailable, DeadlineExceeded and Internal.

Read DeadlineExceeded as "unknown", not "failed". The server may have completed the work and the response simply arrived late. Design side-effecting calls such as reservations to be idempotent (a second reservation with the same order_id shouldn't reserve new stock), otherwise a client retry creates duplicates.

Don't carry downstream statuses upward as they are. Passing the FailedPrecondition Ordering got from Inventory straight on to checkout is sometimes right, but internal hostnames, internal ids or another service's error messages can leak out. Re-create the error at the boundary according to your own contract.

Common Pitfalls

A new channel per call. We saw a threefold difference above, locally and without TLS; on a real network it's worse. If you see a using var channel line inside a method, it's most likely in the wrong place.

gRPC calls with no deadline. A single slow downstream holds every thread and connection waiting on it; the failure spreads up the chain.

Not passing context.CancellationToken. When the deadline passes the client leaves, but the query on the server runs to the end. The log says "canceled" while the database is still working.

Http1AndHttp2 on an endpoint without TLS. That's exactly what the HTTP_1_1_REQUIRED error is telling you.

Returning every error as Internal or Unknown. The client's retry decision looks at the status code; return a transient problem as Internal or a permanent validation error as Unavailable and you get either needless retries or needless giving up.

A Small Experiment

★☆☆ Call GetStock for an existing book with curl using --http2-prior-knowledge and look at the trailers that come after the body. Compare with the "trailers-only" shape of the error response.

★★☆ Add a minimal API endpoint GET /orders/check/{bookId} to Ordering and call the same InventoryServiceClient from there. See the error propagation gives you, then fix it with SuppressContextNotFoundErrors and by passing HttpContext.RequestAborted.

★★★ Make ReserveStock idempotent: a second call with the same order_id should return the first reservation's id instead of opening a new reservation. Run the no-propagation scenario again and show that a client retry no longer creates a duplicate reservation.

So far every call has been one request, one response. In the next post we move to server, client and bidirectional streaming to watch stock changes live, send bulk updates from a CSV file and scan barcodes at the till, and we see how a slow consumer puts the brakes on the server.