The Inventory Service and Sync Communication with gRPC: Stock Reservation, Deadlines and Resilience

Version note: The code in this post was built and run with .NET 10, Aspire 13.6, Grpc.AspNetCore and Grpc.Net.ClientFactory 2.85, Google.Protobuf 3.36, Entity Framework Core 10 and Buf CLI 1.75.

Guess First

Inventory has 25 copies of Middlemarch. Twenty customers order one copy each at the same time. For every reservation Inventory reads the stock row, increases the reserved quantity and writes it back; the row's SQL Server rowversion column is the concurrency token. How many of the twenty requests succeed, and how many run into "the row changed after you read it"? Second question: Ordering calls Inventory with a 2-second deadline; Inventory is slow that day and answers every call in 3 seconds. When Ordering gives up after 2 seconds, does Inventory carry on and make the reservation anyway? The answers come from real runs.

inventory.proto: From the gRPC Series' Contract to Bookstore

In the first post of the gRPC and GraphQL series we designed an inventory.proto for Bookstore, the online bookstore used as the shared example across this site's series. The Inventory service uses the same contract; we removed the fields Bookstore's microservices don't need and made their numbers reserved. We covered field numbers and backward compatibility in that post:

syntax = "proto3";

package bookstore.inventory.v1;

option csharp_namespace = "Bookstore.Inventory.V1";

import "google/protobuf/timestamp.proto";

// The contract from the gRPC and GraphQL series, without the fields Bookstore's
// microservices don't use. Their numbers stay reserved.
service InventoryService {
  rpc GetStock (GetStockRequest) returns (GetStockResponse);
  rpc ReserveStock (ReserveStockRequest) returns (ReserveStockResponse);
  rpc ReleaseStock (ReleaseStockRequest) returns (ReleaseStockResponse);
}

message GetStockResponse {
  reserved 5, 7, 8;
  reserved "unit_price", "reorder_level", "warehouse_quantities";
  string book_id = 1;
  int32 available = 2;
  int32 reserved_quantity = 3;
  StockStatus status = 4;
  google.protobuf.Timestamp updated_at = 6;
}

message ReservationLine {
  string book_id = 1;
  int32 quantity = 2;
}

// order_id makes the call idempotent: the same order always gets the same reservation.
message ReserveStockRequest {
  string order_id = 1;
  repeated ReservationLine lines = 2;
}

message ReserveStockResponse {
  string reservation_id = 1;
  google.protobuf.Timestamp expires_at = 2;
}

book_id is the book's ISBN and order_id is the Guid that Ordering generates. The StockStatus and ReleaseStock messages are the same as in the gRPC series, so we don't show them here.

Sharing the contract and catching breaking changes

The proto file lives in Inventory's folder, in its own project: src/Services/Inventory/Bookstore.Inventory.Contracts. Inventory takes the server side from this project and Ordering the client side. The project behaves like a contract package: it has its own version and depends only on the packages protobuf needs:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <!-- The contract package Inventory publishes. Breaking changes need a new proto package (v2), not a new version number. -->
    <Version>1.0.0</Version>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Google.Protobuf" />
    <PackageReference Include="Grpc.Core.Api" />
    <PackageReference Include="Grpc.Tools">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
  </ItemGroup>

  <ItemGroup>
    <Protobuf Include="Protos/bookstore/inventory/v1/inventory.proto" ProtoRoot="Protos" GrpcServices="Both" />
  </ItemGroup>

</Project>

The architecture test we wrote in post 01 caught this project right away. We had required that a project ending in .Contracts depend on no packages at all:

failed Bookstore.ArchitectureTests.SharedCodeRules.Contract_projects_have_no_logic_dependencies (7ms)
  Assert.All() Failure: 3 out of 3 items in the collection did not pass.
  [0]: Item:  "Google.Protobuf"
       Error: Assert.Contains() Failure: Item not found in collection
              Collection: []
              Not found:  "Google.Protobuf"
  [1]: Item:  "Grpc.Core.Api"
       Error: Assert.Contains() Failure: Item not found in collection
              Collection: []
              Not found:  "Grpc.Core.Api"
  [2]: Item:  "Grpc.Tools"
       Error: Assert.Contains() Failure: Item not found in collection

The rule wasn't wrong, it was incomplete. A gRPC contract needs the protobuf runtime, gRPC's base types and the code generator; none of them is logic or infrastructure. We added those three to the allow list on purpose, and that's where the test's value lies: changing the rule is a code change and shows up in the pull request.

// A contracts project is a description of messages, not a place for logic or infrastructure.
// gRPC contracts need the protobuf runtime, the gRPC base types and the code generator; nothing else.
private static readonly string[] AllowedContractPackages = ["Google.Protobuf", "Grpc.Core.Api", "Grpc.Tools"];

In the mono-repo, Ordering depends on this project with a project reference; when Inventory changes the proto, Ordering is rebuilt in the same commit. That convenience carries a risk too: a breaking change may not cause a compile error and only show up in the running system. A changed field number or type compiles in C#, but misreads the message of a service still running the old version. In CI, the Buf CLI's breaking command compares against the proto on main. For a change in which we made three mistakes on purpose (renaming book_id to isbn, making quantity an int64, giving the reserved number 5 to a new field) the output is this:

buf breaking --against ../main

bookstore/inventory/v1/inventory.proto:29:1:Previously present reserved name "unit_price" on message "GetStockResponse" was deleted.
bookstore/inventory/v1/inventory.proto:29:1:Previously present reserved range "[5]" on message "GetStockResponse" is missing values: [5] were removed.
bookstore/inventory/v1/inventory.proto:41:3:Field "1" with name "isbn" on message "ReservationLine" changed option "json_name" from "bookId" to "isbn".
bookstore/inventory/v1/inventory.proto:41:10:Field "1" on message "ReservationLine" changed name from "book_id" to "isbn".
bookstore/inventory/v1/inventory.proto:42:3:Field "2" with name "quantity" on message "ReservationLine" changed type from "int32" to "int64".
exit=100

The buf.yaml in the proto folder only turns on the breaking: use: [FILE] rule; FILE is the strictest category and counts renames as breaking too. The non-zero exit code stops the pipeline. If a breaking change is really needed, the way described in the gRPC series is to publish a bookstore.inventory.v2 package next to v1 and serve both for a while.

The Inventory Service: Stock, Reservations and Idempotency

In the monolith, Inventory had a StockItem table that knew a book by its BookId, and a ReserveAsync method that ran inside a single transaction. Three things changed in the service: a book is identified by its ISBN, a reservation is a separate record (StockReservation), and every reservation has an expiry time. The reason: in the monolith the reservation and the order were in the same transaction, and if one failed, both were rolled back. Now the reservation lives in Inventory's database and the order in Ordering's. There's no transaction between them; that's exactly the topic of the "No Transactions in Microservices" post of the Microservices series.

The heart of the reservation is StockReservations.ReserveAsync:

public async Task<StockReservation> ReserveAsync(Guid orderId, IReadOnlyList<StockRequest> lines, CancellationToken ct)
{
    Dictionary<string, int> wanted = lines.GroupBy(l => l.Isbn).ToDictionary(g => g.Key, g => g.Sum(l => l.Quantity));

    for (int attempt = 1; ; attempt++)
    {
        // Idempotent: a retried call for the same order gets the reservation it already has
        StockReservation? existing = await db.Reservations.AsNoTracking().FirstOrDefaultAsync(r => r.OrderId == orderId, ct);
        if (existing is not null)
        {
            return existing;
        }

        try
        {
            return await TryReserveAsync(orderId, wanted, ct);
        }
        catch (DbUpdateConcurrencyException) when (attempt < MaxAttempts)
        {
            // Another reservation changed one of these rows after we read it: read again and retry
            logger.LogWarning("Concurrency conflict for order {OrderId} on attempt {Attempt}, retrying", orderId, attempt);
            db.ChangeTracker.Clear();
            // A short random pause so that the competing requests do not collide again in lockstep
            await Task.Delay(TimeSpan.FromMilliseconds(Random.Shared.Next(10, 50) * attempt), ct);
        }
        catch (DbUpdateConcurrencyException)
        {
            throw new RpcException(new Status(StatusCode.Aborted, "Stock is changing too fast; retry the reservation."));
        }
        catch (DbUpdateException ex) when (ex.InnerException is SqlException { Number: 2601 or 2627 })
        {
            // The same order was reserved by a parallel call; the next loop returns that reservation
            db.ChangeTracker.Clear();
        }
    }
}

Idempotency is the first requirement of this design. Ordering's gRPC call can be repeated after a network failure: the standard resilience handler from ServiceDefaults resends the request after a connection error, or the user refreshes the page and confirms the same order again. The reservations table has a unique index on OrderId. A second call for the same order either finds the existing reservation at the start of the loop or, if it came in parallel, hits the unique index violation (SQL Server error 2601/2627) and finds it on the next round. The tools/inventory-probe.cs file in the repository is one of .NET 10's file-based apps: a small gRPC client for trying Inventory by hand. We called it twice with the same order id:

dotnet run tools/inventory-probe.cs -- http://localhost:60906 reserve c05805c1-7a9c-4df5-86d7-08ef3de0e32e 9780141439549 2
reservation=01a1da49-69db-754a-9f2c-644a4fc3fb4f expiresAt=2026-11-14T13:55:06.7479460Z
dotnet run tools/inventory-probe.cs -- http://localhost:60906 reserve c05805c1-7a9c-4df5-86d7-08ef3de0e32e 9780141439549 2
reservation=01a1da49-69db-754a-9f2c-644a4fc3fb4f expiresAt=2026-11-14T13:55:06.7479460Z
dotnet run tools/inventory-probe.cs -- http://localhost:60906 stock 9780141439549
9780141439549: available=3 reserved=22 status=Low
dotnet run tools/inventory-probe.cs -- http://localhost:60906 release c05805c1-7a9c-4df5-86d7-08ef3de0e32e
releasedLines=1
dotnet run tools/inventory-probe.cs -- http://localhost:60906 release c05805c1-7a9c-4df5-86d7-08ef3de0e32e
releasedLines=0

Both calls got the same reservation id, and only 2 copies were taken from stock, once. Releasing is idempotent too: the second release changed no lines. In post 07 the saga's compensation step will call this RPC, and processing the same message twice won't give the stock back twice. A book with no stock comes back with gRPC's FailedPrecondition status, and the missing books are in the unavailable-books trailer:

dotnet run tools/inventory-probe.cs -- http://localhost:60906 reserve <new-guid> 9780060934347 1
FailedPrecondition: Not enough stock for: 9780060934347.

The Same Book at the Same Time: rowversion and Optimistic Concurrency

The stock row's RowVersion column is SQL Server's rowversion type: the database changes it itself every time the row is updated. EF Core makes it a concurrency token with IsRowVersion() and adds it to the WHERE of every UPDATE. If two reservations read the same row and both try to write, the second one's UPDATE updates 0 rows and a DbUpdateConcurrencyException is thrown. The details of the mechanism are in the Concurrency post of the Entity Framework Core series; here we look at its results.

First, the last copy. In the seed data we left a single copy of The Left Hand of Darkness, and five customers sent orders to Ordering at the same time:

Ada -> 201
Alan -> 409
Grace -> 409
Edsger -> 409
Linus -> 409

One order was created, four got "Out of stock". The last copy wasn't sold to two people. Then the scenario from Guess First: twenty parallel orders while there are 25 copies of Middlemarch. On the first try the result was surprising:

  19 201
   1 500

[inventory]       Concurrency conflict for order 01a1da48-24b5-79eb-a520-38b9c2ab4493 on attempt 2, retrying
[inventory]       Error status code 'Aborted' with detail 'Stock is changing too fast; retry the reservation.' raised.
[ordering]       Grpc.Core.RpcException: Status(StatusCode="Aborted", Detail="Stock is changing too fast; retry the reservation.")

There were fourteen conflicts, and one request collided on all three of its three attempts. Inventory behaved correctly and returned Aborted; Ordering, however, didn't recognize that status and gave the customer a 500. We made two fixes. In Inventory we added a short random pause between attempts and raised the number of attempts to five, so requests that collided together don't collide together again on the next round. In Ordering we put Aborted into the "didn't work for now, try again" group as well. The same experiment after the fix:

  20 201

  14 attempt 1
   2 attempt 2
   1 attempt 3

The first line is the distribution of HTTP status codes, the rest the distribution of conflicts in Inventory's log by attempt number. All twenty orders were created and stock went down from 25 to 5 (available=5 reserved=20). There were 17 conflicts in total: fourteen requests collided on the first attempt and continued after reading again, a few went on to a second and third attempt. That's the answer to Guess First: with optimistic concurrency, a conflict isn't an error but part of the normal flow; what matters is that a conflict always ends with the right result.

The monolith's solution was different: a single conditional UPDATE ... SET QuantityReserved = QuantityReserved + @n WHERE QuantityOnHand - QuantityReserved >= @n with ExecuteUpdate. That approach works atomically without reading the row and has no conflicts. Here we chose rowversion, because a reservation now changes more than one row (stock rows and the reservation record) in the same transaction, and reading and checking the rules on the C# side suits us. For a few very hot books (the book everyone buys during a campaign), a conditional UPDATE means fewer conflicts and fewer rounds; it's a place to decide by measuring.

The gRPC Call from Ordering: AddGrpcClient, Service Discovery and Cleartext HTTP/2

Ordering gets Inventory's client through IHttpClientFactory:

// gRPC client through IHttpClientFactory: service discovery and the standard resilience handler apply to it too.
// "_grpc." picks Inventory's endpoint named grpc. A gRPC channel accepts only http/https addresses,
// so the "https+http" fallback form cannot be used here.
builder.Services.AddGrpcClient<InventoryService.InventoryServiceClient>(options =>
    options.Address = new Uri("http://_grpc.inventory"));

We hit two errors before we got to these two lines, and both are worth telling. On the first try the address was https+http://inventory as in post 04; Ordering's log said System.InvalidOperationException: No address resolver configured for the scheme 'https+http'. A gRPC channel tries to resolve the address itself and doesn't know this form; the fix is plain http:// (https:// with TLS in production).

The second error was more instructive. Inventory had a single http endpoint and every order ended with DeadlineExceeded after 2 seconds. Ordering's log showed the reason:

[ordering]       System.Net.Http.HttpRequestException: The HTTP/2 server closed the connection. HTTP/2 error code 'HTTP_1_1_REQUIRED' (0xd). (HttpProtocolError)
[ordering] warn: Polly[0]
[ordering]       Resilience event occurred. EventName: 'OnRetry', Source: '-standard//Standard-Retry', Operation Key: '', Result: 'The HTTP/2 server closed the
connection. HTTP/2 error code 'HTTP_1_1_REQUIRED' (0xd). (HttpProtocolError)'
[ordering] warn: Grpc.Net.Client.Internal.GrpcCall[7]
[ordering]       gRPC call deadline exceeded.

gRPC requires HTTP/2. With TLS, client and server agree on HTTP/2 through ALPN. Without TLS there's no such negotiation: Kestrel's default Http1AndHttp2 setting effectively speaks HTTP/1.1 on an unencrypted port and turns away a client that comes with HTTP/2. There's something else in the log: the standard resilience handler took this error for a transient network error and retried, every attempt got the same error, and in the end the 2-second deadline ran out. A permanent configuration error looked like "a slow Inventory" because of the retries.

The fix is to give Inventory two endpoints: http, speaking HTTP/1.1 for health checks, and grpc, speaking only HTTP/2. The AppHost gives Kestrel the endpoints and their protocols:

// Two endpoints: "http" from the launch profile (health checks) and "grpc". gRPC without TLS needs
// a port that speaks only HTTP/2, so Kestrel gets both endpoints, with their protocols, from the AppHost.
IResourceBuilder<ProjectResource> inventory = builder.AddProject<Projects.Bookstore_Inventory_Grpc>("inventory")
    .WithHttpEndpoint(name: "grpc")
    .WithEndpoint("grpc", endpoint => endpoint.Transport = "http2")
    .WithReference(inventoryDb)
    .WaitForCompletion(inventoryMigrator)
    .WithHttpHealthCheck("/health");
EndpointReference inventoryHttp = inventory.GetEndpoint("http");
EndpointReference inventoryGrpc = inventory.GetEndpoint("grpc");
inventory
    .WithEnvironment("Kestrel__Endpoints__Http__Url",
        ReferenceExpression.Create($"http://localhost:{inventoryHttp.Property(EndpointProperty.TargetPort)}"))
    .WithEnvironment("Kestrel__Endpoints__Grpc__Url",
        ReferenceExpression.Create($"http://localhost:{inventoryGrpc.Property(EndpointProperty.TargetPort)}"))
    .WithEnvironment("Kestrel__Endpoints__Grpc__Protocols", "Http2");

TargetPort is the port the service really listens on behind Aspire's proxy. When Kestrel's endpoints come from configuration, it doesn't use the ASPNETCORE_URLS Aspire provides and writes that to the log: Overriding address(es) 'http://localhost:21781, http://localhost:21780'. Binding to endpoints defined via IConfiguration and/or UseKestrel(). That warning is the expected behavior here. Ordering finds the right port through http://_grpc.inventory, service discovery's notation for "the endpoint named grpc of the inventory service". We will make the same separation in Kubernetes; there, TLS will be handled either by a service mesh or by the ingress, which is the topic of post 14.

There's no start order between Ordering and Inventory: the ordering resource gets WithReference(inventory) but not WaitFor(inventory). One refusing to start while the other restarts is a link we don't want in production either.

Deadlines, and Checkout While Inventory Is Down

Ordering's order handler makes the reservation before saving the order and gives the call its own deadline:

// Sync call with its own deadline: Inventory gets at most this long, then the caller gives up
TimeSpan deadline = configuration.GetValue("Ordering:ReservationDeadline", TimeSpan.FromSeconds(2));
try
{
    ReserveStockResponse reserved = await inventory.ReserveStockAsync(
        new ReserveStockRequest
        {
            OrderId = order.Id.ToString(),
            Lines = { order.Items.Select(i => new ReservationLine { BookId = i.Isbn, Quantity = i.Quantity }) }
        },
        deadline: clock.GetUtcNow().UtcDateTime + deadline,
        cancellationToken: ct);
    order.MarkStockReserved(Guid.Parse(reserved.ReservationId), reserved.ExpiresAt.ToDateTime());
}
catch (RpcException ex) when (ex.StatusCode == StatusCode.FailedPrecondition)
{
    string unavailable = ex.Trailers.GetValue("unavailable-books") ?? "";
    return new PlaceOrderOutcome.OutOfStock(unavailable.Split(',', StringSplitOptions.RemoveEmptyEntries));
}
catch (RpcException ex) when (ex.StatusCode is StatusCode.Unavailable or StatusCode.DeadlineExceeded or StatusCode.Aborted)
{
    // Unknown result: the reservation may exist on Inventory's side. Its TTL releases it
    logger.LogWarning("Stock reservation for order {OrderId} failed with {Status}", order.Id, ex.StatusCode);
    return new PlaceOrderOutcome.InventoryUnavailable(ex.StatusCode);
}

db.Orders.Add(order);

If the reservation succeeds, the order is saved in the StockReserved state and the response includes the reservation's expiry time:

{"id":"01a1da47-d0e5-7251-a185-f2d17b1c70f5","status":"StockReserved","placedAtUtc":"2026-11-14T13:38:22.053555","customerName":"Grace Hopper","customerEmail":"grace@example.com","items":[{"isbn":"9780141439587","title":"Emma","unitPrice":249.90,"quantity":2,"lineTotal":499.80}],"total":499.80,"currency":"TRY","stockReservedUntilUtc":"2026-11-14T13:53:22.336546"}

For the second question in Guess First we added a development-only interceptor to Inventory: when the Inventory:SimulatedLatency setting is given, it holds every call for that long, and it waits with context.CancellationToken. We set the latency to 3 seconds:

{"type":"https://tools.ietf.org/html/rfc9110#section-15.6.4","title":"Checkout is temporarily unavailable","status":503,"detail":"Stock could not be reserved. No order was created; please try again in a moment.","grpcStatus":"DeadlineExceeded","traceId":"00-1e6107605be897b7b15bd8ad9a70447e-981eeca942b13a36-03"}
HTTP 503 in 2.292299s

[inventory]       Simulating 00:00:03 latency; caller deadline is 2026-11-14T13:41:16.9236250Z
[inventory]       Service method 'ReserveStock' canceled.
[inventory]       System.Threading.Tasks.TaskCanceledException: A task was canceled.
[ordering]       gRPC call deadline exceeded.
[ordering]       Execution attempt. Source: '-standard//Standard-Retry', Operation Key: '', Result: 'The operation was canceled.', Handled: 'False', Attempt:

In gRPC a deadline isn't only the time the client waits; it also travels to the server in the grpc-timeout header. The "caller deadline" line in Inventory's log shows that. When the time ran out, the server's context.CancellationToken fired and ReserveStock was cancelled; the reservation was never made. The answer to Guess First: Inventory doesn't carry on, because every await from the handler down to the database carries the same token. Had we forgotten to pass the token to the EF Core query, the reservation would have been made after Ordering gave up. The standard resilience handler's line matters too: Handled: 'False', so it didn't retry the cancellation caused by the deadline. Ordering answered with 503 in 2.3 seconds, and the order wasn't saved.

The result is similar when Inventory is completely down. We stopped it with aspire resource inventory stop and sent an order: HTTP 503 in 2.616873s, again DeadlineExceeded. Locally Aspire's proxy accepts the connection and leaves it unanswered, so the error shows up as a deadline; with a connection to a closed port, the client gets an HttpRequestException instead, and as we saw above, the standard retry repeats it.

Here is the question the product has to answer: what should checkout do while Inventory is down? There are two options. The one we chose: don't create the order, answer 503 with "try again in a moment". Simple and honest, but every Inventory failure means a lost sale. The other: accept the order as Pending and make the reservation later, through a message. The customer sees "your order was received, awaiting confirmation"; when Inventory recovers, the reservation is made, or the order is cancelled if there's no stock. In the Communication Styles post of the Microservices series we picked the second as Bookstore's target: checkout's only synchronous dependency should be the time-boxed payment. When we build events in post 06 and the checkout saga in post 07, the reservation turns from this post's synchronous call into a message-driven step of the saga. gRPC will stay for the live stock query on the book page.

The Reservation's Lifetime: TTL and Release

The comment in the catch block above matters: DeadlineExceeded doesn't mean "Inventory didn't make the reservation", it means "we didn't get the answer in time". Inventory may have made the reservation and sent the answer; or Ordering may crash while saving the order after getting the reservation. In both cases an orphaned reservation stays in Inventory. In the monolith a single transaction rolled this back; here the safety net is the reservation's lifetime, its TTL. Every reservation is valid for 15 minutes, and a background service releases expired reservations:

protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
    using PeriodicTimer timer = new(options.Value.SweepInterval, clock);
    while (await timer.WaitForNextTickAsync(stoppingToken))
    {
        try
        {
            await ExpireDueReservationsAsync(stoppingToken);
        }
        catch (Exception ex) when (ex is not OperationCanceledException)
        {
            logger.LogError(ex, "Expiring reservations failed; trying again on the next tick");
        }
    }
}

To see the behavior, we lowered the TTL to 20 seconds and the sweep interval to 5 seconds and made a reservation:

13:41:27
reservation=01a1da4a-b42f-7a0f-9360-f22afccc4575 expiresAt=2026-11-14T13:41:51.3111920Z
9780141439471: available=23 reserved=2 status=InStock
13:41:55
[inventory]       Reserved 1 line(s) for order b26fb95b-065b-4085-a414-1f31e4f4b687 until 2026-11-14T13:41:51.3111920Z
[inventory]       Reservation 01a1da4a-b42f-7a0f-9360-f22afccc4575 for order b26fb95b-065b-4085-a414-1f31e4f4b687 Expired
9780141439471: available=25 reserved=0 status=InStock

The reservation expired at 13:41:51, was released on the next sweep and stock went back to 25. Fifteen minutes is a business decision: it must be longer than the worst-case duration of the payment step and short enough not to lock stock unnecessarily on a campaign day. The saga in post 07 will make the reservation permanent when the order is confirmed and call ReleaseStock on cancellation; TTL will remain the last resort that kicks in when those steps fail.

In Production

Think about gRPC load balancing in Kubernetes separately. gRPC works over a single long-lived HTTP/2 connection; because Kubernetes' ClusterIP service balances at the connection level, all of Ordering's calls can end up at a single Inventory pod. The solutions (client-side load balancing with a headless service, or a proxy working at the HTTP level, L7) are in the gRPC in Production post of the gRPC and GraphQL series.

Measure the conflict rate. The "Concurrency conflict" log should become a metric. If the attempt limit is reached often for one book on a campaign day, it's time to switch that book to a conditional UPDATE or to queue its reservations.

Derive the deadline from the budget. Ordering's 2 seconds aren't arbitrary: it's an upper bound well above the checkout p95 < 800 ms target we set in the Introduction to Architecture series, and long enough to say "Inventory really isn't answering". When the BFF sets its own timeout in post 09, the deadlines of the calls below it must fit inside it.

Count orphaned reservations. The number of reservations released by TTL is the earliest indicator that something went wrong on the order side. It should be close to zero; if it grows, either Ordering is crashing or the deadlines are too short.

Common Pitfalls

Trying to run cleartext gRPC with Kestrel's default settings. The HTTP_1_1_REQUIRED error above, hidden behind retries and a deadline, can be hunted for hours as "slowness". Without TLS, the gRPC port must be Http2.

Putting a non-idempotent RPC into a pipeline with retries. ServiceDefaults adds retries to every client. If ReserveStock weren't idempotent by order id, a retry after a connection error would take the stock twice.

Not setting a deadline. A gRPC call without a deadline can wait forever by default; even if the timeout in the resilience pipeline sets a limit, the server doesn't know about it and keeps working. A deadline stops the work on both the client and the server.

Not carrying the CancellationToken down to the database on the server. A deadline reaching the server is useless if the server doesn't use it.

Reading DeadlineExceeded as "it didn't happen". The result is unknown; the remedy is a TTL, an idempotent retry or a status query. Assuming the reservation wasn't made and sending the same order again with a different id locks the stock twice.

A Small Experiment

★☆☆ With tools/inventory-probe.cs, reserve 2 copies of War and Peace (3 in stock), then ask for 2 more copies of the same book with a different order id. Which gRPC status comes back? Release the first reservation and repeat the second request.

★★☆ Rewrite StockReservations with a conditional ExecuteUpdateAsync as in the monolith, and compare the Middlemarch experiment with 50 parallel requests on both versions: what happens to the number of conflicts and the total time? What did you have to do to keep the reservation record and the stock update in the same transaction?

★★★ Derive Ordering's deadline from the remaining time of the incoming HTTP request: put a 3-second limit on POST /api/v1/orders with ASP.NET Core's request timeouts, and compute the gRPC deadline as half of the time left. Measure the behavior by giving Inventory 1, 2 and 4 seconds of latency.

Inventory now owns its data and doesn't sell the same last copy to two people; Ordering uses it through a gRPC call with a deadline. But this post's most important finding is a cost: while Inventory is down, no orders can be taken, and there's no transaction between the order and the reservation. In the next post we start closing that gap: integration events with Wolverine and RabbitMQ, a transactional outbox in Catalog and Ordering, keeping Ordering's book copy current with Catalog's events, and ignoring an old event with the version field when message order is broken.