Idempotent Outbound Calls: Sending an Idempotency-Key, Unknown Outcomes and Reconciliation

Version note: The code in this post was built and run with .NET 10, Microsoft.Extensions.Http.Resilience 10.11, Polly 8.9, WireMock.Net 2.31 and xUnit v3 4.0.

Guess First

Bookstore, the online bookstore used as the shared example across this site's series, sends POST /v1/charges to its payment provider. The provider receives the request and charges the card, but answers after 5 seconds. Our attempt timeout is 3 seconds. Three questions:

First: you got a TimeoutRejectedException at second 3. Was the customer's card charged?

Second: if you send the same request with the same Idempotency-Key at second 4, what does the provider return?

Third: in post 05 we tested that the payment POST is never retried. Once you turn retries on, what should replace that test?

The answer to the first is "we don't know", and that answer is the subject of this post. The answer to the second, as the real run below shows, is 409 Conflict: the first request is still being processed at the provider. The answer to the third is "count the retries, but verify they all carry the same key".

A Timeout Is Not an Answer

The most dangerous row in the failure mode table at the start of this series was "unknown outcome after a timeout". Once an HTTP request has left the wire, getting an error does not mean the other side didn't process it. The connection can drop, the response can get lost, or our patience can simply be shorter than the provider's processing time. In every case we hold the same exception, while the truth at the provider can be three different things.

That is why section 9.2.2 of RFC 9110 separates methods: safe methods such as GET, HEAD, OPTIONS and TRACE, plus PUT and DELETE, are idempotent; sending the same request twice has the same intended effect as sending it once. POST is not. According to the RFC, a client should not automatically retry a non-idempotent request unless it has a way to know the request is idempotent after all, or a way to tell that the first request was never applied. In the standard pipeline post we did exactly that with DisableForUnsafeHttpMethods: the payment POST was never retried.

That is safe but incomplete. Without retries every transient error reaches the customer as "payment failed", and on an unknown outcome we return 202 and wait without knowing what for. We need to use the door the RFC leaves open: make the POST idempotent together with the provider.

Idempotency-Key: A Contract With the Provider

Most payment providers solve this with a request header: the client generates a unique key for every payment intent, the provider stores the key with the result of the first request, and repeats with the same key get that first result without the operation running again. The IETF HTTPAPI working group has a draft for this header: version 08 of draft-ietf-httpapi-idempotency-key-header, dated 22 October 2026. It targets the Standards Track but is not an RFC yet, so providers apply the rules in their own documentation. Still, the draft summarizes well how both sides should behave:

Two more details matter. In the draft the header value is a Structured Field String, so it is sent in quotes: Idempotency-Key: "8e03978e-...". Many providers accept the unquoted value too, but following the draft costs nothing. Second, providers don't keep keys forever; the draft asks them to publish that period. After the retention period a request with the same key may be processed as a new payment. That number drives the reconciliation settings below.

How to write the server side, meeting an Idempotency-Key that arrives at our own API with an endpoint filter and a table, is covered in the Idempotency Key post of the Solution Patterns series; its security role in sensitive flows is in the sensitive business flows post of the API Security series. Here we sit on the other side of the table: we are the ones who send the key, store it and interpret the provider's responses.

One Key per Payment Intent

The first decision is what the key represents. If you generate a key per HTTP request, retries carry different keys and idempotency does nothing. A fixed key per order is wrong too: a customer whose card is declined must be able to pay with another card. The right unit is the "payment intent": same order, same amount, same card. In Bookstore we modeled it as PaymentAttempt:

// One payment intent: the same order, amount and card. Every HTTP request for it carries the same key
public sealed class PaymentAttempt
{
    public required Guid Id { get; init; }
    public required int OrderId { get; init; }
    public required string IdempotencyKey { get; init; }
    public required decimal Amount { get; init; }
    public required string Currency { get; init; }
    public required string CardTokenHash { get; init; }
    public required DateTimeOffset CreatedAt { get; init; }
    public PaymentAttemptStatus Status { get; set; } = PaymentAttemptStatus.Open;
    public string? ProviderPaymentId { get; set; }
    public DateTimeOffset? LastCheckedAt { get; set; }

    // What the provider sees as merchant_reference; reconciliation searches by it
    public string Reference => $"bookstore-{OrderId}-{Id:N}";
}

PaymentAttemptStore.GetOrStart returns the open attempt when there is one with the same amount and card; otherwise it creates a new key with Guid.CreateVersion7. We don't store the card token, only compare its SHA-256 hash. If the customer tries another card while an attempt is open, no new key is created:

var open = _attempts.SingleOrDefault(a => a.OrderId == orderId && a.Status == PaymentAttemptStatus.Open);
if (open is not null && open.Amount == amount && open.CardTokenHash == tokenHash)
{
    reused = true;
    return open;
}

if (open is not null)
{
    // A different card or amount while the old attempt may still be charged: refuse
    throw new PaymentAttemptConflictException(orderId);
}

This rule prevents double charges. If you take a payment with the second card while the outcome of the first is unknown, and the first one turns out successful, the customer has paid twice. Instead we return 409 and tell the customer their previous payment is being verified.

Storing the key before the first request is part of the rule. Bookstore's orders live in memory in this series, and so do the attempts. In the real application PaymentAttempts is a table, and the attempt row is written in the same transaction that moves the order to PaymentPending. If the process crashes while the request is in flight, it reads the key from there after the restart. This is the same idea as an outbox: store first, then talk to the outside world. The mechanics are in the Transactional Outbox post of the Working with Message Queues series.

The Adapter: Sending the Header and Reading the Responses

ProviderPaymentGateway now builds its own HttpRequestMessage instead of calling PostAsJsonAsync, because it has to add the header:

var body = new ProviderChargeRequest(
    Amount: ToMinorUnits(request.Amount),
    Currency: request.Currency,
    Source: request.CardToken,
    Description: $"Bookstore order {request.OrderId}",
    MerchantReference: request.Reference);

using var message = new HttpRequestMessage(HttpMethod.Post, "v1/charges")
{
    Content = JsonContent.Create(body, options: Json)
};
// Same key on every attempt for this payment; the draft defines the value as a quoted string
message.Headers.Add(IdempotencyKeyHeader, $"\"{request.IdempotencyKey}\"");

On the response side we translated two new status codes into our own error model:

return response.StatusCode switch
{
    HttpStatusCode.Created or HttpStatusCode.OK => await ReadApprovedAsync(response, cancellationToken),
    // The first request with this key is still running at the provider
    HttpStatusCode.Conflict => throw new PaymentGatewayException(
        PaymentFailureKind.UnknownOutcome, "Payment with this key is still being processed."),
    // Same key, different body: a bug on our side, never retry
    HttpStatusCode.UnprocessableEntity => throw new PaymentGatewayException(
        PaymentFailureKind.ContractViolation, "Idempotency-Key was reused with a different request."),
    HttpStatusCode.PaymentRequired => await ReadDeclinedAsync(response, cancellationToken),
    // ... 429, 503 and 5xx as in the earlier posts
};

We treat 409 as UnknownOutcome because the provider is saying "an operation with this key started but hasn't finished". It may end in success or in a decline; we know no more than after a timeout.

The fake payment provider grew accordingly. FakePaymentProvider in Bookstore.FakeProviders now stores keys, the request's SHA-256 fingerprint and the result; it answers a completed key with the stored response and an Idempotent-Replayed: true header, a key still being processed with 409, and a key that comes with a different body with 422. GET /v1/charges?merchant_reference=... lists the charges for a reference. The new Drop mode imitates a request lost on the way: nothing is processed and the answer comes after the timeout.

Retrying POST Only When It Has a Key

Now the payment POST can be retried, but only when it carries a key. We replaced DisableForUnsafeHttpMethods with our own ShouldHandle:

// Retry POST only when it carries an Idempotency-Key; a 409 means "still running, ask again later"
public static bool ShouldRetry(Outcome<HttpResponseMessage> outcome, HttpRequestMessage? request)
{
    if (request is null || (request.Method == HttpMethod.Post && !request.Headers.Contains(IdempotencyKeyHeader)))
    {
        return false;
    }

    return HttpClientResiliencePredicates.IsTransient(outcome)
        || outcome.Result?.StatusCode == HttpStatusCode.Conflict;
}

payments.AddStandardResilienceHandler()
    .Configure(builder.Configuration.GetSection("Resilience:PaymentProvider"))
    .Configure(options => options.Retry.ShouldHandle = args =>
        ValueTask.FromResult(ProviderPaymentGateway.ShouldRetry(args.Outcome, args.Context.GetRequestMessage())));

On a timeout Outcome.Result is empty, so we reach the request through the ResilienceContext rather than the outcome. GetRequestMessage() is an extension method in Microsoft.Extensions.Http.Resilience; the package's own DisableFor implementation also takes the request from Outcome.Result.RequestMessage first and from the context otherwise. In appsettings.json the attempt timeout stayed at 3 seconds, the total timeout went from 4 to 10 seconds and the retry delay became 1 second; with a 4-second total timeout there was no room left for a retry after a timeout.

I started the fake provider with a 5-second processing time and called checkout:

dotnet run --project src/Bookstore.FakeProviders -- --payments Slow --payments-delay 5
dotnet run --project src/Bookstore.Api

The payment lines in the API log (I removed the Operation Key: '' field for brevity):

Resilience event occurred. EventName: 'OnTimeout', Source: 'payment-provider-standard//Standard-AttemptTimeout'
Execution attempt. Source: 'payment-provider-standard//Standard-Retry', Result: 'The operation didn't complete within the allowed timeout of '00:00:03'.', Handled: 'True', Attempt: '0', Execution Time: 3011.1853ms
Received HTTP response headers after 11.0755ms - 409
Execution attempt. Source: 'payment-provider-standard//Standard-Retry', Result: '409', Handled: 'True', Attempt: '1', Execution Time: 11.2502ms
Received HTTP response headers after 1.6699ms - 201
Execution attempt. Source: 'payment-provider-standard//Standard-Retry', Result: '201', Handled: 'False', Attempt: '2', Execution Time: 1.7926ms
Resilience pipeline executed. Source: 'payment-provider-standard/', Result: '201', Execution Time: 5490.9235ms
Provider replayed the stored response for key 01a1d96c-e23d-77bf-801e-9df969c1d378
Payment approved by provider: ch_000001

Three requests went out, one charge was created. The first attempt timed out at second 3 while the provider kept processing. The retry about a second later got 409, because the first request had not finished at the provider yet. The third attempt at 5.49 seconds received the stored 201. The customer waited 5.5 seconds, but the card was charged once. Without a key the same flow would have meant either three separate charges or telling the customer "payment failed".

When the Pipeline Gives Up: 202 and a Retry With the Same Key

Retries are not always enough. I raised the provider's processing time to 15 seconds; this time the 10-second total timeout never sees the result. In that case CheckoutService doesn't throw; it leaves the attempt open and returns the order as PaymentPending:

catch (PaymentGatewayException ex) when (ex.Kind == PaymentFailureKind.UnknownOutcome)
{
    // Maybe charged, maybe not: keep the attempt open; a retry or reconciliation will find out
    logger.LogWarning("Order {OrderId}: payment outcome unknown, key {IdempotencyKey}", order.Id, attempt.IdempotencyKey);
    return new CheckoutResult(order.Id, order.Status, total, null) { ShippingIsFlatRate = quote.IsFlatRate };
}

The endpoint returns this as 202 Accepted with Location: /api/orders/1001; the client asks for the order's status instead of paying again. I sent the requests in Bookstore.Api.http in order: tok_visa first, right after it tok_mastercard for the same order, then the order status, and tok_visa again 8 seconds later:

$ checkout tok_visa
{"orderId":1001,"status":"PaymentPending","total":548.90,"providerPaymentId":null,"declineReason":null,"shippingIsFlatRate":false} -> 202
$ checkout tok_mastercard
{"type":"https://tools.ietf.org/html/rfc9110#section-15.5.10","title":"A previous payment for this order is still being verified.","status":409,"traceId":"00-c0fcdfbc30189867952bc7745a69a188-e213d374dcca3db5-02"} -> 409
$ GET order
{"id":1001,"status":"PaymentPending","providerPaymentId":null,"attempts":[{"idempotencyKey":"01a1d96d-4807-7712-a456-9925d2b2a082","status":"Open"}]}
$ checkout tok_visa (again)
{"orderId":1001,"status":"Paid","total":548.90,"providerPaymentId":"ch_000001","declineReason":null,"shippingIsFlatRate":false} -> 200

The first checkout's pipeline gave up after one timeout and three 409s at 8.6 seconds and returned 202. The second attempt with a different card got 409. When the customer tried again with the same card, GetOrStart found the open attempt, the request went out with the same key, and the provider returned the charge it had completed in the meantime with Idempotent-Replayed. The log shows it in this order:

Order 1001: payment outcome unknown, key 01a1d96d-4807-7712-a456-9925d2b2a082
Order 1001: resending payment with the stored key 01a1d96d-4807-7712-a456-9925d2b2a082
Provider replayed the stored response for key 01a1d96d-4807-7712-a456-9925d2b2a082
Payment approved by provider: ch_000001

So the customer's "try again" button is now a safe retry. A double click on the payment page, or reloading the page and submitting again, no longer creates a second charge.

Reconciliation: When the Customer Doesn't Come Back

Customers don't always retry; they can close the browser and leave. We need a job that asks the provider about attempts left open. PaymentReconciliationJob is a BackgroundService; a PeriodicTimer driven by TimeProvider wakes it every minute, and it checks every open attempt older than 2 minutes:

foreach (var attempt in attempts.OpenOlderThan(now - options.Value.MinAge))
{
    var order = orders.Find(attempt.OrderId)!;
    attempt.LastCheckedAt = now;
    var charge = await payments.FindChargeAsync(attempt.Reference, cancellationToken);
    if (charge is not null)
    {
        attempt.Status = PaymentAttemptStatus.Succeeded;
        attempt.ProviderPaymentId = charge.ProviderPaymentId;
        order.Status = OrderStatus.Paid;
        order.ProviderPaymentId = charge.ProviderPaymentId;
        logger.LogInformation("Reconciled order {OrderId}: provider has charge {ProviderPaymentId}", order.Id, charge.ProviderPaymentId);
    }
    else if (now - attempt.CreatedAt >= options.Value.AbandonAfter)
    {
        attempt.Status = PaymentAttemptStatus.Abandoned;
        order.Status = OrderStatus.Placed;
        logger.LogWarning("Reconciled order {OrderId}: no charge after {Age}, attempt abandoned", order.Id, now - attempt.CreatedAt);
    }
    else
    {
        logger.LogInformation("Order {OrderId}: no charge at the provider yet, will check again", order.Id);
    }
}

FindChargeAsync is a GET, and the standard pipeline retries it like any GET. If the provider offers no search by key (most don't), sending your own reference with every request matters for exactly this reason: because we can search by merchant_reference, the job knows which charge belongs to which of our attempts.

To try two scenarios I shortened the job's timings from the command line (Interval=5 s, MinAge=10 s, AbandonAfter=40 s). With the provider processing for 15 seconds:

Order 1001: payment outcome unknown, key 01a1d96d-cc25-737a-b939-dfd362d275f1
Order 1001: no charge at the provider yet, will check again
Reconciled order 1001: provider has charge ch_000001

With the provider in Drop mode, where the request is never processed:

Order 1001: payment outcome unknown, key 01a1d96e-5f4b-74c6-8543-5b75ee37b64f
Order 1001: no charge at the provider yet, will check again
...
Reconciled order 1001: no charge after 00:00:44.8674120, attempt abandoned

After the attempt is abandoned the order goes back to Placed and the customer can pay again with a new key. AbandonAfter has two bounds: longer than any time in which a delayed request could still be processed at the provider, and shorter than the provider's key retention. We chose 30 minutes as the default; if the provider keeps keys for 24 hours there is plenty of margin.

The job itself runs in memory on a single instance. With several API instances two of them may query the same attempt; the query is a GET, so that is harmless, but the status update needs to be conditional in the database. For durable and distributed jobs see the Hangfire posts in the Background Jobs series; that series' retry and idempotency post also covers the same key idea from the job side, with an e-mail provider as the example.

There is also the accounting side. Providers publish a daily or weekly settlement report: which charges were captured, which were refunded, what the fees were. Every line in that report should have a Succeeded attempt on our side, and every Succeeded attempt should have a line in the report. Our reconciliation job puts the order in the right state within minutes; the accounting reconciliation runs daily and puts the remaining exceptions (a charge we don't have, a payment missing from the report) in front of a person.

Tests: "Retry With the Same Key" Instead of "Never Retry"

The Payment_post_is_never_retried test from post 05 turned red the moment we enabled retries, which is what we wrote it for. We replaced it with the new definition of the behavior:

[Fact]
public async Task Payment_retries_reuse_one_idempotency_key()
{
    FakePaymentProvider.Configure(_payments, FakeMode.Error);
    await using var api = new BookstoreApiFactory(_payments, _carrier, settings: FastPaymentRetries);

    using var response = await CheckoutAsync(api.CreateClient());

    Assert.Equal(HttpStatusCode.ServiceUnavailable, response.StatusCode);
    Assert.Equal(4, ChargeRequests); // 1 attempt + 3 retries
    var key = Assert.Single(ChargeKeys.Distinct());
}

ChargeKeys reads the Idempotency-Key header of every POST from WireMock's LogEntries; FastPaymentRetries lowers the retry delay to 10 ms in the test. We also rewrote the 20% failure test with 100 separate orders, because a paid order is no longer charged again. We ask the fake's list endpoint how many charges the provider created:

passed Bookstore.Api.Tests.CheckoutResilienceTests.Payment_retries_reuse_one_idempotency_key (732ms)
    charge requests=4, distinct keys=1, key="01a1d970-dc54-712e-b971-20f3eab27bc5"
passed Bookstore.Api.Tests.CheckoutResilienceTests.Checkout_with_twenty_percent_payment_failures (659ms)
    responses: 200 x 70, 503 x 30; charge requests: 103; distinct keys: 71; charges at provider: 70
passed Bookstore.Api.Tests.CheckoutResilienceTests.Unknown_outcome_is_settled_by_reconciliation (1s 934ms)
    checkout=202, after reconciliation: Paid ch_000001, charge requests=5

In the first line the header value WireMock recorded shows its quotes; that is the string form from the draft. The third test builds the unknown outcome end to end: the provider takes 1 second, we give up after 0.5 seconds, checkout returns 202, and the order becomes Paid when the job's RunOnceAsync is called by hand. The second line surprised me. With retries I expected almost every order to be paid, yet 30 orders got 503 and the provider saw only 71 distinct keys. The log had the reason: OnCircuitOpened. The standard pipeline's circuit breaker opens at a 10% failure ratio with at least 100 requests. In post 05 there were no retries, so 100 checkouts produced exactly 100 requests and the threshold was reached only on the last one; once retries raised the request count, 20% failures opened the breaker and the remaining orders got 503 without ever reaching the network. That is the right behavior: if a fifth of the provider's requests fail, we don't want to keep loading it. What the test really verifies did not change: 70 successful responses and 70 charges at the provider, so no order was paid twice.

In Production

Find the key retention period in the provider's documentation and set AbandonAfter from it. After the retention period, a request with the same key can be processed as a new payment even if the first one went through.

Headers like Idempotent-Replayed differ between providers, and some don't send one at all. Don't build logic on this header; use it for logs and metrics only. The real decision belongs to the status code and the body.

Track the number and age of open attempts as a metric. If the reconciliation job stops working, customers start calling to say "I paid but my order is still pending"; before that, an alert on "open attempts older than 30 minutes > 0" should fire.

Think about breaker thresholds together with retries. Retries raise the number of requests the breaker sees; on a low-volume client such as payments, measure MinimumThroughput and FailureRatio against your real traffic.

Common Pitfalls

Generating the key again for every HTTP request. A DelegatingHandler that puts Guid.NewGuid() on each request looks like idempotency but protects no retry.

Not storing the key before sending. If the process crashes while the request is in flight, the key is lost and the same payment goes out again with a new key.

Treating 409 as a failure or a success. A 409 means "I don't know yet"; if the payment is treated as failed and the customer pays again, you get a double charge.

Showing the unknown outcome to the customer as an error. A customer who sees "payment failed" pays again with another card. A 202 and a "your payment is being verified" message are more accurate.

A Small Experiment

★☆☆ Start the fake with --payments Slow --payments-delay 5 and set Resilience:PaymentProvider:Retry:Delay first to 00:00:00.200, then to 00:00:03. How many 409s do you see each time, and how long does checkout take in total?

★★☆ Without changing ShouldRetry, add a DelegatingHandler to the payment client that puts a new Guid on every request. Which assert in Payment_retries_reuse_one_idempotency_key fails?

★★★ Write a test for PaymentReconciliationJob that uses FakeTimeProvider: in Drop mode checkout should return 202, and when you advance the clock by AbandonAfter, the order should go back to Placed and the attempt to Abandoned. Hint: the job's PeriodicTimer and the pipeline's timeouts use the same TimeProvider.

In this post we resolved the unknown outcome in two ways: asking again with the same key, and querying the provider periodically. There is a third way: the provider telling us when the payment is done. To get there we first move to the other side of webhooks. In the next post Bookstore itself sends webhooks: we deliver the order.shipped event to its institutional customers, signed, from an outbox and on a retry schedule.