The Payments Service: A Fake Payment Provider, Idempotent Charges and Webhook Callbacks

Version note: The code in this post was built and run with .NET 10, Aspire 13.6, WolverineFx 6.64, Microsoft.Extensions.Http.Resilience 10.11 and Entity Framework Core 10.

Guess First

Payments sends a charge request to the payment provider and gets no answer within 2 seconds. In fact the provider received the request and charged the card; only its answer was late. Should Payments send PaymentFailed to the checkout saga in that case? Second question: if the provider sends the same "payment succeeded" webhook twice, how many PaymentSucceeded events are published for the order? The answers come from real runs.

A Fake Payment Provider: Why and Where?

In 07 Payments was a single if: it declined anything above 5,000 TRY. The hard parts of a real payment integration are not in an if but on the network: timeouts, retries, late results, forged requests. To see them locally we added a small provider simulator to Bookstore, the online bookstore used as the shared example across this site's series. It imitates no real brand; it offers the behaviour most card payment APIs share: charges that require an Idempotency-Key header, results that arrive asynchronously as webhooks, lookups by merchant reference, and refunds.

The project lives in src/Simulators/Bookstore.FakePaymentProvider, not in src/Services. It is not one of our services; it stands in for the outside world: it uses no ServiceDefaults, shares no contracts and talks to Payments only over HTTP. The architecture test from 01 does not count it as a service either. The AppHost links the two in two directions: Payments finds the provider through service discovery, and the provider sends its webhooks to Payments' address. They share a secret for the webhook signature:

// Shared secret for the provider's webhook signatures (Standard Webhooks: "whsec_" + base64).
// 32 letters and digits are valid base64 (24 bytes); Aspire generates it once and keeps it in user secrets.
IResourceBuilder<ParameterResource> webhookSecret = builder.AddResource(ParameterResourceBuilderExtensions.CreateGeneratedParameter(
    builder, "payment-webhook-secret", secret: true,
    new GenerateParameterDefault { MinLength = 32, Lower = true, Upper = true, Numeric = true, Special = false }));

// Stands in for the external card payment provider. In production this is the provider's URL, not a resource we run.
IResourceBuilder<ProjectResource> paymentProvider = builder.AddProject<Projects.Bookstore_FakePaymentProvider>("payment-provider")
    .WithEnvironment("Webhooks__Secret", ReferenceExpression.Create($"whsec_{webhookSecret}"));

Payments now has tables of its own as well: Payments (one payment per order, its status and the provider's charge id) and an inbox table for webhooks. As in the other services, a payments-migrator resource runs the migrations.

An Anti-Corruption Layer: The Provider's Model Stays at the Border

The provider's API speaks snake_case JSON and reports status with its own strings ("pending", "succeeded", "failed"). We don't want those words to leak into Payments' domain and from there into the saga; if the provider changes tomorrow, only one folder should change. PaymentProviderClient is the one place that translates, and it hands out its own type:

// Anti-corruption layer: the provider's JSON, status strings and error codes stop here.
// The rest of Payments sees only ChargeOutcome.
public abstract record ChargeOutcome
{
    // The provider answered. Pending means "accepted, the result comes as a webhook".
    public sealed record Answered(string ChargeId, ChargeState State, string? FailureReason) : ChargeOutcome;

    // Timeout, 5xx or a broken connection: the card may or may not have been charged.
    public sealed record Unknown(string Reason) : ChargeOutcome;
}

The provider's DTOs (ProviderCharge, ProviderWebhook) are internal and not used outside the Provider folder. The concept itself comes from DDD and came up in the Microservices series too. What matters here is that Unknown is a separate outcome: "failed" and "I don't know" are not the same thing.

Charges with an Idempotency-Key, and No Retries for POST

An Idempotency-Key tells the provider: if a request with this key came before, don't open a new charge, return the first one's result. We don't generate the key randomly; we derive it from the order (charge-{OrderId}). That way every attempt for the same order carries the same key, whether it comes from the first handler or from the reconciliation job days later. We called the simulator directly twice:

HTTP/1.1 202 Accepted
{"id":"ch_c6179d1e0aa6430a906586132e99e6c8","reference":"0f8e2c6a-1111-4c3b-9d1e-000000000001","amount":249.90,"currency":"TRY","status":"pending","failure_code":null}
HTTP/1.1 200 OK
Idempotent-Replayed: true
{"id":"ch_c6179d1e0aa6430a906586132e99e6c8","reference":"0f8e2c6a-1111-4c3b-9d1e-000000000001","amount":249.90,"currency":"TRY","status":"pending","failure_code":null}

The second request returned the same charge; the card was not charged twice. So who should do the repeating? ServiceDefaults adds the standard resilience handler to every HttpClient, and its retry repeats a POST after a timeout too. For a charge we don't want to leave that decision to the HTTP layer: repeating belongs to the message handler and the reconciliation job, both of which record the state in the database. On Payments' client we removed the default handler and put in one that does not repeat POSTs:

// ServiceDefaults gives every client the standard handler, whose retry repeats a POST after a timeout.
// For a charge, repeating is the message handler's decision (with the same Idempotency-Key), not the HTTP layer's.
// RemoveAllResilienceHandlers is still marked experimental in Microsoft.Extensions.Http.Resilience 10.11.
#pragma warning disable EXTEXP0001
builder.Services.AddHttpClient<PaymentProviderClient>(client => client.BaseAddress = new Uri("https+http://payment-provider"))
    .RemoveAllResilienceHandlers()
    .AddStandardResilienceHandler(options =>
    {
        options.Retry.DisableForUnsafeHttpMethods();
        options.AttemptTimeout.Timeout = TimeSpan.FromSeconds(2);
        options.TotalRequestTimeout.Timeout = TimeSpan.FromSeconds(5);
    });
#pragma warning restore EXTEXP0001

On the first order the AuthorizePayment handler never ran, and the message landed in payments.dead-letter:

[payments]       Wolverine.Configuration.InvalidServiceLocationException: Found service locations while generating code for Message Handler for
Bookstore.Contracts.Payments.AuthorizePayment, but ServiceLocationPolicy.NotAllowed is in effect (it is the default).
[payments]       Service Bookstore.Payments.Provider.PaymentProviderClient: The service registration for Bookstore.Payments.Provider.PaymentProviderClient is an
'opaque' lambda factory with the Transient lifetime and requires service location

Wolverine 6 builds a handler's dependencies itself when it generates code, and when it cannot see behind a factory lambda it chooses to stop by default. A typed HttpClient is registered exactly like that. The fix is to allow service location for that type: opts.CodeGeneration.AlwaysUseServiceLocationFor<PaymentProviderClient>(). We moved the message from the DLQ back to payments.commands with the management API, and the answer to the ★★★ exercise from 06 showed up in the log: Discarding a duplicate delivery of envelope ... which the inbox already holds. Wolverine's inbox also keeps the id of a message that went to the dead letter queue; the message was handled only after we sent it with a new message id.

An Unknown Result and the Reconciliation Job

The handler itself is short: create the payment if there is none, call the provider, record the result. What matters is the Unknown branch:

ChargeOutcome outcome = await provider.ChargeAsync(payment.IdempotencyKey, command.OrderId, command.Amount, command.Currency, ct);
switch (outcome)
{
    case ChargeOutcome.Answered { State: ChargeState.Pending } answered:
        payment.MarkSubmitted(answered.ChargeId, clock);
        return null;
    case ChargeOutcome.Answered answered:
        return payment.Complete(answered.State == ChargeState.Succeeded, answered.ChargeId, answered.FailureReason, clock);
    case ChargeOutcome.Unknown unknown:
        // Not a failure: the card may have been charged. Reconciliation finds out; the saga keeps waiting.
        payment.MarkUnknown(unknown.Reason, clock);
        return null;
    default:
        throw new InvalidOperationException($"Unexpected outcome {outcome}");
}

In Payments we use Wolverine's EF Core transactions in Lightweight mode: no database transaction is held open while the handler waits for the provider, and the SaveChanges at the end writes the payment and the outgoing messages together.

We built the situation from Guess First in the simulator: we delayed its answer by 3 seconds and dropped its webhooks. The provider accepted the charge and settled it successfully, but Payments could learn that neither from the answer nor from a webhook:

[payment-provider]       Charge ch_1a068030b95249d48ce9f59f19cde670 for 01a1da89-dc16-79d5-8b68-834d27b5ce70: 219.90 TRY accepted (key charge-01a1da89-dc16-79d5-8b68-834d27b5ce70)
[payments]       Charge charge-01a1da89-dc16-79d5-8b68-834d27b5ce70 has an unknown result: TimeoutRejectedException
[payment-provider]       Webhook evt_7c184bbe3ec2464281460b5ab11985ee for ch_1a068030b95249d48ce9f59f19cde670 dropped (simulation)
[payments]       Reconciling order 01a1da89-dc16-79d5-8b68-834d27b5ce70 (Unknown): provider says Answered { ChargeId = ch_1a068030b95249d48ce9f59f19cde670, State = Succeeded, FailureReason =  }

14:50:33 order=StockReserved payment=Unknown
14:50:37 order=StockReserved payment=Unknown
...
14:50:47 order=StockReserved payment=Unknown
14:50:50 order=Confirmed payment=Succeeded

The first answer to Guess First: sending PaymentFailed would have been wrong. The saga would have cancelled the order and released the stock, while money had been taken from the customer's card. Unknown tells the saga nothing; the saga keeps waiting until its 15-minute timeout. The reconciliation job periodically picks up payments that are Unknown or have been Submitted for a long time and asks the provider by order reference. If the provider knows the charge, the job applies its result; if not (the request never reached the provider), it sends the charge again with the same Idempotency-Key. Locally the interval is 10 seconds; the default in appsettings.json is one minute. The general form of the mechanism is in the Idempotent Outbound Calls post of the External Service Integration and Resilience series.

Webhooks: Signatures, Replay Protection and an Inbox

The webhook endpoint is a door open to the internet: anyone can send it a JSON body that says "payment succeeded". The simulator uses the format of the Standard Webhooks specification: the webhook-id, webhook-timestamp and webhook-signature headers, and an HMAC signature over id.timestamp.body, using HMAC-SHA256. The core of the verification:

// An old timestamp is a replay of a captured request, even when its signature is correct.
if (!long.TryParse(timestamp, NumberStyles.None, CultureInfo.InvariantCulture, out long seconds) ||
    (clock.GetUtcNow() - DateTimeOffset.FromUnixTimeSeconds(seconds)).Duration() > Tolerance)
{
    error = "timestamp outside tolerance";
    return false;
}

string secret = configuration["PaymentProvider:WebhookSecret"]
    ?? throw new InvalidOperationException("PaymentProvider:WebhookSecret is not configured.");
byte[] key = Convert.FromBase64String(secret["whsec_".Length..]);
byte[] expected = HMACSHA256.HashData(key, Encoding.UTF8.GetBytes($"{id}.{timestamp}.{body}"));

// The header may carry several signatures ("v1,abc v1,def") while the secret is being rotated.
foreach (string candidate in signatures.Split(' ', StringSplitOptions.RemoveEmptyEntries))
{
    if (candidate.StartsWith("v1,", StringComparison.Ordinal) &&
        TryFromBase64(candidate[3..], out byte[] actual) &&
        CryptographicOperations.FixedTimeEquals(actual, expected))
    {
        error = "";
        return true;
    }
}

The comparison uses FixedTimeEquals, because a plain equality check stops at the first different byte and the timing difference can leak the signature. The signature covers the raw body, so the endpoint reads the body as a string before turning it into JSON. After verification the endpoint writes the webhook id to the ProcessedWebhooks table and, through the outbox, the payment's new status and the event for the saga in the same transaction. We sent four requests, with signatures computed from the secret the AppHost generated:

tampered signature
HTTP 401
valid signature, timestamp 10 minutes old
HTTP 401
valid signature, first delivery
HTTP 200
valid signature, same webhook-id again
HTTP 200

[payments]       Rejected webhook evt_manual_1: signature mismatch
[payments]       Rejected webhook evt_manual_2: timestamp outside tolerance
[payments]       Webhook evt_manual_3: payment for order 01a1da88-0849-7911-9bb1-f0cbbd3ea21c is Succeeded
[payments]       Webhook evt_manual_3 already processed

The second answer to Guess First: a single PaymentSucceeded. When the same webhook-id came a second time the inbox table recognized it and returned 200; providers resend until they see a 2xx, so 200 is the right answer. Even a webhook with a different id for the same payment (the third request was this order's second "succeeded" notice) publishes nothing new, because Payment.Complete returns null once the payment has its final result. The details of receiving webhooks are in the Receiving Webhooks post of the External Service Integration and Resilience series.

End to End: A Late Payment and a Refund

In the last experiment we combined the timeout scenario from 07 with the real provider flow: we lowered Ordering's payment timeout to 5 seconds and left the simulator slow and without webhooks again:

14:53:11 order=StockReserved payment=Unknown
14:53:14 order=Cancelled payment=Unknown
14:53:17 order=Cancelled payment=Unknown
...
14:53:27 order=Cancelled payment=Refunded

[payments]       Charge charge-01a1da8c-4371-7776-ad13-7cceaac4b095 has an unknown result: TimeoutRejectedException
[payments]       Reconciling order 01a1da8c-4371-7776-ad13-7cceaac4b095 (Unknown): provider says Answered { ChargeId = ch_c19a4102926b4b56b141bc8aea4594d0, ...
[payments]       Refunded 219.90 TRY for order 01a1da8c-4371-7776-ad13-7cceaac4b095: order-cancelled
[payment-provider]       Refund re_7be787c21f194399aa28389a554b0e9a for charge ch_c19a4102926b4b56b141bc8aea4594d0 (key refund-01a1da8c-4371-7776-ad13-7cceaac4b095)

The saga cancelled the order after 5 seconds. Reconciliation found that the payment had in fact succeeded and published PaymentSucceeded. The saga was gone; the NotFound method from 07 saw that the order was cancelled and sent RefundPayment, and Payments passed the refund to the provider with its own Idempotency-Key. The first time we tried this chain the refund got a 500: the simulator did not read the refund request's charge_id field as snake_case. The message went to the DLQ without a single retry, because Payments had no rule for HttpRequestException. We fixed the simulator and added a rule to Payments: when the provider is down or returns 5xx, the message is retried after 1, 5 and 30 seconds. That is safe, because every call carries an Idempotency-Key derived from the order.

In Production

We hold no card data. Payments stores only the amount, the order reference and the provider's charge id. The card number goes from the browser straight to the provider; that is the most effective way to keep the PCI scope small.

Alert on Unknown payments. An Unknown or Submitted payment older than an hour that reconciliation could not settle means something between you and the provider is broken, and a person needs to look at it.

Plan secret rotation from the start. The verifier accepts several signatures in the header; the provider signs with the old and the new secret for a while, then the old one is removed. In production the webhook endpoint should be open over HTTPS only.

A refund moves money. Who refunded what, when and why belongs in an audit log; the Reason field in the RefundPayment message is a start.

Test against the real provider's sandbox too. The simulator is good for teaching the behaviour and building failure scenarios, but only the real API shows its real error codes and webhook types.

Common Pitfalls

Treating a timeout as a failure. Calling an unknown result a failure and cancelling the order means not shipping to a customer whose money was taken.

Generating a new Idempotency-Key on every attempt. A random key opens a new charge on retry. The key should be derived from the business operation (the order) and stay the same across attempts.

Stacking HTTP retries and message retries. If both layers make their own attempts, the number of requests multiplies, and the log does not even show which layer repeated.

Parsing the webhook body into JSON before checking the signature. Re-serialized JSON has different bytes and the signature never matches; worse, unsigned fields have already been read.

Carrying the provider's model into the domain. Comparing the string "succeeded" in four places in Payments means changing four places when the provider changes.

A Small Experiment

★☆☆ Stop the payment-provider resource and send an order. What happens to the payment's status in Payments? After starting the provider, find the no charge, submitted again line in the reconciliation job's log.

★★☆ Add a temporary failure to Payments so that the simulator's webhook gets a 500 on its first two attempts (for example, reject the first two requests based on a setting). What do the provider's retries and the ProcessedWebhooks table show together?

★★★ Add partial refunds: give RefundPayment an amount, stop Payments from refunding more than the charge in total, and decide how each refund gets its own Idempotency-Key. What should happen if the same refund message arrives twice?

Checkout now withstands the failures of a real payment flow. But all the services still sit on separate ports, at separate addresses and without authentication. In the next post we build Angular's single entry point: the Bookstore.Bff project with YARP, the /api/catalog and /api/orders routes, an aggregation endpoint for the book page that combines Catalog and Inventory and still works when one of them does not answer, and per-route rate limits and timeouts.