Sending Webhooks: Standard Webhooks, Signing, Outbox-Based Delivery and Retry Schedules

Version note: The code in this post was built and run with .NET 10, Microsoft.Extensions.Http.Resilience 10.11, Polly 8.9, WireMock.Net 2.31 and xUnit v3 4.0; the signatures were also verified with standardwebhooks 1.1, the official Standard Webhooks JavaScript library.

Guess First

Bookstore, the online bookstore used as the shared example across this site's series, wants to tell its institutional customers, schools and libraries, when their orders are shipped. One school's webhook endpoint waits 20 seconds before answering. Three questions:

First: how long does POST /api/orders/1003/ship, the request that ships the order, take?

Second: the first delivery got a 500 and is retried 5 seconds later. Does the webhook-id header stay the same, and do webhook-timestamp and the signature stay the same too?

Third: the customer's endpoint returns 301 Moved Permanently. Should you follow the redirect?

The answers come from the real runs later in this post: the ship request takes 3.5 milliseconds, because the webhook is not sent inside the request. webhook-id is the same on every attempt, while the timestamp and the signature are new each time. The redirect is not followed; a 3xx counts as a failed delivery.

In the previous post Bookstore was the customer of an external service and dealt with unknown outcomes. In this post we switch roles: Bookstore is the side that calls its institutional customers' systems. Sending webhooks means being an external service for someone else. Everything we expected from the other side in the first posts of this series (a clear contract, sensible timeouts, consistency across retries) is now ours to provide.

Designing a Webhook as a Product

A webhook is an API your customers write code against. Treat events the way you treat endpoints you version and document. The event catalog Bookstore publishes for institutional customers has one row for now:

For the payload we use the structure Standard Webhooks recommends. Standard Webhooks is a community-driven open source specification (version 1.0.0, Apache 2.0 licensed) that collects existing industry practice into one contract, together with libraries for many languages; it is not an RFC. The payload it recommends has three fields: a full-stop delimited type, a timestamp in ISO 8601 format saying when the event happened, and data carrying the actual content.

How much of the record to send is a separate decision. The specification describes two approaches: a "full" payload carries the whole state of the record, a "thin" payload only the identifiers. With thin payloads the customer calls our API for details; the data goes through authorization and access can be logged, so we know which institution read which order and when. In Bookstore we chose the middle: the identifiers plus the one field customers almost always want, the tracking number. The address, the amount and the list of books are not in the payload. The specification recommends keeping payloads small (usually under 20 KB); for large data, sending a link is better.

The simplest rule for versioning is in the table: adding fields is fine; removing a field or changing its meaning means a new event type (such as order.shipped.v2), and the old type keeps being sent in parallel for a while. Customers choose which events they want when they subscribe; filtering on the sending side spares them traffic they don't care about.

The Signature: webhook-id, webhook-timestamp and webhook-signature

A webhook endpoint is a URL open to the internet, and anyone can send requests to it. So that the customer can say "this request really came from Bookstore and wasn't changed on the way", we sign every request. Standard Webhooks defines three headers: webhook-id is the unique identifier of the event, webhook-timestamp is the Unix time (in seconds) of the attempt, and webhook-signature is the signature itself. The signed content is the three joined with full stops: id.timestamp.body. That way the signature protects not just the body but the id and the time as well:

// Standard Webhooks, symmetric scheme: HMAC-SHA256 over "id.timestamp.body", sent as "v1,<base64>"
public static class WebhookSignature
{
    public const string SecretPrefix = "whsec_";

    public static string NewSecret() => SecretPrefix + Convert.ToBase64String(RandomNumberGenerator.GetBytes(32));

    // One "v1,..." per secret, space separated: during rotation both the old and the new secret sign
    public static string Sign(string messageId, long timestamp, ReadOnlySpan<byte> body, IEnumerable<string> secrets)
    {
        var signatures = new List<string>();
        foreach (var secret in secrets)
        {
            signatures.Add("v1," + Convert.ToBase64String(ComputeHmac(secret, messageId, timestamp, body)));
        }

        return string.Join(' ', signatures);
    }

    public static byte[] ComputeHmac(string secret, string messageId, long timestamp, ReadOnlySpan<byte> body)
    {
        var key = Convert.FromBase64String(secret.StartsWith(SecretPrefix, StringComparison.Ordinal) ? secret[SecretPrefix.Length..] : secret);
        var prefix = Encoding.UTF8.GetBytes($"{messageId}.{timestamp}.");
        var signed = new byte[prefix.Length + body.Length];
        prefix.CopyTo(signed, 0);
        body.CopyTo(signed.AsSpan(prefix.Length));
        return HMACSHA256.HashData(key, signed);
    }
}

Each subscription gets its own secret: 32 random bytes, base64 encoded, with a whsec_ prefix. The specification asks for 24 to 64 bytes and recommends never sharing a secret between endpoints; if one customer's secret leaks, only that customer's webhooks can be forged. v1 is the label of the symmetric signature made with HMAC-SHA256. The specification also defines an asymmetric scheme with ed25519 (v1a), where the customer only knows the public key and no secret has to be shared. Because we can share secrets with institutional customers over a secure channel, we started with the symmetric scheme.

The best way to be sure your own signing code is right is to verify it with someone else's code. I captured the request that reached the fake customer endpoint:

webhook-id: msg_01a1d979b39d7343bd2e231853640e0b
webhook-timestamp: 1794649995
webhook-signature: v1,0Q0CW0BC2MMOVnkJAiCN3qBBe8cyp+t9IYB/z6EJIWc=
{"type":"order.shipped","timestamp":"2026-11-14T09:53:14.141046+00:00","data":{"orderId":1003,"shipmentId":"SHP-1003","trackingNumber":"LC-7781240"}}

Then I passed these headers and the body to the official Standard Webhooks JavaScript library (the standardwebhooks npm package). The library verified the signature, and rejected it once I changed a single digit of the tracking number in the body:

first delivery, old secret: verified
tampered body: No matching signature found

The timestamp in the payload is when the event happened; webhook-timestamp in the header is when this attempt was made. They are not the same thing. In the next post we will see the receiving side base its replay protection on the header.

Don't Send in the Request: Write to an Outbox

The easiest implementation is to POST to the customer's URL as soon as the order ships. That makes the customer's endpoint part of our own request: if it's slow, the ship request is slow; if it's down, the ship request fails or the event is lost. Instead we write the webhook to an outbox:

order.Status = OrderStatus.Shipped;
var shipmentId = $"SHP-{orderId}";
var queued = 0;
if (order.InstitutionId is { } institutionId)
{
    // Only an outbox write here: the request does not wait for the customer's endpoint
    queued = webhooks.Publish(institutionId, WebhookEventTypes.OrderShipped,
        new OrderShippedData(order.Id, shipmentId, trackingNumber));
}

WebhookPublisher.Publish creates one WebhookMessage for every subscription to the event. The payload is serialized once, here, and stored as a byte array. This detail matters: if you rebuild the payload on retries, field order, number formatting or data changed in the meantime can make you send different bytes, and the same webhook-id goes out with two different contents.

// Standard Webhooks payload: type, timestamp (when it happened) and data
var payload = JsonSerializer.SerializeToUtf8Bytes(new { type = eventType, timestamp = now, data }, Json);
outbox.Add(new WebhookMessage
{
    Id = $"msg_{Guid.CreateVersion7(now):N}",
    SubscriptionId = subscription.Id,
    EventType = eventType,
    Payload = payload,
    CreatedAt = now,
    NextAttemptAt = now
});

Like the attempts in the previous post, the outbox lives in memory in this sample. In the real Bookstore WebhookMessages is a table, written in the same transaction that sets the order to Shipped; if the transaction doesn't commit there is no webhook, and if it commits the webhook will certainly be sent. How that guarantee is built is covered in the Transactional Outbox post of the Working with Message Queues series. There are also ready-made products for webhook delivery (open source or managed webhook services); when choosing one, look for the same list as in this post: signing, a retry schedule, disabling endpoints, a delivery log.

I put the customer endpoint into the mode that waits 20 seconds (--receiver Slow) and shipped the order:

{"orderId":1003,"status":"Shipped","shipmentId":"SHP-1003","webhooksQueued":1}
200 0.003514s

The ship request returned in 3.5 milliseconds. The webhook itself was sent in the background a few seconds later and hit the 15-second timeout; we'll see that in the delivery log below.

Delivery: Short Timeouts, No Redirects

A BackgroundService called WebhookDispatcher drains the outbox. Every second it picks the messages that are due and sends up to 8 of them in parallel, so one slow customer doesn't hold the others back. The next round doesn't start before the previous one finishes, so the same message is never sent twice at the same time. The webhook client is configured differently from our other clients:

// Someone else's server: no redirects, short connect timeout, 15 s per attempt, no retries here
// (the outbox schedule is the retry). SSRF filtering of the target address: API Security series, post 08
builder.Services.AddHttpClient(WebhookDispatcher.ClientName)
    .ConfigurePrimaryHttpMessageHandler(() => new SocketsHttpHandler
    {
        AllowAutoRedirect = false,
        ConnectTimeout = TimeSpan.FromSeconds(5),
        PooledConnectionLifetime = TimeSpan.FromMinutes(2)
    })
    .AddResilienceHandler("webhooks", pipeline => pipeline.AddTimeout(TimeSpan.FromSeconds(15)));

Every setting has a reason. Standard Webhooks treats 3xx responses as failed deliveries and recommends that customers update their URL instead of being redirected; redirects are also the classic way around SSRF protection. The specification recommends a timeout between 15 and 30 seconds; we took the lower bound. There is no retry in the pipeline, because retrying on a scale of hours rather than seconds is the outbox's job. The SSRF filter that checks the resolved IP address at connection time was built with ConnectCallback in the API Security series, in the post on APIs that talk to the outside world; the same handler plugs in here.

Two details in the delivery itself. Headers are created again on every attempt; webhook-id is fixed, the timestamp and signature are new. And we never buffer the response body: the request goes out with HttpCompletionOption.ResponseHeadersRead, and on failure only the first 1 KB of the body is read for the delivery log. Someone else's server can send you a 1 GB error page.

request.Headers.Add("webhook-id", message.Id);
request.Headers.Add("webhook-timestamp", timestamp.ToString());
request.Headers.Add("webhook-signature", WebhookSignature.Sign(message.Id, timestamp, message.Payload, subscription.ActiveSecrets(now)));

var started = Stopwatch.GetTimestamp();
int? status = null;
string? error = null;
TimeSpan? retryAfter = null;
try
{
    var client = httpClientFactory.CreateClient(ClientName);
    // Headers only: we never buffer a large response body from someone else's server
    using var response = await client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead, cancellationToken);
    status = (int)response.StatusCode;
    retryAfter = response.Headers.RetryAfter?.Delta;
    if (!response.IsSuccessStatusCode)
    {
        error = await ReadStartAsync(response, cancellationToken);
    }
}
catch (Exception ex) when (ex is HttpRequestException or Polly.Timeout.TimeoutRejectedException)
{
    error = ex.Message;
}

The Retry Schedule and Disabling an Endpoint

Every non-2xx result, every timeout and every connection error is a failed attempt. A schedule decides when the next one happens. We took the example schedule from Standard Webhooks: immediately, 5 seconds, 5 minutes, 30 minutes, then 2, 5, 10, 14, 20 and 24 hours; 10 attempts over about 3 days. Each delay gets up to 10% jitter, so a customer endpoint coming back from an outage isn't hit by every message in the same second. If the response has a Retry-After longer than the scheduled delay, we follow it.

var schedule = options.Value.RetrySchedule;
var failures = ++message.ConsecutiveFailures;
if (failures > schedule.Length)
{
    message.Status = WebhookMessageStatus.Failed;
    logger.LogWarning("Webhook {MessageId} failed after {Attempts} attempts", message.Id, failures);
    return;
}

// Up to 10% jitter so that endpoints recovering from an outage are not hit all at once
var delay = schedule[failures - 1] * (1 + Random.Shared.NextDouble() * 0.1);
if (retryAfter > delay)
{
    delay = retryAfter.Value;
}

message.NextAttemptAt = now + delay;

If an endpoint keeps failing for a long time, continuing to send to it is a burden on both sides. We have two rules: 410 Gone means "I don't want webhooks any more" according to the specification and closes the subscription immediately; an endpoint that fails for 5 days without a single successful delivery is closed too. The specification recommends telling the customer through another channel as well; in Bookstore an e-mail goes to the institution's technical contact, while the sample code writes a log line instead.

I switched the fake endpoint to Error (500) and then to Gone (410) and resent the same message. The API log:

Webhook msg_01a1d979b39d7343bd2e231853640e0b attempt 1 failed (500), next attempt in 00:00:05.3427864
Webhook msg_01a1d979b39d7343bd2e231853640e0b attempt 2 failed (500), next attempt in 00:05:24.2581344
Webhook subscription 01a1d979-b251-709e-825f-c6655eb2a99a disabled: Endpoint answered 410 Gone.

The attempt against the slow endpoint landed in the delivery log like this:

{"at":"2026-11-14T09:54:04.370883+00:00","statusCode":null,"error":"The operation didn't complete within the allowed timeout of '00:00:15'.","durationMs":15021}

Waiting for the schedule in real time is not an option, so a test uses FakeTimeProvider to move the clock to each attempt's NextAttemptAt. The fake endpoint in WireMock returns 500 twice, then 200:

passed Bookstore.Api.Tests.WebhookDeliveryTests.Failed_deliveries_follow_the_schedule_with_the_same_webhook_id (199ms)
    status=Delivered, requests=3, webhook-ids=1, timestamps=1794646800,1794646805,1794647125, delays=00:00:05.3359866, 00:05:20.5658503

Three requests, one webhook-id, three different timestamps; delays of 5 seconds and 5 minutes plus jitter. The second answer from the guessing section is in this line.

Secret Rotation and the "Resend" Button

Secrets change: someone leaves the team, a secret ends up in a log file by mistake, or the security policy requires yearly rotation. So that webhooks don't fail verification until the customer installs the new secret, Standard Webhooks defines a path: the webhook-signature header is a space-separated list, and during the transition every message is signed with both the old and the new secret. The receiver accepts the message if any signature in the list matches.

POST /admin/webhooks/subscriptions/{id}/rotate-secret creates a new secret and gives the old one an expiry 24 hours ahead. The header of a message sent after the rotation:

webhook-signature: v1,/OFCxPttqUZBslFfwkTkU4UODCzD8Ea1x8iWB/Eb0Ns= v1,XIb2YNWlU5pZaXydQ+C5+XHWMVB5iSd9AKNy2IDieXw=

The official library verified this message with both secrets, and the message from before the rotation only with the old one:

first delivery, old secret: verified
first delivery, new secret: No matching signature found
after rotation, old secret: verified
after rotation, new secret: verified

The screen the support team uses most is the delivery log. GET /admin/webhooks/messages returns every message's attempts with status codes, the start of the error body and the duration. When an institution says "we never got the shipping notification", this is the first place to look: if we see a 200, the problem is on the receiving side; if we see a 500 or a timeout, we can give the customer the exact time to look up in their own logs. POST /admin/webhooks/messages/{id}/resend is the endpoint behind the "resend" button: it puts the message back in the queue with the same webhook-id. Keeping the id matters, because a receiver that already processed the message recognizes it by the id and doesn't process it twice. These admin endpoints have no authorization in the sample; in Bookstore they sit behind the store staff policy.

In Production

Accept only HTTPS URLs in production. A signature proves the content wasn't changed, but it doesn't hide it; an order notification sent over HTTP can be read on the way. The registration endpoint in the sample allows HTTP only in Development and only for localhost.

Some institutional customers sit behind firewalls and only accept requests from specific IPs. Sending webhook traffic through a NAT or proxy with fixed outbound IPs and publishing those IPs in the documentation is a common request.

The outbox table grows. Plan a job that deletes delivered messages after a while (as long as support needs them, 30 days for example), and track delivery as metrics: success rate per subscription, age of the oldest pending message and number of disabled subscriptions.

Common Pitfalls

Sending the webhook synchronously inside the request. When the customer's endpoint slows down, so does yours; when it is down, the event is lost.

Acting as if ordering were guaranteed. With retries and parallel delivery, order.shipped can arrive before an event that happened earlier. Give the receiver the timestamp field in the payload, leave ordering to them, and state clearly in the documentation that you don't guarantee order.

Serializing the payload again on every attempt. The same webhook-id goes out with different bytes; the receiver's idempotency check sees the first content, your support team the second.

Following redirects and reading response bodies without a limit. Both let a server you don't control use your resources.

Giving every customer the same secret. With one customer's leaked secret, fake events can be sent to all the others.

A Small Experiment

★☆☆ Add an order.delivered event to WebhookPublisher and to the catalog. Which fields did you decide to send thin and which full, and why?

★★☆ Add a mode to the fake endpoint that returns 503 with Retry-After: 600. In the delivery log, is the next attempt 5 seconds or 10 minutes away?

★★★ Add a limit of at most 2 concurrent deliveries per subscription to WebhookDispatcher (Polly's concurrency limiter or a dictionary of SemaphoreSlims). Show with a test that while 100 messages pile up for one customer, the other customers' deliveries are not delayed.

In this post we were the side sending webhooks. In the next post we cross to the other side of the table: the payment provider sends payment.succeeded to Bookstore. Verifying the signature over the raw body, a timestamp check against replays, an inbox for a fast 2xx, and using webhooks to close the "unknown outcomes" left open two posts ago are all in there.