Asynchronous Processing: Queues, Workers, Fan-Out and Backpressure

Version note: The code in this post was built and run with C# 14 and .NET 10; the example ran with an ASP.NET Core 10 minimal API (Application Programming Interface), System.Threading.Channels and BackgroundService.

Guess First

At Bookstore, corporate customers download their invoices as PDFs at the end of the month. The endpoint is simple: POST /orders/{id}/invoice renders the PDF and returns the file's address. Rendering is CPU (Central Processing Unit) bound, one invoice takes about 400 ms, and the server can render at most four at a time. In the first minute of month-end 100 requests arrive at once; the client-side HttpClient waits 5 seconds and gives up.

How many customers get their invoice? How many PDFs does the server render? And if we turn the endpoint into "drop the work on a queue and reply right away", are invoices rendered faster?

Taking Heavy Work Out of the Request Path

Every piece of work done inside an HTTP (Hypertext Transfer Protocol) request adds to the time the client waits and to the resources the server holds meanwhile. Some work is not part of the response: an invoice PDF, an order confirmation e-mail, updating the search index, preparing a report. What they share is that they are slow, often depend on an external system, and the user seeing the result a few seconds later is fine.

The standard way to take this work out of the request path is a queue plus workers: the API drops a small message describing the work onto a queue and replies right away, and separately running workers process the messages at their own pace. The Cloud Design Patterns catalog on Microsoft Learn describes the two sides of this as separate patterns. Queue-Based Load Leveling is about piling up sudden load in a queue so workers can process it at a steady rate; Asynchronous Request-Reply is about how the client later learns the result. In Bookstore's architecture decisions the order e-mail already went through an in-process Channel<T>; in this post we extend the same idea to invoices.

Experiment: The Synchronous Invoice

First, the current state. We fake PDF rendering with a SemaphoreSlim that allows four jobs at a time and a 400 ms wait:

// PDF rendering is CPU bound: at most 4 at a time, ~400 ms each
public sealed class InvoiceRenderer
{
    private readonly SemaphoreSlim _cores = new(4);
    private readonly ConcurrentDictionary<int, bool> _invoices = new();
    private int _rendered, _skipped;

    public async Task RenderAsync(int orderId)
    {
        if (_invoices.ContainsKey(orderId)) { Interlocked.Increment(ref _skipped); return; }   // invoice already exists
        await _cores.WaitAsync();
        try
        {
            await Task.Delay(400);
            _invoices[orderId] = true;
            Interlocked.Increment(ref _rendered);
        }
        finally { _cores.Release(); }
    }
}

// the old way: the PDF is rendered inside the request
app.MapPost("/orders/{orderId:int}/invoice/sync", async (int orderId, InvoiceRenderer renderer) =>
{
    await renderer.RenderAsync(orderId);
    return Results.Ok($"/invoices/{orderId}.pdf");
});

100 concurrent requests, a 5-second client timeout:

sync : 200 OK 48, timed out 52, p50 2,856 ms, slowest OK 4,866 ms
       server rendered 100 PDFs, 52 for clients that had already given up

The server can render 10 invoices a second (4 ÷ 0.4 s), so the last of 100 invoices is ready at second 10. Since clients give up at second 5, only 48 got an answer; even the median successful request took 2.9 seconds. The second line is worse: the server rendered all 100 PDFs, 52 of them for clients no longer waiting. What does a client do? It retries. The next minute the same 52 requests arrive again on top of the unfinished work; under load the synchronous design grows its own load. (Passing the request's CancellationToken into rendering reduces wasted work but does not change the waiting time.)

Queue, Workers and 202 Accepted

In the new design the endpoint writes the work into a bounded Channel<T> and returns 202 Accepted with a status address. Four workers consume the queue:

// bounded queue: at most 50 jobs can wait, the rest are rejected
builder.Services.AddSingleton(Channel.CreateBounded<InvoiceJob>(new BoundedChannelOptions(50) { FullMode = BoundedChannelFullMode.Wait }));
builder.Services.AddHostedService<InvoiceWorker>();

// the new way: drop the work on the queue, return 202 and a status address
app.MapPost("/orders/{orderId:int}/invoice", (int orderId, HttpRequest request, InvoiceJobs jobs, Channel<InvoiceJob> queue) =>
{
    string key = request.Headers["Idempotency-Key"].FirstOrDefault() ?? Guid.NewGuid().ToString();
    if (jobs.TryGetByKey(key, out InvoiceJob? existing))
        return Results.Accepted($"/invoice-jobs/{existing.Id}");   // the same request came again: no new job

    InvoiceJob job = jobs.Create(key, orderId);
    if (!queue.Writer.TryWrite(job))
    {
        jobs.Remove(job, key);
        request.HttpContext.Response.Headers.RetryAfter = "5";
        return Results.StatusCode(StatusCodes.Status503ServiceUnavailable);   // backpressure
    }
    request.HttpContext.Response.Headers.RetryAfter = "2";
    return Results.Accepted($"/invoice-jobs/{job.Id}");
});

app.MapGet("/invoice-jobs/{id:guid}", (Guid id, InvoiceJobs jobs) =>
    jobs.Find(id) switch
    {
        null => Results.NotFound(),
        { Status: JobStatus.Done } job => new SeeOther($"/invoices/{job.OrderId}.pdf"),   // done: redirect to the result
        { } job => Results.Ok(new { status = job.Status.ToString() }),
    });

public sealed class InvoiceWorker(Channel<InvoiceJob> queue, InvoiceRenderer renderer) : BackgroundService
{
    protected override Task ExecuteAsync(CancellationToken stoppingToken) =>
        Task.WhenAll(Enumerable.Range(0, 4).Select(_ => ConsumeAsync(stoppingToken)));

    private async Task ConsumeAsync(CancellationToken stoppingToken)
    {
        await foreach (InvoiceJob job in queue.Reader.ReadAllAsync(stoppingToken))
        {
            job.Status = JobStatus.Running;
            await renderer.RenderAsync(job.OrderId);
            job.Status = JobStatus.Done;
        }
    }
}

SeeOther is a small IResult that writes 303 See Other and the Location header. The same 100 requests:

async: 202 Accepted 54, 503 rejected 46, p50 8 ms, p99 9 ms
       queue drained in 5.7 s by 4 workers, 54 PDFs rendered

Every accepted request got its answer within 9 ms; the client does not wait, no connection is held, no timeouts. But the answer to the last opening question is no: invoices were not rendered faster. 54 PDFs took 5.7 seconds, still ~10 invoices a second; the workers use the same four "cores" as the synchronous endpoint. Asynchronous processing does not speed up the work, it decouples the work's timing from the request: the user isn't kept waiting, sudden load is spread over time, and capacity can be scaled separately.

The status endpoint behaves the way the Asynchronous Request-Reply pattern on Microsoft Learn suggests: 200 and the state while the work runs, then 303 See Other redirecting to the result when it is done. The real polling output of one job:

poll 1: 200 {"status":"Running"}
poll 2: 200 {"status":"Running"}
poll 3: 200 {"status":"Running"}
poll 4: 303 Location: /invoices/2001.pdf

The reason the pattern recommends 303 rather than 302 is that 303 tells the client to always request the redirected address with GET; with 302 some clients may replay the original POST. The Retry-After in the 202 response tells the client how often to ask. If you want to push the result instead of polling (SignalR, a webhook, an e-mail), that is the topic of the upcoming Real-time Applications series in the Backend category.

Backpressure: A Bounded Queue

The second number in the output is 46 times 503. The queue is limited to 50 jobs; once the four jobs taken by the four workers and the 50 jobs in the queue filled it, new requests were rejected at TryWrite and the client got 503 Service Unavailable with Retry-After: 5. This is called backpressure: sending a "slow down" signal to the producer when the consumer can't keep up.

An unbounded queue does not send that signal, but it does not remove the load either. It only pushes the problem forward: the queue grows in memory, the completion time of an accepted job climbs to minutes or hours, and when a job given "202 Accepted" will finish becomes unknown. If the process restarts, all the work in memory is lost. In a bounded queue the capacity is a design decision: 50 jobs with four workers means at most ~5 seconds of waiting. Telling the client "not now, in 5 seconds" is more honest than saying "accepted" and finishing 10 minutes later.

BoundedChannelOptions.FullMode decides what happens when the queue is full: Wait makes the writer wait (in our example TryWrite returns false without waiting, so we reject), DropOldest and DropNewest throw away a job in the queue, DropWrite silently drops the job being written. For work that must not be lost, like invoices, the silent dropping options are wrong; they can be right for flows where only the newest value matters, like telemetry or "last known position".

Queue Depth and Autoscaling

Queue depth is the most honest metric of the system: it grows when the arrival rate exceeds the processing rate and drains when it doesn't. How many workers you need is told by Little's law from the series' first post: required concurrency = arrival rate × processing time. If 2,000 invoices arrive in 10 minutes at month-end, the arrival rate is 3.3 per second; with 0.4-second processing that is 3.3 × 0.4 = 1.3 concurrent jobs, so two workers are enough in steady state. 100 jobs arriving at once, however, pile up in a queue whatever the capacity, and the drain time is depth ÷ processing rate: 100 ÷ 10 = 10 seconds.

This math can become an autoscaling rule: "add workers when the drain time of the work in the queue exceeds 30 seconds". In Kubernetes, KEDA (Kubernetes Event-driven Autoscaling) does this by changing the pod count based on RabbitMQ, Azure Service Bus or Kafka queue depth. Scaling on CPU is late for this job: CPU is high once the workers are already full, while the number of jobs waiting in the queue shows the problem earlier. A warning: scaling the workers does not scale the database they talk to; the connection pool math from the series' first post applies to the worker count too.

Fan-Out: One Event, Several Jobs

When an order is placed, not only an invoice is needed but also a confirmation e-mail, a stock reservation and an analytics record. Calling all of them in turn from OrderService means touching the order code for every new job. With fan-out the order publishes a single OrderPlaced event and each interested job gets its own copy through its own subscription: publish/subscribe. The difference between a queue (point-to-point) and a topic (pub/sub) matters here: a message in a queue is taken by only one of the workers (competing consumers), while a copy of a message in a topic goes to every subscriber.

Fan-out has two effects on the design. Each subscriber lives at its own pace and with its own failures: when the e-mail provider goes down, invoice rendering is not affected. But each subscriber can fall behind independently, so the question "is the order complete?" is no longer answered in one place. These concepts in depth are in the Messaging and Event-Driven Architecture series post "Channels: Point-to-Point Queues, Publish/Subscribe Topics, and Competing Consumers".

At-Least-Once and Idempotency

Almost all real message brokers deliver at-least-once: if a worker processes a message and crashes before sending its acknowledgement (ack), the message is given to another worker again. So the same invoice job can arrive twice. The design has to withstand that, and idempotency is required in two places.

The first is the API entry. If the client doesn't get the 202 response because of the network, it repeats the request; if the server doesn't know it is the same request, it opens a second job. The fix is for the client to send an Idempotency-Key header for each logical operation and for the server to return the same job for the same key. This header is still a draft at the IETF (Internet Engineering Task Force), but the pattern page on Microsoft Learn recommends exactly this:

idempotency: 202 /invoice-jobs/099377df-abcd-4bbb-a99d-f488c577c8d5 | 202 /invoice-jobs/099377df-abcd-4bbb-a99d-f488c577c8d5 | same job: True

The second is the worker. When the same message arrives a second time, the worker must not do the work again, or doing it again must be harmless. RenderAsync in the example skips an order whose invoice has already been rendered:

redelivery: rendered 1, skipped as duplicate 1

In a real system this check is not an in-memory dictionary but a unique constraint in the database (Invoices.OrderId) or a table of processed messages; the check and the write need to be in the same transaction. The full story is in the Messaging and Event-Driven Architecture series posts "Delivery Guarantees: At-Most-Once, At-Least-Once, and the Truth About "Exactly-Once"" and "Idempotent Consumers: What Happens When the Same Message Arrives Twice?".

Common Pitfalls

Thinking an in-memory queue is durable. The Channel<T> in the example lives in the process's memory; on a deployment or a crash every job in the queue is lost, and the status of jobs that got 202 stays "Queued" forever. Work that must not be lost needs a durable broker (RabbitMQ, Azure Service Bus) and an outbox that writes the message in the same transaction as the database change; the .NET clients are in the upcoming Working with Message Queues series in the Backend category.

Removing backpressure with an unbounded queue. The load doesn't disappear, it piles up in the queue; set a limit and reject honestly when it is full.

Returning 202 without a status endpoint. If the client cannot find out whether the work finished, 202 is not an answer but a black hole.

Retrying a poison message forever. A message that throws an exception every time keeps a worker busy and holds up the work behind it; move it to a dead-letter queue after a certain number of attempts.

Assuming the queue preserves order. With several workers running in parallel, jobs finish in a different order than they arrived; if order matters, send the jobs of the same key to the same worker with a partition key.

Making everything asynchronous. Work whose result the user needs right away (payment confirmation, stock check) fills the user experience with polling and uncertainty when made asynchronous; keep asynchrony for work that is not part of the response.

A Small Experiment

★☆☆ Raise the queue capacity from 50 to 200 and send the same 100 requests. How many were accepted, and how many seconds did the queue take to drain? Explain the effect of a larger capacity on the client's waiting time with Little's law.

★★☆ Raise the worker count from 4 to 8 but leave the semaphore in InvoiceRenderer at 4. Did the drain time change? Why? Think of a real example where the bottleneck is not the worker count but another resource.

★★★ Add a failure scenario to the worker: invoices whose orderId is a multiple of 13 throw an exception. Build a structure that moves the job to a dead-letter list after three attempts and returns Failed on the status endpoint with an error in the RFC (Request for Comments) 9457 Problem Details format.

With this post the first half of the series' building blocks is complete: estimation, scaling, load balancing, caching, the data layer, consistent hashing and asynchronous processing. In the next post we design one of these building blocks end to end: rate limiting algorithms from token bucket to sliding window, and why the ASP.NET Core rate limiter keeping its counters per instance calls for a distributed rate limiter.