Worker Service and an In-Process Job Queue: Channel, PeriodicTimer and Windows Service/systemd

Version note: The code in this post was built and run with .NET 10 and Microsoft.Extensions.Hosting 10.0.14 (including the WindowsServices and Systemd packages).

Guess First

Bookstore's order confirmation emails now go through a queue. The queue's capacity is 5, and the worker sending emails finishes each one in 1 second. A campaign starts and 10 orders are confirmed at the same moment. How many of the 10 requests get 202 Accepted and how many are rejected? Which status code is right for the rejected ones: 429 or 503? And: if the app is restarted right then, what happens to the emails in the queue?

When Does a Separate Worker Process Make Sense?

In the previous post we ran a BackgroundService inside the API (Application Programming Interface). That's often the right start: same deploy, same configuration, same DI (Dependency Injection) container. But some work shouldn't live in the same process as the API. If a CPU (Central Processing Unit) hungry report generation hurts the API's response times; if the background work scales differently from the API (6 API replicas, a single stock sync); or if the work has nothing to do with HTTP (Hypertext Transfer Protocol), a separate process is cleaner.

.NET gives you a template for it:

dotnet new worker -n Bookstore.StockSync --framework net10.0

using Bookstore.StockSync;

var builder = Host.CreateApplicationBuilder(args);
builder.Services.AddHostedService<Worker>();

var host = builder.Build();
host.Run();

The difference is small: Host.CreateApplicationBuilder instead of WebApplication, no Kestrel, and the project SDK (Software Development Kit) is Microsoft.NET.Sdk.Worker. The Worker Services page on Microsoft Learn points out one detail: the worker template doesn't enable server GC (garbage collection) by default; for a long-running, memory-heavy worker you should weigh that in the project file yourself.

We'll see both kinds in this post: an email queue inside the API, and a stock sync running as a separate process.

An In-Process Job Queue: A Bounded Channel

We covered the mechanics of Channel in the C# Async series post "Producer/Consumer with C# Channels: Backpressure, Bounded Channels and Pipelines". Here we hide it behind an application service, so endpoints see an interface that speaks their language rather than the Channel:

public sealed record EmailJob(int OrderId, string To, string Subject);

public interface IBackgroundTaskQueue
{
    bool TryEnqueue(EmailJob job);
    IAsyncEnumerable<EmailJob> DequeueAllAsync(CancellationToken ct);
    int Count { get; }
}

public sealed class EmailQueue : IBackgroundTaskQueue
{
    private readonly Channel<EmailJob> _channel;

    public EmailQueue(int capacity) =>
        _channel = Channel.CreateBounded<EmailJob>(new BoundedChannelOptions(capacity)
        {
            FullMode = BoundedChannelFullMode.Wait, // TryWrite returns false on a full queue
            SingleReader = true
        });

    public bool TryEnqueue(EmailJob job) => _channel.Writer.TryWrite(job);

    public IAsyncEnumerable<EmailJob> DequeueAllAsync(CancellationToken ct) => _channel.Reader.ReadAllAsync(ct);

    public int Count => _channel.Reader.Count;
}

There are two decisions. The first is CreateBounded: a Channel without a capacity grows memory without limit when the worker can't keep up, and you only notice when the process dies of out-of-memory. The second is FullMode. Wait makes WriteAsync wait on a full queue; DropOldest, DropNewest and DropWrite silently throw a job away. Silently dropping an email is unacceptable, and waiting ties up the request thread. So we pick the Wait mode and use TryWrite instead of WriteAsync: if the queue is full, TryWrite returns false without waiting and the endpoint makes the call.

The Endpoint and Backpressure: 503 When Full

builder.Services.AddSingleton<IBackgroundTaskQueue>(new EmailQueue(capacity: 5));
builder.Services.AddHostedService<EmailWorker>();

var app = builder.Build();

app.MapPost("/orders/{id:int}/confirmation-email", (int id, IBackgroundTaskQueue queue, HttpContext http) =>
{
    var job = new EmailJob(id, $"customer{id}@example.com", $"Order #{id} confirmed");
    if (queue.TryEnqueue(job))
        return Results.Accepted();

    // queue is full: instead of making the client wait, tell it "try again later"
    http.Response.Headers.RetryAfter = "10";
    return Results.Problem(statusCode: StatusCodes.Status503ServiceUnavailable,
        title: "Email queue is full");
});

I set the queue capacity to 5 and sent 10 requests at once:

$ seq 1 10 | xargs -P 10 -I{} curl -s -o /dev/null -w "%{http_code}\n" -X POST http://127.0.0.1:17112/orders/100{}/confirmation-email | sort | uniq -c
   5 202
   5 503

Five requests were accepted and five were rejected. The full response of a rejected request:

HTTP/1.1 503 Service Unavailable
Content-Type: application/problem+json
Date: Sun, 08 Nov 2026 01:58:06 GMT
Server: Kestrel
Retry-After: 10
Transfer-Encoding: chunked

{"type":"https://tools.ietf.org/html/rfc9110#section-15.6.4","title":"Email queue is full","status":503}

Why 503 and not 429? 429 Too Many Requests says the client itself exceeded a limit; it's the answer when a rate limiter applies a per-user quota. Here the problem isn't one client but the server's current capacity; by RFC (Request for Comments) 9110's definition, 503 Service Unavailable means "the server is temporarily overloaded". In both cases the Retry-After header tells the client when to try again. ASP.NET Core's ProblemDetails support fills the type field in the body automatically.

The real gain of backpressure is that you choose where overload shows up. With an unbounded queue, everything looks "successful" on campaign morning, while emails are delayed for hours and vanish together if the process restarts. With a bounded queue, the load shows up right away as a measurable 503 rate, and the client, or a gateway in front of it, can handle it with a retry strategy.

The Worker: Draining the Queue and What's Left at Shutdown

public sealed class EmailWorker(IBackgroundTaskQueue queue, ILogger<EmailWorker> logger) : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        try
        {
            await foreach (EmailJob job in queue.DequeueAllAsync(stoppingToken))
            {
                await Task.Delay(1000, stoppingToken); // simulates the SMTP server
                logger.LogInformation("Sent '{Subject}' for order {OrderId}", job.Subject, job.OrderId);
            }
        }
        catch (OperationCanceledException)
        {
            logger.LogWarning("Stopping with {Count} emails still in the queue", queue.Count);
        }
    }
}

Right after the load test above, with two emails sent, I sent SIGTERM (terminate signal):

04:57:42.499 info: Bookstore.Api.EmailWorker[0] Sent 'Order #1004 confirmed' for order 1004
04:57:43.501 info: Bookstore.Api.EmailWorker[0] Sent 'Order #1001 confirmed' for order 1001
04:57:43.718 info: Microsoft.Hosting.Lifetime[0] Application is shutting down...
04:57:43.721 warn: Bookstore.Api.EmailWorker[0] Stopping with 3 emails still in the queue

Two emails went out, three stayed in the queue and vanished with the process. At least there's a log line saying so; the Task.Run from the earlier post didn't even say that. Choosing SingleReader = true is deliberate too, so we can use the Count property: with a single reader, the Channel's reader can count how many items are waiting.

You can try to finish the remaining work on shutdown: once stoppingToken is cancelled, keep reading until the queue is empty, as far as ShutdownTimeout allows. But that only saves deploys; if the process crashes or the machine goes down, the queue is lost anyway. We'll talk about this limit at the end of the post.

Periodic Work with PeriodicTimer

The second example is a separate worker process: Bookstore.StockSync, which periodically pulls stock information from the supplier. We put the PeriodicTimer from the C# Async series post "C# Timers and Waiting: PeriodicTimer, WaitAsync and Task.Delay" into a BackgroundService:

public sealed class StockSyncOptions
{
    public TimeSpan Interval { get; set; } = TimeSpan.FromMinutes(5);
}

public sealed class StockSyncWorker(
    ISupplierClient supplier,
    IOptions<StockSyncOptions> options,
    ILogger<StockSyncWorker> logger) : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        using PeriodicTimer timer = new(options.Value.Interval);
        do
        {
            long start = Stopwatch.GetTimestamp();
            try
            {
                int updated = await supplier.SyncStockAsync(stoppingToken);
                logger.LogInformation("Stock sync updated {Count} books in {Elapsed:F0} ms",
                    updated, Stopwatch.GetElapsedTime(start).TotalMilliseconds);
            }
            catch (Exception ex) when (ex is not OperationCanceledException)
            {
                // one failed run must not take down the worker (and the host)
                logger.LogError(ex, "Stock sync failed, will retry on next tick");
            }
        }
        while (await timer.WaitForNextTickAsync(stoppingToken));
    }
}

The do...while is deliberate: the first run happens as soon as the app starts, then repeats on every tick. Each run is protected by its own try/catch, because, as we saw in the previous post, an unhandled exception stops the whole host with the default setting. We read the interval from configuration; for the demo I set it to 1 second and made the fake supplier client finish its second call in 2.5 seconds and its fourth with an error:

04:57:02.041 info: Microsoft.Hosting.Lifetime[0] Application started. Press Ctrl+C to shut down.
04:57:02.347 info: Bookstore.StockSync.StockSyncWorker[0] Stock sync updated 121 books in 303 ms
04:57:05.547 info: Bookstore.StockSync.StockSyncWorker[0] Stock sync updated 122 books in 2502 ms
04:57:05.849 info: Bookstore.StockSync.StockSyncWorker[0] Stock sync updated 123 books in 302 ms
04:57:06.351 fail: Bookstore.StockSync.StockSyncWorker[0] Stock sync failed, will retry on next tick System.Net.Http.HttpRequestException: Supplier API timed out [...]
04:57:07.346 info: Bookstore.StockSync.StockSyncWorker[0] Stock sync updated 125 books in 301 ms
04:57:07.970 info: Microsoft.Hosting.Lifetime[0] Application is shutting down...

Look at the timestamps. The ticks sit on a fixed 1-second grid: the x.047s. The second run took 2.5 seconds and missed two ticks in the meantime. But when it finished, the third run started immediately (05.547), and only once; the missed ticks didn't pile up. The next run went back to the grid (06.047). That's how PeriodicTimer behaves: it keeps one tick and drops the rest. The old System.Threading.Timer, by contrast, calls the callback again without waiting for the previous run to finish, and runs overlap; we'll see the real output of that in the last post of the series. The error in the fourth run was logged and the worker carried on at the next tick.

Windows Service, systemd and Containers

If you'll run the worker on a machine for a long time, you need to hook it into the operating system's service manager:

using Bookstore.StockSync;

var builder = Host.CreateApplicationBuilder(args);

// Activates when running under the Service Control Manager on Windows or systemd on Linux;
// run from a console, neither does anything
builder.Services.AddWindowsService(o => o.ServiceName = "Bookstore Stock Sync");
builder.Services.AddSystemd();

builder.Services.Configure<StockSyncOptions>(builder.Configuration.GetSection("StockSync"));
builder.Services.AddSingleton<ISupplierClient, FakeSupplierClient>();
builder.Services.AddHostedService<StockSyncWorker>();

var host = builder.Build();
host.Run();

Both calls are context aware. According to the packages' XML (Extensible Markup Language) documentation, AddWindowsService, if the process really runs as a Windows Service, ties the host's lifetime to the Windows Service Control Manager and enables logging to the Windows Event Log under the application name. AddSystemd, if the process runs under systemd, sends systemd "started" and "stopping" notifications and switches the console log format to the one systemd expects; for the notifications to work, the unit file needs Type=notify. Neither does anything in any other environment; the logs above come from the same binary running as a plain console app on macOS. That's why adding both is safe.

I didn't run these Windows and systemd integrations under a real Windows Service or systemd unit for this post; their behavior is based on the documentation above. In a container you need neither: the container itself is the process manager. The rule is simple: one process per container, the app itself receives SIGTERM, logs go to stdout. If you start the container through a shell script, SIGTERM goes to the script and may never reach the .NET process; then graceful shutdown doesn't work and the orchestrator sends SIGKILL (kill signal) once the grace period runs out.

The Limit: When Isn't an In-Memory Queue Enough?

The queue in this post lives in the process's memory. That has three consequences. It empties on restart and deploy; that's how we lost three emails above. It doesn't scale; when you take the API to 3 replicas, each replica has its own queue, and work piling up on one replica doesn't spread to the others. It's invisible; you can only learn what's in the queue and why a job failed from the logs.

If these limits are acceptable (notifications that can be lost, cache warm-ups that can be redone, analytics events), an in-process queue is the cheapest and fastest solution. If not, you need to write the work somewhere durable: if the work itself is a job, the database-backed store of Hangfire or Quartz.NET (the fifth and sixth posts); if it's an event other services will also react to, a message broker (the upcoming Working with Message Queues series). If you need to record the work in the same transaction as a database change, an outbox table is the right tool; the Messaging and Event-Driven Architecture series post "Transactional Outbox and Inbox: Understanding and Designing Around the Dual Write Problem" explains why.

In Production

Publish queue depth as a metric. Wiring Count to a gauge lets you see the queue filling up on campaign morning before the 503s start. The 503 rate should be its own metric too; anything above zero tells you capacity isn't enough.

Pick the capacity by measuring. Capacity sets the longest delay you can accept: if the worker sends 10 emails a second, a queue of 600 means a worst-case delay of one minute and 600 jobs that can be lost on restart. Raising the capacity cuts 503s but raises both numbers.

Choose the worker's parallelism deliberately. A single-reader worker does jobs one after another; if the external service is slow you may need several readers (or Parallel.ForEachAsync), but then you must watch the external service's rate limit.

Common Pitfalls

Starting with Channel.CreateUnbounded. Nothing happens in testing; in production, memory grows the first day the worker slows down, and the problem only shows when the process dies.

Making the request wait on a full queue. Awaiting WriteAsync on a full queue also makes the client and Kestrel's connections wait; instead of reflecting backpressure to the client, you've buried it inside the server.

Putting scoped objects in the queue. Put simple values such as ids in the queue, not a DbContext or entities; let the worker reload what it needs in its own scope.

Starting the container with a shell script and losing SIGTERM. Write ENTRYPOINT in exec form, or use exec dotnet ... at the end of the script.

A Small Experiment

★☆☆ Raise the queue capacity from 5 to 50 and send 100 concurrent requests. How do the 202 and 503 counts change, and how many seconds later does the last email go out?

★★☆ Change EmailWorker to try draining the queue on shutdown: when stoppingToken is cancelled, keep sending the rest with TryRead, but finish before ShutdownTimeout. How many emails can you save?

★★★ Make every call of FakeSupplierClient take 1.5 seconds and leave the interval at 1 second. Study the run start times in the log: what is PeriodicTimer doing here, and how many runs per minute do you get?

In the next post we move to schedules more complex than "every 5 minutes": Unix's 5-field cron, the 6-field format with seconds, Quartz's ? L W # characters, choosing a time zone, and runs that are skipped or repeated across DST (Daylight Saving Time) transitions.