End to End: Bookstore Order Documents — Invoice PDF, Confirmation Email and the Monthly Excel Report as a Job Chain

Version note: The code in this post was built and run with .NET 10, Hangfire 1.8.25, Hangfire.Storage.SQLite 0.4.3, QuestPDF 2026.9.2, Azure.Storage.Blobs 12.30.1, Mailpit 1.33.1 and Azurite 3.37.0.

Guess First

In this post we put the series' pieces together for Bookstore, the online bookstore used as the shared example across this site's series: when an order is placed, an invoice PDF is produced, written to storage, and a confirmation email with the PDF attached goes out. The monthly sales report is also prepared on its own on the first day of every month. Three questions:

First: storage is down while the invoice job runs. Which step does Hangfire retry, and what does the email step do in the meantime?

Second: OrderPlaced arrived twice for the same order. How many PDFs are produced, and how many emails does the customer get?

Third: the report job entered the queue and the API stopped right after. What happens when it starts again?

The answers come from real runs. In the first case only the invoice step is retried, 5 and then 30 seconds apart; the email step waits as Awaiting, and once storage is back the chain completes from where it stopped. In the second, one PDF and one email: the second run logs "already stored" and "already queued". In the third the job waits in the database as Enqueued and runs when the application starts.

The Pieces of the Chain

Every step of the chain was written in an earlier post. InvoiceArchive from the fifth post produces the invoice, IFileStore from the eighth post stores it, and the outbox and dispatcher from the third post send the email. The new code in this post is the two job methods that connect the steps:

// Key: the file path invoices/{year}/{invoiceNumber}.pdf; a second run finds the file and renders nothing
public async Task<string> GenerateInvoiceAsync(int orderId, CancellationToken cancellationToken)
{
    InvoiceModel invoice = await invoices.GetAsync(orderId, cancellationToken);
    bool existed = await archive.ExistsAsync(invoice, cancellationToken);
    string path = await archive.EnsureStoredAsync(invoice, cancellationToken);
    logger.LogInformation("Invoice for order {OrderId}: {Path} ({State})", orderId, path, existed ? "already stored" : "rendered");
    return path;
}

// Key: "order-confirmed:{orderId}" in the outbox; a second run finds the row and queues nothing
public async Task QueueConfirmationEmailAsync(int orderId, CancellationToken cancellationToken)
{
    InvoiceModel invoice = await invoices.GetAsync(orderId, cancellationToken);
    string path = InvoiceArchive.PathFor(invoice);
    ...
    bool queued = await outbox.EnqueueAsync(db, $"order-confirmed:{orderId}", message,
        new StoredAttachment(path, $"invoice-{invoice.InvoiceNumber}.pdf", FileFormat.Pdf.ContentType), cancellationToken);
    await db.SaveChangesAsync(cancellationToken);
    logger.LogInformation("Confirmation email for order {OrderId}: {State}", orderId, queued ? "queued" : "already queued");
}

The rule is simple: every step in the chain has its own idempotency key, and that key is the very thing the step writes. The invoice's key is the file path, the email's key is the outbox row, and on the SMTP side the key is the Message-Id we derived from it in the third post. Retries are the normal case for background jobs. The "at-least-once" reality from the retry and idempotency post of the Background Jobs series applies here too. These classes live in Bookstore.Notifications and do not know Hangfire. The API builds the chain.

The Chain with Hangfire: Enqueue and ContinueJobWith

In the first post we queued the report in an in-memory Channel, and queued work was lost when the application restarted. For the durable version we use Hangfire. Bookstore's real database is SQL Server, where Hangfire.SqlServer is used. This series runs on SQLite, so the jobs live in a separate file with Hangfire.Storage.SQLite:

// Durable jobs: Hangfire keeps them in its own SQLite file (Hangfire.SqlServer next to BookstoreDb in production)
builder.Services.AddHangfire(config => config
    .SetDataCompatibilityLevel(CompatibilityLevel.Version_180)
    .UseSimpleAssemblyNameTypeSerializer()
    .UseRecommendedSerializerSettings()
    .UseSQLiteStorage(builder.Configuration.GetConnectionString("Jobs"), new SQLiteStorageOptions { QueuePollInterval = TimeSpan.FromSeconds(1) })
    .UseFilter(new AutomaticRetryAttribute { Attempts = 5, DelaysInSeconds = [5, 30, 120, 600, 3600] }));
builder.Services.AddHangfireServer(options => options.WorkerCount = 4);

The chain is two lines. The second job, created with ContinueJobWith, waits as Awaiting until the first one is Succeeded:

// Called by the relay that reads OrderPlaced from the Ordering outbox; may be called twice for the same order
app.MapPost("/api/orders/{orderId:int}/placed", (int orderId, IBackgroundJobClient jobs) =>
{
    string invoiceJob = jobs.Enqueue<OrderDocumentJobs<BookstoreContext>>(j => j.GenerateInvoiceAsync(orderId, CancellationToken.None));
    // Runs only after the invoice job succeeded; a failed invoice job is retried on its own
    string emailJob = jobs.ContinueJobWith<OrderDocumentJobs<BookstoreContext>>(invoiceJob,
        j => j.QueueConfirmationEmailAsync(orderId, CancellationToken.None));
    return Results.Accepted($"/api/jobs/{emailJob}", new { invoiceJob, emailJob });
});

CancellationToken.None is only a placeholder; Hangfire puts its own token in its place when it runs the job. In Bookstore the Ordering module's outbox relay calls this endpoint. OrderPlaced is written in the same transaction as the order, and the relay hands it to Hangfire. If the relay crashes it may hand over the same event twice; the mechanics are in the Messaging and Event-Driven Architecture series. The first try:

$ curl -s -w '\nHTTP %{http_code}\n' -X POST http://127.0.0.1:52560/api/orders/1042/placed
{"invoiceJob":"1","emailJob":"2"}
HTTP 202
$ curl -s http://127.0.0.1:52560/api/jobs/1
{"jobId":"1","state":"Succeeded","job":"OrderDocumentJobs.GenerateInvoiceAsync"}
$ curl -s http://127.0.0.1:52560/api/jobs/2
{"jobId":"2","state":"Succeeded","job":"OrderDocumentJobs.QueueConfirmationEmailAsync"}
$ curl -s http://127.0.0.1:52560/api/outbox
[{"idempotencyKey":"order-confirmed:1042","status":"Sent","attempts":1,"nextAttemptAtUtc":"2026-11-15T07:12:24.127751","sentAtUtc":"2026-11-15T07:12:26.077986"}]

A license note: the LICENSE.md in Hangfire's NuGet package says the software can be used under the LGPL v3 or under a paid commercial license. The commercial license is needed for distributing modified copies privately and for the Pro packages. Bookstore uses the library unmodified as a NuGet package. The check from the first post turned red this time too: the license came as a file and the tool could not determine its type. We added a version-pinned LGPL-3.0-only override for the three Hangfire packages and put that license on the allow list. This part of the decision also has to be reviewed when the version is upgraded.

When a Step Fails

We stopped the Azurite container and placed order 1043. In the first attempt Hangfire never stepped in: because of the Azure SDK's own retries, the invoice job stayed Processing for 25 seconds and finished in the same attempt once Azurite was back. That is the cost of two layers of retries. While the SDK waits, Hangfire's worker thread stays busy for that whole time. We shortened the SDK's retries and left decisions about long outages to the job:

// The SDK retries on its own; keep that short so a job's own retry (Hangfire) decides about longer outages
BlobClientOptions blobOptions = new()
{
    Retry = { MaxRetries = 2, Delay = TimeSpan.FromSeconds(1), NetworkTimeout = TimeSpan.FromSeconds(10) },
};

We repeated the experiment with order 1044. The state after 30 seconds:

{"jobId":"5","state":"Scheduled","job":"OrderDocumentJobs.GenerateInvoiceAsync","reason":"Retry attempt 2 of 5: Retry failed after 3 tries. Retry settings can be…"}
{"jobId":"6","state":"Awaiting","job":"OrderDocumentJobs.QueueConfirmationEmailAsync"}

In the log:

warn: Hangfire.AutomaticRetryAttribute[0]
      Failed to process the job '5': an exception occurred. Retry attempt 1 of 5 will be performed in 00:00:05.
      System.AggregateException: Retry failed after 3 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (Connection refused (127.0.0.1:52581)) ...
warn: Hangfire.AutomaticRetryAttribute[0]
      Failed to process the job '5': an exception occurred. Retry attempt 2 of 5 will be performed in 00:00:30.

After we restarted Azurite both jobs became Succeeded. The state history in Hangfire's own table shows the chain step by step:

$ sqlite3 bookstore-jobs.db "select JobId, Name, substr(Reason, 1, 100) from State where JobId in (5, 6) order by Id"
5|Enqueued|
6|Awaiting|
5|Processing|
5|Failed|An exception occurred during performance of the job.
5|Scheduled|Retry attempt 1 of 5: Retry failed after 3 tries. Retry settings can be…
5|Enqueued|Triggered by DelayedJobScheduler
5|Processing|
5|Failed|An exception occurred during performance of the job.
5|Scheduled|Retry attempt 2 of 5: Retry failed after 3 tries. Retry settings can be…
5|Enqueued|Triggered by DelayedJobScheduler
5|Processing|
6|Enqueued|
5|Succeeded|
6|Processing|
6|Succeeded|

Job 6 never failed, because it never started early. The chain does not have to start over at a failed step; the step that failed is retried in its place.

The Same Order Twice

We placed 1043 once more. Hangfire opened two new jobs and both succeeded, but they did no work:

info: Bookstore.Notifications.Jobs.OrderDocumentJobs[0]
      Invoice for order 1043: invoices/2026/BKS-2026-001043.pdf (already stored)
info: Bookstore.Notifications.Jobs.OrderDocumentJobs[0]
      Confirmation email for order 1043: already queued

Mailpit has one email per order, each with one attachment:

Monthly sales report 2026-10 is ready | reports@example.com | attachments 0
Your Bookstore order #1044 is confirmed | ada@example.com | attachments 1
Your Bookstore order #1043 is confirmed | ada@example.com | attachments 1
Your Bookstore order #1042 is confirmed | ada@example.com | attachments 1

If two copies run at the same moment, both may find no file and render it. The invoice is deterministic, so the second write overwrites the same file. On the email side the outbox's IdempotencyKey column has a unique index, so the second insert fails, Hangfire retries the step, and the retry logs "already queued".

Attachments in Storage, Not in the Outbox

In the third post we recommended not embedding attachments in the outbox row. The outbox row keeps only the file's path; the dispatcher reads the attachment from IFileStore when it sends the email:

private async Task<EmailMessage> ToMessageAsync(OutboxEmail email, CancellationToken cancellationToken)
{
    EmailMessage message = email.ToMessage();
    if (email.AttachmentPath is null)
    {
        return message;
    }

    using MemoryStream content = new();
    await using (Stream stored = await files.OpenReadAsync(email.AttachmentPath, cancellationToken))
    {
        await stored.CopyToAsync(content, cancellationToken);
    }

    return message with { Attachments = [new EmailAttachment(email.AttachmentName!, email.AttachmentContentType!, content.ToArray())] };
}

A missing file is a permanent error: retrying will not bring it back. In that case the dispatcher marks the email Failed on the first attempt, and there is a unit test for it. The link in the email is not a storage URL but the application's https://bookstore.example.com/orders/1042/invoice address. When the user clicks it, the application checks ownership and produces the 5-minute URL from the eighth post. A SAS put into an email would be in someone else's hands the moment the email is forwarded.

The Monthly Report: a Recurring Job

The report job follows the same rule. The file path is reports/monthly-sales-2026-10.xlsx, the outbox key monthly-sales:2026-10. For scheduling we use Hangfire's recurring jobs:

// 06:00 Istanbul time on the 1st of every month; AddOrUpdate is idempotent, so it can run on every start
app.Services.GetRequiredService<IRecurringJobManager>().AddOrUpdate<MonthlySalesReportJob<BookstoreContext>>(
    JobEndpoints.MonthlyReportId, j => j.RunAsync(null, CancellationToken.None), "0 6 1 * *",
    new RecurringJobOptions { TimeZone = TimeZoneInfo.FindSystemTimeZoneById("Europe/Istanbul") });

The cron expression and the time zone are stored in Hangfire's table. The next execution time is 1796094000000, that is 03:00 UTC on 1 December 2026, 06:00 in Istanbul. Without the time zone the expression would be read as UTC and the report would run three hours early; DST pitfalls are covered in the cron post of the Background Jobs series. We triggered the job without waiting and got the line Monthly sales report 2026-10: reports/monthly-sales-2026-10.xlsx, email queued. Since the month parameter was null, the job computed the previous month in Istanbul time.

We tested durability too. We queued the September report and stopped the API right away:

$ curl -s -X POST -H 'Content-Type: application/json' -d '{"month":"2026-09"}' http://127.0.0.1:52560/api/reports/monthly-sales
{"jobId":"10"}
$ sqlite3 bookstore-jobs.db "select Id, StateName from Job where Id = 10"
10|Enqueued

A few seconds after the API started again, /api/jobs/10 showed Succeeded, and the log had the line Monthly sales report 2026-09: reports/monthly-sales-2026-09.xlsx, email queued. The Channel from the first post lost the work in the same situation.

Integration Test: Mailpit, Azurite and PdfPig

Hangfire is only the scheduler. What we really want to verify is that the steps work correctly against real infrastructure and can be repeated. The test calls the steps without Hangfire, each of them twice. The invoice goes to Azurite, the email to Mailpit with STARTTLS and a password, and at the end the test downloads the attachment through Mailpit's API and reads it with PdfPig:

for (int run = 0; run < 2; run++)
{
    await jobs.GenerateInvoiceAsync(orderId, ct);
    await jobs.QueueConfirmationEmailAsync(orderId, ct);
}

await using (MailKitEmailSender sender = new(Options.Create(MailpitSmtp()), NullLogger<MailKitEmailSender>.Instance))
{
    OutboxEmailDispatcher<BookstoreContext> dispatcher = new(services.GetRequiredService<IServiceScopeFactory>(), sender, files,
        TimeProvider.System, outboxOptions, NullLogger<OutboxEmailDispatcher<BookstoreContext>>.Instance);
    Assert.Equal(1, await dispatcher.DispatchDueAsync(ct));
    Assert.Equal(0, await dispatcher.DispatchDueAsync(ct));
}

...
// The attachment is the stored PDF: its text has the order number, the Turkish address and the total
byte[] pdfBytes = await mailpit.GetByteArrayAsync($"/api/v1/message/{summary.ID}/part/{attachment.PartID}", ct);
using PdfDocument pdf = PdfDocument.Open(pdfBytes);
string text = string.Join(" ", pdf.GetPage(1).GetWords().Select(w => w.Text));
Assert.Contains($"Order #{orderId}", text);
Assert.Contains("34710 İstanbul, Türkiye", text);
Assert.Contains("Grand total ₺1.702,00", text);

The order number is random on every run, so emails and files from earlier runs do not affect the result. Without the environment variables the test skips itself; in CI it is enough to start the two containers as services and pass the variables:

$ export MAILPIT_API_URL=http://127.0.0.1:52562 MAILPIT_SMTP_PORT=52561
$ export Notifications__Smtp__Password=<password> Notifications__Smtp__TrustedCertificateSha256=<sha256>
$ export AZURITE_CONNECTION_STRING=<connection string>
$ dotnet test --project tests/Bookstore.Notifications.Tests -- --filter-class "*OrderDocumentsChainTests" --filter-class "*MailpitEmailTests"
  total: 2
  failed: 0
  succeeded: 2
  duration: 898ms

At the end of the post the whole test project passed 30 of 31 tests. The one skipped test needs an S3-compatible server and passed against RustFS in the eighth post.

In Production

Keep Hangfire's storage on the same server as the application's database, but in a separate schema. In Bookstore that means Hangfire.SqlServer on BookstoreDb.

Protect the dashboard. MapHangfireDashboard is open only to requests from the same machine by default; in production open it to back office roles only, with an authorization filter. Monitoring jobs and alerting are the subject of the dashboard and monitoring post of the Background Jobs series.

Separate the CPU-heavy work. Cover thumbnails are a Hangfire job now too. Moving document and image work to a separate queue, and if needed a separate worker process, keeps order emails from waiting behind a burst of images.

Know your retry layers. The SDK, the HTTP client, Hangfire and the outbox dispatcher each have their own retries. Keep the inner layers short; let the outer layer decide about long outages.

Do not lose the event. Writing OrderPlaced in the same transaction as the order and handing it to Hangfire through a relay makes the start of the chain reliable as well. Since the steps are idempotent, the relay handing over the same event twice is not a problem.

Common Pitfalls

Starting over when a step in the chain fails. The invoice number gets taken twice, or the customer sees two different PDFs. Every step should be retried in its place.

Leaving idempotency to Hangfire. Hangfire runs a job at least once; it does not guarantee that it never runs twice.

Putting a long-lived storage link into an email. Let the link go to the application, and let the application produce a short-lived URL at that moment.

Embedding the attachment in the outbox row. The database grows and the same PDF lives in two places.

Defining a recurring job without a time zone.

A Small Experiment

★☆☆ Set Attempts of AutomaticRetryAttribute to 1, place an order while Azurite is down and look at the job's final state through /api/jobs/{id}. What happens to the continuation?

★★☆ Add a third step to OrderDocumentJobs: a job that marks the order as "documents ready" after the invoice email has been sent. What should this step's idempotency key be?

★★★ Write a real outbox relay for OrderPlaced: a BackgroundService that reads the event row, hands it to Hangfire and marks the row. What happens if the relay crashes after handing over to Hangfire but before marking the row, and why is that not a problem?

This is where the series ends. Bookstore now produces email with MailKit and Razor templates, Excel with ClosedXML, PDFs with QuestPDF and Chromium, processes covers with SkiaSharp, keeps files on the local disk, in Azure Blob Storage or in S3, and connects all of it with repeatable jobs. A check in CI tracks every library's license. The license changes we met along the way, PDFsharp moving to the AGPL and ImageSharp 4's build key, showed why that check has to run on every restore. To go deeper, return to the Background Jobs series for operating jobs, the Messaging and Event-Driven Architecture series for event and outbox mechanics, and the Cloud Fundamentals series for storage concepts.