Retry and Idempotency in Background Jobs: Living with At-Least-Once

Version note: The code in this post was built and run with .NET 10, Hangfire 1.8.25, Quartz.NET 4.7.2, Entity Framework Core (EF Core) 10.0.14 and SQL (Structured Query Language) Server 2025.

Guess First

Bookstore's invoice email job starts like this: "has the invoice email already been sent for this order? If not, send it and record it." Sounds reasonable. If the same job runs on two workers at the same moment, because a worker crashed or a lock expired, how many emails does the customer get? Given that the check happens in the database and the IdempotencyKey column has a unique index, at most one?

Why At Least Once?

For a job to run exactly once, the job infrastructure would have to record "this work is done" atomically with the work itself. That's usually impossible: the email server, the payment provider or the supplier API (Application Programming Interface) doesn't join your database transaction. So job infrastructures offer one of two options: accept losing the work (at most once) or accept running it again (at least once). Hangfire, and Quartz's RequestRecovery, choose the second.

The sources of double execution are concrete. A worker dies in the middle of a job; someone else picks it up from the start. In the previous post we saw page 3 processed twice in a Quartz cluster, with recovering=True. It happens without a worker dying, too: if the time Hangfire keeps a job "invisible", or a distributed lock's lease, runs out, someone else can pick the work up while it's still running. Then there's retry: if the job sends a request to an external service and times out waiting for the answer, the job counts as failed and retries even if the outcome was a success. The Messaging and Event-Driven Architecture series post "Delivery Guarantees: At-Most-Once, At-Least-Once, and the Truth About "Exactly-Once"" covered this choice for messages; it's exactly the same for jobs.

Retry in Hangfire: AutomaticRetry

In Hangfire every job is protected by AutomaticRetry, registered as a global filter. I printed the defaults myself: Attempts=10 OnAttemptsExceeded=Fail. The wait formula in the 1.8.25 source code, with attempt as the attempt number, is (attempt - 1)^4 + 15 + random(0..29) * attempt seconds: the first retry comes 15–44 seconds later, the tenth more than 6561 seconds later, about two hours. The ten attempts add up to more than half a day. The "Retry attempt 1 of 10 will be performed in 00:00:26" line we saw for the renamed method in the fifth post was the first step of this formula.

You can change the default per job. A job importing a supplier invoice times out on its first two attempts:

public sealed class PermanentJobException(string message) : Exception(message);

public sealed class SupplierInvoiceJobs(ILogger<SupplierInvoiceJobs> logger)
{
    private static int _attempts;

    // transient error: two timeouts, success on the third
    [AutomaticRetry(Attempts = 3, DelaysInSeconds = [2, 5, 10])]
    public Task ImportAsync(int invoiceId)
    {
        int attempt = Interlocked.Increment(ref _attempts);
        logger.LogInformation("Importing supplier invoice {InvoiceId}, attempt {Attempt}", invoiceId, attempt);
        if (attempt < 3)
            throw new TimeoutException("Supplier API did not respond in 10 s");
        return Task.CompletedTask;
    }
}

05:11:18.845 info: Bookstore.Jobs.SupplierInvoiceJobs[0] Importing supplier invoice 7001, attempt 1
05:11:18.855 warn: Hangfire.AutomaticRetryAttribute[0] Failed to process the job '1': an exception occurred. Retry attempt 1 of 3 will be performed in 00:00:02. System.TimeoutException: Supplier API did not respond in 10 s [...]
05:11:20.777 info: Bookstore.Jobs.SupplierInvoiceJobs[0] Importing supplier invoice 7001, attempt 2
05:11:20.780 warn: Hangfire.AutomaticRetryAttribute[0] Failed to process the job '1': an exception occurred. Retry attempt 2 of 3 will be performed in 00:00:05. System.TimeoutException: Supplier API did not respond in 10 s [...]
05:11:25.815 info: Bookstore.Jobs.SupplierInvoiceJobs[0] Importing supplier invoice 7001, attempt 3

Two transient errors, waits of 2 and 5 seconds, success on the third attempt. The job's history in the State table keeps every attempt:

1|Enqueued|NULL|2026-11-08 02:11:18.630
1|Processing|NULL|2026-11-08 02:11:18.840
1|Failed|An exception occurred during performance of the job.|2026-11-08 02:11:18.857
1|Scheduled|Retry attempt 1 of 3: Supplier API did not respond in 10 s|2026-11-08 02:11:18.857
1|Enqueued|Triggered by DelayedJobScheduler|2026-11-08 02:11:20.753
1|Processing|NULL|2026-11-08 02:11:20.773
1|Failed|An exception occurred during performance of the job.|2026-11-08 02:11:20.780
1|Scheduled|Retry attempt 2 of 3: Supplier API did not respond in 10 s|2026-11-08 02:11:20.780
1|Enqueued|Triggered by DelayedJobScheduler|2026-11-08 02:11:25.800
1|Processing|NULL|2026-11-08 02:11:25.810
1|Succeeded|NULL|2026-11-08 02:11:25.820

There's an important detail here: every attempt runs the job from the start. If ImportAsync's first attempt imported half the invoice and then timed out, the second attempt imports that half again. Retry doesn't replace idempotency; it needs it.

Taking Permanent Errors Out of Retries

Not every error deserves a retry. "This invoice has no matching order" will stay the same for ten attempts; the only effect of those ten attempts is that a human sees the problem half a day later. Retry makes sense for transient errors: timeouts, 503s, deadlocks, dropped connections. Not for permanent ones: validation errors, missing records, unauthorized requests, a changed method signature.

// permanent error: retrying won't change the outcome
[AutomaticRetry(Attempts = 3, DelaysInSeconds = [2, 5, 10], ExceptOn = [typeof(PermanentJobException)])]
public Task ReconcileAsync(int invoiceId)
{
    logger.LogInformation("Reconciling invoice {InvoiceId}", invoiceId);
    throw new PermanentJobException($"Invoice {invoiceId} has no matching order");
}

Exceptions in the ExceptOn list don't enter retries. In the real run, job 2 went straight to Failed:

2|Enqueued|NULL|2026-11-08 02:11:18.650
2|Processing|NULL|2026-11-08 02:11:18.803
2|Failed|An exception occurred during performance of the job.|2026-11-08 02:11:18.813

It helps to turn this into a contract: let your domain code throw its own exception type for permanent errors (here PermanentJobException), and let the job filter recognize only that. The opposite works too: OnlyOn retries only for the listed exceptions. OnAttemptsExceeded decides what happens when the attempts run out; Fail leaves the job visible on the dashboard, Delete deletes it. Don't use Delete for work you don't want to lose.

Retry Policies in Quartz

Quartz has no retry by default: when a job throws, that run counts as failed and the trigger waits for its next time. The only tool used to be the refireImmediately flag of JobExecutionException; it ran the job again right away, without waiting, so you had to write the backoff yourself. Quartz 4.0 brought retry policies:

[RetryPolicy(3, "00:00:02", 2.0)]   // 3 retries: 2 s, 4 s, 8 s
public sealed class ExchangeRateImportJob(ILogger<ExchangeRateImportJob> logger) : IJob
{
    private static int _runs;

    public ValueTask Execute(IJobExecutionContext context, CancellationToken cancellationToken)
    {
        int run = Interlocked.Increment(ref _runs);
        logger.LogInformation("Exchange rate import, retry attempt {Attempt}", context.RetryAttempt);
        if (run < 3)
            throw new JobExecutionException(new HttpRequestException("Central bank API returned 503"));
        logger.LogInformation("Imported 34 exchange rates");
        return ValueTask.CompletedTask;
    }
}

05:12:30.558 info: Bookstore.Jobs.ExchangeRateImportJob[0] Exchange rate import, retry attempt 0
05:12:30.562 fail: Quartz.Core.JobRunShell[1052] Job DEFAULT.exchange-rates threw a JobExecutionException:  Parameters: refire = False, unscheduleFiringTrigger = False, unscheduleAllTriggers = False   Quartz.JobExecutionException: Central bank API returned 503 [...]
05:12:30.563 info: Quartz.Core.JobRunShell[1056] Job of trigger DEFAULT.exchange-rates failed; retry 1 of 3 scheduled for 11/08/2026 02:12:32 +00:00
05:12:32.563 info: Bookstore.Jobs.ExchangeRateImportJob[0] Exchange rate import, retry attempt 1
05:12:32.563 info: Quartz.Core.JobRunShell[1056] Job of trigger DEFAULT.exchange-rates failed; retry 2 of 3 scheduled for 11/08/2026 02:12:36 +00:00
05:12:36.567 info: Bookstore.Jobs.ExchangeRateImportJob[0] Exchange rate import, retry attempt 2
05:12:36.567 info: Bookstore.Jobs.ExchangeRateImportJob[0] Imported 34 exchange rates

The wait doubles on each attempt: 2 seconds, 4 seconds. Quartz's XML (Extensible Markup Language) documentation clarifies a few details. MaxAttempts is the number of retries after the first failure; [RetryPolicy(3, ...)] means at most four runs in total. The policy is looked up, in order, on the trigger itself, on the job type's attribute, and in the scheduler default given with UseDefaultRetryPolicy. Most importantly: a retry never displaces the trigger's next regular run; a retry that would land within a second of the regular firing is dropped. So a policy whose waits are longer than the trigger interval quietly does nothing.

Idempotent Design: Claim First

We've reached the opening question. We keep every invoice email sent in a table, with a unique index on the IdempotencyKey column:

public enum EmailStatus { Sending, Sent }

public sealed class OutgoingEmail
{
    public int Id { get; set; }
    public required string IdempotencyKey { get; set; }
    public EmailStatus Status { get; set; }
    public DateTime ClaimedAtUtc { get; set; }
    public string? ProviderMessageId { get; set; }
}

// inside BookstoreDbContext.OnModelCreating
b.Entity<OutgoingEmail>().HasIndex(e => e.IdempotencyKey).IsUnique();

The naive job checks first, then sends, and records last:

// naive: check first, then send, record last
public async Task SendNaiveAsync(int orderId)
{
    string key = $"invoice-email:{orderId}";
    if (await db.OutgoingEmails.AnyAsync(e => e.IdempotencyKey == key))
        return;

    string messageId = await provider.SendAsync($"customer{orderId}@example.com", $"Invoice #{orderId}");
    db.OutgoingEmails.Add(new OutgoingEmail
    {
        IdempotencyKey = key, Status = EmailStatus.Sent, ClaimedAtUtc = DateTime.UtcNow, ProviderMessageId = messageId
    });
    await db.SaveChangesAsync();
}

I ran the same job for order 42 in two separate scopes at the same time:

naive, two concurrent runs for order 42:
  DbUpdateException: Cannot insert duplicate key row in object 'dbo.OutgoingEmails' with unique index 'IX_OutgoingEmails_IdempotencyKey'. The duplicate key value is (invoice-email:42).
  emails delivered: 2

Two emails went out. Both runs saw "no record" in AnyAsync, both did the 200 ms send, and the second hit the unique index while saving. The unique index protected the database but not the customer: in the window between the check and the record, the work with an external effect had already been done. This is a check-then-act race condition.

The fix is to change the order: first claim the work with a unique key, then make the external effect, and finally mark the result.

// idempotent: first "claim" with a unique key, then send, then mark
public async Task SendAsync(int orderId)
{
    string key = $"invoice-email:{orderId}";
    var email = new OutgoingEmail { IdempotencyKey = key, Status = EmailStatus.Sending, ClaimedAtUtc = DateTime.UtcNow };
    db.OutgoingEmails.Add(email);
    try
    {
        await db.SaveChangesAsync();
    }
    catch (DbUpdateException ex) when (ex.InnerException is SqlException { Number: 2601 or 2627 })
    {
        logger.LogInformation("{Key} already claimed by another run, skipping", key);
        return;
    }

    email.ProviderMessageId = await provider.SendAsync($"customer{orderId}@example.com", $"Invoice #{orderId}");
    email.Status = EmailStatus.Sent;
    await db.SaveChangesAsync();
    logger.LogInformation("{Key} sent as {MessageId}", key, email.ProviderMessageId);
}

claim-first, two concurrent runs for order 43:
05:12:16.790 info: Bookstore.Jobs.InvoiceEmailJob[0] invoice-email:43 already claimed by another run, skipping
05:12:17.011 info: Bookstore.Jobs.InvoiceEmailJob[0] invoice-email:43 sent as msg-4570d2f1
  emails delivered: 1

This time one email went out. The second run got SQL Server's duplicate key error 2601 on the INSERT and quietly left. (In both outputs I removed, for brevity, the fail-level log lines EF Core writes for these errors; logging a caught error at Error level can create noise in production.) You can also claim with the record the work already writes rather than a separate table: if you have a natural unique key such as an "invoice number", use it. For the transaction and concurrency mechanics in EF Core, see the Entity Framework Core series posts "Transactions in EF Core: SaveChanges, BeginTransaction and Savepoints" and "Concurrency in EF Core: Tokens, rowversion and DbUpdateConcurrencyException".

A Job That Claims and Dies, and Natural Idempotency

Claiming first has a price too. If the job writes the record as Sending and dies before sending the email, when the retry comes the record is already there and the job says "someone else claimed it" and leaves: the email never goes out. The naive approach carries the risk of two emails; this one carries the risk of zero. Which one you pick depends on the nature of the work: a second confirmation email is annoying, a second payment charge is unacceptable.

To close the zero-email risk, we put a claim time (ClaimedAtUtc) on the record. You can add a job or a retry step that reclaims Sending records older than a certain age (say 10 minutes) with a conditional UPDATE: WHERE Status = 'Sending' AND ClaimedAtUtc < @cutoff. The condition stops two retries from reclaiming at the same time. But before deciding whether to send again, ask the real question: did the external service receive the request? The sturdiest way to know is in the next section.

Some work, on the other hand, is idempotent by nature and needs no table. UPDATE Orders SET Status = 'Shipped' WHERE Id = 42 gives the same result however many times it runs; UPDATE Inventory SET Quantity = Quantity - 1 lowers stock by one on every run. Write jobs as "bring it to this value" (set, upsert) wherever possible, not "change it by this much" (increment). If you must increment, record the movement in a table with its own id and compute the total from there.

An Idempotency-Key for External APIs

If the external service can recognize a request by a key, idempotency is built end to end. Many payment and email providers support this with an Idempotency-Key header; the IETF (Internet Engineering Task Force) has a draft for the header (draft-ietf-httpapi-idempotency-key-header, version 08 published on 22 October 2026, not yet an RFC (Request for Comments)). If you keep the key the same on every attempt, the provider sees the second request not as new work but as a repeat of the first. To see the behavior, I wrote a small endpoint that imitates the provider:

app.MapPost("/provider/emails", async (HttpRequest request, EmailRequest body) =>
{
    string? key = request.Headers["Idempotency-Key"];
    if (key is not null && Seen.TryGetValue(key, out string? existing))
        return Results.Ok(new { messageId = existing, replayed = true });

    await Task.Delay(100);
    string messageId = $"msg-{Interlocked.Increment(ref _deliveries):D4}";
    if (key is not null)
        Seen[key] = messageId;
    return Results.Ok(new { messageId, replayed = false });
});

The job trying twice with the same key:

for (int attempt = 1; attempt <= 2; attempt++)
{
    using var request = new HttpRequestMessage(HttpMethod.Post, "/provider/emails")
    {
        Content = JsonContent.Create(new IdempotencyKeyDemo.EmailRequest("ada@example.com", "Invoice #42"))
    };
    request.Headers.Add("Idempotency-Key", "invoice-email:42");
    using HttpResponseMessage response = await http.SendAsync(request);
    Console.WriteLine($"attempt {attempt}: {(int)response.StatusCode} {await response.Content.ReadAsStringAsync()}");
}

attempt 1: 200 {"messageId":"msg-0001","replayed":false}
attempt 2: 200 {"messageId":"msg-0001","replayed":true}

The second request got the same messageId and the email didn't go out a second time. Derive the key from the identity of the work, not at random: invoice-email:42 is the same on every retry, on every worker, even if the job is queued again. Making Hangfire's job id the key may look tempting, but a job recreated by hand for the same invoice gets a different id. Also remember that providers keep keys only for a limited time; retries beyond that window can still produce duplicate work.

In Production

Choose the retry count and duration by the value of the work. The default 10 attempts, with a total of more than half a day, is too long for an order confirmation email: if the customer gets it the next day, they might as well not. For an order confirmation, 3 attempts over minutes makes sense; for a nightly report, longer intervals.

Put attempts into logs and metrics. The distribution of "which attempt succeeded" shows an external service's health very early; if the first-attempt success rate starts dropping, the problem is visible before it grows. In the tenth post we'll add the job id and attempt count to structured logs.

Clean up idempotency records. The claim table shouldn't grow forever; delete records past the idempotency window (say 30 days) with a periodic job. That job is idempotent itself: deleting the same records twice is harmless.

Common Pitfalls

"Check first, then act". Check-then-act does the work twice on two concurrent runs. Claim first with a unique key.

Retrying permanent errors. A validation error doesn't fix itself in ten attempts; it just gets noticed later. Use a separate exception type for permanent errors and ExceptOn.

Using retry instead of idempotency. Retry doesn't reduce double execution, it increases it; every retry runs the job from the start.

Regenerating the idempotency key on every attempt. A key generated with Guid.NewGuid() changes on retry and protects nothing.

Jobs that write increments. Relative changes such as Quantity = Quantity - 1 are applied again on every repeat. Write absolute values, or record the movement with its own id.

A Small Experiment

★☆☆ Remove DelaysInSeconds from SupplierInvoiceJobs.ImportAsync and change the job to fail on every attempt. Read the waits of the first three attempts from the log; do they match the formula?

★★☆ Add a step to InvoiceEmailJob that reclaims stale claims: claim a Sending record older than 10 minutes with a conditional ExecuteUpdateAsync. Test that only one of two concurrent runs claims it.

★★★ Combine InvoiceEmailJob with the fake provider: send invoice-email:{orderId} as the Idempotency-Key and kill the job right after the email is sent, before the status becomes Sent. On retry, what do the claim record and the provider do, and how many emails does the customer get?

In the next post we look at the most common source of double execution: when you scale the API to three replicas, the BackgroundService in each pod produces the nightly report three times. Distributed locks with SQL Server, PostgreSQL and Redis, leader election, and how long you really wait when the lock's owner dies.