Idempotent Consumers: What Happens When the Same Message Arrives Twice?

Version note: The code in this post was built and run with C# 14, .NET 10, and Entity Framework Core (EF Core) 10, against Structured Query Language (SQL) Server 2025.

Guess First

When OrderPlaced arrives, Inventory reserves stock for the books in the order. We saw two posts ago that the broker delivers at least once, so the same message can arrive twice. Here are Inventory's database and consumer:

using Microsoft.EntityFrameworkCore;

namespace Bookstore.Inventory;

public sealed class StockItem
{
    public Guid BookId { get; init; }
    public required string Title { get; init; }
    public int OnHand { get; set; }
    public int Reserved { get; set; }
}

// Inbox: a record of the messages each consumer has processed
public sealed class InboxMessage
{
    public required string MessageId { get; init; }
    public required string Consumer { get; init; }
    public DateTimeOffset ReceivedAt { get; init; }
    public DateTimeOffset? CompletedAt { get; set; }   // null: in progress (two-phase record)
}

public sealed class InventoryDbContext(DbContextOptions<InventoryDbContext> options) : DbContext(options)
{
    public DbSet<StockItem> StockItems => Set<StockItem>();
    public DbSet<InboxMessage> Inbox => Set<InboxMessage>();

    protected override void OnModelCreating(ModelBuilder model)
    {
        model.Entity<StockItem>().HasKey(s => s.BookId);
        model.Entity<InboxMessage>(inbox =>
        {
            // The database is the arbiter: the same consumer can't write the same message twice
            inbox.HasKey(m => new { m.MessageId, m.Consumer });
            inbox.Property(m => m.MessageId).HasMaxLength(100);
            inbox.Property(m => m.Consumer).HasMaxLength(100);
        });
    }
}

using Microsoft.Data.SqlClient;
using Microsoft.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore.Storage;

namespace Bookstore.Inventory;

public sealed record OrderPlacedMessage(string MessageId, Guid OrderId, Guid BookId, int Quantity);

// Reserves stock on every delivery
public sealed class NaiveReserveStock(InventoryDbContext db)
{
    public Task HandleAsync(OrderPlacedMessage m) =>
        db.StockItems.Where(s => s.BookId == m.BookId)
            .ExecuteUpdateAsync(s => s.SetProperty(x => x.Reserved, x => x.Reserved + m.Quantity));
}

using System.Diagnostics;
using Bookstore.Inventory;
using Microsoft.EntityFrameworkCore;

string connectionString = Environment.GetEnvironmentVariable("BOOKSTORE_SQL")
    ?? "Server=localhost;Database=eda_06;Trusted_Connection=True;TrustServerCertificate=True";
DbContextOptions<InventoryDbContext> options = new DbContextOptionsBuilder<InventoryDbContext>()
    .UseSqlServer(connectionString).Options;
TimeProvider clock = TimeProvider.System;
Guid dune = Guid.CreateVersion7();

await using (InventoryDbContext db = new(options))
{
    await db.Database.EnsureDeletedAsync();
    await db.Database.EnsureCreatedAsync();
}

async Task<int> ResetAsync()
{
    await using InventoryDbContext db = new(options);
    await db.Inbox.ExecuteDeleteAsync();
    await db.StockItems.ExecuteDeleteAsync();
    db.StockItems.Add(new StockItem { BookId = dune, Title = "Dune", OnHand = 10 });
    return await db.SaveChangesAsync();
}

async Task<int> ReservedAsync()
{
    await using InventoryDbContext db = new(options);
    return await db.StockItems.Where(s => s.BookId == dune).Select(s => s.Reserved).SingleAsync();
}

OrderPlacedMessage message = new(MessageId: Guid.CreateVersion7().ToString(), OrderId: Guid.CreateVersion7(), BookId: dune, Quantity: 2);

Ada ordered two copies of Dune; the message was processed once, the ack got lost on the way, and the broker delivered the same message again:

// Naive: the same message twice
await ResetAsync();
for (int delivery = 0; delivery < 2; delivery++)
{
    await using InventoryDbContext db = new(options);
    await new NaiveReserveStock(db).HandleAsync(message);
}
Console.WriteLine($"[naive]              delivered twice -> Dune reserved: {await ReservedAsync()}");

How many copies of Dune will show as reserved? Is adding an if that asks "have I seen this message before?" enough to fix it? What happens if two consumer instances get two copies of the same message in the same millisecond?

The Cost of a Duplicate

[naive]              delivered twice -> Dune reserved: 4

A two-book order reserved four copies. Two copies of Dune sitting in the warehouse now look unsellable; at the campaign peak that means orders lost to "out of stock". As we saw in the previous post, a stock difference (Reserved + 2) is commutative but not idempotent: applied twice, the result changes.

Microsoft's Idempotent Consumer pattern in the Azure Architecture Center describes the solution like this: the consumer keeps a record of the messages it has successfully processed and skips a message when it sees it again. The description is short, but it holds four design decisions: what we use as the key, where we keep the record, how we write the record together with the work, and who wins when two copies arrive at once. First, let's take a step back.

Natural Idempotency First

Some operations are idempotent on their own: an upsert keyed on a business identifier, an update that writes an absolute value ("status = cancelled", "stock = 8"), an HTTP (Hypertext Transfer Protocol) PUT. However many times you apply them, the result is the same. Microsoft's guidance recommends this first too: use dedup records only for operations that can't be made naturally idempotent.

The problem in Inventory is that we write the operation as a difference. In the model from the Domain-Driven Design series, stock reservation was StockItem.TryReserve(OrderId, qty) and was idempotent by order id: a reservation was kept as one record per order. Said in database terms, a StockReservations table that is unique on (OrderId, BookId) moves the dedup record into the business data itself: a second reservation for the same order can't get into the table. That's the second of the two options in Microsoft's guidance, keeping dedup on the business entity. The upside is no separate table; the price is designing the dedup logic separately for every message type.

Not every operation has such a natural key. An email, a notification, or work that touches several tables needs a general mechanism: a separate dedup table, the inbox. The rest of this post designs it.

The Dedup Key: What Do We Remember?

The key must identify the same logical message the same way on every redelivery. There are two candidates: a message id assigned by the producer, or a business key that identifies the operation (an idempotency key). The MessageId in the envelope from post 02 of the series is the first candidate. Two warnings apply.

A correlation id is not a dedup key. A correlation id identifies a business flow and is the same across every message in that flow; make it the key and, once the same order's OrderPlaced is processed, its OrderCancelled is dropped as a "duplicate". If you use CloudEvents, by the specification's definition it's the combination of source and id that uniquely identifies an event.

Values generated by the broker are not dedup keys either. RabbitMQ's delivery tag, Azure Service Bus's lock token, the delivery time in Kafka all change on every redelivery. The Redelivered flag from post 04 of the series isn't enough either: a copy the producer resends enters the broker as a separate message and the flag doesn't light up. The question from that post's last exercise ties in here: if the producer generates the MessageId randomly on every send, a producer retry produces two messages with two different ids, and no consumer can match them. The id should be generated once before sending and stored (we'll do that in the series' outbox post), or derived from the business.

Check-Then-Process: The Failure Window

The first solution that comes to mind is to look before processing and record after processing:

// Check first, then process, then record: three separate steps
public sealed class CheckThenProcessReserveStock(InventoryDbContext db, TimeProvider clock)
{
    public const string Name = "inventory.reserve-stock";
    public Func<Task> AfterStockCommitted { get; init; } = () => Task.CompletedTask;

    public async Task HandleAsync(OrderPlacedMessage m)
    {
        if (await db.Inbox.AnyAsync(x => x.MessageId == m.MessageId && x.Consumer == Name)) return;

        await db.StockItems.Where(s => s.BookId == m.BookId)    // commits in its own transaction
            .ExecuteUpdateAsync(s => s.SetProperty(x => x.Reserved, x => x.Reserved + m.Quantity));
        await AfterStockCommitted();                              // what if the process crashes here?

        db.Inbox.Add(new() { MessageId = m.MessageId, Consumer = Name, ReceivedAt = clock.GetUtcNow(), CompletedAt = clock.GetUtcNow() });
        await db.SaveChangesAsync();
    }
}

// Check-then-process: stock committed, crashed before the dedup record was written, message redelivered
await ResetAsync();
try
{
    await using InventoryDbContext db = new(options);
    await new CheckThenProcessReserveStock(db, clock) { AfterStockCommitted = () => throw new Exception("crash") }
        .HandleAsync(message);
}
catch (Exception) { }
await using (InventoryDbContext db = new(options)) await new CheckThenProcessReserveStock(db, clock).HandleAsync(message);
Console.WriteLine($"[check-then-process] crash after stock commit, redelivered -> Dune reserved: {await ReservedAsync()}");

[check-then-process] crash after stock commit, redelivered -> Dune reserved: 4

There was a check, but the result is still four. The stock update committed in its own transaction via ExecuteUpdateAsync, and the process crashed just as it was about to write the dedup record. On redelivery, the check said "I haven't seen this message", because there was no record. Microsoft's guidance calls this gap the check-then-process failure window: any crash between the work and the record leaves the record unwritten and the work repeated. Flipping the order (record first, then work) doesn't close the gap, it only changes its direction: a crash then means the work is never done, which is at-most-once.

The Inbox: Record and Work in the Same Transaction

The way to close the gap is to write the dedup record and the business data in the same transaction. Both commit together or neither does; there's no moment left in between to crash in:

// Inbox: the dedup record and the work in one transaction, the key is claimed first
public sealed class IdempotentReserveStock(InventoryDbContext db, TimeProvider clock)
{
    public const string Name = "inventory.reserve-stock";
    public Func<Task> BeforeCommit { get; init; } = () => Task.CompletedTask;

    public async Task<bool> HandleAsync(OrderPlacedMessage m, CancellationToken ct = default)
    {
        await using IDbContextTransaction tx = await db.Database.BeginTransactionAsync(ct);
        try
        {
            DateTimeOffset now = clock.GetUtcNow();
            db.Inbox.Add(new() { MessageId = m.MessageId, Consumer = Name, ReceivedAt = now, CompletedAt = now });
            await db.SaveChangesAsync(ct);
        }
        catch (DbUpdateException ex) when (ex.InnerException is SqlException { Number: 2627 or 2601 })
        {
            return false;   // duplicate: another delivery already processed this message
        }

        await db.StockItems.Where(s => s.BookId == m.BookId)
            .ExecuteUpdateAsync(s => s.SetProperty(x => x.Reserved, x => x.Reserved + m.Quantity), ct);
        await BeforeCommit();
        await tx.CommitAsync(ct);   // both or neither
        return true;
    }
}

The order is deliberate: as soon as the transaction opens, the first thing is to write the inbox row. That means "claiming" the key; we'll see shortly why that matters for concurrent duplicates. If the row can't be written (a primary key violation), the message has already been processed and the consumer returns without doing anything; the broker still gets an ack, because skipping a duplicate is a successful outcome. Two deliveries, and a crash before the commit:

// Inbox: two deliveries
await ResetAsync();
List<bool> results = [];
for (int delivery = 0; delivery < 2; delivery++)
{
    await using InventoryDbContext db = new(options);
    results.Add(await new IdempotentReserveStock(db, clock).HandleAsync(message));
}
Console.WriteLine($"[inbox]              delivered twice -> processed: {string.Join(", ", results)}; Dune reserved: {await ReservedAsync()}");

[inbox]              delivered twice -> processed: True, False; Dune reserved: 2

// Inbox: crashed before commit, message redelivered
await ResetAsync();
try
{
    await using InventoryDbContext db = new(options);
    await new IdempotentReserveStock(db, clock) { BeforeCommit = () => throw new Exception("crash") }.HandleAsync(message);
}
catch (Exception) { }
int afterCrash = await ReservedAsync();
await using (InventoryDbContext db = new(options)) await new IdempotentReserveStock(db, clock).HandleAsync(message);
Console.WriteLine($"[inbox]              crash before commit -> reserved {afterCrash}; redelivered -> Dune reserved: {await ReservedAsync()}");

[inbox]              crash before commit -> reserved 0; redelivered -> Dune reserved: 2

The second delivery was skipped as a duplicate and the stock stayed at two. In the crash before the commit, both the inbox row and the stock update were rolled back: stock zero, no record. The redelivered message was processed with a clean slate. In the guidance's words, this is the Inbox pattern, the consumer-side companion of the Transactional Outbox. It's essential here that both live in the same SQL Server database; if the inbox table sits in another database or in a cache, it can't join the same transaction and the failure window comes back.

Concurrent Duplicates: The Unique Constraint as Arbiter

With competing consumers, two copies of the same message can reach two instances at the same time: for example, when a lock expires, a second instance picks up the message while the first is still working. When both look to see "have I seen this message?", the answer is no, because neither has committed yet. A check in application code can't settle this race. The solution is to leave the decision to the database:

// Inbox: two copies of the same message at once in two consumer instances
await ResetAsync();
async Task<(bool Processed, long Ms)> RunAsync(TimeSpan holdBeforeCommit)
{
    Stopwatch watch = Stopwatch.StartNew();
    await using InventoryDbContext db = new(options);
    bool processed = await new IdempotentReserveStock(db, clock) { BeforeCommit = () => Task.Delay(holdBeforeCommit) }
        .HandleAsync(message);
    return (processed, watch.ElapsedMilliseconds);
}
Task<(bool Processed, long Ms)> first = RunAsync(TimeSpan.FromMilliseconds(500));
await Task.Delay(100);   // the second copy arrives while the first is inside its transaction
Task<(bool Processed, long Ms)> second = RunAsync(TimeSpan.Zero);
await Task.WhenAll(first, second);
Console.WriteLine($"[inbox]              two copies at once -> processed: {first.Result.Processed}, {second.Result.Processed}; " +
                  $"Dune reserved: {await ReservedAsync()}; second waited for the first: {second.Result.Ms >= 300}");

[inbox]              two copies at once -> processed: True, False; Dune reserved: 2; second waited for the first: True

The first copy wrote the inbox row and waited half a second before committing. The second copy's INSERT waited on the same primary key; SQL Server blocked the second transaction until the first one's outcome was known. Once the first committed, the second's insert was rejected with error 2627, the consumer read that as a duplicate, and the stock stayed at two. The error's text in SQL Server:

Msg 2627, Level 14, State 1
Violation of PRIMARY KEY constraint 'PK_Inbox'. Cannot insert duplicate key in object 'dbo.Inbox'. The duplicate key value is (m-1, c).

This is where claiming the key as the transaction's first step pays off: the second copy stopped without touching the stock and held no other rows locked while it waited. Had the first transaction rolled back, the second's insert would have succeeded and it would have processed the message. In a system where duplicates are frequent, a cheap AnyAsync check before the transaction can be added to avoid the cost of the exception, but that's only an optimization; what guarantees correctness is the constraint. The same principle applies to cache-based dedup: instead of checking "does it exist?" and then saying "write", use a single atomic operation such as set-if-absent.

Side Effects That Can't Join the Transaction

Not every side effect goes into the database transaction. Sending an email, calling the payment provider, writing to another system's API (Application Programming Interface) can't be rolled back along with a commit. Microsoft's guidance suggests a two-phase record for these: first record the key as "in progress", do the external operation, then mark it "completed".

using Microsoft.EntityFrameworkCore;

namespace Bookstore.Inventory;

public enum EmailOutcome { Sent, AlreadySent, NeedsReconciliation }

public sealed class EmailProvider
{
    public List<string> Sent { get; } = [];
    // With real providers, this id can be used to block a resend or to query the status
    public Task SendAsync(string idempotencyKey, string text) { Sent.Add($"{idempotencyKey}: {text}"); return Task.CompletedTask; }
}

// A side effect that can't join the database transaction: a two-phase record
public sealed class SendStockReservedEmail(InventoryDbContext db, EmailProvider provider, TimeProvider clock)
{
    public const string Name = "notifications.stock-reserved-email";
    public Func<Task> AfterSend { get; init; } = () => Task.CompletedTask;

    public async Task<EmailOutcome> HandleAsync(OrderPlacedMessage m)
    {
        InboxMessage? entry = await db.Inbox.SingleOrDefaultAsync(x => x.MessageId == m.MessageId && x.Consumer == Name);
        if (entry is { CompletedAt: not null }) return EmailOutcome.AlreadySent;
        if (entry is not null) return EmailOutcome.NeedsReconciliation;   // a previous attempt may have been left half-done

        entry = new() { MessageId = m.MessageId, Consumer = Name, ReceivedAt = clock.GetUtcNow() };
        db.Inbox.Add(entry);
        await db.SaveChangesAsync();                                         // phase 1: "in progress"

        await provider.SendAsync(m.MessageId, $"Order {m.OrderId}: {m.Quantity} book(s) reserved");
        await AfterSend();

        entry.CompletedAt = clock.GetUtcNow();
        await db.SaveChangesAsync();                                         // phase 2: "completed"
        return EmailOutcome.Sent;
    }
}

// External side effect: the email was sent, crashed before "completed" was written
await ResetAsync();
EmailProvider provider = new();
try
{
    await using InventoryDbContext db = new(options);
    await new SendStockReservedEmail(db, provider, clock) { AfterSend = () => throw new Exception("crash") }.HandleAsync(message);
}
catch (Exception) { }
EmailOutcome redelivered;
await using (InventoryDbContext db = new(options)) redelivered = await new SendStockReservedEmail(db, provider, clock).HandleAsync(message);
Console.WriteLine($"[email]              crash after send, redelivered -> emails sent: {provider.Sent.Count}, second delivery: {redelivered}");

[email]              crash after send, redelivered -> emails sent: 1, second delivery: NeedsReconciliation

The email went out, and the process crashed before "completed" could be written. The redelivered message found a record in the "in progress" state in the inbox, and instead of blindly sending the email again it returned NeedsReconciliation. That's the right answer, because the consumer can't know where the previous attempt stopped: the email may have gone out, may not have, or another instance may be sending it right now. The best way to reconcile is the external system's own idempotency support: many payment and email providers accept an idempotency key and don't process a second request with the same key again, or let you query its outcome. The Solution Patterns series post "Idempotency Keys: Not Creating the Same Order Twice on Double Clicks and Retries" covered the server side of this mechanism. The guidance has one more suggestion: if the consumer calls other services, it should pass the idempotency key on to them so each tier can deduplicate its own work. Records that can't be reconciled should go somewhere a person will look, such as a dead-letter queue.

Retention, Pub/Sub, and the Broker's Own Dedup

The inbox table shouldn't grow forever, but it shouldn't be pruned too early either. The guidance's rule: keep a record longer than the period during which the broker can redeliver the original message. That period depends on the broker's maximum delivery attempts, the lock or visibility timeout, and the message's time-to-live. The item people often forget is the dead-letter queue: a message an operator resends a week later is a duplicate candidate too. If Bookstore's dead-letter queues are dealt with within a few days at most, keeping inbox records for, say, 14 days and deleting older ones with a nightly ExecuteDeleteAsync is a reasonable start.

Why is the inbox key (MessageId, Consumer)? Because in publish/subscribe every subscriber gets its own copy of the same message. If Inventory and Notification share the same database and the same inbox table, a key on the message id alone lets the first consumer to process it silence the others: Inventory reserves the stock, and Notification's email is dropped as a "duplicate".

Finally, the broker's own dedup. Azure Service Bus's duplicate detection drops a second message with the same MessageId within a window of at most 7 days, and that's useful against the producer resending. But it works on the send side and has a bounded window; it doesn't stop the consumer from processing the same message a second time on redelivery. Microsoft's guidance says so plainly: platform features are there to reduce the number of duplicates, not to replace the idempotent consumer. Frameworks such as NServiceBus and MassTransit offer this pattern ready-made; the full EF Core implementation of the inbox and its cleanup is the subject of the upcoming Backend › Working with Message Queues series.

Common Pitfalls

Keeping the dedup record in a cache and the work in the database

Writing dedup keys to a cache "for speed" splits the record and the work across two systems; the check-then-process gap comes back. A cache can be a pre-filter, but it can't be the arbiter.

Treating a duplicate as an error

Throwing an exception and not acking when a duplicate is detected makes the broker deliver it again and again until it lands in the dead-letter queue. A duplicate is a successfully skipped message: ack it and move on. Different content arriving with the same id is something else; Microsoft's guidance suggests sending that message to the dead-letter queue and raising an alert instead of silently discarding it.

A second SaveChanges or another connection inside the handler

The inbox's guarantee rests on a single transaction. Writing from inside the handler through another DbContext, another connection, or an operation running outside the transaction takes that write outside the guarantee. In the Domain-Driven Design series post "Domain Events: Raise, Collect, and Dispatch on SaveChanges" we saw how a nested SaveChanges breaks the single-transaction promise; the same trap is here.

A random MessageId

If the producer generates a new id on every retry, this whole consumer-side mechanism is blind to producer duplicates. The id should be generated once, when the message is first created, and stay the same across retries.

A Small Experiment

★☆☆ Make the inbox key just MessageId and process the same message first with IdempotentReserveStock, then with a second consumer class whose Name constant is different. What does the second consumer do?

★★☆ In IdempotentReserveStock, write the inbox row after the stock update (claim the key last). Does the concurrent duplicate scenario still give the right result? Which row does the second copy wait on, and why does this order hold more locks?

★★★ Add a StockReservations table for Inventory that is unique on (OrderId, BookId), and make stock reservation naturally idempotent without the inbox. If the same order arrives with two different MessageId values (a producer duplicate), how do the solutions with and without the inbox behave?

The inbox closed the dual write on the consumer side: the dedup record and the work in one transaction. On the producer side the same problem is still open: Ordering commits the order to the database, then publishes OrderPlaced to the broker. What if the commit succeeds and the publish fails? In the next post we'll close that gap with the transactional outbox.