Delivery Guarantees: At-Most-Once, At-Least-Once, and the Truth About "Exactly-Once"

Version note: The code in this post was built and run with C# 14 and .NET 10, against RabbitMQ 4.4 (a Docker container).

Guess First

Notification's worker takes confirmation email messages from a RabbitMQ queue. The rule looks simple and correct: send the email first, then ack the message. Let's set up the experiment; instead of an email provider we use a small class that writes what was sent into a list:

namespace Bookstore.Notifications;

// External side effect: an email, once sent, can't be taken back
public sealed class EmailSender
{
    public List<string> Sent { get; } = [];
    public void Send(string message) => Sent.Add(message);
}

using System.Text;
using Bookstore.Notifications;
using RabbitMQ.Client;
using RabbitMQ.Client.Exceptions;

ConnectionFactory factory = new() { HostName = "localhost" };
await using IConnection admin = await factory.CreateConnectionAsync();
// Publisher confirms on: SendAsync waits until the broker accepts the message
await using IChannel setup = await admin.CreateChannelAsync(
    new CreateChannelOptions(publisherConfirmationsEnabled: true, publisherConfirmationTrackingEnabled: true));
Dictionary<string, object?> quorum = new() { ["x-queue-type"] = "quorum" };

async Task<string> FreshQueueAsync(string name, Dictionary<string, object?> args)
{
    await setup.QueueDeleteAsync(name);
    await setup.QueueDeclareAsync(name, durable: true, exclusive: false, autoDelete: false, arguments: args);
    return name;
}

async Task SendAsync(string queue, string text) =>
    await setup.BasicPublishAsync("", queue, mandatory: false,
        new BasicProperties { MessageId = Guid.CreateVersion7().ToString(), DeliveryMode = DeliveryModes.Persistent },
        Encoding.UTF8.GetBytes(text));

The worker takes Alan's message, sends the email, and just as it's about to ack, the process crashes (we simulate that by dropping the connection with AbortAsync). The worker comes back up and reads the queue:

// At-least-once: work first, then ack. Crash before the ack and the message comes back.
{
    string queue = await FreshQueueAsync("eda04.at-least-once", quorum);
    await SendAsync(queue, "confirm order for Alan");
    EmailSender email = new();

    await using (IConnection consumer = await factory.CreateConnectionAsync())
    {
        IChannel channel = await consumer.CreateChannelAsync();
        BasicGetResult? got = await channel.BasicGetAsync(queue, autoAck: false);
        email.Send(Encoding.UTF8.GetString(got!.Body.Span));
        await consumer.AbortAsync();   // the email went out, the ack didn't
    }

    await using (IConnection consumer = await factory.CreateConnectionAsync())
    {
        IChannel channel = await consumer.CreateChannelAsync();
        BasicGetResult? got = await channel.BasicGetAsync(queue, autoAck: false);
        object? count = got!.BasicProperties.Headers?["x-delivery-count"];
        Console.WriteLine($"[at-least-once] redelivered: {got.Redelivered}, x-delivery-count: {count}");
        email.Send(Encoding.UTF8.GetString(got.Body.Span));
        await channel.BasicAckAsync(got.DeliveryTag, multiple: false);
    }
    Console.WriteLine($"[at-least-once] emails sent: {email.Sent.Count} -> {string.Join(" | ", email.Sent)}");
}

Does the broker deliver the message again? If it does, can the worker tell that this is a second delivery? How many emails does Alan get? And does flipping the rule, acking first and sending the email afterwards, solve the problem?

Three Guarantees, Two Sides

A delivery guarantee is a promise about how many times a message will be processed, and it comes in three levels. At-most-once: a message is processed at most once; it can be lost but isn't repeated. At-least-once: a message isn't lost but may be processed more than once. Exactly-once: each message is processed exactly once. Apache Kafka's design documentation stresses that this question is really two separate problems: the message the producer sends durably reaching the broker, and the consumer processing it. A chain is as strong as its weakest link; if the producer side is at-least-once and the consumer side at-most-once, the end-to-end guarantee is at-most-once.

On both sides, the guarantee is decided by the same thing: the order of the acknowledgement and the work.

The Consumer Side: Where Does the Ack Go?

The answer to the guess:

[at-least-once] redelivered: True, x-delivery-count: 1
[at-least-once] emails sent: 2 -> confirm order for Alan | confirm order for Alan

The broker delivered the message again, set the Redelivered flag to true, and the quorum queue showed the number of failed deliveries as 1 in the x-delivery-count header. Alan got two emails. That's at-least-once: the broker can't know whether a message it got no ack for was processed or not; it plays safe and redelivers. If the crash landed in the millisecond gap between the email and the ack, the result is a duplicate.

So is the Redelivered flag enough to catch duplicates? No. The flag says "this message was given to someone before", not "it was processed"; the worker could just as well have crashed before sending the email. And as we'll see shortly, some duplicates reach the broker as two separate messages, and for those the flag never lights up.

Let's flip the rule: ack first (here autoAck: true, meaning the message counts as acked the moment it's delivered), then the work:

// At-most-once: ack first (autoAck), then work. Crash in between and the message is lost.
{
    string queue = await FreshQueueAsync("eda04.at-most-once", quorum);
    await SendAsync(queue, "confirm order for Ada");
    EmailSender email = new();

    await using (IConnection consumer = await factory.CreateConnectionAsync())
    {
        IChannel channel = await consumer.CreateChannelAsync();
        BasicGetResult? got = await channel.BasicGetAsync(queue, autoAck: true);
        Console.WriteLine($"[at-most-once]  received: {Encoding.UTF8.GetString(got!.Body.Span)}");
        await consumer.AbortAsync();   // "crashed" before sending the email
    }
    Console.WriteLine($"[at-most-once]  emails sent: {email.Sent.Count}, left in queue: {await setup.MessageCountAsync(queue)}");
}

[at-most-once]  received: confirm order for Ada
[at-most-once]  emails sent: 0, left in queue: 0

The duplicate is gone, and the message went with it: Ada's email was never sent and the queue is empty. That's at-most-once. In Kafka the same logic applies: if the consumer commits its offset before the work, it's at-most-once; if it commits after the work, it's at-least-once.

Brokers keep track of "this message is with someone right now, but not done yet" in different ways, but the idea is the same. In RabbitMQ, an unacked message returns to the queue when the consumer's channel or connection closes. In Azure Service Bus's peek-lock mode, the message is locked for a while; the default lock duration is 1 minute, at most 5 minutes, and if the consumer doesn't call Complete before it expires, the message becomes visible again. In Amazon SQS (Simple Queue Service), the same thing is called the visibility timeout. Each has the same catch: if the work takes longer than the lock, a second consumer picks up the same message while the first is still working. That's the sneakiest way to produce a duplicate without any crash at all.

The Producer Side: Confirms and Retries

The producer's question is "did the broker get my message?". In RabbitMQ, the answer is publisher confirms: once confirms are enabled on a channel, the broker reports back when it has durably accepted or rejected the message. In RabbitMQ.Client 7, the task returned by BasicPublishAsync waits for that answer. mandatory: true means "if you can't route it to any queue, give it back to me":

// Producer side: publisher confirm + mandatory, the broker reports back if it can't route the message
{
    await using IChannel confirmed = await admin.CreateChannelAsync(
        new CreateChannelOptions(publisherConfirmationsEnabled: true, publisherConfirmationTrackingEnabled: true));
    try
    {
        await confirmed.BasicPublishAsync("", "eda04.no-such-queue", mandatory: true,
            new BasicProperties(), Encoding.UTF8.GetBytes("confirm order for Grace"));
        Console.WriteLine("[confirm]       published");
    }
    catch (PublishReturnException ex)
    {
        Console.WriteLine($"[confirm]       returned by broker: {ex.ReplyCode} {ex.ReplyText}");
    }
}

foreach (string q in new[] { "eda04.at-most-once", "eda04.at-least-once", "eda04.emails", "eda04.emails.dlq" })
    await setup.QueueDeleteAsync(q);

[confirm]       returned by broker: 312 NO_ROUTE

The broker took the message but found no queue to send it to, and returned it with 312 NO_ROUTE. Without confirms, this message would have disappeared silently: the exchange-without-a-bound-queue behavior we saw in the previous post. Azure Service Bus clients already always wait for the broker's acceptance on send. In Kafka, the producer's acks setting decides; the default is acks=all in the Java producer since Kafka 3.0 and in librdkafka, which .NET uses, meaning a record isn't considered successful until it's written to all in-sync replicas.

Confirms prevent loss but open a new question. The producer sent the message, the broker wrote it, but the confirm got lost on the network or timed out. What should the producer do? If it doesn't resend, the message may be lost; if it resends and the first one was actually written, the broker holds two copies. The safe choice is to resend, which makes the producer side at-least-once as well. That second copy enters the broker as a separate message; the Redelivered flag doesn't light up on the consumer side.

Some brokers try to close this source on their side. Kafka's idempotent producer (available since 0.11; on by default in the Java producer since 3.0, though because of a bug in 3.0 and 3.1 the default only really applies in the fixed releases; in librdkafka, underneath Confluent.Kafka, the default is still off and you turn it on with EnableIdempotence = true) gives each producer an id and sends records with sequence numbers; when the broker sees the same number a second time, it doesn't write it. Azure Service Bus's duplicate detection silently drops a second message arriving with the same MessageId within a window (10 minutes by default, at most 7 days). Both cover only the producer resending.

The Three Sources of Duplicates

Microsoft's Idempotent Consumer guidance groups the sources of duplicates under three headings, and the experiments above show all three:

• Producer retry: the send succeeded but the acknowledgement didn't arrive, and the producer sent again. The broker holds two separate messages.
• Lost ack: the consumer processed the message but the ack didn't reach the broker (the connection dropped, the lock expired). The broker redelivers the same message.
• Crash after processing: the consumer wrote to the database or sent the email, then crashed before acking. Alan's two emails are this one.

What the three have in common is that the other side can't know for certain the answer to "did it happen or not?". That this is a theoretical limit, that two parties can't reach certain agreement over an unreliable channel, is shown by the Two Generals problem, a topic for the upcoming Distributed Systems Fundamentals series. The practical consequence: between two parties talking over a network, if you don't want loss, you have to put up with duplicates.

The Truth About "Exactly-Once"

So what is the feature Kafka presents as "exactly-once semantics"? Kafka 0.11 brought two pieces: the idempotent producer and transactions. A transaction makes a producer's writes to multiple partitions atomic. More importantly, the consumer's offset can be written in the same transaction. In a read from Kafka, process, write to Kafka chain (exactly what Kafka Streams does), the read position and the output are committed together or rolled back together; readers using the read_committed isolation level don't see half-finished transactions.

That's where the line is drawn, and Kafka's own documentation says so plainly: exactly-once applies to operations the broker itself controls. If the output is an external system (a database, an email provider, a payment API (Application Programming Interface)), that system has to cooperate; the approach Kafka suggests is storing the consumer's offset together with the output in that system. No broker setting can send Alan's email "exactly once", because the email provider doesn't take part in Kafka's transaction.

Microsoft's guidance sums the situation up with a term: effectively-once. The broker delivers at least once, the consumer notices duplicates and neutralizes them; from the outside, the result looks as if each message was processed exactly once. The formula: at-least-once delivery + idempotent consumer = effectively-once. The second half of the formula is the subject of post 06 in the series.

Retries, Poison Messages, and Dead-Letter Queues

At-least-once has another face: some messages can never be processed. Malformed JSON (JavaScript Object Notation), an order for a book that no longer exists, a bug in the consumer. Retrying such a message doesn't help; retrying it forever clogs the queue and burns CPU (Central Processing Unit) for nothing. This is called a poison message. In EIP (Enterprise Integration Patterns) terms, the Dead Letter Channel is a separate place where messages that can't be delivered or processed are sent. Let's see the concept in a small experiment, with a small limit:

// Poison message: a message that can never be processed goes to the dead-letter queue once the limit is reached
{
    string dlq = await FreshQueueAsync("eda04.emails.dlq", quorum);
    string queue = await FreshQueueAsync("eda04.emails", new()
    {
        ["x-queue-type"] = "quorum",
        ["x-delivery-limit"] = 3,
        ["x-dead-letter-exchange"] = "",
        ["x-dead-letter-routing-key"] = dlq,
    });
    await SendAsync(queue, "{ not valid json");

    IChannel channel = await admin.CreateChannelAsync();
    int attempts = 0;
    while (await channel.BasicGetAsync(queue, autoAck: false) is { } got)
    {
        attempts++;
        await channel.BasicRejectAsync(got.DeliveryTag, requeue: true);   // "couldn't process it, try again"
    }
    uint dead = 0;
    for (int i = 0; i < 20 && dead == 0; i++)   // dead-lettering is asynchronous: wait briefly
    {
        await Task.Delay(100);
        dead = await setup.MessageCountAsync(dlq);
    }
    Console.WriteLine($"[poison]        attempts: {attempts}, main queue: {await setup.MessageCountAsync(queue)}, " +
                      $"dead-letter queue: {dead}");
}

[poison]        attempts: 4, main queue: 0, dead-letter queue: 1

The message was delivered once, retried three times, and moved to the dead-letter queue when the limit was reached. The main queue is empty; the messages behind it are no longer stuck waiting. I also saw in the experiment that the move is asynchronous: on the first run the counter showed the dead-letter queue as empty, which is why the code waits briefly.

Defaults matter. Since RabbitMQ 4.0, the delivery limit on quorum queues defaults to 20; before that it was unlimited. If no dead-letter target is defined, a message that exceeds the limit is deleted, which is why the RabbitMQ documentation recommends setting up dead-lettering for every quorum queue. The counter is subtle too: since RabbitMQ 4.3, only genuine failures (basic.reject, a dropped connection, a crash) count toward the limit; basic.nack doesn't. In Azure Service Bus, MaxDeliveryCount does the same job (its default is 10). The details of delayed retries, backoff, and dead-letter configuration are in the upcoming Backend › Working with Message Queues series.

What matters architecturally is that a dead-letter queue is not a trash can but a work list. Every message sitting there is an email a customer didn't get or a shipment that wasn't prepared. Someone has to watch it, find the cause, and resend after fixing it. A resent message is, by definition, a duplicate candidate too; that's where the question of how long an idempotent consumer should keep its dedup records comes from.

Which Guarantee Should You Choose?

Most of Bookstore's messages are in the second and third rows. A practical rule: start with at-least-once, ask "what happens if this message arrives twice?" for every consumer, and make the ones whose answer is "it does harm" idempotent.

Common Pitfalls

Turning on autoAck for convenience

autoAck: true shows up often in samples and early prototypes because the code is shorter. But it means accepting that the consumer loses the messages it holds every time it crashes. Unless you chose that deliberately, use manual acks and send the ack after the work is done.

Not sizing the lock to the work

In Azure Service Bus, a consumer doing 90 seconds of work with the default 1-minute lock processes every message twice: the lock drops, the message goes to another consumer, and the first consumer's Complete call fails too because the lock is gone. No crash, an error in the log, and a duplicate. Keep the lock duration above the normal processing time, and renew the lock for long work.

Trusting the exactly-once flag

Turning on transactions in Kafka doesn't deduplicate the rows a consumer writes to a database or the emails it sends. When you see an "exactly-once" setting, ask: which operations are inside this promise, and which are outside it?

Infinite retries and an unwatched dead-letter queue

Requeuing without a limit means a single malformed message keeps a consumer busy for hours. If there is a limit but nobody watches the dead-letter queue, the problem just becomes invisible. Dead-letter queue depth is one of the first architectural signals we'll talk about in the last post of the series.

A Small Experiment

★☆☆ In the poison message experiment, use BasicNackAsync(got.DeliveryTag, multiple: false, requeue: true) instead of BasicRejectAsync. After how many attempts does the loop end? Why, according to the RabbitMQ 4.3+ rule? (Don't forget to put an upper bound on the loop.)

★★☆ In the at-least-once experiment, make the second consumer notice through the MessageId that it's processing the same message a second time: keep processed ids in an in-memory HashSet<string>. Why doesn't this solution work once the worker restarts?

★★★ Simulate a producer-side duplicate: send the same content twice, each time with a new MessageId. How can the consumer tell these two messages apart? What would change if the MessageId weren't random but derived from the business flow (such as the order id + the message type)?

The at-least-once promise says nothing about order; worse, a redelivered message can be processed after the messages behind it. In the next post we'll look at where ordering guarantees begin and end, and how partition keys and sessions preserve order.