Managing Dead-Letter Queues: Finding, Inspecting and Reprocessing Poison Messages

Version note: The code in this post was built and run with .NET 10, RabbitMQ 4.4, RabbitMQ.Client 7.2.2, Azure.Messaging.ServiceBus 7.21 (Service Bus emulator), Confluent.Kafka 2.16 (Kafka 4.2) and MassTransit 8.5.12.

Guess First

Billing's DLQ (Dead-Letter Queue) holds one message; the x-death header RabbitMQ added says "rejected once from billing.order-placed". You believe you have fixed the bug and republish the message to the main queue, but the fix was incomplete and the message is rejected again and comes back to the DLQ. What does the counter in x-death show now: 1 or 2? Can your replay tool decide "I have tried this message three times, that is enough" by looking at that counter?

In the previous post we built retry and saw that some messages will reach the end of the road in the DLQ. In this post we treat the DLQ as a work queue rather than a trash can: why did the message fail, who will notice, and how do we safely send it back after the fix? The concepts of poison messages and dead-lettering are covered in the Messaging and Event-Driven Architecture series; here we write code for three brokers and MassTransit.

RabbitMQ: DLX, Delivery Limit and x-death

In RabbitMQ a DLQ is an ordinary queue; what makes it a DLQ is the DLX (Dead Letter Exchange) setting on the main queue. A message is dead-lettered in four cases: a reject or nack with requeue: false, an expired message TTL (Time-To-Live), an exceeded queue length limit, and an exceeded quorum queue delivery limit. Billing's topology:

await channel.QueueDeclareAsync("billing.order-placed", durable: true, exclusive: false, autoDelete: false,
    arguments: new Dictionary<string, object?>
    {
        ["x-queue-type"] = "quorum",
        ["x-delivery-limit"] = 5,
        ["x-dead-letter-exchange"] = "bookstore.dlx",
        ["x-dead-letter-routing-key"] = "billing.order-placed",
        // at-least-once dead-lettering: the message is not removed until the DLQ copy is confirmed
        ["x-dead-letter-strategy"] = "at-least-once",
        ["x-overflow"] = "reject-publish"
    });
await channel.QueueBindAsync("billing.order-placed", "bookstore.ordering", "order.placed");

await channel.QueueDeclareAsync("billing.order-placed.dlq", durable: true, exclusive: false, autoDelete: false,
    arguments: new Dictionary<string, object?>
    {
        ["x-queue-type"] = "quorum",
        ["x-message-ttl"] = (int)TimeSpan.FromDays(14).TotalMilliseconds,
        ["x-max-length"] = 100_000
    });
await channel.QueueBindAsync("billing.order-placed.dlq", "bookstore.dlx", "billing.order-placed");

By default dead-lettering is at-most-once: the broker sends the message to the DLX without internal publisher confirms, and if the target queue is unavailable at that moment the message is lost. On quorum queues, x-dead-letter-strategy = at-least-once closes that gap; according to the RabbitMQ documentation it requires x-overflow to be reject-publish, and with the default drop-head the strategy silently falls back to at-most-once. We also put two limits on the DLQ itself: a 14-day TTL and a cap of 100,000 messages. We will discuss why in "Production Notes".

I published three messages. The consumer treats msg-1 and msg-3 as permanent failures and rejects them with requeue: false, while it hands msg-2 back with requeue: true every time. A small helper that reads the DLQ summarizes the headers like this:

msg-1: first-death=rejected@billing.order-placed deaths=[billing.order-placed/rejected x1] replay-count=0
msg-3: first-death=rejected@billing.order-placed deaths=[billing.order-placed/rejected x1] replay-count=0
msg-2: first-death=delivery_limit@billing.order-placed deaths=[billing.order-placed/delivery_limit x1] replay-count=0

On the first dead-lettering the broker adds six headers: x-first-death-queue, x-first-death-reason, x-first-death-exchange and their x-last-death-* counterparts. x-death is an array with the most recent entry first; each element has queue, reason, count, time, exchange and routing-keys. The reason is one of four values: rejected, expired, maxlen, delivery_limit. That distinction is the basis of the replay decision: rejected means "the consumer rejected this message on purpose", while delivery_limit means "the consumer struggled with this message again and again and never reached a verdict". The latter usually points to a bug in the consumer itself or a message that crashes the process.

There is a trap when reading the headers in .NET: strings in AMQP (Advanced Message Queuing Protocol) 0-9-1 headers reach the client as byte[], not string:

public sealed record DeadLetterInfo(
    string MessageId, string? FirstDeathReason, string? FirstDeathQueue, IReadOnlyList<string> Deaths, int ReplayCount)
{
    public static DeadLetterInfo From(IReadOnlyBasicProperties props)
    {
        IDictionary<string, object?> headers = props.Headers ?? new Dictionary<string, object?>();
        var deaths = new List<string>();
        if (headers.TryGetValue("x-death", out object? raw) && raw is IList list)
        {
            foreach (IDictionary<string, object?> death in list.OfType<IDictionary<string, object?>>())
                deaths.Add($"{Text(death["queue"])}/{Text(death["reason"])} x{death["count"]}");
        }
        int replayCount = headers.TryGetValue("x-replay-count", out object? rc) && rc is int n ? n : 0;
        return new DeadLetterInfo(props.MessageId ?? "?", Text(Get(headers, "x-first-death-reason")),
            Text(Get(headers, "x-first-death-queue")), deaths, replayCount);
    }

    private static object? Get(IDictionary<string, object?> headers, string key) =>
        headers.TryGetValue(key, out object? value) ? value : null;

    // strings in AMQP 0-9-1 headers reach the .NET client as byte[]
    private static string? Text(object? value) => value switch
    {
        byte[] bytes => Encoding.UTF8.GetString(bytes),
        null => null,
        _ => value.ToString()
    };
}

Azure Service Bus: Every Entity Has Its Own DLQ

In Service Bus you do not create the DLQ; every queue and every subscription has its own dead-letter sub-queue, which cannot be deleted separately. The system writes its own reason codes (MaxDeliveryCountExceeded, TTLExpiredException, HeaderSizeExceeded, Session ID is null, MaxTransferHopCountExceeded), but a consumer that recognizes a permanent failure and sends the message with its own reason leaves a far more readable trail:

ServiceBusReceiver receiver = client.CreateReceiver("orders");
ServiceBusReceivedMessage message = (await receiver.ReceiveMessageAsync(TimeSpan.FromSeconds(5)))!;
await receiver.DeadLetterMessageAsync(message,
    deadLetterReason: "UnsupportedCurrency",
    deadLetterErrorDescription: "Currency XXX is not supported by the payment provider");

To read the DLQ you do not build a separate path; you open the receiver with SubQueue.DeadLetter:

static async Task PrintDeadLettersAsync(ServiceBusClient client)
{
    ServiceBusReceiver dlq = client.CreateReceiver("orders", new ServiceBusReceiverOptions { SubQueue = SubQueue.DeadLetter });
    IReadOnlyList<ServiceBusReceivedMessage> messages = await dlq.PeekMessagesAsync(maxMessages: 10);
    foreach (ServiceBusReceivedMessage m in messages)
    {
        Console.WriteLine($"DLQ {m.MessageId} DeliveryCount={m.DeliveryCount} DeadLetterSource={m.DeadLetterSource ?? "-"}");
        Console.WriteLine($"    DeadLetterReason={m.DeadLetterReason}");
        Console.WriteLine($"    DeadLetterErrorDescription={m.DeadLetterErrorDescription}");
    }
}

Output of two separate runs in the emulator; the first is the end of the abandon loop from the previous post, the second is the manual dead-letter above:

DLQ msg-A DeliveryCount=3 DeadLetterSource=-
    DeadLetterReason=MaxDeliveryCountExceeded
    DeadLetterErrorDescription=Message could not be consumed after 3 delivery attempts.

DLQ msg-A DeliveryCount=0 DeadLetterSource=-
    DeadLetterReason=UnsupportedCurrency
    DeadLetterErrorDescription=Currency XXX is not supported by the payment provider

PeekMessagesAsync neither locks nor removes the message; that is the right choice for an inspection tool. Microsoft recommends putting the exception type in DeadLetterReason and the stack trace in the description; however, on the Standard tier the message size limit is 256 KB, so a long stack trace can push the message over it. Two more important differences: TTL is not applied in the DLQ, and there is no automatic cleanup. A message stays there until you receive and complete it. The easiest way to resubmit is Service Bus Explorer in the Azure portal; in code you copy the message received from the DLQ with new ServiceBusMessage(received), send it to the main entity and then complete the message in the DLQ (I did not run this replay code against the emulator; it follows the same order as the RabbitMQ tool below).

Kafka: Dead-Letter Topic

Kafka has no broker-side concept of dead-lettering. The router from the previous post wrote the message to the bookstore.orders.billing.dlt topic once its attempts ran out. An inspection tool that reads the DLT (dead-letter topic) from the beginning without committing prints the headers:

var config = new ConsumerConfig
{
    BootstrapServers = "localhost:17311",
    GroupId = "billing-dlt-inspector",
    AutoOffsetReset = AutoOffsetReset.Earliest,
    EnableAutoCommit = false
};
using IConsumer<string, byte[]> consumer = new ConsumerBuilder<string, byte[]>(config).Build();
consumer.Subscribe("bookstore.orders.billing.dlt");
while (consumer.Consume(TimeSpan.FromSeconds(10)) is { } result)
{
    Console.WriteLine($"{result.TopicPartitionOffset} key={result.Message.Key[..8]}");
    foreach (IHeader header in result.Message.Headers)
        Console.WriteLine($"    {header.Key}: {Encoding.UTF8.GetString(header.GetValueBytes())}");
}
consumer.Close();

In my first attempt the router wrote result.Topic into a header called dlt-original-topic on the way to the DLT. The output was:

bookstore.orders.billing.dlt [[2]] @0 key=a26ab9df
    message-id: msg-B
    retry-attempt: 3
    dlt-original-topic: bookstore.orders.billing.retry
    dlt-exception-type: System.Net.Http.HttpRequestException
    dlt-exception-message: 503 Service Unavailable from payment provider

The bug is obvious: because the message reached the DLT through the retry topic, the retry topic was recorded as the "original" topic. The position in the main topic had to be captured at the first failure; the final version of the router writes it to the original-topic-partition-offset header when attempt == 1:

bookstore.orders.billing.dlt [[2]] @0 key=8dbe442d
    message-id: msg-B
    original-topic-partition-offset: bookstore.orders [[2]] @0
    retry-attempt: 3
    dlt-exception-type: System.Net.Http.HttpRequestException
    dlt-exception-message: 503 Service Unavailable from payment provider

How long Kafka keeps a DLT is the topic's retention.ms setting; set it deliberately. Writing the DLT with the same key as the main topic matters too: when you replay, messages of the same order land in the same partition again and keep their relative order.

MassTransit: _error and _skipped

When retries are exhausted, MassTransit moves the message to a queue named after the endpoint with _error appended and writes the failure into headers. The XXX order from the previous post as it sits in billing-order-placed_error (I shortened the stack trace and dropped some MT-Host-* lines):

MessageId=6c0a0000-0000-0000-43d4-08df3d3c3457 ContentType=application/vnd.masstransit+json
MT-Fault-ConsumerType: Bookstore.Billing.OrderPlacedConsumer
MT-Fault-ExceptionType: System.InvalidOperationException
MT-Fault-InputAddress: rabbitmq://localhost:17301/billing-order-placed
MT-Fault-Message: Unsupported currency XXX
MT-Fault-MessageType: Bookstore.Contracts.OrderPlaced
MT-Fault-StackTrace:    at Bookstore.Billing.PaymentGateway.ChargeAsync(OrderPlaced order, CancellationToken cancellationToken) ...
MT-Fault-Timestamp: 2026-11-08T02:01:29.5773530Z
MT-Host-Assembly: Lab08Mt
MT-Host-MassTransitVersion: 8.5.12.0
MT-Host-ProcessName: Lab08Mt
MT-Reason: fault

These headers are a ready-made classification for a replay tool: you filter on MT-Fault-ExceptionType and know where to send the message back from MT-Fault-InputAddress. MassTransit also publishes a Fault<OrderPlaced> event for each fault; a consumer listening to it can open a support ticket without waiting for someone to look at the DLQ. The second queue is _skipped: messages that reach the endpoint but have no consumer to handle them go there. A message in _skipped is usually not a bug but a topology or deployment order problem (for example, a new message type published before its consumer went live).

Monitoring the DLQ

A DLQ nobody looks at is only slightly better than a message silently lost. The simplest signal is an ASP.NET Core health check:

public sealed class DlqHealthCheck(IConnection connection, string dlq, uint warnAt) : IHealthCheck
{
    public async Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken ct = default)
    {
        await using IChannel channel = await connection.CreateChannelAsync(cancellationToken: ct);
        QueueDeclareOk ok = await channel.QueueDeclarePassiveAsync(dlq, ct);
        var data = new Dictionary<string, object> { ["queue"] = dlq, ["messages"] = ok.MessageCount };
        return ok.MessageCount >= warnAt
            ? HealthCheckResult.Degraded($"{dlq} has {ok.MessageCount} dead-lettered messages", data: data)
            : HealthCheckResult.Healthy(data: data);
    }
}

Degraded: billing.order-placed.dlq has 3 dead-lettered messages

Returning Degraded rather than Unhealthy is deliberate: messages in the DLQ do not mean the service cannot take traffic, and you do not want the load balancer to take the instance out of rotation. The count from QueueDeclarePassiveAsync covers only ready messages; if an inspection tool is holding messages at that moment, the number looks lower. In production the real alert should come from a metric: RabbitMQ's Prometheus plugin (enabled in the 4.4 management image) can publish per-queue message counts too; in Service Bus you use DeadLetterMessageCount returned by ServiceBusAdministrationClient.GetQueueRuntimePropertiesAsync or Azure Monitor metrics; in Kafka you watch the growth rate of the DLT. Tying the alert to a rate such as "increase in the last 15 minutes" instead of "greater than zero" stops old, known messages from firing the same alert every day. Metric collection is covered in the twelfth post of the series.

A Safe Replay Tool

Replay means sending a message from the DLQ back to the main queue after the fix, and it is one of the riskiest manual operations in production. Bulk tools like "move messages" in the RabbitMQ management UI (user interface) move everything without a filter. The rules for our own tool: dry run first, filter by reason, an upper bound, keep the MessageId, count how many times a message was replayed, never delete from the DLQ before the broker confirms, and send directly to the queue the message failed in, not to an exchange:

public sealed class DlqReplayer(IChannel channel)
{
    public async Task<ReplayResult> ReplayAsync(
        string dlq, Func<DeadLetterInfo, bool> filter, int maxMessages, bool dryRun, CancellationToken ct = default)
    {
        int replayed = 0, skipped = 0;
        var heldBack = new List<ulong>();
        for (int i = 0; i < maxMessages; i++)
        {
            BasicGetResult? result = await channel.BasicGetAsync(dlq, autoAck: false, ct);
            if (result is null) break;

            DeadLetterInfo info = DeadLetterInfo.From(result.BasicProperties);
            if (dryRun || !filter(info) || info.FirstDeathQueue is null)
            {
                Console.WriteLine($"  {(dryRun && filter(info) ? "would replay" : "skip")} {info.MessageId} ({info.FirstDeathReason})");
                heldBack.Add(result.DeliveryTag);
                skipped++;
                continue;
            }

            var props = new BasicProperties(result.BasicProperties)
            {
                Headers = new Dictionary<string, object?>(result.BasicProperties.Headers!)
                {
                    ["x-replay-count"] = info.ReplayCount + 1,
                    ["x-replayed-at"] = DateTimeOffset.UtcNow.ToUnixTimeSeconds()
                }
            };
            // MessageId stays the same: if the consumer is idempotent, processing it again does no harm
            await channel.BasicPublishAsync("", info.FirstDeathQueue, mandatory: true, props, result.Body, ct);
            await channel.BasicAckAsync(result.DeliveryTag, multiple: false, ct);
            Console.WriteLine($"  replayed {info.MessageId} -> {info.FirstDeathQueue}");
            replayed++;
        }
        // hand the skipped ones back to the DLQ (requeue)
        foreach (ulong tag in heldBack)
            await channel.BasicNackAsync(tag, multiple: false, requeue: true, ct);
        return new ReplayResult(replayed, skipped);
    }
}

The caller spells out the filter:

var replayer = new DlqReplayer(channel);
ReplayResult result = await replayer.ReplayAsync("billing.order-placed.dlq",
    // only rejected messages and at most 3 replays: never send the same message back forever
    filter: m => m.FirstDeathReason == "rejected" && m.ReplayCount < 3,
    maxMessages: 100, dryRun: mode == "replay-dry");
Console.WriteLine($"replayed={result.Replayed} skipped={result.Skipped} dryRun={mode == "replay-dry"}");

A dry run first, then the real replay:

== replay-dry
  would replay msg-3 (rejected)
  skip msg-2 (delivery_limit)
  would replay msg-1 (rejected)
replayed=0 skipped=3 dryRun=True
== replay
  replayed msg-3 -> billing.order-placed
  skip msg-2 (delivery_limit)
  replayed msg-1 -> billing.order-placed
replayed=2 skipped=1 dryRun=False

msg-2 was left behind on purpose: a message that failed with delivery_limit, sent back before the consumer itself is fixed, enters the same loop again. Now the opening question. Suppose the fix was incomplete and the consumer rejects the two replayed messages again:

msg-2: first-death=delivery_limit@billing.order-placed deaths=[billing.order-placed/delivery_limit x1] replay-count=0
msg-3: first-death=rejected@billing.order-placed deaths=[billing.order-placed/rejected x1] replay-count=1
msg-1: first-death=rejected@billing.order-placed deaths=[billing.order-placed/rejected x1] replay-count=1

The answer is 1. The message was rejected a second time, yet the count in x-death is still x1. Since RabbitMQ 3.13, headers starting with x- belong to the broker, and an x-death republished by a client does not advance the count; the related discussion on RabbitMQ's GitHub recommends "use your own counter header" for exactly this scenario. Our x-replay-count shows 1, and the filter will keep the message in the DLQ after its third replay. Always tie the replay decision to your own counter.

You also need to know one limit of the tool: replay appends the message to the end of the queue. If newer messages for the same order have been processed in the meantime, the replayed old message can overwrite them. That is why the consumer must be idempotent and must check the order state so that "old cannot overwrite new"; we build that in the next post.

Production Notes

The DLQ holds personal data. OrderPlaced carries a customer ID, maybe an e-mail address; a message waiting indefinitely in the DLQ means an undefined retention period under KVKK (Kişisel Verilerin Korunması Kanunu, the Turkish Personal Data Protection Law) and GDPR (General Data Protection Regulation). In RabbitMQ give the DLQ an x-message-ttl (we used 14 days) and choose deliberately where expired messages go; the Service Bus DLQ does not apply TTL, so you need a periodic cleanup job; in Kafka the DLT's retention.ms sets that period.

Give it an owner. Every DLQ should have an owner (a team), an alert and a runbook: what to do for which reason code, who is allowed to replay, which checks come before a replay. Since the replay tool is a command run by hand in production, record its output (which MessageId, who, when) somewhere.

Cap its size. An unbounded DLQ can fill the broker's disk during a long outage. x-max-length in RabbitMQ, the entity size in Service Bus and retention in Kafka set that ceiling. Decide in advance what happens when it is reached (drop the oldest or reject the newest).

Separate permissions for reading the DLQ. An operator with access to DLQ messages can see data they would never see in the normal flow. Restrict access to the replay tool and the management UI to a separate role.

Common Pitfalls

Nobody watching the DLQ. Without an alert, the DLQ turns into an archive where failures pile up silently, and it is noticed only when a customer calls to say "my invoice never arrived".

Failing again with the same error on replay. Replaying without fixing the cause empties the DLQ only to fill it again a few seconds later. Try a dry run and a handful of messages first, look at the result, then continue.

Relying on x-death as a counter. As you saw, the count does not advance on a message republished by the client; use your own header.

Replaying to the exchange. If you republish the message to bookstore.ordering, every other consumer that never failed, such as Notification, receives it a second time. Replay only to the queue the message failed in.

Acking before publishing. If you delete the message from the DLQ first and publish afterwards, a tool that crashes in between loses the message forever. Publish and confirm first, then ack.

Blindly replaying messages that failed with delivery_limit. These usually point to a bug in the consumer; replaying them before the consumer is fixed starts the same loop over again.

A Small Experiment

★☆☆ Add a filter on the message's Type property to DlqReplayer and replay only bookstore.order-placed.v1 messages. Check that other types show up as "skip" in the dry-run output.

★★☆ Write a command that summarizes the messages in the DLQ grouped by their first reason (like rejected: 2, delivery_limit: 1) and expose it as a minimal API (Application Programming Interface) endpoint. The endpoint must not ack the messages; verify that the DLQ count is unchanged after it runs.

★★★ Write the same replay tool for Service Bus: receive from the DLQ with ReceiveMessagesAsync, filter on DeadLetterReason, copy with new ServiceBusMessage(received), carry your own replay counter in an application property, send it, and complete the DLQ message only after the send succeeds. Try both the dry run and a real replay in the emulator.

Replay and retry lead to the same fact: the same message can arrive more than once. The next post handles that in code: an inbox table keyed by MessageId, a unique constraint in the same transaction as the business data, natural idempotency and MassTransit's built-in inbox; so that Billing never issues two invoices for the same order.