Building a Message Replay Tool for Failed RabbitMQ Messages

RabbitMQ is commonly used to connect services in distributed applications. A producer publishes messages to an exchange, RabbitMQ routes them to queues, and consumers process them independently.

This approach works well until a consumer cannot process a message.

An external API may be unavailable, a database operation may fail, or a message may contain invalid data. RabbitMQ can move failed messages into a dead-letter queue, where developers can investigate them.

However, storing failed messages is only part of the solution. Teams also need a safe way to replay those messages after fixing the problem.

Manually copying messages from a dead-letter queue and publishing them again is difficult to manage. It can also create duplicate operations if the original message was partially processed.

This article explains how to build a message replay tool using RabbitMQ, ASP.NET Core, and C#.

Understand How Dead-Letter Queues Work

A dead-letter queue stores messages that cannot be processed normally.

RabbitMQ can dead-letter a message when:

A consumer rejects or negatively acknowledges it without requeueing.
A message expires because of its time-to-live setting.
A queue exceeds its configured length limit.
A quorum queue delivery limit is reached.

For example, consider an order processing application.

Order Service
      |
      v
Order Exchange
      |
      v
Order Queue
      |
      v
Order Consumer
      |
      v
Database / External API
      |
      | Processing fails
      v
Dead-Letter Exchange
      |
      v
Dead-Letter Queue

The failed message remains available for investigation instead of being repeatedly delivered to the same failing consumer.

A replay tool can retrieve an approved message from the dead-letter queue and publish it back to the original processing route.

The important point is that replay should be a controlled operation. It should not automatically resend every failed message without checking why it failed.

Configure RabbitMQ Queues

Let's create an order processing queue and a dead-letter queue.

The main exchange will receive order events. The order queue will use a dead-letter exchange for rejected messages.

The following example uses RabbitMQ.Client.

Install the package:

dotnet add package RabbitMQ.Client

Create the connection and queue configuration:

using RabbitMQ.Client;

public class RabbitMqSetup
{
    private readonly IConnection _connection;

    public RabbitMqSetup(IConnection connection)
    {
        _connection = connection;
    }

    public async Task ConfigureAsync(
        CancellationToken cancellationToken)
    {
        await using var channel =
            await _connection.CreateChannelAsync(
                cancellationToken: cancellationToken);

        await channel.ExchangeDeclareAsync(
            exchange: "orders.exchange",
            type: ExchangeType.Direct,
            durable: true,
            cancellationToken: cancellationToken);

        await channel.ExchangeDeclareAsync(
            exchange: "orders.dlx",
            type: ExchangeType.Direct,
            durable: true,
            cancellationToken: cancellationToken);

        await channel.QueueDeclareAsync(
            queue: "orders.queue",
            durable: true,
            exclusive: false,
            autoDelete: false,
            arguments: new Dictionary<string, object?>
            {
                ["x-dead-letter-exchange"] =
                    "orders.dlx",

                ["x-dead-letter-routing-key"] =
                    "orders.failed"
            },
            cancellationToken: cancellationToken);

        await channel.QueueDeclareAsync(
            queue: "orders.dead-letter",
            durable: true,
            exclusive: false,
            autoDelete: false,
            cancellationToken: cancellationToken);

        await channel.QueueBindAsync(
            queue: "orders.queue",
            exchange: "orders.exchange",
            routingKey: "orders.created",
            cancellationToken: cancellationToken);

        await channel.QueueBindAsync(
            queue: "orders.dead-letter",
            exchange: "orders.dlx",
            routingKey: "orders.failed",
            cancellationToken: cancellationToken);
    }
}

This configuration routes rejected order messages to orders.dead-letter.

For production deployments, prefer broker-managed policies for dead-letter configuration where appropriate. Policies make it easier to update queue behavior without deleting and recreating queues.

Also consider the dead-lettering guarantees of your RabbitMQ queue type. For important business messages, configure the appropriate quorum queue and dead-letter strategy rather than assuming every dead-letter transfer has identical delivery guarantees.

Understand Message Properties Before Replay

A RabbitMQ message contains more than its body.

It can also include properties such as:

Message ID.
Correlation ID.
Content type.
Delivery mode.
Timestamp.
Headers.
Routing key.

A replay tool should preserve the original business message and relevant metadata.

For example, the original message might contain:

{
  "orderId": 4521,
  "customerId": 1052,
  "amount": 1500,
  "status": "Created"
}

The replay operation should not change the order ID or amount simply because the message is being processed again.

It should also retain the original message ID so consumers can recognize duplicate delivery.

However, preserve the original message ID separately from the replay operation ID. The replay operation needs its own identifier for tracking and auditing.

Create a Replay Request Model

The replay tool should not allow users to publish arbitrary messages to any exchange.

Start with a request model that identifies the failed message and the intended replay operation.

public class MessageReplayRequest
{
    public string MessageId { get; set; }
        = string.Empty;

    public string Reason { get; set; }
        = string.Empty;

    public string RequestedBy { get; set; }
        = string.Empty;
}

The message ID identifies the original message.

The reason explains why an operator wants to replay it.

The authenticated user's identity should come from the application authentication context, not from a value supplied by the browser. The same applies to authorization decisions.

For a production system, store the replay request in a database before executing it. This provides a record of who requested the replay and whether the operation succeeded.

Retrieve Failed Messages From the Dead-Letter Queue

RabbitMQ provides a basic.get operation for retrieving individual messages without starting a continuous consumer.

The following example uses the asynchronous RabbitMQ.Client API:

public class DeadLetterMessageReader
{
    private readonly IConnection _connection;

    public DeadLetterMessageReader(
        IConnection connection)
    {
        _connection = connection;
    }

    public async Task<BasicGetResult?> GetMessageAsync(
        CancellationToken cancellationToken)
    {
        await using var channel =
            await _connection.CreateChannelAsync(
                cancellationToken: cancellationToken);

        return await channel.BasicGetAsync(
            queue: "orders.dead-letter",
            autoAck: false,
            cancellationToken: cancellationToken);
    }
}

Using autoAck: false keeps the message unacknowledged until the application decides what to do with it.

The message should be acknowledged only after the replay tool has safely recorded or transferred it according to its recovery design.

A production implementation should also prevent multiple administrators from selecting and replaying the same message concurrently. One approach is to persist a message inventory and use an atomic claim operation before replay begins.

Publish the Message Back to the Original Exchange

Once a message has been selected and approved, the replay service can publish its body back to the original exchange.

The following method accepts the original message and publishes it again:

public class MessageReplayPublisher
{
    private readonly IConnection _connection;

    public MessageReplayPublisher(
        IConnection connection)
    {
        _connection = connection;
    }

    public async Task PublishAsync(
        byte[] body,
        IReadOnlyBasicProperties originalProperties,
        CancellationToken cancellationToken)
    {
        await using var channel =
            await _connection.CreateChannelAsync(
                cancellationToken: cancellationToken);

        var properties =
            new BasicProperties();

        properties.ContentType =
            originalProperties.ContentType;

        properties.ContentEncoding =
            originalProperties.ContentEncoding;

        properties.DeliveryMode =
            DeliveryModes.Persistent;

        properties.MessageId =
            originalProperties.MessageId;

        properties.CorrelationId =
            originalProperties.CorrelationId;

        properties.Type =
            originalProperties.Type;

        properties.Timestamp =
            originalProperties.Timestamp;

        properties.Headers =
            originalProperties.Headers == null
                ? new Dictionary<string, object?>()
                : new Dictionary<string, object?>(
                    originalProperties.Headers);

        await channel.BasicPublishAsync(
            exchange: "orders.exchange",
            routingKey: "orders.created",
            mandatory: true,
            basicProperties: properties,
            body: body,
            cancellationToken: cancellationToken);
    }
}

This example preserves important message properties and sends the message back to the original exchange.

The original message ID remains unchanged so that consumers can identify duplicate processing attempts.

The routing key is deliberately fixed to orders.created. In a real application, use an allowlisted mapping between supported message types and their original routing destinations. Do not let an operator or API caller provide an unrestricted exchange name or routing key.

The code also illustrates the publishing operation, not a complete reliable replay transaction. Production code must confirm that the broker accepted the publication, for example by using publisher confirms, before acknowledging the original dead-letter message.

Avoid Losing Messages During Replay

There is an important reliability problem when retrieving and publishing messages.

Consider this sequence:

The replay tool retrieves a message.
The tool publishes it to the original exchange.
RabbitMQ accepts the publication.
The application crashes before acknowledging the original dead-letter message.

After restarting, the original message is still in the dead-letter queue. The tool may publish it again.

This creates duplicate delivery.

The opposite order is also unsafe:

The tool retrieves a message.
It acknowledges the original message.
The application crashes before publishing it again.

The message is now lost.

There is no general atomic transaction that combines consuming from one queue and publishing to another through an ordinary application-level workflow.

A safer approach is to use an explicit replay state and publisher confirms.

For example:

Message Retrieved
       |
       v
Replay Record Created
       |
       v
Publish to Original Exchange
       |
       v
Publisher Confirm Received
       |
       v
Acknowledge Original Message
       |
       v
Replay Completed

If the application crashes after publishing but before acknowledging, the replay record and original message help the recovery process identify the uncertain operation.

The consumer must still support idempotent processing because duplicate delivery is possible.

For stronger reliability, use a durable replay job table or outbox-style workflow. Persist the replay request and message body before acknowledging the original delivery, then let a worker publish the stored replay job. This separates message capture from message delivery and makes recovery easier to manage.

Make Consumers Idempotent

A replay tool should assume that the same message may be delivered more than once.

For example, a consumer can store processed message IDs in a database.

public class ProcessedMessage
{
    public string MessageId { get; set; }
        = string.Empty;

    public DateTime ProcessedAtUtc { get; set; }
}

Before performing the business operation, check whether the message has already been processed.

The check and business update should happen in the same database transaction when possible. A unique constraint on MessageId provides an additional safeguard against concurrent duplicate processing.

A simplified approach looks like this:

public async Task ProcessAsync(
    string messageId,
    OrderCreatedEvent order,
    CancellationToken cancellationToken)
{
    var alreadyProcessed = await _context.ProcessedMessages
        .AnyAsync(
            x => x.MessageId == messageId,
            cancellationToken);

    if (alreadyProcessed)
    {
        return;
    }

    await using var transaction =
        await _context.Database.BeginTransactionAsync(
            cancellationToken);

    _context.ProcessedMessages.Add(
        new ProcessedMessage
        {
            MessageId = messageId,
            ProcessedAtUtc = DateTime.UtcNow
        });

    _context.Orders.Add(
        new Order
        {
            OrderId = order.OrderId,
            Amount = order.Amount
        });

    await _context.SaveChangesAsync(
        cancellationToken);

    await transaction.CommitAsync(
        cancellationToken);
}

This example assumes that the order creation and processed-message record belong to the same database transaction.

For an external API call, a database transaction cannot make the remote operation atomic. Use an external idempotency key or a reconciliation process for those operations.

Build an ASP.NET Core Replay Endpoint

A replay tool should be available only to authorized operators.

Create an API endpoint that accepts a replay request:

[ApiController]
[Route("api/admin/message-replay")]
[Authorize(Policy = "MessageReplay")]
public class MessageReplayController : ControllerBase
{
    private readonly IMessageReplayService _replayService;

    public MessageReplayController(
        IMessageReplayService replayService)
    {
        _replayService = replayService;
    }

    [HttpPost("{messageId}")]
    public async Task<IActionResult> ReplayAsync(
        string messageId,
        [FromBody] MessageReplayRequest request,
        CancellationToken cancellationToken)
    {
        if (string.IsNullOrWhiteSpace(request.Reason))
        {
            return BadRequest(
                "A replay reason is required.");
        }

        var requestedBy =
            User.Identity?.Name ?? "Unknown";

        var result = await _replayService.ReplayAsync(
            messageId,
            request.Reason,
            requestedBy,
            cancellationToken);

        return Accepted(result);
    }
}

The authorization policy should restrict access to approved operational roles.

The service should also validate that the message exists, has not already been replayed successfully, and belongs to a supported message type.

For sensitive operations, require a second-person approval before publishing the message again.

Track Replay History

Every replay attempt should be recorded.

A replay audit entity might contain:

public class MessageReplayAudit
{
    public Guid Id { get; set; }

    public string OriginalMessageId { get; set; }
        = string.Empty;

    public string RequestedBy { get; set; }
        = string.Empty;

    public string Reason { get; set; }
        = string.Empty;

    public DateTime RequestedAtUtc { get; set; }

    public DateTime? CompletedAtUtc { get; set; }

    public string Status { get; set; }
        = "Pending";

    public string? ErrorMessage { get; set; }
}

This record helps operators answer important questions:

Who replayed the message?
Why was it replayed?
When did the operation happen?
Did RabbitMQ confirm the publication?
Was another replay attempted?
Did the consumer process the message successfully?

Do not store sensitive message payloads in audit records unless there is a documented requirement and appropriate protection.

Best Practices for Production Replay Tools

Require explicit approval

Do not replay every dead-letter message automatically. Invalid messages may fail repeatedly and create unnecessary load.

Preserve original message identity

Keep the original message ID and correlation ID. Generate a separate replay ID to identify each replay operation.

Use publisher confirms

Do not acknowledge the original message until the replay publication has been confirmed and the replay workflow has recorded the outcome.

Support idempotent consumers

Assume that duplicate delivery can happen, especially after application crashes or uncertain network outcomes.

Apply replay limits

Limit the number of messages replayed in one operation. Large replays can overload downstream services.

Monitor replay activity

Track replay counts, successful publications, failed attempts, processing failures, and repeated message delivery.

Protect administrative access

Use authentication, authorization, audit logging, and approval controls for production replay operations.

Common Mistakes to Avoid

One common mistake is acknowledging a dead-letter message before confirming that its replacement publication succeeded. This can result in message loss.

Another mistake is replaying a message without preserving its original identity. Consumers may then treat a duplicate operation as a new business event.

Developers also sometimes replay all failed messages without checking their error category. A message containing invalid data will usually fail again even after being replayed.

Finally, avoid using the management interface or a custom script as the only recovery process. A controlled replay service provides consistent validation, authorization, auditing, and operational visibility.

Summary

A dead-letter queue helps isolate messages that cannot be processed, but enterprise applications also need a safe way to recover them.

A RabbitMQ replay tool should retrieve approved messages, preserve their original metadata, publish them back through an allowlisted route, and acknowledge the original delivery only after a confirmed handoff.

Use durable replay records, publisher confirms, idempotent consumers, and audit history to handle failures during the replay process itself.

The objective is not simply to send failed messages again. It is to recover them without losing business events or repeating operations that have already succeeded.