Preventing Duplicate Message Processing in RabbitMQ

RabbitMQ is widely used to build asynchronous applications and distributed systems. It allows services to communicate without requiring them to be available at the same time.

For example, an e-commerce application may publish an order-created message after a customer completes a purchase. A background consumer receives the message and performs operations such as updating inventory, generating an invoice, sending notifications, or initiating payment processing.

This architecture improves scalability and separates business operations into independent services.

However, there is an important challenge: RabbitMQ can deliver the same message more than once.

Imagine a consumer receives an order message, successfully updates the database, but crashes before acknowledging the message. RabbitMQ does not know that the database operation was completed. When the consumer disconnects, the unacknowledged message can be delivered again.

If the application does not handle duplicate processing, the same order might be processed twice.

This can lead to duplicate payments, repeated notifications, incorrect inventory updates, and inconsistent business records.

The solution is not simply to prevent RabbitMQ from delivering a message again. Reliable messaging systems must assume that duplicate delivery is possible and design consumers to handle it safely.

This article explains why duplicate processing happens, how RabbitMQ acknowledgements work, and how to build idempotent consumers using C#, database transactions, and message deduplication.

Why RabbitMQ Delivers Messages More Than Once

RabbitMQ supports message acknowledgements to determine whether a consumer has successfully processed a delivery.

When a consumer uses manual acknowledgements, it must explicitly acknowledge a message after processing it.

Consider this workflow:

A publisher sends an order-created message.
RabbitMQ stores the message in a queue.
A consumer receives the message.
The consumer updates the database.
The consumer crashes before sending an acknowledgement.
RabbitMQ detects that the consumer connection or channel has closed.
The unacknowledged message becomes available for redelivery.
Another consumer receives the same message.

The database operation may already have succeeded, but RabbitMQ has no confirmation of that business operation.

From the broker's perspective, the message was never acknowledged.

Example of duplicate processing

Consider an order processing service.

The publisher sends:

{
  "messageId": "msg-1001",
  "orderId": 5001,
  "customerId": 2001,
  "totalAmount": 2500.00,
  "eventType": "OrderCreated"
}

The consumer receives the message and updates the order status:

UPDATE orders
SET status = 'Processing'
WHERE order_id = 5001;

The database successfully executes the update.

However, before the consumer acknowledges the message, the application crashes.

When the consumer reconnects, RabbitMQ can deliver the message again.

If the consumer performs the same operation without checking whether it has already been completed, duplicate business processing may occur.

This is one of the reasons RabbitMQ consumers should be designed to handle redeliveries.

Understanding RabbitMQ Delivery Guarantees

Before implementing duplicate protection, it is important to understand the delivery guarantees provided by RabbitMQ.

Messaging systems commonly discuss three delivery models.

Delivery model	Meaning	Main concern
At-most-once	A message is delivered zero or one time	Messages may be lost
At-least-once	A message is delivered one or more times until acknowledged or otherwise handled	Duplicate delivery is possible
Exactly-once	A business operation produces one effective result despite retries or duplicate delivery	Requires coordination across system boundaries

RabbitMQ supports reliable at-least-once processing patterns through durable messaging, acknowledgements, publisher confirms, and redelivery.

However, RabbitMQ does not automatically guarantee that a consumer's database operation or external API call executes exactly once.

For example, a message may be delivered twice even though the publisher sent it only once.

The application must therefore implement idempotency or deduplication.

Why exactly-once processing is difficult

Consider a consumer that performs two operations:

Updates a database.
Sends a payment request to an external payment provider.

These operations do not automatically share one transaction.

The database may successfully commit the update while the payment request times out.

The application cannot always determine whether the external provider processed the request.

Retrying blindly could create a second payment.

This is why exactly-once delivery and exactly-once business effects are different concepts.

A practical production design focuses on making repeated message processing safe.

Understanding Manual Acknowledgements

RabbitMQ supports automatic and manual acknowledgement modes.

Automatic acknowledgement

With automatic acknowledgement, RabbitMQ considers a message acknowledged when it delivers it to the consumer.

The consumer does not explicitly confirm that processing has completed.

This provides high delivery throughput, but it creates a risk of message loss.

For example:

RabbitMQ delivers message
          |
          v
RabbitMQ considers it acknowledged
          |
          v
Consumer starts processing
          |
          v
Consumer crashes
          |
          v
Message is not redelivered

For important business operations, automatic acknowledgement is generally not appropriate unless message loss is acceptable.

Manual acknowledgement

With manual acknowledgement, the consumer explicitly acknowledges a message after successful processing.

The workflow becomes:

RabbitMQ delivers message
          |
          v
Consumer processes message
          |
          v
Database transaction commits
          |
          v
Consumer sends ACK
          |
          v
RabbitMQ removes message

If the consumer crashes before the acknowledgement, RabbitMQ can redeliver the message.

This protects against losing messages when consumers fail, but it also means duplicate delivery must be handled.

In RabbitMQ's AMQP 0-9-1 protocol, the consumer acknowledgement must be sent on the same channel that delivered the message.

Implementing a Basic RabbitMQ Consumer in C#

The following example demonstrates manual acknowledgement using the RabbitMQ .NET client.

It illustrates the processing sequence. The database operation is represented by a placeholder and should be implemented using the application's database access layer.

using RabbitMQ.Client;
using RabbitMQ.Client.Events;
using System.Text;
using System.Text.Json;

public class OrderConsumer
{
    private readonly IConnection _connection;

    public OrderConsumer(IConnection connection)
    {
        _connection = connection;
    }

    public async Task StartAsync()
    {
        var channel = await _connection.CreateChannelAsync();

        await channel.BasicQosAsync(
            prefetchSize: 0,
            prefetchCount: 10,
            global: false);

        var consumer = new AsyncEventingBasicConsumer(channel);

        consumer.ReceivedAsync += async (_, eventArgs) =>
        {
            try
            {
                var json = Encoding.UTF8.GetString(
                    eventArgs.Body.ToArray());

                var order = JsonSerializer.Deserialize<OrderCreatedEvent>(
                    json);

                if (order is null)
                {
                    await channel.BasicNackAsync(
                        eventArgs.DeliveryTag,
                        multiple: false,
                        requeue: false);

                    return;
                }

                await ProcessOrderAsync(order);

                await channel.BasicAckAsync(
                    eventArgs.DeliveryTag,
                    multiple: false);
            }
            catch (Exception ex)
            {
                Console.WriteLine(
                    $"Order processing failed: {ex.Message}");

                await channel.BasicNackAsync(
                    eventArgs.DeliveryTag,
                    multiple: false,
                    requeue: false);
            }
        };

        await channel.BasicConsumeAsync(
            queue: "orders",
            autoAck: false,
            consumer: consumer);
    }

    private Task ProcessOrderAsync(
        OrderCreatedEvent order)
    {
        // Perform the actual business operation.
        return Task.CompletedTask;
    }
}

public class OrderCreatedEvent
{
    public string MessageId { get; set; } = string.Empty;
    public int OrderId { get; set; }
    public int CustomerId { get; set; }
    public decimal TotalAmount { get; set; }
}

The important setting is:

autoAck: false

It enables manual acknowledgements.

The consumer acknowledges the delivery only after the processing method completes successfully.

The example also sets a prefetch count of 10. This limits the number of unacknowledged deliveries that RabbitMQ can send to this consumer according to the configured QoS scope.

A suitable prefetch value depends on message processing time, consumer concurrency, and available resources.

Important acknowledgement considerations

The example demonstrates the acknowledgement sequence, but production consumers need additional safeguards.

For example:

Do not acknowledge a message before its database transaction commits.
Do not acknowledge a message when processing has only been started in the background.
Do not use the same channel concurrently without ensuring the client library's channel usage requirements are respected.
Avoid unlimited message requeueing after repeated failures.
Keep track of processing attempts and route permanently failed messages to an appropriate dead-letter queue.

There is also an important limitation in the example: if the database operation succeeds and the application crashes before the ACK, RabbitMQ can redeliver the message. Manual acknowledgement alone does not prevent duplicate business processing.

For that, the consumer needs idempotency.

What Is Idempotency in Message Processing?

Idempotency means that executing the same operation multiple times produces the same effective result as executing it once.

For example, consider an operation that marks an order as processed.

A non-idempotent implementation might create a new invoice every time it receives an order-created message.

An idempotent implementation checks whether the message has already been processed before performing the business operation.

The first delivery performs the operation.

The second delivery detects that the message has already been handled and skips the duplicate work.

This is one of the most effective ways to protect applications from repeated message delivery.

Example of a non-idempotent operation

Consider this SQL statement:

INSERT INTO invoices (
    order_id,
    invoice_amount,
    created_at
)
VALUES (
    5001,
    2500.00,
    CURRENT_TIMESTAMP
);

If the same message is processed twice, the application may create two invoices for the same order.

Making the operation idempotent

You can enforce uniqueness at the database level:

CREATE TABLE invoices (
    invoice_id BIGINT GENERATED ALWAYS AS IDENTITY
        PRIMARY KEY,
    order_id BIGINT NOT NULL UNIQUE,
    invoice_amount NUMERIC(12, 2) NOT NULL,
    created_at TIMESTAMPTZ NOT NULL
        DEFAULT CURRENT_TIMESTAMP
);

Now the database prevents multiple invoices from being created for the same order.

However, the consumer must still handle the unique constraint violation appropriately.

For more general message processing, use a dedicated message-processing record.

Using a Processed Messages Table

A common approach is to maintain a table that records messages already processed by a consumer.

For example:

CREATE TABLE processed_messages (
    message_id UUID PRIMARY KEY,
    consumer_name VARCHAR(150) NOT NULL,
    processed_at TIMESTAMPTZ NOT NULL
        DEFAULT CURRENT_TIMESTAMP
);

The message ID uniquely identifies the logical message.

The consumer checks this table before executing the business operation.

If the message ID already exists for that consumer, the application skips the operation.

If it does not exist, the application performs the business operation and records the message as processed.

Why the message ID matters

Consider two messages:

{
  "messageId": "f3f2a2a0-1a44-4c9e-9f6d-2a4f7b001001",
  "orderId": 5001
}

And:

{
  "messageId": "f3f2a2a0-1a44-4c9e-9f6d-2a4f7b001001",
  "orderId": 5001
}

Both messages have the same ID, so the consumer can identify them as repeated deliveries of the same logical event.

Now consider another event:

{
  "messageId": "f3f2a2a0-1a44-4c9e-9f6d-2a4f7b001002",
  "orderId": 5001
}

This is a different message, even though it references the same order.

The application may need to process it because it could represent another valid business event.

Do not use only the order ID as a deduplication key unless the business requirement explicitly allows only one event per order.

Implementing Idempotent Processing With a Database Transaction

The most important rule is that recording a message as processed and performing its business operation should happen in the same database transaction.

Otherwise, the application can still encounter inconsistent states.

Consider this incorrect sequence:

Insert message ID into processed_messages.
Commit the transaction.
Update the order.
Application crashes before updating the order.

When RabbitMQ redelivers the message, the application sees its ID in processed_messages and skips it.

The order update never happens.

The opposite sequence is also unsafe:

Update the order.
Commit the transaction.
Application crashes before recording the message ID.
RabbitMQ redelivers the message.
The order operation executes again.

The correct approach is to combine both database operations into one transaction.

SQL implementation

The following PostgreSQL statement attempts to insert the message ID:

INSERT INTO processed_messages (
    message_id,
    consumer_name
)
VALUES (
    @MessageId,
    @ConsumerName
)
ON CONFLICT (
    message_id,
    consumer_name
)
DO NOTHING
RETURNING message_id;

For this uniqueness rule, the table should use a composite primary key:

CREATE TABLE processed_messages (
    message_id UUID NOT NULL,
    consumer_name VARCHAR(150) NOT NULL,
    processed_at TIMESTAMPTZ NOT NULL
        DEFAULT CURRENT_TIMESTAMP,

    PRIMARY KEY (
        message_id,
        consumer_name
    )
);

If the insert returns a row, the message has not previously been recorded for that consumer.

If it returns no row, the message was already processed.

The application should perform the business operation only when the insert succeeds.

C# implementation using Npgsql

The following example demonstrates how to combine deduplication and a business update in one PostgreSQL transaction.

using Npgsql;
using System.Data;

public class OrderMessageHandler
{
    private readonly NpgsqlDataSource _dataSource;

    public OrderMessageHandler(
        NpgsqlDataSource dataSource)
    {
        _dataSource = dataSource;
    }

    public async Task<bool> HandleAsync(
        OrderCreatedEvent message,
        CancellationToken cancellationToken)
    {
        await using var connection =
            await _dataSource.OpenConnectionAsync(
                cancellationToken);

        await using var transaction =
            await connection.BeginTransactionAsync(
                IsolationLevel.ReadCommitted,
                cancellationToken);

        const string insertProcessedMessage = """
            INSERT INTO processed_messages (
                message_id,
                consumer_name
            )
            VALUES (
                @MessageId,
                @ConsumerName
            )
            ON CONFLICT (
                message_id,
                consumer_name
            )
            DO NOTHING
            RETURNING message_id;
            """;

        await using var insertCommand =
            new NpgsqlCommand(
                insertProcessedMessage,
                connection,
                transaction);

        insertCommand.Parameters.AddWithValue(
            "MessageId",
            Guid.Parse(message.MessageId));

        insertCommand.Parameters.AddWithValue(
            "ConsumerName",
            "OrderConsumer");

        var insertedMessageId =
            await insertCommand.ExecuteScalarAsync(
                cancellationToken);

        if (insertedMessageId is null)
        {
            await transaction.CommitAsync(
                cancellationToken);

            return false;
        }

        const string updateOrder = """
            UPDATE orders
            SET status = 'Processing'
            WHERE order_id = @OrderId;
            """;

        await using var updateCommand =
            new NpgsqlCommand(
                updateOrder,
                connection,
                transaction);

        updateCommand.Parameters.AddWithValue(
            "OrderId",
            message.OrderId);

        var affectedRows =
            await updateCommand.ExecuteNonQueryAsync(
                cancellationToken);

        if (affectedRows != 1)
        {
            throw new InvalidOperationException(
                "The order could not be updated.");
        }

        await transaction.CommitAsync(
            cancellationToken);

        return true;
    }
}

The handler returns:

true when the message is processed for the first time.
false when the message has already been processed.

The transaction ensures that the processed-message record and order update either both commit or both roll back.

If the application crashes before the transaction commits, PostgreSQL rolls back the uncommitted operations. RabbitMQ can then redeliver the message, allowing the consumer to try again.

If the transaction commits but the consumer crashes before sending the ACK, the message can be delivered again. This time, the unique constraint identifies it as a duplicate, and the business operation is skipped.

The consumer can safely acknowledge the duplicate because the original database transaction already committed.

This pattern provides an effective foundation for idempotent message processing.

Why the Redelivered Flag Is Not Enough

RabbitMQ provides a redelivered flag in delivery metadata.

When a message is delivered again after being requeued, the flag is set to true.

Consumers can inspect this property to understand whether a message may have been delivered previously.

However, the flag is only a hint.

A message with redelivered set to false is not necessarily guaranteed to be processed for the first time by the application.

For example, a publisher may retry publishing a message after losing a publisher confirmation. RabbitMQ may accept both publishing attempts, resulting in two messages with the same logical content.

Both deliveries may have redelivered set to false because each is a separate delivery of a published message.

Therefore, use a stable application-generated message ID for deduplication.

Do not rely exclusively on RabbitMQ's redelivered flag.

Preventing Duplicate Messages at the Publisher Level

Duplicate processing can also originate from the publisher.

Consider this scenario:

An application publishes an order-created event.
RabbitMQ receives and stores the message.
RabbitMQ sends a publisher confirmation.
The network connection fails before the publisher receives the confirmation.
The publisher cannot determine whether the message was accepted.
The publisher retries the operation.

The retry may publish the same logical event again.

RabbitMQ publisher confirms improve reliability, but they do not automatically deduplicate repeated publishing attempts.

Use a stable message ID

Generate the message ID once when the business event is created.

var message = new OrderCreatedEvent
{
    MessageId = Guid.NewGuid().ToString(),
    OrderId = 5001,
    CustomerId = 2001,
    TotalAmount = 2500.00m
};

If publishing must be retried, reuse the same message ID.

Do not generate a new ID for every retry.

Otherwise, the consumer cannot recognize that the messages represent the same logical event.

Use the transactional outbox pattern

There is another reliability problem when an application updates its database and publishes a message.

Consider this workflow:

The application creates an order in PostgreSQL.
The database transaction commits.
The application publishes an OrderCreated message to RabbitMQ.

If the application crashes between steps 2 and 3, the order exists but the message is never published.

The transactional outbox pattern addresses this problem.

The application writes the business change and an outgoing event into the same database transaction.

For example:

CREATE TABLE outbox_messages (
    message_id UUID PRIMARY KEY,
    event_type VARCHAR(150) NOT NULL,
    payload JSONB NOT NULL,
    created_at TIMESTAMPTZ NOT NULL
        DEFAULT CURRENT_TIMESTAMP,
    published_at TIMESTAMPTZ NULL
);

The application creates the order and records the outgoing message in one transaction:

BEGIN;

INSERT INTO orders (
    order_id,
    customer_id,
    status
)
VALUES (
    5001,
    2001,
    'Created'
);

INSERT INTO outbox_messages (
    message_id,
    event_type,
    payload
)
VALUES (
    'f3f2a2a0-1a44-4c9e-9f6d-2a4f7b001001',
    'OrderCreated',
    '{"orderId":5001,"customerId":2001}'
);

COMMIT;

A background publisher reads unpublished outbox records and sends them to RabbitMQ.

After receiving a publisher confirmation, it marks the record as published.

However, there is still a possible duplicate scenario: RabbitMQ may accept the message, but the publisher may fail before updating published_at.

The publisher can then send the same outbox record again.

That is why the transactional outbox pattern should be combined with consumer-side idempotency.

The outbox prevents losing events between a database commit and message publication. It does not eliminate every possibility of duplicate delivery.

Handling Failed Messages Without Creating Infinite Redelivery Loops

Not every processing failure should result in an immediate retry.

For example, a consumer may fail because:

A database connection is temporarily unavailable.
An external service is responding slowly.
The message payload is invalid.
A required business record does not exist.
A business validation rule rejects the operation.

These failures require different handling.

Requeueing a failed message

RabbitMQ supports negative acknowledgements.

For example:

await channel.BasicNackAsync(
    deliveryTag,
    multiple: false,
    requeue: true);

Setting requeue to true makes the message available for redelivery.

However, immediately requeueing every failed message can create a redelivery loop.

If the database is unavailable, the consumer may repeatedly receive the same message, fail, and requeue it.

This consumes resources without solving the underlying problem.

Use dead-letter queues

A dead-letter queue provides a separate destination for messages that cannot be processed normally.

For example, an application can route permanently rejected messages to a dead-letter exchange.

A message can be rejected without requeueing:

await channel.BasicNackAsync(
    deliveryTag,
    multiple: false,
    requeue: false);

When the queue is configured with an appropriate dead-letter exchange, the message can be routed to the configured dead-letter destination.

A dead-letter queue helps operators inspect failed messages and decide whether they can be corrected and replayed.

For transient failures, use controlled retry policies with delays and a maximum number of attempts rather than immediately requeueing indefinitely.

RabbitMQ quorum queues also support poison message handling features that can limit repeated unsuccessful deliveries. The appropriate configuration depends on the queue type and RabbitMQ version.

Using Prefetch to Control Consumer Load

RabbitMQ prefetch controls how many messages can be delivered to a consumer without acknowledgement.

For example:

await channel.BasicQosAsync(
    prefetchSize: 0,
    prefetchCount: 10,
    global: false);

This limits the number of unacknowledged deliveries within the configured consumer scope.

Prefetch does not prevent duplicate delivery.

However, it helps control the amount of work assigned to consumers and reduces the risk of overwhelming application resources.

Consider the following configuration:

Prefetch	Behavior
1	One unacknowledged delivery at a time
10	Up to ten unacknowledged deliveries
50	Up to fifty unacknowledged deliveries
0	No explicit prefetch limit

A very high prefetch value can increase memory usage and the number of in-flight messages.

A very low value can reduce throughput.

Choose a value based on message processing time, consumer concurrency, database connection limits, and memory availability.

Monitoring Duplicate Message Processing

A production messaging system should track duplicate deliveries and processing outcomes.

Useful metrics include:

Metric	Why it matters
Messages received	Measures consumer workload
Messages acknowledged	Tracks successful broker acknowledgements
Messages rejected	Identifies failed processing
Redelivered messages	Indicates messages delivered previously
Duplicate message IDs	Measures deduplication activity
Processing duration	Identifies slow consumers
Queue depth	Shows accumulated message backlog
Dead-letter count	Tracks messages that could not be processed
Database transaction failures	Identifies persistence problems

You should also log the message ID, event type, consumer name, and processing result.

For example:

logger.LogInformation(
    "Message {MessageId} processed by {ConsumerName}",
    message.MessageId,
    "OrderConsumer");

For duplicate messages:

logger.LogInformation(
    "Duplicate message {MessageId} skipped",
    message.MessageId);

Avoid logging complete message payloads when they contain customer information, payment data, credentials, or other sensitive values.

A stable message ID makes it easier to trace one logical event across publishers, RabbitMQ, consumers, and database operations.

Common Mistakes That Cause Duplicate Business Operations

Mistake	Why it is a problem	Better approach
Acknowledging before database commit	A failed transaction may result in lost processing	ACK after successful commit
Assuming manual ACK prevents duplicates	Messages can be redelivered after failures	Make consumers idempotent
Using the redelivered flag as the only check	Publisher retries can create separate deliveries	Use stable application message IDs
Generating a new message ID for every retry	Duplicate events appear unrelated	Reuse the original message ID
Recording processed messages outside the business transaction	Processing state and business data can become inconsistent	Use one database transaction
Requeueing every failure immediately	Can create endless redelivery loops	Use controlled retries and dead-letter handling
Ignoring publisher confirms	The publisher may not know whether RabbitMQ accepted a message	Use confirms and reliable publishing
Assuming the outbox eliminates duplicates	Publishing can succeed before the outbox record is marked complete	Combine outbox publishing with idempotent consumers
Using only in-memory deduplication	State disappears after application restarts	Persist deduplication records
Deleting processed-message records too early	Old messages may be processed again	Define a retention period based on replay and retry requirements

Recommended Production Architecture

A reliable RabbitMQ architecture should protect both publishing and consuming operations.

A practical design looks like this:

          Application Service
                  |
                  v
          Database Transaction
            /           \
           v             v
        Orders       Outbox Messages
                          |
                          v
                  Background Publisher
                          |
                          v
                       RabbitMQ
                          |
                          v
                  Consumer Application
                          |
                          v
                 Processed Messages
                          |
                          v
                 Business Transaction
                          |
                          v
                    Database
                          |
                          v
                         ACK

The main responsibilities are:

The application writes business changes and outgoing events in one transaction.
The publisher sends outbox events using publisher confirms.
Every logical message has a stable message ID.
RabbitMQ delivers messages to consumers using manual acknowledgements.
The consumer checks whether the message has already been processed.
The deduplication record and business operation are committed in one database transaction.
The consumer acknowledges the message after the transaction succeeds.
Failed messages follow a controlled retry and dead-letter strategy.

This architecture does not make every network interaction exactly once. Instead, it makes repeated delivery safe and ensures that business operations have controlled, predictable outcomes.

Conclusion

Duplicate message processing is a normal possibility in reliable RabbitMQ applications.

A consumer may successfully update a database and crash before acknowledging the message. A publisher may retry after losing a confirmation. A network failure may interrupt communication at a point where the application cannot determine whether an operation completed.

These situations make duplicate delivery possible even when RabbitMQ is configured correctly.

The most reliable approach is to design consumers around idempotency.

Use manual acknowledgements, stable message IDs, database-backed deduplication, and transactions that combine message tracking with business updates. Use publisher confirms and the transactional outbox pattern to improve publishing reliability. Add controlled retries, dead-letter queues, and monitoring to handle failures without creating endless redelivery loops.

The goal is not to assume that every message will arrive exactly once. It is to ensure that processing the same logical message multiple times does not produce duplicate business effects.