Duplicate Message Detection for an Azure Service Bus

Background

In case any application is sending a message to the service bus queue, sometimes due to some error in the system, it can send the same message multiple times.

By enabling duplicate detection, we can prevent this.

Concepts

When we enable duplicate detection, the MessageId of all the messages is sent into a queue or topic during a specific time window. Message with MessageId is logged during the time window. If any message is sent with an existing MessageId, the message is ignored and dropped. Please note that the message will be reported as accepted, though it will be ignored.

When we enable duplicate detection, we can configure duplicate detection window size. The default value of the window is 10 minutes. We can set any value between 20 seconds to 7 days. Please note that if we enable the size for a longed period, it will have an impact on the throughput as the new message IDs will be checked with all the recorded message IDs.

Enabling Duplicate detection

We can enable duplicate detection by using Azure Portal, Windows PowerShell, Azure CLI, or by using an ARM template.

The below snapshot shows how we can create a service bus topic with duplicate detection using Azure Portal.

Please note we need to enable the duplicate detection option during creation only. After a queue or topic is created, we cannot enable this option.

Setting MessageId

If the sender client application is yours, then you can set up the MeesageId using the broker

MessageId property of a Brokered message.

The following C# code will set the MessageId property to the service bus message.

// Create a TopicClient to connect to the Service Bus topic
TopicClient queueClient = new TopicClient(serviceBusConnectionString, topicName);

// Create a new brokered message to send to the queue
var brokeredMessage = new Message
{
    MessageId = orderId.ToString()
};

// Send the message to the queue
await queueClient.SendAsync(brokeredMessage);

In case the service bus message is coming from an external source, and you do not have access to the code, then you can set up an Api in Azure APIM management, the external source will call the api, and the api will send the message to the service bus. Here, the service bus url will be the backend for the api.

<set-backend-service base-url="https://orderproccessingtoipc.servicebus.windows.net/ordertopic/" />

To set the MessageId, you need to create an inbound policy.

<set-variable name="requestBody" value="@{
    var inBody = context.Request.Body.As<XDocument>();
    return inBody.ToString();
}" />

<set-header name="BrokerProperties" exists-action="override">
    <value>@{
        var json = new JObject();
        var inBody = XDocument.Parse((string)context.Variables.GetValueOrDefault("requestBody"));
        var messageId = inBody.Descendants("orderNumber")
                              .Select(e => e.Value)
                              .FirstOrDefault();
        json.Add("MessageId", messageId);
        return json.ToString(Newtonsoft.Json.Formatting.None);
    }</value>
</set-header>

Benefits

When a duplicate message is received in the service bus, the same message will be unnecessarily processed multiple times. As for an example, an application sending order details, if it sends a duplicate message, then there is a possibility that the same order will be processed multiple times.

Duplicate detection will ensure that the queue or topic discards duplicate copies of the same message.