Configure Dead Letter Queue In Amazon SQS

This article explains what dead letter queues are and how they can be configured using AWS console.

Why do we need Dead Letter Queues?

 

Amazon SQS provides us with a great way of sending and receiving messages asynchronously, where a producer sends a message and a consumer receives it. After receiving a message the consumer processes and deletes it. SQS doesn’t automatically delete a message immediately after sending it to the consumer, the consumer has to delete it himself when he is sure that the processing is successful.

 

However, this is not a perfect world nor is any system, so there are chances that we aren't able to process a message, since we didn't process it, we couldn't delete it, therefore our queue will send us the message again (after the visibility timeout occurs). In real-world scenarios sometimes a consumer might fail to process a message, there can be many reasons behind it like DB failure or some flags are turned off, etc. So, we need to set a limitation on receiving the same message again and again otherwise the consumer would receive it infinitely till the retention period for a message is over which adds unnecessary billings.

 

One way to handle such a situation is to delete the message altogether, but can we really do that? If a message is not processed, deleting it will lead to data loss and we obviously don’t want our messages to go unprocessed or get lost but there is no benefit in keeping the failed messages in the Main queue either. So what to do in such a scenario?

 

What are Dead Letter Queues (DLQ)?

 

Here comes the role of Dead Letter queues. DLQs are used for storing failed or undelivered messages, the messages which could not be processed by the consumer are moved to DLQ so that they can be processed in the future. Dead letter queues are separate queues, they are processed and monitored separately. They are not automatically configured along with your main queue, but Amazon SQS provides you an option to configure them with the Main Queue.

 

Steps to configure a Dead Letter Queue

 

Step 1

 

Create two Queues: MainQueue and DeadLetterQueue.

 

 

Step 2

 

Configure DeadLetterQueue with MainQueue.

 

Queue-> MainQueue-> Edit-> Dead Letter Queue

To configure a DLQ we need to set a Redrive Policy which includes,

• Set the queue to Enabled state.

• Queue ARN
  Set it to the ARN of the Queue you want as a dead letter queue, in this example I will use the ARN DeadLetterQueue created in Step-1.

• Maximum receives
  It signifies the number of times a message is received by the consumer before it is moved to DLQ.

Step 3

 

Send a message from the Main Queue.

 

Step 4

 

Receive the message from MainQueue->SendReceieve Message-> Poll for messages.

You will see the receive count along with the message ID. When you Poll for messages two times you will see the Receive Count as 2.

 

 

Try to poll for the third time, you will not see any message. But you haven't deleted any messages yet, where did the message go? Well, check the Dead Letter Queue maybe.

Step 5

 

Check the number of messages in the Dead letter Queue and the Main Queue.

 

As we can see the message has been moved from the Main Queue to DeadLetterQueue after being received two times since we have set Maximum Receive number as 2 in the Redrive Policy while configuring the Dead Letter Queue. 

 

The same implementation can be easily done using C# or Java. All you need to do is write a trigger for Processing messages from the Dead Letter Queue along with the usual code that we write for processing the messages from the Main Queue, which I will explain the upcoming posts.

 

If you have any doubts and questions, please let me know in the comments section.

Thanks!