How to Make REST APIs Idempotent?

Idempotency

In general, Idempotency means if you repeat a certain action, the result will always be the same, e.g. in mathematics adding zero to any number will always result in the same number, no matter how many times you do that operation.

In the REST API world, if you are an API developer, you must have heard this buzzword from some of your colleagues/architect/lead who is also giving thought from the perspective of making that API more resilient and fault tolerant. Let me give you a some real-world scenario and we will try to understand the problem our API might face if it is not Idempotent.

Scenario I: The user is trying to register on a social media site and there is a register button the user presses to complete registration. Something like the below API request, UI application will make.

Consider the case where API does not have Idempotency check and the below request gets executed twice, this can happen intentionally or unintentionally, the user can press the button two times or there is a request time out due to slowness in the network where the user is created at the server side but the client receives a timeout or server creates resource but fails to give a correct response to a client due to server error and client is having retry mechanism to have fault tolerance and resilience in the application. What will happen in that case? API might create a duplicate user, correct?

POST /users

{
    "name": "John Doe",
    "email": "[email protected]"
}

Scenario II: The user is trying to make a payment against a certain order to fulfill that order, it will have an API call like below.

Here, again same thing can happen, the user might accidentally click the payment button twice (considering there is no UI validation or UI validation is failing due to some reason), or there is a request timeout due to slowness in the network where payment is made at server side but the client receives a timeout or server makes a payment but fails to give a correct response to client due to server error and client is having retry mechanism to have fault tolerance and resilience in the application. What will happen in that case?

POST /order/<order-id>/payment

{
    ... (payment details)
}

Scenario III: In enterprise application, each system is not supposed to be a system of record and there is a need for integration where one business object is pushed from one system to another system and every system has its own identified for its object. For e.g. there is a demand creation system, demand creation system creates a demand and pushes that demand to another system to fulfill that demand. Demand creation and fulfillment systems are two different systems. Below is the API call the demand creation system might make.

POST /demand

{
   "demandpurpose" : "Give me a laptop",
   "brand":"dell",
    "specifications": "16GB ram and core i7"
}

Here, again same thing can happen, the user might accidentally click create demand button twice (considering there is no UI validation or UI validation is failing due to some reason), or there is a request time out due to slowness in the network where demand is made at server side but the client receives a timeout or server creates demand but fails to give a correct response to client due to server error and client is having retry mechanism to have fault tolerance and resilience in the application. What will happen in that case?

I think now you have a better idea of real-world problems that might happen when we miss idempotency handling in API, let us try to understand this concept better and then go toward the solutions to the above-listed problems.

Idempotency with HTTP Methods

An idempotent HTTP method is a method that can be invoked many times without different outcomes. It should not matter if the method has been called only once, or ten times over. The result should always be the same.

If we follow the REST principles in designing our APIs, we will have automatically idempotent REST APIs for GET, PUT, DELETE, HEAD, OPTIONS, and TRACE methods. Only POST and PATCH APIs will not be idempotent.

POST and PATCH are NOT idempotent.

GET, PUT, DELETE, HEAD, OPTIONS, and TRACE are idempotent.

HTTP Methods

GET, HEAD, OPTIONS and TRACE methods NEVER change the resource state on the server. They are purely for retrieving the resource representation or metadata at that point in time.

So invoking multiple requests will not have any write operation on the server, so GET, HEAD, OPTIONS, and TRACE are always idempotent.

Generally – not necessarily – PUT APIs are used to update the resource state. If you execute a PUT API N times, the very first request will update the resource; the other N-1 requests will just overwrite the same resource state again and again – effectively not changing anything.

Hence, PUT is idempotent.

When you execute N similar DELETE requests, the first request will delete the resource and the response will be 200 (OK) or 204 (No Content).

Other N-1 requests will return 404 (Not Found).

The response is different from the first request, but there is no change of state for any resource on the server side because the original resource is already deleted.

So, DELETE is idempotent.

How to solve the Idempotency problem for Post and Patch?

There are various ways to solve this problem, let me list them one by one.

Solution I (Scenario I listed above): If we can identify something in the request that can be used as the primary key on the server side, then we can avoid duplicate resources getting created based on that and return an error for duplicate POST request specifying that the resource is already present.

For example, in the below request, we can have the user email ID as a primary key on the server side.

POST /users

{     
    "name": "John Doe",     
    "email": "[email protected]"
}

Solution II (Scenario II listed above): If there is no possibility of having something unique in the request body then we can check for some relationship between two entities on the server side, for e.g. in this case, each order can have only one successful payment request, so we will check if the Payment request is already there in our system then we will reject the subsequent repeated or duplicate requests from the client side with proper error message.

POST /order/<order-id>/payment

{     ... (payment details)

}

Solution III (Scenario III listed above): Let's say if there is no unique key in the request and also there is no relationship between entities at the target system that we can use to check the idempotency. How to solve the issue?

There is the concept of client ID or idempotency key, that we can ask the client to pass, which can used to check the uniqueness and avoid duplication.

POST /demand

{
   "Idempotency-Key" : "1063ef6e-267b-48fc-b874-dcf1e861a49d"
   "demandpurpose" : "Give me a laptop",
   "brand":"dell",
   "specifications": "16GB ram and core i7"
}

Here, the client/source system needs to first generate this idempotency key and pass that to the server/target system, The target system will create a resource/demand and map that resource/demand with the client ID or idempotency key and all the duplicate request or retried request will be then rejected saying resource/demand is already present in the target system. Here client has a choice of creating demand first in their system with idempotency id and then pushing to the target system or upon receiving a successful response client/source system needs to use the same idempotency key sent to the server to create a resource at their end, servere can send this key in get response of 200 OK or 201 Created.

I hope the concept of Idempotency is clear and you better understand some real-world problems, the third scenario mentioned here is an actual business problem that I have faced in my organization, and the solution mentioned here is in place there to avoid it.

Please provide some comments and feedback to this article which motivates me to write down some more articles here !!

Happy learning !!