How to Prevent Duplicate API Requests with Idempotency Keys

Introduction

Imagine a customer clicks the Pay button on an e-commerce website. The request reaches your API, and the payment provider processes the transaction. However, the network connection drops before the customer receives a response.

The customer does not know whether the payment succeeded, so they click the button again.

If your API processes the second request as a new payment, the customer could be charged twice.

This problem is not limited to payment systems. It can also happen when creating orders, booking appointments, transferring money, or submitting any request that should not perform the same operation more than once.

One way to handle this situation is to use idempotency keys.

An idempotency key lets the client identify a particular operation. When the same request is retried with the same key, the server can recognize it and avoid performing the operation again.

In this article, we will learn how idempotency keys work, implement them in an ASP.NET Core Web API, and discuss the database and production considerations needed to make the solution reliable.

What Is an Idempotency Key?

An idempotency key is a unique identifier sent with an API request to identify one logical operation.

For example, a client might send a payment request with this header:

POST /api/payments
Idempotency-Key: 4f1c8c6d-2a77-4c10-9c43-7f4d8b12a901
Content-Type: application/json

The request body might contain the payment details:

{
  "orderId": 101,
  "amount": 2500
}

If the request succeeds but the response is lost, the client can retry using the same key.

The API checks whether that key has already been used. If the original operation has completed, the API can return the saved result instead of creating another payment.

The important rule is that retries of the same logical operation must reuse the same key. A new key represents a new operation and may be processed separately.

How Idempotency Keys Work

A typical idempotency workflow looks like this:

1. The client generates a unique key for an operation.

2. The client sends the key with the API request.

3. The server checks whether the key already exists.

4. If the operation has completed, the server returns the stored result.

5. If the key is new, the server records the operation and processes the request.

6. The server stores the result so that a retry can return the same outcome.

For example, suppose a customer submits a payment using key payment-123.

The first request processes the payment and stores the result. If the customer retries with payment-123, the API returns the existing result.

If the customer submits a genuinely different payment, the client should generate a different key.

Idempotency keys do not mean that every request to an endpoint should always return the same response. They apply to requests that represent the same logical operation.

Idempotency vs. HTTP Methods

HTTP defines certain methods, such as PUT and DELETE, as idempotent in terms of their intended effect. Repeating an identical request should have the same intended effect as sending it once.

However, POST requests commonly create resources or trigger operations, so repeating a request can create duplicates.

An idempotency key does not automatically make an endpoint safe. The server must store and enforce the key correctly.

Step 1: Create the Request Model

Start by defining the request model for an operation such as creating an order.

using System.ComponentModel.DataAnnotations;

public class CreateOrderRequest
{
    [Required]
    public string ProductCode { get; set; } = string.Empty;

    [Range(1, 1000)]
    public int Quantity { get; set; }
}

This model contains the information required to create an order.

The idempotency key is sent through an HTTP header rather than being placed inside the request body. This keeps the operation identifier separate from the business data.

For production applications, validate the request before processing it. Invalid requests should not create orders or reserve idempotency keys unnecessarily.

Step 2: Store Idempotency Records

The server needs to remember which keys have already been used.

For a production system, store idempotency records in a durable database rather than relying only on an in-memory dictionary.

A simplified SQL Server table might look like this:

CREATE TABLE IdempotencyRecords
(
    Id BIGINT IDENTITY(1,1) PRIMARY KEY,
    IdempotencyKey NVARCHAR(128) NOT NULL,
    RequestHash CHAR(64) NOT NULL,
    Status NVARCHAR(20) NOT NULL,
    ResponseBody NVARCHAR(MAX) NULL,
    CreatedAtUtc DATETIME2 NOT NULL
        DEFAULT SYSUTCDATETIME(),
    UpdatedAtUtc DATETIME2 NOT NULL
        DEFAULT SYSUTCDATETIME(),

    CONSTRAINT UQ_IdempotencyRecords_Key
        UNIQUE (IdempotencyKey)
);

The table stores the key, a hash of the request, the processing status, and the result.

The unique constraint is particularly important. It prevents two concurrent requests from creating separate records with the same key.

In a multi-tenant application, the uniqueness rule should generally include the appropriate client, tenant, or account scope. Otherwise, unrelated clients could collide on the same key.

A production implementation may also need an operation identifier, response status code, expiry policy, and additional metadata.

Step 3: Read and Validate the Idempotency Key

The API should reject requests that do not provide a valid key.

using Microsoft.AspNetCore.Mvc;

[ApiController]
[Route("api/orders")]
public class OrdersController : ControllerBase
{
    [HttpPost]
    public IActionResult CreateOrder(
        [FromHeader(Name = "Idempotency-Key")]
        string? idempotencyKey,
        [FromBody] CreateOrderRequest request)
    {
        if (string.IsNullOrWhiteSpace(idempotencyKey))
        {
            return BadRequest(new
            {
                Message = "The Idempotency-Key header is required."
            });
        }

        if (idempotencyKey.Length > 128)
        {
            return BadRequest(new
            {
                Message = "The idempotency key is too long."
            });
        }

        return Accepted(new
        {
            Message = "Request accepted for processing."
        });
    }
}

This example demonstrates header validation only. It does not yet implement deduplication or create an order.

The key length limit helps keep the input manageable. You should also define an allowed format and enforce the correct client or tenant scope.

In production, do not treat a request as successfully processed merely because the key passed validation.

Step 4: Handle Duplicate Requests Safely

A reliable implementation needs to handle more than a simple lookup.

Consider two requests arriving at almost the same time with the same key. Both might check the database before either has stored a record. If both continue independently, they could create duplicate orders.

A robust implementation should follow this general process:

1. Validate the request and calculate a hash of its relevant contents.

2. Attempt to create an idempotency record using a database uniqueness constraint.

3. If the key already exists, load the existing record.

4. Compare the stored request hash with the new request hash.

5. If the hashes differ, reject the request because the same key was used for different input.

6. If the original operation completed, return its stored result.

7. If the operation is still running, return a defined in-progress response or allow the client to check its status.

8. For a new operation, perform the business transaction and save the result reliably.

The request hash helps prevent accidental reuse of a key for different operations. Calculate it from a stable representation of the validated business input rather than relying on arbitrary JSON formatting.

For example, these requests must not silently become the same operation if the quantity or product differs.

{
  "productCode": "LAPTOP-01",
  "quantity": 1
}

{
  "productCode": "LAPTOP-01",
  "quantity": 5
}

If the same key is used for both requests, the server should detect the mismatch and reject the second request.

Use a database transaction

When the business operation and idempotency record live in the same database, a transaction can help ensure that they are committed together.

For example, creating an order and saving its completed result should not leave the system in a state where the order exists but the idempotency record incorrectly says that nothing happened.

However, a database transaction cannot make an external payment provider part of the same atomic transaction.

For external operations, use the provider's own idempotency mechanism when available, persist the operation state, and design recovery for failures that occur between the external call and the local database update.

Step 5: Return the Original Result

When a request is repeated with the same key and the same business input, the API should return the result associated with the original operation.

For example, the first successful request might return:

{
  "orderId": 501,
  "status": "Confirmed"
}

A retry with the same key should return the stored order result rather than create another order.

The API should define what happens when the original operation is still in progress. Depending on the design, it can return an appropriate conflict response, an accepted response with an operation identifier, or a status endpoint that the client can poll.

The response should distinguish a completed operation from one that has only been accepted for processing.

Best Practices for Production Applications

Generate a new key for each logical operation

The client should generate the key when the user initiates an operation and reuse it for retries of that operation.

Do not generate a new key for every network retry. Doing so defeats the purpose of deduplication.

Use durable storage

An in-memory dictionary cannot reliably prevent duplicates across multiple servers or application restarts.

Use a shared, durable store with an appropriate uniqueness constraint. Redis may help in some designs, but its persistence, eviction, and atomicity settings must match the reliability requirements.

Define a retention policy

Idempotency records do not always need to be stored forever. Decide how long keys and results must remain available based on client retry behavior, business rules, and provider requirements.

Do not expire records while clients may still legitimately retry an operation.

Protect sensitive response data

Stored responses may contain personal or business information. Apply suitable access controls, retention rules, and data protection measures.

Avoid logging raw idempotency keys alongside sensitive request bodies unless there is a clear operational need.

Make business operations idempotent where possible

Idempotency keys are one layer of protection. Database constraints, unique order references, payment-provider safeguards, and other business rules can provide additional protection against duplicate operations.

Test concurrent requests

Test scenarios where multiple requests arrive simultaneously with the same key. Verify that only one business operation is created and that the remaining requests receive the expected response.

Also test process restarts, database failures, and retries after timeouts.

Common Mistakes and Troubleshooting

One particularly important failure occurs when an external payment succeeds but the application crashes before recording the result. Retrying the payment blindly can create another charge. Use the payment provider's idempotency support and reconcile uncertain outcomes before attempting a new operation.

Advantages and Disadvantages

Advantages

• Reduces duplicate operations caused by client retries.

• Improves reliability when network responses are lost.

• Helps protect payment, order, and booking workflows.

• Allows clients to retry requests more safely.

• Provides a consistent way to identify repeated operations.

Disadvantages

• Requires additional storage and request-processing logic.

• Needs careful handling of concurrent requests.

• Requires retention and cleanup policies.

• Does not automatically guarantee exactly-once execution across external systems.

• Adds complexity to recovery and monitoring.

Summary

Idempotency keys help ASP.NET Core APIs handle retries without accidentally repeating important business operations.

The client sends the same key when retrying the same logical request. The server checks its durable record, validates that the request matches the original input, and returns the existing result when appropriate.

For production systems, use durable storage, enforce uniqueness at the database level, handle concurrent requests, and define what happens when an operation is still in progress or its outcome is uncertain.

For payments and other external operations, combine idempotency keys with provider-level safeguards and recovery logic. This is what makes the overall workflow reliable, rather than simply checking whether a key exists.