Why Your API Retry Logic Can Make an Outage Worse

Retries become particularly dangerous when request timeouts are long.

Suppose a client has:

Request timeout: 30 seconds
Retries: 3

A single logical operation could potentially keep a connection or task active for a significant amount of time.

Now imagine thousands of concurrent requests.

The application may accumulate many in-flight operations:

Request 1 → waiting
Request 2 → waiting
Request 3 → waiting
...
Request 10,000 → waiting

When each request eventually retries, the number of active operations can become large.

Timeouts and retries should therefore be designed together.

A retry policy that looks reasonable in isolation may create excessive concurrency when combined with a long timeout.

Retry Only Operations That Are Safe to Repeat

Retrying a GET request is usually easier to reason about than retrying an operation that changes state.

Consider:

POST /payments

If the client sends the request and does not receive a response because of a network timeout, the client does not necessarily know whether the server processed the payment.

Retrying the POST could create a duplicate operation.

For example:

Client
  ↓
POST /payment
  ↓
Server processes payment
  ↓
Network timeout
  ↓
Client sees failure
  ↓
Client retries
  ↓
Second payment may be created

This is why retry design must consider idempotency.

Use Idempotency for Critical Operations

An idempotent operation can safely be repeated without causing an unintended additional effect.

For operations such as payments, order creation, or resource provisioning, an idempotency key can help.

For example:

POST /payments
Idempotency-Key: 7f3e9a1b

The server can associate that key with the original operation.

If the same request is received again:

Same idempotency key
        ↓
Server recognizes previous operation
        ↓
Return original result

This allows the client to retry without blindly creating another operation.

The exact implementation depends on the API, but the concept is important whenever a retry can repeat a business action.

Retries Can Overload Dependencies Too

The API itself may not be the real bottleneck.

Imagine this architecture:

Client
  ↓
API
  ↓
Database
  ↓
External service

If the external service becomes slow, the API may start timing out.

Clients retry the API.

The API then creates additional calls to the already struggling dependency.

The result can become:

Client retries
      ↓
API receives more traffic
      ↓
API calls dependency more often
      ↓
Dependency becomes slower
      ↓
API requests timeout
      ↓
Clients retry again

The retry policy therefore needs to be considered across the entire request path.

Avoid Nested Retry Policies

A particularly difficult problem occurs when multiple layers retry the same operation.

For example:

HTTP client → 3 retries
API service → 3 retries
Database client → 3 retries
External SDK → 3 retries

A single logical operation can generate many physical attempts.

The exact multiplication depends on where failures occur, but the important point is that retries at different layers can amplify each other.

If one operation is retried by several independent components, the total request volume can become much larger than expected.

Retry ownership should therefore be deliberate.

Do not add a retry policy to every layer simply because each layer can experience temporary failures.

Circuit Breakers Can Stop Repeated Calls

A circuit breaker is useful when a dependency is consistently failing.

Instead of continuing to send requests:

Request
  ↓
Failure
  ↓
Retry
  ↓
Failure
  ↓
Retry
  ↓
Failure

a circuit breaker can temporarily stop calls to the unhealthy dependency.

Conceptually:

Healthy
   ↓
Failures increase
   ↓
Circuit opens
   ↓
Calls are rejected or short-circuited
   ↓
Wait period
   ↓
Test dependency
   ↓
Resume if healthy

This gives the downstream service time to recover and prevents the caller from continuously generating traffic.

A circuit breaker is not a replacement for retries. The two patterns often work together.

Bulkheads Limit the Damage

Another useful resilience pattern is bulkheading.

The idea is to prevent one failing dependency from consuming all available application resources.

For example, suppose an API calls three external services:

Payment service
Shipping service
Email service

If the payment service becomes slow, unlimited concurrent calls to it could consume all available threads, connections, or tasks.

A bulkhead can limit the number of concurrent operations for that dependency.

This allows other parts of the application to continue operating.

The goal is not simply to make failed requests succeed. It is to prevent one dependency failure from taking down the entire application.

A Practical Retry Policy

A reasonable API client policy often includes several controls together:

Retry only transient failures
        ↓
Use exponential backoff
        ↓
Add jitter
        ↓
Respect Retry-After
        ↓
Limit attempts
        ↓
Use timeouts
        ↓
Protect non-idempotent operations
        ↓
Use circuit breaking where appropriate

Each control addresses a different failure mode.

For example:

5xx response
   ↓
Is retryable?
   ↓
Yes
   ↓
Check Retry-After
   ↓
Calculate backoff + jitter
   ↓
Retry within attempt limit

This is much safer than simply wrapping every API call in a retry loop.

Common Mistakes

Retrying Immediately

Immediate retries can multiply traffic during an outage.

Use controlled backoff instead.

Retrying Every Status Code

A 400 response usually will not become valid because the client sends the same request again.

Classify errors before retrying.

Ignoring Retry-After

If an API provides explicit retry guidance, clients should consider it.

Using Retries Without Idempotency

Repeating a state-changing request can create duplicate operations.

Use idempotency mechanisms for critical operations where appropriate.

Adding Retries at Every Layer

Multiple retry policies can multiply request volume.

Define where retries belong.

Using Long Timeouts With Many Retries

This can create large numbers of long-lived requests.

Timeout and retry settings should be designed together.

Treating Retries as the Main Reliability Strategy

Retries help with temporary failures. They do not fix overloaded services, persistent configuration problems, or broken dependencies.

Best Practices

Retry only failures that have a reasonable chance of succeeding later.

Use exponential backoff rather than immediate retries.

Add jitter so large groups of clients do not retry simultaneously.

Respect server-provided Retry-After information.

Keep retry counts bounded.

Use short, realistic timeouts.

Design retries around idempotency and business consequences.

Avoid overlapping retry policies across multiple layers.

Use circuit breakers and concurrency limits when dependencies remain unhealthy.

Monitor retry rates separately from normal request rates.

Track the total number of attempts, not just the number of logical requests.

What to Monitor

A service can appear healthy when looking only at successful requests.

For example:

Requests:       10,000
Successful:      9,500
Retries:         4,000

The service may have processed 13,500 physical requests to satisfy 10,000 logical operations.

That additional load matters.

Monitor metrics such as:

• Retry count

• Retry rate

• Attempts per request

• HTTP 429 responses

• HTTP 5xx responses

• Timeout rate

• Circuit breaker state

• Request latency

• Dependency latency

• Concurrent requests

A rising retry rate can be an early warning that a dependency is becoming unhealthy.

Summary

Retries can recover temporary failures, but poorly designed retry logic can amplify an outage.

Use bounded retries, exponential backoff, jitter, proper timeouts, and server retry guidance. Make sure state-changing operations are safe to repeat and avoid stacking independent retry policies across every layer.

The goal of retry logic is not to keep sending requests until something works. It is to give a failing system enough room to recover without making the failure worse.