Retries are one of those features that look harmless until a production workload starts failing.
A Lambda function calls an external API. The request times out. The function retries. The second attempt also fails. Another retry starts. Meanwhile, the original invocation may still be consuming resources, the upstream service may be recovering, and the workload may be generating additional requests.
For .NET applications running on AWS Lambda, retries can improve reliability, but they can also increase execution cost, amplify downstream traffic, extend latency, and make an outage worse.
The right approach is not to remove retries. It is to make them cost-aware, failure-aware, and workload-specific.
This article explains how to design retry policies for .NET Lambda workloads, how to calculate their approximate cost impact, how to combine exponential backoff with jitter, and how to decide when a retry should happen at all.
Why Retries Become Expensive
Consider a Lambda function that normally completes in 500 milliseconds.
Without retries:
Invocation
|
v
External API
|
v
Success
With three retries:
Invocation
|
+--> Attempt 1 --> Failure
|
+--> Attempt 2 --> Failure
|
+--> Attempt 3 --> Failure
|
+--> Attempt 4 --> Success
The function may remain active for significantly longer.
There are also costs outside Lambda execution itself. Every retry can produce another network request, another downstream operation, another log entry, and potentially another database or service interaction.
This creates a useful production principle:
A retry should have an expected reliability benefit that is greater than its additional latency and resource cost.
Start With Failure Classification
Not every failure deserves a retry.
For example, retrying a validation error is usually pointless:
HTTP 400
Invalid request
Invalid JSON
Missing required field
Authentication failure
Authorization failure
The same applies to many application-level exceptions.
Transient failures are different:
HTTP 429
HTTP 502
HTTP 503
HTTP 504
Network timeout
Temporary connection failure
These may succeed if the operation is attempted again after a short delay.
A basic policy therefore starts with classification:
| Failure | Retry? | Typical Reason |
|---|---|---|
| Validation error | No | Request will fail again |
| Authentication failure | Usually no | Credentials must change |
| Authorization failure | No | Permission must change |
| Not found | Usually no | Resource is missing |
| Rate limit | Yes, carefully | Service may recover |
| Timeout | Often | Could be transient |
| 502 | Often | Upstream failure |
| 503 | Often | Temporary unavailability |
| 504 | Often | Gateway timeout |
| Network interruption | Often | Connection may recover |
The exact policy should depend on the API contract.
Understand the Lambda Cost Model
AWS Lambda pricing depends on factors such as request count, configured memory, and execution duration. Additional services used by the function can introduce their own charges.
For retry design, the important observation is simple:
More attempts can mean more execution time.
Suppose a function uses 1 GB of memory and normally executes for 500 ms.
A successful request might consume roughly:
1 GB × 0.5 seconds
of Lambda compute duration.
If the same invocation performs four 500 ms attempts, the execution duration can approach:
1 GB × 2 seconds
before considering backoff delays and other work.
The exact bill depends on the Lambda pricing model and architecture, so benchmark and calculate using the current pricing applicable to your account and region rather than hardcoding a universal cost number.
The engineering lesson remains the same: retry count and execution duration are connected.
Use Exponential Backoff
A retry policy should not immediately repeat the failed request.
This pattern is dangerous:
for (var attempt = 1; attempt <= 5; attempt++)
{
try
{
return await CallServiceAsync(cancellationToken);
}
catch
{
// Retry immediately
}
}
If hundreds of Lambda invocations experience the same transient failure, they can all retry immediately.
That creates a retry storm.
Exponential backoff increases the delay between attempts.
A simple model is:
delay = baseDelay × 2^(attempt - 1)
For example:
Attempt 1 -> 200 ms
Attempt 2 -> 400 ms
Attempt 3 -> 800 ms
Attempt 4 -> 1.6 s
The exact values should depend on the downstream service.
Add Jitter
Exponential backoff alone is not enough when many Lambda invocations fail at approximately the same time.
Imagine 1,000 invocations receiving a 503.
Without jitter:
0 ms -> 1,000 requests
500 ms -> 1,000 retries
1,000 ms -> 1,000 retries
2,000 ms -> 1,000 retries
The downstream service continues receiving synchronized bursts.
Jitter introduces controlled randomness:
Invocation A -> 430 ms
Invocation B -> 517 ms
Invocation C -> 468 ms
Invocation D -> 592 ms
The traffic becomes distributed over time.
A common implementation is:
private static TimeSpan GetRetryDelay(
int attempt,
TimeSpan baseDelay,
TimeSpan maxDelay)
{
var exponentialDelay =
TimeSpan.FromMilliseconds(
baseDelay.TotalMilliseconds *
Math.Pow(2, attempt - 1));
var jitter = Random.Shared.NextDouble() * 250;
var totalDelay =
exponentialDelay.TotalMilliseconds + jitter;
return TimeSpan.FromMilliseconds(
Math.Min(totalDelay, maxDelay.TotalMilliseconds));
}
The important part is not the exact jitter algorithm. The important part is avoiding synchronized retries.
Keep the Maximum Retry Count Small
More retries do not automatically mean better reliability.
Suppose an upstream service is unavailable for 60 seconds.
A Lambda function that retries four times within a few seconds is unlikely to benefit from retrying again.
Instead, the request should usually fail and allow a higher-level mechanism to decide what happens next.
For example:
var maxAttempts = 3;
This is often a better starting point than:
var maxAttempts = 10;
The correct number depends on the operation's importance, latency budget, downstream recovery time, and whether another system will retry the work later.
Separate Lambda Retries From Application Retries
This is one of the most important design considerations.
A Lambda invocation can be retried by the surrounding event-processing architecture, while the application code can also retry the downstream request.
That can multiply attempts.
Imagine:
Event source
|
+--> Lambda invocation
|
+--> HTTP attempt 1
+--> HTTP attempt 2
+--> HTTP attempt 3
If the Lambda invocation itself is later retried:
Lambda invocation #1
3 downstream attempts
Lambda invocation #2
3 downstream attempts
The downstream service may receive six attempts for one logical event.
This is retry multiplication.
Calculate the Worst Case
Suppose:
Lambda-level attempts = 3
HTTP-level attempts = 3
The theoretical maximum downstream attempts can become:
3 × 3 = 9
If another layer also retries, the number can grow further.
This is why retry ownership should be explicit.
A good architecture decides which layer is responsible for recovering from which failure.
For example:
Application retry
-> short-lived transient HTTP failures
Event processing retry
-> longer-lived processing failures
This separation avoids making every layer independently retry everything.
Build a Cost-Aware Retry Policy
A retry policy should consider at least four variables:
Failure type
Retry count
Delay
Operation cost
For example:
public sealed record RetryPolicy(
int MaxAttempts,
TimeSpan BaseDelay,
TimeSpan MaxDelay);
Then:
var policy = new RetryPolicy(
MaxAttempts: 3,
BaseDelay: TimeSpan.FromMilliseconds(200),
MaxDelay: TimeSpan.FromSeconds(5));
This keeps policy configuration separate from the execution logic.
Implement the Retry Loop Carefully
A production-oriented implementation might look like:
public async Task<T> ExecuteAsync<T>(
Func<CancellationToken, Task<T>> operation,
CancellationToken cancellationToken)
{
const int maxAttempts = 3;
var baseDelay = TimeSpan.FromMilliseconds(200);
var maxDelay = TimeSpan.FromSeconds(5);
for (var attempt = 1; attempt <= maxAttempts; attempt++)
{
try
{
return await operation(cancellationToken);
}
catch (HttpRequestException) when (attempt < maxAttempts)
{
var delay = GetRetryDelay(
attempt,
baseDelay,
maxDelay);
await Task.Delay(delay, cancellationToken);
}
}
throw new InvalidOperationException(
"The operation could not be completed.");
}
In a real application, exception classification should be more precise than catching every HttpRequestException.
For example, a permanent error should not be retried simply because it arrived through an HTTP client.

Join the conversation! Your thoughts help the community grow.