Finding the Right Boundary Between Application Retries and Database Retries

Retries are a common way to handle temporary failures in distributed applications. A network request may time out, a database connection may briefly fail, or a service may return a temporary error. Retrying the operation can allow the application to recover without requiring manual intervention.

However, retries can also make an incident worse. When an ASP.NET Core API retries a database operation, and the database client or another layer retries the same operation independently, one request can trigger many attempts. Under heavy load, those attempts can consume connections, increase latency, and put additional pressure on an already struggling database.

The important question is not simply whether to retry. It is which layer should own the retry, what failures are safe to retry, and how should the other layers respond?

This article explains how to choose between application-level and database-level retries, configure Entity Framework Core execution strategies, handle transient failures, and prevent retry logic from creating duplicate operations.

Understanding the Different Retry Boundaries

Retries can exist at several layers of an application. Each layer has a different view of the operation and a different understanding of whether repeating it is safe.

Layer	Typical responsibility	Example
Application service	Retry a complete business operation when its state and side effects are understood	Reattempt an operation after checking its idempotency
Database provider	Retry supported transient database failures	Reconnect after a temporary connection failure
HTTP client	Retry eligible requests to a remote service	Retry a temporary service-unavailable response
Job processor	Retry a failed background task	Reschedule a message after a transient database error

These boundaries should not be treated as interchangeable.

A database provider can recognize certain database-specific transient errors. It cannot necessarily determine whether the entire business operation is safe to repeat.

Likewise, an application service understands the business operation, but it should not blindly retry every SQL exception or network failure.

A good design assigns retry responsibility according to the scope of the operation and the information needed to determine whether repeating it is safe.

Why Multiple Retry Layers Can Cause Problems

Consider an API that saves an order using Entity Framework Core. The application has an outer retry policy with three attempts, while the database execution strategy can make three attempts for an individual operation.

If the outer policy repeats the database operation and every execution-strategy attempt fails, the total number of database attempts can multiply.

For example, with three attempts at each of two nested layers, the worst-case number of attempts can reach nine, assuming both layers exhaust their configured attempts and every outer attempt invokes the inner retry policy.

The result can be more database traffic precisely when the database is already experiencing problems.

Other risks include:

Longer request durations caused by repeated delays.
More connections and database resources consumed by concurrent requests.
Duplicate records when the first attempt succeeded but its response was lost.
Repeated external side effects, such as sending a payment request twice.
Transaction failures when retry logic reuses invalid or completed transaction state.

The solution is not to eliminate every retry. It is to avoid overlapping retry policies that do not coordinate their responsibilities.

Step 1: Classify the Failure Before Retrying

Not every failure is transient. Before adding retry logic, identify which failures can reasonably recover without changing the request.

Common categories include the following.

Transient Failures

These may recover after a short delay:

Temporary network interruptions.
Brief database connectivity failures.
Database throttling or temporary resource pressure.
Deadlocks that cause a transaction to be selected as the victim.
Temporary service unavailability.

The correct response depends on the database provider, exception type, operation, and system configuration.

Permanent Failures

These generally require a change to the request or application rather than another attempt:

Invalid input.
Foreign key violations caused by missing related records.
Unique constraint violations caused by conflicting business data.
SQL syntax errors.
Missing database objects caused by an incorrect deployment.

Retrying a permanent error wastes resources and delays the actual error response.

Ambiguous Outcomes

Some failures are harder to classify. A connection can fail after the database has committed a transaction but before the application receives confirmation.

In this situation, the application may not know whether the operation succeeded.

Blindly repeating the operation can create duplicates. For operations with important business consequences, use idempotency keys, unique constraints, durable operation records, or another mechanism that allows the application to determine whether the work has already been completed.

Step 2: Use EF Core Execution Strategies for Transient Database Failures

Entity Framework Core database providers can supply execution strategies that retry supported transient failures. The behavior and configuration depend on the provider.

For SQL Server, enable the provider's retry-on-failure option in Program.cs:

using Microsoft.EntityFrameworkCore;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddDbContext<AppDbContext>(options =>
    options.UseSqlServer(
        builder.Configuration.GetConnectionString("Default"),
        sqlServerOptions =>
        {
            sqlServerOptions.EnableRetryOnFailure(
                maxRetryCount: 3,
                maxRetryDelay: TimeSpan.FromSeconds(5),
                errorNumbersToAdd: null);
        }));

var app = builder.Build();

app.MapControllers();

app.Run();

This configuration enables the SQL Server provider's retrying execution strategy for supported transient failures. The exact errors considered transient are provider-specific.

The example sets a maximum of three retries and a maximum retry delay of five seconds. These values are illustrative starting points, not universal production defaults. Select them according to request deadlines, database behavior, and the application's latency requirements.

What the Execution Strategy Does

The execution strategy can repeat supported database operations when a recognized transient failure occurs. This can be useful for operations that do not require application-specific recovery decisions.

However, enabling retries does not make every database operation safe to repeat. The application still needs to account for transaction boundaries, ambiguous commit outcomes, and external side effects.

For PostgreSQL or another database provider, use the corresponding provider's documented retry mechanism rather than assuming SQL Server configuration applies.

Step 3: Handle Transactions Correctly

Transactions require special attention when using EF Core retrying execution strategies.

When retries are enabled, manually controlled transactions must be executed within the execution strategy's retry boundary. Otherwise, EF Core can reject the operation because it cannot safely retry a transaction that it does not control.

For example:

using Microsoft.EntityFrameworkCore;

public sealed class OrderService
{
    private readonly AppDbContext _dbContext;

    public OrderService(AppDbContext dbContext)
    {
        _dbContext = dbContext;
    }

    public async Task CreateOrderAsync(
        Order order,
        CancellationToken cancellationToken)
    {
        var strategy = _dbContext.Database
            .CreateExecutionStrategy();

        await strategy.ExecuteAsync(async () =>
        {
            await using var transaction =
                await _dbContext.Database
                    .BeginTransactionAsync(cancellationToken);

            _dbContext.Orders.Add(order);

            await _dbContext.SaveChangesAsync(
                cancellationToken);

            await transaction.CommitAsync(
                cancellationToken);
        });
    }
}

This example illustrates the transaction boundary, but it needs an important safeguard before being used as a production implementation: the same order entity is reused if the execution strategy repeats the delegate, and the commit outcome may be ambiguous.

A safer design should create or restore the operation's state for each attempt and provide a way to verify whether a previous attempt committed successfully. EF Core also provides transaction execution patterns that support verification of a successful commit when a connection failure makes the result uncertain.

Do not assume that placing BeginTransactionAsync, SaveChangesAsync, and CommitAsync inside a retry delegate automatically solves ambiguous outcomes or makes the operation idempotent.

Keep External Side Effects Outside Database Retry Delegates

A database execution strategy may execute its delegate more than once. Sending an email, charging a payment method, or publishing a message directly inside that delegate can repeat the external action.

For example, this pattern is unsafe:

await strategy.ExecuteAsync(async () =>
{
    await using var transaction =
        await dbContext.Database.BeginTransactionAsync();

    dbContext.Orders.Add(order);

    await dbContext.SaveChangesAsync();

    // Unsafe: this can run more than once.
    await paymentClient.ChargeAsync(paymentRequest);

    await transaction.CommitAsync();
});

The payment request is an external side effect. If the payment succeeds but the database transaction later fails, the retry may submit the payment again.

A safer design records the business operation in the database and uses an idempotent payment API or a durable messaging pattern to coordinate external work. For workflows that span services, an outbox or saga may be appropriate, depending on the required consistency guarantees.

Step 4: Decide When Application-Level Retries Are Necessary

Application-level retries are useful when the application needs to make a decision beyond the database provider's transient-error classification.

For example, a background job might retry a complete operation after checking whether a previous attempt already completed. A service might also retry a remote API call using a documented retry policy for temporary failures.

Before implementing an application-level retry, answer these questions:

Is the failure likely to be temporary?
Is the operation safe to repeat?
Can the application determine whether a previous attempt succeeded?
Does the retry fit within the request or job deadline?
Will another layer also retry the same operation?

If the database provider already retries supported transient database failures, the application should generally avoid wrapping the same database call in another broad retry policy.

Instead, let the provider handle the database-specific retry behavior and reserve application-level retries for operations that require business-level decisions.

Example: Retry a Background Job at the Job Boundary

A background job processor may retry a job after a temporary database failure. The job should be designed so that repeating it does not create duplicate business effects.

For example, a job can record a unique operation identifier and enforce uniqueness in the database. Before performing work, it checks the durable operation state. A database uniqueness constraint provides protection even when two attempts run concurrently.

The job processor should also use bounded retries and a dead-letter or failed-job workflow for operations that repeatedly fail.

Do not configure unlimited retries. A permanent failure should eventually become visible for investigation rather than being retried forever.

Step 5: Use Exponential Backoff and Jitter

Retry timing matters when many requests fail at the same time.

A fixed delay can cause many clients to retry simultaneously. This creates a retry wave that adds load to an already unhealthy database or service.

Exponential backoff increases the delay between attempts. Jitter adds a random component to reduce synchronized retries.

A simple delay model is:

[
d_n = \min(d_{\max}, d_0 \times 2^{n-1})
]

Where:

(d_n) is the delay before attempt (n).
(d_0) is the initial delay.
(d_{\max}) is the maximum delay.

For example, an initial delay of 200 milliseconds might grow to 400 milliseconds and then 800 milliseconds, subject to a configured cap.

In production, use a well-tested retry implementation that supports jitter, cancellation, and the required retry conditions. The formula alone is not a complete retry policy.

Also consider the total time budget. If a request has a two-second deadline, a retry policy that can sleep for several seconds is not useful. The policy must fit within the remaining request deadline, and cancellation should stop unnecessary work.

Step 6: Avoid Retrying Errors That Are Not Transient

A retry policy should be selective. Retrying every exception can hide bugs and increase load.

For example, a unique constraint violation usually means the attempted data conflicts with existing data. Repeating the same insert with the same values is unlikely to help.

Likewise, a database command that consistently fails because a required table does not exist needs a deployment or schema fix, not another attempt.

Use the provider's supported transient-error classification where possible. If implementing additional application-level policies, explicitly define which errors qualify and test those conditions.

Be careful with timeouts. A timeout does not always mean that the operation failed. The server may have completed the work even though the client did not receive a response. For writes and external side effects, determine whether the operation can be safely repeated before retrying.

Step 7: Monitor Retry Behavior

Retries can improve resilience, but they should remain visible in production telemetry.

Track metrics such as:

Number of retry attempts.
Retry success rate.
Database command failures.
Connection failures and timeouts.
Request latency and deadline cancellations.
Exhausted retry policies.
Duplicate-operation prevention events.

A rising retry count can indicate a deeper problem, such as connection pool exhaustion, database overload, a failing network path, or an incorrectly configured timeout.

Avoid logging sensitive query parameters or customer data while recording retry details. Include enough context to identify the operation and failure category without exposing confidential information.

Common Mistakes to Avoid

Nesting Broad Retry Policies

Application-level and database-level retries can multiply attempts. Assign clear responsibilities to each layer and calculate the worst-case number of attempts.

Retrying Non-Transient Errors

Invalid data, constraint violations, and programming errors generally need correction rather than repeated execution.

Reusing Invalid Transaction State

A failed transaction or disposed connection may not be reusable. Keep transaction creation and cleanup inside the appropriate retry boundary.

Ignoring Ambiguous Commit Outcomes

A connection failure during commit does not prove that the transaction failed. Use idempotency or commit verification when repeating the operation could produce duplicate effects.

Retrying Without a Deadline

Retries can cause requests to exceed their latency budgets. Use bounded attempts, appropriate delays, cancellation, and an overall deadline.

Repeating External Side Effects

Do not place non-idempotent payment, email, or message-publishing operations inside a database retry delegate without a reliable coordination strategy.

Application Retries vs. Database Retries: Practical Guidelines

Situation	Preferred approach
Supported transient database error	Use the database provider's execution strategy
Invalid SQL or constraint violation	Fix the underlying error; do not retry blindly
Database transaction with ambiguous commit	Use idempotency and commit verification
Remote HTTP service temporarily unavailable	Use a bounded HTTP retry policy when the operation is safe to repeat
Background job fails temporarily	Use the job framework's retry mechanism with idempotent processing
Workflow spans a database and external service	Use durable coordination, such as an outbox or saga where appropriate
Multiple retry layers are already configured	Review ownership, attempt multiplication, and total latency

Summary

The right retry boundary depends on the failure type and the scope of the operation. Database execution strategies are appropriate for supported transient database failures, while application-level retries are useful when the application must evaluate business state, coordinate dependencies, or repeat a complete operation safely.

Avoid overlapping retry policies, use bounded backoff, respect cancellation and deadlines, and distinguish transient errors from permanent failures. For transactions and external side effects, account for ambiguous outcomes and enforce idempotency where necessary.

A reliable retry strategy does more than repeat failed operations. It defines when another attempt is worthwhile, limits the cost of failure, and prevents recovery logic from creating additional problems.