Building Reliable Background Jobs in ASP.NET Core With Idempotency and Retry Handling

Background jobs are common in business applications. An application may need to send thousands of emails, generate reports, process uploaded documents, synchronize customer records, or update external systems.

Executing all this work inside an HTTP request is rarely a good design. Long-running operations increase response times, consume request-handling resources, and can fail when a client disconnects or a request times out.

ASP.NET Core provides BackgroundService and IHostedService for running background processes. These services are useful for scheduled operations and in-process task execution, but building a reliable job-processing system requires more than starting a background loop.

A production system must handle duplicate messages, application restarts, temporary failures, concurrent workers, and jobs that fail repeatedly.

Why Background Jobs Need a Different Execution Model

Consider an order-processing application.

When a customer completes an order, the application may need to:

Save the order.
Reserve inventory.
Generate an invoice.
Send an order confirmation.
Notify an external accounting system.

Some operations can be completed quickly, but others may depend on external services.

If all operations execute inside the same HTTP request, a temporary email provider outage could delay the entire order response.

A better approach is to save the order and record the required background work. A worker can process the remaining operations independently.

The API can then return HTTP 202 Accepted when the work has been successfully accepted for asynchronous processing.

However, returning 202 does not mean the job has completed. The application needs a way to track its status and handle failures.

Choose the Right Background Processing Architecture

There are two common approaches in ASP.NET Core.

In-process background services

BackgroundService is suitable for work that runs within the lifetime of the application process.

For example:

Periodically refreshing reference data.
Cleaning temporary files.
Processing a small in-memory queue.
Running scheduled maintenance tasks.

A basic service looks like this:

public sealed class CleanupWorker : BackgroundService
{
    private readonly ILogger<CleanupWorker> _logger;

    public CleanupWorker(
        ILogger<CleanupWorker> logger)
    {
        _logger = logger;
    }

    protected override async Task ExecuteAsync(
        CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            try
            {
                _logger.LogInformation(
                    "Cleanup job started.");

                await CleanupTemporaryFilesAsync(
                    stoppingToken);

                await Task.Delay(
                    TimeSpan.FromHours(1),
                    stoppingToken);
            }
            catch (OperationCanceledException)
                when (stoppingToken.IsCancellationRequested)
            {
                break;
            }
            catch (Exception ex)
            {
                _logger.LogError(
                    ex,
                    "Cleanup job failed.");

                await Task.Delay(
                    TimeSpan.FromMinutes(1),
                    stoppingToken);
            }
        }
    }

    private Task CleanupTemporaryFilesAsync(
        CancellationToken cancellationToken)
    {
        // Perform cleanup operations.
        return Task.CompletedTask;
    }
}

builder.Services.AddHostedService<CleanupWorker>();

This implementation demonstrates a recurring task with basic exception handling.

It does not provide durable scheduling or guaranteed execution. If the process stops, the current operation can be interrupted. If the application restarts, the service begins again without knowing which in-memory operations were previously pending.

For critical business operations, an in-memory queue alone is insufficient.

Durable background job processing

A durable job-processing system stores jobs outside the worker process.

Common choices include:

Hangfire with a persistent database.
RabbitMQ with a durable message queue.
Azure Service Bus.
Amazon SQS.
A database-backed job table.

These systems allow work to survive application restarts and support multiple workers.

The appropriate choice depends on the workload. A scheduled reporting application may benefit from a job scheduler, while a high-volume event-processing system may need a dedicated message broker.

Use a Durable Job Record

For business-critical work, store the job before attempting to process it.

A job record might contain:

Field	Purpose
JobId	Unique job identifier
JobType	Operation to execute
Payload	Required input data
Status	Pending, Processing, Completed, or Failed
AttemptCount	Number of processing attempts
CreatedAt	Job creation time
NextAttemptAt	Next eligible execution time
LastError	Most recent failure information

A simplified SQL table could look like this:

CREATE TABLE BackgroundJobs
(
    JobId UNIQUEIDENTIFIER PRIMARY KEY,
    JobType NVARCHAR(100) NOT NULL,
    Payload NVARCHAR(MAX) NOT NULL,
    Status NVARCHAR(30) NOT NULL,
    AttemptCount INT NOT NULL DEFAULT 0,
    CreatedAt DATETIME2 NOT NULL,
    NextAttemptAt DATETIME2 NULL,
    LastError NVARCHAR(MAX) NULL
);

The application inserts a pending job into this table. A worker periodically finds eligible jobs and processes them.

The important part is that the job exists independently of the HTTP request and application memory.

Prevent Duplicate Processing With Idempotency

One of the most common problems in distributed job processing is duplicate execution.

Imagine that a worker sends an invoice to an external accounting service. The service successfully receives the invoice, but the network connection drops before the worker receives the response.

The worker cannot confidently determine whether the operation succeeded.

If it retries without checking, the accounting system may receive the same invoice twice.

This is where idempotency becomes important.

An operation is idempotent when repeating it with the same logical input does not create additional unintended effects.

A job can use a stable identifier as an idempotency key.

For example:

public async Task ProcessInvoiceAsync(
    Guid invoiceId,
    CancellationToken cancellationToken)
{
    var existingInvoice =
        await _dbContext.ProcessedInvoices
            .AnyAsync(
                x => x.InvoiceId == invoiceId,
                cancellationToken);

    if (existingInvoice)
    {
        return;
    }

    await _accountingClient.CreateInvoiceAsync(
        invoiceId,
        cancellationToken);

    _dbContext.ProcessedInvoices.Add(
        new ProcessedInvoice
        {
            InvoiceId = invoiceId,
            ProcessedAt = DateTime.UtcNow
        });

    await _dbContext.SaveChangesAsync(
        cancellationToken);
}

This example illustrates the intent, but it is not sufficient for concurrent production processing.

Two workers could both check for an existing record before either inserts it. Both might then execute the external operation.

A stronger implementation requires a database uniqueness constraint and an atomic claim or deduplication strategy. For external APIs, use the provider's idempotency-key mechanism whenever available.

For example, the same invoice identifier should be sent on every retry rather than generating a new identifier for each attempt.

Idempotency does not guarantee that a distributed operation happens only once. It makes repeated attempts safe when the participating systems support the necessary safeguards.

Implement Retry Handling With Backoff

Temporary failures are normal in distributed systems.

An external service may return HTTP 503, a database connection may temporarily fail, or a network request may time out.

Immediately retrying a failed operation can increase pressure on an already unhealthy service.

Use exponential backoff to increase the delay between attempts.

A simplified delay calculation is:

[
Delay = \min(MaxDelay,\ BaseDelay \times 2^{Attempt})
]

For example:

Attempt	Approximate delay
1	2 seconds
2	4 seconds
3	8 seconds
4	16 seconds
5	32 seconds

Add random jitter so that many workers do not retry at exactly the same time.

A basic C# implementation:

public static TimeSpan GetRetryDelay(int attempt)
{
    const int baseDelaySeconds = 2;
    const int maximumDelaySeconds = 300;

    var exponent = Math.Min(attempt - 1, 10);

    var calculatedDelay =
        baseDelaySeconds * Math.Pow(2, exponent);

    var cappedDelay = Math.Min(
        calculatedDelay,
        maximumDelaySeconds);

    var jitter = Random.Shared.NextDouble() * 0.25;

    return TimeSpan.FromSeconds(
        cappedDelay * (1 + jitter));
}

The retry policy should also distinguish transient failures from permanent failures.

For example:

HTTP 503 may be temporary.
A connection timeout may be temporary.
Invalid input is usually permanent.
A missing required customer record may require investigation.
An authentication failure may require credential correction.

Do not retry every exception indefinitely. Set a maximum attempt count and move permanently failing jobs into a failed or dead-letter state.

Control Concurrent Job Execution

A background worker should not process unlimited jobs simultaneously.

Suppose a report-generation service receives 5,000 jobs. If every job starts immediately, the application may exhaust database connections, memory, or CPU.

Limit concurrency according to the resources required by each job type.

For example, a worker might process four report-generation jobs at a time but allow more lightweight email jobs.

A simple semaphore can control concurrency within one process:

public sealed class JobProcessor
{
    private readonly SemaphoreSlim _semaphore =
        new(4);

    public async Task ProcessAsync(
        Func<CancellationToken, Task> operation,
        CancellationToken cancellationToken)
    {
        await _semaphore.WaitAsync(
            cancellationToken);

        try
        {
            await operation(cancellationToken);
        }
        finally
        {
            _semaphore.Release();
        }
    }
}

This protects a single process. If multiple worker instances are running, each instance has its own semaphore.

Global concurrency limits require coordination through a shared queue, distributed semaphore, database lease, or another shared mechanism.

Handle Job Ownership and Worker Crashes

When multiple workers read from the same job table, they must not process the same pending job simultaneously.

A common solution is to claim jobs atomically.

The database operation should select an eligible job and change its status to Processing in one transaction.

For SQL Server, a simplified claim operation can use locking hints:

;WITH NextJob AS
(
    SELECT TOP (1) *
    FROM BackgroundJobs WITH
        (UPDLOCK, READPAST, ROWLOCK)
    WHERE Status = 'Pending'
       OR (
           Status = 'Retry'
           AND NextAttemptAt <= SYSUTCDATETIME()
       )
    ORDER BY CreatedAt
)
UPDATE NextJob
SET
    Status = 'Processing',
    AttemptCount = AttemptCount + 1
OUTPUT inserted.*;

This example demonstrates atomic claiming for SQL Server. A complete implementation should also account for expired processing leases, transaction boundaries, and the application's isolation-level requirements.

What happens if a worker claims a job and then crashes?

Without recovery logic, the job may remain in Processing forever.

Use a lease or processing timestamp. If a job has been Processing longer than its allowed lease period, another worker can reclaim it.

The lease should be renewed for long-running jobs. Otherwise, a slow worker might continue processing after another worker has reclaimed the same job.

Idempotency is still necessary because leases cannot eliminate every duplicate execution scenario.

Use Dependency Injection Correctly

Hosted services are registered as singletons. They should not directly capture scoped dependencies such as Entity Framework Core DbContext instances.

Create a dependency injection scope for each job or processing batch.

public sealed class OrderWorker : BackgroundService
{
    private readonly IServiceScopeFactory _scopeFactory;

    public OrderWorker(
        IServiceScopeFactory scopeFactory)
    {
        _scopeFactory = scopeFactory;
    }

    protected override async Task ExecuteAsync(
        CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            using var scope =
                _scopeFactory.CreateScope();

            var processor = scope.ServiceProvider
                .GetRequiredService<IOrderProcessor>();

            await processor.ProcessNextAsync(
                stoppingToken);

            await Task.Delay(
                TimeSpan.FromSeconds(2),
                stoppingToken);
        }
    }
}

This example shows the scope lifetime. A real worker should avoid continuously polling an empty queue without backoff or a proper queue-notification mechanism.

Creating a scope per processing unit also prevents database contexts and other scoped services from being shared across unrelated jobs.

Monitor Job Processing in Production

A background job system needs operational visibility.

Logging only that a job started or completed is not enough. Track metrics that reveal delays, failures, and processing bottlenecks.

Useful measurements include:

Number of pending jobs.
Oldest pending job age.
Job processing duration.
Successful and failed job counts.
Retry frequency.
Jobs exceeding their execution timeout.
Dead-letter job count.
Worker availability.
Queue throughput.

The oldest pending job age is particularly useful. A queue may contain only a few jobs, but if one has been waiting for several hours, the system may still have a serious processing problem.

Use structured logs with JobId, JobType, AttemptCount, and correlation identifiers. Avoid logging sensitive job payloads or personal information.

Set alerts for growing queue depth, repeated failures, and jobs that remain in Processing beyond their expected duration.

Common Mistakes to Avoid

Treating an in-memory queue as durable storage.
Assuming a job executes exactly once.
Retrying permanent failures.
Allowing unlimited concurrent operations.
Sharing a scoped DbContext across background tasks.
Forgetting to recover jobs after worker crashes.
Returning HTTP 200 when asynchronous work has only been accepted.
Ignoring duplicate messages from external queues.
Logging errors without recording the job identifier.
Running long background tasks without cancellation and shutdown handling.

A reliable job-processing design assumes that interruptions and duplicate deliveries will happen. The system should recover without creating duplicate business operations.

Conclusion

Building background jobs in ASP.NET Core is straightforward with BackgroundService, but production reliability requires additional design decisions.

Use durable storage for important work, implement idempotency before adding retries, control concurrency, and recover jobs that become stuck after worker failures.

The most important principle is to separate job acceptance from job completion. Once an operation becomes asynchronous, the application must provide a reliable way to track, retry, and recover it.

That approach makes background processing easier to scale without making business operations dependent on the lifetime of an HTTP request or a single application instance.