Handling Background Job Failures Without Stopping .NET BackgroundService

Sophia Morales

Background services are commonly used in .NET applications to process queued messages, send notifications, synchronize data, generate reports, and perform scheduled maintenance tasks.

Unlike an ordinary API request, a background service may run continuously for hours or days. A single unhandled exception can therefore have a much wider impact than a failed request.

For example, imagine a worker that checks a database every few seconds and sends pending email notifications. If one email fails because an external provider is temporarily unavailable, the worker should not necessarily stop processing every other notification.

The goal is to handle failures at the right level. A failure in one job should not automatically terminate the entire background service, while serious infrastructure or application failures should still be visible and handled deliberately.

This article explains how to structure BackgroundService implementations, isolate job failures, handle cancellation, implement retries, and monitor background processing in production.

1. Understand Why BackgroundService Can Stop Unexpectedly

In ASP.NET Core, BackgroundService provides a base class for implementing long-running background operations.

A typical implementation looks like this:

public class NotificationWorker : BackgroundService
{
    private readonly ILogger<NotificationWorker> _logger;

    public NotificationWorker(
        ILogger<NotificationWorker> logger)
    {
        _logger = logger;
    }

    protected override async Task ExecuteAsync(
        CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            _logger.LogInformation(
                "Checking pending notifications");

            await Task.Delay(
                TimeSpan.FromSeconds(10),
                stoppingToken);
        }
    }
}

The ExecuteAsync method represents the lifetime of the background service.

If an exception escapes from this method, the hosting infrastructure observes the failure.

Starting with .NET 6, the default BackgroundService exception behavior is StopHost. This means an unhandled exception in a background service causes the host to stop.

This behavior is intentional. It prevents an application from appearing healthy while an important background service has silently stopped working.

Consider the following example:

protected override async Task ExecuteAsync(
    CancellationToken stoppingToken)
{
    while (!stoppingToken.IsCancellationRequested)
    {
        await ProcessNotificationsAsync(stoppingToken);

        await Task.Delay(
            TimeSpan.FromSeconds(10),
            stoppingToken);
    }
}

If ProcessNotificationsAsync throws an exception, the loop exits. The exception reaches the hosting infrastructure, and the host can stop.

The important distinction is that the host does not automatically restart the failed BackgroundService when an exception occurs.

If you want the worker to continue after an individual job fails, you must handle that failure inside the worker.

2. Handle Exceptions at the Individual Job Level

The first improvement is to isolate exceptions around the operation that can fail.

For example, a notification worker should not stop processing all notifications because one notification fails.

protected override async Task ExecuteAsync(
    CancellationToken stoppingToken)
{
    while (!stoppingToken.IsCancellationRequested)
    {
        try
        {
            await ProcessNotificationsAsync(stoppingToken);
        }
        catch (OperationCanceledException)
            when (stoppingToken.IsCancellationRequested)
        {
            break;
        }
        catch (Exception ex)
        {
            _logger.LogError(
                ex,
                "An error occurred while processing notifications");
        }

        await Task.Delay(
            TimeSpan.FromSeconds(10),
            stoppingToken);
    }
}

This implementation catches unexpected exceptions from the processing operation and allows the loop to continue.

However, it is important to understand what this code actually protects.

If ProcessNotificationsAsync processes 100 notifications internally and fails on the 50th notification, the remaining notifications may not be processed during that iteration.

Catching the exception keeps the worker alive, but it does not automatically recover the failed operation or continue from the exact point of failure.

For better isolation, handle exceptions around individual jobs.

3. Process Each Job Independently

A more reliable design separates retrieving jobs from processing them.

Imagine a system that retrieves pending email notifications from a database.

You can process each notification independently:

public async Task ProcessNotificationsAsync(
    CancellationToken cancellationToken)
{
    var notifications = await GetPendingNotificationsAsync(
        cancellationToken);

    foreach (var notification in notifications)
    {
        cancellationToken.ThrowIfCancellationRequested();

        try
        {
            await SendNotificationAsync(
                notification,
                cancellationToken);

            await MarkAsCompletedAsync(
                notification.Id,
                cancellationToken);
        }
        catch (OperationCanceledException)
            when (cancellationToken.IsCancellationRequested)
        {
            throw;
        }
        catch (Exception ex)
        {
            _logger.LogError(
                ex,
                "Failed to process notification {NotificationId}",
                notification.Id);

            await MarkAsFailedAsync(
                notification.Id,
                ex.Message,
                cancellationToken);
        }
    }
}

Now, a failure in one notification does not prevent the worker from processing the next notification.

This approach provides several advantages:

Individual failures are isolated.
Successful jobs can be marked as completed.
Failed jobs can be recorded for later processing.
Logs can identify the exact job that failed.
The background service can continue processing other jobs.

For production systems, recording failure information is important. Simply catching an exception and continuing can hide data-processing problems.

A job should have a clearly defined state, such as Pending, Processing, Completed, or Failed.

4. Implement Controlled Retries for Transient Failures

Not every failure requires manual intervention.

Some failures are temporary. An external API might return a timeout, a database connection might temporarily become unavailable, or a network request might fail because of a short-lived connectivity problem.

Retrying these operations can help.

However, retries should be limited and controlled.

Consider this example:

public async Task SendWithRetryAsync(
    Notification notification,
    CancellationToken cancellationToken)
{
    const int maxAttempts = 3;

    for (int attempt = 1; attempt <= maxAttempts; attempt++)
    {
        try
        {
            await _emailClient.SendAsync(
                notification,
                cancellationToken);

            return;
        }
        catch (HttpRequestException ex)
            when (attempt < maxAttempts)
        {
            _logger.LogWarning(
                ex,
                "Notification attempt {Attempt} failed",
                attempt);

            var delay = TimeSpan.FromSeconds(
                Math.Pow(2, attempt));

            await Task.Delay(delay, cancellationToken);
        }
    }

    throw new InvalidOperationException(
        "Notification delivery failed after all retry attempts.");
}

This example uses exponential backoff.

The delay increases between attempts:

Attempt	Delay
1	2 seconds
2	4 seconds
3	No additional retry

The example is intentionally simple. Production retry policies should also consider:

Whether the failure is transient.
HTTP status codes and Retry-After headers.
Maximum retry duration.
Randomized delay, also called jitter.
Whether the operation is safe to repeat.
Cancellation during retry delays.

Avoid retrying every exception. For example, an invalid email address or an authentication failure will not usually be fixed by repeating the same request.

For more advanced retry and timeout handling, .NET applications can use the Microsoft.Extensions.Http.Resilience package with HttpClient.

5. Configure BackgroundService Exception Behavior Carefully

.NET provides BackgroundServiceExceptionBehavior through HostOptions.

It supports two values:

StopHost: Stops the host when an unhandled BackgroundService exception occurs.
Ignore: Logs the exception but does not stop the host or restart the failed service.

You can configure it as follows:

builder.Services.Configure<HostOptions>(options =>
{
    options.BackgroundServiceExceptionBehavior =
        BackgroundServiceExceptionBehavior.Ignore;
});

This changes the host's response to an unhandled exception.

However, there is an important limitation.

Setting Ignore does not restart the BackgroundService. If ExecuteAsync has already completed because of an exception, the worker does not automatically start again.

The application might continue running while the background worker is no longer processing anything.

This can create a dangerous situation: the API remains available, but its background processing has stopped.

For this reason, Ignore should not be treated as a replacement for proper exception handling or worker recovery.

For most applications, keep the default StopHost behavior and handle recoverable job failures inside the worker.

If the entire worker fails because of a serious infrastructure or application error, stopping the host can be safer than silently continuing without the required background processing.

6. Use IServiceScopeFactory for Scoped Dependencies

BackgroundService instances are generally registered as hosted services and managed as long-lived objects.

However, many application services, such as DbContext, are registered with a scoped lifetime.

Injecting a scoped dependency directly into a long-lived background service can cause lifetime problems.

Use IServiceScopeFactory to create a new dependency injection scope for each processing cycle.

public class NotificationWorker : BackgroundService
{
    private readonly IServiceScopeFactory _scopeFactory;
    private readonly ILogger<NotificationWorker> _logger;

    public NotificationWorker(
        IServiceScopeFactory scopeFactory,
        ILogger<NotificationWorker> logger)
    {
        _scopeFactory = scopeFactory;
        _logger = logger;
    }

    protected override async Task ExecuteAsync(
        CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            try
            {
                await using var scope =
                    _scopeFactory.CreateAsyncScope();

                var processor = scope.ServiceProvider
                    .GetRequiredService<INotificationProcessor>();

                await processor.ProcessAsync(stoppingToken);
            }
            catch (OperationCanceledException)
                when (stoppingToken.IsCancellationRequested)
            {
                break;
            }
            catch (Exception ex)
            {
                _logger.LogError(
                    ex,
                    "Notification processing failed");
            }

            try
            {
                await Task.Delay(
                    TimeSpan.FromSeconds(10),
                    stoppingToken);
            }
            catch (OperationCanceledException)
                when (stoppingToken.IsCancellationRequested)
            {
                break;
            }
        }
    }
}

The scope is disposed after each processing cycle.

This allows scoped services, including database contexts, to be created and released correctly.

It also helps prevent a long-running worker from holding onto the same DbContext indefinitely.

7. Handle Graceful Shutdown and Cancellation Correctly

Background services should respond to cancellation when the application is shutting down.

The stoppingToken passed to ExecuteAsync is triggered when the host requests the service to stop.

A common mistake is treating every OperationCanceledException as an application failure.

For example:

try
{
    await ProcessNotificationsAsync(stoppingToken);
}
catch (Exception ex)
{
    _logger.LogError(ex, "Background processing failed");
}

This catches cancellation exceptions too, potentially recording normal shutdown as an error.

Instead, distinguish application shutdown from unexpected cancellation:

try
{
    await ProcessNotificationsAsync(stoppingToken);
}
catch (OperationCanceledException)
    when (stoppingToken.IsCancellationRequested)
{
    _logger.LogInformation(
        "Notification worker is stopping");
}
catch (Exception ex)
{
    _logger.LogError(
        ex,
        "Notification processing failed");
}

A graceful shutdown should allow the worker to stop taking new jobs and finish or safely abandon the current operation within the configured shutdown period.

For long-running jobs, consider checkpoints or durable job states so that interrupted work can be resumed safely.

Do not assume StopAsync will always execute. Unexpected process termination, machine failure, or forced container shutdown can prevent graceful cleanup.

8. Avoid Duplicate Processing When Jobs Fail

Handling exceptions is only part of the problem.

Imagine a worker sends an email successfully but fails before marking the notification as completed.

When the worker runs again, it may send the same email a second time.

This is a duplicate-processing problem.

A reliable job-processing system should use a durable state transition.

For example:

Pending
   |
   v
Processing
   |
   +---- Success ----> Completed
   |
   +---- Failure ----> Failed
                         |
                         v
                    Retry Pending

A production implementation should consider:

Marking jobs as Processing before executing them.
Recording processing attempts.
Setting a lease expiration for jobs that become stuck.
Using idempotency keys for external operations when supported.
Preventing multiple workers from claiming the same job.
Moving repeatedly failing jobs into a dead-letter or manual-review state.

Database updates that claim jobs should be atomic. For example, a worker can use a transaction or an appropriate database-specific locking strategy to ensure that two workers do not process the same record simultaneously.

Exactly-once execution is difficult to guarantee across a database and an external service. Design jobs to tolerate retries and duplicate delivery instead.

9. Monitor Background Worker Health

A background service can remain running while its actual work is failing.

For example, the worker may successfully execute its loop every ten seconds but fail to process every notification because the external email provider is unavailable.

Checking only whether the process is running is not enough.

Monitor useful operational indicators:

Metric	Why it matters
Jobs processed	Measures actual worker activity
Successful jobs	Shows completed work
Failed jobs	Identifies processing problems
Retry attempts	Highlights temporary service failures
Queue depth	Shows whether work is accumulating
Oldest pending job age	Identifies processing delays
Processing duration	Helps detect slow operations
Last successful execution	Shows whether the worker is making progress

You can also implement a heartbeat or last-successful-processing timestamp.

For example, update a health record after a successful processing cycle. A monitoring system can alert when the worker has not completed work within an expected interval.

Health checks should distinguish between process availability and background processing health.

An application can be healthy enough to serve HTTP requests while a critical background worker is unhealthy. Exposing that distinction helps operations teams respond before a queue grows too large.

10. Common Mistakes to Avoid

Mistake	Why it causes problems	Better approach
Catching exceptions without recording them	Failures become difficult to investigate	Log job identifiers and exception details
Setting BackgroundServiceExceptionBehavior to Ignore	The failed worker is not restarted	Handle recoverable failures and monitor worker health
Retrying every exception	Permanent failures create unnecessary traffic	Retry only appropriate transient failures
Retrying without a limit	Jobs can run indefinitely	Set attempt and time limits
Ignoring cancellation	Shutdown can become slow or unclean	Propagate CancellationToken
Sharing scoped services indefinitely	Can cause incorrect dependency lifetimes	Create scopes with IServiceScopeFactory
Not recording job state	Failed work can be lost or duplicated	Use durable job states and recovery logic
Assuming the worker is healthy because the process runs	A worker can remain alive without making progress	Monitor successful processing and queue health

11. Best Practices for Production Applications

For reliable background processing in .NET, follow these practices:

Catch exceptions around individual jobs whenever possible.
Keep the default StopHost behavior for unexpected failures that escape the worker.
Use bounded retries with exponential backoff and jitter for transient failures.
Use IServiceScopeFactory when background workers need scoped dependencies.
Propagate cancellation tokens through database and HTTP operations.
Store job state in durable storage.
Design processing operations to tolerate duplicate execution.
Monitor queue depth, failure rates, processing duration, and last successful execution.
Separate permanent failures from retryable failures.
Use a dedicated job-processing framework or message broker when requirements include scheduling, persistence, distributed workers, and advanced retry management.

A simple BackgroundService is suitable for many recurring tasks. However, as job-processing requirements grow, a dedicated system may provide better reliability and operational control.

Summary

An unhandled exception in a .NET BackgroundService can stop the application host because the default exception behavior is StopHost. This prevents the application from silently continuing without a background service that may be essential to its operation.

To keep a worker running when an individual job fails, handle exceptions at the job-processing level, record failed work, and continue with the next job when appropriate.

Use bounded retries for transient errors, create dependency injection scopes for scoped services, and handle cancellation separately from application failures. Most importantly, monitor actual processing activity rather than checking only whether the worker process is running.

Reliable background processing is not about preventing every exception. It is about isolating failures, recovering safely, and ensuring that unfinished work is not lost.

ASP.NET Core BackgroundServicebackground servicesexception handlingbackground jobsretry policiescancellation tokens

Comments

Join the conversation! Your thoughts help the community grow.

Handling Background Job Failures Without Stopping .NET BackgroundService

Sophia Morales

1h4.7k00100Article

Background services are commonly used in .NET applications to process queued messages, send notifications, synchronize data, generate reports, and perform scheduled maintenance tasks.

Unlike an ordinary API request, a background service may run continuously for hours or days. A single unhandled exception can therefore have a much wider impact than a failed request.

This article explains how to structure BackgroundService implementations, isolate job failures, handle cancellation, implement retries, and monitor background processing in production.

1. Understand Why BackgroundService Can Stop Unexpectedly

In ASP.NET Core, BackgroundService provides a base class for implementing long-running background operations.

A typical implementation looks like this:

public class NotificationWorker : BackgroundService
{
    private readonly ILogger<NotificationWorker> _logger;

    public NotificationWorker(
        ILogger<NotificationWorker> logger)
    {
        _logger = logger;
    }

    protected override async Task ExecuteAsync(
        CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            _logger.LogInformation(
                "Checking pending notifications");

            await Task.Delay(
                TimeSpan.FromSeconds(10),
                stoppingToken);
        }
    }
}

The ExecuteAsync method represents the lifetime of the background service.

If an exception escapes from this method, the hosting infrastructure observes the failure.

Starting with .NET 6, the default BackgroundService exception behavior is StopHost. This means an unhandled exception in a background service causes the host to stop.

This behavior is intentional. It prevents an application from appearing healthy while an important background service has silently stopped working.

Consider the following example:

protected override async Task ExecuteAsync(
    CancellationToken stoppingToken)
{
    while (!stoppingToken.IsCancellationRequested)
    {
        await ProcessNotificationsAsync(stoppingToken);

        await Task.Delay(
            TimeSpan.FromSeconds(10),
            stoppingToken);
    }
}

If ProcessNotificationsAsync throws an exception, the loop exits. The exception reaches the hosting infrastructure, and the host can stop.

The important distinction is that the host does not automatically restart the failed BackgroundService when an exception occurs.

If you want the worker to continue after an individual job fails, you must handle that failure inside the worker.

2. Handle Exceptions at the Individual Job Level

The first improvement is to isolate exceptions around the operation that can fail.

For example, a notification worker should not stop processing all notifications because one notification fails.

protected override async Task ExecuteAsync(
    CancellationToken stoppingToken)
{
    while (!stoppingToken.IsCancellationRequested)
    {
        try
        {
            await ProcessNotificationsAsync(stoppingToken);
        }
        catch (OperationCanceledException)
            when (stoppingToken.IsCancellationRequested)
        {
            break;
        }
        catch (Exception ex)
        {
            _logger.LogError(
                ex,
                "An error occurred while processing notifications");
        }

        await Task.Delay(
            TimeSpan.FromSeconds(10),
            stoppingToken);
    }
}

This implementation catches unexpected exceptions from the processing operation and allows the loop to continue.

However, it is important to understand what this code actually protects.

If ProcessNotificationsAsync processes 100 notifications internally and fails on the 50th notification, the remaining notifications may not be processed during that iteration.

Catching the exception keeps the worker alive, but it does not automatically recover the failed operation or continue from the exact point of failure.

For better isolation, handle exceptions around individual jobs.

3. Process Each Job Independently

A more reliable design separates retrieving jobs from processing them.

Imagine a system that retrieves pending email notifications from a database.

You can process each notification independently:

public async Task ProcessNotificationsAsync(
    CancellationToken cancellationToken)
{
    var notifications = await GetPendingNotificationsAsync(
        cancellationToken);

    foreach (var notification in notifications)
    {
        cancellationToken.ThrowIfCancellationRequested();

        try
        {
            await SendNotificationAsync(
                notification,
                cancellationToken);

            await MarkAsCompletedAsync(
                notification.Id,
                cancellationToken);
        }
        catch (OperationCanceledException)
            when (cancellationToken.IsCancellationRequested)
        {
            throw;
        }
        catch (Exception ex)
        {
            _logger.LogError(
                ex,
                "Failed to process notification {NotificationId}",
                notification.Id);

            await MarkAsFailedAsync(
                notification.Id,
                ex.Message,
                cancellationToken);
        }
    }
}

Now, a failure in one notification does not prevent the worker from processing the next notification.

This approach provides several advantages:

Individual failures are isolated.
Successful jobs can be marked as completed.
Failed jobs can be recorded for later processing.
Logs can identify the exact job that failed.
The background service can continue processing other jobs.

For production systems, recording failure information is important. Simply catching an exception and continuing can hide data-processing problems.

A job should have a clearly defined state, such as Pending, Processing, Completed, or Failed.

4. Implement Controlled Retries for Transient Failures

Not every failure requires manual intervention.

Retrying these operations can help.

However, retries should be limited and controlled.

Consider this example:

public async Task SendWithRetryAsync(
    Notification notification,
    CancellationToken cancellationToken)
{
    const int maxAttempts = 3;

    for (int attempt = 1; attempt <= maxAttempts; attempt++)
    {
        try
        {
            await _emailClient.SendAsync(
                notification,
                cancellationToken);

            return;
        }
        catch (HttpRequestException ex)
            when (attempt < maxAttempts)
        {
            _logger.LogWarning(
                ex,
                "Notification attempt {Attempt} failed",
                attempt);

            var delay = TimeSpan.FromSeconds(
                Math.Pow(2, attempt));

            await Task.Delay(delay, cancellationToken);
        }
    }

    throw new InvalidOperationException(
        "Notification delivery failed after all retry attempts.");
}

This example uses exponential backoff.

The delay increases between attempts:

Attempt	Delay
1	2 seconds
2	4 seconds
3	No additional retry

The example is intentionally simple. Production retry policies should also consider:

Whether the failure is transient.
HTTP status codes and Retry-After headers.
Maximum retry duration.
Randomized delay, also called jitter.
Whether the operation is safe to repeat.
Cancellation during retry delays.

Avoid retrying every exception. For example, an invalid email address or an authentication failure will not usually be fixed by repeating the same request.

For more advanced retry and timeout handling, .NET applications can use the Microsoft.Extensions.Http.Resilience package with HttpClient.

5. Configure BackgroundService Exception Behavior Carefully

.NET provides BackgroundServiceExceptionBehavior through HostOptions.

It supports two values:

StopHost: Stops the host when an unhandled BackgroundService exception occurs.
Ignore: Logs the exception but does not stop the host or restart the failed service.

You can configure it as follows:

builder.Services.Configure<HostOptions>(options =>
{
    options.BackgroundServiceExceptionBehavior =
        BackgroundServiceExceptionBehavior.Ignore;
});

This changes the host's response to an unhandled exception.

However, there is an important limitation.

Setting Ignore does not restart the BackgroundService. If ExecuteAsync has already completed because of an exception, the worker does not automatically start again.

The application might continue running while the background worker is no longer processing anything.

This can create a dangerous situation: the API remains available, but its background processing has stopped.

For this reason, Ignore should not be treated as a replacement for proper exception handling or worker recovery.

For most applications, keep the default StopHost behavior and handle recoverable job failures inside the worker.

If the entire worker fails because of a serious infrastructure or application error, stopping the host can be safer than silently continuing without the required background processing.

6. Use IServiceScopeFactory for Scoped Dependencies

BackgroundService instances are generally registered as hosted services and managed as long-lived objects.

However, many application services, such as DbContext, are registered with a scoped lifetime.

Injecting a scoped dependency directly into a long-lived background service can cause lifetime problems.

Use IServiceScopeFactory to create a new dependency injection scope for each processing cycle.

public class NotificationWorker : BackgroundService
{
    private readonly IServiceScopeFactory _scopeFactory;
    private readonly ILogger<NotificationWorker> _logger;

    public NotificationWorker(
        IServiceScopeFactory scopeFactory,
        ILogger<NotificationWorker> logger)
    {
        _scopeFactory = scopeFactory;
        _logger = logger;
    }

    protected override async Task ExecuteAsync(
        CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            try
            {
                await using var scope =
                    _scopeFactory.CreateAsyncScope();

                var processor = scope.ServiceProvider
                    .GetRequiredService<INotificationProcessor>();

                await processor.ProcessAsync(stoppingToken);
            }
            catch (OperationCanceledException)
                when (stoppingToken.IsCancellationRequested)
            {
                break;
            }
            catch (Exception ex)
            {
                _logger.LogError(
                    ex,
                    "Notification processing failed");
            }

            try
            {
                await Task.Delay(
                    TimeSpan.FromSeconds(10),
                    stoppingToken);
            }
            catch (OperationCanceledException)
                when (stoppingToken.IsCancellationRequested)
            {
                break;
            }
        }
    }
}

The scope is disposed after each processing cycle.

This allows scoped services, including database contexts, to be created and released correctly.

It also helps prevent a long-running worker from holding onto the same DbContext indefinitely.

7. Handle Graceful Shutdown and Cancellation Correctly

Background services should respond to cancellation when the application is shutting down.

The stoppingToken passed to ExecuteAsync is triggered when the host requests the service to stop.

A common mistake is treating every OperationCanceledException as an application failure.

For example:

try
{
    await ProcessNotificationsAsync(stoppingToken);
}
catch (Exception ex)
{
    _logger.LogError(ex, "Background processing failed");
}

This catches cancellation exceptions too, potentially recording normal shutdown as an error.

Instead, distinguish application shutdown from unexpected cancellation:

try
{
    await ProcessNotificationsAsync(stoppingToken);
}
catch (OperationCanceledException)
    when (stoppingToken.IsCancellationRequested)
{
    _logger.LogInformation(
        "Notification worker is stopping");
}
catch (Exception ex)
{
    _logger.LogError(
        ex,
        "Notification processing failed");
}

A graceful shutdown should allow the worker to stop taking new jobs and finish or safely abandon the current operation within the configured shutdown period.

For long-running jobs, consider checkpoints or durable job states so that interrupted work can be resumed safely.

Do not assume StopAsync will always execute. Unexpected process termination, machine failure, or forced container shutdown can prevent graceful cleanup.

8. Avoid Duplicate Processing When Jobs Fail

Handling exceptions is only part of the problem.

Imagine a worker sends an email successfully but fails before marking the notification as completed.

When the worker runs again, it may send the same email a second time.

This is a duplicate-processing problem.

A reliable job-processing system should use a durable state transition.

For example:

Pending
   |
   v
Processing
   |
   +---- Success ----> Completed
   |
   +---- Failure ----> Failed
                         |
                         v
                    Retry Pending

A production implementation should consider:

Marking jobs as Processing before executing them.
Recording processing attempts.
Setting a lease expiration for jobs that become stuck.
Using idempotency keys for external operations when supported.
Preventing multiple workers from claiming the same job.
Moving repeatedly failing jobs into a dead-letter or manual-review state.

Exactly-once execution is difficult to guarantee across a database and an external service. Design jobs to tolerate retries and duplicate delivery instead.

9. Monitor Background Worker Health

A background service can remain running while its actual work is failing.

For example, the worker may successfully execute its loop every ten seconds but fail to process every notification because the external email provider is unavailable.

Checking only whether the process is running is not enough.

Monitor useful operational indicators:

Metric	Why it matters
Jobs processed	Measures actual worker activity
Successful jobs	Shows completed work
Failed jobs	Identifies processing problems
Retry attempts	Highlights temporary service failures
Queue depth	Shows whether work is accumulating
Oldest pending job age	Identifies processing delays
Processing duration	Helps detect slow operations
Last successful execution	Shows whether the worker is making progress

You can also implement a heartbeat or last-successful-processing timestamp.

For example, update a health record after a successful processing cycle. A monitoring system can alert when the worker has not completed work within an expected interval.

Health checks should distinguish between process availability and background processing health.

An application can be healthy enough to serve HTTP requests while a critical background worker is unhealthy. Exposing that distinction helps operations teams respond before a queue grows too large.

10. Common Mistakes to Avoid

Mistake	Why it causes problems	Better approach
Catching exceptions without recording them	Failures become difficult to investigate	Log job identifiers and exception details
Setting BackgroundServiceExceptionBehavior to Ignore	The failed worker is not restarted	Handle recoverable failures and monitor worker health
Retrying every exception	Permanent failures create unnecessary traffic	Retry only appropriate transient failures
Retrying without a limit	Jobs can run indefinitely	Set attempt and time limits
Ignoring cancellation	Shutdown can become slow or unclean	Propagate CancellationToken
Sharing scoped services indefinitely	Can cause incorrect dependency lifetimes	Create scopes with IServiceScopeFactory
Not recording job state	Failed work can be lost or duplicated	Use durable job states and recovery logic
Assuming the worker is healthy because the process runs	A worker can remain alive without making progress	Monitor successful processing and queue health

11. Best Practices for Production Applications

For reliable background processing in .NET, follow these practices:

Catch exceptions around individual jobs whenever possible.
Keep the default StopHost behavior for unexpected failures that escape the worker.
Use bounded retries with exponential backoff and jitter for transient failures.
Use IServiceScopeFactory when background workers need scoped dependencies.
Propagate cancellation tokens through database and HTTP operations.
Store job state in durable storage.
Design processing operations to tolerate duplicate execution.
Monitor queue depth, failure rates, processing duration, and last successful execution.
Separate permanent failures from retryable failures.
Use a dedicated job-processing framework or message broker when requirements include scheduling, persistence, distributed workers, and advanced retry management.

A simple BackgroundService is suitable for many recurring tasks. However, as job-processing requirements grow, a dedicated system may provide better reliability and operational control.

Summary

To keep a worker running when an individual job fails, handle exceptions at the job-processing level, record failed work, and continue with the next job when appropriate.

Reliable background processing is not about preventing every exception. It is about isolating failures, recovering safely, and ensuring that unfinished work is not lost.

ASP.NET Core BackgroundServicebackground servicesexception handlingbackground jobsretry policiescancellation tokens

Comments

Join the conversation! Your thoughts help the community grow.