Background workers are commonly used to process uploaded files, generate reports, import data, resize images, and execute other long-running tasks. Moving this work outside the main request allows an application to respond quickly while processing continues in the background.
However, background workers can fail while a job is running. A server might restart, a container might be terminated, a database connection might be lost, or an unexpected exception might stop the process.
The difficult part is not restarting the worker. It is determining what work has already completed and what remains to be done.
Imagine a worker processing a CSV file containing 100,000 customer records. It successfully processes 40,000 records before the container crashes. When the worker starts again, should it begin from the first record, continue from record 40,001, or restart the entire job?
Starting from the beginning wastes time and may create duplicate records. Continuing from an incorrectly recorded position can skip data. Restarting the job without checking its previous state can repeat operations that have already completed.
A reliable file-processing system needs durable job state, safe checkpoints, duplicate protection, and a recovery mechanism.
In this article, we will explore how to design resumable file-processing jobs, implement checkpointing in C#, test worker crashes, and prevent data loss or duplicate processing.
Why Background Jobs Need Recovery Support
A background worker usually performs a sequence of operations:
Accept a file or receive a reference to it.
Create a processing job.
Read the file in manageable batches.
Validate and process each batch.
Save the results and record progress.
Mark the job as completed.
A crash can occur between any two operations.
For example, a worker might save 500 customer records and crash before updating the job's checkpoint. When the job restarts, it may process those records again.
The opposite problem is also possible. If the worker updates its checkpoint before saving the records, a crash could cause the job to skip records that were never committed.
This means that checkpointing is not simply a matter of saving the current line number. The application must coordinate progress tracking with the actual processing operation.
A robust implementation should answer these questions:
Where is the job's progress stored?
Which records have been committed successfully?
What happens if the worker crashes during a batch?
Can a job be claimed by multiple workers accidentally?
How are repeated operations handled?
How does the system distinguish a temporary failure from a permanent one?
How can an operator investigate or restart a failed job?
These decisions should be made before implementing the worker's recovery logic.
Store Job Progress Outside the Worker Process
The worker's memory disappears when its process terminates. Therefore, essential job information must be stored in durable storage, such as a relational database or a reliable job-processing system.
A job record might contain the following fields:
Field | Purpose |
|---|
JobId | Identifies the processing job |
FilePath | Identifies the uploaded file or stored object |
Status | Indicates whether the job is pending, running, failed, or completed |
LastCommittedRecord | Records the last successfully committed position |
ProcessedCount | Tracks the number of successfully processed records |
AttemptCount | Records the number of processing attempts |
LeaseExpiresAt | Helps determine whether a worker's claim has expired |
LastError | Stores a safe diagnostic summary |
UpdatedAt | Records the last state change |
The precise fields depend on the application.
For large files, storing the complete input file in the job record is usually unnecessary. Store the file in durable object or file storage and retain a stable reference in the database.
Make sure the file remains available for the entire recovery period. A checkpoint is not useful if the source file has been deleted or replaced.
For stronger integrity, record a file checksum or immutable version identifier. This helps ensure that a restarted job processes the same input rather than a different file at the same location.
Design a Job State Machine
A job should have explicit states instead of relying on a single Boolean flag.
A simple state model could include:
Pending: The job has been created but has not started.
Running: A worker currently owns the job.
Completed: All required processing has finished.
Failed: The job cannot continue without intervention or another defined recovery action.
Cancelled: The job was intentionally stopped.
Some systems also need a RetryPending state or a separate retry schedule.
Define which state transitions are valid. For example, a completed job should not return to running because a worker received an old message.
A job can be marked as failed after a permanent validation error or after its retry policy has been exhausted. Temporary infrastructure failures should generally remain eligible for retry.
The state transition and its associated progress update should be persisted consistently. Otherwise, a worker crash can leave the database showing an incorrect combination of status and progress.
Implement a Resumable File Processor in C#
Consider a CSV import job that processes records in batches. The following example illustrates the processing loop and checkpoint concept.
The actual database implementation is represented by an interface so that the processing logic remains separate from the storage mechanism.
public sealed record ImportRecord(
long RecordNumber,
string CustomerId,
string Email);
public sealed record ImportCheckpoint(
string JobId,
long LastCommittedRecord);
public interface IImportCheckpointStore
{
Task<ImportCheckpoint> GetAsync(
string jobId,
CancellationToken cancellationToken);
Task CommitBatchAsync(
string jobId,
long lastCommittedRecord,
IReadOnlyList<ImportRecord> records,
CancellationToken cancellationToken);
}
public interface ICustomerImporter
{
Task<IReadOnlyList<ImportRecord>> ReadBatchAsync(
string filePath,
long afterRecord,
int batchSize,
CancellationToken cancellationToken);
}
The checkpoint store is responsible for coordinating saved records and progress. The importer reads records after the last committed position.
The processing loop can then use the checkpoint to resume the job:
public sealed class CustomerImportWorker
{
private readonly IImportCheckpointStore _checkpointStore;
private readonly ICustomerImporter _importer;
public CustomerImportWorker(
IImportCheckpointStore checkpointStore,
ICustomerImporter importer)
{
_checkpointStore = checkpointStore;
_importer = importer;
}
public async Task ProcessAsync(
string jobId,
string filePath,
CancellationToken cancellationToken)
{
var checkpoint = await _checkpointStore.GetAsync(
jobId,
cancellationToken);
var lastCommittedRecord =
checkpoint.LastCommittedRecord;
const int batchSize = 500;
while (true)
{
cancellationToken.ThrowIfCancellationRequested();
var batch = await _importer.ReadBatchAsync(
filePath,
lastCommittedRecord,
batchSize,
cancellationToken);
if (batch.Count == 0)
{
break;
}
var newCheckpoint = batch.Max(
record => record.RecordNumber);
await _checkpointStore.CommitBatchAsync(
jobId,
newCheckpoint,
batch,
cancellationToken);
lastCommittedRecord = newCheckpoint;
}
}
}
This example illustrates the recovery flow, but its safety depends on the implementation of CommitBatchAsync.
The method must save the batch's effects and advance the checkpoint consistently. If the records are stored in a database, the record changes and checkpoint update should normally occur in the same database transaction.
The example also assumes that the importer returns records in a stable, increasing order and that record numbers identify positions in the original file. If the input format permits repeated or missing identifiers, use a reliable source position or another durable cursor.
In a production implementation, job completion must also be recorded explicitly. A worker should not assume that merely reaching the end of one read operation is sufficient to mark a job complete if additional validation or finalization remains.
Make Checkpoint Updates Atomic
One of the most important recovery requirements is that processed data and checkpoint progress must not disagree.
Suppose a worker processes records 1,001 through 1,500.
There are two possible failure sequences.
Saving Records Before the Checkpoint
The worker saves the records and then crashes before updating the checkpoint.
When the job resumes, it starts from record 1,001 again.
This can repeat the batch's work unless the operations are idempotent.
Saving the Checkpoint Before the Records
The worker advances the checkpoint to record 1,500 and then crashes before saving the records.
When the job resumes, it starts after record 1,500, even though those records may never have been processed.
This can cause permanent data loss.
For database-backed processing, perform both operations in one transaction:
using var transaction =
await connection.BeginTransactionAsync(cancellationToken);
await SaveProcessedRecordsAsync(
connection,
transaction,
records,
cancellationToken);
await UpdateCheckpointAsync(
connection,
transaction,
jobId,
lastCommittedRecord,
cancellationToken);
await transaction.CommitAsync(cancellationToken);
The example assumes that the database operations accept the same connection and transaction. Those methods must be implemented to participate in that transaction.
If the transaction rolls back, neither the batch's database changes nor the checkpoint update should become committed.
This approach works when both operations can be coordinated within the same transactional database. If processing involves an external API, a file system, or another independent service, a single database transaction cannot automatically make all operations atomic. Those workflows need additional safeguards.
Prevent Duplicate Processing After a Crash
Even with checkpoints, a job may repeat work.
For example, the application may successfully call an external service but crash before recording that the operation completed. When the worker resumes, it may call the service again.
The solution depends on the operation being performed.
Use Unique Constraints for Database Imports
Suppose the import file contains customer records with stable customer identifiers.
A database uniqueness constraint can prevent the same customer from being inserted twice. The import logic can then use an upsert or another explicit conflict-handling strategy.
However, a unique constraint alone does not guarantee that every repeated update is harmless. Updating an existing record with stale input can overwrite newer data. Define the intended behavior for duplicate identifiers and conflicting records.
Use Idempotency Keys for External APIs
If a worker sends requests to an external service, use the service's supported idempotency mechanism when available.
Create a stable key for the logical operation and reuse it if the worker retries that operation.
Do not create a new idempotency key every time the job resumes. Doing so can make the external service treat the retry as a separate operation.
If the external service does not support idempotency, consider maintaining an operation ledger, checking the external result before retrying, or designing a reconciliation process.
The appropriate strategy depends on whether the external operation can be safely repeated.
Handle Worker Leases and Multiple Workers
A recovery mechanism must also prevent two workers from processing the same job simultaneously.
This can happen when a job is queued more than once, a worker loses its connection to the database, or an old worker continues running after another worker claims the job.
A lease is a time-limited claim that identifies which worker currently owns a job.
The system can store a worker identifier and lease expiration time. The worker periodically renews the lease while it is healthy.
If the lease expires, another worker may attempt to claim the job.
However, leases alone are not enough. An old worker might continue processing after its lease expires because of a long pause or network partition.
For sensitive workloads, use an atomic job-claim operation and consider a fencing token or version number that prevents stale workers from committing updates after a newer worker has taken ownership.
The checkpoint update should also verify that the worker still has permission to modify the job.
Test Crash Recovery Scenarios
A resumable job should be tested under failure conditions rather than only through a successful end-to-end run.
Useful scenarios include:
Test scenario | Failure being simulated | Expected result |
|---|
Crash before processing a batch | Worker stops before committing changes | The batch is processed after restart |
Crash during a database transaction | The transaction is interrupted | The transaction rolls back, and the checkpoint remains unchanged |
Crash after a committed batch | The worker stops after saving data and progress | Processing resumes from the next uncommitted position |
Repeated job delivery | The same job is queued more than once | Duplicate processing is prevented |
Lease expiration | A worker loses ownership | A valid worker can reclaim the job safely |
External API timeout | The result of an operation is uncertain | Retry or reconciliation avoids duplicate effects |
Source file changes | The input differs from the original version | The job detects the mismatch and stops safely |
Retry limit reached | Repeated attempts fail | The job enters a defined failure or intervention state |
Simulate a Crash Between Batches
A practical test can use a fake importer or checkpoint store that throws an exception after a selected batch has been committed.
The test then creates a new worker instance and runs the job again.
Verify that:
The previously committed records are not lost.
The worker resumes from the saved checkpoint.
No records are skipped.
Repeated operations do not create duplicate business effects.
The job eventually reaches its completed state.
Do not rely only on an in-memory fake to verify database transaction behavior. Use database integration tests for transactional guarantees, and use controlled fault injection to test the recovery sequence.
For long-running workers, include tests that simulate process restarts rather than merely calling the same method twice on the same object.
Common Mistakes to Avoid
Keeping Progress Only in Memory
A local counter disappears when the worker restarts. Persist progress in durable storage.
Updating Progress Before Saving the Results
This can cause the worker to skip records that were never committed. Coordinate data changes and checkpoint updates transactionally wherever possible.
Assuming Every Operation Can Be Repeated Safely
Sending an email, charging a payment method, or creating an external resource may have effects that cannot be reversed easily. Use idempotency or reconciliation mechanisms appropriate to the operation.
Resuming from an Unstable Position
A line number or byte offset is useful only when the input is unchanged and the processing logic can reliably interpret that position. Record a stable file version or checksum and validate it before resuming.
Allowing Multiple Workers to Own the Same Job
Use atomic job claiming, leases, and appropriate concurrency controls.
Marking a Job Completed Too Early
A job should become completed only after all required records, validations, and finalization steps have succeeded.
Best Practices for Reliable File Processing
Store job state and checkpoints in durable storage.
Commit processed data and checkpoint updates together when possible.
Use stable source positions or record identifiers.
Verify that the source file has not changed before resuming.
Make repeatable operations idempotent.
Use database constraints to enforce important uniqueness rules.
Protect jobs from concurrent processing by multiple workers.
Implement bounded retries for transient failures.
Record useful error details without exposing sensitive data.
Provide a way to inspect, retry, or manually recover failed jobs.
Monitor job duration, retry counts, checkpoint progress, and failure rates.
Test crash scenarios with both unit tests and database integration tests.
Advantages and Limitations
Resumable processing reduces the amount of work lost when a worker crashes. It is particularly useful for large imports, report generation, document processing, and jobs that call external services.
Durable checkpoints make recovery more predictable, while idempotency helps prevent duplicate side effects.
However, checkpointing adds complexity. The system must manage job state, concurrency, transaction boundaries, and source-file consistency. External operations can also make exactly-once processing difficult because the database and external service may not share a transaction.
In many systems, the practical goal is at-least-once execution combined with idempotent operations and reliable recovery. This can provide dependable business outcomes without assuming that every operation executes exactly once.
Summary
A worker crash should not force a file-processing system to start from scratch or risk losing previously completed work.
The foundation of a reliable design is durable job state, a trustworthy checkpoint, and consistent coordination between progress updates and processed data. Idempotency and concurrency controls help prevent duplicate effects when jobs are retried or reclaimed by another worker.
Test failures before, during, and after batch commits. Also verify recovery after process restarts, source-file changes, external API timeouts, and lease expiration.
With these safeguards in place, background workers can resume interrupted jobs predictably while preserving data integrity and reducing unnecessary processing.
Join the conversation! Your thoughts help the community grow.