How to Prevent Duplicate File Processing in a Background Worker

Background workers often process files users upload, import from external systems, or deliver through scheduled jobs. A worker might read a CSV file, validate its contents, save records to a database, and move the file to an archive directory after processing.

The workflow sounds straightforward until the same file is processed twice.

Duplicate processing can happen when a file is uploaded more than once, a scheduler starts overlapping jobs, a worker restarts before recording completion, or a file watcher reports the same change several times. Network shares and cloud storage can add timing issues, especially when files become visible before an upload finishes.

Depending on the application, duplicate processing can create repeated database records, send duplicate notifications, charge customers twice, or overwrite previously imported data.

The solution is not simply to check whether a filename has already been processed. Filenames can be reused, files can change while keeping the same name, and two workers can check the same file simultaneously.

A reliable design combines file identification, atomic claiming, idempotent processing, durable status tracking, and recovery after failures.

This article explains how to prevent duplicate file processing in a C# background worker, how to handle concurrent workers, and how to test the workflow under realistic failure conditions.

Understand Why Duplicate Processing Happens

Before implementing a solution, identify the events that can cause the same logical file to enter the processing pipeline more than once.

Common scenarios include:

• A file watcher raises multiple events for the same file.
• A scheduler starts a new job before the previous job finishes.
• Two worker instances discover the same file simultaneously.
• A worker completes database changes but crashes before recording completion.
• A message broker redelivers a file-processing message.
• An upload is retried after the client fails to receive confirmation.
• A file is copied to an input directory before the upload is complete.
• A previously processed filename is reused for a new file.

These scenarios require different protections.

Debouncing file watcher events can reduce repeated notifications, but it does not prevent two independent workers from processing the same file. A database record can track completed work, but a simple check followed by an insert is vulnerable to concurrency unless the operation is protected by an appropriate uniqueness constraint or atomic claim.

A reliable system must protect both the decision to start processing and the business effects produced by processing.

Choose a Reliable File Identity

The first step is deciding how to determine whether two processing requests refer to the same logical file.

A filename alone is usually insufficient.

For example, these files might be different versions of the same report:

daily-sales.csv
daily-sales.csv

The names are identical, but the contents may differ. Conversely, a file may be copied under a different name while containing exactly the same data.

Several identification strategies are available.

For file imports, a SHA-256 content hash is often a useful deduplication key when identical contents should be processed only once.

However, identical contents are not always the same logical operation. A business system may intentionally submit the same file twice as two separate import jobs. In that case, use a stable upload or job identifier instead of treating the content hash as the complete definition of identity.

Calculate a File Hash in C#

The following example calculates a SHA-256 hash for a file using asynchronous file access.

using System.Security.Cryptography;

public static class FileHash
{
    public static async Task<string> ComputeSha256Async(
        string filePath,
        CancellationToken cancellationToken = default)
    {
        await using var stream = new FileStream(
            filePath,
            FileMode.Open,
            FileAccess.Read,
            FileShare.Read,
            bufferSize: 81920,
            options: FileOptions.Asynchronous |
                     FileOptions.SequentialScan);

        var hash = await SHA256.HashDataAsync(
            stream,
            cancellationToken);

        return Convert.ToHexString(hash);
    }
}

The method returns a hexadecimal representation of the file's SHA-256 hash.

The hash is useful only if the file remains stable while it is being read. If another process is still writing to the file, the worker could calculate a hash for incomplete or changing content.

For this reason, establish a file-completion protocol before calculating the hash.

Ensure the File Is Ready Before Processing

A file appearing in a directory does not necessarily mean that the upload or copy operation has completed.

For example, a producer may create a file and then continue writing its contents for several seconds. If the worker begins processing immediately, it may read incomplete data.

One common approach is to have the producer write to a temporary filename and rename the file to its final name only after the write is complete.

For example:

daily-sales.csv.uploading
daily-sales.csv

The worker processes only files with the final extension or naming convention.

On a filesystem where the rename operation is atomic for the relevant path and storage conditions, this approach provides a useful handoff between the producer and consumer. Cross-filesystem moves, network shares, and cloud object storage may have different semantics, so verify the guarantees of the storage system you use.

If the producer cannot follow this protocol, the worker may need a readiness check, a manifest file, or a storage-specific completion signal.

Do not rely solely on checking file size twice with a short delay. A paused upload can leave the file size unchanged temporarily and then resume.

Track Processing State in a Database

A durable processing table allows the worker to record which files have been discovered, claimed, completed, or failed.

For example, the following SQL Server table stores the file identity, processing status, and relevant timestamps.

CREATE TABLE dbo.FileProcessingJobs
(
    JobId BIGINT IDENTITY(1,1) PRIMARY KEY,
    FileKey NVARCHAR(200) NOT NULL,
    ContentHash CHAR(64) NULL,
    Status NVARCHAR(20) NOT NULL,
    CreatedAt DATETIME2 NOT NULL
        DEFAULT SYSUTCDATETIME(),
    StartedAt DATETIME2 NULL,
    CompletedAt DATETIME2 NULL,
    ErrorMessage NVARCHAR(2000) NULL,

    CONSTRAINT UQ_FileProcessingJobs_FileKey
        UNIQUE (FileKey)
);

This example assumes that FileKey identifies a logical processing job. If identical file contents must be deduplicated independently of filenames, you can enforce uniqueness on ContentHash instead, provided that the business rules support that behavior.

Avoid creating a unique constraint that combines fields without first defining what constitutes a duplicate. A poorly chosen uniqueness rule can reject legitimate new jobs.

The status field might contain values such as Pending, Processing, Completed, and Failed. For more complex workflows, you may also need a retry count, lease expiration, worker identifier, and a separate record of processing attempts.

Claim Work Atomically

A common implementation mistake is to check whether a file has already been processed and then insert a new processing record in a separate operation.

Conceptually, the unsafe workflow is:

1. Query the database for the file.
2. If the file does not exist, begin processing.
3. Insert the processing record.

Two workers can execute the first query at the same time. Both may conclude that the file is new and begin processing it.

A uniqueness constraint helps prevent duplicate records, but it does not automatically undo business work that both workers have already started.

Instead, claim the job atomically before executing its business operations.

For a simple SQL Server queue table, an atomic claim can use a transaction and locking hints appropriate to the queue design. The following example claims one pending job and returns its identifier.

BEGIN TRANSACTION;

;WITH NextJob AS
(
    SELECT TOP (1) *
    FROM dbo.FileProcessingJobs
        WITH (UPDLOCK, READPAST, ROWLOCK)
    WHERE Status = 'Pending'
    ORDER BY JobId
)
UPDATE NextJob
SET
    Status = 'Processing',
    StartedAt = SYSUTCDATETIME()
OUTPUT
    inserted.JobId,
    inserted.FileKey,
    inserted.ContentHash;

COMMIT TRANSACTION;

This pattern is intended for a SQL Server queue table where workers claim pending records. UPDLOCK helps coordinate concurrent claims, and READPAST allows a worker to skip certain rows locked by other transactions.

The exact locking behavior depends on the database configuration, isolation level, indexes, and query plan. Test this pattern against the actual SQL Server version and workload rather than assuming that the hints guarantee all desired concurrency behavior in every configuration.

Also, this example does not automatically handle expired claims or crashed workers. A production queue typically needs a lease or recovery process that can identify jobs left in Processing state after a worker disappears.

Most importantly, do not keep the database transaction open while performing lengthy file parsing or calling external services. Claim the work in a short transaction, commit it, and then process the file.

Make Database Writes Idempotent

Atomic claiming prevents two workers from claiming the same pending job at the same time, but duplicate business effects can still occur after a crash.

Consider this sequence:

1. The worker reads a file.
2. It inserts imported records into the database.
3. The database transaction commits.
4. The worker crashes before marking the job Completed.
5. A recovery process retries the job.

Without additional protection, the imported records may be inserted again.

For database-only processing, one approach is to combine the imported records and completion status in the same transaction.

For example, if a file imports customer records, the application can use a stable import identifier or a suitable business key to enforce uniqueness. The transaction can then insert the records and mark the job complete together.

Conceptually:

BEGIN TRANSACTION;

-- Insert or update imported business records here.
-- Apply appropriate uniqueness constraints to those records.

UPDATE dbo.FileProcessingJobs
SET
    Status = 'Completed',
    CompletedAt = SYSUTCDATETIME(),
    ErrorMessage = NULL
WHERE JobId = @JobId
  AND Status = 'Processing';

COMMIT TRANSACTION;

This example shows the completion update, but the real implementation must perform the business writes in the same transaction and verify that the expected job row was updated.

If the transaction fails, both the business changes and completion update should roll back together.

If processing involves an external API, a file transfer, or another independent system, a local database transaction cannot automatically make all side effects atomic. Use idempotency keys, a durable outbox, reconciliation, or another mechanism appropriate to that external system.

Handle Worker Crashes and Stale Processing Jobs

A job marked Processing may remain in that state if the worker terminates unexpectedly.

A recovery mechanism should distinguish between a job that is genuinely active and one that has been abandoned.

One option is to use a lease. The worker records when its claim expires and periodically renews the lease while it is still processing the file.

A recovery process can identify jobs whose leases have expired and make them eligible for retry, subject to the application's recovery rules.

A lease is not a guarantee that the previous worker has stopped. A paused worker might resume after its lease expires, while another worker has already claimed the job.

For operations that must be protected from stale workers, consider a monotonically increasing fencing token or another concurrency-control mechanism. The receiving system must enforce that protection; merely generating a token does not prevent an old worker from performing a side effect.

Before retrying a stale job, also verify whether its business operations were committed. Recovery should rely on durable state and idempotency, not on the assumption that an expired lease means nothing happened.

Test Duplicate File Events

If the worker uses FileSystemWatcher, test repeated notifications for the same file.

Filesystem watchers can report multiple events for one logical change, and their event streams may be affected by buffering or operating-system behavior.

A useful test should create a file, trigger or simulate repeated notifications, and verify that only one logical job is accepted.

For example, the test can submit the same FileKey several times concurrently and then verify the job table contains only the expected number of records.

Do not treat the watcher event itself as proof that a file must be processed immediately. Use the event to discover possible work, then let the durable job store determine whether the work has already been registered or claimed.

For systems that scan a directory periodically, test repeated scans of the same files as well. The worker should produce the same correct outcome whether a file is discovered once or many times.

Test Concurrent Workers

Duplicate-processing defects often remain hidden when a single worker runs in isolation.

Run two or more worker instances against the same processing table and submit the same job at approximately the same time.

Verify that:

• Only one worker successfully claims the job at a time.
• The uniqueness constraint prevents duplicate job records.
• Business records are not inserted twice.
• A failed claim does not allow a worker to proceed with the business operation.
• A crashed worker's job can be recovered safely.
• A stale worker cannot overwrite the result of a newer attempt.

Use barriers or other test coordination mechanisms to force workers into the same critical section. This is more reliable than attempting to create a race condition through random delays.

Inspect database state and actual business effects after the test. Log messages alone cannot prove that duplicate processing was prevented.

Test External Side Effects Separately

Many file-processing jobs send emails, call APIs, publish messages, or move files to another system.

These operations create failure windows that cannot always be solved with a single database transaction.

For example, a worker may upload a file successfully to remote storage but crash before marking the local job complete. Retrying the upload may create another object or overwrite an existing one, depending on the destination's behavior.

Where supported, use a stable external operation identifier, a deterministic destination key, or an idempotency mechanism provided by the destination service.

For notifications, record delivery intent durably and use a provider or application mechanism that can tolerate repeated requests. Do not assume that a successful HTTP response and a local status update happen atomically.

Test the workflow by simulating failure immediately before and after each external side effect. Verify that retries either reuse the original result or safely determine whether the operation needs to be performed.

Common Mistakes to Avoid

Using filenames as the only duplicate check

A filename can be reused for different content, and identical content can arrive under different filenames. Choose a stable identity that matches the business definition of a duplicate.

Checking for existing jobs without an atomic claim

A separate check and insert can allow concurrent workers to process the same file. Use a durable uniqueness constraint and an atomic claim mechanism.

Marking a job complete before business changes are durable

If the worker records completion too early, a crash can leave missing records while the job appears finished. Commit business changes and completion state together when they belong to the same database transaction.

Assuming a worker crash rolls back everything

A process crash does not reverse a committed database transaction or an external API call. Recovery must account for completed side effects.

Automatically retrying every failed job

Some failures are temporary, while others indicate invalid data or an unrecoverable business condition. Use bounded retries, clear failure states, and manual investigation where necessary.

Best Practices for Production Background Workers

Define the identity of a logical file-processing job before implementing duplicate prevention. Use a stable upload identifier, content hash, or business key based on the actual requirements.

Store processing state durably and claim work atomically. Keep transactions short, enforce uniqueness in the database, and make business writes idempotent.

Use temporary filenames, completion markers, or storage-specific upload protocols to ensure that files are complete before processing begins.

Add recovery for abandoned jobs, including leases or another suitable ownership mechanism. Protect against stale workers when concurrent attempts could otherwise create conflicting side effects.

Build automated tests for repeated file notifications, duplicate submissions, concurrent workers, process termination, and failures between business operations and completion updates.

Finally, monitor pending jobs, processing duration, retry counts, stale claims, and failed files. A reliable worker should make it possible to determine what happened to each file and why a job has not completed.

Advantages and Disadvantages

Advantages

• Prevents repeated business operations caused by duplicate events or message delivery.
• Makes recovery after worker crashes safer.
• Improves visibility into pending, completed, and failed jobs.
• Supports concurrent worker execution with controlled ownership.
• Reduces the risk of duplicate records, notifications, and external operations.

Disadvantages

• Requires durable state and additional database constraints.
• Adds complexity around retries, leases, and recovery.
• Content hashing adds file-reading work for large files.
• External side effects may require separate idempotency or reconciliation mechanisms.
• Incorrect duplicate-identification rules can reject legitimate jobs.

Summary

Preventing duplicate file processing requires more than checking whether a filename exists. A reliable background worker must identify logical jobs consistently, wait until files are complete, claim work atomically, and make business operations safe to repeat.

Use durable processing records, uniqueness constraints, and short transactions to coordinate workers. Protect the gap between successful business operations and completion updates with idempotency and appropriate transaction boundaries.

Test repeated file events, concurrent workers, abrupt process termination, stale claims, and failures around external side effects. These tests reveal whether the system can recover without repeating completed work or losing unfinished work.

The goal is to make file processing predictable even when events are duplicated, workers restart, or external systems fail. With durable state, idempotent operations, and deliberate recovery tests, background workers can handle these conditions without compromising data correctness.