Background jobs handle many tasks that do not need to run during an HTTP request. They send emails, process payments, generate reports, synchronize data, and update records from external systems.
The problem starts when a job runs more than once.
A worker might process a message, update a database record, and then crash before acknowledging the message. When the worker restarts, the same message may be delivered again. A scheduled job might also run twice because of a retry, deployment, or overlapping schedule.
If the job is not designed for repeated execution, it could charge a customer twice, send duplicate emails, or create duplicate records.
The solution is to make background jobs idempotent. In simple terms, running the same operation multiple times should not produce unintended additional effects.
This article explains how idempotency works, how to implement it with C# and SQL Server, and how to handle retries, concurrent workers, and external API calls safely.
What Does Idempotency Mean?
An operation is idempotent when repeating it with the same input produces the same intended result as performing it once.
Consider a job that marks an order as shipped.
A non-idempotent implementation might insert a new shipment record every time it runs. If the job runs three times, the database could contain three shipment records for the same order.
An idempotent implementation checks whether the shipment has already been created or uses a database constraint to prevent duplicates.
Operation | Risk when repeated | Safer approach |
|---|
Create an invoice | Duplicate invoices | Use a stable invoice or operation identifier |
Send a payment request | Duplicate charges | Use the payment provider's idempotency key |
Import a customer | Duplicate customer records | Use a unique external customer ID |
Send an email | Duplicate messages | Track delivery attempts and use provider support where available |
Update an order status | Repeated side effects | Apply guarded state transitions |
Generate a daily report | Duplicate report files | Use a stable report key and controlled replacement |
Idempotency does not mean that a job never runs twice. It means the system handles repeated execution without producing unintended duplicate effects.
Why Background Jobs Run More Than Once
Repeated execution is common in distributed systems because workers and message brokers cannot always determine exactly where a failed operation stopped.
For example, a worker may follow this sequence:
Receive a message from a queue.
Update the database.
Crash before acknowledging the message.
Restart and receive the same message again.
The database update may have succeeded even though the worker never acknowledged the message.
This is one reason message systems commonly support at-least-once delivery. A message can be delivered more than once, so consumers must account for duplicates.
Other causes include:
Automatic retries after temporary failures.
Scheduled jobs overlapping with an earlier execution.
Manual reprocessing by an operator.
Network timeouts where the result of an operation is unknown.
Worker restarts during deployment.
Multiple workers receiving related tasks concurrently.
Trying to eliminate every retry is usually the wrong goal. A better design assumes that retries and duplicate execution will happen.
Step 1: Give Every Operation a Stable Identifier
Start by identifying the logical operation the job performs.
For example, suppose an application receives an event indicating that an order should be invoiced. Each delivery of the same event should carry the same operation identifier.
public sealed record CreateInvoiceJob(
Guid OperationId,
int OrderId);
The OperationId identifies the logical request, not an individual execution attempt.
If the job is retried, the retry must reuse the same OperationId. Generating a new identifier for every attempt defeats deduplication because the system sees each attempt as a different operation.
For scheduled work, the identifier could be based on a business key such as the customer ID and reporting date. For message-driven work, it might be the event ID assigned when the event is first created.
Choose an identifier that represents the operation whose effects must not be duplicated.
Step 2: Add a Database Constraint
An application-level check is useful, but it is not enough to protect against concurrent workers.
Imagine two workers processing the same operation at nearly the same time. Both might check whether an invoice exists, find nothing, and then attempt to create one.
A database constraint can prevent both workers from creating the same logical invoice.
Create an invoice table with a unique operation identifier:
CREATE TABLE dbo.Invoices
(
InvoiceId BIGINT IDENTITY(1,1) PRIMARY KEY,
OperationId UNIQUEIDENTIFIER NOT NULL,
OrderId INT NOT NULL,
CreatedAt DATETIME2 NOT NULL
DEFAULT SYSUTCDATETIME(),
CONSTRAINT UQ_Invoices_OperationId
UNIQUE (OperationId)
);
The unique constraint ensures that SQL Server cannot store two invoice records with the same OperationId.
This is stronger than checking for an existing record in application code because the database enforces the rule even when multiple workers run concurrently.
Before adding the constraint to an existing table, check for duplicates and resolve them. A migration will fail if the data already violates the uniqueness rule.
Step 3: Implement an Idempotent Database Operation
The following C# example uses Entity Framework Core to create an invoice. It assumes the database entity has a unique constraint on OperationId.
public async Task CreateInvoiceAsync(
CreateInvoiceJob job,
CancellationToken cancellationToken)
{
var existingInvoice = await dbContext.Invoices
.SingleOrDefaultAsync(
x => x.OperationId == job.OperationId,
cancellationToken);
if (existingInvoice is not null)
{
return;
}
var invoice = new Invoice
{
OperationId = job.OperationId,
OrderId = job.OrderId,
CreatedAt = DateTime.UtcNow
};
dbContext.Invoices.Add(invoice);
await dbContext.SaveChangesAsync(cancellationToken);
}
This is a useful first step, but the initial existence check alone is not concurrency-safe. Two workers can both pass the check before either saves its invoice.
The unique database constraint remains essential.
In a production implementation, handle the specific unique-constraint violation that occurs when another worker has already created the invoice. Do not treat every database exception as a duplicate, because other failures may require a retry or investigation.
You can also use a database-first approach that performs an atomic insert and handles the duplicate key according to your database provider.
The important design rule is that the database, not only the application code, must enforce the uniqueness of the logical operation.
Step 4: Track Processed Messages
For queue-based systems, a processed-message table can help prevent duplicate processing.
CREATE TABLE dbo.ProcessedMessages
(
MessageId NVARCHAR(200) NOT NULL PRIMARY KEY,
ProcessedAt DATETIME2 NOT NULL
DEFAULT SYSUTCDATETIME()
);
The worker records a message ID after successfully processing the message.
However, recording the message ID separately from the business update creates a failure window. If the business update succeeds but the worker crashes before recording the message ID, the message may be processed again.
The reverse ordering is also dangerous: if the worker records the message as processed and crashes before performing the business update, a retry may incorrectly skip unfinished work.
When both changes are in the same relational database, the safer approach is to perform the business update and insert the processed-message record in one database transaction.
Conceptually, the transaction should:
Check or claim the message ID.
Apply the business changes.
Record successful processing.
Commit both changes together.
If the transaction rolls back, neither change should be committed.
A unique constraint on MessageId helps handle concurrent attempts. When two workers try to claim the same message, only one should successfully create the unique record. The other worker should treat the duplicate as already claimed or processed, according to the transaction design.
Be careful about treating a message as complete before the business changes have actually committed.
Step 5: Use a Transaction for Related Database Changes
Suppose a job updates an order and creates an invoice. Those changes should remain consistent.
With EF Core, you can place both operations inside a transaction:
await using var transaction =
await dbContext.Database.BeginTransactionAsync(
cancellationToken);
try
{
var order = await dbContext.Orders
.SingleAsync(
x => x.Id == job.OrderId,
cancellationToken);
var existingInvoice = await dbContext.Invoices
.SingleOrDefaultAsync(
x => x.OperationId == job.OperationId,
cancellationToken);
if (existingInvoice is null)
{
dbContext.Invoices.Add(new Invoice
{
OperationId = job.OperationId,
OrderId = order.Id,
CreatedAt = DateTime.UtcNow
});
}
order.InvoiceRequested = true;
await dbContext.SaveChangesAsync(cancellationToken);
await transaction.CommitAsync(cancellationToken);
}
catch
{
await transaction.RollbackAsync(cancellationToken);
throw;
}
This example demonstrates transaction boundaries, but it still relies on the unique constraint and appropriate handling of concurrent duplicate operations. It also assumes the invoice request and order update belong in the same transaction.
For operations that need stronger concurrency protection, use an atomic database operation, an appropriate locking strategy, or optimistic concurrency control.
Do not include external API calls inside a database transaction. A database transaction cannot roll back an email already sent or a payment already accepted by an external service.
Step 6: Handle External API Calls with Idempotency Keys
Database transactions solve consistency problems within the database, but background jobs often interact with payment providers, email services, and other external systems.
Consider a payment request that times out. The provider might have processed the payment even though your application did not receive the response.
If the job sends a new payment request with a different identifier, the customer could be charged twice.
When the provider supports idempotency keys, send the same stable key for every retry of the same logical payment operation.
var request = new PaymentRequest
{
OrderId = job.OrderId,
Amount = amount
};
var response = await paymentClient.CreatePaymentAsync(
request,
idempotencyKey: job.OperationId.ToString(),
cancellationToken);
The method signature here is illustrative; actual payment SDKs use provider-specific APIs and parameter names.
A provider's idempotency behavior depends on its documented rules, including key scope, retention period, and how it handles requests with different parameters under the same key.
If the provider does not support idempotency keys, you may need another strategy, such as querying the provider for the operation's status before retrying. This cannot always eliminate uncertainty, so design the workflow around the provider's actual capabilities.
Step 7: Make Scheduled Jobs Safe to Overlap
Scheduled jobs can overlap when one execution takes longer than the schedule interval.
For example, a report job runs every five minutes but sometimes takes eight minutes to finish. A second execution may start before the first one completes.
A simple process-level lock can prevent overlap within one running application instance. However, it does not coordinate multiple servers or containers.
For distributed deployments, use a shared coordination mechanism such as a database-backed lease, a distributed lock with appropriate failure handling, or a job scheduler that supports distributed execution controls.
Even with a distributed lock, retain idempotency in the underlying operation. A worker can lose its lease or crash after making changes, and another worker may need to resume the work.
For recurring jobs, a stable business key can be particularly useful. For example, a daily report operation might be uniquely identified by the report type and reporting date.
Step 8: Test Failure and Retry Scenarios
A job that succeeds once is not necessarily safe to retry. Test what happens when failures occur at different points in the workflow.
Test scenario | Expected behavior |
|---|
The same message is delivered twice | No duplicate business record |
Two workers process the same operation concurrently | Database constraints prevent duplicate creation |
The worker crashes before committing | The transaction rolls back |
The worker crashes after committing but before acknowledging | A retry detects the completed operation |
An external API times out | The retry uses the same provider idempotency key |
A scheduled job overlaps | Coordination prevents unwanted overlap, and repeated work remains safe |
A database constraint rejects a duplicate | The job handles the specific duplicate condition correctly |
Include these scenarios in automated integration tests where practical. For concurrency tests, use separate database contexts and independent connections so the test actually exercises competing operations.
Common Mistakes to Avoid
Generating a new idempotency key for every retry
A new key makes a retry look like a new operation. Generate the identifier once and preserve it across attempts.
Relying only on an existence check
Two workers can both check for a record before either creates it. Use a database uniqueness constraint or another atomic coordination mechanism.
Recording completion outside the business transaction
If the business update and processed-message record are committed separately, a crash can leave the system in an inconsistent state. Use a shared transaction when both changes belong in the same database.
Assuming a distributed lock solves every problem
Locks help coordinate execution, but they do not replace idempotent business operations. Workers can crash, lose leases, or retry after an uncertain result.
Retrying every exception indefinitely
Some failures are permanent, such as invalid input or a violated business rule. Retry transient failures with a limit and an appropriate backoff strategy. Send exhausted messages to a dead-letter queue or another controlled failure workflow when the messaging system supports it.
Assuming database transactions cover external systems
A SQL transaction cannot undo an external payment or email. Use provider-supported idempotency, durable workflow state, and appropriate reconciliation procedures.
Best Practices for Production Applications
Assign a stable identifier to every logical operation.
Enforce uniqueness at the database level.
Commit related database changes in one transaction.
Use provider-supported idempotency keys for external operations.
Make message handlers safe to retry.
Use distributed coordination when multiple instances can run the same scheduled job.
Distinguish transient failures from permanent failures.
Log the operation ID and attempt number for troubleshooting.
Test duplicate delivery, concurrent execution, and crashes at different stages.
Define how failed operations can be inspected, retried, or reconciled.
Summary
Background jobs must be designed for retries, duplicate delivery, and unexpected failures. Preventing a job from running twice is not always possible, but preventing repeated execution from producing duplicate business effects is achievable.
Use stable operation identifiers, database constraints, and transactions for database changes. For external APIs, reuse idempotency keys when the provider supports them. For distributed jobs, combine coordination with idempotent operations rather than relying on locks alone.
These practices make background processing more reliable and help protect important business operations such as payments, invoicing, notifications, and data synchronization.
Join the conversation! Your thoughts help the community grow.