Background jobs are commonly used to move slow or long-running work away from the main API request. Instead of making a user wait while an operation completes, the API can place a job into a queue and let a background worker process it separately.
This improves API responsiveness, but it also creates a different failure scenario: the API or another dependency can go down while a background job is already being processed.
For example, imagine an order-processing API accepts a request and creates a background job:
Client
|
v
API
|
v
Queue
|
v
Background Worker
|
v
Database / External Service
The worker may successfully start the job but fail halfway through the operation because the API, database, external service, or worker itself becomes unavailable.
The important question is not only how to restart the job. The system must also determine whether the previous attempt completed, partially completed, or failed before any useful result was recorded.
What Is a Background Job?
A background job is work that is executed outside the normal HTTP request-response cycle.
Instead of performing everything while the client waits:
HTTP Request
|
v
Process Everything
|
v
HTTP Response
the application can return quickly and process the work separately:
HTTP Request
|
v
Create Job
|
v
Queue
|
v
HTTP Response
Later...
Queue
|
v
Worker
|
v
Process Job
Background jobs are useful for operations such as sending emails, generating reports, processing uploaded files, synchronizing data, publishing events, and performing long-running business operations.
The advantage is that the API does not need to keep an HTTP request open while the work continues.
The trade-off is that the application now needs a reliable way to track job state, handle failures, and decide whether a failed job can safely be retried.
What Can Go Wrong Mid-Process?
Consider a job that performs several steps:
Receive Job
|
v
Validate Order
|
v
Update Database
|
v
Call Payment Service
|
v
Update Order Status
|
v
Send Notification
|
v
Mark Job Complete
Now imagine the worker fails after the payment succeeds but before the order status is updated.
The job is no longer simply "failed."
Something already happened.
If the system retries the entire job without understanding the previous state, it could accidentally charge the customer again or create duplicate side effects.
This is one of the most important problems in background job processing.
The Difference Between Failure and Partial Completion
A job can fail at several different stages.
For example:
Job started
|
v
Database update succeeds
|
v
External API succeeds
|
v
Worker crashes
|
v
Job appears incomplete
The worker process may report failure, but the external system may already have accepted the operation.
This creates uncertainty.
The application needs to answer:
Did the operation actually happen?
That question is often more important than:
Did the worker process crash?
A robust background processing design therefore keeps track of business state separately from the worker's process state.
Why Simply Retrying Can Be Dangerous
Retrying is one of the most common ways to handle temporary failures.
For example:
Job fails
|
v
Wait
|
v
Retry
|
v
Success
This works well when the failure happened before any external side effect occurred.
But consider a payment operation:
Worker
|
v
Send payment request
|
v
Payment succeeds
|
v
Worker crashes
The worker restarts and retries:
Worker
|
v
Send payment request again
If the payment service does not recognize the second request as a duplicate, the customer could be charged twice.
This is why retry logic must be designed around the operation being performed.
Idempotency Makes Retries Safer
Idempotency means that processing the same logical operation more than once produces the same intended business result rather than creating additional unintended effects.
For example, suppose a payment request has an idempotency key:
Order: 10025
Payment request: PAY-10025
The payment service records that request identifier.
If the worker sends the same request again:
PAY-10025
the payment service can recognize that the operation has already been processed and avoid charging the customer again.
The exact implementation depends on the system, but the principle is important:
Same business operation
+
Repeated delivery
=
One intended result
This is especially important for jobs that perform payments, order creation, inventory changes, email delivery, or other operations that have external side effects.
Store Job State Explicitly
A background job should normally have a persistent state that can be inspected independently of the worker process.
For example:
Queued
|
v
Processing
|
+---- Failed
|
+---- Completed
A job record might contain information such as:
JobId
Status
CreatedAt
StartedAt
CompletedAt
AttemptCount
LastError
This gives the application a durable record of what happened.
For example:
JobId: 84721
Status: Processing
AttemptCount: 2
StartedAt: ...
LastError: Timeout
If the worker crashes, another worker can inspect the state and determine whether the job needs to be retried or reconciled.
Do Not Use Worker Memory as the Source of Truth
A common mistake is storing important job state only in the worker process.
For example:
private readonly Dictionary<Guid, string> _jobs = new();
This state disappears when the process restarts.
The same problem occurs if the worker stores important progress only in memory.
Production job state should normally be stored in durable infrastructure such as a database or a reliable job-processing system.
The worker should be treated as an executor, not the permanent source of truth for business state.
Use Transactions for Local Database Changes
Suppose a background job performs several database changes that must succeed together.
For example:
Create Invoice
|
v
Update Order
|
v
Record Payment Status
If the worker crashes halfway through these operations, the database could be left in an inconsistent state.
A database transaction can help when all affected operations belong to the same database and can safely be committed together.
For example:
await using var transaction =
await connection.BeginTransactionAsync();
await CreateInvoiceAsync(connection, transaction);
await UpdateOrderAsync(connection, transaction);
await RecordPaymentStatusAsync(connection, transaction);
await transaction.CommitAsync();
If an error occurs before the commit:
Transaction
|
v
Rollback
the database changes can be reverted together.
However, database transactions do not automatically cover external systems.
You cannot normally put a payment API, an email provider, and your database into one ordinary database transaction.
That is where idempotency, workflow state, and reconciliation become important.
The Problem With External APIs
Consider this workflow:
Database Update
|
v
Payment API
|
v
Notification API
Suppose the database update succeeds and the payment API succeeds, but the notification API fails.
Should the entire job be retried?
Not necessarily.
Repeating the payment operation could be dangerous.
Instead, the application may need to continue from the failed step or verify the previous state before retrying.
This is why complex background jobs are often better represented as explicit workflows rather than one large method containing every operation.
Track Progress for Multi-Step Jobs
For a multi-step job, the application can record progress.
For example:
Job: 84721
Step 1: Validate Order Completed
Step 2: Reserve Inventory Completed
Step 3: Process Payment Completed
Step 4: Create Shipment Failed
Step 5: Send Notification Pending
If the worker crashes during Step 4, the recovery process does not need to repeat Steps 1–3 blindly.
It can inspect the recorded state and continue from the appropriate point.
This approach is particularly useful for long-running workflows.
However, the state transitions themselves must be carefully designed. Recording a step as completed before the underlying operation actually succeeds can create a different type of inconsistency.
What Happens When the API Goes Down?
The phrase "API goes down" can mean several different things.
The API may become unavailable while the background worker continues running.
API DOWN
Worker RUNNING
Queue RUNNING
Database RUNNING
In this case, jobs may continue processing if the worker does not depend on the API.
But the worker may itself call the API:
Worker
|
v
Internal API
|
v
Database
If that API becomes unavailable, the worker's processing may fail.
The worker could then retry, delay the job, or move it into a failure state depending on the configured failure policy.
The important point is to identify which component actually failed.
What Happens When the Worker Goes Down?
This is another common scenario.
Suppose a worker receives a job and starts processing it:
Queue
|
v
Worker
|
v
Processing
The worker process crashes.
If the message acknowledgement happened before processing was complete, the queue may consider the message successfully handled even though the business operation was incomplete.
If acknowledgement happens only after successful processing, the message can generally become available for another delivery after the consumer connection is lost.
This is why acknowledgement timing matters.
A common principle is:
Process successfully
|
v
Acknowledge message
rather than:
Receive message
|
v
Acknowledge immediately
|
v
Process message
The second pattern can lose work if the process crashes after acknowledgement but before processing finishes.
Be Careful With Long-Running Jobs
Long-running jobs create additional failure scenarios.
Suppose a report-generation job takes 30 minutes.
During processing:
Worker starts
|
v
10 minutes
|
v
20 minutes
|
v
Worker crashes
If the job must restart from the beginning, the first 20 minutes of work may be lost.
For expensive operations, consider checkpointing progress where practical.
For example:
Files 1-1000 -> Completed
Files 1001-2000 -> Completed
Files 2001-3000 -> Processing
If the worker fails, the recovery process can continue from the last safe checkpoint.
The checkpoint itself should be stored durably.
Retry With Backoff
When a dependency is temporarily unavailable, retrying immediately can make the outage worse.
Suppose 1,000 jobs fail because a service is unavailable.
If every worker immediately retries:
Failure
|
v
Immediate retry
|
v
Failure
|
v
Immediate retry
the failing service receives another large burst of requests.
A better approach is delayed retry with backoff.
For example:
Attempt 1 -> Fail
Wait 1 second
Attempt 2 -> Fail
Wait 5 seconds
Attempt 3 -> Fail
Wait 30 seconds
The exact timing depends on the workload and dependency, but the objective is to reduce pressure during temporary failures.
Randomized delay can also prevent many workers from retrying at exactly the same time.
Dead-Letter Queues Help Isolate Failed Jobs
A dead-letter queue is a separate queue used to hold messages that cannot be successfully processed after the allowed retry attempts.
For example:
Main Queue
|
v
Worker
|
+---- Success
|
+---- Temporary failure -> Retry
|
+---- Repeated failure -> Dead-Letter Queue
This prevents one permanently failing message from being retried indefinitely.
The dead-letter queue should also be monitored. Moving a message there is not the same as resolving the problem.
Teams should know why messages were dead-lettered and have a controlled process for investigating and replaying them when appropriate.
Observability Is Critical
Background jobs are harder to troubleshoot when there is no clear relationship between the original request and the asynchronous work.
Use a correlation or job identifier that can be followed through the system.
For example:
Request ID: 7f82
Job ID: 84721
Order ID: 10025
Logs can then connect:
API request
|
v
Job created
|
v
Worker started
|
v
Database operation
|
v
External API call
|
v
Job completed
Useful metrics include:
Jobs created
Jobs completed
Jobs failed
Retry count
Processing duration
Queue depth
Oldest queued job
Dead-letter count
Job age
Dependency latency
Without these metrics, a background processing problem can be difficult to distinguish from an API or database problem.
A Practical Failure-Handling Model
A production background job system can follow a model like this:
Job Created
|
v
Queued
|
v
Processing
|
+----------+----------+
| |
Success Failure
| |
v v
Completed Retry Policy
|
+---------+---------+
| |
Retry Available Retry Exhausted
| |
v v
Processing Dead-Letter
For operations with external side effects, add state verification and idempotency to the retry path.
This prevents the recovery mechanism from blindly repeating an operation that may already have succeeded.
Common Mistakes
Acknowledging Before Processing Finishes
If the job is acknowledged too early, a worker crash can cause work to disappear from the queue.
Retrying Everything
Not every failure is temporary. Invalid data and permanent business failures should not be retried indefinitely.
Assuming a Failed Job Had No Side Effects
A worker can fail after an external operation has already succeeded. Always consider partial completion.
Keeping Job State Only in Memory
Process memory disappears when the worker restarts. Important job state should be durable.
Ignoring Idempotency
Retrying non-idempotent operations can create duplicate payments, records, notifications, or other side effects.
Putting the Entire Workflow in One Transaction
Database transactions cannot automatically roll back changes made in external systems.
Monitoring Only Worker Availability
A worker process can be running while every job it receives is failing or taking far too long.
Best Practices
Persist important job state and make the recovery process independent of worker memory.
Acknowledge messages only after successful processing when the messaging system and workload require at-least-once processing.
Design important operations to be idempotent so that retries do not create duplicate side effects.
Use retry limits, backoff, and dead-letter handling for repeated failures.
Keep multi-step workflows explicit when partial completion is possible.
Use transactions for related database changes, but do not assume they protect external systems.
Track correlation IDs, job IDs, processing duration, retry counts, and failure reasons.
Test worker crashes and dependency failures deliberately so the recovery behavior is known before production incidents occur.
Summary
Background jobs can fail after doing part of the work, so simply retrying the entire job is not always safe.
Use durable job state, idempotency, controlled retries, transactions where appropriate, and clear failure handling to make recovery predictable.
The goal is not just to restart failed jobs. It is to recover them without losing work or performing the same side effect twice.
Join the conversation! Your thoughts help the community grow.