Long-running workflows are different from ordinary application requests.
A normal HTTP request might finish in a few milliseconds or seconds. A business workflow may need to run for minutes, hours, or even days while waiting for external systems, approvals, scheduled events, retries, or human interaction.
Trying to keep such workflows alive inside a normal web request creates reliability problems.
The application can restart. A container can be replaced. A network connection can fail. An external API can become unavailable. A process can crash halfway through an operation.
Durable execution addresses this problem by separating workflow state from the lifetime of the process executing it.
Instead of depending on one application instance staying alive, the workflow records its progress so it can continue after interruptions.
For .NET applications, this pattern is particularly useful for order processing, financial workflows, document processing, data pipelines, provisioning, and AI-assisted multi-step operations.
What Is Durable Execution?
Durable execution allows a workflow to preserve its state and continue from a known point after failures.
Consider an order-processing workflow:
Receive Order
↓
Validate Payment
↓
Reserve Inventory
↓
Create Shipment
↓
Send Confirmation
A traditional implementation may execute everything inside one process.
If the process crashes after inventory reservation:
Receive Order
↓
Validate Payment
↓
Reserve Inventory
↓
X Process Crash
The application needs to determine what happened before it can safely continue.
With durable execution, workflow progress becomes persistent:
Workflow State
↓
Step 1 Completed
↓
Step 2 Completed
↓
Step 3 Completed
After a restart, the workflow engine can resume from the appropriate point instead of treating the workflow as completely new.
Why Long-Running Workflows Are Difficult
Long-running workflows introduce several failure modes.
Process Failure
A worker can terminate unexpectedly.
Infrastructure Failure
A container, virtual machine, or server can disappear.
Network Failure
A downstream service may timeout or become temporarily unavailable.
Dependency Failure
A database, payment service, or third-party API may become unavailable.
Partial Completion
The most difficult situation is when the application does not know whether an external operation succeeded.
For example:
Application
↓
Payment Service
↓
Payment succeeds
↓
Network connection fails
The application receives a timeout.
Did the payment fail?
Or did the payment succeed but the response disappear?
Durable workflows need explicit strategies for handling these cases.
Durable Execution vs Background Jobs
Background jobs and durable workflows solve related but different problems.
A background job generally looks like:
Queue
↓
Worker
↓
Execute Job
This works well for independent tasks.
A durable workflow is more structured:
Workflow
↓
Step A
↓
Step B
↓
Wait
↓
Step C
↓
Retry
↓
Step D
The workflow needs to remember:
Which steps completed
Which step is currently executing
Which events have been received
Which retries have occurred
What data is required for the next step
When execution should resume
That state is the key distinction.
A Typical .NET Architecture
A durable workflow architecture can be organized into several components:
┌─────────────────┐
│ .NET API │
└────────┬────────┘
│
▼
┌─────────────────┐
│ Workflow Engine │
└────────┬────────┘
│
┌───────────┼───────────┐
▼ ▼ ▼
Activity A Activity B Activity C
│ │ │
▼ ▼ ▼
Database API Queue
│
▼
┌─────────────────┐
│ Durable State │
└─────────────────┘
The API starts the workflow, while workers execute individual activities.
The durable state allows the workflow to survive worker failures.
Model Workflows as Explicit Steps
A good workflow should have clearly defined activities.
For example:
ProcessOrder
├── ValidateOrder
├── AuthorizePayment
├── ReserveInventory
├── CreateShipment
└── NotifyCustomer
This structure makes failures easier to reason about.
If ReserveInventory fails, the workflow knows which activity failed rather than having to infer the state of an arbitrary block of application code.
Keep Activities Small
A common mistake is putting too much work inside a single activity.
Avoid:
ProcessEverything()
Prefer:
ValidateOrder()
AuthorizePayment()
ReserveInventory()
CreateShipment()
SendNotification()
Smaller activities provide better observability and retry boundaries.
They also make recovery easier.
Retries Need Idempotency
Retries are one of the most important parts of durable execution.
Suppose:
AuthorizePayment()
times out.
The workflow retries the operation.
But the original request might already have succeeded.
You could accidentally charge the customer twice.
The solution is idempotency.
An operation is idempotent when executing it multiple times produces the same intended result as executing it once.
For example:
public async Task<PaymentResult> AuthorizePayment(
PaymentRequest request,
CancellationToken cancellationToken)
{
var existing = await paymentStore
.FindByIdempotencyKeyAsync(
request.IdempotencyKey,
cancellationToken);
if (existing is not null)
{
return existing;
}
var result = await paymentGateway.AuthorizeAsync(
request,
cancellationToken);
await paymentStore.SaveAsync(
request.IdempotencyKey,
result,
cancellationToken);
return result;
}
The idempotency key allows a retry to recognize that the operation has already been processed.
Design Every External Activity for Failure
External calls should never be treated as guaranteed.
Consider:
var response = await httpClient.PostAsJsonAsync(
"/inventory/reserve",
request,
cancellationToken);
The call can fail because of:
Timeout
DNS failure
Connection reset
HTTP 5xx response
Rate limiting
Authentication problems
Service outage
A durable workflow should classify failures instead of retrying everything.
Not Every Error Should Be Retried
Transient failures are usually good retry candidates:
HTTP 408
HTTP 429
HTTP 500
HTTP 502
HTTP 503
HTTP 504
Permanent failures generally require different handling:
Invalid request
Invalid customer
Insufficient inventory
Invalid authentication
Business rule violation
Blind retries can turn a simple failure into a larger outage.
Use Exponential Backoff
A workflow should generally avoid retrying immediately in a tight loop.
Instead:
Attempt 1 → 1 second
Attempt 2 → 2 seconds
Attempt 3 → 4 seconds
Attempt 4 → 8 seconds
A production system should also use jitter so large numbers of workflows do not retry simultaneously.
Conceptually:
Retry Delay =
Exponential Backoff
+
Random Jitter
This reduces synchronized retry traffic during dependency outages.
Long Waits Should Not Consume Workers
One of the strongest benefits of durable execution is handling long waits efficiently.
Imagine a workflow that waits for customer approval.
A poor implementation might keep a worker running:
Worker
↓
Wait 24 Hours
↓
Continue
That wastes execution capacity.
A durable workflow can instead persist its state:
Workflow
↓
Waiting for Approval
↓
Persist State
↓
Release Worker
↓
Approval Event
↓
Resume Workflow
This is particularly valuable for workflows involving:
Event-Driven Workflow Resumption
A workflow can wait for an external event:
Order Created
↓
Payment Pending
↓
Wait
↓
Payment Confirmed
↓
Reserve Inventory
The payment system does not need to keep the original HTTP request alive.
Instead, it can notify the workflow when the payment state changes.
This creates a more resilient architecture:
Payment Service
│
│ Event
▼
Workflow Engine
│
▼
Continue Workflow
Durable State Should Contain Business Progress
The workflow should persist information required to resume execution.
For example:
public sealed record OrderWorkflowState(
Guid OrderId,
bool PaymentAuthorized,
bool InventoryReserved,
bool ShipmentCreated,
string? ShipmentId);
The exact state model depends on the workflow engine, but the principle remains the same.
Persist business state that is necessary for recovery.
Do not assume in-memory variables will still exist after a restart.
Avoid Storing Everything in Workflow State
Durable state is not a replacement for your primary database.
Large objects, files, and frequently changing domain data should generally remain in appropriate storage.
A workflow can store references:
Workflow State
↓
Order ID
↓
Database
↓
Order Details
instead of copying the entire order into workflow state.
This keeps workflow state smaller and easier to manage.
Handling Compensation
Distributed workflows often cannot use a traditional database transaction across every system.
Consider:
Charge Payment
↓
Reserve Inventory
↓
Create Shipment
Suppose shipment creation fails after payment and inventory reservation succeed.
You may need compensation:
Create Shipment
↓
Failure
↓
Release Inventory
↓
Refund Payment
This is commonly associated with the Saga pattern.
The workflow explicitly defines what should happen when a later step cannot complete.
Durable Execution and the Saga Pattern
A durable workflow can act as the coordinator for a Saga:
Step 1
↓
Compensation 1
Step 2
↓
Compensation 2
Step 3
↓
Compensation 3
For example:
Reserve Inventory
↓
Create Shipment
↓
Charge Customer
If charging fails:
Charge Customer
X
↓
Cancel Shipment
↓
Release Inventory
The important point is that compensation is a business operation, not a database rollback.
Observability Is Essential
Long-running workflows are difficult to troubleshoot without good telemetry.
Track:
Workflow ID
Instance ID
Activity name
Attempt number
Start time
Completion time
Failure reason
Retry count
Current state
External dependency
Correlation ID
A useful trace looks like:
Workflow: OrderProcessing
Instance: 8f31...
ValidateOrder 120 ms
AuthorizePayment 340 ms
ReserveInventory 95 ms
CreateShipment 410 ms
SendNotification 80 ms
When a workflow takes 30 minutes instead of 2 seconds, observability should make it clear where the time was spent.
Make Workflow Execution Deterministic
Durable workflow systems often replay workflow logic to reconstruct state.
That means workflow orchestration code should avoid relying directly on nondeterministic values such as:
DateTime.UtcNow
Guid.NewGuid()
Random.Shared
inside replay-sensitive orchestration logic.
Instead, use the workflow framework's supported abstractions for time, identifiers, timers, and external operations.
Activities can perform nondeterministic work and return results to the workflow.
The exact rules depend on the durable execution technology being used, but the general principle is important:
Workflow orchestration should describe durable decisions, while activities perform external side effects.
Durable Execution for AI Workflows
The same pattern becomes increasingly useful for AI-powered systems.
Consider an agent workflow:
Receive Request
↓
Retrieve Context
↓
Call Model
↓
Execute Tool
↓
Validate Result
↓
Request Approval
↓
Execute Action
An AI workflow can fail at almost any point.
A model request may timeout. A tool may become unavailable. A human approval may take hours.
Durable execution allows the workflow to preserve progress between those stages.
This is particularly useful when AI agents interact with external systems and business processes.
Common Mistakes
Treating a Workflow Like an HTTP Request
Long-running operations should not depend on one HTTP request remaining alive.
Retrying Non-Idempotent Operations
Retries can duplicate payments, orders, messages, or other side effects.
Retrying Every Exception
Permanent business failures should not be repeatedly retried.
Keeping Workers Alive During Long Waits
Use durable timers or events rather than consuming worker capacity unnecessarily.
Storing Large Objects in Workflow State
Keep large data in appropriate storage and store references in the workflow.
Ignoring Compensation
Distributed operations often require explicit rollback-like business actions.
Poor Observability
Without workflow and activity identifiers, diagnosing long-running failures becomes difficult.
Best Practices for Production .NET Workflows
Model workflows as explicit business steps.
Keep activities focused and independently retryable.
Make external operations idempotent.
Classify transient and permanent failures.
Use exponential backoff with jitter.
Persist state required for recovery.
Use durable timers for long waits.
Resume workflows through events instead of holding connections open.
Design compensation actions for partially completed workflows.
Keep large payloads outside workflow state.
Add distributed tracing and correlation identifiers.
Monitor workflow duration and retry rates.
Test process crashes and dependency failures.
Test duplicate events and duplicate activity execution.
Verify recovery behavior before deploying to production.
How to Test Failure Recovery
A durable workflow should not be tested only when everything works.
Introduce controlled failures:
Test 1
Step 2 fails
Test 2
Worker crashes after Step 2
Test 3
External API times out
Test 4
External API succeeds but response is lost
Test 5
Workflow receives duplicate event
Test 6
Workflow waits for 24 hours
Test 7
Workflow resumes after application restart
The goal is to verify that the final business state remains correct.
For example:
Payment = Authorized
Inventory = Reserved
Shipment = Created
should remain consistent even when individual execution attempts fail.
Conclusion
Long-running workflows require a different reliability model from ordinary request-response applications.
The central idea behind durable execution is straightforward:
Process Lifetime
≠
Workflow Lifetime
A workflow should be able to survive the failure or replacement of the process executing it.
For .NET applications, this means designing workflows around durable state, explicit activities, retries, idempotency, timers, events, compensation, and observability.
The most important architectural shift is to stop thinking of a workflow as one long-running method.
Instead, treat it as a durable state machine:
Start
↓
Activity
↓
Persist
↓
Activity
↓
Wait
↓
Event
↓
Resume
↓
Complete
That approach makes long-running .NET applications more resilient to infrastructure failures, dependency outages, retries, and interruptions while providing a much clearer foundation for reliable business and AI workflows.
Join the conversation! Your thoughts help the community grow.