AI agents are becoming more capable of handling tasks that take longer than a typical request-response cycle.
An agent might analyze a large codebase, process thousands of records, run a sequence of tools, wait for external systems, or perform a multi-step research workflow.
That creates a problem that does not appear in the same way with short-lived applications:
What happens if the process stops halfway through the task?
Consider an agent that has already completed seven of ten steps:
Start
|
v
Step 1 - Read requirements
|
v
Step 2 - Search data
|
v
Step 3 - Analyze results
|
v
Step 4 - Call API
|
v
Step 5 - Generate output
|
v
Step 6 - Validate
|
v
Step 7 - Update state
|
X Process crashes
|
v
Step 8 - ?
Without durable execution, the application may lose the information required to continue.
With durable execution, the workflow can persist its state and resume from a known point.
What Is Durable Execution?
Durable execution means that the important state of a long-running workflow survives process failures.
Instead of keeping the entire workflow state in application memory:
Agent Process
|
+-- Workflow State
|
+-- Current Step
|
+-- Tool Results
the application persists important state externally:
Agent Process
|
v
Durable Workflow State
|
+-- Current Step
+-- Completed Steps
+-- Tool Results
+-- Retry Information
If the process terminates, another worker can load that state and continue the workflow.
The key idea is that the workflow should not depend on one running process staying alive for the entire task.
Why Long-Running Agents Need It
A five-second agent and a five-hour agent have very different reliability requirements.
A short request might look like:
Request
|
v
Agent
|
v
Response
If the process fails, the user can often retry the request.
A long-running workflow is different:
Start
|
+-- 20 minutes of processing
|
+-- External API call
|
+-- 40 minutes of analysis
|
+-- Database updates
|
X-- Process failure
Restarting from the beginning may waste significant work.
It can also create duplicate side effects.
Durable Execution vs Checkpointing
These concepts are related but not identical.
Checkpointing usually means saving the current state at specific points.
For example:
Step 1
|
v
Checkpoint
|
v
Step 2
|
v
Checkpoint
|
v
Step 3
|
v
Checkpoint
If the process fails after Step 3 begins, the application can resume from the latest checkpoint.
Durable execution is broader. A durable workflow can maintain execution state, retries, timers, external events, and workflow progress as part of the execution model.
For an AI agent, that can be particularly useful because agent workflows often contain many different types of operations.
A Long-Running Agent in C#
A simple workflow might be represented like this:
public sealed class AgentWorkflowState
{
public string WorkflowId { get; init; } = string.Empty;
public int CurrentStep { get; set; }
public List<string> CompletedSteps { get; init; } = [];
public string? LastResult { get; set; }
}
The workflow can persist this state after meaningful operations.
public async Task SaveStateAsync(
AgentWorkflowState state,
CancellationToken cancellationToken)
{
// Persist workflow state in durable storage.
await repository.SaveAsync(
state,
cancellationToken);
}
The important part is not the particular storage implementation.
The important part is that the state exists outside the process.
What Should Be Persisted?
Do not automatically persist everything the agent produces.
Persist information required to safely continue the workflow.
Typical state may include:
State | Why It Matters |
|---|
Workflow ID | Identifies the execution |
Current step | Determines where execution should continue |
Completed steps | Prevents unnecessary repetition |
Tool results | Avoids repeating expensive operations |
Retry count | Controls retry behavior |
External operation IDs | Helps track side effects |
User-approved decisions | Preserves human decisions |
Error state | Allows recovery or intervention |
Large model prompts or intermediate data may require a different storage strategy.
The workflow state should contain references when storing the complete payload would be unnecessarily expensive.
Do Not Store the Entire Agent in Memory
A fragile design looks like this:
var agentState = new AgentState();
await RunStep1Async(agentState);
await RunStep2Async(agentState);
await RunStep3Async(agentState);
await RunStep4Async(agentState);
If the process terminates between Step 3 and Step 4, the in-memory state disappears.
A more durable approach is:
var state =
await repository.LoadAsync(
workflowId,
cancellationToken);
await RunStepAsync(state, cancellationToken);
await repository.SaveAsync(
state,
cancellationToken);
The application can reconstruct the workflow from persisted state.
Idempotency Is Critical
Durable execution introduces another important concept: idempotency.
Suppose an agent performs this operation:
Create customer
The application writes:
INSERT INTO customers (...)
VALUES (...);
The process crashes immediately after the database accepts the operation but before the workflow records that the step completed.
When the workflow resumes, it may execute the same operation again.
You could end up with:
First attempt -> Customer created
Retry -> Customer created again
The workflow needs a way to safely determine whether the operation has already happened.
One approach is an idempotency key:
Workflow ID: wf-1001
Step: CreateCustomer
Operation ID: wf-1001-create-customer
The database can enforce uniqueness:
CREATE UNIQUE INDEX ux_customer_operation
ON customer_operations(operation_id);
The exact design depends on the operation, but the principle is important:
A retry should not accidentally repeat an irreversible side effect.
Separate Work From Side Effects
AI agent workflows often mix computation and external actions.
For example:
Analyze Document
|
v
Generate Recommendation
|
v
Send Email
|
v
Update CRM
The first two operations may be relatively easy to retry.
The last two can create external side effects.
A durable architecture should track those operations separately.
Workflow
|
+-- Analysis
|
+-- Recommendation
|
+-- Email Operation ID
|
+-- CRM Operation ID
This makes recovery easier.
Handling External API Calls
Suppose an agent calls an external API:
var response =
await httpClient.PostAsJsonAsync(
"/api/tasks",
request,
cancellationToken);
The application needs to consider what happens if the network connection fails after the external system receives the request.
There are now two possibilities:
Request sent
|
+---- Response received
|
+---- Connection lost
A timeout does not necessarily mean that the external operation failed.
The workflow should therefore use an operation identifier where the external system supports it.
Operation ID
|
v
External API
|
v
Result
If the agent retries, the external system can recognize the same operation instead of creating a duplicate.
Retry Policies Need Boundaries
Retries are useful when failures are temporary.
For example:
Attempt 1
|
X Timeout
|
Attempt 2
|
X Timeout
|
Attempt 3
|
v
Success
But retrying everything indefinitely is dangerous.
A workflow should distinguish between:
A simple retry policy might look like:
for (var attempt = 1; attempt <= 3; attempt++)
{
try
{
return await ExecuteAsync(
cancellationToken);
}
catch (HttpRequestException)
when (attempt < 3)
{
await Task.Delay(
TimeSpan.FromSeconds(attempt * 2),
cancellationToken);
}
}
throw new InvalidOperationException(
"Operation failed after retries.");
Production systems often use a resilience library or workflow platform rather than implementing all retry behavior manually.
What Happens During a Deployment?
Long-running agents create an operational challenge during deployments.
Suppose:
Version 1
|
v
Agent running for 3 hours
A deployment starts:
Version 2 deployed
If workflow state exists only in memory, the running task may disappear.
With durable state:
Version 1
|
v
Persisted Workflow
|
v
Version 2
|
v
Resume Workflow
This makes deployments less disruptive.
However, workflow state and code changes must remain compatible.
If Version 2 changes the structure or meaning of persisted state, migration or versioning may be necessary.
Workflow Versioning
Consider this state:
{
"workflowId": "wf-1001",
"step": 4,
"status": "Running"
}
A newer application version might expect a different workflow structure.
One way to manage this is to include a workflow version:
{
"workflowId": "wf-1001",
"workflowVersion": 2,
"step": 4,
"status": "Running"
}
The application can then determine how to interpret the stored state.
This becomes increasingly important as workflows remain active for hours or days.
Durable Execution Does Not Mean Durable Everything
It is tempting to persist every intermediate model response, tool call, and application object.
That can create unnecessary storage and operational complexity.
Instead, define the minimum state required for recovery.
For example:
Persist:
- Workflow state
- Step status
- Important tool results
- External operation IDs
- User decisions
Do not automatically persist:
- Temporary objects
- Reconstructable data
- Unnecessary duplicate model output
The correct boundary depends on the workflow.
Common Mistakes
Keeping Workflow State Only in Memory
A process restart can destroy the workflow's progress.
Retrying Non-Idempotent Operations
A retry can create duplicate records, payments, messages, or other side effects.
Treating Every Error as Temporary
Some errors should fail immediately instead of being retried.
Saving State Too Infrequently
If checkpoints are too far apart, a failure can force the workflow to repeat substantial work.
Saving Everything
Excessive persistence can increase storage costs and make state management harder.
Ignoring Deployment Changes
Long-running workflows may outlive the application version that created them.
Forgetting Cancellation
Users and operators should have a way to stop long-running workflows safely.
Best Practices for Production
Give Every Workflow a Stable ID
A unique workflow identifier makes state, logs, retries, and external operations easier to correlate.
Persist Meaningful Checkpoints
Checkpoint after operations where losing progress would be expensive.
Make Side Effects Idempotent
Use operation IDs, unique constraints, or equivalent mechanisms where appropriate.
Keep Transactions Short
Do not hold database transactions open while waiting for model inference or external services.
Separate Durable State From Large Payloads
Store large artifacts separately and keep references in the workflow state when appropriate.
Record Failure Information
Persist enough information to understand why a workflow stopped.
Support Resume and Cancellation
A durable workflow should have explicit states such as:
Pending
Running
Waiting
Completed
Failed
Cancelled
Monitor Workflow Age
A workflow that has been running for hours may be legitimate, but one that remains stuck for days should be visible to operators.
Advantages and Disadvantages
Advantages
Survives application restarts
Reduces repeated work after failures
Supports long-running workflows
Makes retries safer
Improves deployment resilience
Provides better operational visibility
Supports human intervention and recovery
Disadvantages
Adds state-management complexity
Requires durable storage
Requires careful idempotency design
Workflow versioning becomes important
Debugging can involve multiple components
Poorly designed persistence can create unnecessary storage overhead
Troubleshooting Long-Running Agent Failures
When an agent fails after running for a long time, check:
Identify the workflow ID.
Load the latest persisted state.
Determine the last completed step.
Check whether the current operation created an external side effect.
Verify whether that operation is idempotent.
Review retry history.
Check external API status.
Check database transactions and locks.
Verify workflow version compatibility.
Resume only after determining whether the failed step is safe to repeat.
Do not blindly restart the entire workflow.
The correct recovery point depends on the last durable checkpoint and the side effects already produced.
Summary of the Article
Long-running AI agents cannot safely depend on a single application process remaining alive until the task finishes. Processes restart, machines fail, deployments happen, networks disconnect, and external services become unavailable.
Durable execution addresses this problem by persisting the important state of the workflow so that another process can continue the task after a failure.
For C# applications, a durable agent architecture should maintain a stable workflow ID, persist meaningful checkpoints, handle retries carefully, and make external side effects idempotent wherever possible. Database operations, API calls, model execution, and workflow state should be treated as separate concerns rather than one long-running in-memory operation.
The most important design principle is simple: a long-running AI agent should be able to lose its process without losing the workflow itself.
Join the conversation! Your thoughts help the community grow.