Why AI Agents Need Durable Execution When Tasks Run for Hours

AI agents are becoming more capable of handling tasks that take longer than a typical request-response cycle.

An agent might analyze a large codebase, process thousands of records, run a sequence of tools, wait for external systems, or perform a multi-step research workflow.

That creates a problem that does not appear in the same way with short-lived applications:

What happens if the process stops halfway through the task?

Consider an agent that has already completed seven of ten steps:

Start
  |
  v
Step 1 - Read requirements
  |
  v
Step 2 - Search data
  |
  v
Step 3 - Analyze results
  |
  v
Step 4 - Call API
  |
  v
Step 5 - Generate output
  |
  v
Step 6 - Validate
  |
  v
Step 7 - Update state
  |
  X  Process crashes
  |
  v
Step 8 - ?

Without durable execution, the application may lose the information required to continue.

With durable execution, the workflow can persist its state and resume from a known point.

What Is Durable Execution?

Durable execution means that the important state of a long-running workflow survives process failures.

Instead of keeping the entire workflow state in application memory:

Agent Process
     |
     +-- Workflow State
     |
     +-- Current Step
     |
     +-- Tool Results

the application persists important state externally:

Agent Process
     |
     v
Durable Workflow State
     |
     +-- Current Step
     +-- Completed Steps
     +-- Tool Results
     +-- Retry Information

If the process terminates, another worker can load that state and continue the workflow.

The key idea is that the workflow should not depend on one running process staying alive for the entire task.

Why Long-Running Agents Need It

A five-second agent and a five-hour agent have very different reliability requirements.

A short request might look like:

Request
  |
  v
Agent
  |
  v
Response

If the process fails, the user can often retry the request.

A long-running workflow is different:

Start
 |
 +-- 20 minutes of processing
 |
 +-- External API call
 |
 +-- 40 minutes of analysis
 |
 +-- Database updates
 |
 X-- Process failure

Restarting from the beginning may waste significant work.

It can also create duplicate side effects.

Durable Execution vs Checkpointing

These concepts are related but not identical.

Checkpointing usually means saving the current state at specific points.

For example:

Step 1
 |
 v
Checkpoint
 |
 v
Step 2
 |
 v
Checkpoint
 |
 v
Step 3
 |
 v
Checkpoint

If the process fails after Step 3 begins, the application can resume from the latest checkpoint.

Durable execution is broader. A durable workflow can maintain execution state, retries, timers, external events, and workflow progress as part of the execution model.

For an AI agent, that can be particularly useful because agent workflows often contain many different types of operations.

A Long-Running Agent in C#

A simple workflow might be represented like this:

public sealed class AgentWorkflowState
{
    public string WorkflowId { get; init; } = string.Empty;

    public int CurrentStep { get; set; }

    public List<string> CompletedSteps { get; init; } = [];

    public string? LastResult { get; set; }
}

The workflow can persist this state after meaningful operations.

public async Task SaveStateAsync(
    AgentWorkflowState state,
    CancellationToken cancellationToken)
{
    // Persist workflow state in durable storage.
    await repository.SaveAsync(
        state,
        cancellationToken);
}

The important part is not the particular storage implementation.

The important part is that the state exists outside the process.

What Should Be Persisted?

Do not automatically persist everything the agent produces.

Persist information required to safely continue the workflow.

Typical state may include:

State	Why It Matters
Workflow ID	Identifies the execution
Current step	Determines where execution should continue
Completed steps	Prevents unnecessary repetition
Tool results	Avoids repeating expensive operations
Retry count	Controls retry behavior
External operation IDs	Helps track side effects
User-approved decisions	Preserves human decisions
Error state	Allows recovery or intervention

Large model prompts or intermediate data may require a different storage strategy.

The workflow state should contain references when storing the complete payload would be unnecessarily expensive.

Do Not Store the Entire Agent in Memory

A fragile design looks like this:

var agentState = new AgentState();

await RunStep1Async(agentState);
await RunStep2Async(agentState);
await RunStep3Async(agentState);
await RunStep4Async(agentState);

If the process terminates between Step 3 and Step 4, the in-memory state disappears.

A more durable approach is:

var state =
    await repository.LoadAsync(
        workflowId,
        cancellationToken);

await RunStepAsync(state, cancellationToken);

await repository.SaveAsync(
    state,
    cancellationToken);

The application can reconstruct the workflow from persisted state.

Idempotency Is Critical

Durable execution introduces another important concept: idempotency.

Suppose an agent performs this operation:

Create customer

The application writes:

INSERT INTO customers (...)
VALUES (...);

The process crashes immediately after the database accepts the operation but before the workflow records that the step completed.

When the workflow resumes, it may execute the same operation again.

You could end up with:

First attempt  -> Customer created
Retry          -> Customer created again

The workflow needs a way to safely determine whether the operation has already happened.

One approach is an idempotency key:

Workflow ID: wf-1001
Step: CreateCustomer
Operation ID: wf-1001-create-customer

The database can enforce uniqueness:

CREATE UNIQUE INDEX ux_customer_operation
ON customer_operations(operation_id);

The exact design depends on the operation, but the principle is important:

A retry should not accidentally repeat an irreversible side effect.

Separate Work From Side Effects

AI agent workflows often mix computation and external actions.

For example:

Analyze Document
      |
      v
Generate Recommendation
      |
      v
Send Email
      |
      v
Update CRM

The first two operations may be relatively easy to retry.

The last two can create external side effects.

A durable architecture should track those operations separately.

Workflow
   |
   +-- Analysis
   |
   +-- Recommendation
   |
   +-- Email Operation ID
   |
   +-- CRM Operation ID

This makes recovery easier.

Handling External API Calls

Suppose an agent calls an external API:

var response =
    await httpClient.PostAsJsonAsync(
        "/api/tasks",
        request,
        cancellationToken);

The application needs to consider what happens if the network connection fails after the external system receives the request.

There are now two possibilities:

Request sent
   |
   +---- Response received
   |
   +---- Connection lost

A timeout does not necessarily mean that the external operation failed.

The workflow should therefore use an operation identifier where the external system supports it.

Operation ID
     |
     v
External API
     |
     v
Result

If the agent retries, the external system can recognize the same operation instead of creating a duplicate.

Retry Policies Need Boundaries

Retries are useful when failures are temporary.

For example:

Attempt 1
   |
   X Timeout
   |
Attempt 2
   |
   X Timeout
   |
Attempt 3
   |
   v
Success

But retrying everything indefinitely is dangerous.

A workflow should distinguish between:

Temporary network failures
Rate limits
Service unavailability
Invalid input
Authentication failures
Permanent business errors

A simple retry policy might look like:

for (var attempt = 1; attempt <= 3; attempt++)
{
    try
    {
        return await ExecuteAsync(
            cancellationToken);
    }
    catch (HttpRequestException)
        when (attempt < 3)
    {
        await Task.Delay(
            TimeSpan.FromSeconds(attempt * 2),
            cancellationToken);
    }
}

throw new InvalidOperationException(
    "Operation failed after retries.");

Production systems often use a resilience library or workflow platform rather than implementing all retry behavior manually.

What Happens During a Deployment?

Long-running agents create an operational challenge during deployments.

Suppose:

Version 1
   |
   v
Agent running for 3 hours

A deployment starts:

Version 2 deployed

If workflow state exists only in memory, the running task may disappear.

With durable state:

Version 1
   |
   v
Persisted Workflow
   |
   v
Version 2
   |
   v
Resume Workflow

This makes deployments less disruptive.

However, workflow state and code changes must remain compatible.

If Version 2 changes the structure or meaning of persisted state, migration or versioning may be necessary.

Workflow Versioning

Consider this state:

{
  "workflowId": "wf-1001",
  "step": 4,
  "status": "Running"
}

A newer application version might expect a different workflow structure.

One way to manage this is to include a workflow version:

{
  "workflowId": "wf-1001",
  "workflowVersion": 2,
  "step": 4,
  "status": "Running"
}

The application can then determine how to interpret the stored state.

This becomes increasingly important as workflows remain active for hours or days.

Durable Execution Does Not Mean Durable Everything

It is tempting to persist every intermediate model response, tool call, and application object.

That can create unnecessary storage and operational complexity.

Instead, define the minimum state required for recovery.

For example:

Persist:
- Workflow state
- Step status
- Important tool results
- External operation IDs
- User decisions

Do not automatically persist:
- Temporary objects
- Reconstructable data
- Unnecessary duplicate model output

The correct boundary depends on the workflow.

Common Mistakes

Keeping Workflow State Only in Memory

A process restart can destroy the workflow's progress.

Retrying Non-Idempotent Operations

A retry can create duplicate records, payments, messages, or other side effects.

Treating Every Error as Temporary

Some errors should fail immediately instead of being retried.

Saving State Too Infrequently

If checkpoints are too far apart, a failure can force the workflow to repeat substantial work.

Saving Everything

Excessive persistence can increase storage costs and make state management harder.

Ignoring Deployment Changes

Long-running workflows may outlive the application version that created them.

Forgetting Cancellation

Users and operators should have a way to stop long-running workflows safely.

Best Practices for Production

Give Every Workflow a Stable ID

A unique workflow identifier makes state, logs, retries, and external operations easier to correlate.

Persist Meaningful Checkpoints

Checkpoint after operations where losing progress would be expensive.

Make Side Effects Idempotent

Use operation IDs, unique constraints, or equivalent mechanisms where appropriate.

Keep Transactions Short

Do not hold database transactions open while waiting for model inference or external services.

Separate Durable State From Large Payloads

Store large artifacts separately and keep references in the workflow state when appropriate.

Record Failure Information

Persist enough information to understand why a workflow stopped.

Support Resume and Cancellation

A durable workflow should have explicit states such as:

Pending
Running
Waiting
Completed
Failed
Cancelled

Monitor Workflow Age

A workflow that has been running for hours may be legitimate, but one that remains stuck for days should be visible to operators.

Advantages and Disadvantages

Advantages

Survives application restarts
Reduces repeated work after failures
Supports long-running workflows
Makes retries safer
Improves deployment resilience
Provides better operational visibility
Supports human intervention and recovery

Disadvantages

Adds state-management complexity
Requires durable storage
Requires careful idempotency design
Workflow versioning becomes important
Debugging can involve multiple components
Poorly designed persistence can create unnecessary storage overhead

Troubleshooting Long-Running Agent Failures

When an agent fails after running for a long time, check:

Identify the workflow ID.
Load the latest persisted state.
Determine the last completed step.
Check whether the current operation created an external side effect.
Verify whether that operation is idempotent.
Review retry history.
Check external API status.
Check database transactions and locks.
Verify workflow version compatibility.
Resume only after determining whether the failed step is safe to repeat.

Do not blindly restart the entire workflow.

The correct recovery point depends on the last durable checkpoint and the side effects already produced.

Summary of the Article

Long-running AI agents cannot safely depend on a single application process remaining alive until the task finishes. Processes restart, machines fail, deployments happen, networks disconnect, and external services become unavailable.

Durable execution addresses this problem by persisting the important state of the workflow so that another process can continue the task after a failure.

For C# applications, a durable agent architecture should maintain a stable workflow ID, persist meaningful checkpoints, handle retries carefully, and make external side effects idempotent wherever possible. Database operations, API calls, model execution, and workflow state should be treated as separate concerns rather than one long-running in-memory operation.

The most important design principle is simple: a long-running AI agent should be able to lose its process without losing the workflow itself.

AI agents are becoming more capable of handling tasks that take longer than a typical request-response cycle.

An agent might analyze a large codebase, process thousands of records, run a sequence of tools, wait for external systems, or perform a multi-step research workflow.

That creates a problem that does not appear in the same way with short-lived applications:

What happens if the process stops halfway through the task?

Consider an agent that has already completed seven of ten steps:

Start
  |
  v
Step 1 - Read requirements
  |
  v
Step 2 - Search data
  |
  v
Step 3 - Analyze results
  |
  v
Step 4 - Call API
  |
  v
Step 5 - Generate output
  |
  v
Step 6 - Validate
  |
  v
Step 7 - Update state
  |
  X  Process crashes
  |
  v
Step 8 - ?

Without durable execution, the application may lose the information required to continue.

With durable execution, the workflow can persist its state and resume from a known point.

What Is Durable Execution?

Durable execution means that the important state of a long-running workflow survives process failures.

Instead of keeping the entire workflow state in application memory:

Agent Process
     |
     +-- Workflow State
     |
     +-- Current Step
     |
     +-- Tool Results

the application persists important state externally:

Agent Process
     |
     v
Durable Workflow State
     |
     +-- Current Step
     +-- Completed Steps
     +-- Tool Results
     +-- Retry Information

If the process terminates, another worker can load that state and continue the workflow.

The key idea is that the workflow should not depend on one running process staying alive for the entire task.

Why Long-Running Agents Need It

A five-second agent and a five-hour agent have very different reliability requirements.

A short request might look like:

Request
  |
  v
Agent
  |
  v
Response

If the process fails, the user can often retry the request.

A long-running workflow is different:

Start
 |
 +-- 20 minutes of processing
 |
 +-- External API call
 |
 +-- 40 minutes of analysis
 |
 +-- Database updates
 |
 X-- Process failure

Restarting from the beginning may waste significant work.

It can also create duplicate side effects.

Durable Execution vs Checkpointing

These concepts are related but not identical.

Checkpointing usually means saving the current state at specific points.

For example:

Step 1
 |
 v
Checkpoint
 |
 v
Step 2
 |
 v
Checkpoint
 |
 v
Step 3
 |
 v
Checkpoint

If the process fails after Step 3 begins, the application can resume from the latest checkpoint.

Durable execution is broader. A durable workflow can maintain execution state, retries, timers, external events, and workflow progress as part of the execution model.

For an AI agent, that can be particularly useful because agent workflows often contain many different types of operations.

A Long-Running Agent in C#

A simple workflow might be represented like this:

public sealed class AgentWorkflowState
{
    public string WorkflowId { get; init; } = string.Empty;

    public int CurrentStep { get; set; }

    public List<string> CompletedSteps { get; init; } = [];

    public string? LastResult { get; set; }
}

The workflow can persist this state after meaningful operations.

public async Task SaveStateAsync(
    AgentWorkflowState state,
    CancellationToken cancellationToken)
{
    // Persist workflow state in durable storage.
    await repository.SaveAsync(
        state,
        cancellationToken);
}

The important part is not the particular storage implementation.

The important part is that the state exists outside the process.

What Should Be Persisted?

Do not automatically persist everything the agent produces.

Persist information required to safely continue the workflow.

Typical state may include:

State	Why It Matters
Workflow ID	Identifies the execution
Current step	Determines where execution should continue
Completed steps	Prevents unnecessary repetition
Tool results	Avoids repeating expensive operations
Retry count	Controls retry behavior
External operation IDs	Helps track side effects
User-approved decisions	Preserves human decisions
Error state	Allows recovery or intervention

Large model prompts or intermediate data may require a different storage strategy.

The workflow state should contain references when storing the complete payload would be unnecessarily expensive.

Do Not Store the Entire Agent in Memory

A fragile design looks like this:

var agentState = new AgentState();

await RunStep1Async(agentState);
await RunStep2Async(agentState);
await RunStep3Async(agentState);
await RunStep4Async(agentState);

If the process terminates between Step 3 and Step 4, the in-memory state disappears.

A more durable approach is:

var state =
    await repository.LoadAsync(
        workflowId,
        cancellationToken);

await RunStepAsync(state, cancellationToken);

await repository.SaveAsync(
    state,
    cancellationToken);

The application can reconstruct the workflow from persisted state.

Idempotency Is Critical

Durable execution introduces another important concept: idempotency.

Suppose an agent performs this operation:

Create customer

The application writes:

INSERT INTO customers (...)
VALUES (...);

The process crashes immediately after the database accepts the operation but before the workflow records that the step completed.

When the workflow resumes, it may execute the same operation again.

You could end up with:

First attempt  -> Customer created
Retry          -> Customer created again

The workflow needs a way to safely determine whether the operation has already happened.

One approach is an idempotency key:

Workflow ID: wf-1001
Step: CreateCustomer
Operation ID: wf-1001-create-customer

The database can enforce uniqueness:

CREATE UNIQUE INDEX ux_customer_operation
ON customer_operations(operation_id);

The exact design depends on the operation, but the principle is important:

A retry should not accidentally repeat an irreversible side effect.

Separate Work From Side Effects

AI agent workflows often mix computation and external actions.

For example:

Analyze Document
      |
      v
Generate Recommendation
      |
      v
Send Email
      |
      v
Update CRM

The first two operations may be relatively easy to retry.

The last two can create external side effects.

A durable architecture should track those operations separately.

Workflow
   |
   +-- Analysis
   |
   +-- Recommendation
   |
   +-- Email Operation ID
   |
   +-- CRM Operation ID

This makes recovery easier.

Handling External API Calls

Suppose an agent calls an external API:

var response =
    await httpClient.PostAsJsonAsync(
        "/api/tasks",
        request,
        cancellationToken);

The application needs to consider what happens if the network connection fails after the external system receives the request.

There are now two possibilities:

Request sent
   |
   +---- Response received
   |
   +---- Connection lost

A timeout does not necessarily mean that the external operation failed.

The workflow should therefore use an operation identifier where the external system supports it.

Operation ID
     |
     v
External API
     |
     v
Result

If the agent retries, the external system can recognize the same operation instead of creating a duplicate.

Retry Policies Need Boundaries

Retries are useful when failures are temporary.

For example:

Attempt 1
   |
   X Timeout
   |
Attempt 2
   |
   X Timeout
   |
Attempt 3
   |
   v
Success

But retrying everything indefinitely is dangerous.

A workflow should distinguish between:

Temporary network failures
Rate limits
Service unavailability
Invalid input
Authentication failures
Permanent business errors

A simple retry policy might look like:

for (var attempt = 1; attempt <= 3; attempt++)
{
    try
    {
        return await ExecuteAsync(
            cancellationToken);
    }
    catch (HttpRequestException)
        when (attempt < 3)
    {
        await Task.Delay(
            TimeSpan.FromSeconds(attempt * 2),
            cancellationToken);
    }
}

throw new InvalidOperationException(
    "Operation failed after retries.");

Production systems often use a resilience library or workflow platform rather than implementing all retry behavior manually.

What Happens During a Deployment?

Long-running agents create an operational challenge during deployments.

Suppose:

Version 1
   |
   v
Agent running for 3 hours

A deployment starts:

Version 2 deployed

If workflow state exists only in memory, the running task may disappear.

With durable state:

Version 1
   |
   v
Persisted Workflow
   |
   v
Version 2
   |
   v
Resume Workflow

This makes deployments less disruptive.

However, workflow state and code changes must remain compatible.

If Version 2 changes the structure or meaning of persisted state, migration or versioning may be necessary.

Workflow Versioning

Consider this state:

{
  "workflowId": "wf-1001",
  "step": 4,
  "status": "Running"
}

A newer application version might expect a different workflow structure.

One way to manage this is to include a workflow version:

{
  "workflowId": "wf-1001",
  "workflowVersion": 2,
  "step": 4,
  "status": "Running"
}

The application can then determine how to interpret the stored state.

This becomes increasingly important as workflows remain active for hours or days.

Durable Execution Does Not Mean Durable Everything

It is tempting to persist every intermediate model response, tool call, and application object.

That can create unnecessary storage and operational complexity.

Instead, define the minimum state required for recovery.

For example:

Persist:
- Workflow state
- Step status
- Important tool results
- External operation IDs
- User decisions

Do not automatically persist:
- Temporary objects
- Reconstructable data
- Unnecessary duplicate model output

The correct boundary depends on the workflow.

Common Mistakes

Keeping Workflow State Only in Memory

A process restart can destroy the workflow's progress.

Retrying Non-Idempotent Operations

A retry can create duplicate records, payments, messages, or other side effects.

Treating Every Error as Temporary

Some errors should fail immediately instead of being retried.

Saving State Too Infrequently

If checkpoints are too far apart, a failure can force the workflow to repeat substantial work.

Saving Everything

Excessive persistence can increase storage costs and make state management harder.

Ignoring Deployment Changes

Long-running workflows may outlive the application version that created them.

Forgetting Cancellation

Users and operators should have a way to stop long-running workflows safely.

Best Practices for Production

Give Every Workflow a Stable ID

A unique workflow identifier makes state, logs, retries, and external operations easier to correlate.

Persist Meaningful Checkpoints

Checkpoint after operations where losing progress would be expensive.

Make Side Effects Idempotent

Use operation IDs, unique constraints, or equivalent mechanisms where appropriate.

Keep Transactions Short

Do not hold database transactions open while waiting for model inference or external services.

Separate Durable State From Large Payloads

Store large artifacts separately and keep references in the workflow state when appropriate.

Record Failure Information

Persist enough information to understand why a workflow stopped.

Support Resume and Cancellation

A durable workflow should have explicit states such as:

Pending
Running
Waiting
Completed
Failed
Cancelled

Monitor Workflow Age

A workflow that has been running for hours may be legitimate, but one that remains stuck for days should be visible to operators.

Advantages and Disadvantages

Advantages

Survives application restarts
Reduces repeated work after failures
Supports long-running workflows
Makes retries safer
Improves deployment resilience
Provides better operational visibility
Supports human intervention and recovery

Disadvantages

Adds state-management complexity
Requires durable storage
Requires careful idempotency design
Workflow versioning becomes important
Debugging can involve multiple components
Poorly designed persistence can create unnecessary storage overhead

Troubleshooting Long-Running Agent Failures

When an agent fails after running for a long time, check:

Identify the workflow ID.
Load the latest persisted state.
Determine the last completed step.
Check whether the current operation created an external side effect.
Verify whether that operation is idempotent.
Review retry history.
Check external API status.
Check database transactions and locks.
Verify workflow version compatibility.
Resume only after determining whether the failed step is safe to repeat.

Do not blindly restart the entire workflow.

The correct recovery point depends on the last durable checkpoint and the side effects already produced.

Summary of the Article

Durable execution addresses this problem by persisting the important state of the workflow so that another process can continue the task after a failure.

The most important design principle is simple: a long-running AI agent should be able to lose its process without losing the workflow itself.