A long-running AI agent may spend minutes or hours working through a task.

During that time, it may call APIs, query databases, execute tools, process files, wait for external events, and communicate with other services.

Now imagine the network connection disappears halfway through the workflow.

AI Agent
   |
   +--> Database
   |
   +--> API
   |
   +--> Tool
   |
   X
Connection Lost

The important question is not simply how to reconnect.

The application needs to determine what the agent was doing when the connection failed, whether the operation completed, and whether it is safe to continue or retry it.

A reliable agent should treat connection loss as an expected failure condition rather than an exceptional event that automatically terminates the entire workflow.

Why Connection Loss Is Different for Long-Running Agents

For a short API request, a network failure may simply result in:

Request
   |
   X
Connection Lost
   |
   v
Retry

Long-running agents have more state.

Consider:

Start
 |
 v
Read Database
 |
 v
Call AI Model
 |
 v
Update Record
 |
 v
Call External API
 |
 X
Connection Lost

At this point, the application needs to know:

Without persisted state, answering these questions becomes difficult.

The Agent Should Have a Durable Workflow State

A useful architecture separates the agent process from its workflow state.

                Durable State
                     |
                     v
              +--------------+
              | Agent State  |
              +------+-------+
                     ^
                     |
              +------+-------+
              | Agent Worker |
              +--------------+
                     |
        +------------+------------+
        |            |            |
        v            v            v
     Database       API          Tools

If the worker loses its connection, another worker can load the state and continue.

For example:

public sealed class AgentWorkflowState
{
    public string WorkflowId { get; init; } = string.Empty;

    public string Status { get; set; } = "Running";

    public int CurrentStep { get; set; }

    public string? LastCompletedOperation { get; set; }
}

The exact state model depends on the workflow, but the principle is consistent: important progress should exist outside the process memory.

Connection Loss Does Not Always Mean Operation Failure

This is one of the most important concepts in distributed systems.

Suppose the agent sends:

POST /payments

The external service receives and processes the request.

But before the response reaches the agent:

External Service
      |
      v
Payment Processed
      |
      X
Network Connection Lost
      |
      v
Agent

From the agent's perspective, the request failed.

From the external service's perspective, it succeeded.

If the agent simply retries, the operation could happen twice.

The workflow therefore needs to distinguish between:

Operation failed

and:

Operation result unknown

The second case requires much more careful recovery.

Use Idempotency for External Operations

Where an external service supports idempotency, give each logical operation a stable identifier.

For example:

Workflow:
wf-10042

Operation:
send-report

Idempotency Key:
wf-10042-send-report

The agent sends the same identifier when retrying.

Conceptually:

Agent
  |
  | Operation ID: wf-10042-send-report
  v
External Service
  |
  +--> First request processed
  |
  +--> Retry recognized as same operation

The external service can then return the existing result instead of performing the operation again, assuming it supports this behavior.

C# Example of an Idempotent Operation

A C# service might represent an operation like this:

public sealed record AgentOperation(
    string WorkflowId,
    string OperationId);

The application can generate the identifier once:

var operation = new AgentOperation(
    workflowId,
    $"{workflowId}-send-report");

The important part is that the identifier should remain stable across retries.

Do not generate a new random operation ID every time the request is retried.

Detecting Transient Failures

Not every connection error requires the same response.

Common transient failures include:

Other failures are usually not fixed by retrying:

A retry policy should therefore classify errors instead of retrying everything.

For example:

for (var attempt = 1; attempt <= 3; attempt++)
{
    try
    {
        return await client.SendAsync(
            request,
            cancellationToken);
    }
    catch (HttpRequestException)
        when (attempt < 3)
    {
        await Task.Delay(
            TimeSpan.FromSeconds(attempt * 2),
            cancellationToken);
    }
}

throw new InvalidOperationException(
    "The operation could not be completed.");

Production applications often use a dedicated resilience library or workflow framework to handle retry policies more consistently.

Use Exponential Backoff

Retrying immediately can make an overloaded service even worse.

For example:

Failure
  |
  +--> Retry immediately
  |
  X
Failure
  |
  +--> Retry immediately
  |
  X
Failure

A better pattern introduces increasing delays:

Attempt 1
   |
   X
Wait 1 second
   |
Attempt 2
   |
   X
Wait 2 seconds
   |
Attempt 3
   |
   X
Wait 4 seconds
   |
Attempt 4

The exact values should depend on the service and workload.

Jitter can also be added so that many agents do not retry simultaneously.

Do Not Retry Forever

A long-running agent should not become an infinite retry loop.

A workflow should define limits such as:

Maximum attempts
Maximum elapsed time
Maximum retry delay

After those limits are reached, the workflow can move into a recoverable state:

Running
   |
   X
Repeated failures
   |
   v
WaitingForRetry
   |
   v
Failed

The workflow can then be reviewed or resumed later.

What Happens to the Agent When the Connection Returns?

A connection being restored does not necessarily mean the workflow can simply continue from the current line of code.

For example:

Step 4
 |
 v
External API Call
 |
 X
Connection Lost
 |
 v
Application Reconnects

Before continuing, the application should determine whether Step 4 completed.

If the external system supports status lookup:

Check Operation Status
        |
   +----+----+
   |         |
Found       Not Found
   |         |
Continue    Retry

This is much safer than automatically executing the operation again.

Database Operations Need Special Handling

Database transactions provide strong guarantees, but the application still needs to handle connection failures correctly.

Consider:

await using var transaction =
    await connection.BeginTransactionAsync(
        cancellationToken);

await UpdateAgentStateAsync(
    connection,
    transaction,
    cancellationToken);

await SaveResultAsync(
    connection,
    transaction,
    cancellationToken);

await transaction.CommitAsync(
    cancellationToken);

If the connection fails during the transaction, the application should not assume that every operation completed.

The transaction boundary determines which operations were committed.

A new connection can then be used to inspect the durable workflow state.

Keep Transactions Short

Never keep a database transaction open while the agent waits for a model response.

Avoid:

BEGIN TRANSACTION
      |
      v
Call AI Model
      |
      v
Wait several minutes
      |
      v
Call API
      |
      v
COMMIT

Prefer:

Read Database
     |
     v
Close Transaction
     |
     v
Call AI Model
     |
     v
Call External Service
     |
     v
Short Database Transaction
     |
     v
Commit

Short transactions reduce lock duration and make recovery easier.

Store the Last Known Workflow State

Suppose the workflow has these steps:

1. Load Data
2. Analyze Data
3. Generate Recommendation
4. Save Recommendation
5. Notify User

The application can persist progress:

{
  "workflowId": "wf-5001",
  "status": "Running",
  "completedStep": 3
}

If the connection disappears after Step 3, another worker can load the state.

Load Workflow
     |
     v
Completed Step = 3
     |
     v
Continue at Step 4

This is much safer than restarting from Step 1.

Handle Connection Loss With Workflow States

A useful state model might look like:

Pending
   |
   v
Running
   |
   +----> Waiting
   |
   +----> ConnectionLost
   |
   +----> Completed
   |
   +----> Failed
   |
   +----> Cancelled

For example:

public enum AgentStatus
{
    Pending,
    Running,
    Waiting,
    ConnectionLost,
    Completed,
    Failed,
    Cancelled
}

The exact states should reflect the application's recovery requirements.

A dedicated ConnectionLost state can be useful when the system wants to distinguish infrastructure failures from business failures.

Reconnection Should Not Be the Agent's Responsibility

It is tempting to write:

while (true)
{
    try
    {
        await RunAgentAsync();
        break;
    }
    catch
    {
        await ReconnectAsync();
    }
}

This approach can become difficult to control.

The workflow engine or application infrastructure should generally manage:

The agent itself should focus on the task it is responsible for.

Handling Long Network Outages

Suppose the database or external API remains unavailable for 30 minutes.

The agent should not continuously retry every few seconds.

Instead:

Connection Failure
       |
       v
Persist Workflow State
       |
       v
Schedule Retry
       |
       v
Worker Released
       |
       v
Service Available
       |
       v
Workflow Resumes

This prevents a large number of agents from consuming application resources while waiting for the same unavailable service.

Durable Waiting Is Useful

Long-running workflows often need to wait.

For example:

Agent
  |
  v
Submit Approval
  |
  v
Wait for Human
  |
  v
Continue

The worker does not need to remain active during the entire waiting period.

The workflow state can record:

Status:
Waiting

Reason:
HumanApproval

Workflow:
wf-5001

When the event occurs, the workflow resumes.

This is more efficient than keeping a process alive just to wait.

What If the Agent Loses Its Model Connection?

The same principles apply to AI model calls.

Suppose:

Agent
 |
 v
AI Model
 |
 X
Connection Lost

The application should determine:

If the model operation cannot be safely identified, the application should treat the result as uncertain rather than automatically assuming failure.

Common Mistakes

Restarting the Entire Workflow

A connection failure near the end of a two-hour workflow should not necessarily send the agent back to the beginning.

Retrying Every Failed Request

Some failures are permanent, and some operations are not safe to repeat.

Generating a New ID for Every Retry

This can turn one logical operation into multiple independent operations.

Holding Connections Open During AI Processing

Database and network connections should not remain occupied while waiting for model inference.

Ignoring Unknown Outcomes

A timeout does not always mean that the external operation failed.

Keeping Workers Alive During Long Waits

If the workflow can wait durably, release the worker and resume later.

Relying Only on In-Memory State

A process restart can destroy recovery information.

Best Practices for Production

Persist Before Risky Operations

Store enough state to identify what the workflow is about to do.

Persist After Successful Operations

Record successful completion so that recovery does not repeat completed work.

Use Stable Operation IDs

Use workflow and operation identifiers for external side effects.

Separate Retryable and Non-Retryable Errors

Do not apply the same retry policy to every failure.

Use Backoff and Jitter

Avoid synchronized retry storms when many agents fail simultaneously.

Put Limits on Retries

Define maximum attempts and maximum elapsed time.

Release Resources While Waiting

Do not keep database connections or worker threads occupied unnecessarily.

Make Recovery Observable

Log:

This makes production troubleshooting much easier.

Advantages and Disadvantages

Advantages

Disadvantages

Troubleshooting Connection Loss

When a long-running agent loses its connection:

  1. Identify the workflow ID.

  2. Find the last durable checkpoint.

  3. Determine the operation that was in progress.

  4. Check whether the operation may have completed.

  5. Query the external system when status lookup is available.

  6. Check database transaction state.

  7. Determine whether the operation is idempotent.

  8. Apply the retry policy only if the operation is safe to retry.

  9. Persist the recovery decision.

  10. Resume the workflow from the correct step.

The key is to avoid treating every connection failure as a simple restart.

Summary

A long-running AI agent should assume that network connections will eventually fail. The important part is not preventing every failure but designing the workflow so that a temporary connection problem does not destroy the entire task.

The agent should persist meaningful workflow state, use stable operation identifiers, distinguish retryable failures from permanent errors, and use idempotency when external operations can produce side effects. Database transactions should remain short, and workers should not hold connections open while waiting for AI inference or external events.

When a connection is lost, the safest recovery process is to identify the last durable state, determine whether the interrupted operation actually completed, and then either resume, retry, wait, or fail based on that information.

The core principle is simple: connection loss should interrupt execution, not erase the workflow.