A long-running AI agent may spend minutes or hours working through a task.
During that time, it may call APIs, query databases, execute tools, process files, wait for external events, and communicate with other services.
Now imagine the network connection disappears halfway through the workflow.
AI Agent
|
+--> Database
|
+--> API
|
+--> Tool
|
X
Connection Lost
The important question is not simply how to reconnect.
The application needs to determine what the agent was doing when the connection failed, whether the operation completed, and whether it is safe to continue or retry it.
A reliable agent should treat connection loss as an expected failure condition rather than an exceptional event that automatically terminates the entire workflow.
Why Connection Loss Is Different for Long-Running Agents
For a short API request, a network failure may simply result in:
Request
|
X
Connection Lost
|
v
Retry
Long-running agents have more state.
Consider:
Start
|
v
Read Database
|
v
Call AI Model
|
v
Update Record
|
v
Call External API
|
X
Connection Lost
At this point, the application needs to know:
Which steps completed?
Was the external API request sent?
Did the database update commit?
Was the model response already generated?
Can the failed operation safely be retried?
Should the workflow resume or restart?
Without persisted state, answering these questions becomes difficult.
The Agent Should Have a Durable Workflow State
A useful architecture separates the agent process from its workflow state.
Durable State
|
v
+--------------+
| Agent State |
+------+-------+
^
|
+------+-------+
| Agent Worker |
+--------------+
|
+------------+------------+
| | |
v v v
Database API Tools
If the worker loses its connection, another worker can load the state and continue.
For example:
public sealed class AgentWorkflowState
{
public string WorkflowId { get; init; } = string.Empty;
public string Status { get; set; } = "Running";
public int CurrentStep { get; set; }
public string? LastCompletedOperation { get; set; }
}
The exact state model depends on the workflow, but the principle is consistent: important progress should exist outside the process memory.
Connection Loss Does Not Always Mean Operation Failure
This is one of the most important concepts in distributed systems.
Suppose the agent sends:
POST /payments
The external service receives and processes the request.
But before the response reaches the agent:
External Service
|
v
Payment Processed
|
X
Network Connection Lost
|
v
Agent
From the agent's perspective, the request failed.
From the external service's perspective, it succeeded.
If the agent simply retries, the operation could happen twice.
The workflow therefore needs to distinguish between:
Operation failed
and:
Operation result unknown
The second case requires much more careful recovery.
Use Idempotency for External Operations
Where an external service supports idempotency, give each logical operation a stable identifier.
For example:
Workflow:
wf-10042
Operation:
send-report
Idempotency Key:
wf-10042-send-report
The agent sends the same identifier when retrying.
Conceptually:
Agent
|
| Operation ID: wf-10042-send-report
v
External Service
|
+--> First request processed
|
+--> Retry recognized as same operation
The external service can then return the existing result instead of performing the operation again, assuming it supports this behavior.
C# Example of an Idempotent Operation
A C# service might represent an operation like this:
public sealed record AgentOperation(
string WorkflowId,
string OperationId);
The application can generate the identifier once:
var operation = new AgentOperation(
workflowId,
$"{workflowId}-send-report");
The important part is that the identifier should remain stable across retries.
Do not generate a new random operation ID every time the request is retried.
Detecting Transient Failures
Not every connection error requires the same response.
Common transient failures include:
Other failures are usually not fixed by retrying:
Invalid authentication
Invalid request
Missing resource
Permission failure
Invalid business data
A retry policy should therefore classify errors instead of retrying everything.
For example:
for (var attempt = 1; attempt <= 3; attempt++)
{
try
{
return await client.SendAsync(
request,
cancellationToken);
}
catch (HttpRequestException)
when (attempt < 3)
{
await Task.Delay(
TimeSpan.FromSeconds(attempt * 2),
cancellationToken);
}
}
throw new InvalidOperationException(
"The operation could not be completed.");
Production applications often use a dedicated resilience library or workflow framework to handle retry policies more consistently.
Use Exponential Backoff
Retrying immediately can make an overloaded service even worse.
For example:
Failure
|
+--> Retry immediately
|
X
Failure
|
+--> Retry immediately
|
X
Failure
A better pattern introduces increasing delays:
Attempt 1
|
X
Wait 1 second
|
Attempt 2
|
X
Wait 2 seconds
|
Attempt 3
|
X
Wait 4 seconds
|
Attempt 4
The exact values should depend on the service and workload.
Jitter can also be added so that many agents do not retry simultaneously.
Do Not Retry Forever
A long-running agent should not become an infinite retry loop.
A workflow should define limits such as:
Maximum attempts
Maximum elapsed time
Maximum retry delay
After those limits are reached, the workflow can move into a recoverable state:
Running
|
X
Repeated failures
|
v
WaitingForRetry
|
v
Failed
The workflow can then be reviewed or resumed later.
What Happens to the Agent When the Connection Returns?
A connection being restored does not necessarily mean the workflow can simply continue from the current line of code.
For example:
Step 4
|
v
External API Call
|
X
Connection Lost
|
v
Application Reconnects
Before continuing, the application should determine whether Step 4 completed.
If the external system supports status lookup:
Check Operation Status
|
+----+----+
| |
Found Not Found
| |
Continue Retry
This is much safer than automatically executing the operation again.
Database Operations Need Special Handling
Database transactions provide strong guarantees, but the application still needs to handle connection failures correctly.
Consider:
await using var transaction =
await connection.BeginTransactionAsync(
cancellationToken);
await UpdateAgentStateAsync(
connection,
transaction,
cancellationToken);
await SaveResultAsync(
connection,
transaction,
cancellationToken);
await transaction.CommitAsync(
cancellationToken);
If the connection fails during the transaction, the application should not assume that every operation completed.
The transaction boundary determines which operations were committed.
A new connection can then be used to inspect the durable workflow state.
Keep Transactions Short
Never keep a database transaction open while the agent waits for a model response.
Avoid:
BEGIN TRANSACTION
|
v
Call AI Model
|
v
Wait several minutes
|
v
Call API
|
v
COMMIT
Prefer:
Read Database
|
v
Close Transaction
|
v
Call AI Model
|
v
Call External Service
|
v
Short Database Transaction
|
v
Commit
Short transactions reduce lock duration and make recovery easier.
Store the Last Known Workflow State
Suppose the workflow has these steps:
1. Load Data
2. Analyze Data
3. Generate Recommendation
4. Save Recommendation
5. Notify User
The application can persist progress:
{
"workflowId": "wf-5001",
"status": "Running",
"completedStep": 3
}
If the connection disappears after Step 3, another worker can load the state.
Load Workflow
|
v
Completed Step = 3
|
v
Continue at Step 4
This is much safer than restarting from Step 1.
Handle Connection Loss With Workflow States
A useful state model might look like:
Pending
|
v
Running
|
+----> Waiting
|
+----> ConnectionLost
|
+----> Completed
|
+----> Failed
|
+----> Cancelled
For example:
public enum AgentStatus
{
Pending,
Running,
Waiting,
ConnectionLost,
Completed,
Failed,
Cancelled
}
The exact states should reflect the application's recovery requirements.
A dedicated ConnectionLost state can be useful when the system wants to distinguish infrastructure failures from business failures.
Reconnection Should Not Be the Agent's Responsibility
It is tempting to write:
while (true)
{
try
{
await RunAgentAsync();
break;
}
catch
{
await ReconnectAsync();
}
}
This approach can become difficult to control.
The workflow engine or application infrastructure should generally manage:
Retry limits
Backoff
Connection lifecycle
Workflow state
Cancellation
Recovery
The agent itself should focus on the task it is responsible for.
Handling Long Network Outages
Suppose the database or external API remains unavailable for 30 minutes.
The agent should not continuously retry every few seconds.
Instead:
Connection Failure
|
v
Persist Workflow State
|
v
Schedule Retry
|
v
Worker Released
|
v
Service Available
|
v
Workflow Resumes
This prevents a large number of agents from consuming application resources while waiting for the same unavailable service.
Durable Waiting Is Useful
Long-running workflows often need to wait.
For example:
Agent
|
v
Submit Approval
|
v
Wait for Human
|
v
Continue
The worker does not need to remain active during the entire waiting period.
The workflow state can record:
Status:
Waiting
Reason:
HumanApproval
Workflow:
wf-5001
When the event occurs, the workflow resumes.
This is more efficient than keeping a process alive just to wait.
What If the Agent Loses Its Model Connection?
The same principles apply to AI model calls.
Suppose:
Agent
|
v
AI Model
|
X
Connection Lost
The application should determine:
Was the request accepted?
Was a response generated?
Can the request be safely retried?
Does the provider expose an operation identifier?
Is the model request deterministic enough for the application?
Should the workflow resume from an earlier checkpoint?
If the model operation cannot be safely identified, the application should treat the result as uncertain rather than automatically assuming failure.
Common Mistakes
Restarting the Entire Workflow
A connection failure near the end of a two-hour workflow should not necessarily send the agent back to the beginning.
Retrying Every Failed Request
Some failures are permanent, and some operations are not safe to repeat.
Generating a New ID for Every Retry
This can turn one logical operation into multiple independent operations.
Holding Connections Open During AI Processing
Database and network connections should not remain occupied while waiting for model inference.
Ignoring Unknown Outcomes
A timeout does not always mean that the external operation failed.
Keeping Workers Alive During Long Waits
If the workflow can wait durably, release the worker and resume later.
Relying Only on In-Memory State
A process restart can destroy recovery information.
Best Practices for Production
Persist Before Risky Operations
Store enough state to identify what the workflow is about to do.
Persist After Successful Operations
Record successful completion so that recovery does not repeat completed work.
Use Stable Operation IDs
Use workflow and operation identifiers for external side effects.
Separate Retryable and Non-Retryable Errors
Do not apply the same retry policy to every failure.
Use Backoff and Jitter
Avoid synchronized retry storms when many agents fail simultaneously.
Put Limits on Retries
Define maximum attempts and maximum elapsed time.
Release Resources While Waiting
Do not keep database connections or worker threads occupied unnecessarily.
Make Recovery Observable
Log:
This makes production troubleshooting much easier.
Advantages and Disadvantages
Advantages
Long-running agents can survive temporary network failures
Completed work does not need to be repeated unnecessarily
Durable state supports recovery across process restarts
Retry behavior becomes predictable
External side effects can be protected with idempotency
Workers can be released during long waits
Disadvantages
Recovery logic adds architectural complexity
Idempotency can be difficult for some external operations
Durable state requires additional storage
Incorrect retry policies can still cause duplicate operations
Workflow debugging requires good observability
Troubleshooting Connection Loss
When a long-running agent loses its connection:
Identify the workflow ID.
Find the last durable checkpoint.
Determine the operation that was in progress.
Check whether the operation may have completed.
Query the external system when status lookup is available.
Check database transaction state.
Determine whether the operation is idempotent.
Apply the retry policy only if the operation is safe to retry.
Persist the recovery decision.
Resume the workflow from the correct step.
The key is to avoid treating every connection failure as a simple restart.
Summary
A long-running AI agent should assume that network connections will eventually fail. The important part is not preventing every failure but designing the workflow so that a temporary connection problem does not destroy the entire task.
The agent should persist meaningful workflow state, use stable operation identifiers, distinguish retryable failures from permanent errors, and use idempotency when external operations can produce side effects. Database transactions should remain short, and workers should not hold connections open while waiting for AI inference or external events.
When a connection is lost, the safest recovery process is to identify the last durable state, determine whether the interrupted operation actually completed, and then either resume, retry, wait, or fail based on that information.
The core principle is simple: connection loss should interrupt execution, not erase the workflow.
Join the conversation! Your thoughts help the community grow.