AI agent workflows are often demonstrated as a simple sequence:
Request
|
v
Agent
|
v
Tool
|
v
Response
Production systems are rarely that simple.
A real agent workflow may contain:
User Request
|
v
Planner
|
+------> Database
|
+------> API
|
+------> Search
|
+------> Document Processing
|
v
Decision
|
v
Final Response
Every external dependency introduces another failure point.
A database can time out. An API can return an error. A tool can produce invalid output. A model call can exceed its deadline. A workflow can be interrupted after successfully completing one step but before recording the next state.
For declarative agent workflows, this creates an important engineering problem:
How do you recover from failures without duplicating side effects, losing workflow state, or allowing an agent to make unsafe recovery decisions?
The answer is not simply "add retries."
Production-grade recovery requires a combination of:
Durable state
Retry policies
Timeouts
Idempotency
Compensation
Checkpoints
Failure classification
Human intervention
Recovery-aware workflow design
What Is a Declarative Agent Workflow?
In an imperative workflow, application code explicitly controls execution:
var result = await StepOneAsync();
var next = await StepTwoAsync(result);
await StepThreeAsync(next);
A declarative workflow describes what should happen and allows the workflow runtime to manage execution.
Conceptually:
Workflow
|
+--> Step A
|
+--> Step B
|
+--> Step C
The runtime can manage:
Execution
State
Scheduling
Retries
Pausing
Resuming
Failures
This separation becomes particularly valuable when workflows run for longer periods or depend on multiple external systems.
Why Agent Workflows Need Failure Recovery
An agent workflow can fail at several layers:
Workflow
|
+---------------+---------------+
| | |
v v v
Model Tool Data
| | |
Timeout API Error Database
| | |
+---------------+---------------+
|
v
Failure
A failure is not always equivalent to "stop everything."
For example:
Search Tool Failed
|
v
Retry
|
v
Success
may be appropriate.
But:
Payment Failed
|
v
Retry
|
v
Retry
|
v
Retry
could be dangerous if the operation creates duplicate charges.
The recovery strategy must depend on the type of operation.
Failure Taxonomy
Start by classifying failures.
| Failure Type | Example | Typical Strategy |
|---|---|---|
| Transient | Timeout | Retry |
| Rate limit | Too many requests | Backoff |
| Authentication | Expired credential | Refresh or stop |
| Authorization | Permission denied | Stop |
| Validation | Invalid input | Correct or request input |
| Dependency | Service unavailable | Retry or fallback |
| Business | Operation rejected | Compensate or stop |
| Model | Invalid structured output | Retry with validation |
| Infrastructure | Worker crash | Resume from checkpoint |
| Unknown | Unexpected exception | Escalate |
This classification should happen before designing retry logic.
The First Principle: Not Every Failure Should Be Retried
A common implementation is:
try
Execute
catch
Retry
This is too broad.
A better approach is:
Failure
|
v
Classify
|
+--> Transient ------> Retry
|
+--> Rate Limited ---> Backoff
|
+--> Invalid -------> Correct
|
+--> Unauthorized --> Stop
|
+--> Business ------> Compensate
|
+--> Unknown ------> Escalate
Retries should be based on failure semantics.
Retry Policies
A basic retry policy can be represented as:
public sealed class RetryPolicy
{
public int MaxAttempts { get; init; }
public TimeSpan InitialDelay { get; init; }
public double BackoffMultiplier { get; init; }
public TimeSpan MaxDelay { get; init; }
}
For example:
var policy = new RetryPolicy
{
MaxAttempts = 4,
InitialDelay = TimeSpan.FromSeconds(1),
BackoffMultiplier = 2,
MaxDelay = TimeSpan.FromSeconds(30)
};
The resulting delays could be approximately:
Attempt 1
|
1 second
|
Attempt 2
|
2 seconds
|
Attempt 3
|
4 seconds
|
Attempt 4
The exact values should be chosen based on the dependency.
Exponential Backoff
Without backoff, multiple workflow workers may retry simultaneously:
Service Fails
|
+--> Worker A retries
+--> Worker B retries
+--> Worker C retries
+--> Worker D retries
This can create a retry storm.
With exponential backoff:
Worker A -> 1s
Worker B -> 2s
Worker C -> 4s
Worker D -> 8s
Add jitter when many workers may retry the same dependency.
Conceptually:
Delay = BaseDelay + RandomJitter
This reduces synchronized retries.
Retry Only Idempotent Operations Automatically
Consider:
GetCustomer
Retrying is generally safer than:
CreatePayment
because the second operation may produce another side effect.
The workflow should classify operations by idempotency.
For example:
| Operation | Automatic Retry |
|---|---|
| Read customer | Usually safe |
| Search documents | Usually safe |
| Get order | Usually safe |
| Create record | Requires idempotency |
| Send email | Requires care |
| Issue refund | Requires strong idempotency |
| Delete resource | Requires careful policy |
The exact behavior depends on the underlying API.
Idempotency Keys
For side-effecting operations, use an idempotency key.
For example:
var idempotencyKey =
$"workflow:{workflowId}:refund:{stepId}";
The same operation can then be identified across retries.
Conceptually:
Workflow
|
v
Refund Request
|
+--> Attempt 1
|
+--> Timeout
|
+--> Attempt 2
|
v
Same Key
|
v
Existing Result
The downstream service can recognize that the operation has already been processed.
Why Timeouts Are Required
A retry without a timeout can make a workflow wait indefinitely.
For example:
Agent
|
v
External API
|
+---- waiting...
|
+---- waiting...
|
+---- waiting...
Define an explicit timeout:
using var timeout =
new CancellationTokenSource(
TimeSpan.FromSeconds(15));
var result = await client
.GetAsync(
requestUri,
timeout.Token);
The workflow can then classify the timeout and decide whether to retry.
Workflow Timeout vs Activity Timeout
These are different.
Activity Timeout
Controls one operation:
Tool Call
|
+-- Maximum 15 seconds
Workflow Timeout
Controls the complete operation:
Entire Workflow
|
+-- Maximum 10 minutes
You may need both.
For example:
Workflow
|
+--> Tool A: 15 sec
|
+--> Tool B: 20 sec
|
+--> Tool C: 30 sec
The total workflow still needs a broader execution limit.
Durable State Is the Foundation
A workflow that can resume after failure must know where it was.
Instead of:
Memory
|
v
Current Step
use durable state:
Workflow
|
v
Persistent State
|
+--> Current Step
+--> Inputs
+--> Outputs
+--> Status
+--> Retry Count
+--> Correlation ID
A simplified model might be:
public sealed class WorkflowState
{
public required string WorkflowId { get; init; }
public required string CurrentStep { get; set; }
public string Status { get; set; } = "Running";
public int RetryCount { get; set; }
public DateTimeOffset UpdatedAt { get; set; }
}
The actual state model will depend on the workflow runtime.
Checkpoints
A checkpoint records successful progress.
Consider:
Step A
|
v
Checkpoint
|
v
Step B
|
v
Checkpoint
|
v
Step C
If Step C fails, the workflow can resume from the appropriate point instead of starting from Step A.
Without checkpoints:
Failure at Step C
|
v
Restart Step A
With checkpoints:
Failure at Step C
|
v
Resume Step C
This is particularly valuable for long-running workflows.
Checkpoint Too Early and Too Late
Checkpoint placement matters.
Suppose:
Create Order
|
Checkpoint
|
Send Confirmation
If the checkpoint is written before the order is actually committed, recovery may incorrectly assume the operation completed.
On the other hand, checkpointing after every tiny operation can add overhead.
A useful checkpoint represents a meaningful, durable state transition.
Workflow State Machine
Recovery becomes easier when workflow states are explicit.
For example:
Created
|
v
Planning
|
v
Executing
|
+----> Waiting
|
v
Completed
Failure states can be represented separately:
Executing
|
v
Retrying
|
+----> Executing
|
v
Failed
|
v
Compensating
|
v
Compensated
This prevents the workflow from relying on ambiguous flags.
Example State Model
public enum WorkflowStatus
{
Created,
Running,
Waiting,
Retrying,
Completed,
Failed,
Compensating,
Cancelled
}
A workflow can then explicitly transition:
Running -> Retrying
Retrying -> Running
Running -> Completed
Running -> Failed
Failed -> Compensating
Not every state transition should be allowed.
Validate State Transitions
Avoid code that permits:
Completed -> Running
unless the workflow explicitly supports reopening.
A state transition method can enforce valid transitions:
public void MoveTo(WorkflowStatus next)
{
if (!IsValidTransition(Status, next))
{
throw new InvalidOperationException(
$"Invalid transition: {Status} -> {next}");
}
Status = next;
}
This makes recovery logic more predictable.
Compensation Instead of Rollback
Distributed workflows cannot always perform a traditional database rollback.
Consider:
Create Customer
|
v
Create Subscription
|
v
Send Notification
The database transaction may not span all three operations.
If notification fails, you cannot necessarily roll back the external subscription transactionally.
Instead, use compensation:
Notification Failed
|
v
Cancel Subscription
This is a saga-style approach.
Example Compensation Flow
Step 1
Create Customer
|
v
Step 2
Create Subscription
|
v
Step 3
Send Notification
|
X
Failure
|
v
Compensate Step 2
|
v
Cancel Subscription
The compensation operation should itself be designed to be idempotent.
Compensation Is Not Perfect Rollback
This distinction matters.
A database rollback can restore transactional state.
A compensation workflow performs another business operation to reduce or reverse the effect.
For example:
Charge Customer
cannot always be "rolled back" in the database.
A refund is a new business operation:
Charge
|
v
Refund
The system should model this explicitly.
Agent Workflows Add Another Failure Dimension
An agent can choose the wrong tool.
For example:
User Request
|
v
Agent
|
v
Wrong Tool
This is different from:
Correct Tool
|
v
Temporary Network Failure
Retrying the wrong tool does not solve the problem.
Agent workflows therefore need to distinguish:
Execution failure
from:
Decision failure
Validate Agent Tool Selection
Before executing a tool call, validate:
Tool exists
Tool is approved
Agent has permission
Input matches schema
Tenant context is valid
Risk policy allows execution
Conceptually:
Agent Decision
|
v
Policy Validation
|
+---- Denied
|
v
Tool Execution
The agent should not bypass this layer.
Validate Structured Model Output
Suppose the workflow expects:
{
"tool": "invoice.search",
"customerId": "123"
}
The model might produce:
{
"tool": "invoice.search",
"customerId": 123,
"includeAllTenants": true
}
Schema validation should reject unexpected or invalid fields.
A strongly typed model helps:
public sealed class ToolRequest
{
public required string Tool { get; init; }
public required string CustomerId { get; init; }
}
Then validate before execution.
Recovery From Invalid Model Output
An invalid model response should not necessarily terminate the workflow immediately.
A possible strategy is:
Invalid Output
|
v
Validation Error
|
v
Structured Retry
|
v
Model Generates Correct Format
But limit the number of attempts.
For example:
Attempt 1 -> Invalid
Attempt 2 -> Invalid
Attempt 3 -> Invalid
|
v
Failed
Do not allow indefinite model retries.
Model Retry Should Include the Failure Reason
Instead of simply asking:
Try again.
provide structured validation feedback:
The generated tool request is invalid.
Required field:
customerId: string
Unknown field:
includeAllTenants
This gives the model a precise correction target.
Recovery Should Preserve Original Intent
One danger of adaptive agents is that recovery can change the original task.
For example:
Original:
"Find pending invoices for customer A."
After a tool failure, the agent should not decide:
"Search all invoices."
just because that tool happens to work.
Recovery should preserve:
User Intent
+
Authorization
+
Tenant Scope
+
Business Constraints
while changing only the execution strategy.
Fallback Strategies
A fallback can be useful when a dependency is unavailable.
For example:
Primary Search
|
X
Unavailable
|
v
Secondary Search
But fallback data must have equivalent security requirements.
Do not automatically switch to a broader data source simply because the primary source failed.
Fallback Quality Levels
A workflow can define:
Primary
|
v
Full Result
Fallback
|
v
Reduced Result
No Safe Fallback
|
v
Failure
For example:
Primary:
Real-time inventory
Fallback:
Cached inventory from a known timestamp
The response should clearly indicate the data freshness if the business workflow requires it.
Circuit Breakers
If a dependency repeatedly fails:
Request
|
v
API
|
X
Failure
retrying every workflow can increase pressure.
A circuit breaker changes the behavior:
Closed
|
v
Failures increase
|
v
Open
|
v
Reject quickly
|
v
Half-Open
|
v
Test dependency
This protects both the workflow system and the failing dependency.
Retry and Circuit Breaker Work Together
A practical pattern is:
Tool Call
|
v
Circuit Check
|
+---- Open ---> Fail Fast
|
v
Execute
|
+---- Success
|
+---- Transient Failure
|
v
Retry
Retries handle individual transient failures.
Circuit breakers handle repeated dependency failures.
They solve different problems.
Bulkheads
One failing dependency should not consume all workflow capacity.
For example:
Workflow Workers
|
+--> Search
|
+--> Billing
|
+--> Documents
Give high-risk or unreliable dependencies separate capacity limits.
Join the conversation! Your thoughts help the community grow.