AI agent workflows are often demonstrated as a simple sequence:
Request
|
v
Agent
|
v
Tool
|
v
Response
Production systems are rarely that simple.
A real agent workflow may contain:
User Request
|
v
Planner
|
+------> Database
|
+------> API
|
+------> Search
|
+------> Document Processing
|
v
Decision
|
v
Final Response
Every external dependency introduces another failure point.
A database can time out. An API can return an error. A tool can produce invalid output. A model call can exceed its deadline. A workflow can be interrupted after successfully completing one step but before recording the next state.
For declarative agent workflows, this creates an important engineering problem:
How do you recover from failures without duplicating side effects, losing workflow state, or allowing an agent to make unsafe recovery decisions?
The answer is not simply "add retries."
Production-grade recovery requires a combination of:
What Is a Declarative Agent Workflow?
In an imperative workflow, application code explicitly controls execution:
var result = await StepOneAsync();
var next = await StepTwoAsync(result);
await StepThreeAsync(next);
A declarative workflow describes what should happen and allows the workflow runtime to manage execution.
Conceptually:
Workflow
|
+--> Step A
|
+--> Step B
|
+--> Step C
The runtime can manage:
Execution
State
Scheduling
Retries
Pausing
Resuming
Failures
This separation becomes particularly valuable when workflows run for longer periods or depend on multiple external systems.
Why Agent Workflows Need Failure Recovery
An agent workflow can fail at several layers:
Workflow
|
+---------------+---------------+
| | |
v v v
Model Tool Data
| | |
Timeout API Error Database
| | |
+---------------+---------------+
|
v
Failure
A failure is not always equivalent to "stop everything."
For example:
Search Tool Failed
|
v
Retry
|
v
Success
may be appropriate.
But:
Payment Failed
|
v
Retry
|
v
Retry
|
v
Retry
could be dangerous if the operation creates duplicate charges.
The recovery strategy must depend on the type of operation.
Failure Taxonomy
Start by classifying failures.
| Failure Type | Example | Typical Strategy |
|---|
| Transient | Timeout | Retry |
| Rate limit | Too many requests | Backoff |
| Authentication | Expired credential | Refresh or stop |
| Authorization | Permission denied | Stop |
| Validation | Invalid input | Correct or request input |
| Dependency | Service unavailable | Retry or fallback |
| Business | Operation rejected | Compensate or stop |
| Model | Invalid structured output | Retry with validation |
| Infrastructure | Worker crash | Resume from checkpoint |
| Unknown | Unexpected exception | Escalate |
This classification should happen before designing retry logic.
The First Principle: Not Every Failure Should Be Retried
A common implementation is:
try
Execute
catch
Retry
This is too broad.
A better approach is:
Failure
|
v
Classify
|
+--> Transient ------> Retry
|
+--> Rate Limited ---> Backoff
|
+--> Invalid -------> Correct
|
+--> Unauthorized --> Stop
|
+--> Business ------> Compensate
|
+--> Unknown ------> Escalate
Retries should be based on failure semantics.
Retry Policies
A basic retry policy can be represented as:
public sealed class RetryPolicy
{
public int MaxAttempts { get; init; }
public TimeSpan InitialDelay { get; init; }
public double BackoffMultiplier { get; init; }
public TimeSpan MaxDelay { get; init; }
}
For example:
var policy = new RetryPolicy
{
MaxAttempts = 4,
InitialDelay = TimeSpan.FromSeconds(1),
BackoffMultiplier = 2,
MaxDelay = TimeSpan.FromSeconds(30)
};
The resulting delays could be approximately:
Attempt 1
|
1 second
|
Attempt 2
|
2 seconds
|
Attempt 3
|
4 seconds
|
Attempt 4
The exact values should be chosen based on the dependency.
Exponential Backoff
Without backoff, multiple workflow workers may retry simultaneously:
Service Fails
|
+--> Worker A retries
+--> Worker B retries
+--> Worker C retries
+--> Worker D retries
This can create a retry storm.
With exponential backoff:
Worker A -> 1s
Worker B -> 2s
Worker C -> 4s
Worker D -> 8s
Add jitter when many workers may retry the same dependency.
Conceptually:
Delay = BaseDelay + RandomJitter
This reduces synchronized retries.
Retry Only Idempotent Operations Automatically
Consider:
GetCustomer
Retrying is generally safer than:
CreatePayment
because the second operation may produce another side effect.
The workflow should classify operations by idempotency.
For example:
| Operation | Automatic Retry |
|---|
| Read customer | Usually safe |
| Search documents | Usually safe |
| Get order | Usually safe |
| Create record | Requires idempotency |
| Send email | Requires care |
| Issue refund | Requires strong idempotency |
| Delete resource | Requires careful policy |
The exact behavior depends on the underlying API.
Idempotency Keys
For side-effecting operations, use an idempotency key.
For example:
var idempotencyKey =
$"workflow:{workflowId}:refund:{stepId}";
The same operation can then be identified across retries.
Conceptually:
Workflow
|
v
Refund Request
|
+--> Attempt 1
|
+--> Timeout
|
+--> Attempt 2
|
v
Same Key
|
v
Existing Result
The downstream service can recognize that the operation has already been processed.
Why Timeouts Are Required
A retry without a timeout can make a workflow wait indefinitely.
For example:
Agent
|
v
External API
|
+---- waiting...
|
+---- waiting...
|
+---- waiting...
Define an explicit timeout:
using var timeout =
new CancellationTokenSource(
TimeSpan.FromSeconds(15));
var result = await client
.GetAsync(
requestUri,
timeout.Token);
The workflow can then classify the timeout and decide whether to retry.
Workflow Timeout vs Activity Timeout
These are different.
Activity Timeout
Controls one operation:
Tool Call
|
+-- Maximum 15 seconds
Workflow Timeout
Controls the complete operation:
Entire Workflow
|
+-- Maximum 10 minutes
You may need both.
For example:
Workflow
|
+--> Tool A: 15 sec
|
+--> Tool B: 20 sec
|
+--> Tool C: 30 sec
The total workflow still needs a broader execution limit.
Durable State Is the Foundation
A workflow that can resume after failure must know where it was.
Instead of:
Memory
|
v
Current Step
use durable state:
Workflow
|
v
Persistent State
|
+--> Current Step
+--> Inputs
+--> Outputs
+--> Status
+--> Retry Count
+--> Correlation ID
A simplified model might be:
public sealed class WorkflowState
{
public required string WorkflowId { get; init; }
public required string CurrentStep { get; set; }
public string Status { get; set; } = "Running";
public int RetryCount { get; set; }
public DateTimeOffset UpdatedAt { get; set; }
}
The actual state model will depend on the workflow runtime.
Checkpoints
A checkpoint records successful progress.
Consider:
Step A
|
v
Checkpoint
|
v
Step B
|
v
Checkpoint
|
v
Step C
If Step C fails, the workflow can resume from the appropriate point instead of starting from Step A.
Without checkpoints:
Failure at Step C
|
v
Restart Step A
With checkpoints:
Failure at Step C
|
v
Resume Step C
This is particularly valuable for long-running workflows.
Checkpoint Too Early and Too Late
Checkpoint placement matters.
Suppose:
Create Order
|
Checkpoint
|
Send Confirmation
If the checkpoint is written before the order is actually committed, recovery may incorrectly assume the operation completed.
On the other hand, checkpointing after every tiny operation can add overhead.
A useful checkpoint represents a meaningful, durable state transition.
Workflow State Machine
Recovery becomes easier when workflow states are explicit.
For example:
Created
|
v
Planning
|
v
Executing
|
+----> Waiting
|
v
Completed
Failure states can be represented separately:
Executing
|
v
Retrying
|
+----> Executing
|
v
Failed
|
v
Compensating
|
v
Compensated
This prevents the workflow from relying on ambiguous flags.
Example State Model
public enum WorkflowStatus
{
Created,
Running,
Waiting,
Retrying,
Completed,
Failed,
Compensating,
Cancelled
}
A workflow can then explicitly transition:
Running -> Retrying
Retrying -> Running
Running -> Completed
Running -> Failed
Failed -> Compensating
Not every state transition should be allowed.
Validate State Transitions
Avoid code that permits:
Completed -> Running
unless the workflow explicitly supports reopening.
A state transition method can enforce valid transitions:
public void MoveTo(WorkflowStatus next)
{
if (!IsValidTransition(Status, next))
{
throw new InvalidOperationException(
$"Invalid transition: {Status} -> {next}");
}
Status = next;
}
This makes recovery logic more predictable.
Compensation Instead of Rollback
Distributed workflows cannot always perform a traditional database rollback.
Consider:
Create Customer
|
v
Create Subscription
|
v
Send Notification
The database transaction may not span all three operations.
If notification fails, you cannot necessarily roll back the external subscription transactionally.
Instead, use compensation:
Notification Failed
|
v
Cancel Subscription
This is a saga-style approach.
Example Compensation Flow
Step 1
Create Customer
|
v
Step 2
Create Subscription
|
v
Step 3
Send Notification
|
X
Failure
|
v
Compensate Step 2
|
v
Cancel Subscription
The compensation operation should itself be designed to be idempotent.
Compensation Is Not Perfect Rollback
This distinction matters.
A database rollback can restore transactional state.
A compensation workflow performs another business operation to reduce or reverse the effect.
For example:
Charge Customer
cannot always be "rolled back" in the database.
A refund is a new business operation:
Charge
|
v
Refund
The system should model this explicitly.
Agent Workflows Add Another Failure Dimension
An agent can choose the wrong tool.
For example:
User Request
|
v
Agent
|
v
Wrong Tool
This is different from:
Correct Tool
|
v
Temporary Network Failure
Retrying the wrong tool does not solve the problem.
Agent workflows therefore need to distinguish:
Execution failure
from:
Decision failure
Validate Agent Tool Selection
Before executing a tool call, validate:
Tool exists
Tool is approved
Agent has permission
Input matches schema
Tenant context is valid
Risk policy allows execution
Conceptually:
Agent Decision
|
v
Policy Validation
|
+---- Denied
|
v
Tool Execution
The agent should not bypass this layer.
Validate Structured Model Output
Suppose the workflow expects:
{
"tool": "invoice.search",
"customerId": "123"
}
The model might produce:
{
"tool": "invoice.search",
"customerId": 123,
"includeAllTenants": true
}
Schema validation should reject unexpected or invalid fields.
A strongly typed model helps:
public sealed class ToolRequest
{
public required string Tool { get; init; }
public required string CustomerId { get; init; }
}
Then validate before execution.
Recovery From Invalid Model Output
An invalid model response should not necessarily terminate the workflow immediately.
A possible strategy is:
Invalid Output
|
v
Validation Error
|
v
Structured Retry
|
v
Model Generates Correct Format
But limit the number of attempts.
For example:
Attempt 1 -> Invalid
Attempt 2 -> Invalid
Attempt 3 -> Invalid
|
v
Failed
Do not allow indefinite model retries.
Model Retry Should Include the Failure Reason
Instead of simply asking:
Try again.
provide structured validation feedback:
The generated tool request is invalid.
Required field:
customerId: string
Unknown field:
includeAllTenants
This gives the model a precise correction target.
Recovery Should Preserve Original Intent
One danger of adaptive agents is that recovery can change the original task.
For example:
Original:
"Find pending invoices for customer A."
After a tool failure, the agent should not decide:
"Search all invoices."
just because that tool happens to work.
Recovery should preserve:
User Intent
+
Authorization
+
Tenant Scope
+
Business Constraints
while changing only the execution strategy.
Fallback Strategies
A fallback can be useful when a dependency is unavailable.
For example:
Primary Search
|
X
Unavailable
|
v
Secondary Search
But fallback data must have equivalent security requirements.
Do not automatically switch to a broader data source simply because the primary source failed.
Fallback Quality Levels
A workflow can define:
Primary
|
v
Full Result
Fallback
|
v
Reduced Result
No Safe Fallback
|
v
Failure
For example:
Primary:
Real-time inventory
Fallback:
Cached inventory from a known timestamp
The response should clearly indicate the data freshness if the business workflow requires it.
Circuit Breakers
If a dependency repeatedly fails:
Request
|
v
API
|
X
Failure
retrying every workflow can increase pressure.
A circuit breaker changes the behavior:
Closed
|
v
Failures increase
|
v
Open
|
v
Reject quickly
|
v
Half-Open
|
v
Test dependency
This protects both the workflow system and the failing dependency.
Retry and Circuit Breaker Work Together
A practical pattern is:
Tool Call
|
v
Circuit Check
|
+---- Open ---> Fail Fast
|
v
Execute
|
+---- Success
|
+---- Transient Failure
|
v
Retry
Retries handle individual transient failures.
Circuit breakers handle repeated dependency failures.
They solve different problems.
Bulkheads
One failing dependency should not consume all workflow capacity.
For example:
Workflow Workers
|
+--> Search
|
+--> Billing
|
+--> Documents
Give high-risk or unreliable dependencies separate capacity limits.
Conceptually:
Search Pool
##########
Billing Pool
####
Document Pool
######
If document processing becomes slow, it should not necessarily block all billing workflows.
Concurrency Limits
Agent workflows can dynamically generate multiple tool calls.
For example:
Agent
|
+--> Search 1
+--> Search 2
+--> Search 3
+--> Search 4
+--> Search 5
+--> Search 6
Without limits, this can overload dependencies.
Use bounded concurrency:
var semaphore =
new SemaphoreSlim(4);
await semaphore.WaitAsync(
cancellationToken);
try
{
await ExecuteToolAsync(
cancellationToken);
}
finally
{
semaphore.Release();
}
The correct limit depends on the dependency and workload.
Cancellation
Users may cancel a long-running request.
The workflow should propagate cancellation:
User
|
X
Cancel
|
v
Agent
|
v
Workflow
|
+--> Stop New Work
|
+--> Cancel Safe Operations
|
+--> Persist State
Cancellation is not the same as failure.
The workflow may need to enter:
Cancelled
rather than:
Failed
Graceful Shutdown
Workers can be restarted during deployment or infrastructure maintenance.
A durable workflow should handle:
Worker
|
X
Shutdown
without losing state.
A good architecture is:
Persistent Workflow State
|
v
New Worker
|
v
Resume
The worker should not be the source of truth for workflow state.
Exactly-Once Is Difficult
Distributed systems frequently use the phrase:
Exactly once
but implementing true exactly-once side effects across independent systems is difficult.
A safer model is:
At-least-once execution
+
Idempotent operations
+
Durable state
For example:
Workflow retries
|
v
Same idempotency key
|
v
Downstream detects duplicate
This provides practical protection against duplicate effects.
At-Least-Once vs At-Most-Once
At-Most-Once
Execute
|
X
Failure
|
v
Do not retry
You reduce duplicate operations but may lose work.
At-Least-Once
Execute
|
X
Unknown result
|
v
Retry
You increase the chance of completing the operation but must handle duplicates.
For many workflows, at-least-once execution combined with idempotency is more practical.
Unknown Outcomes Are Dangerous
Consider:
Create Order
|
v
Server processes request
|
v
Response lost
|
v
Workflow sees timeout
The workflow does not know whether the order was created.
Blindly retrying could create a duplicate.
Instead:
Timeout
|
v
Check operation status
|
+--> Exists ---> Continue
|
+--> Does not exist ---> Retry
This pattern is extremely important for side-effecting operations.
Use Operation IDs
Create an operation identifier before the side effect:
var operationId =
$"{workflowId}:{stepId}";
The downstream system can store:
OperationId
Status
Result
CreatedAt
A repeated request with the same operation ID can return the existing result.
Durable Workflow State Example
A simplified persistence model might be:
CREATE TABLE workflow_instances
(
id UUID PRIMARY KEY,
workflow_type VARCHAR(200) NOT NULL,
status VARCHAR(50) NOT NULL,
current_step VARCHAR(200),
retry_count INTEGER NOT NULL DEFAULT 0,
input JSONB NOT NULL,
output JSONB,
created_at TIMESTAMP NOT NULL,
updated_at TIMESTAMP NOT NULL
);
Step execution can be tracked separately:
CREATE TABLE workflow_steps
(
id UUID PRIMARY KEY,
workflow_id UUID NOT NULL,
step_name VARCHAR(200) NOT NULL,
status VARCHAR(50) NOT NULL,
attempt INTEGER NOT NULL,
idempotency_key VARCHAR(300),
started_at TIMESTAMP,
completed_at TIMESTAMP,
error_code VARCHAR(100)
);
This makes recovery and auditing easier.
Store Errors as Structured Data
Avoid storing only:
Something went wrong.
Store:
Error Category
Error Code
Dependency
Attempt
Timestamp
Retryable
Correlation ID
For example:
{
"category": "Transient",
"code": "DEPENDENCY_TIMEOUT",
"dependency": "invoice-api",
"retryable": true,
"attempt": 2
}
Structured failures make automated recovery much more reliable.
Dead-Letter Workflows
Some workflows cannot recover automatically.
Instead of retrying forever:
Failure
|
v
Retry
|
v
Retry
|
v
Retry
|
v
Retry forever
move the workflow to a recoverable failure state:
Failure
|
v
Retry Limit
|
v
Dead-Letter / Manual Review
The workflow can then be investigated and resumed manually.
Human-in-the-Loop Recovery
Some operations should require human approval.
For example:
Agent
|
v
Refund Request
|
v
Policy Check
|
v
High Value
|
v
Human Approval
|
+---- Denied
|
v
Execute
This is safer than allowing an agent to automatically retry a high-impact operation.
Recovery Should Be Observable
Every retry should be measurable.
Capture:
Workflow ID
Step
Attempt
Failure Type
Delay
Dependency
Outcome
For example:
Workflow: 123
Step: invoice.search
Attempt: 2
Failure: timeout
Delay: 4 seconds
Result: success
This allows operators to identify recurring dependency problems.
Metrics That Matter
Track:
| Metric | Why It Matters |
|---|
| Workflow success rate | Overall reliability |
| Retry rate | Dependency stability |
| Recovery success rate | Recovery effectiveness |
| Average retry count | Retry pressure |
| Dead-letter count | Manual workload |
| Workflow duration | User experience |
| Step failure rate | Problematic components |
| Compensation count | Business failures |
| Cancellation rate | Workflow behavior |
| Duplicate prevention count | Idempotency effectiveness |
A high retry rate may indicate that the underlying dependency needs attention.
Distributed Tracing
A workflow can span multiple services:
Workflow
|
+--> Agent
|
+--> Tool
|
+--> API
|
+--> Database
Use a shared correlation or trace identifier.
For example:
Trace ID: abc123
Workflow
|
+--> Planner
|
+--> Search Tool
|
+--> Customer API
|
+--> Database
This makes failures much easier to investigate.
Recovery Policies Should Be Declarative Too
If the workflow itself is declarative, recovery rules should ideally be represented as configuration or workflow policy.
For example:
{
"step": "customer.search",
"retry": {
"maxAttempts": 3,
"backoff": "exponential",
"jitter": true
},
"timeoutSeconds": 15,
"onFailure": "fallback-search"
}
This keeps the recovery behavior visible alongside the workflow definition.
Separate Business Recovery From Infrastructure Recovery
These are different.
Infrastructure Recovery
Network timeout
API unavailable
Worker restart
Database connection failure
Usually handled with:
Retry
Backoff
Resume
Circuit breaker
Business Recovery
Payment declined
Credit limit exceeded
Customer not eligible
Order already cancelled
Usually requires:
Business rule
Compensation
Alternative path
Human intervention
Do not treat a business rejection as a transient infrastructure failure.
Recovery Decision Matrix
| Failure | Retry | Fallback | Compensation | Human |
|---|
| Network timeout | Yes | Maybe | No | No |
| Rate limit | Yes | Maybe | No | No |
| Invalid input | Usually no | No | No | Maybe |
| Permission denied | No | No | No | Maybe |
| Payment declined | Usually no | Maybe | Depends | Maybe |
| Duplicate request | No | No | No | No |
| Worker crash | Resume | No | No | No |
| Unknown outcome | Carefully | Maybe | Depends | Maybe |
| High-value operation failure | Carefully | Maybe | Often | Yes |
The exact policy should be defined by business risk.
Common Mistakes
Retrying Every Exception
This can amplify failures and create duplicate side effects.
No Idempotency for Writes
Retries can create duplicate records or transactions.
Keeping Workflow State Only in Memory
Worker failure can then destroy the recovery point.
Restarting From the Beginning
Long-running workflows may repeat expensive or dangerous operations.
Treating Compensation as Rollback
Compensation is another business operation, not necessarily a transactional undo.
Allowing Unlimited Retries
A permanently failing dependency can consume resources indefinitely.
Ignoring Unknown Outcomes
A timeout does not always mean the operation did not happen.
Letting the Agent Choose Recovery Without Policy
The agent should not decide that a dangerous alternative operation is acceptable simply because the original tool failed.
Using One Timeout for Everything
Model calls, APIs, databases, and entire workflows have different timing requirements.
Hiding Failed Workflows
A workflow that cannot recover automatically should be visible to operators.
Troubleshooting
The Workflow Executes a Step Twice
Check:
Idempotency key
Checkpoint timing
Retry behavior
Worker restart handling
Duplicate message handling
Workflow Restarts From the Beginning
Check whether workflow state and completed step information are persisted durably.
Retries Make the Dependency Worse
Check:
Backoff
Jitter
Maximum attempts
Circuit breaker
Concurrency limits
Workflow Says "Success" After a Failed Operation
Inspect the checkpoint location.
The workflow may be recording completion before the side effect is actually durable.
A Timeout Causes Duplicate Records
The operation likely needs an idempotency key or status-reconciliation mechanism.
Agent Keeps Calling a Failed Tool
Add:
Failure classification
Circuit breaking
Tool health state
Retry limits
Alternative-path policy
Compensation Fails
Compensation should have its own retry and escalation strategy.
Do not assume compensation always succeeds on the first attempt.
Workflow Becomes Stuck
Inspect:
Current state
Current step
Retry count
Pending activity
Worker availability
Timeout
Dependency status
A workflow watchdog can identify instances that have remained in one state beyond an expected threshold.
Best Practices
Classify failures before deciding how to recover.
Retry only operations that are safe to retry.
Use exponential backoff and jitter for transient failures.
Give every external operation an explicit timeout.
Persist workflow state durably.
Use meaningful checkpoints.
Design side-effecting operations to be idempotent.
Use operation IDs or idempotency keys for retried writes.
Treat unknown outcomes differently from confirmed failures.
Use compensation for distributed business operations.
Do not confuse compensation with transactional rollback.
Limit retry attempts.
Use circuit breakers for repeatedly failing dependencies.
Use bounded concurrency to protect downstream services.
Persist state before allowing a workflow to resume.
Separate infrastructure recovery from business recovery.
Validate agent-generated tool requests before execution.
Do not allow recovery logic to bypass authorization.
Provide a dead-letter or manual recovery path.
Measure retries, failures, recovery time, and compensation activity.
A Production-Oriented Recovery Architecture
Putting the major pieces together:
User Request
|
v
Agent Workflow
|
+---------+---------+
| |
v v
Durable Policy
State Checks
| |
+---------+---------+
|
v
Workflow Step
|
v
Tool / API
|
+--------+--------+
| |
Success Failure
| |
v v
Checkpoint Failure Classifier
| |
| +--------+--------+
| | | |
| v v v
| Retry Fallback Stop
| |
| v
| Backoff
| |
+--------+
|
v
Next Step
For business operations:
Failure
|
v
Compensation
|
v
Recovery State
|
+----> Completed
|
+----> Manual Review
This gives the workflow runtime explicit recovery paths instead of relying on exception handling alone.
A Practical Recovery Contract
A useful abstraction is:
public sealed record RecoveryDecision
{
public required RecoveryAction Action { get; init; }
public TimeSpan? Delay { get; init; }
public string? Reason { get; init; }
}
public enum RecoveryAction
{
Retry,
Fallback,
Compensate,
Fail,
Escalate
}
The workflow engine can then make recovery behavior explicit:
var decision =
recoveryPolicy.Decide(
failure,
workflowContext);
switch (decision.Action)
{
case RecoveryAction.Retry:
await ScheduleRetryAsync(
decision.Delay,
cancellationToken);
break;
case RecoveryAction.Fallback:
await ExecuteFallbackAsync(
cancellationToken);
break;
case RecoveryAction.Compensate:
await ExecuteCompensationAsync(
cancellationToken);
break;
case RecoveryAction.Fail:
await MarkFailedAsync(
cancellationToken);
break;
case RecoveryAction.Escalate:
await EscalateAsync(
cancellationToken);
break;
}
The advantage is that recovery becomes a policy decision rather than scattered try/catch blocks.
Designing for Resume
A durable workflow should be able to answer:
Where was I?
What already completed?
What failed?
How many times did I retry?
What side effects may have happened?
What can safely execute next?
If the system cannot answer those questions after a worker restart, recovery is incomplete.
A strong workflow state should therefore preserve:
Workflow ID
Current state
Current step
Completed steps
Step outputs
Attempt count
Idempotency keys
Failure information
Correlation ID
Created timestamp
Updated timestamp
The Recovery Hierarchy
A useful way to think about recovery is:
Level 1
-------
Retry
Level 2
-------
Fallback
Level 3
-------
Resume
Level 4
-------
Compensate
Level 5
-------
Manual Recovery
The workflow should use the least disruptive safe mechanism.
For example:
Temporary timeout
|
v
Retry
but:
Business operation rejected
|
v
Business recovery
and:
Unknown high-risk outcome
|
v
Manual investigation
Conclusion
Declarative agent workflows make complex AI processes easier to describe and orchestrate, but production reliability requires much more than defining a sequence of steps.
Failures are inevitable.
The important question is what the workflow does when they occur.
A robust recovery architecture combines:
Durable State
+
Checkpoints
+
Failure Classification
+
Retries
+
Backoff
+
Timeouts
+
Idempotency
+
Fallbacks
+
Compensation
+
Circuit Breakers
+
Concurrency Limits
+
Human Recovery
The most important principle is:
Recovery must be designed around the semantics of the failed operation, not around the fact that an exception occurred.
A temporary network timeout may deserve a retry.
A permission failure should normally stop.
An unknown result from a side-effecting operation may require reconciliation.
A completed business operation may require compensation rather than rollback.
And a high-risk failure may require human intervention.
For AI agents, there is one additional rule:
Never let the agent decide that a recovery action is safe merely because it is technically possible.
The workflow's policy, authorization layer, and business rules must remain in control.
When declarative workflows combine durable state with explicit recovery policies, agent systems become far more resilient to worker failures, dependency outages, model errors, retries, and distributed-system uncertainty.
Frequently Asked Questions
Why are retries not enough for agent workflows?
Retries only address certain transient failures. They do not solve duplicate side effects, worker crashes, unknown outcomes, business failures, or compensation requirements.
What is the most important feature for workflow recovery?
Durable workflow state is fundamental. Without persistent state, a restarted worker may not know which steps completed or which operations are safe to repeat.
When should an agent workflow retry?
Retry when the failure is known to be transient and the operation is safe to repeat or protected by idempotency.
What is an idempotency key?
An idempotency key uniquely identifies a logical operation so that repeated requests can be recognized as the same operation rather than separate operations.
What should happen when an API times out after processing a request?
Do not automatically assume the operation failed. Reconcile the operation using its identifier or status endpoint before deciding whether to retry.
What is compensation?
Compensation is a separate business operation that reverses or mitigates the effect of an earlier successful operation when a distributed workflow cannot perform a traditional transaction rollback.
Should an AI agent choose its own fallback?
Not for unrestricted operations. Fallbacks should be defined by workflow policy, authorization, and business rules.
How many times should a workflow retry?
There is no universal number. Use a bounded retry policy based on dependency behavior, operation risk, timeout limits, and business requirements.
What happens when automatic recovery fails?
Move the workflow into an explicit failed, dead-letter, or manual-review state so that it can be investigated and potentially resumed without restarting the entire workflow.
How can duplicate side effects be prevented?
Use idempotency keys, operation identifiers, durable step state, and downstream deduplication for side-effecting operations.
Should workflow state be stored in memory?
Not for workflows that must survive worker restarts or long execution periods. Durable state should be persisted outside the worker process.
What is the difference between workflow failure and tool failure?
A tool failure is a failure of an individual operation. Workflow failure means the overall process cannot continue safely. A tool failure may be recoverable without failing the entire workflow.