AI workflows are easy to demonstrate when everything works.
An agent receives a request, calls a tool, produces an answer, and the workflow completes successfully.
Production systems are different.
An external API can time out. An MCP tool can return an error. An AI model can produce an invalid response. A downstream service can become temporarily unavailable. A human approval step can be rejected. A retry can succeed on the second attempt but fail repeatedly under sustained load.
These scenarios are particularly important when using declarative workflows because orchestration logic is defined outside traditional application code.
Microsoft Agent Framework declarative workflows allow .NET developers to define workflow behavior using YAML rather than implementing every orchestration path directly in C#. The YAML definition is loaded into a standard Workflow, which can then be executed and composed with code-based workflows. Declarative workflows support actions for control flow, agent and tool invocation, HTTP and MCP integration, and human-in-the-loop interactions.
That makes declarative workflows easier to review and modify, but it also creates a testing requirement:
How do you prove that the workflow behaves correctly when individual steps fail?
One effective answer is failure injection.
Instead of waiting for production failures, deliberately introduce controlled failures during testing and verify that the workflow follows the expected recovery path.
What Is Failure Injection?
Failure injection is a testing technique in which a controlled dependency failure is introduced to verify how an application behaves.
For an AI workflow, the dependency might be:
For example, consider:
User Request
|
v
Triage Agent
|
v
Customer API
|
v
Response Agent
A happy-path test verifies:
Triage → API → Response → Success
A failure-injection test might force:
Triage → API → Timeout
|
v
Retry
|
v
Success
Or:
Triage → API → Failure
|
v
Recovery Branch
|
v
User Message
The objective is not merely to verify that an exception is thrown.
The objective is to verify that the workflow reaches the correct state after the failure.
Why Declarative Workflows Need Failure Testing
Declarative workflows move orchestration logic into YAML.
A simplified workflow might look like:
kind: Workflow
trigger:
kind: OnConversationStart
id: support_workflow
actions:
- kind: InvokeAzureAgent
id: classify_request
agent:
name: TriageAgent
- kind: InvokeMcpTool
id: lookup_customer
tool:
name: get_customer
- kind: InvokeAzureAgent
id: generate_response
agent:
name: ResponseAgent
- kind: SendActivity
id: send_response
activity:
text: =Local.response
Microsoft's current declarative workflow documentation describes actions as the building blocks of a workflow, with actions executed according to the workflow definition. C# declarative workflows use a trigger-based YAML structure, and expressions can reference workflow state through namespaces such as Local.* and System.*.
The problem is that a visually correct YAML file does not prove that every failure path works.
A workflow can have excellent happy-path behavior and still fail badly when:
A tool times out.
An agent returns an unusable result.
A branch receives an unexpected value.
A retry executes incorrectly.
A checkpoint cannot be persisted.
A downstream dependency remains unavailable.
Define the Failure Contract First
Before injecting failures, define what the workflow is supposed to do.
For example:
| Failure | Expected Behavior |
|---|
| First API call fails | Retry |
| API remains unavailable | Return controlled error |
| Agent returns empty result | Branch to fallback |
| MCP tool fails | Do not continue with invalid data |
| Approval rejected | Stop privileged operation |
| Dependency timeout | Record failure and recover |
| Checkpoint failure | Surface workflow failure |
| Invalid input | Reject before expensive operations |
This becomes your test contract.
Without an expected outcome, failure injection only proves that something broke.
Build a Testable Workflow
A useful workflow should make important state observable.
For example:
kind: Workflow
trigger:
kind: OnConversationStart
id: customer_support
actions:
- kind: SetVariable
id: initialize
variable: Local.status
value: started
- kind: InvokeAzureAgent
id: classify_request
displayName: Classify customer request
agent:
name: TriageAgent
- kind: SetVariable
id: mark_classified
variable: Local.status
value: classified
- kind: SendActivity
id: send_status
activity:
text: =Concat("Workflow status: ", Local.status)
The exact agent configuration depends on the Agent Framework environment.
The important testing principle is that meaningful state transitions should be identifiable.
Instead of testing only:
Output == "Success"
test the workflow's intermediate behavior as well:
Started
↓
Classified
↓
Dependency Called
↓
Recovered
↓
Completed
Inject Failures at Dependency Boundaries
Do not randomly corrupt the entire workflow.
Inject failures at boundaries where external dependencies can realistically fail.
For example:
Workflow
|
+-- Agent
|
+-- MCP Tool ← Failure injection
|
+-- HTTP API ← Failure injection
|
+-- Database ← Failure injection
|
+-- Approval ← Failure injection
This makes failures deterministic and repeatable.
A good test should be able to say:
Given API failure on attempt 1,
when the workflow executes,
then it retries,
and the second attempt succeeds.
That is far more useful than a test that occasionally fails because a real external service happened to be unavailable.
Use a Failure-Injection Test Double
One practical pattern is to place a controllable wrapper around a dependency.
For example:
public interface ICustomerService
{
Task<Customer?> GetCustomerAsync(
string customerId,
CancellationToken cancellationToken);
}
The production implementation might call an HTTP service.
The test implementation can deliberately fail:
public sealed class FaultInjectingCustomerService
: ICustomerService
{
private readonly ICustomerService _inner;
private int _remainingFailures;
public FaultInjectingCustomerService(
ICustomerService inner,
int failuresBeforeSuccess)
{
_inner = inner;
_remainingFailures = failuresBeforeSuccess;
}
public Task<Customer?> GetCustomerAsync(
string customerId,
CancellationToken cancellationToken)
{
if (_remainingFailures > 0)
{
_remainingFailures--;
throw new HttpRequestException(
"Injected test failure.");
}
return _inner.GetCustomerAsync(
customerId,
cancellationToken);
}
}
Now a test can configure:
failuresBeforeSuccess = 1
and verify that the workflow's retry behavior works.
For persistent failure:
failuresBeforeSuccess = 100
can simulate a dependency that remains unavailable throughout the test.
The important characteristic is determinism.
Inject Timeouts Separately From Errors
A timeout is not identical to an HTTP 500 response.
A workflow may handle them differently.
A test double can deliberately delay a dependency:
public sealed class SlowCustomerService
: ICustomerService
{
public async Task<Customer?> GetCustomerAsync(
string customerId,
CancellationToken cancellationToken)
{
await Task.Delay(
TimeSpan.FromSeconds(30),
cancellationToken);
return null;
}
}
A test can then use a shorter timeout and verify that cancellation propagates correctly.
For example:
using var cts =
new CancellationTokenSource(
TimeSpan.FromSeconds(2));
await Assert.ThrowsAsync<OperationCanceledException>(
() => service.GetCustomerAsync(
"customer-123",
cts.Token));
The exact exception type should be verified against the application's HTTP/client implementation rather than assumed.
The important test is:
Dependency becomes slow
↓
Timeout occurs
↓
Workflow does not hang indefinitely
↓
Expected recovery path executes
Test Agent Failures Without Depending on a Real Model
AI model behavior is inherently variable.
A workflow test should not require a real LLM to produce exactly the same response every time.
Instead, separate orchestration tests from model-quality tests.
For orchestration testing, use a deterministic test agent or provider that can return predefined results.
For example:
public sealed class FakeAgent
{
private readonly string _response;
public FakeAgent(string response)
{
_response = response;
}
public Task<string> RunAsync(
string input,
CancellationToken cancellationToken)
{
return Task.FromResult(_response);
}
}
Tests can then simulate:
Valid response
Empty response
Malformed response
Expected refusal
Dependency error
Timeout
This allows the workflow itself to be tested independently of model variability.
Test Conditional Branches
Declarative workflows support conditional control flow. Microsoft documents actions such as If, ConditionGroup, Foreach, BreakLoop, ContinueLoop, and GotoAction for workflow control.
Suppose a workflow checks whether an agent produced a result:
- kind: If
id: validate_result
condition: =IsBlank(Local.AgentResult)
then:
- kind: SetVariable
id: mark_failure
variable: Local.status
value: failed
- kind: SendActivity
id: send_failure
activity:
text: "The request could not be completed."
else:
- kind: SetVariable
id: mark_success
variable: Local.status
value: completed
You should test both branches.
Test Case 1: Valid Result
AgentResult = valid
Expected status = completed
Test Case 2: Empty Result
AgentResult = empty
Expected status = failed
Test Case 3: Unexpected Result
AgentResult = malformed
Expected behavior = controlled failure
Testing only the first case leaves the most important workflow logic unverified.
Test Retry Behavior
Retries deserve dedicated tests.
Consider a dependency that fails twice and succeeds on the third attempt:
Attempt 1 → Failure
Attempt 2 → Failure
Attempt 3 → Success
The test should verify:
Number of calls = 3
Final workflow state = success
Then test permanent failure:
Attempt 1 → Failure
Attempt 2 → Failure
Attempt 3 → Failure
Expected:
Number of calls = configured retry limit
Final workflow state = controlled failure
Do not only assert that the final output is correct.
Also assert that the workflow did not exceed its retry policy.
Avoid Testing Retries With Real Delays
A common mistake is to make tests wait for actual retry delays.
For example:
Retry delay = 30 seconds
Retries = 5
could turn a unit test into a multi-minute test.
Use an injectable delay abstraction where appropriate:
public interface IRetryDelay
{
Task DelayAsync(
TimeSpan delay,
CancellationToken cancellationToken);
}
The production implementation performs the actual delay.
The test implementation can complete immediately.
This gives you deterministic tests without sacrificing production retry behavior.
Test MCP Tool Failures
If the workflow invokes MCP tools, treat the MCP server as an external dependency.
Test at least:
Tool succeeds
Tool returns expected error
Tool times out
Tool returns malformed data
Tool becomes unavailable
A useful test matrix is:
| MCP Scenario | Expected Workflow Behavior |
|---|
| Valid result | Continue |
| Empty result | Validate and branch |
| Tool error | Recover or fail cleanly |
| Timeout | Apply timeout policy |
| Connection failure | Retry if appropriate |
| Invalid schema | Reject result |
| Unauthorized | Stop operation |
The goal is to ensure that an MCP failure does not accidentally become an apparently valid AI response.
Test Failure Propagation
A particularly important scenario is an error that occurs deep in the workflow.
Consider:
User
↓
Agent A
↓
MCP Tool
↓
HTTP API
↓
Database
If the database fails, the test should verify what happens at every layer.
For example:
Database failure
↓
HTTP API returns error
↓
MCP tool reports failure
↓
Agent does not fabricate data
↓
Workflow enters recovery path
↓
User receives controlled response
The exact implementation will depend on the application's error-handling architecture, but the test should validate the complete propagation path.
Test Human-in-the-Loop Failures
Declarative workflows support human-in-the-loop actions, including questions and external input requests.
That means approval itself is a failure mode.
Test:
Approval requested
↓
Approved
↓
Continue
and:
Approval requested
↓
Rejected
↓
Stop privileged operation
Also test:
Approval requested
↓
No response
↓
Timeout
↓
Controlled workflow state
A rejected approval should never be interpreted as a transient technical error unless the business logic explicitly defines it that way.
Test Checkpoint Recovery
Long-running workflows introduce another important failure scenario: the process can terminate after some work has already completed.
Microsoft Agent Framework workflows support checkpointing so workflow state can be saved and resumed.
A useful recovery test looks like:
Step 1 → Success
Step 2 → Success
Checkpoint
Step 3 → Process crashes
↓
Restart
↓
Restore checkpoint
↓
Resume Step 3
The test should verify that completed operations are not unintentionally repeated.
This is particularly important when a workflow performs non-idempotent operations such as:
Create payment
Send email
Create ticket
Submit order
Provision resource
A retry after checkpoint restoration must not accidentally duplicate the operation.
Test Idempotency
Failure injection often exposes duplicate-operation bugs.
Consider:
await paymentService.CreatePaymentAsync(
orderId,
amount,
cancellationToken);
If the request succeeds but the workflow crashes before recording the result, a retry might create another payment.
Use an idempotency key:
await paymentService.CreatePaymentAsync(
orderId,
amount,
idempotencyKey: $"payment:{orderId}",
cancellationToken);
The exact API depends on the downstream service.
The test should simulate:
Payment succeeds
↓
Workflow crashes
↓
Workflow resumes
↓
Payment operation retried
↓
Exactly one payment exists
This is a production reliability concern, not merely a workflow-testing concern.
Build a Failure Matrix
A mature test suite should maintain a failure matrix.
| Component | Failure | Expected Result | Test Type |
|---|
| Agent | Empty response | Fallback | Integration |
| Agent | Exception | Controlled failure | Integration |
| MCP | Timeout | Retry | Integration |
| MCP | Invalid result | Validation failure | Integration |
| HTTP API | 500 | Retry | Integration |
| HTTP API | 401 | Stop | Integration |
| Database | Timeout | Recovery | Integration |
| Approval | Rejected | Stop | Integration |
| Checkpoint | Restore | Resume | Integration |
| Workflow | Invalid input | Reject | Unit |
| Workflow | Invalid branch | Fallback | Unit |
This matrix becomes a living reliability specification.
Separate Unit, Integration, and Resilience Tests
Do not put every failure scenario into one enormous test suite.
Unit Tests
Test individual components:
Authorization
Validation
Retry policy
Failure classification
Transformation
Integration Tests
Test workflow behavior with controlled dependencies:
Workflow
+ Test Agent
+ Test MCP Server
+ Test HTTP API
+ Test Database
Resilience Tests
Test system-level behavior:
Multiple instances
Network failures
Dependency outages
Process termination
Checkpoint recovery
Load
This separation keeps normal CI tests fast while allowing deeper resilience tests to run in dedicated pipelines.
Validate the YAML Before Runtime Testing
A malformed declarative workflow should fail before a production deployment.
Keep YAML definitions under source control and validate them as part of CI.
The current declarative workflow model treats YAML as a versioned workflow definition that is loaded into a standard workflow at runtime.
A CI pipeline can therefore follow:
Pull Request
↓
YAML Validation
↓
Workflow Construction
↓
Unit Tests
↓
Failure-Injection Tests
↓
Integration Tests
↓
Deployment
This is significantly safer than discovering an invalid workflow definition after deployment.
Add Observability to Test Failures
A failure test should make it obvious where the failure occurred.
Record:
Workflow ID
Execution ID
Action ID
Tenant ID
Correlation ID
Failure type
Retry attempt
Elapsed time
Final state
For example:
logger.LogWarning(
"Workflow action failed. Workflow={WorkflowId}, " +
"Action={ActionId}, Attempt={Attempt}, " +
"FailureType={FailureType}",
workflowId,
actionId,
attempt,
failureType);
Avoid logging sensitive prompts, model responses, credentials, or customer information merely to make a test easier to debug.
Common Mistakes
Testing Only the Happy Path
This is the biggest mistake.
A workflow is not production-ready because one successful input produces the expected output.
Injecting Random Failures
Random failures make tests difficult to reproduce.
Prefer deterministic scenarios:
Fail once
Fail twice
Always fail
Delay once
Return malformed response
Using Real Production Dependencies
Do not deliberately break production systems to validate ordinary CI behavior.
Use controlled test doubles, mocks, local infrastructure, or dedicated resilience environments.
Testing Only Exceptions
Not every failure is an exception.
Also test:
Empty responses
Invalid data
Authorization failures
Timeouts
Rejected approvals
Partial results
Unexpected state
Ignoring Side Effects
A workflow can report failure while still creating a duplicate external operation.
Always test side effects and idempotency for operations that modify external systems.
Coupling Tests to Exact LLM Wording
Do not make orchestration tests depend on one exact generated sentence.
Prefer structured outcomes, tool calls, state transitions, and business assertions.
Troubleshooting Failed Workflow Tests
The Test Is Flaky
Check whether the test depends on:
A real model
Real network calls
Timing
Random data
Shared state
External services
Replace uncontrolled dependencies with deterministic test implementations.
The Retry Test Takes Too Long
Inject the retry-delay mechanism and make the test implementation complete immediately.
The production delay should remain unchanged.
The Workflow Continues After a Tool Failure
Verify whether the failure is represented as:
Exception
or:
Error result
Your workflow may be checking only for exceptions while the dependency returns a valid HTTP response containing an application-level error.
A Recovery Test Duplicates Data
Inspect the checkpoint boundary and idempotency strategy.
If an external operation completed before the checkpoint was written, replaying that operation can produce duplicates unless the operation is idempotent.
YAML Changes Are Not Being Tested
Ensure the test loads the same YAML file used by the deployment rather than recreating the workflow entirely in C#.
Otherwise, the test may validate a different workflow than the one deployed.
A Practical CI Strategy
A reasonable pipeline can use four stages:
Stage 1
YAML validation
↓
Stage 2
Fast unit tests
↓
Stage 3
Deterministic failure-injection integration tests
↓
Stage 4
Extended resilience tests
For pull requests, run the first three stages.
For scheduled or pre-release validation, run the full resilience suite.
This balances developer feedback speed with production confidence.
Recommended Failure-Injection Scenarios
For a new declarative AI workflow, start with these scenarios:
Valid input.
Missing input.
Invalid input.
Agent failure.
Empty agent response.
MCP tool failure.
MCP tool timeout.
HTTP 500 response.
HTTP timeout.
Authentication failure.
Authorization failure.
Malformed dependency response.
Retry succeeds.
Retry limit exceeded.
Human approval rejected.
Human approval timeout.
Workflow process termination.
Checkpoint restoration.
Duplicate operation after recovery.
Downstream dependency remains unavailable.
This list covers substantially more than the traditional "does the workflow return the expected answer?" test.
Frequently Asked Questions
What is failure injection in an AI workflow?
Failure injection deliberately introduces controlled failures into dependencies or workflow steps to verify that the workflow handles errors, retries, recovery, and termination correctly.
Should failure injection be used with real AI models?
For model-quality testing, real models are appropriate. For deterministic orchestration tests, controlled agent implementations are usually preferable because they allow tests to reproduce specific responses and failures.
Can declarative workflows be tested like normal .NET workflows?
Yes. Microsoft Agent Framework loads declarative YAML definitions into standard Workflow instances, allowing them to be executed and composed with other workflow infrastructure.
What should I test first?
Start with deterministic dependency failures:
Success
Failure once
Failure always
Timeout
Invalid response
Then move to checkpoint recovery, side-effect duplication, and multi-instance resilience.
Should every failure be retried?
No.
Transient infrastructure failures may be retryable, while authentication failures, authorization failures, invalid input, and explicit business rejection often should not be retried automatically.
Retry policy should classify failures rather than blindly retrying every error.
Is checkpointing enough to guarantee safe recovery?
No.
Checkpointing can preserve workflow state, but external side effects still require idempotency or another consistency strategy. A workflow can recover correctly while an external operation is accidentally repeated.
Conclusion
Declarative workflows move orchestration logic into a format that is easier to review, version, and change. Microsoft Agent Framework's current .NET declarative workflow implementation supports structured actions, branching, integrations, human-in-the-loop scenarios, and checkpointing, making it suitable for complex multi-step AI applications.
But declarative syntax does not eliminate failure modes.
A production workflow must be tested against the conditions it will eventually encounter:
Agent failure
↓
Tool failure
↓
Timeout
↓
Retry
↓
Recovery
↓
Checkpoint
↓
Process restart
↓
Idempotent continuation
The most effective approach is to make those failures deterministic and repeatable.
Instead of waiting for an MCP server to become unavailable or an external API to time out in production, create controlled test dependencies that reproduce those conditions during CI.
The objective is not simply to prove that the workflow can fail.
It is to prove that it fails predictably, recovers safely, does not duplicate side effects, and produces a controlled outcome for the user.
That is the difference between testing an AI workflow's happy path and testing its production reliability.