Testing an AI agent API is different from testing a traditional ASP.NET Core API.
A normal API usually receives a known request, executes predictable application logic, and returns a response that can be compared with an expected result. An AI agent may call a language model, choose different tools, generate different wording, retry an operation, or take a different path based on the model's response.
That does not mean AI agent APIs cannot be tested reliably. It means the test strategy needs to separate deterministic application behavior from non-deterministic model behavior.
One practical approach is to replace the real AI model with deterministic mocks during automated tests and record representative conversations that can be replayed later. This allows developers to test the workflow, tool execution, validation, authorization, error handling, and persistence without depending on a live model.
This article explains how to build that testing approach in an ASP.NET Core application.
Why Traditional API Tests Are Not Enough for AI Agents
Consider an API endpoint:
POST /api/agent/chatThe user sends:
{
"message": "Check my order and tell me whether it has shipped."
}The application may perform this workflow:
User Request
|
v
ASP.NET Core API
|
v
AI Model
|
v
Tool Selection
|
v
Order API
|
v
AI Model
|
v
Final ResponseThere are several points where the result can vary.
The model may choose a different tool.
The model may produce different wording.
The model may decide that additional information is required.
A tool may fail.
A retry may happen.
The model provider may change its response format.
If the automated test expects the exact final text, the test can become unreliable.
For example, this test is fragile:
Assert.Equal(
"Your order has been shipped and is currently in transit.",
response.Message);A perfectly valid model response could instead be:
Your order has shipped and is currently in transit.The business result is the same, but the string is different.
The better approach is to test the behavior that matters.
Separate Model Behavior From Application Behavior
A useful testing architecture separates the AI model behind an interface.
public interface IAgentModel
{
Task<AgentModelResponse> CompleteAsync(
AgentModelRequest request,
CancellationToken cancellationToken);
}The production application can use a real model implementation.
IAgentModel
|
+-- ProductionAgentModel
|
+-- DeterministicMockAgentModel
|
+-- ReplayAgentModelThe application workflow does not need to know which implementation is being used.
This is the foundation for reliable testing.
Create a Deterministic Model Response
A test model should return a predefined response.
For example:
public sealed class DeterministicAgentModel : IAgentModel
{
private readonly AgentModelResponse _response;
public DeterministicAgentModel(AgentModelResponse response)
{
_response = response;
}
public Task<AgentModelResponse> CompleteAsync(
AgentModelRequest request,
CancellationToken cancellationToken)
{
return Task.FromResult(_response);
}
}The test controls the model response completely.
Suppose the application expects the model to request an order lookup:
var modelResponse = new AgentModelResponse
{
ToolCall = new AgentToolCall
{
Name = "get_order",
Arguments = new Dictionary<string, object>
{
["orderId"] = "48291"
}
}
};The test can now verify what the application does when the model makes that specific tool request.
Test the Agent Workflow Instead of the Model
The goal of the application test should not be to determine whether the language model is intelligent.
The goal should be to determine whether the application correctly handles the model's output.
For example:
Model requests get_order
|
v
Application validates tool call
|
v
Application calls OrderService
|
v
Application sends result back to model
|
v
Application returns responseThe model is simply a controlled dependency in this test.
This makes failures much easier to understand.
If the test fails because the order service was not called, the problem belongs to application code.
Use ASP.NET Core Integration Tests
For an agent API, integration tests are often more useful than testing every class independently.
ASP.NET Core's WebApplicationFactory can start the application using a test configuration.
A simplified test setup can look like this:
public class AgentApiTests
: IClassFixture<WebApplicationFactory<Program>>
{
private readonly WebApplicationFactory<Program> _factory;
public AgentApiTests(WebApplicationFactory<Program> factory)
{
_factory = factory;
}
[Fact]
public async Task Agent_Should_Return_Order_Status()
{
var client = _factory.CreateClient();
var response = await client.PostAsJsonAsync(
"/api/agent/chat",
new
{
message = "Check order 48291"
});
response.EnsureSuccessStatusCode();
var result =
await response.Content.ReadFromJsonAsync<AgentResponse>();
Assert.NotNull(result);
}
}The important part is replacing the real AI dependency before the request reaches the application.
Replace the AI Service During Testing
If the application registers the model through dependency injection:
builder.Services.AddScoped<IAgentModel, ProductionAgentModel>();the test application can replace it.
Conceptually:
builder.Services.AddScoped<IAgentModel, DeterministicAgentModel>();A more flexible test factory can accept the model implementation as a dependency.
public class AgentTestFactory
: WebApplicationFactory<Program>
{
protected override void ConfigureWebHost(
IWebHostBuilder builder)
{
builder.ConfigureServices(services =>
{
services.RemoveAll<IAgentModel>();
services.AddScoped<IAgentModel>(
_ => CreateTestModel());
});
}
private IAgentModel CreateTestModel()
{
return new DeterministicAgentModel(
new AgentModelResponse
{
Message = "Order is currently in transit."
});
}
}Now the API test runs against the actual ASP.NET Core application pipeline while controlling the AI dependency.
That is a useful balance between unit testing and full end-to-end testing.
Test Tool Calls Separately
Agent applications often depend heavily on tools.
For example:
get_order
check_payment
check_shipping
create_ticket
send_emailEach tool should have a clear contract.
public interface IOrderTool
{
Task<OrderResult> GetOrderAsync(
string orderId,
CancellationToken cancellationToken);
}The test can replace the real implementation with a controlled fake.
public sealed class FakeOrderTool : IOrderTool
{
public Task<OrderResult> GetOrderAsync(
string orderId,
CancellationToken cancellationToken)
{
return Task.FromResult(
new OrderResult
{
OrderId = orderId,
Status = "Shipped"
});
}
}Now the test can verify whether the agent correctly handles the tool result.
This is much more reliable than calling the real order service from every automated test.
Validate Tool Arguments
One of the most important agent testing scenarios is invalid tool input.
Suppose the model produces:
{
"tool": "get_order",
"arguments": {
"orderId": ""
}
}The application should not blindly execute the tool.
Test this behavior directly.
[Fact]
public async Task Empty_Order_Id_Should_Not_Call_Order_Tool()
{
var result = await agent.ExecuteAsync(
new AgentRequest
{
Message = "Check my order"
},
CancellationToken.None);
Assert.Equal(
AgentExecutionStatus.InvalidToolRequest,
result.Status);
}This type of validation is especially important because the model is an untrusted input source from an application security perspective.
The application should validate tool names, argument types, required fields, authorization, and allowed operations before executing anything.
Replayable Conversations
A single test case is not enough for complex agents.
A better approach is to store representative conversations as test fixtures.
For example:
{
"name": "order-shipping-status",
"messages": [
{
"role": "user",
"content": "Check order 48291"
},
{
"role": "assistant",
"tool": "get_order",
"arguments": {
"orderId": "48291"
}
},
{
"role": "tool",
"name": "get_order",
"content": {
"status": "Shipped"
}
},
{
"role": "assistant",
"content": "Your order has shipped."
}
]
}The exact fixture format is less important than keeping the conversation reproducible.
The replay test loads the conversation and feeds the same sequence into the application.
Recorded Conversation
|
v
Replay Test
|
v
Agent Workflow
|
v
Expected Tool Calls
Expected State
Expected ResultThis allows developers to reproduce a production bug without repeatedly calling the live model.
Replay the Important Parts, Not Just the Final Message
A replay system becomes much more useful when it captures tool interactions.
Instead of recording only:
User -> Assistantrecord:
User
|
v
Model
|
v
Tool Call
|
v
Tool Result
|
v
Model
|
v
Final ResponseNow a test can verify that the agent followed the correct execution path.
For example:
Assert.Collection(
execution.ToolCalls,
call =>
{
Assert.Equal("get_order", call.Name);
Assert.Equal("48291", call.Arguments["orderId"]);
});
Join the conversation! Your thoughts help the community grow.