Introduction
AI applications are harder to test than traditional applications.
With a normal API, the same input usually produces a predictable result. If a test sends an HTTP request and expects a specific status code, the test either passes or fails.
AI systems are different.
The same prompt can produce slightly different answers. An agent may choose different tools. A retrieval system may return different documents. A tool call can succeed while the final answer is still wrong.
That makes traditional unit tests useful, but not enough.
For an AI application, we also need to understand what happened during the request.
Which model was called? Which tools were used? What documents were retrieved? How many tokens were consumed? How long did each step take? Did the agent call a tool it should not have used?
This is where AI traces become valuable.
Instead of evaluating only the final answer, we can evaluate the complete execution trace and use those evaluations as a CI/CD quality gate.
The basic idea is:
Code Change
↓
AI Test Cases
↓
Run Application
↓
Capture Trace
↓
Evaluate Trace
↓
Pass / Fail
↓
Deploy
This approach gives development teams a way to catch AI regressions before they reach production.
What Is an AI Trace?
A trace represents the execution of an AI request.
For a simple application, it might look like:
User Request
↓
LLM
↓
Response
An agentic application can be much more complicated:
User Request
↓
Agent
↓
Planner
↓
Tool: Search
↓
Retriever
↓
LLM
↓
Tool: Database
↓
LLM
↓
Final Response
A trace records these steps.
A simplified trace might contain:
{
"traceId": "trace-123",
"operation": "customer-support",
"model": "local-model",
"steps": [
{
"type": "llm",
"durationMs": 420
},
{
"type": "tool",
"name": "search_customer",
"durationMs": 85
},
{
"type": "llm",
"durationMs": 510
}
]
}
The exact format can vary, but the concept is the same.
The trace tells us how the answer was produced, not just what the answer was.
Why Final-Answer Testing Is Not Enough
Imagine an AI support agent produces this response:
Your order has been cancelled successfully.
A basic test might mark this as a successful response.
But the trace could reveal:
Agent
↓
search_order
↓
get_order_status
↓
cancel_order
Now we have a problem.
The user only asked:
What is the status of my order?
The agent performed a cancellation.
The final answer may look reasonable, but the execution was unsafe.
This is why trace-based evaluation is especially useful for agents.
What Should We Evaluate?
A useful AI evaluation pipeline can check several dimensions.
Final Answer Quality
Does the response answer the user's question?
Tool Selection
Did the agent use the correct tool?
Tool Parameters
Were the correct parameters passed?
Tool Count
Did the agent make unnecessary calls?
Retrieval Quality
Did the system retrieve relevant information?
Latency
Did the request stay within the expected response time?
Token Usage
Did the request consume an unexpected amount of tokens?
Safety
Did the agent attempt a restricted operation?
Error Handling
Did the application recover correctly from tool or model failures?
These checks can be converted into automated CI/CD gates.
Trace-Based Testing Architecture
A practical architecture looks like this:
CI/CD Pipeline
|
AI Test Suite
|
Application
|
Trace Collector
|
Evaluation Engine
|
Quality Gate
/ \
PASS FAIL
| |
Deploy Stop Build
The important part is that evaluation happens after execution but before deployment.
Start With Deterministic Checks
Not every evaluation needs another AI model.
Some of the most valuable checks can be deterministic.
For example:
if (trace.ToolCalls.Count > 5)
{
return EvaluationResult.Fail(
"Too many tool calls.");
}
Or:
if (trace.Tools.Any(t => t.Name == "delete_customer"))
{
return EvaluationResult.Fail(
"Restricted tool was called.");
}
These tests are fast and predictable.
Use deterministic checks wherever possible.
Example Trace Model in C#
A simple internal trace model could look like this:
public sealed record AiTrace(
string TraceId,
string Operation,
TimeSpan Duration,
IReadOnlyList<AiStep> Steps);
public sealed record AiStep(
string Type,
string Name,
TimeSpan Duration);
Now the evaluator can inspect the execution:
public sealed class TraceEvaluator
{
public void Evaluate(AiTrace trace)
{
if (trace.Duration > TimeSpan.FromSeconds(5))
{
throw new InvalidOperationException(
"AI latency exceeded the CI threshold.");
}
}
}
The actual evaluation system can become more sophisticated as the application grows.
Build Evaluation Rules
Instead of having one large test, create independent rules.
For example:
Trace
├── Latency Rule
├── Tool Rule
├── Token Rule
├── Safety Rule
├── Retrieval Rule
└── Answer Rule
Each rule can return:
PASS
WARN
FAIL
For example:
public sealed record EvaluationResult(
string Rule,
bool Passed,
string Message);
Then the CI pipeline can determine whether a deployment should continue.
Example: Tool Usage Gate
Suppose an order agent is allowed to use:
get_order
search_orders
create_support_ticket
It should not use:
delete_order
refund_payment
A trace evaluator can enforce this:
var allowedTools = new HashSet<string>
{
"get_order",
"search_orders",
"create_support_ticket"
};
foreach (var tool in trace.Tools)
{
if (!allowedTools.Contains(tool.Name))
{
throw new InvalidOperationException(
$"Unexpected tool: {tool.Name}");
}
}
This is much safer than checking only the final response.
Example: Tool Parameter Validation
Tool selection alone is not enough.
Suppose the agent correctly calls:
refund_payment
but sends:
{
"amount": 1000000
}
The tool itself may be legitimate, but the parameters are not.
A trace can capture the arguments, allowing a test to verify:
Refund amount <= allowed limit
Currency is valid
Payment belongs to test user
Approval requirement was satisfied
For sensitive tools, parameter-level testing is essential.
Example: Latency Gate
AI latency can change after a model, prompt, or retrieval implementation changes.
Suppose the baseline is:
P50 = 1.2 seconds
P95 = 2.4 seconds
A new version produces:
P50 = 1.4 seconds
P95 = 4.8 seconds
The average may still appear acceptable.
But P95 has doubled.
A CI gate can detect this:
if (metrics.P95.TotalSeconds > 3)
{
return EvaluationResult.Fail(
"P95 latency exceeded 3 seconds.");
}
This prevents a performance regression from quietly reaching production.
Compare Against a Baseline
Absolute thresholds are useful, but baseline comparison is often better.
For example:
Previous version:
P50 = 800 ms
Current version:
P50 = 1,120 ms
The regression is:
(1120 - 800) / 800
= 40%
A team could define:
Allowed regression: 15%
The CI pipeline would then fail the build.
This works well when AI workloads change frequently.
Token Usage Can Be a Quality Gate
AI applications can become unexpectedly expensive or slow because of prompt growth.
Imagine:
Version A
Input tokens: 1,500
Output tokens: 300
Version B
Input tokens: 8,000
Output tokens: 450
The final response might look identical.
But the second implementation is doing significantly more work.
A trace can expose this regression.
Join the conversation! Your thoughts help the community grow.