Testing an AI App When Its Web Search Results Keep Changing

Testing an AI application is already different from testing a traditional application. A method that receives the same input does not always produce exactly the same response because language models can generate different answers.

The challenge becomes more noticeable when web search is involved.

Imagine building an application that answers questions about the latest .NET releases. Today, a search might return three official documentation pages. Tomorrow, the search index may return different pages, a new announcement may appear, or an existing page may have changed.

Even when the user submits exactly the same question, the generated answer can be different.

This creates a problem for developers who rely on traditional automated tests. A test that compares the complete AI response with a previously saved string may pass today and fail tomorrow without any application code changing.

The solution is not to stop testing AI applications. It is to change what we test.

In this article, we will explore how to test a search-enabled AI application using Amazon Bedrock Web Search, C#, and ASP.NET Core. We will focus on repeatable tests, controlled test data, citation validation, and production monitoring.

Why Web Search Makes AI Testing More Difficult

A traditional API usually produces predictable output when its input and dependencies remain unchanged.

For example, a calculation endpoint that receives two numbers should return the same result every time.

A search-enabled AI application has more variables:

• Search results can change over time.
• Web pages can be updated or removed.
• Search ranking can change.
• The model may formulate different search queries.
• The model may interpret retrieved information differently.
• Generated answers can vary even when the underlying information is similar.

Amazon Bedrock Web Search adds another important detail: the model decides whether it needs current information and can perform one or more searches during a request.

The search results come from an Amazon-maintained web index. If the initial results are insufficient, the model can reformulate its query and search again.

This means the same user question does not guarantee the same search observations or final response.

A useful testing strategy must separate application behavior from external search behavior.

Identify What Should Be Tested

Before writing tests, divide the application into separate responsibilities.

For a typical AI application, the architecture might look like this:

User
  |
  v
ASP.NET Core API
  |
  v
AI Application Service
  |
  v
Amazon Bedrock Responses API
  |
  v
Web Search Tool
  |
  v
Generated Answer and Citations
  |
  v
Response Processing
  |
  v
User

Each component has different testing requirements.

The most important design decision is to avoid making every application test depend on a live web search.

If the search provider returns different information, your controller validation test should not fail. The controller should be tested independently from the changing search results.

Create an Abstraction Around the AI Service

A good starting point is to define an interface for the AI functionality.

This allows the application to use Amazon Bedrock in production and a controlled implementation during automated tests.

Create an interface:

public interface IAIAnswerService
{
    Task<AIAnswerResponse> AskAsync(
        string question,
        CancellationToken cancellationToken);
}

Define the response models:

public class AIAnswerResponse
{
    public string Answer { get; set; } = string.Empty;

    public List<SourceReference> Sources { get; set; } = new();
}

public class SourceReference
{
    public string Title { get; set; } = string.Empty;

    public string Url { get; set; } = string.Empty;
}

The production implementation communicates with Amazon Bedrock. The test implementation returns predefined responses.

Register the production implementation in Program.cs:

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddHttpClient<BedrockSearchService>();

builder.Services.AddScoped<IAIAnswerService, BedrockSearchService>();

builder.Services.AddControllers();

var app = builder.Build();

app.MapControllers();

app.Run();

The BedrockSearchService should implement IAIAnswerService.

This separation makes it possible to test the API without sending real requests to Amazon Bedrock.

Use a Fake Response for Repeatable Tests

A fake service returns controlled data instead of calling the actual model.

For example:

public class FakeAIAnswerService : IAIAnswerService
{
    public Task<AIAnswerResponse> AskAsync(
        string question,
        CancellationToken cancellationToken)
    {
        var response = new AIAnswerResponse
        {
            Answer = "The latest documentation describes " +
                     "several platform improvements.",

            Sources =
            {
                new SourceReference
                {
                    Title = "Official Documentation",
                    Url = "https://example.com/documentation"
                }
            }
        };

        return Task.FromResult(response);
    }
}

The source URL is illustrative test data, not a real search result.

Now an API test can use this fake service and verify the application's behavior.

For example, with xUnit:

using Xunit;

public class AIControllerTests
{
    [Fact]
    public async Task AskAsync_ReturnsAnswerAndSources()
    {
        var service = new FakeAIAnswerService();

        var result = await service.AskAsync(
            "What changed in .NET?",
            CancellationToken.None);

        Assert.NotNull(result);

        Assert.NotEmpty(result.Answer);

        Assert.NotEmpty(result.Sources);

        Assert.Equal(
            "Official Documentation",
            result.Sources[0].Title);
    }
}

This test is repeatable because it does not depend on search ranking, page availability, or model-generated wording.

The objective is to verify that the application correctly handles an AI response, not to test the live search engine every time the test suite runs.

Test the Bedrock Response Parser Separately

Another important area is response parsing.

Amazon Bedrock returns generated text and citation annotations through the Responses API. The application must extract these values and convert them into its own response model.

A parser should be tested using saved response fixtures.

For example, create a test JSON file:

{
  "output": [
    {
      "type": "message",
      "content": [
        {
          "type": "output_text",
          "text": "The platform includes new improvements.",
          "annotations": [
            {
              "type": "url_citation",
              "title": "Release Documentation",
              "url": "https://example.com/release",
              "start_index": 0,
              "end_index": 45
            }
          ]
        }
      ]
    }
  ]
}