Comparing Coding Models on Amazon Bedrock With the Same Test Set

{"id":"CSHARP-001","category":"CSharp","prompt":"Create a C# email validation method. Handle null, empty input, and invalid email formats. Include unit test examples.","expected":"The implementation must reject null and empty input, validate the supplied address, and include meaningful tests."}
{"id":"CSHARP-002","category":"CSharp","prompt":"Implement pagination for an ASP.NET Core product API using Entity Framework Core. Return the requested page and total record count.","expected":"The implementation must use database-side pagination, validate page values, and return the total record count."}
{"id":"CSHARP-003","category":"Async","prompt":"Write an asynchronous C# service method that retrieves customer information from a database. Support cancellation and handle missing records.","expected":"The implementation must use async database operations, pass the cancellation token, and handle missing records."}

You are an experienced software engineer.

Generate production-oriented code based on the supplied requirement.

Follow these instructions:
- Use the requested programming language.
- Follow established coding conventions.
- Handle invalid input.
- Avoid unnecessary dependencies.
- Include relevant error handling.
- Explain important implementation decisions.
- Do not invent APIs or methods.
- Mention assumptions when information is missing.

public interface ICodingModel
{
    string ModelName { get; }

    Task<ModelResult> GenerateAsync(
        string prompt,
        CancellationToken cancellationToken);
}

public class ModelResult
{
    public string ModelName { get; set; } = string.Empty;

    public string GeneratedCode { get; set; } = string.Empty;

    public long ResponseTimeMs { get; set; }

    public int? InputTokens { get; set; }

    public int? OutputTokens { get; set; }
}

Each model adapter implements ICodingModel.

For example, one adapter can use an OpenAI-compatible Bedrock endpoint, while another can use the Converse API.

This approach keeps model-specific authentication, request construction, response parsing, and token extraction outside the evaluation logic.

It also makes it easier to add another model without changing the test runner.

Run the Same Test Cases Against Every Model

Now create a test runner that loads the dataset and sends each prompt to every configured model.

The following example shows the evaluation flow:

using System.Diagnostics;

public class CodingModelEvaluator
{
    private readonly IReadOnlyList<ICodingModel> _models;

    public CodingModelEvaluator(
        IReadOnlyList<ICodingModel> models)
    {
        _models = models;
    }

    public async Task EvaluateAsync(
        IEnumerable<string> testCases,
        CancellationToken cancellationToken)
    {
        foreach (var testCase in testCases)
        {
            Console.WriteLine($"Test: {testCase}");

            foreach (var model in _models)
            {
                var stopwatch = Stopwatch.StartNew();

                var result = await model.GenerateAsync(
                    testCase,
                    cancellationToken);

                stopwatch.Stop();

                result.ResponseTimeMs =
                    stopwatch.ElapsedMilliseconds;

                Console.WriteLine(
                    $"Model: {model.ModelName}");

                Console.WriteLine(
                    $"Response time: {result.ResponseTimeMs} ms");

                Console.WriteLine(
                    result.GeneratedCode);
            }
        }
    }
}

This example demonstrates the comparison flow. A complete implementation should deserialize JSONL records into typed test cases, store each result, and capture errors separately.

For repeatable evaluation, run every test case multiple times when the model's generation behavior is nondeterministic.

Also, avoid running all models concurrently without considering their service quotas and throttling limits. Sequential execution or controlled concurrency can make the test environment easier to manage.

Measure More Than Response Time

Response time is useful, but it should not be the only metric.

A model that generates code in two seconds but produces incorrect SQL is not necessarily better than a model that takes five seconds and generates a working implementation.

Use a combination of technical and operational measurements.

Metric	What it measures
Compilation success	Whether generated code builds
Unit test pass rate	Whether the implementation satisfies expected behavior
Functional correctness	Whether the code meets the requirements
Code quality	Readability, maintainability, and design
Security	Whether the implementation introduces vulnerabilities
Response latency	Time taken to generate the response
Input tokens	Prompt-processing usage
Output tokens	Generated-response usage
Estimated cost	Inference cost for the evaluation

For code generation, compilation and automated test results should carry significant weight.

For example, you can save generated C# code into an isolated temporary project and run:

dotnet build -c Release
dotnet test -c Release

Run generated code in a restricted environment. Do not execute arbitrary model-generated code directly on a production server or a developer's unrestricted machine.

The test environment should have limited filesystem access, no unnecessary network access, and appropriate execution time and resource limits.

Use Amazon Bedrock Model Evaluation

Amazon Bedrock also provides a managed Model Evaluation capability.

It supports programmatic evaluations, human-based evaluations, and evaluations that use another language model as a judge.

For coding tasks, LLM-as-a-Judge can help assess aspects that are difficult to measure using simple automated tests.

Some available metrics include:

Metric	Purpose
Correctness	Evaluates whether the response is correct
Completeness	Checks whether the response addresses the full requirement
Helpfulness	Evaluates usefulness and clarity
Following Instructions	Checks whether the model followed the supplied instructions
Coherence	Evaluates logical consistency
Harmfulness	Evaluates potentially harmful content

For example, two models may both generate code that compiles, but one may produce a more maintainable implementation.

A judge model can help assess that difference, although its scores should not replace compiler and unit-test results.

Amazon Bedrock supports custom prompt datasets stored in Amazon S3 using JSON Lines format.

There is also an option to provide your own model responses instead of asking Bedrock to invoke the models during evaluation.

An important limitation is that a single evaluation dataset using supplied model responses supports one or two model responses per prompt. For larger comparisons, split the evaluation into suitable jobs or use your own evaluation harness to collect the results.

Human review is also valuable when evaluating architecture, readability, and maintainability.

Calculate a Practical Model Score

Once the results are available, you can create a scoring system that reflects your application's priorities.

For example:

Evaluation area	Weight
Compilation success	15%
Unit test pass rate	30%
Functional correctness	25%
Code quality	15%
Security	10%
Response time	5%

These weights are illustrative. A production team should define them according to its actual requirements.

For example, a coding assistant that generates database queries may place more importance on correctness and security than response speed.

A model that performs well across most categories may be a better choice than one that performs exceptionally well in only one area.

Also, keep the raw results. A single overall score can hide important differences between models.

Avoid Common Evaluation Mistakes

Even a well-designed test dataset can produce misleading results if the evaluation process is inconsistent.

Some mistakes to avoid include:

• Giving different prompts to different models.
• Comparing models with different output limits without documenting the difference.
• Evaluating only one or two programming tasks.
• Measuring response time without checking correctness.
• Ignoring failed requests and throttling.
• Comparing model prices without including input and output token usage.
• Running generated code without isolation.
• Treating an LLM judge score as absolute truth.
• Selecting a model using benchmark results that do not represent your application.

Another important consideration is prompt leakage. If the evaluation dataset is used repeatedly to tune prompts, the results may no longer represent how the model performs on new tasks.

Maintain a separate set of unseen test cases for final validation.

Summary

Comparing coding models on Amazon Bedrock requires a consistent evaluation process rather than relying on model descriptions or benchmark rankings.

Start with a common test dataset, use the same instructions, and record the model configuration for every run. Evaluate generated code through compilation, automated tests, security checks, and human review.

Amazon Bedrock Model Evaluation can help assess correctness, completeness, helpfulness, and other response qualities. A custom C# evaluation harness can complement it by measuring programming-specific results such as build success, test pass rate, response latency, and token usage.

The best coding model is not necessarily the fastest or the one with the largest context window. It is the model that consistently meets your application's requirements at an acceptable cost and level of reliability.