How to Measure Whether a Code Change Makes Your Application Slower

A code change can improve functionality while making an application slower. A new database query, an additional API call, a more complex calculation, or an unnecessary object allocation may increase response times without causing any obvious errors.

These performance regressions are not always noticeable during development. An operation that takes a few extra milliseconds in a local environment may become a serious problem when thousands of requests arrive simultaneously.

The challenge is determining whether a code change actually caused the slowdown. Comparing two response times is not enough because performance also depends on hardware, network conditions, database activity, caching, background jobs, and concurrent traffic.

A reliable approach requires a repeatable baseline, a controlled test environment, consistent workloads, and measurements that reveal more than average response time.

In this article, we will explore how to establish a performance baseline, compare application versions, measure latency and throughput, identify regressions, and use profiling tools to understand why a change makes an application slower.

Why Performance Regressions Are Difficult to Detect

Consider an ASP.NET Core API that returns order details.

Before a code change, the endpoint responds in approximately 120 milliseconds under a particular test workload. After the change, the response time increases to 155 milliseconds.

At first glance, the change appears to have made the endpoint slower by 35 milliseconds.

However, the comparison is meaningful only if both measurements were collected under similar conditions. The second test might have experienced a cold cache, a busy database, a different request distribution, or additional background activity.

A performance regression becomes more credible when the result is repeatable and the test conditions are controlled.

Common causes of performance regressions include:

Additional database queries.
Unnecessary object allocations.
Synchronous operations that block request processing.
Inefficient loops or data transformations.
Larger API responses.
Repeated serialization or deserialization.
Cache misses caused by a code change.
Lock contention between concurrent requests.
Increased logging or telemetry overhead.
Changes to database query plans.

The objective is to distinguish the effect of the code change from normal variation in the system.

Establish a Performance Baseline

Before measuring a new implementation, record how the current version performs.

A baseline provides a reference for deciding whether the new version is faster, slower, or statistically indistinguishable under the tested conditions.

Choose a Representative Workload

Start by identifying the operations that matter most to users.

For a web application, these might include:

Loading a frequently used page.
Retrieving customer or order information.
Creating a transaction.
Searching a large collection.
Uploading a file.
Processing a background job.

Select realistic inputs and request patterns. A benchmark that repeatedly requests the same cached record may not represent an endpoint that normally retrieves different records.

For example, a database-backed endpoint should be tested with representative data volumes and query patterns. Testing against an almost empty development database may hide performance problems that appear with production-sized data.

Record the Environment

Document the environment used for each test.

Important details include:

Application version or Git commit.
Runtime and framework version.
CPU and memory limits.
Database version and configuration.
Database size and data distribution.
Cache configuration.
Test concurrency.
Request rate and duration.
Background processing activity.

Keep these details consistent between the baseline and the candidate version whenever possible.

If the application is deployed in containers, ensure that both versions receive comparable CPU and memory resources. Otherwise, resource differences may be mistaken for code-level regressions.

Warm Up the Application

The first requests may behave differently from later requests because of runtime compilation, dependency initialization, connection establishment, and cache population.

For a .NET application, initial requests may include just-in-time compilation and initialization costs.

Run a warm-up phase before collecting steady-state measurements if your objective is to compare normal production performance.

However, do not discard startup measurements when startup time is itself important. Cold-start performance should be measured separately from steady-state performance.

Measure the Right Performance Metrics

Response time is important, but it does not provide a complete picture of application performance.

Response Time and Percentiles

Response time measures how long an operation takes from the perspective of the measurement point.

Suppose a test produces these results:

Metric	Before change	After change
Average response time	120 ms	145 ms
p50 latency	95 ms	105 ms
p95 latency	210 ms	280 ms
p99 latency	420 ms	650 ms

The average increased by approximately 21%, but the p99 latency increased by approximately 55%.

The p50 represents the median response time. The p95 represents the response time at or below which approximately 95% of measured requests completed. The p99 describes the corresponding 99th-percentile threshold.

Percentiles help reveal slow requests that an average can hide. However, they do not explain the cause of the slowdown, and high-percentile estimates can be unstable when the sample size is too small.

Use enough requests to measure the percentiles meaningfully and compare them under equivalent workloads.

Throughput

Throughput measures how much work the application completes over a period.

For an HTTP service, it is commonly expressed as requests per second.

An application might have similar response times at low traffic but process fewer requests per second after a code change. This can indicate reduced capacity or increased resource consumption.

Conversely, a test that sends more requests to the candidate version may increase its response time even if the implementation has not become less efficient.

Keep the offered load consistent when comparing versions.

Resource Consumption

Measure resource use alongside latency and throughput.

Useful measurements include:

CPU utilization.
Memory usage.
Garbage collection frequency and pause duration.
Allocation rate.
Database query count and execution time.
Network traffic.
Lock contention.
Thread pool activity.

These metrics help distinguish different causes.

For example, increased CPU consumption may indicate additional computation. Higher allocation rates may suggest unnecessary temporary objects. Increased database time may indicate extra queries or a less efficient query plan.

Compare Two Versions Under the Same Conditions

A useful performance test compares the baseline version with the candidate version using the same workload and environment.

For example, suppose you are evaluating a change to an order-processing endpoint.

The test should use the same:

Dataset and data distribution.
Request payloads.
Concurrency level.
Application resource limits.
Database configuration.
Cache conditions.
Measurement duration.

Run each version multiple times and record the results.

Avoid running competing benchmarks on the same machine if they interfere with CPU, disk, or network resources. When possible, use isolated environments or alternate test order to reduce systematic bias.

Calculate the Performance Difference

A simple percentage change can help summarize the results.

For latency, calculate:

Percentage change = ((New latency - Baseline latency) / Baseline latency) × 100

For example, if p95 latency increases from 200 milliseconds to 240 milliseconds:

Percentage change = ((240 - 200) / 200) × 100 = 20%

The candidate version has a 20% higher p95 latency in this test.

A positive latency change means slower responses, while a negative change means faster responses.

This calculation is useful for reporting, but it does not prove that the code change caused the difference. Repeatability, sample size, and environmental consistency still matter.

Create a Simple Benchmark in C#

For a small isolated operation, a benchmark framework can help measure execution time and allocations.

Consider a method that transforms a collection of orders.

public sealed class Order
{
    public int Id { get; init; }
    public decimal Total { get; init; }
}

public sealed class OrderService
{
    public decimal CalculateTotal(
        IEnumerable<Order> orders)
    {
        decimal total = 0;

        foreach (var order in orders)
        {
            total += order.Total;
        }

        return total;
    }
}

You can benchmark the method using BenchmarkDotNet.

First, add the BenchmarkDotNet package to a benchmark project. Then create a benchmark class:

using BenchmarkDotNet.Attributes;

[MemoryDiagnoser]
public class OrderBenchmarks
{
    private readonly OrderService _service = new();

    private readonly List<Order> _orders =
        Enumerable.Range(1, 10000)
            .Select(id => new Order
            {
                Id = id,
                Total = id * 1.25m
            })
            .ToList();

    [Benchmark]
    public decimal CalculateTotal()
    {
        return _service.CalculateTotal(_orders);
    }
}

Configure the benchmark project to run the benchmark class using BenchmarkDotNet's standard runner.

The MemoryDiagnoser attribute adds allocation measurements to the benchmark results.

BenchmarkDotNet performs warm-up and repeated measurements to reduce some common sources of measurement error. It can also report statistical information about the results.

However, a microbenchmark measures a small operation in isolation. It does not automatically predict end-to-end API latency, database performance, or behavior under production concurrency.

Use microbenchmarks to investigate focused code paths, and use load tests to evaluate complete application behavior.

Test API Performance with a Repeatable Load

For an HTTP API, use a load-testing tool to send requests at a controlled rate and collect latency, throughput, and error measurements.

For example, a k6 script might look like this:

import http from 'k6/http';
import { check } from 'k6';

export const options = {
  vus: 10,
  duration: '60s',
};

export default function () {
  const response = http.get(
    'https://staging.example.com/api/orders');

  check(response, {
    'status is 200': (r) => r.status === 200,
  });
}

This example runs ten virtual users for one minute. It checks whether requests return HTTP 200 responses.

The URL is illustrative and should be replaced with an appropriate test endpoint.

To compare application versions, run the same script against equivalent deployments with the same data and resource limits.

The example uses a fixed number of virtual users. It does not guarantee a fixed requests-per-second rate because the request rate depends on response time and the time spent between iterations.

If the test requires a specific request rate, configure an appropriate rate-based executor and ensure the load generator itself has enough capacity.

Monitor response time, error rate, throughput, and application resource usage throughout the test.

Do not run aggressive load tests against production without an approved plan, explicit traffic limits, and safeguards for dependent services.

Identify the Cause with Profiling

Once a repeatable regression has been confirmed, profiling helps explain what changed.

A performance comparison tells you that an operation became slower. A profiler helps identify where execution time or resources are being consumed.

CPU Profiling

A CPU profile shows which methods consume processing time.

If a new validation method appears near the top of the profile, inspect whether it performs unnecessary calculations or repeats work for every request.

Compare profiles from the baseline and candidate versions using equivalent workloads. A method that consumes a large percentage of CPU in one profile may not actually be responsible for the regression if the overall workload differs.

Allocation Profiling

Allocation profiling identifies where the application creates objects and how much memory it allocates.

Excessive temporary allocations can increase garbage collection activity, especially under high request rates.

For example, a code change that repeatedly converts a large collection to a new list may increase memory pressure even when the final result remains the same.

Measure both allocation rate and garbage collection behavior before deciding whether an optimization is necessary.

Database Profiling

For database-backed applications, compare query counts, query durations, and execution plans.

A change that introduces one additional query per request may appear harmless during a small test. Under high traffic, the additional database work can become a major bottleneck.

Check whether the candidate version performs repeated queries, retrieves unnecessary columns, or prevents the database from using an effective index.

The database may also be the source of the slowdown even when the application code itself has not become more expensive.

Establish Performance Regression Thresholds

A team can define thresholds that determine when a change should trigger investigation or block a release.

For example:

Metric	Example regression threshold	Response
p95 response time	More than 10% higher	Investigate and compare repeated runs.
p99 response time	More than 15% higher	Review tail latency and outliers.
Error rate	Any increase beyond the agreed limit	Investigate before release.
CPU per request	More than 10% higher	Review CPU profiles and additional processing.
Allocations per request	Significant unexplained increase	Inspect allocation profiles and garbage collection.
Throughput	More than 10% lower at the same load	Investigate capacity and resource use.

These are example thresholds, not universal standards. Select limits according to application requirements, measurement variability, and the cost of a regression.

A small internal tool may tolerate a slight latency increase if the change substantially improves maintainability. A payment API or user-facing service with strict latency objectives may need tighter limits.

Avoid blocking every build because of tiny measurement differences. Use repeatable tests, appropriate sample sizes, and a defined policy for handling inconclusive results.

Common Mistakes to Avoid

Comparing Different Environments

Different CPU limits, database sizes, runtime versions, and cache states can produce misleading results.

Keep the environment consistent or explicitly account for the differences.

Relying Only on Average Latency

Average response time can hide slow requests. Compare p50, p95, and p99 alongside throughput and error rates.

Testing with Too Little Data

A small number of requests may not reveal tail latency, memory pressure, or concurrency-related bottlenecks.

Use a workload large enough to represent the behavior you need to measure.

Optimizing Before Identifying the Bottleneck

Changing code based on intuition can add complexity without improving performance.

Confirm the regression, profile the application, and target the operation responsible for the measured difference.

Ignoring Measurement Noise

Operating system activity, background jobs, network variation, and shared resources can affect results.

Repeat tests and investigate whether the observed difference is consistent before drawing conclusions.

Best Practices for Reliable Performance Testing

Establish a baseline before making performance-sensitive changes.
Use realistic workloads and representative data.
Record application versions and test environment details.
Separate cold-start measurements from steady-state measurements.
Compare latency percentiles, throughput, errors, and resource consumption.
Repeat tests and account for measurement variability.
Use microbenchmarks for isolated methods and load tests for complete services.
Profile the application before attempting complex optimizations.
Define regression thresholds based on application service-level objectives.
Include performance checks in CI/CD for critical code paths.
Investigate regressions before accepting or rejecting a change.
Retest after optimization to confirm that the change produced a measurable improvement.

Advantages and Limitations of Performance Regression Testing

Aspect	Advantage	Limitation
Early detection	Identifies slowdowns before they reach production.	Test environments may not reproduce every production condition.
Repeatability	Makes comparisons between versions easier.	Results still depend on workload and measurement quality.
Resource analysis	Helps identify CPU, memory, and database bottlenecks.	Profiling can introduce some overhead.
Release confidence	Provides measurable evidence for performance-sensitive changes.	Passing tests cannot guarantee that every workload will perform well.
Automated checks	Helps prevent recurring regressions.	Strict thresholds can produce false alarms if variability is not considered.

Summary

Measuring whether a code change makes an application slower requires more than comparing two response times. Establish a baseline, use a representative workload, keep the test environment consistent, and measure latency percentiles, throughput, errors, and resource consumption.

Use microbenchmarks to examine isolated operations and load tests to evaluate complete application behavior. When a regression is repeatable, use CPU, allocation, or database profiling to identify the underlying cause.

Finally, establish performance thresholds that reflect the application's requirements and automate checks for important workloads. This approach helps developers make performance decisions based on evidence rather than assumptions and reduces the chance that a seemingly small code change will create a production bottleneck.