Why Average API Response Time Can Be Misleading: Using p95 and p99

When an API becomes slow, developers often start by checking its average response time. If the average is 150 milliseconds, the API may appear to be performing well. However, some users might be waiting two seconds or longer for the same operation to finish.

This happens because an average does not show how response times are distributed across individual requests. A small number of very slow requests can affect the user experience without making the average look particularly concerning. In other cases, a large number of fast requests can hide serious delays affecting a smaller group of users.

To understand these problems, engineering teams use percentile metrics such as p50, p95, and p99. These measurements help reveal what typical requests experience and how slow the requests near the end of the distribution can become.

In this article, we'll examine why average API latency can be misleading, how percentiles work, how to calculate them, and how to use them when monitoring and troubleshooting production APIs.

Why Average Response Time Does Not Tell the Whole Story

Suppose an API receives ten requests. Nine finish in 100 milliseconds, while one takes 2,000 milliseconds.

The average response time is calculated by adding all response times and dividing the total by the number of requests.

Response times:
100, 100, 100, 100, 100,
100, 100, 100, 100, 2000

Total response time = 2900 milliseconds
Number of requests = 10

Average = 2900 / 10
        = 290 milliseconds

An average of 290 milliseconds may look acceptable for some applications. However, one request took two seconds, which could be noticeable during a login, checkout, search, or payment workflow.

The average combines every request into a single number. It does not tell you whether all requests took roughly the same amount of time or whether some users experienced much longer delays.

Consider two APIs that both report an average response time of 200 milliseconds.

The first API might return nearly every response in approximately 200 milliseconds. The second might return most responses in 50 milliseconds while a smaller group takes several seconds.

Both APIs have the same average, but their performance characteristics are very different.

This is why latency monitoring should use more than one statistic.

Understand p50, p95, and p99

Percentiles describe the position of a value within a distribution of measurements.

When response times are sorted from fastest to slowest, each percentile indicates a point in that ordered distribution.

The most useful percentiles for API monitoring are often p50, p95, and p99.

p50: The median response time

The p50 latency is the 50th percentile, also called the median.

Approximately half of the measured requests complete at or below this value, while the other half complete at or above it, subject to the percentile calculation method and ties.

If an API has a p50 latency of 80 milliseconds, a typical request is relatively fast. However, this does not mean that every request finishes within 80 milliseconds.

The median is useful for understanding the experience near the middle of the request distribution.

p95: The slower end of common traffic

The p95 latency is the 95th percentile.

Approximately 95% of measured requests have latency at or below this value, leaving about 5% above it.

If the p95 is 800 milliseconds, most requests finish within 800 milliseconds, but a meaningful portion take longer.

This metric is particularly useful for identifying performance problems that affect more than just a handful of requests.

p99: The slowest end of the distribution

The p99 latency is the 99th percentile.

Approximately 99% of measured requests finish at or below this value, while around 1% take longer.

If an API handles one million requests during a measurement period, roughly 10,000 requests may fall above the p99 threshold. The exact count depends on the percentile definition, the data, and how values are grouped.

A p99 of four seconds may indicate that a substantial number of requests experience delays, even if the average and p50 remain low.

The p99 is especially useful for high-traffic systems, where even a small percentage of slow requests can represent thousands of affected operations.

Compare Average Latency with Percentiles

Consider an API with the following latency measurements:

Metric	Response time
Average	180 ms
p50	90 ms
p95	650 ms
p99	2,400 ms

The average suggests that the API is reasonably fast. The percentiles reveal a different picture.

Half of the requests finish in approximately 90 milliseconds or less, but the slowest 5% extend beyond the p95 threshold, and the slowest 1% extend beyond the p99 threshold.

This does not automatically mean that the API is unhealthy. The acceptable latency depends on the endpoint, user expectations, service-level objectives, and the business impact of delays.

For example, a background reporting endpoint may tolerate a few seconds of latency, while a payment authorization request may need a much tighter response-time target.

The important point is to compare the measurements against the requirements of the application rather than judging performance using the average alone.

Calculate Percentiles from Sample Data

You can calculate percentiles from a collection of response times to understand how the measurements work.

Consider the following Python example:

import statistics

response_times = [
    45, 50, 55, 60, 65,
    70, 75, 80, 90, 100,
    120, 150, 200, 350, 800
]

average = statistics.mean(response_times)
median = statistics.median(response_times)

sorted_times = sorted(response_times)

print(f"Average: {average:.2f} ms")
print(f"Median: {median:.2f} ms")
print(f"Sorted response times: {sorted_times}")

This calculates the average and median and displays the sorted measurements.

For percentiles, you can use a statistics library such as NumPy:

import numpy as np

response_times = [
    45, 50, 55, 60, 65,
    70, 75, 80, 90, 100,
    120, 150, 200, 350, 800
]

p50, p95, p99 = np.percentile(
    response_times,
    [50, 95, 99]
)

print(f"p50: {p50:.2f} ms")
print(f"p95: {p95:.2f} ms")
print(f"p99: {p99:.2f} ms")

Install NumPy if it is not already available:

pip install numpy

Different tools can calculate percentiles using different interpolation methods. For a small dataset, the calculated result may differ depending on the selected method.

Production monitoring systems also often use histograms or other aggregated data rather than storing every request duration. Their percentile estimates depend on how that data is collected and aggregated.

For that reason, do not assume that percentiles from two monitoring systems will match exactly unless their measurement and aggregation methods are compatible.

Monitor Percentiles in ASP.NET Core

For a .NET application, you can record request duration using built-in ASP.NET Core diagnostics, OpenTelemetry, or custom instrumentation.

A simple middleware example demonstrates how to measure elapsed request time:

using System.Diagnostics;

var builder = WebApplication.CreateBuilder(args);

var app = builder.Build();

app.Use(async (context, next) =>
{
    var stopwatch = Stopwatch.StartNew();

    try
    {
        await next();
    }
    finally
    {
        stopwatch.Stop();

        var elapsedMilliseconds =
            stopwatch.Elapsed.TotalMilliseconds;

        app.Logger.LogInformation(
            "HTTP {Method} {Path} completed with status {StatusCode} in {ElapsedMs:F2} ms",
            context.Request.Method,
            context.Request.Path,
            context.Response.StatusCode,
            elapsedMilliseconds);
    }
});

app.MapGet("/api/products", () =>
{
    return Results.Ok(new[]
    {
        new { Id = 1, Name = "Keyboard" },
        new { Id = 2, Name = "Mouse" }
    });
});

app.Run();

This records the elapsed time for each request handled by the middleware. The measurement includes the work performed by downstream middleware and the endpoint, but it does not necessarily include every part of the request's journey through a load balancer, gateway, or client network.

This example is useful for understanding request timing, but logging every request individually can generate a large volume of data in a busy application. Avoid using raw request logs as your only long-term latency monitoring strategy.

In production, a metrics system should aggregate request durations into histograms or another suitable distribution representation. You can then calculate or estimate p50, p95, and p99 for the relevant measurement window.

OpenTelemetry instrumentation for ASP.NET Core can provide standard HTTP server duration metrics, which can be exported to a compatible metrics backend.

When instrumenting production APIs, use low-cardinality labels such as the route template and HTTP method. Avoid using raw URLs containing user IDs, order IDs, or other unique values as metric labels, because they can create excessive numbers of distinct time series.

Understand Why p95 and p99 Increase

Percentiles help identify a problem, but they do not explain its cause. A rising p95 or p99 is a reason to investigate the request path.

Several common issues can cause the slowest requests to take much longer than usual.

Slow database queries

Some requests may trigger expensive queries, wait for database locks, or retrieve much larger result sets than others.

Look for database calls with unusually long durations, query plans that change under different parameter values, and periods when connection pools become exhausted.

Compare database execution time with the total request duration. If the database accounts for only a small portion of a slow request, the problem may be elsewhere.

Connection pool exhaustion

Applications commonly reuse database connections and outbound HTTP connections through pools.

If too many operations compete for a limited number of connections, requests may spend time waiting for an available connection before any useful work begins.

Monitor pool utilization, connection acquisition time, and the number of concurrent operations.

Thread pool starvation

In .NET, blocking asynchronous operations or excessive synchronous work can reduce the number of threads available to process requests efficiently.

This may cause some requests to wait much longer than others.

Use traces, runtime metrics, and diagnostic tools to investigate thread pool behavior rather than increasing thread counts without understanding the cause.

Downstream service delays

An API may depend on another service for authentication, pricing, inventory, or payment processing.

If that dependency becomes slow, the API's tail latency may increase even when its own code has not changed.

Distributed traces help identify which dependency contributes most to the delay.

Traffic spikes and resource contention

CPU saturation, memory pressure, garbage collection pauses, and increased request concurrency can all affect response times.

Compare latency percentiles with CPU usage, memory metrics, request volume, and dependency timings over the same period.

A p99 increase during a traffic spike may indicate that the application needs more capacity, better concurrency control, or more efficient processing.

Set Latency Targets with Service-Level Objectives

Percentiles are most useful when they are tied to measurable performance goals.

For example, a service-level objective might require 95% of successful API requests to complete within 300 milliseconds over a rolling measurement window.

Another objective might require 99% of requests to complete within one second.

These targets should reflect the needs of the application. A requirement that is too strict can create unnecessary infrastructure costs, while a target that is too loose may allow users to experience unacceptable delays.

It is also important to define which requests are included in the measurement.

Decide whether the objective covers all requests, only successful responses, or a specific class of operations. A metric that excludes failed requests can hide a serious problem if the application becomes fast only because requests are failing early.

Monitor latency alongside error rates, throughput, and saturation. A low latency value is not necessarily good if the server is returning errors instead of completing useful work.

Avoid Common Percentile Monitoring Mistakes

One common mistake is calculating the average of percentile values from several application instances. The average of multiple p95 values is not generally the true p95 of the combined traffic.

For example, one instance may handle many requests while another handles only a few. Averaging their p95 measurements does not account correctly for the different traffic volumes or the underlying latency distributions.

Use a monitoring system that can aggregate compatible histogram data across instances before calculating the overall percentile.

Another mistake is combining measurements from unrelated endpoints. A fast health-check endpoint can make the overall API latency look better even when a critical checkout endpoint is slow.

Track latency by route template or operation where appropriate, and compare equivalent measurement windows.

Finally, avoid reacting to a single unusually high p99 value without examining request volume and the time window. Percentile estimates can be unstable when very few requests are available. Use enough observations to support meaningful conclusions and investigate sustained changes rather than relying on isolated measurements.

Conclusion

Average API response time is useful, but it does not show the full distribution of request latency. An API can have a low average while a meaningful number of users experience slow responses.

The p50 metric helps describe typical performance, p95 reveals slower requests affecting a larger portion of traffic, and p99 highlights delays near the slow end of the distribution.

Combine these measurements with error rates, request volume, resource metrics, and distributed traces to understand both the severity and the cause of performance problems.

For production APIs, define clear latency targets, monitor each important endpoint, and investigate changes in the latency distribution rather than relying on a single average. This gives your team a more accurate view of application performance and a better starting point for troubleshooting.