Measuring Local AI Latency Across CPU, GPU, and NPU Hardware

Riya Patel
6d
1.4k
0
1

Article

Introduction

Running an AI model locally sounds straightforward until you run the same model on different hardware.

A model that feels responsive on a machine with a dedicated GPU may be noticeably slower on a CPU-only laptop. A newer device with an NPU can change the picture again, especially for workloads designed around low-power AI inference.

For .NET developers building local AI applications, this creates an important question: Which hardware actually gives the best latency for the workload?

The answer is not always "the fastest GPU."

Different processors have different strengths. CPU inference may be perfectly reasonable for a small model. A GPU can provide much higher parallel throughput. An NPU can potentially deliver efficient AI inference while consuming less power.

Instead of guessing, it is better to measure.

This article walks through a practical approach to benchmarking local AI latency from C#, comparing CPU, GPU, and NPU execution, and interpreting the results correctly.

CPU, GPU, and NPU: What Changes?

The three types of processors are designed differently.

A CPU is a general-purpose processor. It is good at handling a wide variety of application workloads and is available on virtually every development machine.

A GPU is designed to execute many operations in parallel. That makes it particularly useful for neural network workloads with large numbers of matrix operations.

An NPU is specifically designed for AI workloads and can provide efficient inference for supported models and operations.

A simplified architecture looks like this:

                Local AI Application
                       |
        +--------------+--------------+
        |              |              |
       CPU            GPU            NPU
        |              |              |
    General         Parallel       AI-focused
   Processing       Compute       Inference

The important word is supported.

A model cannot automatically use an NPU simply because the computer contains one. The runtime, model format, operators, drivers, and execution provider all need to support the workload.

Why Latency Is More Complicated Than It Looks

Suppose you run the same prompt three times:

CPU:  820 ms
GPU:  190 ms
NPU:  240 ms

It might appear that the GPU is the obvious winner.

But what if the GPU needs to transfer data between system memory and device memory while the NPU can process the workload with much lower power consumption?

For a desktop application that may not matter.

For a battery-powered laptop running an assistant all day, it could matter considerably.

This is why a useful benchmark should measure more than inference time.

Consider these metrics:

Metric	Why it matters
Cold-start latency	Measures first-use experience
Warm inference latency	Measures steady-state performance
P50 latency	Represents typical performance
P95 latency	Shows slower requests
Throughput	Measures work completed over time
Memory usage	Shows system resource consumption
CPU utilization	Shows CPU pressure
GPU utilization	Shows accelerator usage
NPU utilization	Shows whether the NPU is actually being used
Power consumption	Important for mobile devices

Define the Benchmark Before Running It

A benchmark becomes difficult to interpret when the workload changes between tests.

Use the same:

Model
Model version
Input
Output length
Runtime
Precision
Context size
Hardware configuration

For example:

Model:           Local language model
Input:           256 tokens
Expected output: 128 tokens
Precision:       FP16
Context:         4K
Runs:            30

Then run the same workload against each execution provider.

CPU
GPU
NPU

This gives you a controlled comparison.

Cold Start vs Warm Inference

The first request often behaves differently from subsequent requests.

A typical sequence might look like this:

Application Start
      ↓
Load Runtime
      ↓
Load Model
      ↓
Initialize Hardware
      ↓
First Inference

The first request may therefore take significantly longer.

After initialization:

Request
  ↓
Inference
  ↓
Response

may be much faster.

For example:

Test	CPU	GPU	NPU
Cold start	4.8 s	2.1 s	1.9 s
Warm P50	780 ms	180 ms	220 ms

The cold-start number matters for command-line tools and short-lived processes.

The warm number matters more for applications that keep the model loaded.

Always report them separately.

Measuring Latency in C#

For a simple benchmark, Stopwatch is sufficient.

using System.Diagnostics;

var stopwatch = Stopwatch.StartNew();

var response = await RunInferenceAsync(input);

stopwatch.Stop();

Console.WriteLine(
    $"Inference: {stopwatch.Elapsed.TotalMilliseconds:F2} ms");

Do not create a new model instance for every measurement.

Instead, initialize the runtime once:

Initialize
   ↓
Warm Up
   ↓
Benchmark
   ↓
Dispose

Otherwise, model initialization will distort the inference measurements.

Build a Reusable Benchmark Result

A small result type makes it easier to compare execution providers.

public sealed record InferenceResult(
    string Provider,
    double LatencyMs,
    long AllocatedBytes);

Then your benchmark can return:

return new InferenceResult(
    providerName,
    stopwatch.Elapsed.TotalMilliseconds,
    GC.GetAllocatedBytesForCurrentThread());

The exact metrics you collect can be expanded later.

Run a Warm-Up Phase

Before measuring, run several requests that are excluded from the final results.

for (var i = 0; i < 3; i++)
{
    await RunInferenceAsync(input);
}

Then start measuring:

var results = new List<double>();

for (var i = 0; i < 30; i++)
{
    var stopwatch = Stopwatch.StartNew();

    await RunInferenceAsync(input);

    stopwatch.Stop();

    results.Add(stopwatch.Elapsed.TotalMilliseconds);
}

This avoids treating runtime initialization as normal inference behavior.

Use Percentiles Instead of Only Averages

Suppose your 10 measurements are:

The average is heavily affected by the 700 ms outlier.

For interactive applications, P50 and P95 are often more informative.

P50 tells you roughly how a typical request behaves.

P95 tells you what slower users may experience.

A benchmark report could look like:

CPU
P50:  760 ms
P95:  910 ms

GPU
P50:  185 ms
P95:  230 ms

NPU
P50:  225 ms
P95:  270 ms

Now the comparison is much more useful.

CPU Benchmarking

CPU inference is the simplest baseline because it is available almost everywhere.

It is also useful as a reference point.

For example:

CPU
P50 = 820 ms

Then you can calculate relative improvement.

If the GPU produces:

GPU
P50 = 210 ms

the approximate speedup is:

820 / 210 = 3.9x

This gives a more useful comparison than saying the GPU is "faster."

However, CPU performance can depend heavily on:

Number of cores
Instruction set
Memory bandwidth
Thermal limits
Runtime optimizations
Model size
Quantization

So CPU results should always include the processor configuration.

GPU Benchmarking

GPUs are well suited to workloads with high parallelism.

But GPU inference introduces additional considerations.

There may be overhead from:

CPU
 ↓
Memory Transfer
 ↓
GPU
 ↓
Inference
 ↓
Memory Transfer
 ↓
CPU

For a large workload, that overhead may be small compared with the actual computation.

For a very small model, it can become a meaningful part of total latency.

This is why a GPU does not automatically win every benchmark.

Measure the complete request path rather than only accelerator execution time.

NPU Benchmarking

NPUs are particularly interesting for local AI applications because they are designed for AI inference.

However, NPU benchmarking requires additional care.

A model may contain operations that are not supported by the NPU execution provider.

When that happens, the runtime may split the workload:

Model
  ↓
Supported Operations → NPU
Unsupported Operations → CPU

This can make the benchmark confusing.

You might believe that the NPU is processing the complete model when it is actually handling only part of the graph.

Therefore, always verify which execution provider is being used.

Verify That the Accelerator Is Actually Used

One of the easiest benchmarking mistakes is comparing:

CPU

against a configuration that was supposed to use:

GPU

but silently fell back to CPU.

The benchmark may still produce a valid result.

It is simply not the result you intended to measure.

Record the execution provider explicitly:

Provider: CPU
Provider: GPU
Provider: NPU

Also inspect runtime logs or profiling information where available.

A benchmark should never assume accelerator usage just because an accelerator exists in the machine.

Test Different Model Sizes

Hardware differences become more visible as workload size changes.

For example:

Small Model
Medium Model
Large Model

You might see:

Model	CPU	GPU	NPU
Small	120 ms	90 ms	95 ms
Medium	540 ms	170 ms	190 ms
Large	2.4 s	520 ms	610 ms

The smaller model may not benefit much from accelerator execution because overhead becomes a larger part of total latency.

The larger model has enough computation to take advantage of parallel hardware.

Test Different Input Sizes

Input size matters for language models.

Test a range such as:

128 tokens
512 tokens
1,024 tokens
2,048 tokens
4,096 tokens

The results can reveal how latency scales.

For example:

Input Size → Latency

128       → 120 ms
512       → 180 ms
1024      → 270 ms
2048      → 450 ms
4096      → 820 ms

This is more useful than testing one fixed prompt.

It also helps estimate how the application will behave when real users send larger requests.

Measure Token Generation Separately

For generative AI workloads, total latency can be split into different components.

A useful conceptual model is:

Total Latency
    =
Time to First Token
    +
Token Generation Time

Time to first token tells you how quickly the system starts responding.

Tokens-per-second tells you how quickly the remaining response is generated.

For example:

TTFT:  180 ms
Speed: 42 tokens/sec
Output: 100 tokens

Two systems can have similar total latency but very different user experiences.

One might start responding immediately.

Another might wait longer and then generate quickly.

For interactive applications, both measurements matter.

Benchmark Batch Size

If the application processes multiple requests together, test batch size separately.

For example:

Batch 1
Batch 2
Batch 4
Batch 8

A GPU may benefit significantly from batching because it can process more work in parallel.

An interactive application, however, may normally operate at batch size 1.

Do not use a large batch benchmark to make conclusions about single-user interactive latency.

Memory Usage Matters

A model that is fast but consumes almost all available memory may not be practical.

Track memory during:

Startup
Model Loading
Warm-Up
Inference
Repeated Inference

In C#, basic process information can be captured with:

using System.Diagnostics;

var process = Process.GetCurrentProcess();

long memoryMb =
    process.WorkingSet64 / (1024 * 1024);

Console.WriteLine(
    $"Working Set: {memoryMb} MB");

For detailed production profiling, use dedicated operating-system and runtime profiling tools rather than relying only on this value.

Watch for Thermal Throttling

Long-running AI benchmarks can heat the machine.

A system may initially produce:

GPU latency = 180 ms

After several minutes:

GPU latency = 240 ms

The difference may not be caused by the model.

The hardware may be reducing its clock speed to control temperature.

This is especially important for laptops and compact devices.

Run long enough to observe sustained behavior.

A five-second benchmark can tell you about peak performance.

A 30-minute benchmark can tell you about sustained performance.

Both are useful, but they answer different questions.

Measure Power When It Matters

Power consumption becomes particularly important for laptops and edge devices.

Imagine:

CPU
Latency: 700 ms
Power:   Low

GPU
Latency: 180 ms
Power:   High

NPU
Latency: 220 ms
Power:   Low

For a desktop workstation, the GPU may be the obvious choice.

For a battery-powered assistant, the NPU may provide a better overall experience.

This is why "lowest latency wins" is not always the correct conclusion.

The real goal is often:

Best latency
+
Acceptable power consumption
+
Acceptable memory usage

Keep the Benchmark Reproducible

A useful benchmark should document the environment.

Record:

Operating System
.NET Version
Model
Model Format
Quantization
Runtime Version
Execution Provider
CPU
GPU
NPU
RAM
Input Size
Output Size
Number of Runs

For example:

CPU:       12-core processor
GPU:       Dedicated accelerator
NPU:       Available
RAM:       32 GB
Model:     Local 7B model
Precision: 4-bit
Runs:      30
Warm-up:   3

Without this information, benchmark numbers are difficult to reproduce.

A Practical Benchmark Structure

A clean benchmark project can use this structure:

LocalAiBenchmark/
│
├── Models/
├── Audio/
├── Benchmarks/
│   ├── CpuBenchmark.cs
│   ├── GpuBenchmark.cs
│   └── NpuBenchmark.cs
│
├── Results/
│
└── Program.cs

The goal is to keep model execution separate from result collection.

That makes it easier to change the runtime or model without rewriting the entire benchmark.

Example Benchmark Output

A useful console result might look like:

Local AI Benchmark
==================

Model: LocalModel
Input: 1024 tokens
Output: 256 tokens
Warm-up: 3
Runs: 30

CPU
P50:  820 ms
P95:  980 ms
Memory: 4.1 GB

GPU
P50:  210 ms
P95:  260 ms
Memory: 5.8 GB

NPU
P50:  235 ms
P95:  290 ms
Memory: 3.9 GB

This immediately tells you more than a single "GPU is 4x faster" statement.

Common Benchmarking Mistakes

Comparing Different Models

The hardware comparison becomes meaningless if the model changes.

Measuring Only One Request

One request is not representative.

Ignoring Warm-Up

Initialization can significantly distort results.

Assuming NPU or GPU Usage

Always verify the execution provider.

Ignoring Model Quantization

Precision and quantization can dramatically change performance and memory usage.

Using Only Average Latency

Outliers can have a major impact on interactive applications.

Testing Only Short Workloads

Long-running workloads can expose thermal throttling and memory problems.

Ignoring Power

A small latency improvement may not justify significantly higher power consumption.

Best Practices

Use exactly the same model across hardware comparisons.
Keep input and output sizes consistent.
Separate cold-start and warm inference measurements.
Run multiple iterations.
Report P50 and P95 latency.
Measure time to first token for streaming workloads.
Measure tokens per second for generative models.
Verify actual CPU, GPU, or NPU execution.
Record memory usage.
Test sustained workloads to identify thermal throttling.
Record model precision and quantization.
Document the complete hardware and software environment.
Test realistic workloads instead of synthetic prompts alone.
Consider power consumption when targeting laptops or edge devices.
Avoid treating one machine's benchmark as a universal result.

FAQs

Is an NPU always faster than a CPU?

No. Performance depends on the model, runtime, supported operators, and hardware. A small workload may not benefit enough from NPU execution to overcome runtime overhead.

Is a GPU always the best option?

Not necessarily. GPUs are often excellent for larger parallel workloads, but power consumption, memory requirements, and workload size can change the trade-off.

Should I compare CPU, GPU, and NPU using the same model?

Yes. Using the same model, input, output, precision, and runtime configuration makes the comparison much more meaningful.

What is more important: P50 or P95 latency?

Both are useful. P50 represents typical performance, while P95 shows how slower requests behave. For interactive applications, P95 can reveal user-visible latency problems that averages hide.

Should I benchmark power consumption?

If the application will run on laptops, mobile devices, or edge hardware, yes. Power efficiency can be just as important as raw inference speed.

How many benchmark runs should I perform?

There is no universal number, but a warm-up phase followed by at least a few dozen consistent runs gives a much better picture than one or two measurements.

Conclusion

CPU, GPU, and NPU benchmarking is really about understanding the trade-offs of running AI locally. A GPU may provide the lowest latency for a large model, while an NPU may offer a better balance between performance and power on a laptop. CPU execution can still be perfectly practical for smaller workloads or machines without dedicated AI hardware. The important thing is to test the same workload across the available execution providers, separate cold-start behavior from warm inference, measure P50 and P95 latency, verify that the intended accelerator is actually being used, and include memory and power considerations where they matter. Once those measurements are available, choosing the right hardware becomes an engineering decision based on the application's actual requirements rather than assumptions about which processor is supposed to be faster.