Benchmarking Local LLM Inference with Quantized Models on Windows

Running a large language model locally on Windows has become much more practical as consumer GPUs and optimized inference runtimes have improved.

For developers, local inference can be useful when an application needs lower latency, offline operation, predictable infrastructure costs, or stronger control over sensitive data. But choosing a model is only part of the problem.

The way the model is quantized, the amount of available GPU memory, the inference runtime, context length, batch configuration, and hardware all affect performance.

A model that looks attractive on paper may perform poorly on a particular Windows machine.

This makes benchmarking important.

Instead of asking only:

Which local model is fastest?

a useful benchmark should answer:

Which model and quantization level provides the best balance of quality, memory usage, throughput, latency, and stability on the target hardware?

This article explains how to build a repeatable local LLM benchmark on Windows, how quantization affects inference, and how to compare configurations without drawing misleading conclusions.

What Is Quantization?

Quantization reduces the numerical precision used to represent model parameters.

A model may originally use higher-precision values such as:

FP32
FP16
BF16

Quantized versions can use lower-precision representations such as:

INT8
INT6
INT5
INT4

The exact formats and implementation details depend on the model and inference runtime.

The main goal is to reduce:

Model size
GPU memory usage
Memory bandwidth requirements

while retaining as much model quality as possible.

A simplified example looks like this:

Original model
     |
     v
FP16 parameters
     |
     v
Quantization
     |
     +----> 8-bit
     |
     +----> 6-bit
     |
     +----> 4-bit

Lower precision can make a model fit on hardware that could not otherwise run it.

However, lower precision does not automatically mean better performance.

Why Quantization Matters for Windows Developers

Local inference is often constrained by hardware.

Suppose a model requires more memory than the available GPU VRAM.

The options may include:

Use a smaller model
Use a more aggressive quantization
Use CPU inference
Use partial GPU offloading
Use a machine with more VRAM

Quantization can therefore change whether a model is practical at all.

For example:

Configuration	Relative Memory	Potential Trade-off
FP16	High	Higher memory requirement
INT8	Medium-high	Better memory efficiency
INT6	Medium	Balance
INT4	Low	Greater quality risk depending on model

These are conceptual comparisons rather than universal measurements. Actual memory usage depends on architecture, runtime, context length, and implementation.

Hardware Should Be Part of the Benchmark

Do not publish or compare inference numbers without documenting the hardware.

At minimum, capture:

CPU
GPU
GPU VRAM
System RAM
Windows version
Inference runtime
Runtime version
Model
Quantization
Context length
Driver version

For example:

CPU:
Desktop 16-core processor

GPU:
NVIDIA GPU

VRAM:
16 GB

RAM:
64 GB

OS:
Windows 11

Runtime:
Local inference runtime

Model:
7B-class model

Quantization:
4-bit

The exact hardware matters because inference performance is strongly dependent on memory bandwidth, compute capability, and whether the model fits entirely in GPU memory.

Define What You Want to Measure

A useful local LLM benchmark should measure several metrics.

Time to First Token

This measures how long the system takes to produce the first generated token.

It is especially important for interactive applications.

Request
  |
  v
Model processing
  |
  v
First token

A lower value generally produces a more responsive user experience.

Tokens Per Second

This measures generation throughput.

Tokens Per Second =
Generated Tokens / Generation Time

It is useful for comparing steady-state generation performance.

Total Latency

Measure the complete request:

Request
+
Model processing
+
Generation
+
Response

This is more representative of real application behavior than generation speed alone.

Memory Usage

Track:

GPU VRAM
System RAM
Peak memory
Idle memory

A configuration that generates 30 tokens per second but constantly exhausts available VRAM may not be practical.

Benchmark Prompt Processing Separately

LLM workloads usually have at least two different performance phases.

Prompt processing
       |
       v
Token generation

A long prompt may take significant time to process even when generation itself is fast.

For example:

Input:
12,000 tokens

Output:
300 tokens

The workload is very different from:

Input:
500 tokens

Output:
3,000 tokens

Benchmark both scenarios.

This is particularly important for coding assistants, RAG applications, and agents where context can become large.

Use a Fixed Benchmark Dataset

Do not compare models using random prompts.

Create a fixed dataset containing representative tasks.

For a developer-focused benchmark, use categories such as:

Code generation
Code explanation
Bug fixing
SQL generation
JSON generation
Summarization
Reasoning
Long-context analysis

For example:

[
  {
    "id": "code-001",
    "category": "code-generation",
    "prompt": "Create a C# method that..."
  },
  {
    "id": "code-002",
    "category": "debugging",
    "prompt": "Identify the issue in this code..."
  },
  {
    "id": "summary-001",
    "category": "summarization",
    "prompt": "Summarize the following technical document..."
  }
]

The exact benchmark prompts should reflect the workloads you expect in production.

Warm Up the Model Before Measuring

The first request is often different from subsequent requests.

The runtime may need to:

Load model
Allocate memory
Initialize kernels
Initialize GPU resources
Build caches

If you include model loading in every measurement, you are measuring startup behavior rather than steady-state inference.

Use a warm-up phase:

Load model
    |
    v
Warm-up requests
    |
    v
Benchmark requests

Measure cold-start and warm inference separately.

Measure Cold Start

Cold-start performance still matters for applications that frequently load and unload models.

Record:

Model load time
Memory allocation time
First request latency
Time to first token

A model that is excellent after warm-up may still be inconvenient for desktop applications that start inference on demand.

Keep Runtime Settings Constant

When comparing quantizations, keep other variables fixed.

For example:

Model architecture
Prompt dataset
Context length
Temperature
Maximum output tokens
GPU layers
Batch size
Sampling settings
Runtime version

Otherwise, you cannot determine what caused the performance difference.

A fair benchmark changes one major variable at a time.

Compare Quantization Levels

A basic benchmark matrix might look like:

Quantization	Model Size	VRAM Usage	Generation Speed	Quality
FP16	Highest	Highest	Baseline	Highest reference
INT8	Lower	Lower	Measure	Usually close
INT6	Lower	Lower	Measure	Measure
INT4	Lowest	Lowest	Measure	Measure

Do not treat this as a universal performance ranking.

The runtime and GPU can produce very different results.

The benchmark must measure the actual configurations.

Quality Must Be Measured Too

A faster model is not automatically better.

Suppose:

Model A:
42 tokens/sec
High coding accuracy

Model B:
55 tokens/sec
Noticeably worse coding accuracy

For a coding assistant, Model A may be the better choice.

Include quality evaluation.

For code-generation workloads, useful metrics include:

Compilation success
Unit-test success
Functional correctness
Syntax correctness
Instruction adherence

For structured output:

Valid JSON rate
Schema compliance
Required-field accuracy

For summarization:

Factual consistency
Coverage
Human evaluation

Performance and quality should be evaluated together.

Use Repeatable Code Evaluation

For coding models, generated code should not be evaluated only by reading it.

A stronger approach is:

Prompt
  |
  v
Model generates code
  |
  v
Compile
  |
  v
Run tests
  |
  v
Record result

For a .NET benchmark:

dotnet build
dotnet test

This gives an objective signal.

For example:

Model	Quantization	Compile Rate	Test Pass Rate
Model A	8-bit	98%	94%
Model A	4-bit	96%	91%
Model B	8-bit	97%	93%
Model B	4-bit	95%	89%

These values are illustrative. Actual benchmark results should be generated from the test dataset.

Measure GPU Utilization

GPU utilization can help explain unexpected performance.

Track:

GPU utilization
VRAM utilization
Power usage
Temperature
Clock behavior

Suppose one quantization configuration produces low GPU utilization while another keeps the GPU busy.

That difference can help identify whether the workload is:

Compute-bound
Memory-bound
CPU-bound
Transfer-bound

Without utilization metrics, a raw tokens-per-second number can be difficult to interpret.

Watch CPU-GPU Offloading

A model can sometimes run partly on the GPU and partly on the CPU.

This may make a larger model technically runnable, but performance can change substantially.

Conceptually:

Model
 |
 +--> GPU layers
 |
 +--> CPU layers

Data movement between CPU and GPU can become a bottleneck.

Benchmark at least:

Full GPU
Partial GPU
CPU-only

when the runtime supports these configurations.

Do not assume that putting more layers on the GPU always produces a linear improvement.

Context Length Changes Memory Usage

A common benchmarking mistake is to report model memory usage without specifying context length.

The model weights are only part of the memory footprint.

The runtime also needs memory for intermediate state and the KV cache used during generation.

As context grows:

Context
  |
  +--> KV cache
  |
  v
Higher memory requirement

Therefore, benchmark realistic context sizes.

For example:

2K tokens
8K tokens
16K tokens
32K tokens

if the model and runtime support them.

Benchmark Long-Context Workloads Separately

A model may perform well with:

1,000-token prompt

but struggle with:

30,000-token prompt

Long-context benchmarks should record:

Prompt processing time
Peak VRAM
Generation speed
Time to first token
Total latency

This is especially relevant for:

RAG
Coding assistants
Document analysis
Agent memory
Repository analysis

Avoid Single-Run Benchmarks

One run is not enough.

Performance can vary because of:

Background applications
GPU temperature
Power state
CPU scheduling
Memory pressure
Windows processes
GPU clock behavior

Run each configuration multiple times.

For example:

Warm-up:
3 runs

Benchmark:
10 runs

Then report:

Median
P95
Minimum
Maximum

The median is often more useful than a single best result.

Report Variability

Consider these results:

Run 1: 42 tok/s
Run 2: 44 tok/s
Run 3: 43 tok/s
Run 4: 43 tok/s
Run 5: 42 tok/s

The system is stable.

Now compare:

Run 1: 51 tok/s
Run 2: 32 tok/s
Run 3: 48 tok/s
Run 4: 29 tok/s
Run 5: 46 tok/s

The average alone hides the instability.

Report distribution statistics when possible.

Example Benchmark Harness

A simple benchmark harness can record request-level timing.

using System.Diagnostics;

public sealed record BenchmarkResult(
    double TotalMilliseconds,
    int OutputTokens);

public static async Task<BenchmarkResult> RunBenchmarkAsync(
    Func<Task<string>> inference)
{
    var stopwatch = Stopwatch.StartNew();

    string response = await inference();

    stopwatch.Stop();

    int outputTokens = EstimateTokenCount(response);

    return new BenchmarkResult(
        stopwatch.Elapsed.TotalMilliseconds,
        outputTokens);
}

static int EstimateTokenCount(string text)
{
    if (string.IsNullOrWhiteSpace(text))
        return 0;

    return text.Split(
        ' ',
        StringSplitOptions.RemoveEmptyEntries).Length;
}

The token estimator above is intentionally simple and should not be treated as an exact tokenizer.

For a serious benchmark, use the tokenizer associated with the model or runtime so token counts are measured consistently.

Record Structured Benchmark Results

Store benchmark results in a machine-readable format.

For example:

{
  "model": "example-model",
  "quantization": "4-bit",
  "contextTokens": 8192,
  "runs": 10,
  "medianTokensPerSecond": 42.8,
  "p95LatencyMs": 1850,
  "peakVramMb": 11240,
  "taskSuccessRate": 0.94
}

This makes it easier to compare runs over time.

Build a Benchmark Matrix

A useful matrix might include:

Model	Quantization	Context	GPU Offload	Median tok/s	P95 Latency	Peak VRAM	Quality
Model A	8-bit	8K	Full	Measure	Measure	Measure	Measure
Model A	4-bit	8K	Full	Measure	Measure	Measure	Measure
Model A	4-bit	16K	Full	Measure	Measure	Measure	Measure
Model B	8-bit	8K	Full	Measure	Measure	Measure	Measure
Model B	4-bit	8K	Full	Measure	Measure	Measure	Measure

This is much more useful than saying:

Model A runs at 40 tokens per second.

The additional context tells readers what that number actually means.

Common Benchmarking Mistakes

Comparing Different Hardware

A GPU with significantly different memory bandwidth or compute capability can produce completely different results.

Mixing Runtime Versions

Runtime optimizations can change performance.

Including Cold Start in Some Tests

Cold-start and warm inference are different metrics.

Changing Multiple Variables

Changing quantization, context, batch size, and GPU offloading simultaneously makes the result difficult to interpret.

Measuring Only Tokens Per Second

Speed without quality, memory, and latency does not tell the full story.

Using One Prompt

One prompt cannot represent a production workload.

Using One Run

Single measurements can be affected by background system activity.

Ignoring Context Length

Memory consumption can change significantly with larger contexts.

Reporting Peak Performance Only

A best-case number is less useful than a reproducible median and variability range.

A Practical Benchmarking Workflow

A repeatable Windows benchmark can follow this process:

1. Record hardware and software versions
2. Select representative models
3. Select quantization levels
4. Create a fixed prompt dataset
5. Define context sizes
6. Load the model
7. Warm up the runtime
8. Run repeated measurements
9. Capture latency and throughput
10. Capture GPU and system memory
11. Evaluate output quality
12. Store structured results
13. Repeat for every configuration
14. Compare results

Keep the environment as stable as possible between runs.

How to Choose the Best Configuration

The fastest configuration is not necessarily the best one.

Imagine three options:

Configuration A
High quality
High memory
Moderate speed

Configuration B
Good quality
Moderate memory
High speed

Configuration C
Lower quality
Low memory
Very high speed

For an interactive coding assistant, B may be the best balance.

For offline batch processing, C might be attractive.

For sensitive code-generation tasks where correctness matters more than latency, A may be preferable.

The benchmark should therefore support a workload-specific decision rather than produce one universal winner.

Frequently Asked Questions

Does lower-bit quantization always make inference faster?

No. Lower precision can reduce memory requirements and sometimes improve throughput, but actual performance depends on the model architecture, GPU, runtime, kernels, memory bandwidth, and workload.

Is 4-bit quantization always worse than 8-bit?

Not necessarily. The quality difference depends on the model and quantization method. The correct approach is to benchmark both performance and task quality.

Should I benchmark CPU inference?

Yes, if CPU execution is a realistic deployment option. It provides a useful baseline and can reveal whether GPU offloading is actually helping.

Why does context length affect performance?

Longer context increases the amount of data the runtime must process and can increase memory consumption, including KV-cache requirements.

How many benchmark runs should I perform?

There is no universal number, but multiple warm runs are necessary. Ten runs per configuration is a reasonable starting point for a small engineering benchmark, followed by median and percentile analysis.

Can tokens per second be compared across different models?

Only with care. Tokenization, model architecture, workload, context length, runtime, and hardware all affect the measurement.

Conclusion

Local LLM inference on Windows is increasingly practical, but selecting a model based only on its parameter count or advertised speed can produce misleading results. Quantization, context length, GPU memory, CPU-GPU offloading, runtime configuration, and workload characteristics all influence the real performance you will experience.

A useful benchmark measures more than tokens per second. Capture time to first token, generation throughput, total latency, memory consumption, GPU utilization, cold-start behavior, and task quality. Use a fixed workload, warm up the runtime, repeat measurements, and report enough environment details that another developer can understand the conditions behind the numbers.

Most importantly, choose a configuration based on the workload. A slightly slower quantized model that fits comfortably in memory and produces more reliable code may be a better production choice than a faster configuration that sacrifices quality or stability.

The purpose of local LLM benchmarking is not to find the biggest or fastest model. It is to find the configuration that delivers the right combination of quality, responsiveness, memory efficiency, and reliability for the application you actually need to build.