Benchmarking Zstandard Compression in .NET Applications

followed by compression.

The streaming design can avoid requiring the entire input in memory.

Benchmark both approaches when the application has a choice.

Measure Memory Usage

Add:

[MemoryDiagnoser]

to the benchmark.

Then examine:

• Allocated bytes

• Gen 0 collections

• Gen 1 collections

• Gen 2 collections

For high-throughput services, allocation behavior can matter as much as raw compression time.

A compression implementation that is slightly faster but creates significantly more temporary allocations may behave differently under sustained concurrency.

Avoid Benchmarking Only MemoryStream

MemoryStream is useful because it removes storage latency.

But production applications may use:

FileStream
NetworkStream
Pipe
HTTP response stream
Cloud storage stream

After measuring compression itself, benchmark the actual stream used by the application.

The test progression can be:

Phase 1
MemoryStream
     |
     v
Algorithm cost

Phase 2
FileStream
     |
     v
Algorithm + disk

Phase 3
Network
     |
     v
Algorithm + network

This helps separate bottlenecks.

Test Compression Levels

Compression APIs expose a CompressionLevel abstraction, and Zstandard also provides more detailed configuration through ZstandardCompressionOptions. Microsoft documents constructors for Zstandard streams using both CompressionLevel and explicit Zstandard options.

Start with the common levels:

CompressionLevel.Fastest
CompressionLevel.Optimal

Then test more detailed Zstandard settings if the application needs them.

Do not assume Optimal is the right production setting.

The correct configuration depends on:

CPU budget
Payload size
Bandwidth
Latency target
Compression ratio
Request frequency

Build a Compression Matrix

A useful benchmark table is:

The values should come from the actual benchmark environment.

Zstandard Dictionaries

Zstandard also supports dictionaries.

The concept is useful when many small payloads share common structure.

For example:

Payload 1
{
  "customerId": ...
  "transactionId": ...
}

Payload 2
{
  "customerId": ...
  "transactionId": ...
}

A dictionary can contain patterns that frequently occur in the data.

.NET 11 exposes ZstandardDictionary and related encoder/decoder APIs.

Conceptually:

Training Data
     |
     v
Dictionary
     |
     +--------+
     |        |
     v        v
Encoder    Decoder
     |        |
     v        v
Payload    Payload

Dictionary-based compression should be benchmarked separately from ordinary Zstandard compression.

When Dictionaries May Help

Dictionaries are particularly interesting for:

• Small messages

• Repeated schemas

• Protocol records

• RPC payloads

• Structured logs

• Similar JSON objects

But dictionary effectiveness depends strongly on whether the dictionary represents the actual production data.

A dictionary generated from unrelated data may provide little benefit.

Benchmark Dictionary-Based Compression

A simplified example using the explicit encoder API could look like:

using System.IO.Compression;

using var dictionary =
    new ZstandardDictionary(
        dictionaryBytes);

using var encoder =
    new ZstandardEncoder(
        dictionary);

The exact encoder API should be used according to the .NET 11 SDK version being tested because the Zstandard API is part of the evolving .NET 11 surface. Microsoft's current API documentation lists dictionary-aware encoder and stream constructors.

For a production benchmark, record:

Dictionary size
Training dataset
Payload size
Compression ratio
Compression time
Decompression time

Benchmark Highly Compressible Data

Create a dataset with repeated structures.

For example:

var text = string.Join(
    Environment.NewLine,
    Enumerable.Repeat(
        "INFO User authenticated successfully",
        100_000));

Convert it to bytes:

byte[] data =
    Encoding.UTF8.GetBytes(text);

This represents a highly repetitive workload.

It is useful for testing compression effectiveness, but it should not be the only dataset.

Benchmark Low-Compressibility Data

Use data that already has high entropy.

For example:

var data =
    RandomNumberGenerator.GetBytes(
        10 * 1024 * 1024);

This helps answer a practical question:

How much CPU does compression consume when the input does not compress well?

This can be important for applications processing encrypted or already-compressed data.

Do Not Compress Everything

Compression is not always beneficial.

Avoid blindly compressing:

JPEG
PNG
MP4
ZIP
GZIP
Zstandard
Encrypted payloads

Many of these formats are already compressed or have high entropy.

Compression can add CPU cost without significantly reducing size.

A better strategy is to understand the data first.

Compression and Encryption

Encryption should generally happen after compression when both operations are required:

Original Data
     |
     v
Compression
     |
     v
Encryption
     |
     v
Network / Storage

Compressing encrypted data is generally ineffective because encrypted output should not retain the predictable structure compression algorithms depend on.

For sensitive data, also consider whether compression introduces side-channel risks in the specific protocol design.

Do not add compression to security-sensitive protocols without reviewing the threat model.

HTTP Compression

For HTTP APIs, compression is often negotiated between the client and server.

Conceptually:

Accept-Encoding: zstd, br, gzip

The server can select a supported encoding.

The exact HTTP framework and infrastructure configuration determines whether Zstandard can be negotiated and emitted automatically.

Do not assume that adding ZstandardStream automatically enables HTTP content encoding.

A custom implementation may require explicit middleware or server configuration.

Storage Workloads

Compression can reduce storage consumption.

For example:

Raw Data
   |
   v
Zstandard
   |
   v
Object Storage

This can be useful for:

• Data exports

• Backups

• Logs

• Archive files

• Intermediate pipeline data

But storage cost is only one variable.

Measure:

Compression CPU
+
Storage reduction
+
Upload time
+
Download time
+
Decompression time

A smaller file is not automatically a better architecture if compression consumes excessive CPU during every request.

Common Benchmarking Mistakes

Using Random Data Only

Random data often compresses poorly.

You need representative structured datasets.

Using Highly Repetitive Data Only

Highly repetitive data can make compression appear unrealistically effective.

Use multiple datasets.

Measuring Only Compression Time

Measure output size and decompression as well.

Including Disk I/O Without Realizing It

Use MemoryStream to isolate algorithm cost.

Then separately benchmark real storage.

Using Different Input Data Per Algorithm

Every algorithm must receive identical input.

Ignoring Allocations

High-throughput services can be affected by allocation and GC pressure.

Testing One Payload Size

Small messages and large files can behave very differently.

Publishing Unverified Benchmarks

Do not report specific throughput, CPU, or compression-ratio numbers unless they were actually measured.

Troubleshooting

Zstandard Output Is Larger Than Expected

Check the input.

Already-compressed or high-entropy data may not benefit much from additional compression.

Also verify the selected compression settings.

Compression Uses Too Much CPU

Test a faster compression configuration.

Then measure whether the larger output causes an acceptable network or storage trade-off.

Decompression Is the Bottleneck

Benchmark the consumer separately.

The fastest compressor is not automatically the best choice if clients have constrained CPU resources.

Memory Usage Is High

Check whether the application buffers the complete input and output.

Prefer streaming APIs when the workload supports them.

Dictionary Compression Does Not Improve Results

Verify that the dictionary was trained on representative data.

Dictionary effectiveness depends on similarity between training and production payloads.

Production Decision Framework

A compression decision can be expressed as:

             Is bandwidth expensive?
                    |
              +-----+-----+
             Yes          No
              |            |
              v            v
        Evaluate ratio   Is CPU limited?
                              |
                         +----+----+
                        Yes        No
                         |          |
                         v          v
                   Prefer speed   Benchmark

This is only a starting point.

Actual decisions should use measurements.

For example:

Scenario A
CPU expensive
Network cheap
    |
    v
Favor faster compression

Scenario B
CPU available
Network expensive
    |
    v
Favor stronger compression

Scenario C
Small repeated messages
    |
    v
Evaluate dictionaries

Scenario D
Already compressed data
    |
    v
Consider skipping compression

Best Practices

1. Benchmark Zstandard against the algorithms already used by the application.

2. Use production-shaped datasets.

3. Test multiple payload sizes.

4. Measure compression and decompression separately.

5. Measure compressed size and compression ratio.

6. Measure allocations and memory usage.

7. Use MemoryStream to isolate algorithm performance.

8. Then test the actual production stream.

9. Evaluate compression levels independently.

10. Use dictionaries only when the workload justifies them.

11. Do not compress already-compressed data blindly.

12. Consider both CPU and network/storage costs.

13. Document hardware, runtime, input data, and configuration.

14. Do not publish fabricated benchmark numbers.

Frequently Asked Questions

Does .NET 11 support Zstandard natively?

Yes. .NET 11 adds Zstandard APIs to System.IO.Compression, including ZstandardStream and ZstandardEncoder.

Is Zstandard faster than GZip?

There is no universal result that applies to every payload and configuration.

Compression level, input data, hardware, implementation, and workload all affect the outcome.

Benchmark your actual data.

Is Zstandard better than Brotli?

Neither is universally better.

The appropriate choice depends on compression ratio, compression speed, decompression speed, compatibility, and application requirements.

Should I use Zstandard for every API response?

No.

Evaluate payload size, client support, CPU cost, network savings, and infrastructure compatibility before enabling it broadly.

Are Zstandard dictionaries useful?

They can be useful for small, structurally similar messages.

Their effectiveness depends on the quality of the dictionary and similarity between training and production data.

Should I benchmark compression with real files?

Yes.

If your production workload processes files, benchmark representative files.

Also use MemoryStream when you need to isolate compression cost from storage performance.

Conclusion

Zstandard support in .NET 11 gives developers another native compression option without requiring a separate compression abstraction. ZstandardStream, ZstandardEncoder, dictionaries, and related configuration APIs are now part of System.IO.Compression.

But the existence of a new compression algorithm does not answer the architectural question.

The correct choice depends on the complete workload:

Input Data
    |
    v
Compression
    |
    +-- CPU
    +-- Memory
    +-- Time
    |
    v
Compressed Data
    |
    +-- Storage
    +-- Network
    |
    v
Decompression
    |
    v
Consumer

A useful benchmark therefore measures more than compression ratio.

It should compare:

Compression Time
+
Decompression Time
+
Output Size
+
CPU
+
Memory
+
Allocations
+
End-to-End Cost

The most important principle is:

Do not choose a compression algorithm because it produces the smallest file or the fastest benchmark in isolation. Choose it because its compression, decompression, CPU, memory, and storage/network characteristics fit the actual workload.

.NET 11's native Zstandard APIs make that evaluation easier to perform directly within the .NET ecosystem. The next step for any production team should be a controlled benchmark using its own representative data rather than relying on generic algorithm rankings.