Introduction
Applications often receive data from APIs, files, IoT devices, and other services. When the data becomes large, loading the entire payload into memory before processing it can increase memory usage and delay the response.
CBOR, which stands for Concise Binary Object Representation, is a binary data format designed to represent structured information in a compact form. It is useful when applications need to exchange data efficiently, especially when payload size and processing overhead matter.
In .NET, developers can work with CBOR data using APIs provided by the System.Formats.Cbor package. When data arrives in smaller chunks, incremental processing can help an application avoid waiting for the complete payload before beginning its work.
This article explains how CBOR works, how to read CBOR data incrementally in .NET, and how to handle incomplete input safely. The examples focus on the core parsing concepts and the practical considerations involved in processing streamed binary data.
One important detail: the availability of specific APIs depends on the target framework and package version. Verify the APIs supported by the version used in your project rather than assuming that every CBOR feature is built into .NET 11.
What Is CBOR?
CBOR is a binary serialization format standardized for representing structured data. It supports values such as integers, strings, byte strings, arrays, maps, Boolean values, and null.
Unlike JSON, which represents data using readable text, CBOR encodes values in a binary format.
For example, a JSON message might look like this:
{
"deviceId": 42,
"temperature": 26.5,
"active": true
}
CBOR can represent the same information in a compact binary structure.
The binary representation is not intended to be read directly by a person. An application uses a CBOR decoder to interpret its contents.
CBOR vs. JSON
Feature | CBOR | JSON |
|---|
Representation | Binary | Text |
Human readability | Requires decoding tools | Readable without special tools |
Data types | Supports binary values and additional native data types | Primarily supports objects, arrays, strings, numbers, Booleans, and null |
Payload size | Can be smaller for suitable data | Depends on the structure and textual representation |
Debugging | Often requires a decoder or specialized tools | Easy to inspect in logs and text editors |
Common use cases | Binary protocols, constrained devices, structured binary messages | Web APIs, configuration files, and general data exchange |
CBOR is not automatically smaller or faster for every payload. The result depends on the data, encoding choices, and application implementation.
Why Read CBOR Incrementally?
Consider an application that receives sensor information from thousands of devices. Each device sends a CBOR message containing its identifier, timestamp, and measurements.
A straightforward implementation might collect the entire message into a byte array and then decode it.
For small messages, this is often sufficient. However, large messages or many concurrent requests can increase memory pressure.
Incremental reading allows an application to process data as it becomes available.
The main advantages include:
Processing data without requiring one large input buffer.
Starting work before the entire input has arrived.
Managing memory more carefully for large payloads.
Supporting network protocols that deliver data in multiple chunks.
Incremental reading does not mean that every individual CBOR value can be processed before its encoding is complete. The parser still needs enough bytes to recognize and decode the current value.
It is also important to distinguish incremental parsing from network streaming. A CBOR decoder processes encoded values, while the network layer determines how bytes arrive.
Understanding CBOR Parsing in .NET
The System.Formats.Cbor package provides the CborReader class for reading CBOR data.
A reader can decode values from a byte buffer. For example, the following program reads a CBOR array containing a device identifier, a temperature, and a status flag.
First, install the package if it is not already available in your project:
dotnet add package System.Formats.Cbor
The package version should be selected according to your project's target framework and dependency requirements.
Reading a Complete CBOR Message
The following example creates a CBOR message and reads its values.
using System;
using System.Formats.Cbor;
byte[] payload;
var writer = new CborWriter();
writer.WriteStartArray(3);
writer.WriteInt32(42);
writer.WriteDouble(26.5);
writer.WriteBoolean(true);
writer.WriteEndArray();
payload = writer.Encode();
var reader = new CborReader(payload);
int arrayLength = reader.ReadStartArray()
?? throw new FormatException("An array length is required.");
if (arrayLength != 3)
{
throw new FormatException("Unexpected CBOR array length.");
}
int deviceId = reader.ReadInt32();
double temperature = reader.ReadDouble();
bool active = reader.ReadBoolean();
reader.ReadEndArray();
if (reader.BytesRemaining != 0)
{
throw new FormatException("Unexpected trailing CBOR data.");
}
Console.WriteLine($"Device ID: {deviceId}");
Console.WriteLine($"Temperature: {temperature}");
Console.WriteLine($"Active: {active}");
The writer is included to make the example self-contained. In a real application, the payload would usually come from a file, a network connection, or another service.
The reader processes the array in the order in which its values were encoded.
The length check helps reject unexpected structures. The final check ensures that the example does not silently ignore extra bytes after the expected message.
This example reads a complete payload. To process data incrementally, the application needs to handle incomplete input and preserve the bytes that have not yet been consumed.
Handling CBOR Data That Arrives in Chunks
A network stream does not guarantee that a complete CBOR value will arrive in a single read operation.
For example, a message may contain 100 bytes, but the first network read may return only 20 bytes. The remaining bytes may arrive in later reads.
The application must not assume that one read returns one complete message.
A common approach is to maintain a buffer, append newly received bytes, and parse complete values while retaining incomplete data.
For low-level incremental parsing, .NET provides System.Formats.Cbor reader APIs, but the exact approach depends on the package version and the required CBOR structure.
The following example demonstrates the buffering principle using a stream and a memory buffer. It collects one complete payload before decoding it, so it is useful for understanding partial reads but is not a fully incremental CBOR parser.
using System;
using System.IO;
using System.Threading;
using System.Threading.Tasks;
static async Task<byte[]> ReadPayloadAsync(
Stream stream,
int maximumSize,
CancellationToken cancellationToken = default)
{
using var buffer = new MemoryStream();
byte[] chunk = new byte[4096];
while (true)
{
int bytesRead = await stream.ReadAsync(
chunk.AsMemory(),
cancellationToken);
if (bytesRead == 0)
{
break;
}
if (buffer.Length + bytesRead > maximumSize)
{
throw new InvalidDataException(
"The payload exceeds the permitted size.");
}
await buffer.WriteAsync(
chunk.AsMemory(0, bytesRead),
cancellationToken);
}
return buffer.ToArray();
}
This helper reads until the stream ends and rejects payloads larger than the configured limit.
It is suitable only when the stream's end indicates the end of the payload. If the connection stays open or contains multiple messages, the application needs a framing protocol that identifies where each message ends.
The helper also retains the complete payload in memory. For large messages, a more advanced parser should process complete CBOR values as bytes arrive.
Using SequenceReader and Pipelines for Incremental Processing
For high-throughput applications, System.IO.Pipelines can help manage incoming bytes efficiently.
A pipeline provides a PipeReader that exposes available input as a ReadOnlySequence<byte>. The application can examine the available data, process what it understands, and retain the remainder for the next read.
The general process is:
Read available bytes from the pipeline.
Attempt to decode the next CBOR value.
Record how many bytes were successfully consumed.
Preserve incomplete bytes for the next read.
Release consumed memory and continue processing.
SequenceReader<byte> can help navigate a sequence of bytes, but it does not decode CBOR values by itself. The CBOR parser must still understand the format's encoding rules.
A production implementation should use the incremental or sequence-based APIs supported by the selected CBOR library version. If those APIs are unavailable, use a well-tested parser or buffer complete framed messages before decoding them.
Avoid manually implementing the CBOR binary format unless you have a strong reason to do so. Incorrect handling of lengths, nesting, or special values can lead to parsing errors and security problems.
Handling Incomplete and Invalid Data
Incremental parsers must distinguish between incomplete input and invalid input.
Suppose a message begins with an array declaration but ends before all the array elements arrive. The data may be valid but incomplete.
By contrast, a message may contain a value that does not match the expected structure. That is a validation failure rather than a temporary lack of bytes.
A robust implementation should account for the following situations:
Situation | Recommended Handling |
|---|
More bytes are needed | Preserve the incomplete input and read more data |
Invalid CBOR encoding | Reject the message and record an appropriate error |
Unexpected value type | Reject the message if it violates the expected schema |
Payload exceeds the configured limit | Stop processing and reject the input |
Unexpected trailing bytes | Reject the message or handle them according to the protocol |
Network connection closes early | Treat the message as incomplete if the required data has not arrived |
The parser's exception behavior depends on the specific API. Do not treat every parsing exception as proof that more bytes are needed. Some exceptions indicate malformed data and should not trigger endless retries.
Common Mistakes to Avoid
Assuming One Read Returns the Entire Message
A stream read may return fewer bytes than requested. Always account for partial reads.
Loading Unlimited Data Into Memory
An application that accepts unlimited payload sizes can consume excessive memory. Configure a maximum message size and enforce it while reading.
Ignoring Message Boundaries
CBOR describes data values, but a network protocol must still establish how messages are separated. Use an explicit length prefix, a protocol-defined boundary, or another reliable framing mechanism.
Confusing Incomplete Input With Invalid Input
Incomplete data may become valid when additional bytes arrive. Invalid data should be rejected rather than retried indefinitely.
Accepting Unexpected Data
Check that decoded values have the expected types, lengths, and structure. Reject trailing bytes when the protocol expects exactly one message.
Best Practices for Production Applications
Use these practices when building a CBOR-processing service:
Set a maximum payload size based on the application's requirements.
Use asynchronous stream operations for network-bound workloads.
Preserve unconsumed bytes between parsing operations.
Validate the decoded structure before using its values.
Handle cancellation and network failures explicitly.
Avoid logging complete payloads when they may contain sensitive information.
Test with fragmented input, malformed messages, and unexpected value types.
Use a maintained CBOR library rather than writing a custom decoder.
Measure memory usage and throughput under realistic workloads before choosing a buffering strategy.
These practices improve reliability and help protect the application from malformed or unexpectedly large inputs.
Advantages and Disadvantages
Advantages
Incremental CBOR processing can reduce the need to buffer an entire payload before starting useful work. It can improve memory management and support applications that receive data gradually.
It is particularly useful when processing large messages or handling many concurrent connections.
Disadvantages
Incremental parsing requires more careful buffer management than decoding a complete byte array. Developers must handle incomplete input, message boundaries, parser state, and error recovery correctly.
The implementation can also become more complex if the chosen CBOR library does not expose the required sequence-based or incremental APIs.
Summary
CBOR provides a compact binary format for structured data, and .NET applications can decode it with the System.Formats.Cbor package.
For small payloads, reading the complete message into memory and using CborReader is often the simplest solution. For larger or continuously arriving payloads, incremental processing can reduce buffering overhead, but it requires careful handling of incomplete data and message boundaries.
The most important considerations are to choose APIs supported by your package version, enforce payload limits, validate the decoded structure, and distinguish incomplete input from malformed data.
Start with a simple implementation that matches your message sizes and performance requirements. Introduce incremental parsing when the application's workload justifies the additional complexity.
Join the conversation! Your thoughts help the community grow.