How to Read CBOR Data Incrementally in .NET 11

Applications often receive data in small chunks instead of getting an entire message at once. This happens when reading from network streams, processing messages from connected devices, receiving data from APIs, or handling large files. If the application waits until the complete message is available before parsing it, memory usage can increase and processing may be delayed.

Concise Binary Object Representation (CBOR) is a binary data format designed to represent structured information with relatively little overhead. It is useful in systems that exchange compact messages, including IoT applications, embedded devices, and network protocols.

.NET provides the System.Formats.Cbor library for reading and writing CBOR data. In .NET 11, new incremental-reading APIs allow CborReader to process CBOR input as it arrives in chunks, without requiring the complete document to be copied into one contiguous buffer first.

The main benefit is that an application can begin reading available data, preserve its parsing position when more bytes are needed, and continue when the next chunk arrives. This is particularly useful when messages are large or their size is not known in advance.

This article explains how incremental CBOR reading works, how to use CborReader with partial input, how to handle incomplete data safely, and what developers should consider before using this approach in production.

What Is CBOR and Why Does Incremental Reading Matter?

CBOR is a binary format for representing values such as integers, strings, byte arrays, arrays, maps, Boolean values, and null. Its data model is similar to JSON in many common scenarios, but CBOR encodes values as bytes rather than readable text.

For example, the following JSON represents a small array of numbers:

[10, 20]

The same two-element array can be encoded in CBOR using three bytes:

82 0A 14

The first byte, 82, identifies an array containing two elements. The next byte, 0A, represents the integer 10, and 14 represents the integer 20.

The binary representation is compact, but the reader still needs to understand the structure of the data. If only the first two bytes have arrived, the array has started and its first value is available, but the second value has not yet arrived.

A traditional parsing approach often expects a complete buffer. The application collects all the bytes, constructs a reader, and then parses the document. Incremental reading allows the application to begin processing earlier and provide additional input when the reader needs it.

When incremental CBOR reading is useful

Consider an IoT gateway receiving sensor messages over a network connection. The gateway might receive the beginning of a CBOR message in one network read and the remaining bytes in another.

Without incremental parsing, the application may need to accumulate the complete message before it can inspect any values. With incremental parsing, it can read the available structure and continue when more data arrives.

The same approach can help with:

Network protocols where messages arrive in multiple reads.
Large CBOR documents that should not be copied into a single large buffer.
Streaming systems that process incoming binary messages.
Applications that need to detect malformed input before accepting an entire payload.
Services that receive data at unpredictable intervals.

Incremental reading does not automatically make every application faster or eliminate all buffering. Its main advantage is that the reader can preserve parsing state across input chunks, allowing the application to manage incoming data more efficiently.

What Changes in .NET 11?

The existing CborReader provides a stateful, forward-only API. It reads values in sequence and maintains information about the current position in the CBOR document.

The incremental-reading additions in .NET 11 allow the reader to distinguish between input that is complete and input that may continue arriving.

The main concepts are:

API or concept	Purpose
CborReaderOptions	Configures reader behavior, including the CBOR conformance mode
isFinalBlock	Indicates whether the supplied input contains the final available portion of the document
NeedsMoreData	Indicates that the reader cannot continue parsing with the bytes currently available
BytesRemaining	Reports the number of unread bytes in the current input buffer
SlideData	Supplies a new buffer while preserving the reader's parsing context

These features work together. The application creates a reader with an initial chunk, tells it whether more input may arrive, and checks the reader's state when parsing cannot continue.

When more bytes are available, the application passes the remaining unconsumed bytes together with the new input to SlideData. This preserves the parsing context instead of starting from the beginning of the document.

One important limitation is that incremental reading currently requires CborConformanceMode.Lax. Other conformance modes reject a non-final input block. Because Lax mode performs fewer conformance checks, applications that require strict validation must account for this limitation before adopting incremental parsing.

The APIs are part of the .NET 11 release work. When using a preview or release-candidate build, verify the package version and documentation for the exact build you have installed.

Set Up the CBOR Reader

To use the CBOR APIs, add the System.Formats.Cbor package to your project.

For a .NET 11 project, the package reference can be declared as follows:

<PackageReference Include="System.Formats.Cbor"
                  Version="11.0.0-rc.2.*" />

This illustrates a release-candidate package version rather than a recommendation to use a floating version in production. Select the exact package version available for your target .NET 11 build and pin it in your project.

The following namespace provides the reader and its related types:

using System.Formats.Cbor;

Once the package is available, the application can create a CborReader using an initial buffer and the required reader options.

For incremental input, the reader must be initialized with the final-block flag set to false. This tells it that the document may contain additional bytes that have not arrived yet.

Read CBOR Data in Multiple Chunks

Suppose an application receives a CBOR array containing the integers 10 and 20. The first chunk contains the array header and the first integer, while the second chunk contains the remaining integer.

The complete CBOR payload is three bytes long, but the application initially has only two bytes.

The following example demonstrates how to continue reading when the next chunk arrives.

using System.Formats.Cbor;

ReadOnlyMemory<byte> firstChunk =
    new byte[] { 0x82, 0x0A };

ReadOnlyMemory<byte> nextChunk =
    new byte[] { 0x14 };

var options = new CborReaderOptions
{
    ConformanceMode = CborConformanceMode.Lax
};

var reader = new CborReader(
    firstChunk,
    options,
    isFinalBlock: false);

reader.ReadStartArray();

int firstValue = reader.ReadInt32();

Console.WriteLine(firstValue);

if (reader.PeekState() == CborReaderState.NeedsMoreData)
{
    if (reader.BytesRemaining != 0)
    {
        throw new InvalidOperationException(
            "This example expects no unconsumed bytes.");
    }

    reader.SlideData(
        nextChunk,
        isFinalBlock: true);
}

int secondValue = reader.ReadInt32();

reader.ReadEndArray();

Console.WriteLine(secondValue);

The output is:

10
20

This example deliberately uses a chunk boundary where no unconsumed bytes remain after reading the first integer. Real network input can split data at other positions, so production code must also preserve any unread bytes when supplying the next chunk.

Understand what happens at each stage

First, the reader receives the bytes 82 and 0A. The first byte identifies an array containing two items. The next byte represents the integer 10.

The application calls ReadStartArray to enter the array and ReadInt32 to read the first item. At this point, the reader has consumed all the available bytes, but the array still contains another value.

The application then checks PeekState. When the reader reports NeedsMoreData, it cannot continue with the current input. This is not automatically a malformed-document error. The reader was told that more data might arrive, so it can wait for the next chunk.

Next, SlideData supplies the remaining byte and marks it as the final block. The reader can then read the integer 20 and close the array with ReadEndArray.

The important point is that the reader retains its knowledge of the array and its position. The application does not need to parse the first integer again.

Handle Unconsumed Bytes Correctly

The previous example covers a simple chunk boundary. In a real application, the reader may still have unread bytes when it requests more input.

For example, the buffer may contain the beginning of the next CBOR value, but not enough bytes to finish reading it. The application must preserve those unconsumed bytes and append the new input.

SlideData expects the replacement buffer to begin with the unread portion of the previous buffer, followed by the newly received bytes.

The following example demonstrates how to build that replacement buffer:

using System.Formats.Cbor;

static ReadOnlyMemory<byte> CombineRemainingAndNewData(
    CborReader reader,
    ReadOnlyMemory<byte> currentBuffer,
    ReadOnlyMemory<byte> incomingChunk)
{
    int remaining = reader.BytesRemaining;

    if (remaining == 0)
    {
        return incomingChunk;
    }

    byte[] combined = new byte[
        remaining + incomingChunk.Length];

    currentBuffer
        .Slice(currentBuffer.Length - remaining)
        .CopyTo(combined);

    incomingChunk.CopyTo(
        combined.AsMemory(remaining));

    return combined;
}

This helper assumes that currentBuffer is the exact buffer currently supplied to the reader. It uses BytesRemaining to identify the unread suffix and combines that suffix with the new input.

For example, if three bytes remain unread and the next network read provides five more bytes, the replacement buffer must contain those original three bytes followed by the five new bytes.

The helper allocates a new array to make the example easy to follow. A high-throughput application may use pooled buffers or another buffer-management strategy to reduce allocations. If you use pooled memory, ensure that it remains valid while the reader needs it and is not returned to the pool too early.

Also remember that this helper only prepares the replacement buffer. The application must call SlideData with that buffer and the correct final-block value before continuing to read.

Distinguish Incomplete Input From Invalid CBOR

One of the most important details in incremental parsing is deciding whether the input is incomplete or invalid.

Suppose the reader encounters the beginning of an integer or string but does not have all the bytes needed to decode it. If the application has declared that more input may arrive, the reader can report NeedsMoreData.

However, if the application declares that the current buffer is the final block and the encoded value is incomplete, the reader cannot wait for additional bytes. The document must be treated as incomplete or invalid.

The application should distinguish between these situations:

Situation	Correct response
More bytes may arrive and the reader needs them	Preserve unread bytes and request the next chunk
The final block arrives and the document is complete	Finish parsing and validate the expected structure
The final block arrives but a value is incomplete	Reject the payload
A value has an invalid CBOR encoding	Handle the parsing error
The document contains unexpected values or structure	Reject it according to the application's schema

Do not catch every parsing exception and automatically wait for more data. That can cause a permanently malformed message to remain in a waiting state or consume resources unnecessarily.

Likewise, do not interpret NeedsMoreData as proof that the incoming payload is valid. It only indicates that the reader needs additional input to continue.

Validate the Document After Reading It

Reading CBOR successfully does not necessarily mean the application should accept the document.

For example, a device might send a valid CBOR map containing a temperature value and a device identifier. The binary structure can be valid even if the temperature is outside the application's permitted range or the device identifier is unknown.

Because incremental reading currently uses Lax conformance mode, application-level validation is especially important.

Check the following after decoding the relevant fields:

Required fields are present.
Values have the expected CBOR types.
Numeric values fall within allowed ranges.
Strings and byte arrays do not exceed configured size limits.
Nested arrays and maps remain within an acceptable depth.
Unexpected fields are handled according to the protocol.
The complete document contains exactly the expected number of root-level values.

For example, a sensor message may require a device ID and a temperature. The application should verify that the device ID is recognized and that the temperature falls within a sensible range before storing the reading.

Also consider the difference between format validation and business validation. Format validation checks whether the bytes represent a valid CBOR structure. Business validation checks whether the decoded values make sense for your application.

Both are necessary when processing untrusted input.

Common Mistakes When Using Incremental CBOR Reading

Treating every NeedsMoreData result as an error

When the reader was initialized with isFinalBlock set to false, NeedsMoreData can be an expected part of normal processing. The application should obtain more bytes and continue parsing rather than treating the result as an immediate failure.

Losing unread bytes between chunks

If the application replaces the input buffer without preserving its unread portion, part of the encoded value may be lost. SlideData requires the new buffer to contain the unread bytes followed by the newly available input.

Marking the final block too early

The final-block flag tells the reader whether more bytes can arrive. If the application marks a chunk as final when the network message is not complete, the reader may reject an otherwise valid message.

Using incremental reading with a strict conformance mode

The current incremental-reading implementation requires Lax mode for non-final blocks. Do not assume that a stricter mode can be used interchangeably. Review the validation requirements of your protocol and add the appropriate application-level checks.

Allowing unbounded message growth

Incremental reading does not eliminate the need for input-size limits. A client could keep sending data without completing a document, causing the application to retain buffers and processing state for too long.

Set maximum message sizes, timeouts, and limits on concurrent incomplete messages. Release buffers and associated resources when a connection closes or parsing fails.

Advantages and Disadvantages

Advantages

Incremental parsing allows the application to begin processing available CBOR data before the complete document arrives.
The reader preserves its parsing context across chunks, avoiding the need to restart parsing from the beginning.
Applications can reduce the need to collect and copy an entire payload into one contiguous buffer.
The approach is useful for network protocols and streaming systems where data arrives at unpredictable intervals.
It provides a clear way to distinguish between needing more input and reaching the end of the document.

Disadvantages

Buffer management is more complicated because unread bytes must be preserved correctly.
Incremental reading currently requires Lax conformance mode for non-final blocks.
Applications still need to manage memory limits, timeouts, and incomplete messages.
Incorrect final-block handling can cause valid input to be rejected or malformed input to be handled incorrectly.
Performance improvements depend on the application's message sizes, input patterns, and buffer-management strategy.

Best Practices for Production Applications

Use incremental reading when your application genuinely receives data in chunks or needs to avoid assembling large payloads before parsing. For small, already-complete CBOR messages, the conventional reader may be simpler.

Keep the input buffer and reader state synchronized. When more data arrives, preserve the unread portion of the current buffer, append the new bytes, and pass the replacement buffer to SlideData. Mark the final block accurately.

Set limits for payload size, nesting depth, incomplete-message duration, and concurrent parsing operations. Handle invalid CBOR separately from incomplete input, and validate decoded values against the application's schema.

Finally, test chunk boundaries deliberately. Split a valid payload at different byte positions, including inside encoded values, and confirm that the application can reconstruct and parse it correctly. Test truncated final input, malformed encodings, unexpectedly large payloads, and cancellation during network reads.

Summary

Incremental CBOR reading in .NET 11 gives developers a way to process binary input as it arrives instead of waiting for an entire document to be available in one buffer. The CborReader APIs support this by reporting when more data is required and allowing the application to supply additional bytes while preserving the parsing context.

The essential steps are to initialize the reader with a non-final block, parse the available values, check for NeedsMoreData, preserve unread bytes, and use SlideData to continue when new input arrives. When the final block is received, the application must ensure that the document is complete and that its contents satisfy the required validation rules.

For production use, buffer management and input validation are just as important as the parsing API. When these details are handled correctly, incremental reading can make CBOR processing more suitable for streaming and network-based applications without requiring every payload to be copied into a single contiguous buffer.