Reading CBOR Data Incrementally with the New .NET 11 APIs

Applications that exchange structured binary data often need to process messages received over a network, read files, or communicate with external devices. When the input arrives in multiple chunks, loading the entire payload into a single buffer before parsing it can add unnecessary memory pressure and delay processing.

.NET 11 RC 2 introduces incremental reading support in System.Formats.Cbor. The updated CborReader can process a CBOR document that arrives in chunks without requiring the entire document to be copied into contiguous memory first.

The new functionality is useful for developers building network protocols, data ingestion services, and applications that handle structured binary messages. It also introduces an important responsibility: the application must recognize when more input is needed and provide the next chunk without losing unread data.

This article explains how incremental CBOR reading works, how to use the new APIs, and what to consider when implementing a reliable parser.

What Is CBOR?

CBOR stands for Concise Binary Object Representation. It is a binary data format designed to represent structured values such as integers, strings, arrays, maps, Boolean values, and other data types.

Unlike JSON, which represents values using readable text, CBOR uses a compact binary encoding. This can make it suitable for protocols where message size and efficient data exchange matter.

For example, a device might send a measurement containing a sensor identifier and a numeric reading. A CBOR array could represent those values as two elements:

using System.Formats.Cbor;

var writer = new CborWriter();

writer.WriteStartArray(2);
writer.WriteTextString("sensor-17");
writer.WriteInt32(28);
writer.WriteEndArray();

byte[] payload = writer.Encode();

The example creates a two-element CBOR array containing a sensor identifier and a reading. The resulting byte array can be transmitted or stored.

Before .NET 11 RC 2, developers using CborReader generally needed to provide the encoded data through a complete input buffer. Incremental reading changes how the reader can consume input when a document arrives in separate chunks.

What Is New in .NET 11 RC 2?

The updated CborReader supports input that is delivered incrementally.

Three elements are particularly important:

isFinalBlock: Indicates whether the supplied input represents the final block of the CBOR document.
NeedsMoreData: Indicates that the reader cannot continue because additional input is required.
SlideData: Supplies another input block while allowing the reader to continue processing the document.

Incremental reading requires CborConformanceMode.Lax. Other conformance modes reject a non-final block because they do not support this incremental input behavior.

The main advantage is that the parser no longer needs to wait for the entire encoded document before it can begin reading its contents.

However, incremental reading does not automatically make every application memory-efficient. The application must still manage incoming buffers, preserve unread bytes, and avoid accumulating the complete payload unnecessarily.

How Incremental CBOR Reading Works

Consider a CBOR array containing two integers: 10 and 20.

Its encoded representation can be split across two input chunks. The first chunk contains the array header and the first integer, while the second chunk contains the remaining integer.

The reader can process the first value immediately and request more input when it reaches the incomplete portion of the document.

Step 1: Create a Reader With an Incomplete Input Block

The first chunk is supplied with isFinalBlock set to false.

This tells the reader that the current buffer may not contain the entire document.

Step 2: Read the Available Values

The reader processes the array header and reads the first integer.

When it reaches the second integer, the reader determines whether sufficient input is available to continue.

Step 3: Check Whether More Data Is Required

Use PeekState() to inspect the next reader state without consuming the next value.

If the state is CborReaderState.NeedsMoreData, the application must provide another chunk before continuing.

Step 4: Supply the Next Chunk

Call SlideData with the additional bytes. Set isFinalBlock to true when the supplied block completes the document.

The reader can then continue parsing from its current position.

Practical C# Example

The following example demonstrates the incremental reading pattern introduced in .NET 11 RC 2.

using System;
using System.Formats.Cbor;

ReadOnlyMemory<byte> firstChunk =
    new byte[] { 0x82, 0x0A };

ReadOnlyMemory<byte> nextChunk =
    new byte[] { 0x14 };

var reader = new CborReader(
    firstChunk,
    new CborReaderOptions
    {
        ConformanceMode = CborConformanceMode.Lax
    },
    isFinalBlock: false);

reader.ReadStartArray();

int first = reader.ReadInt32();

if (reader.PeekState() == CborReaderState.NeedsMoreData)
{
    if (reader.BytesRemaining == 0)
    {
        reader.SlideData(
            nextChunk,
            isFinalBlock: true);
    }
    else
    {
        throw new InvalidOperationException(
            "Additional buffer management is required.");
    }
}

int second = reader.ReadInt32();

reader.ReadEndArray();

Console.WriteLine(first);
Console.WriteLine(second);

The expected output is:

10
20

The first byte, 0x82, represents an array containing two elements. The following byte, 0x0A, represents the integer 10. The final byte, 0x14, represents the integer 20.

The reader processes the first integer before the second chunk is supplied.

The example also checks BytesRemaining. This matters because a production implementation cannot assume that every incomplete read occurs exactly at the end of its current buffer. If unread bytes remain, the application must preserve them when managing the next input block.

This sample illustrates the API pattern with a deliberately simple message. A real network parser needs a more complete buffer-management strategy.

Handling Network Streams in Production

Incremental CBOR reading is especially relevant when a message arrives through a network connection.

A network read does not necessarily correspond to one complete CBOR value. A single message may arrive in several reads, and one read may contain bytes belonging to more than one logical value.

For that reason, a production parser should separate transport handling from CBOR decoding.

Manage Incomplete Input

Keep track of the bytes that have not yet been consumed. When additional input arrives, combine or slide the remaining data into the next reader buffer according to the API's buffer-management requirements.

Do not discard unread bytes simply because another network read has completed.

Handle End-of-Stream Correctly

If the transport ends before the CBOR document is complete, the application should report an incomplete message rather than treating it as a valid document.

Set isFinalBlock to true only when the application knows that no more bytes belong to the current document.

Validate the Parsed Data

Successful CBOR decoding does not mean that the input satisfies your application's business rules.

For example, a decoded sensor reading might still be outside the permitted range. Validate required fields, numeric limits, message structure, and any protocol-specific constraints before acting on the data.

Apply Input Limits

Incremental processing does not eliminate the possibility of oversized messages or deeply nested data.

Set appropriate limits on message size, nesting depth, and processing time. Reject malformed input early and avoid allocating large buffers based solely on untrusted length information.

Incremental Reading Compared With Complete-Buffer Reading

Consideration	Complete-buffer reading	Incremental reading
Input requirement	Complete encoded document	Input can arrive in chunks
Initial parsing	Starts after the complete buffer is available	Can start with an incomplete block
Buffer management	Generally simpler	Requires careful handling of unread data
Memory usage	May require the entire document in memory	Can reduce the need for a complete contiguous copy
Implementation complexity	Lower	Higher
Suitable scenarios	Small messages and files	Streamed or chunked input

Incremental reading is not automatically faster for every workload. For small messages, a complete-buffer approach may remain simpler and perfectly adequate.

The new APIs are most valuable when input naturally arrives in chunks or when avoiding an additional contiguous copy matters to the application.

Advantages and Disadvantages

Advantages

Earlier processing: Applications can begin decoding values before the entire document arrives.

Reduced buffering requirements: The parser does not need to receive the complete document in one contiguous memory block.

Better support for streamed data: The API fits workflows involving network messages and other chunked input sources.

Explicit incomplete-input handling: NeedsMoreData provides a clear indication that the reader requires additional bytes.

Disadvantages

More complicated buffer management: Applications must preserve unread bytes and provide subsequent blocks correctly.

Conformance restrictions: Incremental reading requires lax CBOR conformance mode.

No automatic transport integration: Developers still need to read from streams or network connections and manage the incoming data.

Validation remains necessary: The API decodes CBOR but does not replace application-level security and data validation.

Common Mistakes and Troubleshooting

The Reader Reports `NeedsMoreData`

This means the current input is insufficient to continue decoding. Obtain the next chunk and supply it through SlideData.

Do not attempt to read the next value until the required input is available.

The Parser Loses Data Between Chunks

Review the buffer-management logic. Any unread bytes must remain available when the next chunk is supplied.

Test with documents split at different byte positions, including boundaries inside strings, integers, and nested structures.

Incremental Reading Fails With a Different Conformance Mode

Use CborConformanceMode.Lax for incremental input. Other conformance modes reject non-final blocks.

If your application requires stricter conformance, evaluate whether you can buffer the complete document and validate it using the required mode.

The Application Still Uses Too Much Memory

Check whether the application accumulates every received chunk in a MemoryStream or another growing buffer before parsing.

Incremental decoding only reduces unnecessary buffering when the surrounding application also manages memory appropriately.

Summary

.NET 11 RC 2 adds incremental reading support to CborReader, allowing developers to process CBOR documents delivered in chunks without first copying the entire document into contiguous memory.

The key elements are the isFinalBlock constructor argument, CborReaderState.NeedsMoreData, and SlideData. Together, they allow a parser to read available values, request additional input, and continue decoding as more bytes arrive.

For production applications, the most important considerations are correct buffer management, proper end-of-stream handling, input limits, and application-level validation. Incremental reading is a useful option for streamed binary data, but complete-buffer parsing may remain the better choice for small and self-contained messages.