Benchmarking Chunking Strategies for Enterprise RAG Systems

The Developer Problem: Unoptimized Document Chunking in RAG Pipelines

When building enterprise Retrieval-Augmented Generation (RAG) systems, developers often focus primarily on selecting Large Language Models (LLMs) or vector database providers. However, the initial text-chunking strategy—how source documents are split into discrete segments before generating vector embeddings—has a massive impact on overall retrieval precision, contextual accuracy, and token costs.

Applying arbitrary or unmeasured chunking strategies introduces critical failure modes in production:

Context Fragmentation: Splitting text mid-sentence or cutting across logical section boundaries truncates critical context, causing the LLM to generate incomplete or inaccurate responses.
Semantic Dilution: Overly large chunks (e.g., 2,000+ tokens) pack too many disparate concepts into a single vector embedding, diluting specific semantic signals and lowering cosine similarity search relevance.
High Token Waste: Ingesting large, loosely relevant document chunks into LLM prompt context windows inflates API costs and increases generation latency.
Loss of Document Structure: Naive character-count splitters strip essential metadata—such as markdown headings, table relationships, and code block boundaries—rendering structured technical documentation difficult to retrieve accurately.

To build reliable enterprise RAG architectures, developers must evaluate chunking techniques empirically across measurable performance dimensions: Mean Reciprocal Rank (MRR@K), Context Precision, Context Recall, vector storage footprint, and retrieval latency.

Benchmarking Methodology: Evaluating Core Chunking Strategies

A robust evaluation framework benchmarks chunking strategies against a domain-specific gold-standard dataset using automated evaluation metrics.

┌─────────────────────────────────────────────────────────────┐
│                 Source Enterprise Documents                 │
└──────────────────────────────┬──────────────────────────────┘
                               │
                               ▼
┌─────────────────────────────────────────────────────────────┐
│                 Chunking Strategy Engine                    │
│   (Fixed-Size / Sentence / Markdown / Semantic Splitting)   │
└──────────────────────────────┬──────────────────────────────┘
                               │
                               ▼
┌─────────────────────────────────────────────────────────────┐
│              Vector Ingestion & K-NN Retrieval              │
└──────────────────────────────┬──────────────────────────────┘
                               │
                               ▼
┌─────────────────────────────────────────────────────────────┐
│                  RAG Evaluation Framework                   │
│   (Evaluates MRR@K, Context Precision, Recall, and Costs)   │
└─────────────────────────────────────────────────────────────┘

The table below contrasts popular document chunking strategies across standard engineering evaluation dimensions:

Chunking Strategy	Splitting Logic	Context Retention	Ingestion Overhead	Ideal Enterprise Use Case
Fixed-Size Chunking	Fixed character/token count with fixed overlap (e.g., 512 tokens, 50 overlap).	Low to Moderate; frequently cuts mid-sentence or mid-paragraph.	Ultra-Fast	Unstructured plain text logs or rapid prototyping.
Sentence-Based Chunking	Splits along sentence boundaries (., !, ?) up to a token limit.	Moderate; preserves sentence integrity but ignores document hierarchy.	Fast	Narrative prose, articles, and general correspondence.
Markdown / Structural Chunking	Respects document headers (#, ##), sections, tables, and code blocks.	High; preserves structural layout and semantic boundaries.	Moderate	Technical documentation, API specs, and wikis.
Semantic Distance Chunking	Uses embedding distance variance between adjacent sentences to detect topic shifts.	Very High; groups semantically cohesive sentences dynamically.	Slow (Requires embedding calls)	Complex financial reports, legal contracts, and dense policy manuals.

Implementing an Automated Chunking Benchmark Harness in .NET

The following step-by-step implementation demonstrates how to build a benchmarking engine in C# to compare Fixed-Size, Structural, and Sentence chunking strategies using Microsoft.Extensions.AI.

Step 1: Install Required Packages

Add the required AI extensions and text tokenization packages:

Bash

dotnet add package Microsoft.Extensions.AI
dotnet add package Microsoft.Extensions.AI.OpenAI
dotnet add package Microsoft.ML.Tokenizers

Step 2: Define Chunking Abstractions and Strategy Implementations

Create unified interfaces and strategy algorithms for splitting source text.

using System.Text.RegularExpressions;
using Microsoft.ML.Tokenizers;

public record TextChunk(string Content, int TokenCount, int ChunkIndex);

public interface IChunkingStrategy
{
    string StrategyName { get; }
    List<TextChunk> CreateChunks(string sourceText);
}

// 1. Fixed-Size Token Chunking Strategypublic class FixedSizeChunkingStrategy : IChunkingStrategy
{
    private readonly Tokenizer _tokenizer = Tokenizer.CreateTiktokenForModel("gpt-4o");
    private readonly int _chunkSize;
    private readonly int _overlap;

    public string StrategyName => $"FixedSize ({_chunkSize} tokens, {_overlap} overlap)";

    public FixedSizeChunkingStrategy(int chunkSize = 256, int overlap = 32)
    {
        _chunkSize = chunkSize;
        _overlap = overlap;
    }

    public List<TextChunk> CreateChunks(string sourceText)
    {
        var tokens = _tokenizer.EncodeToIds(sourceText);
        var chunks = new List<TextChunk>();
        int index = 0;

        for (int i = 0; i < tokens.Count; i += (_chunkSize - _overlap))
        {
            var segment = tokens.Skip(i).Take(_chunkSize).ToList();
            string text = _tokenizer.Decode(segment);
            chunks.Add(new TextChunk(text, segment.Count, index++));
            
            if (i + _chunkSize >= tokens.Count) break;
        }

        return chunks;
    }
}

// 2. Structural Markdown Chunking Strategypublic class MarkdownStructuralChunkingStrategy : IChunkingStrategy
{
    public string StrategyName => "Structural Markdown Splitting";

    public List<TextChunk> CreateChunks(string sourceText)
    {
        var chunks = new List<TextChunk>();
        // Split by Markdown headers (H1, H2, H3)
        var sections = Regex.Split(sourceText, @"(?=^#{1,3}\s)", RegexOptions.Multiline);
        int index = 0;

        foreach (var section in sections)
        {
            if (string.IsNullOrWhiteSpace(section)) continue;
            
            // Basic estimation or exact tokenizer count
            int tokenCount = section.Length / 4; 
            chunks.Add(new TextChunk(section.Trim(), tokenCount, index++));
        }

        return chunks;
    }
}

Step 3: Implement the Benchmark Evaluation Harness

Construct the evaluation engine to split source text across strategies, generate embeddings, and measure retrieval relevance against ground-truth queries.

using System.Diagnostics;
using System.Numerics.Tensors;
using Microsoft.Extensions.AI;

public record ChunkBenchmarkResult(
    string StrategyName,
    int TotalChunksGenerated,
    double AverageChunkTokenSize,
    double MeanReciprocalRank,
    double ProcessingTimeMs);

public class ChunkingBenchmarkEngine
{
    private readonly IEmbeddingGenerator<string, Embedding<float>> _embeddingGenerator;

    public ChunkingBenchmarkEngine(IEmbeddingGenerator<string, Embedding<float>> embeddingGenerator)
    {
        _embeddingGenerator = embeddingGenerator;
    }

    public async Task<ChunkBenchmarkResult> EvaluateStrategyAsync(
        IChunkingStrategy strategy,
        string sourceDocument,
        List<(string Query, string TargetSubString)> testQueries)
    {
        var sw = Stopwatch.StartNew();

        // 1. Generate Chunks
        var chunks = strategy.CreateChunks(sourceDocument);

        // 2. Embed Chunks
        var chunkTexts = chunks.Select(c => c.Content).ToList();
        var embeddings = await _embeddingGenerator.GenerateAsync(chunkTexts);
        sw.Stop();

        // 3. Evaluate Retrieval Precision (MRR)
        double totalReciprocalRank = 0.0;

        foreach (var (query, targetSubString) in testQueries)
        {
            var queryVector = (await _embeddingGenerator.GenerateAsync(new[] { query }))[0].Vector;

            // Compute Cosine Similarity against generated chunks
            var searchResults = embeddings.Select((emb, idx) => new
            {
                Chunk = chunks[idx],
                Similarity = TensorPrimitives.CosineSimilarity(queryVector.Span, emb.Vector.Span)
            })
            .OrderByDescending(x => x.Similarity)
            .ToList();

            // Find rank of the first chunk containing the target expected text
            for (int rank = 0; rank < searchResults.Count; rank++)
            {
                if (searchResults[rank].Chunk.Content.Contains(targetSubString, StringComparison.OrdinalIgnoreCase))
                {
                    totalReciprocalRank += 1.0 / (rank + 1);
                    break;
                }
            }
        }

        double mrr = totalReciprocalRank / testQueries.Count;
        double avgTokens = chunks.Count > 0 ? chunks.Average(c => c.TokenCount) : 0;

        return new ChunkBenchmarkResult(
            StrategyName: strategy.StrategyName,
            TotalChunksGenerated: chunks.Count,
            AverageChunkTokenSize: avgTokens,
            MeanReciprocalRank: mrr,
            ProcessingTimeMs: sw.ElapsedMilliseconds);
    }
}

Architectural Advantages and Disadvantages

Advantages

Data-Driven Chunk Sizing: Replaces guesswork with empirical metrics (MRR@K, context precision) specific to your document domain.
Optimized Storage and Compute: Identifies the smallest effective chunk size needed for accuracy, minimizing vector database hosting costs.
Reduced Hallucinations: Structurally coherent chunks ensure the LLM receives complete, un-truncated context during generation cycles.

Disadvantages

Increased Upfront Benchmarking Effort: Requires creating a representative set of test queries matched to source document sections.
Complex Pipeline Operations: Advanced strategies (like semantic distance splitting) introduce additional compute overhead during document ingestion pipelines.

Enterprise Best Practices

Match Strategy to Document Type: Use structural Markdown splitters for API specs and wikis, semantic distance splitters for legal agreements, and fixed-size splitters for raw event log streams.
Maintain Context Overlaps: When using fixed-size chunking, include a 10% to 15% token overlap between adjacent chunks to prevent cutting off key terms at boundary edges.
Prepend Parent Metadata Headers: When chunking sub-sections, attach higher-level document titles or section headings (e.g., [Document: Security Policy > Section: MFA Requirements]) to the top of each text chunk before embedding.
Evaluate Multi-Chunk Aggregation: Test whether returning two smaller 256-token chunks achieves better retrieval accuracy than returning a single monolithic 1,024-token chunk.

Common Mistakes to Avoid

Splitting Without Token Awareness: Splitting text by simple character count (string.Substring(0, 1000)) instead of using model-specific tokenizers leads to truncated sub-word tokens.
Ignoring Table Layouts: Running character-count splitters over Markdown or HTML tables destroys table row and column relationships. Always isolate tables into dedicated single chunks.
Over-Chunking Into Tiny Fragments: Generating tiny 32-token chunks strips surrounding context, making vectors non-distinct and degrading retrieval quality.

Troubleshooting Guide

Issue 1: High Retrieval Relevance But Incomplete LLM Answers

Root Cause: Chunks are too small (e.g., 64 tokens), so while the vector search locates the exact term, the retrieved text lacks the surrounding context necessary to answer the question completely.
Resolution: Increase chunk size (e.g., to 256 or 512 tokens) or implement a Parent-Child Retrieval Strategy (searching over small child chunks but returning the larger parent chunk to the LLM).

Issue 2: Low Cosine Similarity Scores Across All Queries

Root Cause: Chunks are too large (e.g., 2,000+ tokens), diluting specific topic vectors with unrelated text.
Resolution: Reduce maximum chunk token bounds and re-run retrieval benchmarks.

Issue 3: Ingestion Pipeline Out-of-Memory Errors

Root Cause: Semantic distance chunking algorithms generating excessive intermediate vector embeddings for huge single documents synchronously.
Resolution: Process large documents using batch streaming pipelines or combine sentence-grouping heuristics before generating vector embeddings.

Frequently Asked Questions (FAQs)

1. What is the most common chunk size for enterprise RAG applications?

A chunk size of 256 to 512 tokens with a 10% to 15% overlap serves as the standard baseline for general text documents. However, optimal sizing varies depending on your document structure and embedding model.

2. What is Parent-Child Chunking?

Parent-Child Chunking splits documents into small "child" chunks (e.g., 128 tokens) for fine-grained vector similarity search, but links each child to a larger "parent" chunk (e.g., 1,024 tokens) that is actually passed to the LLM context window.

3. How does Semantic Chunking work?

Semantic Chunking splits text into individual sentences, generates embeddings for each, calculates the cosine distance between consecutive sentences, and creates a new chunk boundary whenever the semantic distance spikes above a configured threshold.

Conclusion

Benchmarking chunking strategies transforms RAG document processing from an arbitrary setup step into a data-driven optimization process. By evaluating fixed-size, structural, and semantic chunking using a structured .NET evaluation harness, engineering teams can maximize retrieval accuracy, minimize prompt token waste, and build highly reliable enterprise AI applications.