Benchmarking Semantic Chunking for Enterprise Document Retrieval

Retrieval-augmented generation (RAG) systems are often described in terms of embeddings, vector databases, and large language models. But one of the most important decisions happens before any of those components are used: how the source documents are divided into chunks.

If chunks are too large, retrieval may return a lot of irrelevant information. If they are too small, important context can be separated across multiple chunks. Traditional fixed-size chunking is easy to implement, but enterprise documents often have structure that does not align with arbitrary character or token boundaries.

Semantic chunking attempts to solve this problem by creating chunks around meaningful changes in content rather than simply cutting a document every N tokens.

For enterprise RAG systems, however, semantic chunking should not be adopted simply because it sounds more intelligent. It should be benchmarked against simpler approaches using the actual document corpus and retrieval workload.

This article explains how semantic chunking works, how it differs from fixed and structural chunking, and how to build a practical benchmark for enterprise document retrieval.

Introduction

Consider a 100-page technical document containing:

Architecture descriptions
API specifications
Configuration examples
Troubleshooting procedures
Security requirements
Frequently asked questions

A fixed chunking strategy might split the document every 800 tokens:

Document
   |
   +---- Chunk 1: 800 tokens
   +---- Chunk 2: 800 tokens
   +---- Chunk 3: 800 tokens
   +---- Chunk 4: 800 tokens
   ...

The boundaries are convenient, but they may not correspond to the document's logical structure.

A semantic strategy attempts to identify related passages:

Document
   |
   +---- Authentication Concepts
   |
   +---- Authentication Configuration
   |
   +---- Authentication Troubleshooting
   |
   +---- Authorization Requirements
   |
   +---- API Security

The second approach can produce more meaningful retrieval units, but it also introduces additional processing, complexity, and tuning.

The right question is therefore not:

"Is semantic chunking better?"

It is:

"Which chunking strategy produces the best retrieval quality for this workload at an acceptable cost and latency?"

What Is Document Chunking?

Chunking is the process of dividing a document into smaller units that can be indexed and retrieved independently.

A typical RAG pipeline looks like this:

Documents
    |
    v
Parsing
    |
    v
Chunking
    |
    v
Embeddings
    |
    v
Vector Store
    |
    v
Retriever
    |
    v
LLM
    |
    v
Answer

The chunk becomes the basic retrieval unit.

If the chunk does not contain enough context to answer a question, retrieval may return technically relevant but practically incomplete information.

For example, imagine a document containing:

## Connection Pooling

The default pool size is 100.

## Configuration

The maximum pool size can be changed using:

MaxPoolSize=500

If the chunk boundary separates the configuration from the explanation, a question such as:

How do I increase the connection pool size?

may retrieve only the configuration value without enough surrounding context.

Fixed-Size Chunking

The simplest strategy is fixed-size chunking.

For example:

public static IEnumerable<string> ChunkBySize(
    string text,
    int chunkSize)
{
    for (int i = 0; i < text.Length; i += chunkSize)
    {
        yield return text[
            i..Math.Min(i + chunkSize, text.Length)];
    }
}

In a real RAG pipeline, token-based chunking is generally more appropriate than character-based chunking because model context and pricing are token-oriented.

Fixed-size chunking has several advantages:

Simple implementation
Predictable chunk sizes
Low processing overhead
Easy benchmarking
Easy operational tuning

Its biggest weakness is that it does not understand document meaning.

Overlapping Chunks

A common improvement is to introduce overlap.

For example:

Chunk 1
|------------------------|
            Chunk 2
            |------------------------|
                        Chunk 3
                        |------------------------|

If the chunk size is 800 tokens and the overlap is 100 tokens, the next chunk begins before the previous one completely ends.

Overlap helps preserve context around boundaries.

However, it also increases:

Number of chunks
Embedding operations
Storage requirements
Retrieval candidates
Potential duplicate context

The overlap should therefore be treated as a benchmark parameter rather than a universal best practice.

Structural Chunking

Many enterprise documents already contain useful structure.

Examples include:

Headings
Sections
Paragraphs
Tables
Lists
Code blocks
Page boundaries
Metadata fields

A structural chunker can use these boundaries.

For example:

Document
  |
  +-- Chapter
       |
       +-- Section
            |
            +-- Paragraph
            +-- Paragraph
            +-- Table

Instead of cutting text arbitrarily, the chunker attempts to preserve the document's logical organization.

This can be particularly useful for technical documentation, policies, manuals, and reports.

What Is Semantic Chunking?

Semantic chunking uses the meaning of adjacent passages to determine where one chunk should end and another should begin.

A simplified approach is:

Sentence 1
Sentence 2
Sentence 3
Sentence 4
       |
       v
Calculate semantic similarity
       |
       v
Detect topic transition
       |
       v
Create chunk boundary

Suppose the document contains:

The application uses OAuth 2.0.

Access tokens are issued by the identity provider.

Tokens expire after 60 minutes.

The database uses connection pooling.

Connections are reused across requests.

The semantic relationship between the first three sentences is stronger than the relationship between the third and fourth sentences.

A semantic chunker may therefore produce:

Chunk 1:
OAuth authentication and token expiration

Chunk 2:
Database connection pooling

The important point is that semantic chunking is not simply "larger chunks with embeddings." It uses semantic relationships to determine boundaries.

A Basic Semantic Similarity Approach

A simple implementation can represent each sentence as an embedding.

Conceptually:

var embeddings =
    await embeddingService.GenerateAsync(sentences);

for (int i = 1; i < embeddings.Count; i++)
{
    var similarity =
        CosineSimilarity(
            embeddings[i - 1],
            embeddings[i]);

    if (similarity < threshold)
    {
        CreateBoundary();
    }
}

A simplified cosine similarity implementation is:

public static double CosineSimilarity(
    IReadOnlyList<float> a,
    IReadOnlyList<float> b)
{
    double dot = 0;
    double magnitudeA = 0;
    double magnitudeB = 0;

    for (int i = 0; i < a.Count; i++)
    {
        dot += a[i] * b[i];
        magnitudeA += a[i] * a[i];
        magnitudeB += b[i] * b[i];
    }

    if (magnitudeA == 0 || magnitudeB == 0)
    {
        return 0;
    }

    return dot /
        (Math.Sqrt(magnitudeA) *
         Math.Sqrt(magnitudeB));
}

This is a conceptual implementation. Production systems should also consider batching, embedding throughput, memory usage, and model-specific behavior.

The Chunking Threshold Matters

The semantic threshold determines when a topic transition is considered significant.

For example:

Similarity >= 0.85
    Keep sentences together

Similarity < 0.85
    Consider boundary

But 0.85 is not a universal value.

A threshold that works well for technical manuals may behave differently for:

Legal contracts
Financial reports
Product documentation
Support tickets
Research papers

This is one reason benchmarking is necessary.

Why Enterprise Documents Are Difficult

Enterprise documents are rarely plain paragraphs.

They may contain:

Heading
Paragraph
Table
Bullet list
Code sample
Warning
Footnote
Image
Caption
Appendix

Semantic chunking applied directly to extracted text can lose important relationships.

For example:

Table:
Status | Meaning
200    | Success
404    | Not Found
500    | Server Error

If the table is converted into an unstructured text sequence, the retrieval representation may become less useful.

Therefore, a serious benchmark should evaluate not just the chunking algorithm but also the document parsing and structure preservation pipeline.

Chunk Metadata Matters

A chunk should carry metadata describing where it came from.

For example:

public sealed record DocumentChunk(
    string Id,
    string Text,
    string DocumentId,
    string Section,
    int PageNumber,
    int ChunkIndex);

Metadata enables:

Filtering
Citation generation
Debugging
Document-level grouping
Section-aware retrieval
Retrieval analysis

For example:

Document: SecurityGuide.pdf
Page: 42
Section: Authentication
Chunk: 17

When a retrieved chunk produces a poor answer, this metadata makes the failure easier to investigate.

Designing the Benchmark

The benchmark should compare multiple strategies using the same corpus.

At minimum, consider:

Strategy A:
Fixed-size chunks

Strategy B:
Fixed-size + overlap

Strategy C:
Structural chunks

Strategy D:
Semantic chunks

Keep other variables constant where possible:

Embedding model
Vector store
Query set
Retrieval algorithm
Top-K
Generation model
Prompt
Evaluation criteria

Otherwise, it becomes difficult to determine whether a change in results actually came from chunking.

Build a Representative Evaluation Dataset

Create questions from real document content.

For example:

Question:
What is the default connection pool size?

Expected Source:
Database Configuration section

Question:
How is OAuth token expiration configured?

Expected Source:
Authentication section

Question:
What happens when a request receives HTTP 429?

Expected Source:
Rate Limiting section

The dataset should include both simple and difficult questions.

Useful categories include:

Question Type	Example
Direct fact	What is the default timeout?
Configuration	How do I configure X?
Multi-part	What are the requirements for X and Y?
Cross-section	How does X affect Y?
Procedural	What steps are required?
Table-based	What does status code 429 mean?
Long-context	Summarize the authentication requirements

Retrieval Metrics

The benchmark should measure retrieval separately from answer generation.

Useful metrics include:

Recall@K

Did the correct source appear in the top K retrieved chunks?

Recall@5 =
Relevant queries with answer source in top 5
------------------------------------------------
Total evaluated queries

Precision@K

How many retrieved chunks were actually relevant?

Precision@5 =
Relevant retrieved chunks
-------------------------
Total retrieved chunks

Mean Reciprocal Rank

MRR evaluates how high the first relevant result appears.

If the correct chunk is ranked first:

MRR = 1

If it is ranked fifth:

MRR = 0.2

These metrics help identify whether a chunking strategy improves retrieval ranking rather than simply increasing the number of returned results.

Answer-Level Evaluation

Retrieval quality is not the final goal.

The system ultimately needs to answer questions correctly.

Evaluate:

Correctness
Completeness
Groundedness
Citation accuracy
Context relevance

A useful benchmark structure is:

Question
   |
   +--> Retrieval Evaluation
   |
   +--> Context Evaluation
   |
   +--> Answer Evaluation

This separation is important.

A chunking strategy can improve Recall@10 but still produce worse answers if it returns too much irrelevant context.

Measure Chunk Statistics

Do not evaluate only retrieval accuracy.

Capture basic chunk statistics:

Metric	Why It Matters
Chunk count	Storage and embedding cost
Average tokens	Context size
Median tokens	Typical retrieval size
p95 tokens	Large-chunk behavior
Overlap tokens	Duplicate content
Empty/invalid chunks	Pipeline quality
Metadata completeness	Filtering and debugging

For example, two strategies might achieve similar retrieval quality, but one could produce twice as many chunks.

That difference matters operationally.

Benchmark Embedding Cost

Semantic chunking typically requires additional semantic processing before indexing.

A simplified cost model is:

Total Indexing Cost =
Parsing Cost
+
Chunking Cost
+
Embedding Cost
+
Storage Cost

If semantic chunking requires embeddings for many small units before constructing final chunks, the indexing pipeline can become more expensive.

That does not make semantic chunking a bad choice. It means the benchmark should measure the trade-off.

Benchmark Retrieval Latency

Measure retrieval latency under the same conditions.

At minimum, capture:

p50
p95
p99

A useful benchmark table is:

Strategy	Recall@5	MRR	Avg Tokens	p95 Latency	Indexing Cost
Fixed	Measure	Measure	Measure	Measure	Measure
Fixed + Overlap	Measure	Measure	Measure	Measure	Measure
Structural	Measure	Measure	Measure	Measure	Measure
Semantic	Measure	Measure	Measure	Measure	Measure

Do not fill these values with assumed results. They should come from the actual corpus and infrastructure being evaluated.

A Practical Benchmark Harness

A simple abstraction can keep benchmark execution consistent.

public interface IChunkingStrategy
{
    string Name { get; }

    Task<IReadOnlyList<DocumentChunk>> ChunkAsync(
        Document document,
        CancellationToken cancellationToken);
}

Implementations could include:

FixedChunkingStrategy
OverlapChunkingStrategy
StructuralChunkingStrategy
SemanticChunkingStrategy

Then benchmark them using the same documents:

foreach (var strategy in strategies)
{
    var stopwatch = Stopwatch.StartNew();

    var chunks = await strategy.ChunkAsync(
        document,
        cancellationToken);

    stopwatch.Stop();

    results.Add(
        new ChunkingBenchmarkResult(
            strategy.Name,
            chunks.Count,
            stopwatch.Elapsed));
}

The benchmark harness should keep raw results so that experiments can be repeated and compared later.

Choosing Chunk Size

There is no universal ideal chunk size.

A useful experiment might evaluate several configurations:

Small
Medium
Large

For each configuration, compare:

Retrieval Quality
Answer Quality
Latency
Storage
Embedding Cost

The best configuration is the one that fits the application's workload, not the one with the smallest chunk or highest retrieval score in isolation.

Common Failure Modes

Chunks Are Too Small

Symptoms include:

Missing context
Incomplete answers
Many retrieved chunks required
High duplication

Chunks Are Too Large

Symptoms include:

Irrelevant context
Larger prompts
Increased generation cost
Lower context precision

Semantic Boundaries Are Too Aggressive

The chunker may split closely related concepts into separate chunks.

Semantic Boundaries Are Too Weak

Unrelated topics may remain in the same chunk.

Tables Are Destroyed During Parsing

A chunking algorithm cannot recover semantic structure that was lost during document extraction.

Metadata Is Missing

Without document and section metadata, debugging retrieval failures becomes difficult.

Troubleshooting Poor Retrieval

When retrieval quality is low, do not immediately change the embedding model.

Inspect the pipeline in order:

Document Parsing
      |
      v
Chunk Boundaries
      |
      v
Chunk Metadata
      |
      v
Embeddings
      |
      v
Indexing
      |
      v
Query Processing
      |
      v
Retrieval Ranking

Ask:

Was the relevant content extracted correctly?
Does the relevant information exist inside a single chunk?
Are chunk boundaries separating related concepts?
Are metadata filters excluding the correct document?
Is the embedding representation appropriate?
Is Top-K too low?
Is reranking required?
Is the final LLM receiving too much irrelevant context?

This prevents chunking from becoming a catch-all explanation for every RAG problem.

Advantages of Semantic Chunking

Can preserve meaningful topic boundaries.
May improve retrieval for concept-oriented queries.
Can reduce arbitrary fragmentation of related content.
Can adapt better to documents with changing topics.
Provides a useful alternative to purely size-based chunking.

Disadvantages of Semantic Chunking

More complex than fixed-size chunking.
Requires additional processing.
Threshold selection can be difficult.
Results can vary by embedding model.
May perform poorly on highly structured documents if structure is discarded first.
Can increase indexing cost.
Requires benchmarking against simpler alternatives.

Best Practices

Benchmark semantic chunking against fixed and structural strategies.
Use the real enterprise document corpus whenever possible.
Preserve headings, tables, pages, and other useful metadata.
Evaluate retrieval and answer quality separately.
Measure Recall@K and MRR rather than relying only on subjective answers.
Track chunk size distributions.
Measure indexing cost and retrieval latency.
Tune semantic thresholds using evaluation data.
Avoid assuming that larger or smaller chunks are inherently better.
Keep chunking strategy configurable.
Store enough metadata to trace every retrieved chunk back to its source.
Test different document types independently.
Inspect failed retrieval cases manually.
Evaluate the complete RAG pipeline, not just the vector search layer.
Re-run benchmarks whenever the embedding model, parser, chunker, or retrieval strategy changes.

Frequently Asked Questions

Is semantic chunking always better than fixed-size chunking?

No. Semantic chunking can improve retrieval for some document types, but fixed-size or structural chunking may perform equally well or better for other workloads.

Does semantic chunking require an LLM?

Not necessarily. Semantic chunking can use embeddings and similarity measurements to identify topic transitions. More sophisticated implementations may use language models, but that adds processing cost and complexity.

What is the best chunk size for RAG?

There is no universal value. The appropriate size depends on document structure, query patterns, embedding model, context limits, and the desired balance between retrieval precision and context completeness.

Should I use overlap with semantic chunks?

Possibly, but overlap should be evaluated rather than assumed. Semantic boundaries already attempt to preserve related information, so excessive overlap can create unnecessary duplication.

How do I know whether chunking is causing poor RAG results?

Inspect failed queries and determine whether the required information is contained in retrieved chunks. Retrieval metrics such as Recall@K and MRR can help identify whether the correct source is being retrieved.

Should tables be chunked separately?

Often, yes. Tables can have different semantic and structural characteristics from normal prose. Preserving table structure can be more important than applying the same chunking strategy used for paragraphs.

Conclusion

Semantic chunking addresses an important weakness of fixed-size document segmentation: document meaning does not always follow arbitrary token boundaries. By detecting topic transitions and keeping related content together, semantic chunking can produce retrieval units that better reflect the underlying document.

However, semantic chunking should not become another assumption in the RAG pipeline. It introduces additional processing, configuration, and evaluation requirements. In some enterprise workloads, a carefully tuned fixed-size or structural strategy may provide an equally strong result with significantly less complexity.

The most reliable approach is to treat chunking as an engineering experiment. Build a representative evaluation dataset, compare multiple strategies using the same retrieval and generation pipeline, measure Recall@K, MRR, answer quality, latency, indexing cost, and chunk statistics, and inspect failed queries.

For enterprise RAG systems, the best chunking strategy is not the one with the most sophisticated algorithm. It is the one that consistently preserves the information users need, retrieves it efficiently, and operates within the application's cost and performance constraints.