Retrieval-augmented generation (RAG) systems are often described in terms of embeddings, vector databases, and large language models. But one of the most important decisions happens before any of those components are used: how the source documents are divided into chunks.
If chunks are too large, retrieval may return a lot of irrelevant information. If they are too small, important context can be separated across multiple chunks. Traditional fixed-size chunking is easy to implement, but enterprise documents often have structure that does not align with arbitrary character or token boundaries.
Semantic chunking attempts to solve this problem by creating chunks around meaningful changes in content rather than simply cutting a document every N tokens.
For enterprise RAG systems, however, semantic chunking should not be adopted simply because it sounds more intelligent. It should be benchmarked against simpler approaches using the actual document corpus and retrieval workload.
This article explains how semantic chunking works, how it differs from fixed and structural chunking, and how to build a practical benchmark for enterprise document retrieval.
Introduction
Consider a 100-page technical document containing:
Architecture descriptions
API specifications
Configuration examples
Troubleshooting procedures
Security requirements
Frequently asked questions
A fixed chunking strategy might split the document every 800 tokens:
Document
|
+---- Chunk 1: 800 tokens
+---- Chunk 2: 800 tokens
+---- Chunk 3: 800 tokens
+---- Chunk 4: 800 tokens
...
The boundaries are convenient, but they may not correspond to the document's logical structure.
A semantic strategy attempts to identify related passages:
Document
|
+---- Authentication Concepts
|
+---- Authentication Configuration
|
+---- Authentication Troubleshooting
|
+---- Authorization Requirements
|
+---- API Security
The second approach can produce more meaningful retrieval units, but it also introduces additional processing, complexity, and tuning.
The right question is therefore not:
"Is semantic chunking better?"
It is:
"Which chunking strategy produces the best retrieval quality for this workload at an acceptable cost and latency?"
What Is Document Chunking?
Chunking is the process of dividing a document into smaller units that can be indexed and retrieved independently.
A typical RAG pipeline looks like this:
Documents
|
v
Parsing
|
v
Chunking
|
v
Embeddings
|
v
Vector Store
|
v
Retriever
|
v
LLM
|
v
Answer
The chunk becomes the basic retrieval unit.
If the chunk does not contain enough context to answer a question, retrieval may return technically relevant but practically incomplete information.
For example, imagine a document containing:
## Connection Pooling
The default pool size is 100.
## Configuration
The maximum pool size can be changed using:
MaxPoolSize=500
If the chunk boundary separates the configuration from the explanation, a question such as:
How do I increase the connection pool size?
may retrieve only the configuration value without enough surrounding context.
Fixed-Size Chunking
The simplest strategy is fixed-size chunking.
For example:
public static IEnumerable<string> ChunkBySize(
string text,
int chunkSize)
{
for (int i = 0; i < text.Length; i += chunkSize)
{
yield return text[
i..Math.Min(i + chunkSize, text.Length)];
}
}
In a real RAG pipeline, token-based chunking is generally more appropriate than character-based chunking because model context and pricing are token-oriented.
Fixed-size chunking has several advantages:
Simple implementation
Predictable chunk sizes
Low processing overhead
Easy benchmarking
Easy operational tuning
Its biggest weakness is that it does not understand document meaning.
Overlapping Chunks
A common improvement is to introduce overlap.
For example:
Chunk 1
|------------------------|
Chunk 2
|------------------------|
Chunk 3
|------------------------|
If the chunk size is 800 tokens and the overlap is 100 tokens, the next chunk begins before the previous one completely ends.
Overlap helps preserve context around boundaries.
However, it also increases:
The overlap should therefore be treated as a benchmark parameter rather than a universal best practice.
Structural Chunking
Many enterprise documents already contain useful structure.
Examples include:
Headings
Sections
Paragraphs
Tables
Lists
Code blocks
Page boundaries
Metadata fields
A structural chunker can use these boundaries.
For example:
Document
|
+-- Chapter
|
+-- Section
|
+-- Paragraph
+-- Paragraph
+-- Table
Instead of cutting text arbitrarily, the chunker attempts to preserve the document's logical organization.
This can be particularly useful for technical documentation, policies, manuals, and reports.
What Is Semantic Chunking?
Semantic chunking uses the meaning of adjacent passages to determine where one chunk should end and another should begin.
A simplified approach is:
Sentence 1
Sentence 2
Sentence 3
Sentence 4
|
v
Calculate semantic similarity
|
v
Detect topic transition
|
v
Create chunk boundary
Suppose the document contains:
The application uses OAuth 2.0.
Access tokens are issued by the identity provider.
Tokens expire after 60 minutes.
The database uses connection pooling.
Connections are reused across requests.
The semantic relationship between the first three sentences is stronger than the relationship between the third and fourth sentences.
A semantic chunker may therefore produce:
Chunk 1:
OAuth authentication and token expiration
Chunk 2:
Database connection pooling
The important point is that semantic chunking is not simply "larger chunks with embeddings." It uses semantic relationships to determine boundaries.
A Basic Semantic Similarity Approach
A simple implementation can represent each sentence as an embedding.
Conceptually:
var embeddings =
await embeddingService.GenerateAsync(sentences);
for (int i = 1; i < embeddings.Count; i++)
{
var similarity =
CosineSimilarity(
embeddings[i - 1],
embeddings[i]);
if (similarity < threshold)
{
CreateBoundary();
}
}
A simplified cosine similarity implementation is:
public static double CosineSimilarity(
IReadOnlyList<float> a,
IReadOnlyList<float> b)
{
double dot = 0;
double magnitudeA = 0;
double magnitudeB = 0;
for (int i = 0; i < a.Count; i++)
{
dot += a[i] * b[i];
magnitudeA += a[i] * a[i];
magnitudeB += b[i] * b[i];
}
if (magnitudeA == 0 || magnitudeB == 0)
{
return 0;
}
return dot /
(Math.Sqrt(magnitudeA) *
Math.Sqrt(magnitudeB));
}
This is a conceptual implementation. Production systems should also consider batching, embedding throughput, memory usage, and model-specific behavior.
The Chunking Threshold Matters
The semantic threshold determines when a topic transition is considered significant.
For example:
Similarity >= 0.85
Keep sentences together
Similarity < 0.85
Consider boundary
But 0.85 is not a universal value.
A threshold that works well for technical manuals may behave differently for:
Legal contracts
Financial reports
Product documentation
Support tickets
Research papers
This is one reason benchmarking is necessary.
Why Enterprise Documents Are Difficult
Enterprise documents are rarely plain paragraphs.
They may contain:
Heading
Paragraph
Table
Bullet list
Code sample
Warning
Footnote
Image
Caption
Appendix
Semantic chunking applied directly to extracted text can lose important relationships.
For example:
Table:
Status | Meaning
200 | Success
404 | Not Found
500 | Server Error
If the table is converted into an unstructured text sequence, the retrieval representation may become less useful.
Therefore, a serious benchmark should evaluate not just the chunking algorithm but also the document parsing and structure preservation pipeline.
Chunk Metadata Matters
A chunk should carry metadata describing where it came from.
For example:
public sealed record DocumentChunk(
string Id,
string Text,
string DocumentId,
string Section,
int PageNumber,
int ChunkIndex);
Metadata enables:
Filtering
Citation generation
Debugging
Document-level grouping
Section-aware retrieval
Retrieval analysis
For example:
Document: SecurityGuide.pdf
Page: 42
Section: Authentication
Chunk: 17
When a retrieved chunk produces a poor answer, this metadata makes the failure easier to investigate.
Designing the Benchmark
The benchmark should compare multiple strategies using the same corpus.
At minimum, consider:
Strategy A:
Fixed-size chunks
Strategy B:
Fixed-size + overlap
Strategy C:
Structural chunks
Strategy D:
Semantic chunks
Keep other variables constant where possible:
Embedding model
Vector store
Query set
Retrieval algorithm
Top-K
Generation model
Prompt
Evaluation criteria
Otherwise, it becomes difficult to determine whether a change in results actually came from chunking.
Build a Representative Evaluation Dataset
Create questions from real document content.
For example:
Question:
What is the default connection pool size?
Expected Source:
Database Configuration section
Question:
How is OAuth token expiration configured?
Expected Source:
Authentication section
Question:
What happens when a request receives HTTP 429?
Expected Source:
Rate Limiting section
The dataset should include both simple and difficult questions.
Useful categories include:
| Question Type | Example |
|---|
| Direct fact | What is the default timeout? |
| Configuration | How do I configure X? |
| Multi-part | What are the requirements for X and Y? |
| Cross-section | How does X affect Y? |
| Procedural | What steps are required? |
| Table-based | What does status code 429 mean? |
| Long-context | Summarize the authentication requirements |
Retrieval Metrics
The benchmark should measure retrieval separately from answer generation.
Useful metrics include:
Recall@K
Did the correct source appear in the top K retrieved chunks?
Recall@5 =
Relevant queries with answer source in top 5
------------------------------------------------
Total evaluated queries
Precision@K
How many retrieved chunks were actually relevant?
Precision@5 =
Relevant retrieved chunks
-------------------------
Total retrieved chunks
Mean Reciprocal Rank
MRR evaluates how high the first relevant result appears.
If the correct chunk is ranked first:
MRR = 1
If it is ranked fifth:
MRR = 0.2
These metrics help identify whether a chunking strategy improves retrieval ranking rather than simply increasing the number of returned results.
Answer-Level Evaluation
Retrieval quality is not the final goal.
The system ultimately needs to answer questions correctly.
Evaluate:
Correctness
Completeness
Groundedness
Citation accuracy
Context relevance
A useful benchmark structure is:
Question
|
+--> Retrieval Evaluation
|
+--> Context Evaluation
|
+--> Answer Evaluation
This separation is important.
A chunking strategy can improve Recall@10 but still produce worse answers if it returns too much irrelevant context.
Measure Chunk Statistics
Do not evaluate only retrieval accuracy.
Capture basic chunk statistics:
| Metric | Why It Matters |
|---|
| Chunk count | Storage and embedding cost |
| Average tokens | Context size |
| Median tokens | Typical retrieval size |
| p95 tokens | Large-chunk behavior |
| Overlap tokens | Duplicate content |
| Empty/invalid chunks | Pipeline quality |
| Metadata completeness | Filtering and debugging |
For example, two strategies might achieve similar retrieval quality, but one could produce twice as many chunks.
That difference matters operationally.
Benchmark Embedding Cost
Semantic chunking typically requires additional semantic processing before indexing.
A simplified cost model is:
Total Indexing Cost =
Parsing Cost
+
Chunking Cost
+
Embedding Cost
+
Storage Cost
If semantic chunking requires embeddings for many small units before constructing final chunks, the indexing pipeline can become more expensive.
That does not make semantic chunking a bad choice. It means the benchmark should measure the trade-off.
Benchmark Retrieval Latency
Measure retrieval latency under the same conditions.
At minimum, capture:
p50
p95
p99
A useful benchmark table is:
| Strategy | Recall@5 | MRR | Avg Tokens | p95 Latency | Indexing Cost |
|---|
| Fixed | Measure | Measure | Measure | Measure | Measure |
| Fixed + Overlap | Measure | Measure | Measure | Measure | Measure |
| Structural | Measure | Measure | Measure | Measure | Measure |
| Semantic | Measure | Measure | Measure | Measure | Measure |
Do not fill these values with assumed results. They should come from the actual corpus and infrastructure being evaluated.
A Practical Benchmark Harness
A simple abstraction can keep benchmark execution consistent.
public interface IChunkingStrategy
{
string Name { get; }
Task<IReadOnlyList<DocumentChunk>> ChunkAsync(
Document document,
CancellationToken cancellationToken);
}
Implementations could include:
FixedChunkingStrategy
OverlapChunkingStrategy
StructuralChunkingStrategy
SemanticChunkingStrategy
Then benchmark them using the same documents:
foreach (var strategy in strategies)
{
var stopwatch = Stopwatch.StartNew();
var chunks = await strategy.ChunkAsync(
document,
cancellationToken);
stopwatch.Stop();
results.Add(
new ChunkingBenchmarkResult(
strategy.Name,
chunks.Count,
stopwatch.Elapsed));
}
The benchmark harness should keep raw results so that experiments can be repeated and compared later.
Choosing Chunk Size
There is no universal ideal chunk size.
A useful experiment might evaluate several configurations:
Small
Medium
Large
For each configuration, compare:
Retrieval Quality
Answer Quality
Latency
Storage
Embedding Cost
The best configuration is the one that fits the application's workload, not the one with the smallest chunk or highest retrieval score in isolation.
Common Failure Modes
Chunks Are Too Small
Symptoms include:
Chunks Are Too Large
Symptoms include:
Semantic Boundaries Are Too Aggressive
The chunker may split closely related concepts into separate chunks.
Semantic Boundaries Are Too Weak
Unrelated topics may remain in the same chunk.
Tables Are Destroyed During Parsing
A chunking algorithm cannot recover semantic structure that was lost during document extraction.
Metadata Is Missing
Without document and section metadata, debugging retrieval failures becomes difficult.
Troubleshooting Poor Retrieval
When retrieval quality is low, do not immediately change the embedding model.
Inspect the pipeline in order:
Document Parsing
|
v
Chunk Boundaries
|
v
Chunk Metadata
|
v
Embeddings
|
v
Indexing
|
v
Query Processing
|
v
Retrieval Ranking
Ask:
Was the relevant content extracted correctly?
Does the relevant information exist inside a single chunk?
Are chunk boundaries separating related concepts?
Are metadata filters excluding the correct document?
Is the embedding representation appropriate?
Is Top-K too low?
Is reranking required?
Is the final LLM receiving too much irrelevant context?
This prevents chunking from becoming a catch-all explanation for every RAG problem.
Advantages of Semantic Chunking
Can preserve meaningful topic boundaries.
May improve retrieval for concept-oriented queries.
Can reduce arbitrary fragmentation of related content.
Can adapt better to documents with changing topics.
Provides a useful alternative to purely size-based chunking.
Disadvantages of Semantic Chunking
More complex than fixed-size chunking.
Requires additional processing.
Threshold selection can be difficult.
Results can vary by embedding model.
May perform poorly on highly structured documents if structure is discarded first.
Can increase indexing cost.
Requires benchmarking against simpler alternatives.
Best Practices
Benchmark semantic chunking against fixed and structural strategies.
Use the real enterprise document corpus whenever possible.
Preserve headings, tables, pages, and other useful metadata.
Evaluate retrieval and answer quality separately.
Measure Recall@K and MRR rather than relying only on subjective answers.
Track chunk size distributions.
Measure indexing cost and retrieval latency.
Tune semantic thresholds using evaluation data.
Avoid assuming that larger or smaller chunks are inherently better.
Keep chunking strategy configurable.
Store enough metadata to trace every retrieved chunk back to its source.
Test different document types independently.
Inspect failed retrieval cases manually.
Evaluate the complete RAG pipeline, not just the vector search layer.
Re-run benchmarks whenever the embedding model, parser, chunker, or retrieval strategy changes.
Frequently Asked Questions
Is semantic chunking always better than fixed-size chunking?
No. Semantic chunking can improve retrieval for some document types, but fixed-size or structural chunking may perform equally well or better for other workloads.
Does semantic chunking require an LLM?
Not necessarily. Semantic chunking can use embeddings and similarity measurements to identify topic transitions. More sophisticated implementations may use language models, but that adds processing cost and complexity.
What is the best chunk size for RAG?
There is no universal value. The appropriate size depends on document structure, query patterns, embedding model, context limits, and the desired balance between retrieval precision and context completeness.
Should I use overlap with semantic chunks?
Possibly, but overlap should be evaluated rather than assumed. Semantic boundaries already attempt to preserve related information, so excessive overlap can create unnecessary duplication.
How do I know whether chunking is causing poor RAG results?
Inspect failed queries and determine whether the required information is contained in retrieved chunks. Retrieval metrics such as Recall@K and MRR can help identify whether the correct source is being retrieved.
Should tables be chunked separately?
Often, yes. Tables can have different semantic and structural characteristics from normal prose. Preserving table structure can be more important than applying the same chunking strategy used for paragraphs.
Conclusion
Semantic chunking addresses an important weakness of fixed-size document segmentation: document meaning does not always follow arbitrary token boundaries. By detecting topic transitions and keeping related content together, semantic chunking can produce retrieval units that better reflect the underlying document.
However, semantic chunking should not become another assumption in the RAG pipeline. It introduces additional processing, configuration, and evaluation requirements. In some enterprise workloads, a carefully tuned fixed-size or structural strategy may provide an equally strong result with significantly less complexity.
The most reliable approach is to treat chunking as an engineering experiment. Build a representative evaluation dataset, compare multiple strategies using the same retrieval and generation pipeline, measure Recall@K, MRR, answer quality, latency, indexing cost, and chunk statistics, and inspect failed queries.
For enterprise RAG systems, the best chunking strategy is not the one with the most sophisticated algorithm. It is the one that consistently preserves the information users need, retrieves it efficiently, and operates within the application's cost and performance constraints.