Azure Content Understanding: Benchmarking Semantic Chunking for Enterprise RAG

Introduction

Retrieval-Augmented Generation, or RAG, is often described as a simple workflow: take a document, split it into smaller pieces, search for relevant pieces, and give those results to a language model.

In real enterprise applications, the difficult part is often the second step.

How should a document be divided before it is indexed?

If chunks are too small, important context can be lost. If they are too large, retrieval can return a lot of irrelevant information and increase the amount of content sent to the model.

Semantic chunking approaches this problem by considering the meaning and structure of the content rather than splitting a document only after a fixed number of characters or tokens.

Recent Azure Content Understanding capabilities have expanded document-processing workflows around semantic chunking, contextualization, and agentic document reasoning. The interesting engineering question is not simply whether semantic chunking is available, but whether it actually improves retrieval quality for an enterprise RAG workload.

The best way to answer that is through a controlled benchmark.

What Is Semantic Chunking?

Traditional chunking usually divides content according to a fixed rule.

For example:

Document
    |
    v
Every 500 tokens
    |
    +--> Chunk 1
    +--> Chunk 2
    +--> Chunk 3

This is straightforward, but the boundary may occur in the middle of an important idea.

Consider:

The customer can cancel the subscription within 30 days.

The cancellation request must be submitted through the
account portal and must include the original order number.

A fixed-size splitter might separate these sentences.

The first chunk contains the cancellation rule, while the second contains the process required to use that rule.

Semantic chunking attempts to keep related information together.

Conceptually:

Document
    |
    v
Understand Content
    |
    v
Identify Related Sections
    |
    +--> Policy Chunk
    +--> Procedure Chunk
    +--> Exception Chunk

The exact implementation depends on the document-processing system and configuration.

Why Chunking Matters in RAG

A typical enterprise RAG pipeline looks like this:

Documents
    |
    v
Content Processing
    |
    v
Chunking
    |
    v
Embeddings / Index
    |
    v
User Query
    |
    v
Retrieval
    |
    v
Relevant Context
    |
    v
Language Model
    |
    v
Answer

Chunking affects the retrieval stage directly.

If the correct information is split across unrelated chunks, the retrieval system may return only part of the required context.

That can lead to answers that are technically based on the source material but incomplete.

Fixed-Size vs Semantic Chunking

A useful benchmark should compare at least two strategies.

Area	Fixed-Size Chunking	Semantic Chunking
Boundary	Size-based	Meaning/structure-based
Implementation	Simple	More sophisticated
Context preservation	Can vary	Designed to preserve related content
Processing complexity	Lower	Higher
Retrieval quality	Workload-dependent	Workload-dependent
Cost	Generally easier to control	Requires additional processing
Best use	Predictable documents	Structurally rich documents

Neither approach should automatically be considered better.

The correct choice depends on the documents and retrieval questions.

Building a Benchmark Dataset

Start with a representative enterprise document collection.

Include different document types such as:

Policies
Product documentation
Technical manuals
Procedures
Contracts
Frequently asked questions
Internal knowledge articles

Avoid creating a benchmark from only short, clean documents.

A RAG system often struggles most with long documents containing multiple topics, tables, references, and exceptions.

A useful dataset might look like:

benchmark/
    policies/
    procedures/
    manuals/
    technical-docs/
    contracts/

The exact dataset should represent the workload the application will serve.

Creating Evaluation Questions

Documents alone are not enough.

You also need questions whose answers can be verified.

For example:

Document:
Employee Expense Policy

Question:
What is the maximum amount that can be claimed
for a hotel stay without additional approval?

Each test case should contain:

Document
Question
Expected Answer
Relevant Source Section

This gives the benchmark an objective reference.

Establishing a Baseline

Before testing semantic chunking, establish a baseline using the existing chunking strategy.

For example:

Documents
   |
   v
Fixed-Size Chunking
   |
   v
Index
   |
   v
Evaluation Questions

Record the results.

Then run the same dataset using semantic chunking:

Documents
   |
   v
Semantic Chunking
   |
   v
Index
   |
   v
Same Evaluation Questions

The only major variable should be the chunking strategy.

This makes the comparison much more meaningful.

Metrics to Measure

A useful benchmark should measure more than response quality.

Consider the following metrics:

Metric	What It Measures
Retrieval Recall	Whether relevant information was retrieved
Retrieval Precision	How much retrieved content was relevant
Answer Correctness	Whether the final answer is correct
Context Completeness	Whether required information was retrieved together
Chunk Count	Number of chunks produced
Average Chunk Size	Distribution of chunk sizes
Processing Time	Time required to create chunks
Index Size	Storage required
Token Usage	Context sent to the model
Cost	Overall processing and inference cost

Not every project needs every metric.

Choose the measurements that match the application's requirements.

Measuring Retrieval Recall

Suppose the correct answer exists in three source sections.

The retrieval system returns:

Section A
Section C
Section X
Section Y

Two of the three relevant sections were retrieved.

That means retrieval recall is incomplete.

A simple conceptual calculation is:

Recall =
Relevant Retrieved Items
-------------------------
All Relevant Items

For example:

2 / 3 = 0.67

This is an illustrative calculation rather than a benchmark result.

The important point is that semantic chunking should be evaluated by whether it helps the retriever find the information required to answer the question.

Measuring Retrieval Precision

Precision asks a different question.

If the system retrieves ten chunks and only three are relevant:

Precision =
3 / 10
=
0.30

High recall with very low precision can still create problems because the language model receives a large amount of irrelevant context.

A useful RAG system needs a practical balance between recall and precision.

Measuring Context Completeness

Some questions require multiple pieces of information.

Consider a policy:

Section 1:
Employees can claim travel expenses.

Section 2:
Claims above a certain amount require approval.

Section 3:
Approval must be obtained before reimbursement.

A question about the complete reimbursement process may require all three sections.

If retrieval returns only Section 1, the answer may be incomplete even though the retrieved chunk is technically relevant.

Semantic chunking can be particularly interesting for these multi-part questions.

Testing Tables

Enterprise documents often contain tables.

For example:

Expense Type	Maximum Amount	Approval
Hotel	Defined limit	Required above limit
Meals	Defined limit	Not normally required
Transport	Defined limit	Based on category

A chunking system should preserve the relationship between the table heading, rows, and surrounding explanation.

A benchmark should therefore include questions whose answers depend on table content.

For example:

"What approval is required when the hotel expense
exceeds the standard limit?"

The answer may require information from both the table and nearby explanatory text.

Testing Long Documents

Short documents can hide chunking problems.

Include long documents in the benchmark.

For example:

50-page policy
100-page technical manual
Large product specification
Long operational procedure

The exact sizes should reflect the application's real workload.

Test questions at different positions:

Beginning
Middle
End
Appendix
Referenced section

This helps determine whether retrieval quality changes depending on where information appears.

Testing Cross-Section Questions

Some of the most useful enterprise questions require information from multiple sections.

For example:

"What is the approval process for a purchase
that exceeds the standard limit?"

The spending limit might appear in one section while the approval workflow appears somewhere else.

These questions are useful because they test whether the chunking and retrieval pipeline preserves enough context for multi-step reasoning.

Benchmarking Chunk Size

Do not assume that semantic chunking produces the ideal chunk size automatically.

Measure the distribution.

For example:

Chunk 1 -> 320 tokens
Chunk 2 -> 540 tokens
Chunk 3 -> 410 tokens
Chunk 4 -> 890 tokens

Then calculate useful statistics:

Minimum chunk size
Maximum chunk size
Average size
Median size
Number of chunks
Percentage above the preferred context size

This helps identify whether semantic boundaries are producing practical retrieval units.

Measuring Index Growth

Different chunking strategies can produce different numbers of chunks.

For example:

Fixed Chunking
10,000 documents
       |
       v
100,000 chunks

while another strategy might produce:

Semantic Chunking
10,000 documents
       |
       v
65,000 chunks

These numbers are illustrative only.

The benchmark should measure actual output.

Fewer chunks do not automatically mean better retrieval.

The question is whether the chunks contain useful and retrievable information.

Measuring Processing Cost

Semantic processing can require additional computation.

Measure the time required to process the same dataset.

Record:

Dataset
Start Time
End Time
Chunk Count
Processing Duration

For example:

Processing Time =
End Time - Start Time

Run the benchmark more than once where practical and use a consistent environment.

Avoid reporting a single execution as a universal performance characteristic.

Measuring End-to-End Cost

The complete RAG pipeline can have several cost components:

Document Processing
        +
Chunking
        +
Indexing
        +
Retrieval
        +
Model Inference

A semantic approach could improve retrieval quality while increasing processing cost.

That does not automatically make it a bad choice.

For an internal knowledge system where documents are processed once and queried thousands of times, a higher ingestion cost may be reasonable if retrieval quality improves.

For constantly changing documents, the trade-off may look different.

Common Mistakes

Testing Only Answer Quality

A correct answer does not tell you why the system succeeded.

Measure retrieval quality separately.

Using Only One Document Type

Chunking behavior can vary significantly between policies, manuals, tables, and technical documents.

Using Synthetic Questions Only

Real user questions often contain ambiguity and incomplete terminology.

Include realistic queries.

Changing Multiple Variables

If chunking, embeddings, retrieval configuration, and model are all changed simultaneously, it becomes difficult to identify what caused the improvement.

Ignoring Processing Cost

A quality improvement may come with additional ingestion or indexing overhead.

Measure both.

Assuming Larger Chunks Are Better

Large chunks may preserve context but can introduce unnecessary information into the model context.

Troubleshooting Poor Retrieval

If semantic chunking produces poor results, investigate:

Chunk boundaries.
Chunk size distribution.
Document structure.
Metadata.
Embedding configuration.
Retrieval parameters.
Query formulation.
Tables and structured content.
Cross-section dependencies.
Source-document quality.

Inspect the actual retrieved chunks.

Do not evaluate only the final model response.

For example:

Question
   |
   v
Retrieved Chunks
   |
   +--> Relevant?
   +--> Complete?
   +--> Correct section?
   |
   v
Final Answer

This makes troubleshooting much easier.

Production Considerations

Before adopting semantic chunking across an enterprise RAG platform, evaluate the actual document lifecycle.

Ask:

How frequently do documents change?
How many documents need processing?
How expensive is reprocessing?
How large is the search index?
How important is retrieval accuracy?
Are documents mostly structured or unstructured?
Do users ask questions requiring multiple sections?
Are tables important?
Are citations required?

A strategy that works well for technical documentation may not behave the same way for contracts or financial policies.

Best Practices

Build a Representative Evaluation Set

Use documents and questions that reflect real production usage.

Separate Retrieval From Generation

Measure whether the correct information was retrieved before judging the final answer.

Include Difficult Questions

Test cross-section, table-based, and context-dependent questions.

Track Chunk Distribution

Measure how semantic processing changes chunk count and size.

Measure Quality and Cost Together

A better answer is useful, but the improvement should be considered alongside processing and inference cost.

Keep the Comparison Controlled

Change the chunking strategy while keeping other important variables stable.

Inspect Failed Cases

The most useful benchmark insights often come from questions where the system retrieves the wrong or incomplete context.

Advantages

Can preserve related information across natural document boundaries.
Can improve context completeness for complex documents.
Provides a potentially better foundation for enterprise RAG retrieval.
Can reduce the problem of arbitrary fixed-size boundaries.
Works well with documents whose structure carries important meaning.

Disadvantages

Processing can be more complex than simple fixed-size splitting.
Additional processing can increase ingestion cost.
Chunk sizes may become less predictable.
Results can vary significantly depending on document structure.
Semantic chunking does not guarantee better retrieval for every workload.

Conclusion

Chunking is one of the most important design decisions in an enterprise RAG pipeline, but it is often treated as a minor preprocessing step.

That is a mistake.

A chunk determines what information becomes available to the retrieval system and, ultimately, what context the language model receives.

Semantic chunking provides an opportunity to create more meaningful retrieval units by considering document structure and related content. However, its value should be demonstrated through measurement rather than assumed from the name of the feature.

A strong benchmark compares semantic and fixed-size chunking against the same documents and evaluation questions. Measure retrieval recall, precision, context completeness, chunk distribution, processing time, index size, and end-to-end cost.

Most importantly, inspect failed retrievals directly.

If semantic chunking consistently helps the system retrieve the right information while maintaining acceptable processing and storage costs, it can be a strong choice for enterprise RAG workloads. If the improvement is small or comes with significant operational overhead, a simpler chunking strategy may remain the better engineering decision.

The benchmark should make that decision based on evidence from the actual workload rather than assumptions about which chunking strategy sounds more sophisticated.