Chunking Strategies for Enterprise RAG

When building Retrieval-Augmented Generation (RAG) systems, junior developers obsess over the LLM. Senior engineers obsess over the chunking strategy.

You can use the most advanced LLM in the world, but if your chunking strategy destroys the semantic and structural context of your documents, your RAG pipeline will fail. In large, complex domain knowledge bases, naive chunking (like blindly splitting text every 500 tokens) is a death sentence for accuracy.

Today, we will explore how to choose and implement an advanced chunking strategy for massive, highly structured domain knowledge, and how to orchestrate the retrieval of those chunks using LangGraph.

The Real-Time Use Case: Pharmaceutical Manufacturing & Regulatory Compliance

Imagine you are the Lead AI Architect for a global pharmaceutical company. You are building a Manufacturing Copilot for plant engineers and quality assurance (QA) teams.

The Knowledge Base:

15,000+ pages of FDA regulatory guidelines.
5,000+ Standard Operating Procedures (SOPs) for drug synthesis and packaging.
Highly structured data: Nested headings, complex parameter tables, and strict hierarchical workflows.

The Problem:
An engineer asks: "What is the maximum temperature tolerance for storing Compound X during Phase 2 packaging, and what is the required QA sign-off step?"

If you use a naive RecursiveCharacterTextSplitter, the text is chopped alphabetically. The "Storage Requirements" table gets separated from the "Phase 2 Packaging" heading. The LLM receives a fragmented table without context and hallucinates a temperature, or worse, misses the QA sign-off requirement entirely. In pharma, this isn't just a bad user experience; it's a compliance violation.

Choosing the Winning Chunking Strategy

For large, structured domain knowledge bases, we must abandon naive splitting. We need a hybrid approach: Structure-Aware Chunking combined with Parent-Child (Auto-Merging) Retrieval.

1. Structure-Aware Chunking (The Ingestion Phase)

Instead of splitting by character count, we split by document structure. We use Markdown headers to understand the hierarchy.

A chunk isn't just a block of text; it retains its metadata: {"H1": "Phase 2 Packaging", "H2": "Storage Requirements"}.
This ensures that a table about temperature is permanently bound to the context of "Phase 2 Packaging".

2. Parent-Child / Auto-Merging (The Retrieval Phase)

This is the secret weapon for enterprise RAG.

Child Chunks (Small - ~250 tokens): We embed these small, highly specific chunks into the Vector Database. This ensures high precision in semantic search.
Parent Chunks (Large - ~1500 tokens): We store the larger, context-rich chunks in a fast Key-Value store (like Redis).
The Magic: When a user queries the system, we search the Child chunks. But instead of passing the tiny child chunk to the LLM, we fetch its corresponding Parent chunk. The LLM gets the precise match plus the full surrounding context.

End-to-End Architecture with LangGraph

While ingestion is often a batch process, the retrieval and generation of Parent-Child chunks requires complex state management. This is where LangGraph shines.

We will build a LangGraph RAG pipeline that:

Retrieves the precise Child chunks.
Expands the context by fetching the Parent chunks.
Grades the expanded context for relevance.
Generates the final answer.

Step 1: Define the LangGraph State

from typing import TypedDict, List, Literal
from langchain_core.documents import Document

class PharmaRAGState(TypedDict):
    query: str
    child_documents: List[Document]      # Small, precise chunks from Vector DB
    parent_documents: List[Document]     # Large, context-rich chunks from KV Store
    final_context: List[Document]        # The context actually passed to the LLM
    answer: str

Step 2: The Ingestion Strategy (Conceptual Setup)

Before querying, we set up the Parent-Child index using LangChain's ParentDocumentRetriever.

from langchain.retrievers import ParentDocumentRetriever
from langchain.storage import InMemoryStore # Use Redis in production
from langchain_text_splitters import MarkdownHeaderTextSplitter, RecursiveCharacterTextSplitter

# 1. Structure-aware splitter (Headers)
header_splitter = MarkdownHeaderTextSplitter(headers_to_split_on=[("#", "H1"), ("##", "H2")])

# 2. Child splitter (Small chunks for precise vector search)
child_splitter = RecursiveCharacterTextSplitter(chunk_size=250, chunk_overlap=20)

# Initialize the Parent-Child Retriever
parent_child_retriever = ParentDocumentRetriever(
    vectorstore=vector_db,
    docstore=InMemoryStore(), # Maps child IDs to Parent chunks
    child_splitter=child_splitter,
    parent_splitter=header_splitter
)
# Add documents to the pipeline
parent_child_retriever.add_documents(raw_pharma_docs)

Step 3: Building the LangGraph Query Pipeline

Now, we orchestrate the query flow. Notice how we explicitly separate the retrieval of child chunks from the expansion of parent chunks.

from langgraph.graph import StateGraph, START, END
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model="gpt-4o", temperature=0)

# Node 1: Retrieve precise Child Chunks
def retrieve_child_chunks(state: PharmaRAGState):
    # This hits the Vector DB using the small child chunks
    child_docs = parent_child_retriever.vectorstore.similarity_search(state["query"], k=4)
    return {"child_documents": child_docs}

# Node 2: Expand to Parent Chunks (The Auto-Merging Magic)
def expand_to_parent_context(state: PharmaRAGState):
    child_docs = state["child_documents"]
    parent_docs = []
    
    for child in child_docs:
        # Fetch the large parent chunk from the KV store using the child's metadata
        parent_id = child.metadata.get("parent_doc_id")
        if parent_id:
            parent_doc = parent_child_retriever.docstore.mget([parent_id])[0]
            # Deduplicate in case multiple child chunks belong to the same parent
            if parent_doc not in parent_docs:
                parent_docs.append(parent_doc)
                
    return {"parent_documents": parent_docs}

# Node 3: Context Grading (Self-Correction)
def grade_context(state: PharmaRAGState):
    parent_docs = state["parent_documents"]
    query = state["query"]
    
    # Use a fast LLM to ensure the expanded parent chunks actually contain the answer
    # If the parent chunk is too broad and dilutes the context, we might need to route differently
    relevant_docs = []
    for doc in parent_docs:
        # Simplified grading logic for the example
        if query.lower() in doc.page_content.lower() or "temperature" in doc.page_content.lower():
            relevant_docs.append(doc)
            
    return {"final_context": relevant_docs if relevant_docs else parent_docs}

# Node 4: Generate Answer
def generate_answer(state: PharmaRAGState):
    context = "\n\n".join([doc.page_content for doc in state["final_context"]])
    
    prompt = f"""You are an expert Pharmaceutical QA Engineer. 
    Answer the question based ONLY on the provided SOP context. 
    If the answer is not in the context, state 'Information not found in SOPs'.
    
    Context: {context}
    Question: {state["query"]}"""
    
    response = llm.invoke(prompt)
    return {"answer": response.content}

Step 4: Compile the Graph

workflow = StateGraph(PharmaRAGState)

# Add Nodes
workflow.add_node("retrieve_children", retrieve_child_chunks)
workflow.add_node("expand_parents", expand_to_parent_context)
workflow.add_node("grade_context", grade_context)
workflow.add_node("generate", generate_answer)

# Define Edges
workflow.add_edge(START, "retrieve_children")
workflow.add_edge("retrieve_children", "expand_parents")
workflow.add_edge("expand_parents", "grade_context")
workflow.add_edge("grade_context", "generate")
workflow.add_edge("generate", END)

# Compile the App
rag_app = workflow.compile()

Execution: How it Solves the Pharma Use Case

Let's trace our engineer's query: "What is the maximum temperature tolerance for storing Compound X during Phase 2 packaging?"

Retrieve Children: The Vector DB finds a small child chunk containing the exact table row: "Compound X | Max Temp: 4°C | Phase 2".
Expand Parents: LangGraph takes that child chunk, finds its parent_doc_id, and pulls the full 1500-token Parent chunk. This parent chunk includes the crucial context: "Phase 2 Packaging requires QA sign-off by the Shift Supervisor before storage."
Grade Context: The grader confirms both the temperature data and the QA sign-off requirement are present in the expanded context.
Generate: The LLM generates a perfect, compliant answer: "The maximum temperature tolerance for Compound X during Phase 2 packaging is 4°C. Additionally, Phase 2 packaging requires mandatory QA sign-off by the Shift Supervisor prior to storage."

Output:

Chunking Strategies for Enterprise RAG-1

Chunking Strategies for Enterprise RAG-2

Production Metrics & The Senior Engineer's Takeaway

By shifting from naive chunking to Structure-Aware Parent-Child Chunking orchestrated via LangGraph, the pharmaceutical company achieved:

Context Retention: 98% (Up from 62%). Tables and headers were no longer fragmented.
Hallucination Rate: Dropped to near zero. The Parent chunks provided the strict boundaries the LLM needed to stay grounded.
Retrieval Precision: Maintained high precision because the Vector DB was still searching on small, highly specific child chunks.

The Takeaway

Choosing a chunking strategy is not just an ingestion task; it dictates your entire retrieval architecture. Naive chunking allows for naive, linear RAG chains. But advanced chunking (like Parent-Child) requires a state machine to manage the retrieval expansion and context grading.

By combining structural chunking with LangGraph, you bridge the gap between precise vector search and rich, contextual generation, turning your RAG pipeline into a true enterprise-grade asset.