Tuning Chunking, Retrieval, and LangGraph for Institutional-Grade AI

In the financial sector, a hallucination isn't just a bad user experience—it can lead to regulatory fines, poor investment decisions, and millions of dollars in losses. When building Retrieval-Augmented Generation (RAG) systems for financial documents like 10-Ks, 10-Qs, and earnings call transcripts, standard "out-of-the-box" RAG pipelines fail.

Financial documents are dense, highly structured, and rely heavily on exact numerical precision and domain-specific jargon.

In this end-to-end guide, we will explore how to tune chunk size, overlap, and retriever parameters specifically for financial documents. We will then build a Real-Time Use Case using LangGraph, moving beyond simple linear RAG into a stateful, self-correcting agentic workflow.

The Real-Time Use Case: The "Alpha Analyst" Assistant

The Scenario: You are building an AI assistant for a hedge fund. The user asks a complex, multi-hop question:

"Compare NVIDIA's gross margin guidance given in the Q4 2025 earnings call with the actual gross margin reported in the Q1 2026 10-Q. Did they beat guidance, and what management cited as the primary driver for the variance?"

The Challenge:

The answer requires cross-referencing two different document types (Transcript vs. SEC Filing).
It requires exact number retrieval (margins).
It requires semantic understanding of management's reasoning.

A standard RAG pipeline will likely retrieve a chunk about NVIDIA's general revenue, miss the specific margin numbers, and hallucinate the "primary driver." We need a tuned, agentic approach.

Step 1: Tuning Chunk Size and Overlap for Finance

Standard RAG tutorials suggest chunk sizes of 500 tokens with a 50-token overlap. In finance, this is a recipe for disaster.

1. The Chunk Size Dilemma

Financial documents contain massive tables (e.g., Consolidated Statements of Operations) and long, legally binding caveats.

Too small (e.g., 300 tokens): You sever the table header from the data rows. The LLM sees "15,400" but doesn't know if it's Revenue, COGS, or Net Income.
Too large (e.g., 2000+ tokens): You dilute the embedding vector. If a chunk contains three different quarters' data, the vector representation becomes a "mush" of conflicting contexts, ruining retrieval precision.

The Financial Sweet Spot: 800 to 1200 tokens.
This is large enough to keep a standard financial table intact with its headers, but small enough to maintain high vector precision.

2. The Overlap Strategy

Overlap prevents the "mid-sentence cut-off" problem. In legal/financial text, a single sentence can contain a critical caveat (e.g., "Revenue grew 20%, excluding the impact of the new supply chain tariffs..."). If you cut right before "excluding", the LLM will draw the wrong conclusion.

The Financial Sweet Spot: 10% to 15% overlap (approx. 100-150 tokens).

3. The Secret Weapon: Markdown-Aware & Parent-Child Chunking

For financial docs, do not use naive RecursiveCharacterTextSplitter.

Convert to Markdown first: Use tools like unstructured or LlamaParse to convert PDFs to Markdown. This preserves table structures and headers.
Use Parent-Child (Small-to-Big) Retrieval: Chunk the document into small, highly precise embeddings (e.g., 300 tokens) for retrieval, but map them back to a larger parent chunk (1000 tokens) to pass to the LLM for generation. This gives you the best of both worlds: pinpoint retrieval and rich context.

Step 2: Tuning Retriever Parameters

Dense vector embeddings (like OpenAI's text-embedding-3-large) are great at semantic meaning, but they are terrible at exact keyword matching. If a user asks for "EBITDA", a pure vector search might return a chunk about "Operating Income" because they are semantically similar, but financially distinct.

1. Hybrid Search (BM25 + Dense Vector)

You must use Hybrid Search for financial documents. BM25 (keyword search) ensures exact matches for tickers (NVDA), acronyms (EBITDA, GAAP), and specific financial terms, while Dense vectors capture the semantic intent.

Tuning: Set the weighting to 0.3 BM25 / 0.7 Dense (or use Reciprocal Rank Fusion - RRF).

2. Top-K and Score Thresholds

Top-K for Retrieval: Fetch 10 to 15 documents initially. Financial queries are often broad; casting a wider net ensures the right data is in the pool.
Score Threshold: Implement a similarity threshold (e.g., score > 0.75). If the highest-scoring document is below this, the retriever should trigger a query rewrite rather than passing garbage to the LLM.

3. Metadata Filtering

Financial datasets are massive. Always attach metadata to your chunks: {"ticker": "NVDA", "doc_type": "10-Q", "date": "2026-05-15", "section": "MD&A"}.
Force the retriever to filter by doc_type and date before doing the vector search. This reduces noise by 90%.

4. Reranking (The Game Changer)

After retrieving 10 chunks via Hybrid Search, pass them through a Cross-Encoder Reranker (like Cohere Rerank 3 or BGE-Reranker-v2). Rerankers are computationally heavy but highly accurate. They will re-sort the 10 chunks and allow you to confidently pass only the Top 3 to the LLM, saving context window space and reducing hallucinations.

Step 3: Building the LangGraph RAG Pipeline

Linear RAG (Retrieve -> Generate) fails when the initial retrieval misses the mark. LangGraph allows us to build a cyclic graph where the system can grade its own retrieval and rewrite the query if necessary.

Here is the architecture for our "Alpha Analyst" LangGraph pipeline:

Retrieve: Fetch documents using Hybrid Search + Metadata.
Grade Documents: An LLM grades if the retrieved chunks actually contain the financial data requested.
Route:
- If Relevant -> Go to Generate.
- If Irrelevant -> Go to Rewrite Query.
Rewrite Query: Reformulate the query to be more specific (e.g., adding "gross margin" and "Q1 2026").
Generate: Produce the final answer with strict citations.

The Python Implementation

import os
from typing import List, TypedDict, Literal
from langchain_core.documents import Document
from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain.retrievers import EnsembleRetriever
from langchain_community.retrievers import BM25Retriever
from langchain_community.vectorstores import FAISS
from langgraph.graph import END, StateGraph

# --- 1. Define the Graph State ---
class GraphState(TypedDict):
    question: str
    generation: str
    documents: List[Document]
    iterations: int

# --- 2. Initialize Components (Mocked for brevity) ---
# In production, load your FAISS index and BM25 index from disk/DB
vectorstore = FAISS.load_local("faiss_financial_index", OpenAIEmbeddings())
vector_retriever = vectorstore.as_retriever(search_kwargs={"k": 10})

# Mock BM25 retriever (requires pre-computed corpus)
bm25_retriever = BM25Retriever.from_documents(documents_corpus, k=10)

# Hybrid Retriever (Ensemble)
hybrid_retriever = EnsembleRetriever(
    retrievers=[bm25_retriever, vector_retriever],
    weights=[0.3, 0.7]
)

llm = ChatOpenAI(model="gpt-4o", temperature=0)

# --- 3. Define the Nodes ---

def retrieve_documents(state: GraphState):
    """Retrieves documents using the hybrid retriever."""
    print("---RETRIEVING DOCUMENTS---")
    question = state["question"]
    
    # Apply metadata filtering here in production!
    # e.g., vectorstore.as_retriever(search_kwargs={"filter": {"ticker": "NVDA"}})
    documents = hybrid_retriever.invoke(question)
    
    return {"documents": documents, "iterations": state.get("iterations", 0) + 1}

def grade_documents(state: GraphState):
    """Determines if the retrieved documents are relevant to the financial question."""
    print("---GRADING DOCUMENTS---")
    question = state["question"]
    documents = state["documents"]
    
    grading_prompt = ChatPromptTemplate.from_template(
        """You are a financial analyst grading the relevance of retrieved document chunks 
        to a user's question. 
        Question: {question}
        Document: {doc}
        
        Does the document contain specific financial data, numbers, or context to answer the question?
        Answer 'yes' or 'no'."""
    )
    
    # Simple grading logic (in production, use structured output / JSON mode)
    relevant_docs = []
    for doc in documents:
        grade = (grading_prompt | llm).invoke({"question": question, "doc": doc.page_content})
        if "yes" in grade.content.lower():
            relevant_docs.append(doc)
            
    return {"documents": relevant_docs}

def rewrite_query(state: GraphState):
    """Rewrites the question to improve retrieval if initial docs were poor."""
    print("---REWRITING QUERY---")
    question = state["question"]
    
    rewrite_prompt = ChatPromptTemplate.from_template(
        """The initial retrieval for the question failed to find relevant financial data.
        Original Question: {question}
        Rewrite the question to be more specific, focusing on exact financial terms, 
        tickers, and timeframes (e.g., 'Q1 2026 gross margin'). 
        Return ONLY the rewritten question."""
    )
    
    new_question = (rewrite_prompt | llm).invoke({"question": question})
    return {"question": new_question.content}

def generate_answer(state: GraphState):
    """Generates the final financial analysis based on relevant documents."""
    print("---GENERATING ANSWER---")
    question = state["question"]
    documents = state["documents"]
    
    context = "\n\n".join([doc.page_content for doc in documents])
    
    gen_prompt = ChatPromptTemplate.from_template(
        """You are an expert financial analyst. Answer the question based ONLY on the provided context.
        If the exact numbers or reasoning are not in the context, state clearly that the data is unavailable.
        Do not hallucinate financial figures.
        
        Context: {context}
        Question: {question}
        
        Answer:"""
    )
    
    generation = (gen_prompt | llm).invoke({"context": context, "question": question})
    return {"generation": generation.content}

# --- 4. Define Routing Logic ---

def route_query(state: GraphState) -> Literal["generate", "rewrite"]:
    """Decides whether to generate an answer or rewrite the query."""
    documents = state["documents"]
    iterations = state.get("iterations", 0)
    
    # If no relevant docs found, and we haven't looped too many times, rewrite
    if len(documents) == 0 and iterations < 2:
        return "rewrite"
    return "generate"

# --- 5. Build the LangGraph ---

workflow = StateGraph(GraphState)

# Add nodes
workflow.add_node("retrieve", retrieve_documents)
workflow.add_node("grade", grade_documents)
workflow.add_node("rewrite", rewrite_query)
workflow.add_node("generate", generate_answer)

# Set entry point
workflow.set_entry_point("retrieve")

# Add edges
workflow.add_edge("retrieve", "grade")
workflow.add_conditional_edges(
    "grade",
    route_query,
    {
        "generate": "generate",
        "rewrite": "rewrite",
    },
)
workflow.add_edge("rewrite", "retrieve") # Loop back to retrieve with new query
workflow.add_edge("generate", END)

# Compile the graph
app = workflow.compile()

# --- 6. Run the Pipeline ---
if __name__ == "__main__":
    initial_query = "Did NVIDIA beat its gross margin guidance from the last earnings call and why?"
    
    result = app.invoke({
        "question": initial_query,
        "documents": [],
        "iterations": 0
    })
    
    print("\n--- FINAL ANSWER ---")
    print(result["generation"])

Step 4: Evaluation and Iteration

You cannot tune what you do not measure. For financial RAG, standard metrics like BLEU or ROUGE are useless. You must use an LLM-as-a-judge framework like Ragas or TruLens.

Focus on these three metrics:

Context Precision: Did the retriever actually find the chunk containing the Q1 2026 margin? (If this is low, tune your Hybrid Search weights or Chunk Size).
Faithfulness: Did the LLM's answer strictly adhere to the retrieved numbers without hallucinating? (If this is low, lower the LLM temperature to 0 and strengthen your generation prompt).
Answer Relevance: Did the final answer actually address the user's specific question about the variance driver?

Conclusion

Building RAG for financial documents requires a paradigm shift from "good enough" consumer chatbots to institutional-grade precision.

By utilizing larger, markdown-aware chunk sizes (800-1200 tokens) with 10-15% overlap, you preserve the structural integrity of financial tables. By implementing Hybrid Search and Reranking, you ensure exact financial terminology is never lost in semantic "mush." Finally, by orchestrating the pipeline with LangGraph, you give your system the ability to self-correct, ensuring that when the first retrieval fails, the agent intelligently rewrites its query until it finds the exact financial truth. In finance, AI isn't just about speed; it's about accuracy. Tuning these parameters is the bridge between a cool demo and a deployable financial product.