Hybrid Retrieval (BM25 + Vector + Reranking) with LangGraph

When building Retrieval-Augmented Generation (RAG) systems, relying solely on Vector Search is a common trap. Vector databases are excellent at capturing semantic meaning (e.g., matching "canine companions" to "dogs"), but they often fail at exact keyword matching (e.g., searching for a specific error code like 0x8004DE40 or a product SKU).

Conversely, traditional keyword search (BM25) is perfect for exact matches but fails at semantic understanding.

The solution? Hybrid Retrieval. By combining BM25, Vector Search, and a Cross-Encoder Reranker, you get the best of both worlds. But orchestrating this efficiently—especially running the retrievers in parallel and managing the state—can get messy.

Enter LangGraph. In this article, we will build a Hybrid RAG pipeline using LangGraph to orchestrate parallel retrieval, merging, reranking, and generation.

The Real-World Use Case: SaaS IT Helpdesk Agent

Imagine you are building an AI IT Helpdesk for a company. The knowledge base contains a mix of:

Policy Documents: "How to request a new laptop" (Semantic)
Troubleshooting Guides: "Fixing Windows Error 0x8004DE40" (Keyword-heavy)

The User Query: "I'm getting Error 0x8004DE40 on my machine, and I think I just want to return it and get a new one."

Vector Search alone will find the return policy but might miss the specific troubleshooting guide for the error code.
BM25 alone will find the error code guide but might miss the broader return policy context.
Hybrid Search will fetch both, and the Reranker will sort them by actual relevance to the user's dual intent.

Architecture Overview

We will build a LangGraph State Machine with the following flow:

START
Parallel Fan-out:
- Node A: Vector Search
- Node B: BM25 Search
Fan-in (Merge): Combine and deduplicate documents from both retrievers.
Rerank: Pass the merged pool through a Cross-Encoder Reranker.
Generate: Pass the top reranked documents to the LLM.
END

Prerequisites

Install the required libraries. We will use FAISS for vectors, rank-bm25 for keyword search, and FlashRank for a fast, local, free reranker.

pip install langgraph langchain-openai langchain-community faiss-cpu rank-bm25 flashrank

Step 1: Setup and Mock Knowledge Base

First, let's create our mock IT Helpdesk knowledge base.

from langchain_core.documents import Document

# Mock Knowledge Base
documents = [
    Document(page_content="To return a defective laptop, fill out form HR-44 and mail it to the IT depot within 14 days.", metadata={"source": "return_policy.pdf"}),
    Document(page_content="Error 0x8004DE40 occurs when the Windows Hello camera driver is corrupted. Reinstall driver version 4.2.1 to fix.", metadata={"source": "win_hello_troubleshoot.md"}),
    Document(page_content="General laptop troubleshooting: 1. Restart the device. 2. Check for Windows Updates. 3. Run the hardware diagnostic tool.", metadata={"source": "general_it.md"}),
    Document(page_content="To request a new laptop, submit a ticket to the IT Helpdesk with your manager's approval.", metadata={"source": "hardware_request.pdf"}),
]

Step 2: Initialize the Retrievers

We need to set up both our Vector Retriever and our BM25 Retriever.

from langchain_community.vectorstores import FAISS
from langchain_community.retrievers import BM25Retriever
from langchain_openai import OpenAIEmbeddings
import os

# Set your OpenAI API key
os.environ["OPENAI_API_KEY"] = "your-api-key-here"

# 1. Vector Retriever
embeddings = OpenAIEmbeddings()
vectorstore = FAISS.from_documents(documents, embeddings)
vector_retriever = vectorstore.as_retriever(search_kwargs={"k": 3})

# 2. BM25 Retriever
bm25_retriever = BM25Retriever.from_documents(documents)
bm25_retriever.k = 3

Step 3: Define the LangGraph State and Nodes

LangGraph relies on a State object that is passed between nodes. We will define the state and the functions (nodes) that manipulate it.

from typing import TypedDict, Annotated
from langgraph.graph import StateGraph, START, END
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_community.cross_encoders import FlashrankRerank
import operator

# --- 1. Define the Graph State ---
class GraphState(TypedDict):
    query: str
    vector_docs: list[Document]
    bm25_docs: list[Document]
    merged_docs: list[Document]
    reranked_docs: list[Document]
    answer: str

# --- 2. Define the Nodes ---

def retrieve_vector(state: GraphState):
    """Node for Vector Search"""
    docs = vector_retriever.invoke(state["query"])
    return {"vector_docs": docs}

def retrieve_bm25(state: GraphState):
    """Node for BM25 Keyword Search"""
    docs = bm25_retriever.invoke(state["query"])
    return {"bm25_docs": docs}

def merge_and_deduplicate(state: GraphState):
    """Node to combine results and remove exact duplicates"""
    all_docs = state["vector_docs"] + state["bm25_docs"]
    
    # Deduplicate based on page_content
    seen = set()
    unique_docs = []
    for doc in all_docs:
        if doc.page_content not in seen:
            seen.add(doc.page_content)
            unique_docs.append(doc)
            
    return {"merged_docs": unique_docs}

def rerank_documents(state: GraphState):
    """Node to rerank the merged documents using FlashRank"""
    reranker = FlashrankRerank(top_n=3) # Keep top 3 after reranking
    
    # Flashrank expects a specific input format
    reranked = reranker.compress_documents(
        documents=state["merged_docs"], 
        query=state["query"]
    )
    return {"reranked_docs": reranked}

def generate_answer(state: GraphState):
    """Node to generate the final LLM response"""
    llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
    
    prompt = ChatPromptTemplate.from_template(
        """You are an IT Helpdesk AI. Answer the user's question based ONLY on the provided context.
        
        Context:
        {context}
        
        User Question: {query}
        
        Answer:"""
    )
    
    # Format context
    context = "\n\n".join([doc.page_content for doc in state["reranked_docs"]])
    
    chain = prompt | llm | StrOutputParser()
    answer = chain.invoke({"context": context, "query": state["query"]})
    
    return {"answer": answer}

Step 4: Build and Compile the LangGraph

Here is where LangGraph shines. We define the edges, including the parallel fan-out from the START node and the fan-in to the merge node.

# Initialize the StateGraph
workflow = StateGraph(GraphState)

# Add Nodes
workflow.add_node("vector_search", retrieve_vector)
workflow.add_node("bm25_search", retrieve_bm25)
workflow.add_node("merge", merge_and_deduplicate)
workflow.add_node("rerank", rerank_documents)
workflow.add_node("generate", generate_answer)

# Define Edges
# 1. Fan-out: START triggers both retrievers in parallel
workflow.add_edge(START, "vector_search")
workflow.add_edge(START, "bm25_search")

# 2. Fan-in: Both retrievers must finish before merging
workflow.add_edge("vector_search", "merge")
workflow.add_edge("bm25_search", "merge")

# 3. Sequential flow for the rest of the pipeline
workflow.add_edge("merge", "rerank")
workflow.add_edge("rerank", "generate")
workflow.add_edge("generate", END)

# Compile the graph
hybrid_rag_app = workflow.compile()

Step 5: Run the Pipeline

Let's test our graph with the tricky query that requires both semantic and keyword understanding.

# The tricky query
user_query = "I'm getting Error 0x8004DE40 on my machine, and I think I just want to return it and get a new one."

# Invoke the graph
result = hybrid_rag_app.invoke({"query": user_query})

print("--- FINAL ANSWER ---")
print(result["answer"])

print("\n--- RETRIEVED & RERANKED CONTEXT ---")
for i, doc in enumerate(result["reranked_docs"]):
    print(f"[{i+1}] {doc.page_content[:80]}... (Source: {doc.metadata['source']})")

Expected Output:

--- FINAL ANSWER ---
To resolve the Error 0x8004DE40, you need to reinstall the Windows Hello camera driver (version 4.2.1), as it indicates a corrupted driver. If you prefer to return the defective laptop instead of fixing it, you can do so by filling out form HR-44 and mailing it to the IT depot within 14 days.

--- RETRIEVED & RERANKED CONTEXT ---
[1] Error 0x8004DE40 occurs when the Windows Hello camera driver is corrupted. Reinstall driver version 4.2.1 to fix. (Source: win_hello_troubleshoot.md)
[2] To return a defective laptop, fill out form HR-44 and mail it to the IT depot within 14 days. (Source: return_policy.pdf)
[3] To request a new laptop, submit a ticket to the IT Helpdesk with your manager's approval. (Source: hardware_request.pdf)

Notice how the Reranker perfectly placed the exact error code fix at the top, followed by the specific return policy, pushing the generic "request a new laptop" document to the bottom.

Why use LangGraph for this instead of standard LangChain LCEL?

You might wonder why we didn't just use LangChain's EnsembleRetriever. While EnsembleRetriever is great, LangGraph provides distinct advantages for production systems:

True Parallel Execution: In LangGraph, defining multiple edges from START executes vector_search and bm25_search concurrently, cutting your retrieval latency in half.
State Visibility & Debugging: Every intermediate step (the raw vector docs, the raw BM25 docs, the merged list) is saved in the GraphState. You can easily inspect why the reranker made its choices.
Conditional Routing: If you want to add a step later like, "If the reranked documents have a score below 0.5, trigger a web search instead," LangGraph makes adding conditional edges trivial. LCEL chains become incredibly brittle when adding complex conditional logic.
Human-in-the-Loop: LangGraph natively supports pausing the graph, allowing a human IT agent to review the retrieved documents before the LLM generates the final email to the customer.

Conclusion

By combining BM25 for exact keyword matching, Vector Search for semantic understanding, and a Cross-Encoder for precise sorting, you eliminate the biggest blind spots in standard RAG. Wrapping this in LangGraph ensures your pipeline is fast (parallel execution), robust, and ready for complex production routing.