Introduction
In the evolution of Retrieval-Augmented Generation (RAG), the industry has moved past the "Naive RAG" phase. We quickly learned that simply chunking documents, embedding them, and fetching the top-K results leads to a critical bottleneck: Retrieval Noise.
When your vector database returns irrelevant, contradictory, or out-of-context chunks, the LLM’s context window gets polluted. This results in hallucinations, confused reasoning, and frustrated users.
To solve this, we need to move from linear pipelines to Agentic RAG. By using LangGraph, we can introduce self-correction, evaluation loops, and dynamic routing into our retrieval process.
In this end-to-end article, we will explore how to reduce retrieval noise using a real-world, real-time use case: an Enterprise SaaS Technical Support Assistant.
The Real-Time Use Case: "CloudFlow" Support Assistant
The Scenario
CloudFlow is a B2B SaaS platform for data engineering. They process thousands of live support tickets daily. The engineering team built a Naive RAG bot to deflect Tier-1 tickets.
The Problem: Retrieval Noise in the Wild
In a real-time production environment, the Naive RAG bot was failing.
Semantic Ambiguity
A user asks:
"Why is my pipeline failing with a 404 error?"
The vector database retrieves documentation about the 404 Not Found error for the Web Dashboard UI, but the user is actually asking about the Data Ingestion API.
Version Drift
The user is on CloudFlow v3.2, but the retriever fetches highly relevant (but obsolete) troubleshooting steps for v1.0.
Context Bleed
A retrieved chunk contains the solution, but also includes three paragraphs of unrelated background theory, pushing the actual solution out of the LLM's attention span.
The Goal
Build a real-time LangGraph pipeline that:
Intercepts noisy retrievals
Grades retrieved documents
Dynamically rewrites queries
Applies metadata filtering
Prevents irrelevant context from reaching the LLM
The Strategy: How LangGraph Reduces Noise
LangGraph allows us to model our RAG pipeline as a cyclic graph with state.
To eliminate retrieval noise, we implement four key nodes:
Query Routing & Rewriting
Standardize user queries and inject implicit context, such as the user's software version.
Contextual Retrieval
Retrieve documents using strict metadata filtering.
Retrieval Grader (The Noise Filter)
Use an LLM-based evaluator to score each retrieved chunk and discard irrelevant content.
Hallucination Grader
Ensure the final generated answer is grounded entirely in the cleaned context.
End-to-End Implementation with LangGraph
1. Setup and State Definition
First, define the graph state. The state carries the user question, retrieved documents, and generated answer through the workflow.
from typing import List, TypedDict, Literal
from langchain_core.documents import Document
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.pydantic_v1 import BaseModel, Field
from langgraph.graph import StateGraph, END
# Define the Graph State
class GraphState(TypedDict):
question: str
generation: str
documents: List[Document]
iterations: int # Prevent infinite loops
2. The Nodes: Building the Noise-Reduction Engine
Node A: Query Rewriting (Pre-Retrieval)
Users often ask vague questions. We rewrite them into highly specific search queries and enrich them with real-time user metadata.
def rewrite_query(state: GraphState):
"""Rewrites the user query to be more specific for the vector DB."""
question = state["question"]
# In production, fetch user metadata dynamically
user_context = "User is on CloudFlow v3.2, Enterprise Tier."
prompt = ChatPromptTemplate.from_template(
"You are a query rewriter. Convert the following user question "
"into a specific, optimized search query for a vector database. "
"Include this context: {context}\n\n"
"Original Question: {question}"
)
chain = prompt | llm
rewritten = chain.invoke({
"context": user_context,
"question": question
})
return {
"question": rewritten.content,
"iterations": state.get("iterations", 0) + 1
}
Node B: Retrieval (with Metadata Filtering)
Strict metadata filtering prevents version-drift issues.
def retrieve_documents(state: GraphState):
"""Retrieves documents with metadata filtering."""
question = state["question"]
docs = vectorstore.similarity_search(
question,
k=5,
filter={
"version": "3.2",
"doc_type": "troubleshooting"
}
)
return {"documents": docs}
Node C: The Retrieval Grader (The Core Noise Filter)
This node is the heart of the solution.
Vector similarity alone cannot determine actual relevance. We use an LLM or reranker model to grade retrieved documents and filter out noise.
Relevance Scoring Model
class GradeDocuments(BaseModel):
"""Binary relevance score."""
binary_score: str = Field(
description="Documents are relevant to the question, 'yes' or 'no'"
)
Document Grading Node
def grade_documents(state: GraphState):
"""Filters irrelevant retrievals."""
question = state["question"]
documents = state["documents"]
llm_with_structured_output = (
llm.with_structured_output(
GradeDocuments
)
)
prompt = ChatPromptTemplate.from_template(
"You are a grader assessing relevance "
"of a retrieved document to a user question.\n"
"Retrieved document:\n\n{document}\n\n"
"User question: {question}\n\n"
"If the document contains keywords or semantic meaning "
"related to the user question, grade it as relevant.\n"
"Return only 'yes' or 'no'."
)
grading_chain = (
prompt
| llm_with_structured_output
)
filtered_docs = []
for d in documents:
score = grading_chain.invoke({
"question": question,
"document": d.page_content
})
if score.binary_score == "yes":
filtered_docs.append(d)
return {"documents": filtered_docs}
Node D: Generation and Hallucination Control
Once the noisy documents are removed, the final answer is generated from the cleaned context.
def generate_answer(state: GraphState):
"""Generate answer from clean context."""
question = state["question"]
documents = state["documents"]
context = "\n\n".join([
doc.page_content
for doc in documents
])
prompt = ChatPromptTemplate.from_template(
"Answer the question based ONLY on "
"the following context:\n"
"{context}\n\n"
"Question: {question}"
)
chain = prompt | llm
generation = chain.invoke({
"context": context,
"question": question
})
return {
"generation": generation.content
}
3. Wiring the LangGraph Pipeline
Now we connect all nodes using conditional routing.
If every retrieved document is graded as noise, the workflow loops back to query rewriting.
If relevant documents remain, the workflow proceeds to answer generation.
Initialize LLM
llm = ChatOpenAI(
model="gpt-4o",
temperature=0
)
Conditional Routing Logic
def decide_to_generate(state: GraphState):
"""Determine next action."""
filtered_documents = state["documents"]
iterations = state.get("iterations", 0)
if not filtered_documents:
if iterations < 2:
return "rewrite"
return "generate_empty"
return "generate"
Build the Graph
workflow = StateGraph(GraphState)
# Nodes
workflow.add_node(
"rewrite_query",
rewrite_query
)
workflow.add_node(
"retrieve",
retrieve_documents
)
workflow.add_node(
"grade_documents",
grade_documents
)
workflow.add_node(
"generate",
generate_answer
)
# Entry Point
workflow.set_entry_point(
"rewrite_query"
)
# Flow
workflow.add_edge(
"rewrite_query",
"retrieve"
)
workflow.add_edge(
"retrieve",
"grade_documents"
)
# Noise Reduction Loop
workflow.add_conditional_edges(
"grade_documents",
decide_to_generate,
{
"rewrite": "rewrite_query",
"generate": "generate",
"generate_empty": END
}
)
workflow.add_edge(
"generate",
END
)
# Compile
app = workflow.compile()
Productionizing for Real-Time Performance
Running agentic workflows inside customer support systems introduces strict latency requirements.
Users will not wait 15 seconds for an answer.
Use Smaller Models for Grading
Do not use GPT-4o for relevance grading.
Prefer:
GPT-4o Mini
bge-reranker-v2-m3
Cross-Encoder rerankers
This can reduce grading latency from approximately 800ms to 150ms.
Streaming Responses
LangGraph supports streaming execution.
Benefits:
Semantic Caching
Store:
Rewritten queries
Filtered document IDs
Using Redis or similar caches prevents repeated grading loops for common issues.
Early Exit Fallbacks
Notice the following condition:
if iterations < 2
This prevents infinite rewrite-retrieve loops.
When the system cannot find relevant documents after multiple attempts:
Escalate to a human agent
Provide fallback guidance
Avoid user-facing timeouts
The Impact: Before and After
When CloudFlow deployed this Agentic RAG architecture, key metrics improved significantly.
Retrieval Precision
The retrieval grader successfully filtered irrelevant UI documentation when API documentation was needed.
Hallucination Rate
Removing contradictory and outdated chunks dramatically improved answer quality.
Ticket Deflection Rate
More customer questions were resolved without human intervention.
Conclusion
Retrieval noise is one of the biggest challenges in production RAG systems.
Vector databases excel at finding similar content, but similarity alone does not guarantee relevance. Without additional controls, irrelevant context pollutes the LLM's reasoning process and increases hallucinations.
By leveraging LangGraph, we move beyond simple retrieval pipelines and create an Agentic RAG workflow that actively evaluates, filters, and improves retrieved context before generation occurs.
Through:
Query rewriting
Metadata filtering
Retrieval grading
Conditional routing
Self-correction loops
we can dramatically improve retrieval quality and answer accuracy.
In enterprise environments where trust and correctness matter, Agentic RAG is no longer an optional enhancement—it is a core architectural requirement.