How to Build a RAG Evaluation Pipeline With Python: Metrics, RAGAS, and Testing
A retrieval-augmented generation (RAG) application can appear to work correctly while producing answers that are incomplete, irrelevant, or unsupported by the documents it retrieves.
Consider an AI-powered technical support assistant. A customer asks how long deleted workspace data is retained. The system retrieves a document about account deletion but misses the policy describing data retention. The language model then generates a convincing answer based on incomplete information
The response might sound reasonable, yet it could be wrong.
This is why a RAG evaluation pipeline matters. Testing whether an application generates readable answers is not enough. Developers also need to measure whether the correct information was retrieved, whether the response follows that information, and whether it answers the original question.
In this guide, we will explore important RAG evaluation metrics, build a small evaluation dataset, implement automated RAG evaluation with Python and Ragas, and examine how to use evaluation results during development.
Why RAG Evaluation Matters in Production ?
A RAG system typically retrieves relevant information from an external knowledge source and passes that information to a language model to generate an answer.
Although retrieval improves access to domain-specific knowledge, it does not guarantee that the final response will be correct.
Several problems can occur during this process:
Retrieval failure: The system retrieves documents that do not contain the information needed to answer the question.
Missing evidence: Relevant information exists in the knowledge base but is excluded from the retrieved context.
Hallucination: The language model generates claims that are not supported by the retrieved material.
Poor answer relevance: The response discusses a related topic without addressing the user's actual question.
Quality regression: Changes to prompts, embeddings, chunking, or retrieval settings reduce performance.
For example, imagine a customer asks:
"How long is deleted workspace data retained?"
The correct policy might state that deleted workspace data is retained for 30 days before permanent deletion.
If the retriever returns only general account-deletion instructions, the language model might generate an unsupported retention period.
A useful evaluation process should identify whether the problem originated in retrieval, answer generation, or both.
How Do You Evaluate a RAG Pipeline in Python?
To evaluate a RAG pipeline in Python, create a dataset containing user questions, retrieved document contexts, generated responses, and reference answers. Use a framework such as Ragas to measure faithfulness, context precision, context recall, and answer relevancy. Analyze individual scores, investigate failures, and repeat the evaluation after making changes to the system.
Understanding the Key RAG Evaluation Metrics
RAG evaluation requires examining two different parts of the system: retrieval quality and response-generation quality.
A system may retrieve excellent documents but generate an unsupported response. Conversely, a language model may produce a reasonable answer despite receiving incomplete evidence.
Four metrics help distinguish these situations.
Faithfulness measures whether the claims in a generated response are supported by the retrieved context.
Suppose the retrieved document states:
"Customers may request a refund within 14 days of purchase."
But the AI responds:
"Customers can request a refund within 30 days."
The response contradicts the retrieved evidence and should receive a poor faithfulness score.
Faithfulness measures grounding, not necessarily truth against an external source.
Context precision evaluates how effectively the retriever ranks relevant information ahead of irrelevant content.
Imagine a user asks about refund eligibility. The retrieval system returns five documents, but the actual refund policy appears only in the final result.
Even though the relevant information was retrieved, the ranking is inefficient.
Higher context precision generally indicates that useful evidence appears earlier in the retrieval results.
Context recall measures whether the retrieved information covers the important facts needed to answer a question.
For example, an account cancellation policy might specify cancellation eligibility, notice requirements, and refund restrictions.
If retrieval captures only eligibility but misses the other conditions, context recall may be low.
In reference-based Ragas evaluation, this metric checks how much of the reference answer is supported by the retrieved contexts.
Answer relevancy evaluates whether a generated response addresses the user's question.
A user might ask:
"Is customer support available on weekends?"
The AI could respond with a detailed explanation of weekday support hours without answering whether weekend assistance is available.
The response may be grounded in retrieved documents while still being insufficiently relevant.
Answer relevancy measures alignment with the question rather than factual correctness.
Comparing RAG Evaluation Metrics
Metric | Evaluates | Helps Detect |
|---|
Faithfulness | Grounding of generated claims | Unsupported or hallucinated statements |
Context Precision | Ranking of retrieved evidence | Irrelevant or poorly ranked chunks |
Context Recall | Coverage of required evidence | Missing important information |
Answer Relevancy | Alignment with the user question | Off-topic or incomplete responses |
These metrics complement one another. A response can be faithful but irrelevant, or relevant but unsupported.
For dependable RAG pipeline testing, developers should evaluate both retrieval and generation.
Setting Up the Python Evaluation Environment
We will use Python, Ragas, pandas, and an LLM provider for evaluation.
Ragas provides metrics for assessing retrieved contexts and generated responses. Pandas helps organize, analyze, and export the results.
Step 1: Create a Python Virtual Environment
Create a new project directory:
mkdir rag-evaluation
cd rag-evaluation
python -m venv .venv
Activate the environment.
On Windows:
.venv\Scripts\activate
On macOS or Linux:
source .venv/bin/activate
Step 2: Install Dependencies
python -m pip install --upgrade pip
python -m pip install "ragas>=0.4,<0.5" openai pandas python-dotenv
This example uses the Ragas 0.4 collections-based metrics interface.
Ragas APIs have changed across releases, so verify the installed version and its documentation before running the implementation.
python -m pip show ragas
The version range identifies the intended API family rather than guaranteeing compatibility with every future package release.
Step 3: Configure Your API Key
Create a .env file in the project directory:
OPENAI_API_KEY="your_api_key_here"
Never hardcode production credentials into Python source files or commit the .env file to version control.
Evaluation calls can consume tokens and generate API charges, so start with a small dataset.
Building a RAG Evaluation Dataset
An evaluation dataset contains examples that represent the questions an application is expected to answer.
For basic RAG evaluation with Python, each example should include:
User input: The original question.
Retrieved contexts: The document passages returned by the retrieval system.
Response: The answer generated by the RAG application.
Reference: A verified answer used to assess the result.
The following dataset uses three hypothetical technical support scenarios: refund eligibility, weekend support availability, and delivery timing.
Create a file named evaluation.py.
evaluation_data = [
{
"id": "refund_policy",
"user_input": "What is the refund period?",
"retrieved_contexts": [
"Customers can request refunds within 14 days "
"of the original purchase.",
"Refund requests are reviewed by the billing team."
],
"response": (
"Customers can request a refund within "
"14 days of purchase."
),
"reference": (
"Refunds may be requested within 14 days "
"of the original purchase."
),
},
{
"id": "weekend_support",
"user_input": "Is customer support available on weekends?",
"retrieved_contexts": [
"Live support is available Monday through Friday.",
"Customers can submit support tickets at any time."
],
"response": (
"Live support is available every day, "
"including weekends."
),
"reference": (
"Live support is available Monday through Friday. "
"Tickets may be submitted at any time."
),
},
{
"id": "delivery_time",
"user_input": "How long does standard delivery take?",
"retrieved_contexts": [
"Standard delivery usually takes 3 to 5 business days.",
"Express delivery is a separate shipping option."
],
"response": (
"Standard delivery usually takes "
"3 to 5 business days."
),
"reference": (
"Standard delivery usually takes "
"3 to 5 business days."
),
},
]
The weekend-support example intentionally contains a response that conflicts with the retrieved evidence.
This makes it useful for checking whether faithfulness evaluation detects unsupported statements.
In a production application, the dataset should be captured from representative questions and verified against approved source material.
It should also include ambiguous questions, missing information, contradictory documents, and situations where the system is expected to abstain.
Implementing RAG Evaluation With Python and Ragas
Now we can evaluate the dataset using four metrics.
The implementation below follows the collections-based Ragas API documented for the 0.4 series. It uses an asynchronous OpenAI client to configure the evaluator and embeddings.
Step 1: Import Libraries and Configure the Evaluator
Add the following code below evaluation_data in evaluation.py:
import asyncio
import os
from pathlib import Path
import pandas as pd
from dotenv import load_dotenv
from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.embeddings.base import embedding_factory
from ragas.metrics.collections import (
Faithfulness,
ContextPrecision,
ContextRecall,
AnswerRelevancy,
)
load_dotenv()
if not os.getenv("OPENAI_API_KEY"):
raise RuntimeError("OPENAI_API_KEY is not configured.")
client = AsyncOpenAI()
evaluator_llm = llm_factory(
"gpt-4o-mini",
client=client,
)
evaluator_embeddings = embedding_factory(
"openai",
model="text-embedding-3-small",
client=client,
)
The evaluator LLM assesses retrieved information and generated responses. The embedding model supports answer relevancy evaluation.
Both dependencies are part of the evaluation process, not necessarily the underlying RAG application's generation architecture.
Step 2: Initialize Evaluation Metrics
metrics = {
"faithfulness": Faithfulness(
llm=evaluator_llm
),
"context_precision": ContextPrecision(
llm=evaluator_llm
),
"context_recall": ContextRecall(
llm=evaluator_llm
),
"answer_relevancy": AnswerRelevancy(
llm=evaluator_llm,
embeddings=evaluator_embeddings,
),
}
Each metric evaluates a different aspect of the pipeline.
Context precision and recall use the reference answer to assess retrieval quality. Faithfulness evaluates grounding, while answer relevancy examines whether the response addresses the question.
Step 3: Evaluate Individual Examples
Define an asynchronous function that evaluates one dataset entry:
async def evaluate_example(example):
user_input = example["user_input"]
contexts = example["retrieved_contexts"]
response = example["response"]
reference = example["reference"]
results = await asyncio.gather(
metrics["faithfulness"].ascore(
response=response,
retrieved_contexts=contexts,
),
metrics["context_precision"].ascore(
user_input=user_input,
reference=reference,
retrieved_contexts=contexts,
),
metrics["context_recall"].ascore(
reference=reference,
retrieved_contexts=contexts,
),
metrics["answer_relevancy"].ascore(
user_input=user_input,
response=response,
),
)
return {
"id": example["id"],
"question": user_input,
"response": response,
"reference": reference,
"faithfulness": results[0].value,
"context_precision": results[1].value,
"context_recall": results[2].value,
"answer_relevancy": results[3].value,
}
Ragas returns metric result objects, and their .value properties contain numerical scores.
The function collects these scores alongside the original example information, making individual failures easier to inspect.
Step 4: Create a Results DataFrame
Now add a function to run the evaluation, identify potentially problematic examples, and save the results.
async def main():
rows = []
for example in evaluation_data:
try:
result = await evaluate_example(example)
rows.append(result)
except Exception as exc:
print(
f"Evaluation failed for {example['id']}: {exc}"
)
raise
df = pd.DataFrame(rows)
score_columns = [
"faithfulness",
"context_precision",
"context_recall",
"answer_relevancy",
]
print("\nRAG Evaluation Results")
print(
df[["id", *score_columns]].to_string(
index=False
)
)
# Example screening threshold, not a universal standard.
review_threshold = 0.70
poor_examples = df[
df[score_columns].lt(review_threshold).any(axis=1)
]
print("\nExamples Requiring Review")
print(
poor_examples[
["id", "question", *score_columns]
].to_string(index=False)
)
output_dir = Path("results")
output_dir.mkdir(exist_ok=True)
df.to_csv(
output_dir / "rag_evaluation_results.csv",
index=False,
)
poor_examples.to_csv(
output_dir / "poor_quality_examples.csv",
index=False,
)
print("\nEvaluation results exported.")
if __name__ == "__main__":
asyncio.run(main())
Run the script:
python evaluation.py
The 0.70 threshold is only an example screening rule. It must be calibrated against the application's requirements and human-reviewed evaluation data.
The code has been reconstructed using documented Ragas interfaces; it has not been executed against a live API in this article. Exact compatibility should be verified in the target environment.
How to Interpret RAG Evaluation Results
RAGAS evaluation becomes useful when developers examine why individual examples receive different scores.
The following table contains hypothetical results for illustration only. These values are not measured outputs from the preceding Python script
Test Case | Faithfulness | Context Precision | Context Recall | Answer Relevancy |
|---|
Refund policy | 0.98 | 0.92 | 1.00 | 0.95 |
Weekend support | 0.25 | 0.91 | 1.00 | 0.88 |
Delivery time | 0.99 | 0.96 | 1.00 | 0.97 |
The refund-policy example shows strong performance across all four dimensions.
The weekend-support example illustrates a different failure. The retrieved contexts contain the relevant policy, but the generated answer incorrectly claims that live support is available every day.
This should lower faithfulness even though context recall remains high.
A low context precision score suggests that retrieval ranking needs investigation. A low context recall score suggests that important evidence may be missing.
Low answer relevancy indicates that the response may not address the original question adequately.
Are There Standard Passing Scores?
There is no universally accepted threshold that guarantees RAG quality.
A faithfulness score suitable for an internal knowledge assistant may not be sufficient for a high-stakes financial or legal workflow.
Teams should establish evaluation requirements using representative test data, human review, and the consequences of incorrect answers.
Aggregate scores also need careful interpretation. A high average can hide a small number of serious failures.
Automating RAG Evaluation in Development Workflows
Manual testing becomes difficult when a RAG application changes frequently.
Prompt modifications, chunking strategies, new embedding models, document updates, and retrieval adjustments can all influence results.
Automated RAG evaluation helps teams identify regressions before releasing changes.
A practical workflow includes:
Version the evaluation dataset: Store representative questions, reference answers, and source-document versions.
Establish a baseline: Evaluate the existing application and save its scores.
Introduce one controlled change: Modify a prompt, retriever, embedding model, or chunking configuration.
Run the evaluation again: Compare results with the established baseline.
Inspect regressions: Review examples with reduced retrieval quality, grounding, or relevance.
Integrate with CI/CD: Execute a suitable evaluation suite before deployment and track results over time.
For larger datasets, teams may run a small evaluation subset during pull requests and perform broader testing before production releases.
Because LLM-based evaluations introduce latency and API costs, evaluation scheduling should reflect the development workflow.
Store model identifiers, evaluator versions, prompt versions, and dataset revisions with each run to improve reproducibility.
Common Mistakes When Evaluating RAG Systems
Even a technically correct evaluation pipeline can produce misleading conclusions when the testing methodology is incomplete.
Common mistakes include:
Using only answer accuracy: A correct-looking answer may still contain unsupported claims. Evaluate grounding separately.
Testing with too few examples: A handful of successful questions cannot represent the diversity of real user requests.
Ignoring retrieval metrics: When answers are incorrect, the problem may originate in missing or poorly ranked evidence.
Overtrusting LLM evaluators: Evaluator models can make mistakes, so important judgments should be compared with human assessments.
Using unrepresentative datasets: Test sets should include difficult, ambiguous, incomplete, and realistic production scenarios.
Ignoring latency and evaluation costs: Better scores may introduce unacceptable response times or operational expenses.
Applying uncalibrated thresholds: Fixed score cutoffs may not reflect the actual risks and quality requirements of an application.
Developers should also examine whether their evaluation data contains outdated documents, inconsistent references, or information unavailable to the production retriever.
Reliable evaluation depends on the quality of the test dataset as much as the metrics themselves.
Final Thoughts
Building a RAG application is only the beginning. Maintaining its reliability requires evaluating retrieval performance, generated responses, and changes introduced throughout development.
A practical RAG evaluation pipeline combines metrics such as faithfulness, context precision, context recall, and answer relevancy with representative datasets and repeatable Python testing.
Frameworks such as Ragas make these evaluations easier to implement, but numerical scores should support engineering judgment rather than replace it.
Teams also need reproducible test cases, careful error analysis, and ongoing monitoring.
At THE TISA, this evaluation-first engineering perspective is relevant to building AI applications that must operate reliably with business data and existing software systems.
Ultimately, the goal is not simply to generate fluent answers. It is to ensure that a RAG system retrieves appropriate evidence, produces grounded responses, and improves through measurable testing.
Frequently Asked Questions
1. What are the most important RAG evaluation metrics?
Faithfulness, context precision, context recall, and answer relevancy are four useful starting metrics. Together, they help evaluate retrieval quality, answer grounding, and relevance to the user's question.
2. How does RAGAS evaluate RAG pipelines?
Ragas uses evaluation metrics to assess retrieved contexts and generated responses. Depending on the metric, it may use an LLM, embeddings, reference answers, or other scoring methods to generate evaluation results.
3. Can you evaluate RAG without reference answers?
Yes. Metrics such as faithfulness and answer relevancy can evaluate certain response characteristics without reference answers. However, reference-based context recall requires additional ground-truth information.
4. How often should developers evaluate a RAG pipeline?
Evaluate after meaningful changes to prompts, retrieval settings, chunking, embeddings, or model configurations. Production systems should also undergo periodic evaluation using representative and newly observed failure cases
Join the conversation! Your thoughts help the community grow.