TopolearnWithout measurements, every chunking, model or prompt change is a guess that may make things worse.
RAG has two parts to measure: did retrieval find the right chunks, and did the LLM answer well from them?
Retrieval metrics need questions paired with the node ids that should be found:
- Hit rate: how often a correct node appears in the top k.
- MRR (mean reciprocal rank): rewards finding it near the top; 1 if first, 0.5 if second, and so on.
evaluator = RetrieverEvaluator.from_metric_names(["hit_rate", "mrr"], retriever=retriever)
result = await evaluator.aevaluate(query="...", expected_ids=[node_id])These need no LLM, so they are cheap to run on hundreds of questions.
Response evaluators use an LLM as a judge:
FaithfulnessEvaluator: is the answer supported by the retrieved context, or did the model make things up?RelevancyEvaluator: do the answer and context actually address the question?CorrectnessEvaluator: compares the answer with a reference answer and gives a score.
Each returns an EvaluationResult with passing, score and feedback. BatchEvalRunner runs several evaluators over many questions concurrently.
Building a test set: write 30 to 100 real user questions with expected sources and answers. An LLM can draft extra questions from your documents, but review them by hand. Include questions whose answer is not in the data, to test "I don't know" behaviour.
Use a strong model as judge, check a sample of its verdicts yourself, and compare versions on the same set. External tools (such as Ragas, DeepEval or observability platforms) integrate with LlamaIndex for larger evaluation setups.
Example
import asyncio
from llama_index.core import Document, VectorStoreIndex
from llama_index.core.evaluation import FaithfulnessEvaluator, RetrieverEvaluator
from llama_index.llms.openai import OpenAI
index = VectorStoreIndex.from_documents([
Document(text="Refunds take 5 business days.", id_="refunds"),
Document(text="Shipping is free over 50 EUR.", id_="shipping")])
retriever = index.as_retriever(similarity_top_k=2)
refund_node = next(n.node_id for n in index.docstore.docs.values() if "Refunds" in n.text)
async def main():
r_eval = RetrieverEvaluator.from_metric_names(["hit_rate", "mrr"], retriever=retriever)
res = await r_eval.aevaluate(query="How long until I get my money back?",
expected_ids=[refund_node])
print(res.metric_vals_dict) # {'hit_rate': 1.0, 'mrr': 1.0}
judge = FaithfulnessEvaluator(llm=OpenAI(model="gpt-4o"))
response = index.as_query_engine().query("How long do refunds take?")
verdict = await judge.aevaluate_response(response=response)
print(verdict.passing, verdict.score) # True 1.0
asyncio.run(main())Remember
- Retrieval: hit rate and MRR against expected node ids (no LLM needed)
- Answers: Faithfulness, Relevancy and Correctness evaluators (LLM judge)
- Keep a reviewed test set, including unanswerable questions
Watch out for
- Evaluating only answer quality, so you can't tell whether a regression came from retrieval or generation.