Module 6 of 6 · 35 minutes

Evaluating and improving RAG

You can't improve what you don't measure. This module shows how to test a RAG system properly — and how to make it better, one fix at a time.

Learning objectives

Separate retrieval failures from generation failures
Build a representative RAG test set
Compute recall@k and MRR
Use an LLM judge responsibly
Run an evidence-based improvement loop

6.1 Evaluate retrieval and generation separately

A wrong answer could be the retriever's fault (the right passage never arrived) or the generator's (it was there, but ignored or misread). Measure both.

Stage	Metric	Question it answers
Retrieval	Recall@k (hit rate)	Is a relevant passage in the top k results?
Retrieval	Mean reciprocal rank (MRR)	How high does the first relevant passage appear?
Generation	Faithfulness (groundedness)	Is every claim supported by the retrieved passages?
	Answer relevance / correctness	Does it actually answer the question, correctly?
	Citation accuracy	Do citations point to passages that support the claim?

Three relationships to evaluate in RAG: is the context relevant to the question, is the answer faithful to the context, and does the answer address the question?

6.2 Build a test set

Collect 50–200 realistic questions with the passage(s) that answer them and a reference answer. Include: common questions, paraphrases, exact identifiers, questions spanning two documents, questions the documents can't answer, and tricky or adversarial ones. You can use an LLM to draft questions from your documents — then review them by hand.

Python · recall@k and MRRtest_set = [
    {"q": "Can I return a cracked board?", "relevant": "Returns Policy v7"},
    {"q": "How long is UK delivery?",     "relevant": "Delivery FAQ"},
]

def evaluate_retrieval(test_set, k=3):
    hits, rr = 0, 0.0
    for t in test_set:
        sources = [c["source"] for c in retrieve(t["q"], k=k)]
        if t["relevant"] in sources:
            hits += 1
            rr += 1 / (sources.index(t["relevant"]) + 1)
    n = len(test_set)
    return {"recall@k": hits / n, "MRR": rr / n}

print(evaluate_retrieval(test_set))

6.3 LLM-as-judge for answers

Grading answer quality by hand doesn't scale. A common approach is an LLM judge with a clear rubric — for example, "Is every factual claim in the answer supported by the sources? Answer SUPPORTED or UNSUPPORTED, then explain." Anthropic's guidance on evaluations recommends detailed rubrics, simple outputs (e.g. correct/incorrect or 1–5), letting the grader reason first, and using a different model to grade than the one that generated. Always check a sample of the judge's verdicts against human judgement before trusting it.

6.4 The improvement loop

Run the test set; record metrics.
Look at failures and classify them: retrieval miss, bad chunk, ignored context, hallucination, refusal when it shouldn't.
Fix the biggest category — chunking, hybrid search, reranking, prompt or metadata.
Re-run and compare. Keep changes that help.

Evidence-based improvement: never change three things at once and guess which one helped.

6.5 Beyond basic RAG

Agentic RAG: an LLM decides when and what to search, runs several searches, and checks whether it has enough information before answering.
Structured data: for questions about numbers in databases, let the model write and run queries (with strict, read-only permissions) instead of embedding table rows.
Knowledge graphs: represent entities and relationships for questions that need connecting many facts.
Long-context models: very large context windows sometimes make it practical to include whole documents; RAG still wins on cost, freshness, scale and citations for large collections.

6.6 Production checklist

Permissions enforced at retrieval time
Documents labelled as untrusted data in the prompt
Fresh index with versioning and deletion
Visible citations and an "I can't find that" path
Retrieval and answer metrics tracked on a test set
User feedback captured and reviewed
Personal data handled in line with data protection law

Key takeaways

Measure retrieval (recall@k, MRR) and generation (faithfulness, correctness, citations) separately.
Test sets need paraphrases, identifiers, multi-document and unanswerable questions.
LLM judges need clear rubrics and human spot-checks.
Classify failures, fix the biggest category, re-measure.
Production RAG needs permissions, freshness, citations and feedback.