Evaluating RAG
Measure retrieval and generation separately so you know which part to fix.
Evolving: the basics hold, details shift. Reviewed 17 Oct 2026.
Key ideas
- Retrieval: recall@k, MRR, nDCG against labelled question-to-passage pairs.
- Generation: faithfulness (grounded in context), answer relevance, correctness against a reference.
- Build a golden set of 50 to 200 real questions with expected sources; grow it from production failures.
- Synthetic question generation helps bootstrap, but review samples by hand.
Common mistakes
- Only judging final answers; you cannot tell whether retrieval or the prompt failed.