Evaluating RAG: Retrieval Scores and Answer Scores

Retrieval eval asks 'did we fetch the right chunks': recall@k (of the relevant chunks, what fraction were retrieved), MRR (how high the first relevant hit ranks), nDCG (graded ordering quality). It needs labeled query→relevant-chunk pairs — built from production logs, hand-labeled, or synthetically generated and verified (synthetic queries skew easy; they ask the way the generator thinks, not the way users do).

Generation eval asks 'was the answer right': faithfulness (claims supported by retrieved context), answer relevance (did it address the question), context utilization (did it use what it was given). RAGAS-style frameworks automate this trio — each measures a different stage's health.