Why RAG pipelines fail silently — and how to catch it
Most RAG evaluation only looks at the final answer. That hides whether the failure came from retrieval or generation — here's a sharper way to look at it.
A retrieval-augmented generation pipeline has two places to fail: the retriever can return the wrong context, or the generator can ignore good context and answer badly anyway. Scoring only the final output collapses these into one number, which makes debugging close to impossible.
Splitting evaluation into a retrieval-quality check and a generation-faithfulness check — even a rough one — makes failure analysis tractable. You start to see whether your error budget is a retriever problem or a prompting problem, which are fixed in completely different ways.
This is the core idea behind the clinical RAG failure classifier in my projects section: a small model whose only job is to label *why* an answer might be wrong, sitting alongside the main pipeline rather than inside it.