Retrieval-augmented generation is two systems in a trench coat, and almost every evaluation problem comes from measuring them as one. Here is the method we use, and why each part of it exists.
Measure the two halves separately
A RAG system retrieves passages, then generates an answer from them. When the answer is wrong, exactly one of two things happened: the right passage never came back, or the right passage came back and the model failed to use it. These have completely different fixes, and an end-to-end score cannot tell you which one you are looking at.
So run two evaluations. The retrieval one asks: for this question, did the passage containing the answer come back, and at what rank? The generation one asks: given the correct passage, did the model produce a faithful answer from it? Fix retrieval first — in our experience it is the cause of the clear majority of wrong answers, and generation problems often evaporate once it is sound.
Build the test set from cases you already have
The best test set is not written; it is harvested. Resolved support tickets, historical queries, the questions the front desk actually gets asked. Real questions carry the phrasing, the typos, the missing context, and the domain shorthand that invented test questions never do.
A hundred cases is enough to be useful and small enough to actually assemble. Cover the common path, but deliberately include the awkward ones: questions whose answer is not in the corpus at all, questions that span two documents, questions with a false premise. Those three categories are where systems fail in production and where a clean test set will never look.
Label each case with the passage that answers it and the answer a competent colleague would give. That labelling week is the cost of admission to every measurement that follows.
Three numbers, not one
Track accuracy, refusal rate, and citation coverage separately, because they trade against each other and a combined score hides the trade.
Accuracy is the share of answered questions that are correct. Refusal rate is the share of questions the system declined. Citation coverage is the share of answers whose claims are actually supported by the passages cited — which is not the same as having citations attached, and checking the difference is where most surprises live.
A system can reach high accuracy by refusing everything hard, or high coverage by answering only the trivial. Watching all three keeps a change from improving one at the silent expense of another.
Retrieval metrics worth having
Recall at k is the workhorse: of the questions whose answer exists in the corpus, for how many did the answering passage appear in the top k results? If recall at 10 is poor, no amount of prompt engineering downstream will save the system, because the information is not in the context window at all.
Mean reciprocal rank tells you where in the list the right passage landed. Recall can look acceptable while the answer sits at rank 9, which matters because models attend unevenly across a long context and the useful passage may be effectively ignored. Reranking is the usual fix and it is worth measuring before and after rather than adopting on faith.
Track the no-answer cases too. For questions the corpus genuinely cannot answer, the correct retrieval behaviour is a low top score — and that low score is what your refusal threshold keys off.
Set the refusal threshold deliberately
The threshold below which the system declines to answer is a product decision with a measurable cost on both sides, and it should be chosen by looking at the curve rather than by picking a round number.
Sweep the threshold across your test set and plot accuracy against refusal rate. You will see the knee: the point past which refusing more questions stops buying meaningful accuracy. Where you sit relative to that knee depends on the domain — a clinical or financial context justifies refusing far more aggressively than an internal documentation search does.
Make it a gate, not a report
An evaluation that runs when someone remembers is an evaluation that stops running. Wire the suite into the deploy so a prompt change, a model upgrade, or a chunking tweak cannot reach production without the numbers being compared to the previous release.
This is what makes model upgrades routine rather than frightening. When a new model appears, you run the suite, read three numbers, and know within an hour whether it is an improvement for your corpus — instead of shipping on the vendor's benchmark and discovering the regression through customer complaints.
What good looks like
A test set of real cases including the awkward ones. Retrieval measured separately with recall at k and MRR. Accuracy, refusal rate, and citation coverage tracked as three numbers. A refusal threshold chosen off the curve. The whole suite gating every deploy.
Nothing here requires a specialist evaluation platform. It requires a week of labelling and the discipline to keep the gate closed, and it is the difference between a system you can improve deliberately and one you can only change and hope.