How I graded a RAG system before trusting it

Separate report cards for retrieval and generation, refusal traps where declining is the right answer, and why a fluent answer is not evidence of anything.

A RAG system will answer almost anything you ask it, fluently, with citations that look right. That is precisely the problem. Before putting mine in front of anyone, I wanted a number for how often the citations actually point at the truth, and a separate number for how often the prose respects the citations. This entry is the grading system, because the grading turned out to be more work, and more interesting, than the pipeline.

The system, in one paragraph

Six public engineering documents (DOE handbooks on thermodynamics, heat transfer, and mechanical components, NASA’s systems engineering handbook, OSHA’s process safety booklet, and one federal efficiency rule I happen to have complied with in a past job at Rheem): 778 pages, split into 2,091 chunks of 300 tokens, retrieved with dense embeddings and BM25 fused by reciprocal rank, optionally reranked by a cross-encoder, and answered by an LLM that is instructed to use only the retrieved excerpts and to cite document and page for everything.

One exam, two report cards

The eval set is 31 hand-labeled questions, each with the document and pages that contain the answer. Building it was tedious and it is the most valuable artifact in the repo.

Retrieval gets graded against the labeled pages: did the right chunk show up in the top 3 (Recall@3 came out 0.923), and how high did it rank (MRR 0.817). Generation gets graded against the retrieved text, not against the world: does every claim in the answer trace to the excerpts (faithfulness 0.928), and does the answer address the question (relevancy 0.960).

The reason for two report cards is that each one can fail alone. A fluent, faithful answer built on the wrong chunks is confidently misgrounded, and retrieval metrics catch it. Perfect chunks summarized with invented numbers is hallucination, and faithfulness catches it. A single blended score would have hidden every failure I actually found while tuning, including the chunk size: 300 tokens won an ablation against larger sizes, and I would not have seen why without the retrieval-side metrics standing alone.

The refusal traps

Five of the 31 questions are traps: engineering questions whose answers are not in the corpus, sitting close enough to the domain to be tempting. The correct behavior is declining, and the system prompt instructs the exact sentence to use. The system declined 5 of 5. There are also five borderline questions living in the gray zone between answerable and not, because a system that only ever sees clean cases is a system you know nothing about.

Refusal-as-a-feature changes how you demo. Most RAG demos hope you ask something the corpus covers. This one includes questions where the right answer is “I don’t have information on that in the provided documents,” and shows the system saying so.

What the honest numbers cost

I expected hybrid retrieval to win because engineers ask about exact strings such as acronyms, standard numbers, and valve designations. The result went the other way. Dense-only retrieval reached 0.923 Recall@3 and 0.817 MRR. Hybrid RRF reached 0.885 and 0.774. Adding the cross-encoder recovered Recall@3 to 0.923 but lowered MRR to 0.741 and tripled latency. The API keeps both controls for comparison; the free deployment leaves the reranker off.

Every run prints latency and generation cost. Production uses Groq’s free plan for generation and OpenAI only for retrieval embeddings, so a synthesized query retains one small paid embedding call.

The limitation I would state before anyone asks: 31 queries is a small exam. It is big enough to expose retrieval regressions and refusal failures during development, and too small to certify anything. Growing it is the roadmap, and every number above says “on 31 labeled queries” wherever it appears.

Ask it a graded question yourself

The live demo at rag.alvinalias.com has a button that draws from the same graded exam: it asks the question, then prints the labeled source next to whatever the answer cited, hit or miss, and one rotation in the set is a trap where the correct outcome is a refusal. The eval set, protocol, and pipeline are in the repo.

Related reading

• I stopped asking an LLM to write every RAG answer, the V2 routing, verified facts path, classifier benchmark, and provider call map.