I stopped asking an LLM to write every RAG answer
“What is the threshold quantity for chlorine?” and “Why is phosgene’s threshold lower than ammonia’s?” mention the same kind of fact. They should not use the same answer path.
The first question has one controlled value in an OSHA table. Letting a language model rewrite it adds a failure mode. The second question asks for an explanation, so retrieval and generation are useful. My first RAG build sent both through retrieval and an LLM. V2 classifies the question first.
Three routes
The router chooses lookup, interpret, or clarify before retrieval.
| Intent | What the system does |
|---|---|
lookup |
Reads a verified value, unit, source quote, and page from JSON. No model generates the answer. |
interpret |
Embeds the query, retrieves passages, asks a model for a cited answer, then validates the citations and numbers. |
clarify |
Lists the competing stored meanings and asks the user to choose. |
“What is the TQ for chlorine?” now returns 1,500 pounds from OSHA page 46 through a Python template. “What is the TQ for hydrogen?” matches several hydrogen compounds and asks which one the user means. “Explain the difference between a relief valve and a safety valve” keeps the full retrieval and generation path.
The failure direction stays simple. If the classifier fails, deterministic rules take over. If a fact lookup misses, the query falls through to retrieval. The system returns to the measured V1 behavior instead of dropping the question.
The facts file took more work than the router
The lookup path uses 88 verified facts from two documents: 41 from the 2017 DOE central air-conditioner and heat-pump rule, and 47 from OSHA 3132. Each record carries the value, unit, page, source wording, and search terms.
The source documents resisted clean extraction. Federal Register tables use a multi-column layout that scrambles the PDF text layer. OSHA prints “Hodrogen Cyanide” and shows the Hydrogen Sulfide CAS as 7783=06-4. I kept those strings in the evidence quotes and normalized the structured fields used for lookup.
The frozen factual evaluation has 50 questions. The lookup returns the correct value and page on 50 of 50. That result is narrow by design. It says the deterministic path passes this test set. New phrasing can still miss and fall through to synthesis.
I benchmarked the router instead of picking the fanciest option
The intent set contains 168 reviewed questions with a fixed 135/33 train and holdout split. I ran four backends on the same holdout.
| Backend | Correct | Wilson 95% interval | Median latency |
|---|---|---|---|
| Rules | 21/33 | 46.6% to 77.8% | 0 ms |
| Groq zero-shot | 29/33 | 72.7% to 95.2% | 330 ms |
| GPT-4o-mini zero-shot | 31/33 | 80.4% to 98.3% | 571 ms |
| DistilBERT + LoRA | 27/33 | 65.6% to 91.4% | 21 ms |
GPT-4o-mini had the highest point estimate. The difference from Groq was two questions, and their confidence intervals overlap. That test does not establish a reliable quality advantage. I kept Groq in production because its free plan avoids a classifier charge and it was faster in this run. The local LoRA model remains an offline option.
Which API gets called
The production split is easy to lose track of because Groq and OpenAI do different jobs.
| Query path | Groq | OpenAI |
|---|---|---|
| Verified fact lookup | Intent classification | No call |
| Clarification | Intent classification | No call |
| Synthesized answer | Intent classification and answer generation | Query embedding |
| Health check | No call | No call |
OpenAI remains on the synthesized path because the committed Chroma index was built with text-embedding-3-small. A query has to use the same vector space as the stored documents. The current price is $0.02 per million input tokens, so the remaining paid call is small and only occurs when retrieval runs.
Groq serves openai/gpt-oss-20b for classification and generation. The openai/ prefix is the model ID, not the API provider. Requests go to Groq, use the Groq key, and follow the Groq free-plan limits.
GPT-4o-mini was called during the classifier benchmark. It is not the selected production classifier or generator.
Checking the generated path
Interpretation still needs an LLM, so V2 checks the answer after generation.
The validator confirms that each cited document and page appeared in the retrieved chunks. It checks that every number in the answer appears in those chunks. It also measures token overlap between each answer sentence and the retrieved text.
The first two checks catch fabricated citations and unsupported numbers. The sentence check is coarser. A fluent paraphrase can pass token overlap without being fully supported, and a correct sentence with different wording can be flagged. I keep the offline Ragas faithfulness score for the finer evaluation and use the runtime validator as a fast guardrail.
What changed in the project
V2 added more evaluation work than model work: 88 source-checked facts, 168 reviewed intent labels, four classifier benchmarks, 50 deterministic factual tests, 56 automated tests, and three local API smoke tests covering lookup, synthesis, and clarification.
The live interface shows which route produced each response. A recruiter can see whether a value came from the verified facts file or from cited synthesis, then open the retrieved passages beside it.
Try it at rag.alvinalias.com. The code, facts schema, intent set, and benchmark outputs are in the repository.
Citation
@online{alias2026,
author = {Alias, Alvin},
title = {I Stopped Asking an {LLM} to Write Every {RAG} Answer},
date = {2026-07-10},
url = {https://alvinalias.com/notes/posts/not-every-rag-query-needs-an-llm.html},
langid = {en}
}