Three routes

The router chooses lookup, interpret, or clarify before retrieval.

“What is the TQ for chlorine?” now returns 1,500 pounds from OSHA page 46 through a Python template. “What is the TQ for hydrogen?” matches several hydrogen compounds and asks which one the user means. “Explain the difference between a relief valve and a safety valve” keeps the full retrieval and generation path.

The failure direction stays simple. If the classifier fails, deterministic rules take over. If a fact lookup misses, the query falls through to retrieval. The system returns to the measured V1 behavior instead of dropping the question.

The facts file took more work than the router

The lookup path uses 88 verified facts from two documents: 41 from the 2017 DOE central air-conditioner and heat-pump rule, and 47 from OSHA 3132. Each record carries the value, unit, page, source wording, and search terms.

The source documents resisted clean extraction. Federal Register tables use a multi-column layout that scrambles the PDF text layer. OSHA prints “Hodrogen Cyanide” and shows the Hydrogen Sulfide CAS as 7783=06-4. I kept those strings in the evidence quotes and normalized the structured fields used for lookup.

The frozen factual evaluation has 50 questions. The lookup returns the correct value and page on 50 of 50. That result is narrow by design. It says the deterministic path passes this test set. New phrasing can still miss and fall through to synthesis.

I benchmarked the router instead of picking the fanciest option

The intent set contains 168 reviewed questions with a fixed 135/33 train and holdout split. I ran four backends on the same holdout.

GPT-4o-mini had the highest point estimate. The difference from Groq was two questions, and their confidence intervals overlap. That test does not establish a reliable quality advantage. I kept Groq in production because its free plan avoids a classifier charge and it was faster in this run. The local LoRA model remains an offline option.

Which API gets called

The production split is easy to lose track of because Groq and OpenAI do different jobs.

OpenAI remains on the synthesized path because the committed Chroma index was built with text-embedding-3-small. A query has to use the same vector space as the stored documents. The current price is $0.02 per million input tokens, so the remaining paid call is small and only occurs when retrieval runs.

Groq serves openai/gpt-oss-20b for classification and generation. The openai/ prefix is the model ID, not the API provider. Requests go to Groq, use the Groq key, and follow the Groq free-plan limits.

GPT-4o-mini was called during the classifier benchmark. It is not the selected production classifier or generator.

Checking the generated path

Interpretation still needs an LLM, so V2 checks the answer after generation.

The validator confirms that each cited document and page appeared in the retrieved chunks. It checks that every number in the answer appears in those chunks. It also measures token overlap between each answer sentence and the retrieved text.

The first two checks catch fabricated citations and unsupported numbers. The sentence check is coarser. A fluent paraphrase can pass token overlap without being fully supported, and a correct sentence with different wording can be flagged. I keep the offline Ragas faithfulness score for the finer evaluation and use the runtime validator as a fast guardrail.

What changed in the project

V2 added more evaluation work than model work: 88 source-checked facts, 168 reviewed intent labels, four classifier benchmarks, 50 deterministic factual tests, 56 automated tests, and three local API smoke tests covering lookup, synthesis, and clarification.

The live interface shows which route produced each response. A recruiter can see whether a value came from the verified facts file or from cited synthesis, then open the retrieved passages beside it.

Try it at rag.alvinalias.com. The code, facts schema, intent set, and benchmark outputs are in the repository.

Related reading

• How I graded a RAG system before trusting it, the V1 evaluation design, refusal traps, and retrieval ablations.
• The RAG Engineering Assistant system card, the current routes, limits, and provider behavior.

The accuracy decoy

The dataset (AI4I 2020, ten thousand machine records) has failures in 3.4% of rows. A model that predicts “no failure” every single time scores 96.6% accuracy and catches nothing. So accuracy is not reported anywhere in this project, and any imbalanced-classification result quoted without that disclaimer deserves a raised eyebrow.

The metrics that survive imbalance: PR-AUC (0.841 for the XGBoost model, against 0.820 random forest and 0.455 logistic baselines) and recall at whatever operating point you choose. The operating point is the story.

Charging each error its price

Model the two mistakes in money. A missed failure becomes unplanned downtime: modeled here at $50,000. A false alarm becomes a scheduled maintenance call that wasn’t needed: $2,000. Both numbers are stated assumptions, not measurements, and everything downstream inherits them, which is why they sit in the open on the demo page.

Then sweep the threshold and charge every mistake at every setting:

for threshold in np.arange(0.1, 0.9, 0.05):
    preds = (failure_proba >= threshold).astype(int)
    fn_cost = ((preds == 0) & (y_test == 1)).sum() * 50_000
    fp_cost = ((preds == 1) & (y_test == 0)).sum() * 2_000
    total_cost = fn_cost + fp_cost

The minimum lands at 0.775. At that threshold, on the held-out test set: 62 of 68 real failures caught (91.2% recall), 6 missed, 83 false alarms tolerated, $466K modeled total cost. Compare the intuition to the default: because a miss costs 25 times a false alarm, you might expect the threshold to drop below 0.5 to catch more failures. It moves the other way, up to 0.775, because this model’s probabilities are well separated: the failures it can catch, it catches with high confidence, so raising the bar sheds expensive false alarms while giving up almost no real catches. That non-obvious direction is exactly why you sweep instead of reasoning by vibes.

The part that generalizes

The threshold is not a modeling constant; it is a business input wearing a decimal point. Change the cost assumptions and the optimum moves. Double the false-alarm cost (a plant where techs are scarce) and the optimum climbs higher; make misses catastrophic (safety-critical equipment) and it slides down. The model did not change. The economics did.

That is also why the live demo now lets you move the costs yourself: the page ships the real per-threshold error counts from the same test set the published numbers come from, and recomputes the cost curve and its minimum in front of you. The one non-negotiable behind that feature: the exported counts had to reproduce the published confusion matrix at 0.775 exactly, 6 and 83, or the export fails the build. Re-derivations must agree with published numbers before they earn trust.

Owning the misses

91.2% recall means about 9 in 100 failures slip past. The demo draws real records from the dataset with their outcomes hidden, predicts, then reveals what actually happened, and when a run lands on a miss, the page says so in plain text. A maintenance manager deciding whether to trust a model needs to know what its misses look like more than they need another decimal of AUC.

Try it at machine-failure.alvinalias.com; the sweep, the notebooks, and the model card are in the repo.

The setup

The corpus is 3,000 synthetic work orders with a calibrated noise layer: typos, technician shorthand like “repl” and “brg” and “RTS,” vague observations, and terse one-liners, because clean synthetic text makes every extractor score a meaningless 100%. The taxonomy and vocabulary come from twelve years of writing real work orders across HVAC, subsea, and manufacturing. Ground truth is nulled per-field wherever the noise layer removed a fact from the text, so no extractor gets penalized for information that is not there.

Three extractors ran on the same 100-record evaluation sample: rule-based regex (free, under a millisecond), LLM-only (GPT-4o-mini with a Pydantic schema, about 1.1 seconds per record), and the hybrid, which escalated to the LLM whenever the regex’s self-reported confidence fell below 0.7.

The results, including the embarrassing column

On equipment tags, regex nearly matches the LLM (82% against 99%), because tags follow an XX-NNN pattern that regex was born for. On failure mode, the narrative field, regex collapses to 13% while the LLM reaches 70%: typos, shorthand, and paraphrase defeat keyword matching.

The hybrid cut LLM calls by 56%. If the gate were working, accuracy should sit most of the way toward the LLM’s numbers. On failure mode it landed at 23%: barely above regex, nowhere near 70%.

The autopsy

The gate assumed regex confidence correlates with regex correctness. It does not, and the direction of the failure is the interesting part: on clean, well-structured text the regex is confident and right, and on noisy text it is confident and wrong, because a keyword pattern that happens to match garbage reports the same confidence as one matching the real thing. The records that most needed escalation were exactly the ones that never got it.

That is a calibration failure, and it generalizes far beyond regex. Any pipeline that routes work based on a component’s self-reported confidence inherits that component’s calibration, and cheap extractors are almost never calibrated on the distribution that matters, which is the messy tail.

What a production version needs

An escalation signal that does not come from the extractor grading its own homework. The candidates I would try, in order: field-completeness checks (empty or suspiciously short extracted fields are a cheap, honest tell), a small classifier trained to predict extraction failure from surface features of the text, or spot-check sampling with drift monitoring. The cost at stake is real but small: LLM-only extraction runs roughly $0.10 to $0.15 per thousand records at GPT-4o-mini prices, so the hybrid has to be nearly free to be worth its complexity.

Negative results like this one rarely make it into portfolios, which is exactly why this one is in mine: the measurement worked, the architecture failed, and the failure has a nameable cause with a testable fix.

See the pipeline

The classifier this corpus trains (DistilBERT with LoRA, 94.4% F1 while updating 1.1% of parameters) and the similar-case retrieval run live at workorders.alvinalias.com; the extraction study, with the per-field table and the eval script, is in the repo.

Features from physics, not from a menu

The feature set came from what I watched machines actually do, not from a time-series feature catalog: COP (cooling delivered per watt drawn, the best single efficiency signal in refrigeration), supply-side delta-T (narrows as coils foul), refrigerant-circuit delta-T (widens as charge depletes), load ratio (high load plus falling COP is the zone where things break), and rolling statistics of those over 24 hours and 7 days.

An Isolation Forest scores each reading for anomaly structure; the score inverts and rescales to a 0 to 100 health number. Standard so far. The two decisions worth writing down are what made it defensible.

Decision one: judge each unit against its own baseline

A raw anomaly score answers “is this reading strange for the fleet?” That is the wrong question. Some units always ran hot; some buildings always ran heavy loads. At Rheem you never judged a compressor against the fleet’s vibration baseline, only against its own commissioning signature.

So the score is normalized per unit: each building’s raw scores are mapped through that building’s own 5th and 95th percentiles, so health 20 means “bad for this unit,” not “different from other units.” The units with unusual but stable operating profiles stopped false-alarming the moment this landed.

Decision two: keep a second detector as witness

With no labels, there is no accuracy number, and anyone who quotes one should be asked where it came from. What you can do is triangulate. A Local Outlier Factor model, which defines anomaly by local density rather than isolation depth, scores the same readings independently. The two detectors agree on 91.3% of a 100,000-reading sample.

Agreement is not correctness. What it establishes is that the anomaly structure mostly does not depend on which definition of “strange” you pick, and the 8.7% where they disagree is exactly the set where the call is genuinely ambiguous. The contamination setting got the same treatment: run at 0.02, 0.05, and 0.10, and check that everything flagged at the strict setting stays flagged at the middle one. It does, which means the flagged population is not an artifact of one knob.

The finding I didn’t expect

The strongest SHAP contributor across the fleet is not COP level. It is the 24-hour rolling standard deviation of COP: volatility, not value. In hindsight it is obvious from the shop floor: a machine that is failing intermittently swings before it sags. A compressor short-cycling, a valve hunting, a sensor going flaky: they all show up as variance first. The model found the intermittent-fault signature on its own, and that, more than any metric, is what made me trust the anomaly structure.

What this can and cannot say

It cannot say “this unit will fail in N days”; that needs labels or run-to-failure history (my turbofan project is that problem, with NASA’s labels). It can rank 497 units by how far each has drifted from its own normal, with a second detector countersigning most calls, and that ranking is what a maintenance planner actually needs on Monday morning: which ten units get a technician this week.

The live fleet board at hvac.alvinalias.com shows the real scored units and prints the detector cross-check with every score. The repo, with the sensitivity checks and the normalization code, is at github.com/aalias01/hvac-equipment-health.

The system, in one paragraph

Six public engineering documents (DOE handbooks on thermodynamics, heat transfer, and mechanical components, NASA’s systems engineering handbook, OSHA’s process safety booklet, and one federal efficiency rule I happen to have complied with in a past job at Rheem): 778 pages, split into 2,091 chunks of 300 tokens, retrieved with dense embeddings and BM25 fused by reciprocal rank, optionally reranked by a cross-encoder, and answered by an LLM that is instructed to use only the retrieved excerpts and to cite document and page for everything.

One exam, two report cards

The eval set is 31 hand-labeled questions, each with the document and pages that contain the answer. Building it was tedious and it is the most valuable artifact in the repo.

Retrieval gets graded against the labeled pages: did the right chunk show up in the top 3 (Recall@3 came out 0.923), and how high did it rank (MRR 0.817). Generation gets graded against the retrieved text, not against the world: does every claim in the answer trace to the excerpts (faithfulness 0.928), and does the answer address the question (relevancy 0.960).

The reason for two report cards is that each one can fail alone. A fluent, faithful answer built on the wrong chunks is confidently misgrounded, and retrieval metrics catch it. Perfect chunks summarized with invented numbers is hallucination, and faithfulness catches it. A single blended score would have hidden every failure I actually found while tuning, including the chunk size: 300 tokens won an ablation against larger sizes, and I would not have seen why without the retrieval-side metrics standing alone.

The refusal traps

Five of the 31 questions are traps: engineering questions whose answers are not in the corpus, sitting close enough to the domain to be tempting. The correct behavior is declining, and the system prompt instructs the exact sentence to use. The system declined 5 of 5. There are also five borderline questions living in the gray zone between answerable and not, because a system that only ever sees clean cases is a system you know nothing about.

Refusal-as-a-feature changes how you demo. Most RAG demos hope you ask something the corpus covers. This one includes questions where the right answer is “I don’t have information on that in the provided documents,” and shows the system saying so.

What the honest numbers cost

I expected hybrid retrieval to win because engineers ask about exact strings such as acronyms, standard numbers, and valve designations. The result went the other way. Dense-only retrieval reached 0.923 Recall@3 and 0.817 MRR. Hybrid RRF reached 0.885 and 0.774. Adding the cross-encoder recovered Recall@3 to 0.923 but lowered MRR to 0.741 and tripled latency. The API keeps both controls for comparison; the free deployment leaves the reranker off.

Every run prints latency and generation cost. Production uses Groq’s free plan for generation and OpenAI only for retrieval embeddings, so a synthesized query retains one small paid embedding call.

The limitation I would state before anyone asks: 31 queries is a small exam. It is big enough to expose retrieval regressions and refusal failures during development, and too small to certify anything. Growing it is the roadmap, and every number above says “on 31 labeled queries” wherever it appears.

Ask it a graded question yourself

The live demo at rag.alvinalias.com has a button that draws from the same graded exam: it asks the question, then prints the labeled source next to whatever the answer cited, hit or miss, and one rotation in the set is a trap where the correct outcome is a refusal. The eval set, protocol, and pipeline are in the repo.

Related reading

• I stopped asking an LLM to write every RAG answer, the V2 routing, verified facts path, classifier benchmark, and provider call map.

The rule

When you score a transaction, every history feature must be computed from rows strictly before that transaction. Lifetime return rate, return velocity, average days to return, order counts, tenure: all of them get a cutoff at the moment being scored, none of them may see the row itself or anything after it.

The implementation shape that makes this cheap: sort by customer and timestamp, then build history features with expanding or windowed aggregations that shift by one row, so each transaction sees only its own past. In pandas that is a groupby plus shift before any expanding aggregation; in SQL it is a window frame ending at 1 PRECEDING. The moment you see a history feature built without a shift, assume leakage until proven otherwise.

Why the split has to be temporal too

Point-in-time features fix leakage inside rows; the split fixes it between training and evaluation. A random split on transaction data puts January’s rows in training and December’s siblings in test, for the same customers, in both directions of time. The model partly memorizes customers instead of learning behavior, and the test score inflates.

My split: train on December 2009 through June 2011, test on July through December 2011. The model never sees the future of any customer it is judged on. The classifier holds 0.992 ROC-AUC under that discipline, which is exactly why the number is worth quoting; the same architecture under a random split would report something flattering and useless.

The self-check: a rolling backtest

One temporal split is one snapshot. To know whether performance is stable rather than lucky, re-fit and score across rolling monthly windows: train up to month k, score month k+1, slide, repeat. Eighteen windows on this dataset gave a mean top-decile precision of 0.181, and, just as important, a spread you can look at. If a single test-set number is much better than your rolling mean, the snapshot is flattering you.

The checklist

1. Every history feature has an explicit as-of cutoff, enforced by a shift or window frame, not by intention.
2. The train/test boundary is a date, not a random seed.
3. Customers do not appear on both sides of the boundary with the model expected to generalize to them as strangers, unless that is deliberately your setting; know which setting you are in.
4. A rolling backtest exists, and the headline metric survives it.
5. Anything the model consumes at serving time is something you could actually have known at that moment: if the feature table is rebuilt nightly, your features are as of last night, and evaluation should mimic that staleness.

Item five is the one that separates paper pipelines from production ones, and it is the first question I would ask anyone presenting a transaction model.

Where this is applied

The live return-risk demo at returns.alvinalias.com scores curated real invoice lines with features built under exactly these rules. The visible ledger shows customer, invoice, stock code, quantity, price, and country. The API also receives the historical context that would have been known at that invoice moment, including the quantity and price z-scores, weekend flag, month-end proximity, and product return rate. The repo with the feature pipeline, temporal split, demo-case builder, and backtest notebook is at github.com/aalias01/retail-returns-intelligence.

Changelog

• 2026-07: updated the live-demo note after moving the frontend to curated real invoice samples.
• 2026-05: first published.

The problem, briefly

NASA’s C-MAPSS FD001 dataset simulates 100 turbofan engines running to failure, 20,631 training cycles in all, each cycle a row of 21 sensor readings: temperatures, pressures, shaft speeds, fuel flow. The task is remaining useful life: given an engine’s sensor history, predict how many cycles remain. You train on complete run-to-failure trajectories and predict on engines cut off at an arbitrary point, scored against the official held-back answers.

Seven of the 21 sensors barely move in FD001, so they carry nothing. That leaves 14 informative channels, and one convention worth knowing: remaining life is capped at 125 cycles, because a healthy engine’s sensors look identical whether it has 130 or 300 cycles left. Predicting “125 or more” is the honest ceiling.

Decision one: split by engine, never randomly

The most common way to get a flattering RUL number is a random train/validation split. Rows from the same engine land on both sides, the model memorizes each engine’s personal drift, and validation error collapses. It isn’t prediction, it’s recognition.

Every split in this project is grouped by engine: an engine’s entire trajectory is either training or validation, never both. The validation score got worse the day I did this, which is exactly the point. The number stopped lying.

If you take one thing from this entry: on any dataset where rows belong to units that degrade, group your splits by unit and expect the honest number to be uglier.

Decision two: size the window to the physics

The failure mode in FD001 is compressor fouling. It is slow. Cycle to cycle, the sensor drift is smaller than the noise; over tens of cycles, it is unmistakable. So the features are 30-cycle rolling means and standard deviations of each sensor, and 30 wasn’t tuned by grid search: it approximates one fouling cycle, the timescale on which the degradation physically accumulates.

This is the part I’d defend in any review: the window is where the domain knowledge lives. An LSTM is asked to discover that timescale from data. A rolling mean is told the timescale by someone who knows the machine. On a dataset this size, telling beats discovering:

Worth saying plainly: transformer and hybrid models published since have pushed FD001 below all of these. 15.85 is a strong classical result, not the state of the art. What makes it interesting is the ratio of result to machinery.

The model agrees with the thermodynamics

The top SHAP feature for most predictions is the 30-cycle mean of sensor 3, the HPC outlet temperature. That is the textbook fouling signature: as the compressor fouls, efficiency drops and outlet temperature climbs. The next features in the ranking are the upstream and downstream sensors that respond to the same fault. Nobody told the model any thermodynamics; checking that its explanations recover the known physics is the closest thing tabular ML has to a sanity proof.

What I’d redo

The prediction ships with a ±15 cycle band, and that band is a heuristic, labeled as such in the API schema itself. It is not a calibrated interval, and pretending otherwise would be worse than the crude band. The fix is known (conformal prediction or quantile regression) and it is the top of the V2 list, along with extending past FD001’s single operating condition, where this feature set would need regime clustering before any rolling statistics make sense.

Try it against the official answers

The live demo at turbofan.alvinalias.com does something I wish more ML demos did: one click loads a real engine from NASA’s held-back test set, the model commits to an estimate, and then the official answer prints next to it, error included. Some runs miss. That’s what 15.85 RMSE means, and watching it miss honestly tells you more about the model than any metric table. Code, notebooks, and the model card are in the repo.