Regex confidence is a poor escalation signal

My hybrid extractor cut LLM calls by 56% and kept far less than 56% of the accuracy gain. The autopsy is more useful than the result.

Industrial companies sit on years of maintenance work orders: free text written by technicians about what broke and what they did. Extracting structured fields from that text (equipment tag, failure mode, parts, root cause) is a classic job for an LLM, and a classic place to worry about cost. The obvious architecture is a hybrid: run a cheap regex extractor first, escalate to the LLM only when the regex reports low confidence. I built exactly that, and its most useful output was a negative result.

The setup

The corpus is 3,000 synthetic work orders with a calibrated noise layer: typos, technician shorthand like “repl” and “brg” and “RTS,” vague observations, and terse one-liners, because clean synthetic text makes every extractor score a meaningless 100%. The taxonomy and vocabulary come from twelve years of writing real work orders across HVAC, subsea, and manufacturing. Ground truth is nulled per-field wherever the noise layer removed a fact from the text, so no extractor gets penalized for information that is not there.

Three extractors ran on the same 100-record evaluation sample: rule-based regex (free, under a millisecond), LLM-only (GPT-4o-mini with a Pydantic schema, about 1.1 seconds per record), and the hybrid, which escalated to the LLM whenever the regex’s self-reported confidence fell below 0.7.

The results, including the embarrassing column

On equipment tags, regex nearly matches the LLM (82% against 99%), because tags follow an XX-NNN pattern that regex was born for. On failure mode, the narrative field, regex collapses to 13% while the LLM reaches 70%: typos, shorthand, and paraphrase defeat keyword matching.

The hybrid cut LLM calls by 56%. If the gate were working, accuracy should sit most of the way toward the LLM’s numbers. On failure mode it landed at 23%: barely above regex, nowhere near 70%.

The autopsy

The gate assumed regex confidence correlates with regex correctness. It does not, and the direction of the failure is the interesting part: on clean, well-structured text the regex is confident and right, and on noisy text it is confident and wrong, because a keyword pattern that happens to match garbage reports the same confidence as one matching the real thing. The records that most needed escalation were exactly the ones that never got it.

That is a calibration failure, and it generalizes far beyond regex. Any pipeline that routes work based on a component’s self-reported confidence inherits that component’s calibration, and cheap extractors are almost never calibrated on the distribution that matters, which is the messy tail.

What a production version needs

An escalation signal that does not come from the extractor grading its own homework. The candidates I would try, in order: field-completeness checks (empty or suspiciously short extracted fields are a cheap, honest tell), a small classifier trained to predict extraction failure from surface features of the text, or spot-check sampling with drift monitoring. The cost at stake is real but small: LLM-only extraction runs roughly $0.10 to $0.15 per thousand records at GPT-4o-mini prices, so the hybrid has to be nearly free to be worth its complexity.

Negative results like this one rarely make it into portfolios, which is exactly why this one is in mine: the measurement worked, the architecture failed, and the failure has a nameable cause with a testable fix.

See the pipeline

The classifier this corpus trains (DistilBERT with LoRA, 94.4% F1 while updating 1.1% of parameters) and the similar-case retrieval run live at workorders.alvinalias.com; the extraction study, with the per-field table and the eval script, is in the repo.