Why 0.775 is the right threshold when a miss costs $50K

Accuracy is a decoy on imbalanced failure data. Charge each error its real price, sweep the threshold, and let the minimum tell you where to act.

Every classifier ships with a default decision threshold of 0.5, and on industrial failure data that default is quietly making an economic decision nobody signed off on: it prices a missed machine failure and an unnecessary maintenance call as equal mistakes. In any plant I ever worked in, they are not within an order of magnitude of equal. This entry is about moving one number, and why that move is the most defensible modeling decision in the project.

The accuracy decoy

The dataset (AI4I 2020, ten thousand machine records) has failures in 3.4% of rows. A model that predicts “no failure” every single time scores 96.6% accuracy and catches nothing. So accuracy is not reported anywhere in this project, and any imbalanced-classification result quoted without that disclaimer deserves a raised eyebrow.

The metrics that survive imbalance: PR-AUC (0.841 for the XGBoost model, against 0.820 random forest and 0.455 logistic baselines) and recall at whatever operating point you choose. The operating point is the story.

Charging each error its price

Model the two mistakes in money. A missed failure becomes unplanned downtime: modeled here at $50,000. A false alarm becomes a scheduled maintenance call that wasn’t needed: $2,000. Both numbers are stated assumptions, not measurements, and everything downstream inherits them, which is why they sit in the open on the demo page.

Then sweep the threshold and charge every mistake at every setting:

for threshold in np.arange(0.1, 0.9, 0.05):
    preds = (failure_proba >= threshold).astype(int)
    fn_cost = ((preds == 0) & (y_test == 1)).sum() * 50_000
    fp_cost = ((preds == 1) & (y_test == 0)).sum() * 2_000
    total_cost = fn_cost + fp_cost

The minimum lands at 0.775. At that threshold, on the held-out test set: 62 of 68 real failures caught (91.2% recall), 6 missed, 83 false alarms tolerated, $466K modeled total cost. Compare the intuition to the default: because a miss costs 25 times a false alarm, you might expect the threshold to drop below 0.5 to catch more failures. It moves the other way, up to 0.775, because this model’s probabilities are well separated: the failures it can catch, it catches with high confidence, so raising the bar sheds expensive false alarms while giving up almost no real catches. That non-obvious direction is exactly why you sweep instead of reasoning by vibes.

The part that generalizes

The threshold is not a modeling constant; it is a business input wearing a decimal point. Change the cost assumptions and the optimum moves. Double the false-alarm cost (a plant where techs are scarce) and the optimum climbs higher; make misses catastrophic (safety-critical equipment) and it slides down. The model did not change. The economics did.

That is also why the live demo now lets you move the costs yourself: the page ships the real per-threshold error counts from the same test set the published numbers come from, and recomputes the cost curve and its minimum in front of you. The one non-negotiable behind that feature: the exported counts had to reproduce the published confusion matrix at 0.775 exactly, 6 and 83, or the export fails the build. Re-derivations must agree with published numbers before they earn trust.

Owning the misses

91.2% recall means about 9 in 100 failures slip past. The demo draws real records from the dataset with their outcomes hidden, predicts, then reveals what actually happened, and when a run lands on a miss, the page says so in plain text. A maintenance manager deciding whether to trust a model needs to know what its misses look like more than they need another decimal of AUC.

Try it at machine-failure.alvinalias.com; the sweep, the notebooks, and the model card are in the repo.