What a test-set threshold sweep can and cannot tell you

Cost assumptions move an operating threshold, but selecting and reporting it on the same labels makes the result exploratory.

Every binary classifier needs an operating threshold, and 0.5 is already an implicit decision about the tradeoff between misses and false alarms. In this project I swept that threshold under stated cost assumptions. The exercise is useful, but the original workflow selected 0.775 and reported its performance on the same test labels. That makes the threshold-dependent result exploratory, not an unbiased deployment recommendation.

The accuracy decoy

The dataset (AI4I 2020, ten thousand machine records) has failures in 3.4% of rows. A model that predicts “no failure” every single time scores 96.6% accuracy and catches nothing. So accuracy is not reported anywhere in this project, and any imbalanced-classification result quoted without that disclaimer deserves a raised eyebrow.

PR-AUC is more informative than accuracy for this imbalanced split. XGBoost scored 0.841, compared with 0.820 for random forest and 0.455 for logistic regression. Recall, false alarms, and cost depend on the operating threshold, so they need a separate selection protocol.

Charging each error its price

Model the two mistakes in money. A missed failure becomes unplanned downtime: modeled here at $50,000. A false alarm becomes a scheduled maintenance call that wasn’t needed: $2,000. Both numbers are stated assumptions, not measurements, and everything downstream inherits them, which is why they sit in the open on the demo page.

Then sweep the threshold and charge every mistake at every setting:

for threshold in np.arange(0.1, 0.9, 0.05):
    preds = (failure_proba >= threshold).astype(int)
    fn_cost = ((preds == 0) & (y_test == 1)).sum() * 50_000
    fp_cost = ((preds == 1) & (y_test == 0)).sum() * 2_000
    total_cost = fn_cost + fp_cost

On this test-set sweep, equal miss and false-alarm counts favor a threshold near 0.95. Under the assumed 25:1 cost ratio, the modeled minimum moves to 0.775. At that point the same test data show 62 of 68 failures caught, 6 missed, 83 false alarms, and $466K of modeled total cost. Those values describe the sweep. They do not estimate how a threshold chosen without access to final labels would perform on new data.

The part that generalizes

The threshold is not a modeling constant. Change the cost assumptions and the minimizing point moves. The model did not change; the decision rule did. A production workflow should select that rule on a validation set or inside nested evaluation, then report it once on untouched final data.

The live demo lets you move the two assumed costs and recomputes the curve from exported test-set error counts. It is an educational sensitivity analysis. Reproducing the 6 misses and 83 false alarms at 0.775 checks implementation consistency, but it does not repair the fact that the test labels selected the threshold.

Owning the misses

The demo also reveals misses on sampled records. That is useful for inspecting failure modes, but the 91.2% recall remains a descriptive test-sweep result until the threshold is selected independently and evaluated on untouched data.

Try it at machine-failure.alvinalias.com; the sweep, the notebooks, and the model card are in the repo.