A temporal split did not save my transaction model

An audit of future-history and cancellation-sign leakage, why chronological evaluation still failed, and the checks I now require before trusting a transaction model.

last verified 2026-07-16

I thought I had protected a retail transaction model from leakage. I used a chronological train/test split, described the features as point in time, and added a rolling backtest. The evaluation looked unusually strong.

Then I audited the matrix that the model actually consumed.

The audit found two failures. Customer and category aggregates had been computed from the full dataset before they were joined onto earlier rows. Negative quantity and line revenue also identified cancellation rows, so parts of the feature vector exposed the outcome itself. I withdrew the classifier and backtest claims and quarantined the retained model.

The important lesson is not merely that random splits are risky. A chronological split cannot rescue a row whose features already contain the future or the target.

What looked correct

The intended evaluation had several good ingredients:

1. Training rows came before test rows in calendar time.
2. Historical behavior was supposed to be available only before each invoice.
3. A rolling monthly backtest supplemented the single holdout period.
4. The live demo used real invoice rows rather than invented combinations.

Those choices were necessary, but the implementation did not satisfy the assumptions behind them. The split controlled which rows entered train and test. It did not control how the upstream aggregate tables had been built.

Failure 1: future history was joined onto past rows

Every customer-history feature has a hidden clock. “What is this customer’s return rate?” is a different question at every invoice time. If I compute one lifetime value after loading all transactions and join it back by customer, an invoice from 2010 can inherit behavior from 2011.

The safe implementation must carry an explicit as-of cutoff. In pandas, history usually needs a customer sort plus a shift before an expanding or rolling aggregation. In SQL, the window must end at 1 PRECEDING, not the current row. For category or product aggregates, the same rule applies. A lookup table computed once over the full dataset is not point in time merely because the final model uses a date split.

One test catches this class of bug well: add or modify a future transaction, rebuild the features, and assert that every earlier row remains byte-for-byte unchanged.

Failure 2: cancellation signs exposed the outcome

In UCI Online Retail II, cancellations are represented by C-prefixed invoice numbers and negative transaction values. If the prediction row includes negative quantity or negative line revenue, the model does not need to forecast a later return. It can recognize a cancellation that has already happened.

This exposed a deeper target-definition problem. A forward model should start from an original purchase at a real decision time, then label whether a later cancellation can be matched to it within a stated horizon. Cancellation rows belong in the label-building process, not in the feature matrix presented for prediction.

The first schema assertion for the rebuild is therefore simple: every candidate scoring row must represent a purchase, and no field may encode a later cancellation or refund state.

Why the rolling backtest also failed

A rolling backtest is only as honest as the data construction inside each fold. If every fold reads an aggregate table built from the full period, moving the cutoff does not remove the leak. It repeats the same contaminated experiment at several dates.

Each fold must rebuild labels and features using only information available by that fold’s cutoff. Validation chooses models and thresholds. A later untouched period estimates final performance. Anything selected after looking at that final period is part of a new experiment and needs another holdout.

The repair standard

I will not restore a predictive claim until the rebuild passes all of these checks:

1. Prediction unit: each row is an original purchase, not a cancellation row.
2. Decision time: the exact moment of scoring is defined before feature work begins.
3. Outcome link: a later cancellation is matched to the purchase under a documented rule and horizon.
4. As-of features: customer, product, and category aggregates use only rows strictly before the decision time.
5. No outcome signs: invoice prefixes, negative quantity, negative revenue, and post-outcome status fields are excluded from predictors.
6. Fold-local construction: every validation and backtest fold rebuilds its feature tables from that fold’s available history.
7. Untouched evaluation: model choices and thresholds are frozen before the final temporal test.
8. Leakage tests: changing future rows cannot alter past features, and a feature availability audit proves each input could exist when the API receives a request.

I would also run an ablation on every suspicious feature family. If performance collapses after removing a cancellation sign or a full-history aggregate, that is evidence about the old shortcut, not evidence that the safer model should restore the number.

What remains public

The repository keeps the failed classifier, notebooks, API contract, and other project components so the audit trail is inspectable. The live demo labels the classifier as quarantined. Its legacy return_probability and risk_tier field names remain for compatibility, but the displayed number is an audit-only retained-model score. It is not a valid forward return estimate and should not support a customer decision.

Publishing the correction is more useful than leaving an impressive metric in place. The failure is a concrete reminder to audit the matrix that ran, not the feature story I intended to build.

Changelog

• 2026-07: rewrote the note after the feature audit, withdrew the classifier claims, and documented the rebuild standard.
• 2026-05: first published.