A measurement of how logistics ML's favourite benchmark number is made — and what the honest number underneath it looks like.
There is a number that keeps appearing in supply-chain machine learning. Papers report it, master's theses defend it, GitHub portfolios headline it: about 97% accuracy at predicting late deliveries on the DataCo SMART Supply Chain dataset, the field's most popular public benchmark. A peer-reviewed 2026 paper reports 97.47% with a metaheuristically tuned XGBoost. A thesis reports 97.58% and adds, proudly, that there were no false negatives at all. Dozens of public notebooks land within a point of the same figure.
Here is the thing about that number. The dataset's target column,
Late_delivery_risk, is not an observation. It is an arithmetic consequence of
two other columns in the same table: it equals 1 exactly when Days for
shipping (real) exceeds Days for shipment (scheduled). I measured this
directly on all 180,519 rows: the comparison reproduces the label 97.55% of the
time, and every single one of the 4,423 disagreements is a canceled shipment,
where the label is forced to zero. The label's full recipe is: late if the
shipment took longer than scheduled, and wasn't canceled.
If a model is allowed to see the "real shipping days" column — how long the delivery actually took — then predicting lateness is not prediction. It is subtraction.
To check that this explains the literature's results rather than merely casting suspicion on them, I ran the same three model families the papers use (logistic regression, random forest, gradient boosting) on the same data, four ways. Same seed, same splits, same code — the only thing that changes is which columns the models are allowed to see.
Give the models everything, including the Delivery Status column — which,
I measured, maps to the label with zero exceptions — and all three models score
a perfect 100.0% accuracy, F1 of 1.0, AUC of 1.0. No model deserves credit for
this; the answer key is in the input.
Drop Delivery Status but keep "Days for shipping (real)" — and all three
models land between 97.2% and 97.7%.
That second band should look familiar. It is the literature's number. The 97.4% that gets published is not "impressively close to perfect" — it is the agreement rate between the leaked column and the label (97.55%), achieved by models that learned the label's own derivation, with the canceled-shipment exception accounting for the missing two and a half points. The models learned nothing about deliveries. They learned the formula that generated the answer column.
Now remove what a real system couldn't know. Keep only what exists at order time — product, quantities, prices, destination, shipping mode, scheduled days, payment type, date — and accuracy falls to 69.6–77.5% depending on the model, with the best AUC at 0.88.
Even that overstates it. The 77.5% belongs to a random forest evaluated on a random split, which lets the same customers and routes appear on both sides of the train/test line — the model partly memorises entities instead of learning transferable structure. Score it honestly, training on the first 80% of orders by date and testing on the rest, and every model converges to about 69% accuracy, AUC 0.74.
So the full gradient, measured end to end on identical data with identical models:
| What the model sees | Accuracy |
|---|---|
| Everything, incl. the status column | 100.0% |
| Everything except status | 97.2–97.7% ← the published number |
| Order-time features, random split | 69.6–77.5% |
| Order-time features, time-ordered split | ~69% |
Twenty-eight accuracy points separate the number that gets published from the number a deployed system would produce. Each methodological shortcut buys a tranche of the difference: the status column buys the last 2.5 points to perfection, the real-shipping-days column buys 28 points, and the random split buys a final 8-point illusion even after the leaked columns are gone.
To find out whether this is a few careless notebooks or the field's default, I censused the public work on this dataset: 18 academic works (papers, book chapters, theses) and 47 public code artifacts (35 GitHub repositories, 12 Kaggle notebooks) — 65 works in all. Every verdict required verbatim evidence: a quoted feature list, a quoted code line. Works whose features could not be verified (mostly paywalled papers) are recorded as unverifiable, not assumed guilty.
The result: 28 of 65 works verifiably train on leaked columns; 20 are clean; 17 could not be verified. Among the code artifacts alone, where the evidence is a grep away, 53% leak. The claimed accuracies sort themselves accordingly: the leaked cluster runs from 0.75 to a flat 1.00, with its mode at 97–100%; the clean cluster runs 0.69 to 0.85. Every audited accuracy claim above roughly 90% traced back to a leaked column or an engineered real-minus-scheduled feature. Above that line, in this literature, a high score is not a signal of a good model. It is a signal of the leak.
Among the works whose feature lists could be verified:
- A 2026 paper in Applied Sciences (MDPI) reports 97.47% and presents a SHAP interpretability analysis whose dominant features are the shipping-days and date columns — the label's own ingredients, celebrated as insight.
- A 2024 master's thesis reports 97.58% and states, verbatim, that "Days for shipping (real)" and "Days for shipment (scheduled)" "were found to be the main factors in the prediction." They are not the main factors. They are the definition.
- A 2024 arXiv paper on causal root-cause discovery keeps the real shipping days, the status column, and the label itself among its causal-graph variables, then reports that the strongest causal edges into delivery risk are — the status column (strength 10.7) and the shipping days (10.3), an order of magnitude above any genuine driver. The method rediscovered the label's algebra and presented it as root-cause analysis.
The code tells the same story with fewer manners. The most-voted Kaggle notebook on this task (166 votes, plus a 123-vote fork) claims 99.37% accuracy for late-delivery prediction; its drop-list removes the status column but lets both shipping-day columns through. Then, remarkably, the same notebook runs its own ablation — dropping the two day columns — and reports "nearly 84%." The leak and its refutation coexist in one artifact, fifteen points apart, and the 99 is the number in the headline.
A GitHub project hits 97.4% across all three of its models after dropping
Delivery Status "because it directly correlates with the target" — and
then deliberately keeps the real shipping days to "preserve its predictive
power." The predictive power being preserved is the label. Another repository
reports a flat 1.0 accuracy, and ships a Streamlit web app whose prediction
form asks the user to type in — "Days for shipping (real)" — how long the
delivery actually took, as an input for predicting whether it will be late.
The leak survived all the way to the user interface.
The census also caught the number spreading socially: one specific accuracy, 0.9766507866164414, appears digit-for-digit in three separate artifacts — an original notebook and two forks — propagated by copy-paste through an architecture that, at one point, regresses the shipping-day columns from features that include the label itself. And a caution for anyone auditing by score alone: several artifacts have the leaked column verifiably in their feature matrix but score only 0.62–0.82, because a linear or shallow model fails to cash the leak in. Leakage must be judged from the code, not inferred from the score.
Two works get it right. The EAGLE paper (arXiv 2604.05254) explicitly documents the leakage — noting the >0.99 feature-label correlation — rebuilds the task with temporal windows, and lands roughly ten points below the leaked cluster. And one lone GitHub author independently measured the same algebraic identity this audit starts from, canceled-shipment remainder and all. Neither has made the headline numbers go away.
This would be an amusing benchmark curiosity if the same claim-shape did not run through the logistics industry's commercial layer, where the numbers are bigger and the methodology is invisible.
Real-time visibility platforms — the systems shippers pay to predict arrival times — advertise their accuracy in public marketing: over 90% within a two-hour window on the final transit day; 95% ETA accuracy; "six times more accurate than carrier ETAs"; 90% of road delays predicted twelve hours ahead. None of these publish the sample, the baseline, the time horizon distribution, or the selection rules behind the figure. They are not auditable — which is precisely the point of showing what happens on the one logistics dataset where the claim can be audited. On public data, with the method in the open, the gap between the advertised number and the deployable number measured 28 accuracy points, and the advertised number turned out to be the label's own reflection.
And at the other end of the industry, the operators typically publish nothing at all: warehouse volumes, floor area, trucks per day — capacity figures, not one number about delivery reliability or picking accuracy. Between vendors who publish unauditable accuracy and operators who publish none, the freight industry's measurement layer is either marketing or missing.
If you buy prediction — ETA feeds, delay-risk scores, forecast accuracy — the questions this audit suggests are short: What exactly was the input at prediction time? What would a naive baseline (the carrier's schedule) have scored? Was the evaluation split by time? On the one public dataset where those questions can be forced, they moved the number from 97% to 69%.
Methods and measured facts: label derivation verified on all 180,519 rows; reproduction with scikit-learn (logreg / RF-200 / HistGB), stratified and temporal 80/20 splits, seed 42. Census verdicts assigned only on quoted evidence (feature lists, code lines); works whose features could not be verified are recorded as UNCLEAR, not assumed. Full tables and scripts: this repository.
The dataset is not redistributed here. Download DataCoSupplyChainDataset.csv
from Kaggle (shashwatwork/dataco-smart-supply-chain-for-big-data-analysis)
into data/, then:
cd repro
python repro_v1.py # setups A (leaked), B (clean random), C (clean temporal)
python repro_a2.py # setup A2 (Days-real kept, Delivery Status dropped)
Requires Python 3.10+, pandas, scikit-learn. Results land in results_*.json;
the runs behind this article are committed.
census/ holds the 65-work census (JSONL, one work per line, schema in
SCHEMA.md). Every LEAKED/CLEAN verdict carries verbatim evidence (a quoted
feature list or code line); works that could not be verified are UNCLEAR.
Corrections welcome — file an issue with the evidence and the record will be
amended.