TopolearnLeakage is the most common reason a model that scored well in development fails in production.
Data leakage means information that would not be available at prediction time reaches training or validation. Scores look great; production does not. Common sources:
- preprocessing fitted on all data before splitting (fixed by pipelines),
- features computed from the future or from the target, such as "account closed date" when predicting churn,
- duplicate or related rows in both train and validation.
The last one is about the splitter. Random K-fold assumes rows are independent. When they are not, pick a splitter that mirrors how the model will be used:
- Groups: several rows per patient, customer or device.
GroupKFold(n_splits)withgroups=passed tocross_val_scorekeeps each group in one fold, so you measure performance on unseen customers.StratifiedGroupKFoldalso balances classes. - Time:
TimeSeriesSplit(n_splits, gap=...)always validates on data after the training window.gapleaves a buffer between train and test when features use lagged windows.
A quick leakage check: if one feature alone gives near-perfect predictions, or if the CV score is far above what domain experts consider plausible, investigate before celebrating. Also compare random-CV and group/time-CV scores; a large drop means random CV was overly optimistic.
Example
import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GroupKFold, KFold, TimeSeriesSplit, cross_val_score
rng = np.random.default_rng(0)
groups = np.repeat(np.arange(50), 10) # 50 patients, 10 rows each
patient_effect = rng.normal(size=(50, 2))[groups] # rows of a patient look alike
X = patient_effect + rng.normal(0, 0.1, (500, 2))
y = (rng.normal(size=50)[groups] > 0).astype(int) # label is per patient, unrelated to X
rf = RandomForestClassifier(random_state=0)
print(cross_val_score(rf, X, y, cv=KFold(5, shuffle=True, random_state=0)).mean().round(2)) # 0.86 (leak!)
print(cross_val_score(rf, X, y, cv=GroupKFold(5), groups=groups).mean().round(2)) # 0.58
for tr, te in TimeSeriesSplit(n_splits=3, gap=1).split(np.arange(10)):
print(tr, te)
# [0 1 2] [4 5]
# [0 1 2 3 4] [6 7]
# [0 1 2 3 4 5 6] [8 9] (validation always after training, gap of 1)Remember
- Pick the splitter that mirrors deployment: groups for entities, time order for forecasts
- Pass
groups=to cross-validation functions when using group splitters - Implausibly high scores are a leakage signal
Watch out for
- Shuffling time-ordered data in K-fold, which trains on the future.