TopolearnA single split is noisy; cross-validation gives a more reliable score and its spread.
k-fold cross-validation splits the training data into k folds, trains on k-1 of them and validates on the remaining one, k times. You get k scores; their mean is the estimate and their spread tells you how much to trust it.
cross_val_score(model, X, y, cv=5, scoring="roc_auc")returns the array of scores.cross_validate(...)returns a dict with fit times, scores for several metrics (scoring=["accuracy", "f1"]), optional training scores and fitted estimators.cross_val_predict(...)returns out-of-fold predictions for every sample, handy for plots and error analysis but not a substitute for a score.
With an integer cv and a classifier, scikit-learn uses StratifiedKFold, which keeps class proportions in each fold; for regressors it uses KFold. Neither shuffles by default. If your rows are sorted (by date, by class), pass a splitter explicitly: KFold(5, shuffle=True, random_state=0).
Pass the whole pipeline as the model, so preprocessing is refitted inside each fold. Cross-validation is for choosing models and hyperparameters on the training set; the final test set is still kept aside for one last check.
n_jobs=-1 runs folds in parallel. Five or ten folds is typical; more folds means less bias but more computation and noisier individual scores.
Example
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, cross_validate, StratifiedKFold
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, y = load_breast_cancer(return_X_y=True)
pipe = make_pipeline(StandardScaler(), LogisticRegression())
scores = cross_val_score(pipe, X, y, cv=5)
print(scores.round(3), scores.mean().round(3))
# [0.982 0.982 0.974 0.974 0.991] 0.981
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
res = cross_validate(pipe, X, y, cv=cv, scoring=["accuracy", "roc_auc"])
print(sorted(res)) # ['fit_time', 'score_time', 'test_accuracy', 'test_roc_auc']
print(res["test_roc_auc"].mean().round(3)) # 0.995Remember
- Report the mean and the spread of fold scores
- Integer
cvon a classifier means stratified folds without shuffling - Cross-validate the whole pipeline, not pre-transformed data
Watch out for
- Using cross_val_predict output to compute a single score and calling it the CV score.