TopolearnThe best score from a search is optimistic, and the bias grows with the number of candidates tried.
When you choose hyperparameters by maximising a CV score, the winning score is partly luck: you picked the candidate whose noise was most favourable. With many candidates and little data, best_score_ can be well above what the model achieves on new data. This is selection bias, a mild form of overfitting to the validation folds.
Nested cross-validation separates the two jobs:
- an inner loop (the search object) chooses hyperparameters on each outer training split,
- an outer loop evaluates the whole procedure, search included, on data the search never saw.
In scikit-learn it is a one-liner because a search object is an estimator: cross_val_score(GridSearchCV(pipe, grid, cv=inner), X, y, cv=outer). The outer score estimates the performance of "tune then fit", not of one specific configuration, and the best parameters may differ between outer folds, which itself tells you how stable the choice is.
Nested CV multiplies cost (outer × inner × candidates fits). With enough data, a simple train/test split where the search uses only the training part is equivalent in spirit. Other habits that keep evaluation honest: decide the metric before looking, keep a final untouched test set, and report the spread of scores, not just the mean. When comparing two models' CV scores, remember fold scores are correlated, so naive t-tests overstate significance.
Example
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import GridSearchCV, KFold, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = load_breast_cancer(return_X_y=True)
X, y = X[:150], y[:150] # small data makes the bias visible
pipe = make_pipeline(StandardScaler(), SVC())
grid = {"svc__C": [0.01, 0.1, 1, 10, 100], "svc__gamma": [1e-4, 1e-3, 1e-2, 0.1, 1]}
inner = KFold(5, shuffle=True, random_state=1)
outer = KFold(5, shuffle=True, random_state=2)
search = GridSearchCV(pipe, grid, cv=inner).fit(X, y)
print(round(search.best_score_, 3)) # 0.953 (optimistic)
nested = cross_val_score(GridSearchCV(pipe, grid, cv=inner), X, y, cv=outer)
print(nested.mean().round(3), nested.std().round(3)) # 0.94 0.044Remember
best_score_is biased upward by the selection itself- Nested CV = a search object evaluated by an outer
cross_val_score - The outer score rates the tuning procedure, not one configuration
Watch out for
- Comparing many models by their best_score_ and reporting the winner's score as its expected performance.