TopolearnRegularisation is the main lever for linear models with many or correlated features.
Regularisation adds a penalty on coefficient size to the loss, trading a little bias for much lower variance.
Ridge(L2 penalty,alpha * sum(w²)) shrinks all coefficients smoothly and handles correlated features well.Lasso(L1 penalty,alpha * sum(|w|)) pushes some coefficients to exactly zero, performing feature selection.ElasticNetmixes both withl1_ratio.
Larger alpha means stronger regularisation. Classifiers such as LogisticRegression and LinearSVC use the inverse convention: C is 1/alpha, so smaller C means stronger regularisation. LogisticRegression uses a pure L2 penalty by default; since 1.8 you choose the mix with l1_ratio (1 means L1) instead of the deprecated penalty argument, and L1 needs a compatible solver such as "saga".
Because the penalty compares coefficient sizes directly, scale features first; otherwise a feature measured in large units gets penalised less. Put the scaler and model in one pipeline.
Pick the strength by cross-validation. Built-in CV variants do it efficiently: RidgeCV(alphas=np.logspace(-3, 3, 13)), LassoCV, ElasticNetCV and LogisticRegressionCV search a path of values and expose the winner as alpha_ or C_. Search on a log scale: the useful range spans several orders of magnitude.
Example
import numpy as np
from sklearn.datasets import make_regression
from sklearn.linear_model import LinearRegression, LassoCV, RidgeCV
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
# 60 samples, 50 features, only 5 of them useful
X, y = make_regression(n_samples=60, n_features=50, n_informative=5,
noise=10, random_state=0)
for model in [LinearRegression(), RidgeCV(alphas=np.logspace(-3, 3, 13)),
LassoCV(random_state=0)]:
pipe = make_pipeline(StandardScaler(), model)
print(type(model).__name__, cross_val_score(pipe, X, y, cv=5).mean().round(3))
# LinearRegression 0.786
# RidgeCV 0.852
# LassoCV 0.991
lasso = make_pipeline(StandardScaler(), LassoCV(random_state=0)).fit(X, y)
print((lasso[-1].coef_ != 0).sum()) # 14 (most of the 45 useless features dropped)Remember
- Ridge shrinks, Lasso zeroes out coefficients, ElasticNet mixes both
alphaup orCdown means stronger regularisation- Scale features first and choose the strength on a log scale by CV
Watch out for
- Assuming larger C means more regularisation in LogisticRegression; it is the opposite.