TopolearnGood features often matter more than the model, and doing it in the pipeline keeps it leak-free.
Feature engineering belongs in the pipeline so it is refitted per fold and shipped with the model. scikit-learn provides building blocks:
FunctionTransformer(func)wraps any stateless function, such asnp.log1pfor skewed amounts, or a function that derivesprice_per_m2from a DataFrame. Passfeature_names_outso names survive.PolynomialFeatures(degree=2, interaction_only=False)adds squares and products, letting linear models fit curves and interactions. The number of columns grows quickly.SplineTransformer(n_knots, degree)creates smooth piecewise-polynomial bases per feature: often a better way to model non-linear effects than high-degree polynomials.extrapolation="periodic"suits cyclic features such as hour of day.KBinsDiscretizerbuckets a numeric feature.TargetEncoderreplaces each category with a smoothed mean of the target. It is excellent for high-cardinality categories (zip codes, product IDs) where one-hot would create thousands of columns. Itsfit_transformuses internal cross fitting so a row's encoding never uses its own target, which prevents leakage (control the folds withcv=). That meansfit(X, y).transform(X)gives different values fromfit_transform(X, y)on training data; always usefit_transformin training, which pipelines do automatically.
Any stateful step (one that learns from data) must be a transformer, not a pandas operation done beforehand on the full dataset.
Example
import numpy as np
import pandas as pd
from sklearn.preprocessing import FunctionTransformer, PolynomialFeatures, TargetEncoder
log = FunctionTransformer(np.log1p, feature_names_out="one-to-one")
print(log.fit_transform(pd.DataFrame({"amount": [0, 9, 99]})).round(2).values.ravel())
# [0. 2.3 4.61]
poly = PolynomialFeatures(degree=2, include_bias=False).fit(np.zeros((1, 2)))
print(poly.get_feature_names_out(["a", "b"])) # ['a' 'b' 'a^2' 'a b' 'b^2']
rng = np.random.default_rng(0)
zip_code = rng.choice(["z1", "z2", "z3"], size=300).reshape(-1, 1)
y = (zip_code.ravel() == "z1") * 1.0 + rng.normal(0, 0.1, 300)
enc = TargetEncoder()
Xt = enc.fit_transform(zip_code, y) # cross-fitted on training data
print(Xt.shape, np.round(enc.encodings_[0], 2)) # (300, 1) [0.98 0. 0. ]Remember
- Wrap stateless functions in
FunctionTransformer, stateful logic in real transformers - Splines or polynomials let linear models fit non-linear effects
TargetEncoderhandles high-cardinality categories with built-in cross fitting
Watch out for
- Computing target means per category with pandas on the full data, which leaks the target.