TopolearnTree ensembles are the default high-performing choice for tabular data.
Random forests (RandomForestClassifier/Regressor) train many deep trees, each on a bootstrap sample of rows and considering a random subset of features at each split (max_features). Averaging decorrelated trees cuts variance dramatically. They are robust with defaults, rarely overfit badly as you add trees, and parallelise with n_jobs=-1. oob_score=True gives a free validation estimate from the rows each tree did not see.
Gradient boosting builds shallow trees one after another, each fitting the errors of the ensemble so far, scaled by learning_rate. It usually reaches higher accuracy than a forest but needs more tuning and can overfit if it runs too long.
scikit-learn has two implementations. Prefer HistGradientBoostingClassifier/Regressor: it bins features into histograms, is orders of magnitude faster on large data, handles missing values natively, supports categorical features (categorical_features="from_dtype" with pandas category columns), and stops early automatically on data with more than 10,000 samples (early_stopping="auto"). The older GradientBoostingClassifier is slower and best kept for small datasets.
Key knobs for boosting: learning_rate and max_iter trade off together (lower rate, more iterations), max_leaf_nodes or max_depth control tree complexity, and l2_regularization and min_samples_leaf regularise. XGBoost and LightGBM are dedicated libraries built on the same idea.
Example
from sklearn.datasets import make_friedman1
from sklearn.ensemble import RandomForestRegressor, HistGradientBoostingRegressor
from sklearn.model_selection import train_test_split
X, y = make_friedman1(n_samples=20_000, noise=1.0, random_state=0) # non-linear target
X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=0)
rf = RandomForestRegressor(n_estimators=100, n_jobs=-1, oob_score=True, random_state=0)
rf.fit(X_tr, y_tr)
print(round(rf.oob_score_, 3), round(rf.score(X_te, y_te), 3)) # 0.918 0.92
hgb = HistGradientBoostingRegressor(learning_rate=0.1, max_iter=500, random_state=0)
hgb.fit(X_tr, y_tr)
print(hgb.n_iter_, round(hgb.score(X_te, y_te), 3)) # 141 0.95 (stopped early)Remember
- Forests average deep, decorrelated trees; boosting adds shallow trees that correct errors
- Prefer
HistGradientBoosting*overGradientBoosting*for speed and NaN support - Lower learning rate needs more iterations; early stopping picks the count
Watch out for
- Tuning a random forest's n_estimators as if more trees could overfit; more trees mostly costs time.