TopolearnThese variants can add accuracy or speed for specific data, and you will meet them in competition code.
boosting_type (alias boosting) and related options change how the ensemble is built.
DART (boosting_type="dart") applies dropout to trees: at each iteration some existing trees are temporarily dropped, so new trees cannot simply rely on them. It can improve accuracy on some problems but is slower, and early stopping does not work reliably because earlier trees are rescaled as training continues. Key parameters: drop_rate, skip_drop, max_drop.
GOSS (Gradient-based One-Side Sampling) keeps all rows with large gradients and samples a fraction of the rest, re-weighting them to stay unbiased. It speeds up training on large data. Since LightGBM 4.0 it is a data sampling strategy, set with data_sample_strategy="goss" (top_rate, other_rate), not a boosting type; GOSS is incompatible with bagging.
Random forest mode (boosting_type="rf") builds independent trees on bagged samples, requiring subsample < 1 with subsample_freq > 0 (or feature fraction < 1).
Linear trees (linear_tree=True) fit a linear model in each leaf instead of a constant, which helps when the target changes smoothly with features (extrapolation-like behaviour within leaves). Not available with categorical-only splits on all features and slower to train.
Start with plain gbdt; reach for these when you have a measured reason.
Example
import lightgbm as lgb
import numpy as np
from sklearn.model_selection import cross_val_score
rng = np.random.default_rng(3)
X = rng.uniform(0, 10, size=(3000, 3))
y = 3 * X[:, 0] + 2 * X[:, 1] + rng.normal(0, 1, 3000) # smooth, linear target
configs = {"gbdt": {}, "linear_tree": {"linear_tree": True},
"goss": {"data_sample_strategy": "goss"},
"rf": {"boosting_type": "rf", "subsample": 0.7, "subsample_freq": 1}}
for name, extra in configs.items():
m = lgb.LGBMRegressor(n_estimators=200, num_leaves=15, verbose=-1, **extra)
print(name, round(cross_val_score(m, X, y, cv=5, scoring="neg_root_mean_squared_error").mean(), 3))
# gbdt -1.122
# linear_tree -1.083 (leaves fit the linear trend)
# goss -1.129
# rf -2.37 (no boosting: weaker on this task)Remember
- DART can help accuracy but breaks early stopping
- GOSS is
data_sample_strategy="goss"in LightGBM 4 linear_tree=Truesuits smooth targets;rfmode needs bagging enabled
Watch out for
- Using boosting_type="goss" from old tutorials; in 4.x use data_sample_strategy.