TopolearnUnderstanding the algorithm explains what every hyperparameter really does.
For each boosting round, XGBoost computes for every training row the gradient g and hessian h (first and second derivative) of the loss with respect to the current prediction. For squared error, g = prediction - y and h = 1; for log loss, g = p - y and h = p(1 - p).
A new tree is fitted to these. For a leaf containing rows I, with G = sum of g and H = sum of h, the optimal leaf weight is w = -G / (H + lambda), and the leaf's contribution to the objective is -G² / (2(H + lambda)). A split into left and right children has
gain = 1/2 * [G_L²/(H_L+lambda) + G_R²/(H_R+lambda) - (G_L+G_R)²/(H_L+H_R+lambda)] - gamma
So reg_lambda shrinks leaf weights and gains, gamma is a fixed price per extra leaf, min_child_weight is a lower bound on H in each child, and reg_alpha soft-thresholds G. Leaf weights are multiplied by learning_rate before being added.
Split finding: the default tree_method="hist" buckets each feature into at most max_bin (256) quantile bins and accumulates gradient histograms per node, so finding the best split costs O(bins) rather than O(rows). A child's histogram is obtained by subtracting its sibling's from the parent's.
Sparsity-aware splits: missing values are excluded from the histograms and their G and H are tried on each side, giving the learned default direction.
base_score is the starting prediction, estimated from the target in recent versions.
Example
import numpy as np
import xgboost as xgb
X = np.array([[1.0], [2.0], [3.0], [4.0]])
y = np.array([1.0, 1.0, 3.0, 3.0])
params = {"max_depth": 1, "eta": 1.0, "lambda": 1.0, "base_score": 0.0,
"min_child_weight": 0, "objective": "reg:squarederror"}
booster = xgb.train(params, xgb.DMatrix(X, label=y), num_boost_round=1)
# by hand: g = pred - y = -y, h = 1; left {1,2}: G=-2, H=2; right {3,4}: G=-6, H=2
lam = 1.0
print(-(-2) / (2 + lam), -(-6) / (2 + lam)) # 0.6666666666666666 2.0 (leaf weights)
gain = 0.5 * (4 / 3 + 36 / 3 - 64 / 5)
print(round(gain, 4)) # 0.2667
df = booster.trees_to_dataframe()
print(df.loc[0, "Split"], round(df.loc[0, "Gain"], 4)) # 3.0 0.5333 (reported without the 1/2)
print(df.loc[1:, "Gain"].round(4).tolist()) # [0.6667, 2.0] (leaf values)Remember
- Each tree fits gradients and hessians; leaf weight = -G/(H + lambda)
lambda,gamma,min_child_weightappear directly in the gain formula- Histogram binning (
max_bin) makes split finding fast
Watch out for
- Thinking learning_rate only slows training; it shrinks every leaf weight, acting as regularisation.