TopolearnKnowing the internals explains max_bin, sparse-data speed and memory use, and when settings matter.
LightGBM's speed comes from a few ideas from its original paper.
Histogram binning. Before training, each feature is bucketed into at most max_bin (default 255) bins using quantiles computed on up to bin_construct_sample_cnt rows (default 200,000). Split finding then scans bins instead of sorted values: building a node's histogram costs O(rows in node), and finding its best split costs O(bins). Binned values fit in 8 bits, cutting memory. min_data_in_bin and max_bin_by_feature refine it; more bins mean finer splits and more memory.
Histogram subtraction. A child's histogram equals the parent's minus its sibling's, so LightGBM builds only the smaller child's histogram and derives the other.
Exclusive Feature Bundling (EFB). Sparse features that are rarely non-zero at the same time (such as one-hot columns) are bundled into a single feature by offsetting their bin ranges. Thousands of sparse columns may become a few dense ones, which is why LightGBM is fast on sparse data. enable_bundle and max_conflict_rate control it.
GOSS keeps the top top_rate fraction of rows by gradient magnitude, samples other_rate of the rest, and multiplies the sampled small-gradient rows by (1 - top_rate) / other_rate so gain estimates stay approximately unbiased.
Parallel learning (feature, data and voting parallel) distributes histogram building across machines.
Example
import lightgbm as lgb
import numpy as np
from scipy import sparse
rng = np.random.default_rng(4)
n = 20000
cats = rng.integers(0, 2000, n) # one-hot of 2000 categories
X = sparse.csr_matrix((np.ones(n), (np.arange(n), cats)), shape=(n, 2000))
y = (cats % 7 == 0).astype(float) + rng.normal(0, 0.1, n)
params = {"objective": "regression", "max_bin": 63, "min_data_in_leaf": 5, "verbosity": -1}
ds = lgb.Dataset(X, label=y, params=params).construct()
print(ds.num_feature()) # 2000 (original features)
booster = lgb.train(params, ds, num_boost_round=50)
pred = booster.predict(X[:4])
print(np.round(pred, 2), (cats[:4] % 7 == 0).astype(int)) # [0.08 0.08 0.08 0.55] [0 0 0 1]Remember
- Features are binned into at most
max_binquantile bins before training - Histogram subtraction and EFB make sparse, wide data fast
- GOSS reweights sampled small-gradient rows to keep gains unbiased
Watch out for
- Raising max_bin to thousands for tiny gains while multiplying memory use.