TopolearnThe tree structure explains CatBoost's very fast prediction, its regularising effect and its memory profile.
By default CatBoost grows oblivious (symmetric) trees: every node at the same depth uses the same split (feature and threshold). A depth-6 tree is therefore just 6 binary conditions and a table of 64 leaf values. A row's leaf index is the 6-bit number formed by its condition results.
Consequences:
- Prediction is extremely fast and branch-free: evaluate d comparisons, build an index, look up a value. It vectorises well on CPUs and GPUs.
- The structure is a strong regulariser. Each level must find a split that is good for all branches at once, so individual trees are weaker, and the ensemble generalises well with less tuning.
- Some patterns need more trees or depth than with leaf-wise trees.
grow_policy changes this: "Depthwise" (level by level, splits differ per node) and "Lossguide" (best leaf first, like LightGBM) with max_leaves. They can fit some data better but lose the speed advantage and some model-analysis features.
Before training, numeric features are quantized into at most border_count bins (254 on CPU by default), with borders chosen by feature_border_type (GreedyLogSum by default). Splits are only considered at these borders, which makes split search fast and memory-light. A Pool can be quantized once and reused.
Leaf values are computed with a Newton or gradient step (leaf_estimation_method), possibly several iterations per tree (leaf_estimation_iterations), scaled by the learning rate.
Example
import numpy as np
from catboost import CatBoostRegressor
rng = np.random.default_rng(0)
X = rng.uniform(0, 10, (500, 3))
y = np.sin(X[:, 0]) + 0.5 * X[:, 1] + rng.normal(0, 0.1, 500)
model = CatBoostRegressor(iterations=50, depth=3, border_count=32, verbose=0,
random_seed=0, allow_writing_files=False).fit(X, y)
borders = model.get_borders()
print({f: len(b) for f, b in borders.items()}) # at most 32 borders per feature
print(model.get_tree_leaf_counts()[:3]) # [8 8 8]: always 2**depth
leaves = model.calc_leaf_indexes(X[:2]) # leaf index per row per tree
print(leaves.shape, leaves[:, 0]) # (2, 50) and values in 0..7
lossguide = CatBoostRegressor(iterations=50, grow_policy="Lossguide", max_leaves=16,
verbose=0, allow_writing_files=False).fit(X, y)
print(lossguide.get_tree_leaf_counts()[:3]) # leaf counts vary per treeRemember
- Oblivious trees use one split per level: depth d gives 2^d leaves
- Prediction is a few comparisons plus a table lookup, very fast
- Symmetric structure regularises; Depthwise/Lossguide trade that for flexibility
- Numeric features are quantized into border_count bins before training
Watch out for
- Raising border_count to thousands expecting better accuracy; it mostly slows training.