TopolearnA handful of parameters account for most of the gains; knowing them saves hours of blind tuning.
CatBoost's defaults are tuned to be robust, so tune deliberately and in this rough order:
- `learning_rate` and `iterations`: the step size of each tree and the number of trees. They trade off: halve the rate, roughly double the trees. Fix a reasonable rate (0.03 to 0.1) and let early stopping choose iterations.
- `depth` (default 6): depth of each symmetric tree; a tree has 2^depth leaves. Typical range 4 to 10. Deeper trees capture more interactions but overfit faster and train slower.
- `l2_leaf_reg` (default 3): L2 regularisation on leaf values. Increase it (5 to 30) when overfitting.
- `random_strength`: noise added to split scores to diversify trees; higher values regularise.
- Row sampling:
bootstrap_typeisMVSby default on CPU (Bayesianon GPU). Tunesubsamplefor MVS and Bernoulli, orbagging_temperaturefor Bayesian, where higher means more aggressive sampling. - `border_count`: number of bins for numeric features (254 on CPU by default). Fewer bins train faster.
- `one_hot_max_size`, `max_ctr_complexity`: control categorical encoding and feature combinations.
Speed knobs: thread_count (defaults to all cores), task_type="GPU", and keeping boosting_type="Plain" rather than "Ordered" on large data.
model.get_all_params() shows every value actually used, including those chosen automatically such as the learning rate. Always tune against a validation set or cross-validation, never the training score.
Example
from catboost import CatBoostRegressor
from sklearn.datasets import make_friedman1
from sklearn.model_selection import train_test_split
X, y = make_friedman1(n_samples=2000, noise=1.0, random_state=0)
X_tr, X_val, y_tr, y_val = train_test_split(X, y, random_state=0)
def val_rmse(**params):
m = CatBoostRegressor(iterations=2000, early_stopping_rounds=100, verbose=0,
random_seed=0, allow_writing_files=False, **params)
m.fit(X_tr, y_tr, eval_set=(X_val, y_val))
return round(m.get_best_score()["validation"]["RMSE"], 3), m.tree_count_
print("default lr:", val_rmse())
print("lr=0.03:", val_rmse(learning_rate=0.03)) # (rmse, trees) to compare
print("depth=3:", val_rmse(depth=3))
print("depth=10:", val_rmse(depth=10))
print("l2=20:", val_rmse(l2_leaf_reg=20))
m = CatBoostRegressor(iterations=500, verbose=0, allow_writing_files=False).fit(X_tr, y_tr)
p = m.get_all_params()
print(p["depth"], p["l2_leaf_reg"], round(p["learning_rate"], 3)) # 6 3 <auto-chosen rate>Remember
- Fix a learning rate and let early stopping choose iterations
- depth (4-10) and l2_leaf_reg are the main capacity/regularisation knobs
- random_strength and row sampling (subsample, bagging_temperature) regularise
- get_all_params shows the values actually used, including auto-chosen ones
Watch out for
- Tuning learning_rate and iterations independently without early stopping, which mostly finds the same trade-off twice.