TopolearnHigh-cardinality and interacting categorical features are where CatBoost wins or overfits, and these settings decide which.
CatBoost calls the numeric encodings of categorical features CTRs (a term from click-through-rate modelling). Each categorical feature can generate several CTRs:
Borders: ordered target statistics over target bins (default for classification and regression)Counter: how frequent the category is, which uses no labels and also works on test dataBucketsandBinarizedTargetMeanValue, plus GPU-specific types such asFloatTargetMeanValue
Settings use a compact syntax, for example simple_ctr=["Borders:Prior=0.5", "Counter"]. simple_ctr applies to single features, combinations_ctr to combinations, and per_feature_ctr targets specific columns. counter_calc_method="Full" counts frequencies over train and test data together.
`one_hot_max_size`: features with at most this many distinct values are one-hot encoded instead of getting CTRs. Raising it (say to 10) can help low-cardinality features; raising it far is costly.
Feature combinations: when a tree already splits on categorical features, CatBoost greedily tries combining them with other categorical features, creating new CTR features such as city x device. max_ctr_complexity limits how many features one combination may contain; 1 disables combinations, and low values speed training and reduce overfitting on noisy categories.
model.get_feature_importance(prettified=True) reports importance for the original features, while get_feature_importance(type="Interaction") and the model's CTR tables reveal which combinations were used.
Example
import numpy as np
import pandas as pd
from catboost import CatBoostClassifier
rng = np.random.default_rng(0)
n = 3000
df = pd.DataFrame({"city": rng.choice([f"c{i}" for i in range(40)], n),
"device": rng.choice(["ios", "android", "web"], n),
"plan": rng.choice(["free", "pro"], n)})
# the label depends on an INTERACTION of city and device
y = ((df["city"].str[-1].astype(int) % 2 == 0) == (df["device"] == "ios")).astype(int)
def fit(**kw):
m = CatBoostClassifier(iterations=200, verbose=0, random_seed=0,
allow_writing_files=False, **kw)
m.fit(df[:2400], y[:2400], cat_features=[0, 1, 2], eval_set=(df[2400:], y[2400:]))
return round(m.get_best_score()["validation"]["Logloss"], 3)
print("default:", fit()) # ~0.0: combos find it
print("no combos:", fit(max_ctr_complexity=1)) # ~0.57: much worse
print("one-hot small:", fit(one_hot_max_size=4)) # device and plan one-hot
print("custom ctr:", fit(simple_ctr=["Borders:Prior=0.5", "Counter"]))Remember
- CTRs are numeric encodings of categories: Borders, Counter, Buckets...
- simple_ctr / combinations_ctr / per_feature_ctr configure them
- one_hot_max_size decides which features are one-hot encoded
- max_ctr_complexity limits greedy categorical combinations
Watch out for
- Setting max_ctr_complexity=1 to speed up training on data whose signal lives in category interactions.