TopolearnTrees capture non-linear rules with no scaling needed, and they are the building block of forests and boosting.
A decision tree splits the data with simple questions such as "is bmi <= 0.03?", choosing at each node the feature and threshold that best separate the target (by Gini impurity or entropy for classifiers, squared error for regressors). A new sample follows the questions down to a leaf, whose majority class or mean value is the prediction.
Trees are attractive because they:
- handle non-linear relationships and interactions automatically,
- need no feature scaling,
- can be read:
export_text(tree, feature_names=...)prints the rules.
Their weakness is overfitting. Left alone (max_depth=None), a tree grows until leaves are pure, memorising noise: training accuracy near 100%, mediocre test accuracy. Limit growth with:
max_depth: maximum number of levels,min_samples_leaf: each leaf must hold at least this many samples,ccp_alpha: cost-complexity pruning after growing.
Single trees are also unstable: a small change in data can produce a completely different tree. That is why in practice you mostly use ensembles of trees (random forests, gradient boosting), which keep the strengths and average away the instability. The fitted feature_importances_ attribute gives a quick, if biased, view of which features the tree used.
Example
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier, export_text
data = load_breast_cancer()
X_tr, X_te, y_tr, y_te = train_test_split(data.data, data.target, random_state=0)
for depth in [None, 3]:
tree = DecisionTreeClassifier(max_depth=depth, random_state=0).fit(X_tr, y_tr)
print(depth, tree.get_depth(), round(tree.score(X_tr, y_tr), 3), round(tree.score(X_te, y_te), 3))
# None 8 1.0 0.881
# 3 3 0.977 0.937
print(export_text(tree, feature_names=list(data.feature_names), max_depth=1))
# |--- mean concave points <= 0.05
# | |--- worst area <= 952.90
# | | |--- truncated branch of depth 2
# ...Remember
- Trees need no scaling and model non-linear interactions
- Unlimited depth memorises the training set; constrain
max_depthormin_samples_leaf - Single trees are unstable; ensembles fix that
Watch out for
- Judging a deep tree by its near-perfect training accuracy.