TopolearnBusiness losses (asymmetric costs, special likelihoods) can be optimised directly instead of approximated.
Gradient boosting needs only the first and second derivatives of the loss per row, so you can supply your own.
scikit-learn API: pass a callable as objective:
def objective(y_true, y_pred): # y_pred: raw scores
return grad, hess
lgb.LGBMRegressor(objective=objective)Native API (LightGBM 4.x): put the callable in the params dict, params["objective"] = f, where f(preds, train_data) returns (grad, hess) and labels come from train_data.get_label().
Things to get right:
- Predictions passed in are raw scores (before any sigmoid or exp). For classification-like losses apply the link yourself; and
predictreturns raw scores for custom objectives, so apply the inverse link to get probabilities. - Hessians must be positive. For non-convex losses clip them to a small value.
- Set a sensible starting point: LightGBM's
boost_from_averagedoesn't know your loss, so passinit_score(native) orinit_score=infit, or accept a slower start from zero. - Pair it with a custom metric that reports the real loss, so early stopping optimises the right thing.
Verify gradients numerically on a few points; a sign error trains a model that gets steadily worse while appearing to run fine.
Example
import lightgbm as lgb
import numpy as np
UNDER_COST = 3.0 # under-forecasting stock costs 3x more
def asym_l2(y_true, y_pred): # exactly (y_true, y_pred); y_pred = raw score
r = y_pred - y_true
w = np.where(r < 0, UNDER_COST, 1.0)
return 2 * w * r, 2 * w # gradient, positive hessian
rng = np.random.default_rng(5)
X = rng.uniform(0, 10, size=(5000, 1))
y = 2 * X[:, 0] + rng.normal(0, 3, 5000)
plain = lgb.LGBMRegressor(n_estimators=200, verbose=-1).fit(X, y)
custom = lgb.LGBMRegressor(objective=asym_l2, n_estimators=200, verbose=-1)
custom.fit(X, y, init_score=np.full(len(y), y.mean())) # sensible start
pred = custom.predict(X) + y.mean() # raw output excludes init_score: add it back
print(round(float((plain.predict(X) < y).mean()), 2)) # 0.5 under-predicted share
print(round(float((pred < y).mean()), 2)) # 0.34 (biased upward, as intended)Remember
- Custom objectives return per-row gradient and positive hessian of the raw score
predictreturns raw scores; apply links and add back any init_score- Provide a starting point with init_score and a matching custom metric
Watch out for
- Forgetting that predictions exclude the init_score you supplied at fit time.