Gradient Boosting for Classification

Gradient boosting for binary classification works in log-odds space. Start from $F_0 = \log\frac{\bar y}{1 - \bar y}$; each round:

1. residuals $r = y - \sigma(F)$
2. fit a regression stump to the residuals: try every feature and every midpoint between consecutive distinct sorted values, splitting into x <= t and x > t, and keep the split with the lowest squared error of the residuals around each side's mean (ties: lower feature, then lower threshold)
3. set each leaf's value with a Newton step: $\gamma = \frac{\sum r}{\sum p(1-p)}$ over the leaf's samples
4. $F \leftarrow F + \eta\,\gamma_{\text{leaf}(x)}$