Main ideas
- Sequentially build small trees
- Fit each tree to the residuals
- Prediction:
- Sum of the trees’ predictions
- Each times a small shrinkage \(\nu\)
compressive_strengthcementblast_furnace_slagfly_ashwatersuperplasticizercoarse_aggregatefine_aggregateage# Boosting from scratch: B trees of depth 4, with shrinkage nu.
B <- 1000
nu <- 0.1
# Start from the fit 0, so the residuals are the strengths.
r <- y_train
trees <- vector("list", B)
# Fit a tree to the residuals, then subtract nu times its predictions.
for (b in seq_len(B)) {
data_b <- mutate(x_train, r = r)
trees[[b]] <- rpart(r ~ ., data = data_b, control = small)
r <- r - nu * predict(trees[[b]], x_train)
}# The training and test data in xgboost's format.
x_test <- select(test, -compressive_strength)
d_train <- xgb.DMatrix(x_train, label = y_train, nthread = 1)
d_test <- xgb.DMatrix(x_test, label = y_test, nthread = 1)
# B = 1,000 trees of depth 4 with shrinkage 0.1, and the test MSE.
params <- xgb.params(learning_rate = 0.1, max_depth = 4, nthread = 1)
fit <- xgb.train(params, d_train, nrounds = 1000)
mean((y_test - predict(fit, d_test))^2)[1] 24.0684
\[\frac{\sum_{i \in \text{leaf}} r_i}{n_{\text{leaf}} + \texttt{lambda}},\]
lambda — penalty that shrinks the leaf’s prediction toward 0; default 1| Fit | Settings | Training MSE | Test MSE |
|---|---|---|---|
| Loop of rpart() fits | start at 0 | 0.42 | 23.7 |
| xgboost, loop settings | lambda = 0, base_score = 0, tree_method = "exact" |
0.41 | 25.0 |
| xgboost, defaults | lambda = 1, start at the mean, tree_method = "hist" |
0.58 | 24.1 |
nrounds) — how many trees are addedlearning_rate) — how fast the model learnsmax_depth) — how complex each tree isxgboost settings (lambda, min_child_weight, gamma, subsample): defaults


\[\text{CV}_{(10)}(B) = \frac{1}{10}\sum_{\ell=1}^{10}\text{MSE}_\ell(B), \qquad \text{MSE}_\ell(B) = \frac{1}{n_\ell}\sum_{i \in \mathcal{F}_\ell}\left[y_i - \widehat f_B^{(-\ell)}(x_i)\right]^2,\]
\[\text{CV}_{(10)}(B) = \frac{1}{10}\sum_{\ell=1}^{10}\text{RMSE}_\ell(B)^2 = \overline{\text{RMSE}}(B)^2 + \frac{9}{10}\,s_{\text{RMSE}}(B)^2,\]
test_rmse_mean)test_rmse_std)xgb.cv() records only the mean and standard deviation of the fold RMSEsparams: \(\nu = 0.1\) and \(\delta = 4\) fixed
# 10-fold cross-validation for every number of trees up to 5,000.
cv_fit <- xgb.cv(params, d_train, nrounds = 5000, nfold = 10, verbose = 0)
cv_log <- cv_fit$evaluation_log
# The CV estimate of the test MSE after each number of trees; the best B.
cv_mse <- cv_log$test_rmse_mean^2 + 0.9 * cv_log$test_rmse_std^2
B_cv <- which.min(cv_mse)
c(B = B_cv, cv = cv_mse[B_cv]) B cv
1194.00000 20.03447

\(\nu = 0.1\), \(\delta = 4\), \(B = 1{,}194\)
\(\nu = 0.1\), \(\delta = 4\), \(B = 1{,}194\)
[1] 24.14392
| Method | Tuning | CV/OOB MSE | Test MSE |
|---|---|---|---|
| Least squares, main effects | none | – | 106.4 |
| Least squares with log(age) | none | – | 48.5 |
| Default tree | rpart() stopping rules | – | 86.8 |
| Full tree | none | – | 40.3 |
| Pruned tree | cp by 10-fold CV | 60.8 | 42.2 |
| Bagging, \(m = 8\) | none | 27.3 | 26.9 |
| Random forest, \(m = 5\) | \(m\) by OOB error | 26.8 | 27.5 |
| Boosting, \(\nu = 0.1\), \(\delta = 4\) | \(\nu\) and \(\delta\) fixed in advance, \(B = 1{,}194\) by 10-fold CV | 20.0 | 24.1 |
\[\frac{n_1 n_2}{n_1 + n_2}\left(\bar r_1 - \bar r_2\right)^2,\]
xgboost calls the decrease a split’s gain; the Gain column of xgb.importance() is each variable’s share\[\text{gain} = \frac{S_1^2}{n_1 + \texttt{lambda}} + \frac{S_2^2}{n_2 + \texttt{lambda}} - \frac{S^2}{n_1 + n_2 + \texttt{lambda}},\]
lambda — the leaf penalty of the Leaf predictions slidelambda = 0: the gain equals the decrease in RSS Feature Gain
<char> <num>
1: age 0.36214097
2: cement 0.29837558
3: superplasticizer 0.08909573
4: water 0.08908287
5: blast_furnace_slag 0.06942685
6: fine_aggregate 0.05293820
7: fly_ash 0.01997255
8: coarse_aggregate 0.01896725


\[\widehat f_B(x) = \sum_{j=1}^p a_j(x), \qquad a_j(x) = \sum_{b:\, \widehat f^b \text{ splits on } X_j} \nu\,\widehat f^b(x),\]

xgboost adds more, such as lambdaxgboost starts from the mean and shrinks leaf predictionsDS 3030