compressive_strengthcementblast_furnace_slagfly_ashwatersuperplasticizercoarse_aggregatefine_aggregateage\[ \begin{aligned} Var\left[\frac{1}{B}\sum_{b=1}^B \widehat f_b(x)\right] &= \frac{1}{B^2}\left\{\sum_{b=1}^B Var\left[\widehat f_b(x)\right] + \sum_{b \ne b'} Cov\left[\widehat f_b(x), \widehat f_{b'}(x)\right]\right\} \\ &= \frac{B\sigma^2 + B\left(B - 1\right)\rho\sigma^2}{B^2} = \rho\sigma^2 + \frac{1 - \rho}{B}\sigma^2. \end{aligned} \]
Probability that a given observation is missed by all \(n\) draws
\[\left(1 - \frac{1}{n}\right)^n \to e^{-1} \approx 0.368\]

\[\widehat f_{\text{bag}}(x) = \frac{1}{B}\sum_{b=1}^B \widehat f^{*b}(x),\]
# Bagging from scratch: B full trees, each fit to a bootstrap sample.
B <- 300
grow <- rpart.control(cp = 0, minsplit = 2, minbucket = 1, xval = 0)
boots <- map(seq_len(B), \(b) sample(n_train, replace = TRUE))
trees <- map(boots, \(boot) {
rpart(compressive_strength ~ ., data = train[boot, ], control = grow)
})
tree_pred <- sapply(trees, predict, newdata = test)
c(test_mse = mean((y_test - rowMeans(tree_pred))^2))test_mse
26.76253

\[\text{MSE}_{\text{OOB}} = \frac{1}{n}\sum_{i=1}^n \left(y_i - \widehat f_{\text{OOB}}(x_i)\right)^2,\]
\[\widehat f_{\text{OOB}}(x_i) = \frac{1}{\left|\mathcal{O}_i\right|}\sum_{b \in \mathcal{O}_i} \widehat f^{*b}(x_i),\]
oob_mse
27.03195


randomForest(): \(m = \lfloor p/3 \rfloor\) for a numeric response| Method | \(m\) | \(\widehat\sigma^2\) | \(\widehat\rho\) | \(\widehat\rho\widehat\sigma^2\) |
|---|---|---|---|---|
| Bagging | 8 | 62 | 0.09 | 5.8 |
| Random forest | 2 | 105 | 0.04 | 4.4 |
Call:
randomForest(formula = compressive_strength ~ ., data = train, importance = TRUE)
Type of random forest: regression
Number of trees: 500
No. of variables tried at each split: 2
Mean of squared residuals: 32.1988
% Var explained: 88.07
test_mse
34.24721


age blast_furnace_slag cement water
5 2 4 3
IncNodePurity)
%IncMSE)
%IncMSE IncNodePurity
cement 128.80824 38055.659
blast_furnace_slag 40.40708 10765.080
fly_ash 30.74548 9337.777
water 95.72834 22962.469
superplasticizer 68.02636 15163.163
coarse_aggregate 32.06984 13284.344
fine_aggregate 44.18691 14802.340
age 164.19623 54983.954

| Method | Test MSE |
|---|---|
| Least squares, main effects | 106.4 |
| Least squares, main effects, log(age) | 48.5 |
| Default tree | 86.8 |
| Pruned tree | 42.2 |
| Full tree | 40.3 |
| Bagging, rpart() loop | 26.8 |
| Bagging, randomForest() | 27.1 |
| Random forest, m = 2 | 34.2 |
| Random forest, m = 5 | 26.7 |

DS 3030