Sale_Price, in dollars; the models use its logarithm
| Method | Tuning | Nonzero estimates | 10-fold CV estimate | Test MSE |
|---|---|---|---|---|
| Least squares | None | 122 | 0.0505 | 0.0462 |
| Forward stepwise | \(d = 18\) | 18 | 0.0231 | 0.0269 |
| Ridge | \(\lambda = 100.0\) | 122 | 0.0187 | 0.0241 |
| Lasso | \(\lambda = 2.51\) | 57 | 0.0177 | 0.0228 |
| PCR | \(M = 62\) | 122 | 0.0184 | 0.0262 |
| PLS | \(M = 3\) | 122 | 0.0184 | 0.0246 |
North_Ames for Neighborhood
| Method | Mean test MSE | Median test MSE | Samples won | Median change from least squares |
|---|---|---|---|---|
| Least squares | 0.0614 | 0.0504 | 0 | |
| Forward stepwise | 0.0303 | 0.0295 | 0 | −42% |
| Ridge | 0.0242 | 0.0240 | 39 | −53% |
| Lasso | 0.0242 | 0.0239 | 52 | −53% |
| PCR | 0.0262 | 0.0258 | 1 | −49% |
| PLS | 0.0237 | 0.0233 | 108 | −54% |

| Property | Least squares | Forward stepwise | Ridge | Lasso | PCR | PLS |
|---|---|---|---|---|---|---|
| Estimates | Unconstrained | Kept: least squares; dropped: 0 | All shrunk toward 0, none 0 | Shrunk; some exactly 0 | In the span of the first \(M\) components | In the span of \(M\) directions |
| Selects features | No | Yes | No | Yes | No | No |
| Tuning parameter | None | Size \(d\) | \(\lambda\) | \(\lambda\) | \(M\) | \(M\) |
| Standardization | Not needed | Not needed | Needed | Needed | Needed | Needed |
| Needs from \(n\) and \(p\) | \(n > p\) | \(d < n\) | Any \(p\) | Any \(p\) | \(M\) at most the rank of \(\tilde{\mathbf{X}}\) | \(M\) at most the rank of \(\tilde{\mathbf{X}}\) |
| Computation | One fit | \(1 + p(p + 1)/2\) fits | One SVD for every \(\lambda\) | Coordinate descent for each \(\lambda\) | One SVD for every \(M\) | One step per direction |
| Interpretation | Every estimate | A least squares model on \(d\) features | Every estimate, shrunken | Some shrunken estimates | Directions; needs every feature | Directions; needs every feature |
DS 3030