Method Comparison

Today

  • Ames home sales
    • Data
    • Training homes
    • Candidate features
  • Training-sample fits
    • Six fits
    • Cross-validation curves
    • Test evaluation
    • Kept dummies
  • Repeated training samples
    • Method ranking
    • Kept features
  • Six methods compared
    • Properties
    • Expected settings

Ames home sales

Data

  • Question: which of the six methods predicts the sale price of a home best, and what does each give up?
  • Observation: a home
  • Response variable: Sale_Price, in dollars; the models use its logarithm
  • Explanatory variables: 33 numeric and 40 categorical
  • Training sample: 200 homes, with \(p = 122\) candidate features
  • Test homes: 2,725

Training homes

Candidate features

Training-sample fits

Six fits

  • Least squares: all \(p\) features, no tuning parameter
  • Forward stepwise selection: size \(d = 0, 1, \ldots, p\)
  • Ridge regression: minimize \(RSS + \lambda\sum_{j=1}^p\beta_j^2\)
  • The lasso: minimize \(RSS + \lambda\sum_{j=1}^p\left|\beta_j\right|\)
  • PCR: first \(M\) principal components of the standardized features
  • PLS: first \(M\) PLS directions
  • \(d\), \(\lambda\) or \(M\) chosen by 10-fold cross-validation
  • Standardization, forward stepwise path, components and directions recomputed in each fold

Cross-validation curves

Test evaluation

Method Tuning Nonzero estimates 10-fold CV estimate Test MSE
Least squares None 122 0.0505 0.0462
Forward stepwise \(d = 18\) 18 0.0231 0.0269
Ridge \(\lambda = 100.0\) 122 0.0187 0.0241
Lasso \(\lambda = 2.51\) 57 0.0177 0.0228
PCR \(M = 62\) 122 0.0184 0.0262
PLS \(M = 3\) 122 0.0184 0.0246

Kept dummies

  • Lasso keeps 57 of the 122 features, forward stepwise selection 18
  • Neighborhood dummies: 15; the lasso keeps 8, forward stepwise selection 3
  • Some but not all dummies of a variable kept: lasso 15 variables, forward stepwise selection 4
  • Ridge regression, PCR and PLS: every dummy nonzero
  • A dropped dummy merges its level into the reference, North_Ames for Neighborhood

Repeated training samples

Method ranking

Method Mean test MSE Median test MSE Samples won Median change from least squares
Least squares 0.0614 0.0504 0
Forward stepwise 0.0303 0.0295 0 −42%
Ridge 0.0242 0.0240 39 −53%
Lasso 0.0242 0.0239 52 −53%
PCR 0.0262 0.0258 1 −49%
PLS 0.0237 0.0233 108 −54%

Kept features

Six methods compared

Properties

Property Least squares Forward stepwise Ridge Lasso PCR PLS
Estimates Unconstrained Kept: least squares; dropped: 0 All shrunk toward 0, none 0 Shrunk; some exactly 0 In the span of the first \(M\) components In the span of \(M\) directions
Selects features No Yes No Yes No No
Tuning parameter None Size \(d\) \(\lambda\) \(\lambda\) \(M\) \(M\)
Standardization Not needed Not needed Needed Needed Needed Needed
Needs from \(n\) and \(p\) \(n > p\) \(d < n\) Any \(p\) Any \(p\) \(M\) at most the rank of \(\tilde{\mathbf{X}}\) \(M\) at most the rank of \(\tilde{\mathbf{X}}\)
Computation One fit \(1 + p(p + 1)/2\) fits One SVD for every \(\lambda\) Coordinate descent for each \(\lambda\) One SVD for every \(M\) One step per direction
Interpretation Every estimate A least squares model on \(d\) features Every estimate, shrunken Some shrunken estimates Directions; needs every feature Directions; needs every feature

Expected settings

  • Least squares: \(n\) large relative to \(p\)
  • Forward stepwise selection and the lasso: a few features carry the signal, the rest are noise
  • Ridge regression: many features with small effects of similar size
  • PCR: response related to the directions in which the features vary most
  • PLS: response related to directions that are not the leading principal components
  • Expectations from reasoning, not results: one dataset does not establish them

Conclusion

  • The methods differ in what they do to the estimates
  • Only forward stepwise selection and the lasso select features
  • Cross-validate \(d\), \(\lambda\) or \(M\), recomputing the fit’s data-dependent steps in each fold
  • A cross-validation estimate can miss the test MSE in either direction
  • One training sample ranks the methods for that sample only
  • One dataset cannot establish the expected settings