Sex (M: \(y = 1\), F: \(y = -1\))Hwt: heart weight (g)Bwt: body weight (kg)F, 65 M)
\[ y_i\left(\beta_0 + \beta^\top x_i\right) \ge M \]
Support vector classifier, constraints for all \(i = 1, \ldots, n\):
\[ \begin{aligned} \underset{\beta_0, \beta_1, \ldots, \beta_p, \epsilon_1, \ldots, \epsilon_n, M}{\text{maximize}} \quad & M \\ \text{subject to} \quad & \sum_{j=1}^p \beta_j^2 = 1, \\ & y_i\left(\beta_0 + \beta_1 x_{i1} + \cdots + \beta_p x_{ip}\right) \ge M(1 - \epsilon_i), \\ & \epsilon_i \ge 0, \quad \sum_{i=1}^n \epsilon_i \le C, \end{aligned} \]
With \(f(x_i) = \beta_0 + \beta^\top x_i\):
Call:
svm(formula = Sex ~ Hwt + Bwt, data = train, kernel = "linear", cost = 1,
tolerance = 1e-08)
Parameters:
SVM-Type: C-classification
SVM-Kernel: linear
cost: 1
Number of Support Vectors: 58
e1071::svm() solves
\[ \underset{w_0, w, \epsilon_1, \ldots, \epsilon_n}{\text{minimize}} \quad \tfrac{1}{2}\|w\|^2 + \texttt{cost}\sum_{i=1}^n \epsilon_i \quad \text{subject to} \quad y_i\left(w_0 + w^\top x_i\right) \ge 1 - \epsilon_i, \quad \epsilon_i \ge 0, \]
cost \(> 0\) — price of one unit of slackcost: slack is expensive, a small budgetcost: slack is cheap, a large budgetcost is a different budgetsvm() does by default (scale = TRUE):\[ \tilde x_{ij} = \frac{x_{ij} - \bar x_j}{s_j}, \]
beta0 Hwt Bwt M
0.5519 -0.1502 0.9887 0.6426

\[ \widehat\epsilon_i = \max\left(0,\ 1 - \frac{y_i \widehat f(x_i)}{\widehat M}\right), \]
cost: budget \(C\) no larger, margin \(\widehat M\) no widercost \(\ne C\): the \(C\) a cost gives depends on the datacost \(\to \infty\) on separable data: the maximal margin classifiercost |
\(\sum_i\widehat\epsilon_i\) | \(\widehat M\) | Support vectors | Inside the margin | Wrong side | Training error rate |
|---|---|---|---|---|---|---|
| 0.01 | 61.68 | 17.68 | 62 | 31 | 31 | 0.323 |
| 0.10 | 58.80 | 1.77 | 62 | 34 | 28 | 0.292 |
| 1.00 | 54.55 | 0.64 | 58 | 33 | 22 | 0.229 |
| 10.00 | 54.41 | 0.57 | 56 | 32 | 21 | 0.219 |


cost error dispersion
1 0.01000000 0.3255556 0.1478747
2 0.01778279 0.3255556 0.1478747
3 0.03162278 0.3255556 0.1478747
cost
10 1.778279


Predicted
Truth F M
F 11 5
M 4 28
MM: 33.3%cost)# Fits to the training cats; each gives a probability that a cat is M.
set.seed(20261114)
logit_fit <- glm(Sex ~ Hwt + Bwt, data = train, family = binomial)
lda_fit <- MASS::lda(Sex ~ Hwt + Bwt, data = train)
qda_fit <- MASS::qda(Sex ~ Hwt + Bwt, data = train)
rf_fit <- randomForest::randomForest(Sex ~ Hwt + Bwt, data = train)
# Probabilities of F and M that LDA gives the first three test cats.
head(predict(lda_fit, test)$posterior, 3) F M
1 0.7219232 0.2780768
6 0.6667787 0.3332213
10 0.6774822 0.3225178
\[ \text{log loss} = -\frac{1}{n}\sum_{i=1}^n \log \widehat p_i, \]
| Classifier | CV error rate (%) | CV log loss | Test error rate (%) | Test log loss |
|---|---|---|---|---|
| Logistic, main effects | 22.9 | 0.510 | 16.7 | 0.418 |
| Logistic, interaction | 26.0 | 0.513 | 16.7 | 0.417 |
| Logistic, quadratic | 25.0 | 0.550 | 16.7 | 0.402 |
| LDA | 24.0 | 0.509 | 16.7 | 0.429 |
| QDA | 27.1 | 0.543 | 14.6 | 0.377 |
| Random forest | 22.9 | 0.567 | 20.8 | 0.406 |
| Support vector classifier | 24.0 | — | 18.8 | — |
cost: budget no larger, margin no wider; features standardizedcost by cross-validation; judged by the test error rateDS 3030