place, the city (Sacramento: \(y = 1\), Other: \(y = -1\))longitude (degrees)latitude (degrees)Other, 294 Sacramento)
Support vector classifier on the degree-2 features:
\[ \widehat w_0 + \widehat w_1X_1 + \widehat w_2X_2 + \widehat w_3X_1^2 + \widehat w_4X_1X_2 + \widehat w_5X_2^2 = 0 \]
[1] 0.06602254
Monomials of degree at most \(d\) in \(p\) explanatory variables, without the constant:
\[ q = \binom{p + d}{d} - 1 \]
\[ \widehat f(x) = \widehat w_0 + \sum_{i \in \mathcal{S}} \widehat\alpha_i \langle x, x_i\rangle, \]
Support vector machine: the support vector classifier with a kernel
\[ f(x) = w_0 + \sum_{i \in \mathcal{S}} \alpha_i K(x, x_i). \]
Linear (the inner product):
\[ K(x_i, x_{i'}) = \sum_{j=1}^p x_{ij}x_{i'j} \]
Polynomial of degree \(d\), a positive integer:
\[ K(x_i, x_{i'}) = \left(1 + \sum_{j=1}^p x_{ij}x_{i'j}\right)^d. \]
Radial, with \(\gamma > 0\):
\[ K(x_i, x_{i'}) = \exp\left(-\gamma\sum_{j=1}^p (x_{ij} - x_{i'j})^2\right). \]
\(p = 2\) and \(d = 2\), expanding the square:
\[ \begin{aligned} \left(1 + x_{i1}x_{i'1} + x_{i2}x_{i'2}\right)^2 &= 1 + 2x_{i1}x_{i'1} + 2x_{i2}x_{i'2} + x_{i1}^2x_{i'1}^2 + 2x_{i1}x_{i2}x_{i'1}x_{i'2} + x_{i2}^2x_{i'2}^2 \\ &= \langle h(x_i), h(x_{i'})\rangle, \end{aligned} \]
where
\[ h(x_i) = \left(1,\ \sqrt2 x_{i1},\ \sqrt2 x_{i2},\ x_{i1}^2,\ \sqrt2 x_{i1}x_{i2},\ x_{i2}^2\right)^\top \]
\[ \widehat f\left(x^*\right) = \widehat w_0 + \sum_{i \in \mathcal{S}}\widehat\alpha_i K\left(x^*, x_i\right) \]
Call:
svm(formula = place ~ longitude + latitude, data = train, kernel = "radial",
cost = 1, gamma = 1)
Parameters:
SVM-Type: C-classification
SVM-Kernel: radial
cost: 1
Number of Support Vectors: 147
cost gamma
23 10 100

$radial
Predicted
Truth Other Sacramento
Other 165 2
Sacramento 5 139
$linear
Predicted
Truth Other Sacramento
Other 129 38
Sacramento 32 112
Other: 46.3%
\(K\) classes, \(K > 2\):
svm(): one versus one whenever the class has more than two levelsElk Grove, Other, Sacramento Predicted
Truth Elk Grove Other Sacramento
Elk Grove 34 0 0
Other 0 131 2
Sacramento 0 5 139
Support vector classifier, without slack variables:
\[ \underset{w_0, w}{\text{minimize}} \quad \sum_{i=1}^n \max\left[0, 1 - y_i f(x_i)\right] + \lambda\sum_{j=1}^p w_j^2, \qquad \lambda = \frac{1}{2\,\texttt{cost}}, \]

Logistic regression, negative log-likelihood:
\[ \sum_{i=1}^n \log\left(1 + e^{-y_i f(x_i)}\right). \]
# Logistic regression on longitude and latitude, classified at 0.5.
logistic_fit <- glm(place ~ longitude + latitude, data = train,
family = binomial)
p_test <- predict(logistic_fit, test, type = "response")
logistic_class <- if_else(p_test > 0.5, "Sacramento", "Other")
mean(logistic_class != test$place)[1] 0.221865
cost and \(\gamma\) by cross-validation; judged by the test error rateDS 3030