Classifier Evaluation

Today

• Credit card classifiers
• Confusion matrix
• Sensitivity and specificity
• Threshold choice
• ROC curve
• AUC
• Student strata

Distilled from the chapter’s three learning objectives. Multiple classes is not on the agenda because it is optional.

Credit Card Classifiers

Credit card default data

ISLR2::Default · Who defaults? Which mistakes? Separate models for students?

Default	Customers	Proportion	Mean balance ($)	SD of balance ($)	Zero balances
No	9,667	0.9667	804	456	499
Yes	333	0.0333	1,748	341	0

The question the chapter asks, in three parts: how well LDA and logistic regression identify the customers who will default, which kinds of mistake they make, and whether separate models for students and non-students do better. The data are the ones from the LDA lecture, so this is the same table. Only 333 of 10,000 customers defaulted, which matters for every error rate that follows.

Income and balance

Defaulters at larger balances; the classes overlap heavily

Optional. The chapter’s closing sentence of the data subsection: income runs across nearly the whole range for both classes. The next slide draws the same cloud with the boundaries.

Fitted classifiers

LDA (solid) and logistic regression (dashed): boundaries at \(\widehat p(x) = 0.5\)

The LDA lecture fit four classifiers (LDA and logistic regression, each with balance alone and with balance and income). Each gives every customer a posterior probability, and this lecture evaluates those probabilities as scores. The two boundaries are nearly vertical and close together, logistic regression’s slightly to the left; many defaulters lie to the left of both. The figure shows only the two-feature classifiers’ boundaries.

Classifier Evaluation

LDA scores within each class

Dotted line: \(\widehat p(x) = 0.5\) · any threshold in the overlap makes two kinds of mistake

The opening figure of the section. The defaulters’ scores sit well to the right of the non-defaulters’ but the histograms overlap, so defaulters to the left of a threshold are missed and non-defaulters to its right are false alarms. The chapter quotes the share of each class on the wrong side of 0: 76.9% of defaulters score at or below 0, and 0.2% of non-defaulters above it.

Confusion matrix

Truth	Predicted positive	Predicted negative
Positive	True positive (TP)	False negative (FN)
Negative	False positive (FP)	True negative (TN)

Positive: the class of interest · predict positive when \(\widehat p(x) > t\)

Error rate: \((FP + FN)/n\)

With two classes call the class of interest positive; a threshold t turns probabilities into classes. TP and FN are the positives on the right and left of the dotted line on the previous slide, FP and TN the negatives. The diagonal is the correct classifications.

Credit card confusion matrix

Truth	Predicted Yes	Predicted No	Total
Yes	77	256	333
No	20	9,647	9,667

LDA, balance and income · \(t = 0.5\) · training counts

Error rate 2.76%, against 3.33% for predicting No for everyone

256 of the 276 errors are missed defaulters

The error rate is small because the 9,667 non-defaulters dominate it, even though most errors are missed defaulters. These are training error rates, optimistic for new customers, as the classification lecture discussed; estimating test error from the training data alone is the resampling unit.

Sensitivity and specificity

\[\text{Sensitivity} = \frac{TP}{TP + FN}, \qquad \text{Specificity} = \frac{TN}{TN + FP}\]

Sensitivity: positives caught, the true positive rate

Specificity: negatives left alone, the true negative rate

Credit card: sensitivity 23.1%, specificity 99.8%

Dividing each row of the confusion matrix by its total gives two rates that condition on the true class; their complements are the false negative rate and the false positive rate (1 minus specificity). For the credit card customers the classifier flags 77 of the 333 who defaulted and correctly leaves alone 9,647 of the 9,667 who did not.

Classification Threshold

Classification threshold

rateNames = ["Overall error rate", "Sensitivity (true positive rate)",
             "Specificity (true negative rate)",
             "False positive rate (1 - specificity)"]
rateStyles = ({
  "Overall error rate": {stroke: "black", dash: "none"},
  "Sensitivity (true positive rate)": {stroke: "#D55E00", dash: "8,4"},
  "Specificity (true negative rate)": {stroke: "#0072B2", dash: "2,4"},
  "False positive rate (1 - specificity)": {stroke: "#009E73", dash: "12,4,2,4"}
})
viewof shownRates = Inputs.checkbox(rateNames, {
  value: rateNames.slice(0, 3),
  format: name => html`<span style="display: inline-flex; align-items: center; gap: 0.4em">
    <svg width="36" height="10"><line x1="2" y1="5" x2="34" y2="5"
      stroke="${rateStyles[name].stroke}" stroke-width="2.5"
      stroke-dasharray="${rateStyles[name].dash}" stroke-linecap="round"/></svg>${name}</span>`
})

Lowering \(t\): sensitivity rises or stays, specificity falls or stays · no threshold improves both

{
  const rates = transpose(threshold_rates);
  return Plot.plot({
    width: 740, height: 500, style: {fontSize: "18px"},
    marginTop: 24,
    x: {domain: [0, 0.5], label: "Threshold t"},
    y: {domain: [0, 1], label: "Rate", grid: true},
    marks: [
      Plot.ruleX(threshold_marks, {stroke: "gray", strokeDasharray: "2,3"}),
      Plot.text(threshold_marks, {x: d => d, y: 1, text: d => `t = ${d}`,
                                  dy: -12, fill: "gray"}),
      ...rateNames.filter(r => shownRates.includes(r)).map(r =>
        Plot.line(rates.filter(d => d.rate === r),
                  {x: "t", y: "value", stroke: rateStyles[r].stroke,
                   strokeDasharray: rateStyles[r].dash, strokeWidth: 2.5,
                   strokeLinecap: "round"}))
    ]
  });
}

Work the checkboxes live: the four rates against t for the two-feature LDA classifier, with dotted lines at t = 0.2 and t = 0.5. The false positive rate starts unchecked because it mirrors specificity. The error rate follows the false positive rate, nearly flat over most of the range and climbing sharply as t approaches 0. Lowering t enlarges the set of features classified positive, so every observation predicted positive before still is: TP and FP can only grow, FN and TN only shrink. The figure is interactive and needs an internet connection (the Observable runtime loads from the web), as does the notes page. While a checkbox has keyboard focus the arrow keys do not change slides; click an empty part of the slide to return to slide navigation.

Lowering the threshold

Model	Threshold	TP	FN	FP	TN	Error rate	Sensitivity	Specificity
LDA, balance	0.5	76	257	24	9,643	0.0281	0.2282	0.9975
LDA, balance	0.2	195	138	236	9,431	0.0374	0.5856	0.9756
LDA, balance + income	0.5	77	256	20	9,647	0.0276	0.2312	0.9979
LDA, balance + income	0.2	189	144	240	9,427	0.0384	0.5676	0.9752
Logistic regression, balance + income	0.5	108	225	38	9,629	0.0263	0.3243	0.9961
Logistic regression, balance + income	0.2	200	133	271	9,396	0.0404	0.6006	0.9720

Lower \(t\): more defaulters caught, more false alarms

Lowering the threshold from 0.5 to 0.2 raises the two-feature LDA classifier’s sensitivity from 23.1% to 56.8%, at the cost of specificity falling from 99.8% to 97.5% and the error rate rising from 2.76% to 3.84%. Adding income changes LDA very little. At both thresholds logistic regression flags more customers than the two-feature LDA classifier (146 against 97 at t = 0.5) and has the higher sensitivity and the lower specificity. The overall error rate need not be smallest at t = 0.5: LDA’s training error rate is lowest near t = 0.427, at 2.61%.

ROC curve

ROC curve: sensitivity against \(1 - \text{specificity}\) at every threshold

Only the ordering of the observations matters

\[\widehat{p}(x) > t\]

Predict positive: \(t\) is therefore a threshold for a probability

\[c = \log\left[\frac{t}{1 - t}\right]\]

At t = 1 nothing is predicted positive, the point (0, 0); at t = 0 everything is, the point (1, 1). Any strictly increasing transformation of the score (the log posterior odds or the estimated probability itself) sorts the observations the same way, produces the same confusion matrices and traces the same curve. The rule says predict positive when the estimated probability exceeds t, so t is a threshold for a probability; predicting positive when the estimated probability exceeds t is predicting positive when the log posterior odds exceed c.

ROC curve, threshold slider

viewof t = Inputs.range([0.01, 0.99], {value: 0.5, step: 0.01, label: "Threshold t", width: 640})

{
  const bins = transpose(slider_bins);
  const outline = transpose(slider_outline);
  const thresholds = transpose(slider_thresholds);
  const roc = transpose(slider_roc);
  const now = thresholds[Math.round(t * 100) - 1];
  const classes = ["No (negative)", "Yes (positive)"];
  const flagged = k => bins
    .filter(d => d.class === k && d.x1 > now.cutoff)
    .map(d => ({...d, x0: Math.max(d.x0, now.cutoff)}));

  const densityPlot = Plot.plot({
    width: 700, height: 400, style: {fontSize: "16px"},
    x: {label: "LDA log posterior odds of default"},
    y: {label: "Density within class"},
    color: {domain: classes, range: slider_colors, legend: true},
    marks: [
      Plot.rect(flagged(classes[0]),
                {x1: "x0", x2: "x1", y1: 0, y2: "density", fill: "class",
                 fillOpacity: 0.2}),
      Plot.rect(flagged(classes[1]),
                {x1: "x0", x2: "x1", y1: 0, y2: "density", fill: "class",
                 fillOpacity: 0.45}),
      Plot.line(outline.filter(d => d.class === classes[0]),
                {x: "x", y: "density", stroke: "class", strokeWidth: 2}),
      Plot.line(outline.filter(d => d.class === classes[1]),
                {x: "x", y: "density", stroke: "class", strokeDasharray: "6,4",
                 strokeWidth: 2}),
      Plot.ruleX([now.cutoff], {strokeDasharray: "2,3"})
    ]
  });

  const rocPlot = Plot.plot({
    width: 420, height: 400, style: {fontSize: "16px"},
    x: {domain: [0, 1], label: "False positive rate (1 − specificity)"},
    y: {domain: [0, 1], label: "True positive rate (sensitivity)"},
    marks: [
      Plot.line([[0, 0], [1, 1]], {stroke: "gray", strokeDasharray: "4,4"}),
      Plot.line(roc, {x: "fpr", y: "tpr", strokeWidth: 2}),
      Plot.dot([now], {x: "fpr", y: "tpr", r: 6, fill: "white", stroke: "black",
                       strokeWidth: 2})
    ]
  });

  return html`<p style="text-align: center"><b>t = ${t.toFixed(2)}</b>,
    cutoff c = ${now.cutoff.toFixed(2)}; sensitivity = ${now.tpr.toFixed(3)},
    specificity = ${(1 - now.fpr).toFixed(3)}</p>
    <div style="display: flex; flex-wrap: wrap; gap: 1rem; justify-content: center">
      ${densityPlot}${rocPlot}
    </div>`;
}

Lower \(t\): the cutoff slides left, the point climbs toward \((1, 1)\)

Drag the slider from t = 0.9 to t = 0.1. The cutoff slides left, the shaded share of the defaulters’ histogram (sensitivity) and of the non-defaulters’ (false positive rate) both grow, and the point climbs toward (1, 1). The dashed diagonal is a classifier that ignores the features and predicts positive at random; the LDA curve bows far toward the top-left corner. The figure is interactive and needs an internet connection (the Observable runtime loads from the web). Click the slider, then use the mouse or the left and right arrow keys to move it: while the slider has focus, reveal.js ignores the arrow keys, so they move the slider and not the slides. Click an empty part of the slide to return to slide navigation.

ROC curves, balance + income

Almost on top of one another, far above the diagonal

Lowering t from 0.5 to 0.2 moves each model’s marked point up and slightly right along its curve; logistic regression’s point sits further along its nearly shared curve than LDA’s. The crossover discussion (the balances where the two posterior curves cross at the income quartiles, and the order reversing at a low threshold) is in the notes.

Area under the ROC curve

AUC = 0.94907 for balance and income

The left panel of the chapter’s two-panel figure. The area is the AUC; the next slide computes it and the slide after gives the AUCs of the fitted classifiers.

Computing the AUC

\[\text{AUC} = \sum_{m=1}^{M}\left(\text{FPR}_m - \text{FPR}_{m-1}\right)\frac{\text{TPR}_m + \text{TPR}_{m-1}}{2}\]

• \(\left(\text{FPR}_m, \text{TPR}_m\right)\) — point \(m\) on the ROC curve, by increasing \(\text{FPR}\)
• \(m = 0\) is \((0, 0)\); \(m = M\) is \((1, 1)\)

Trapezoid rule: join consecutive points by lines and add up the areas beneath

The chapter’s one computation of the AUC. Each term is the width of a strip times the average of its two heights, the area of one trapezoid under the line joining consecutive ROC points. The chapter’s code builds the points from the sorted scores and applies this sum.

AUC of the fitted classifiers

Model	AUC
LDA, balance	0.94798
Logistic regression, balance	0.94798
LDA, balance + income	0.94907
Logistic regression, balance + income	0.94905

Generally, larger is better, at most 1

With balance alone LDA and logistic regression have identical AUCs, since both posterior probabilities are increasing functions of balance, so they order the customers identically and trace the same ROC curve. With balance and income the orderings differ slightly, and the AUCs 0.94907 and 0.94905 differ by 2.1 times 10 to the minus 5. An AUC of 0.949 is close to 1: the curve bows far toward the top-left corner.

ROC curves, same customers

AUC ranks classifiers without a threshold

The right panel of the chapter’s two-panel figure. Income alone barely separates the classes (AUC 0.533), its curve staying near the diagonal. The balance-only and two-feature curves nearly coincide and cross: the two-feature curve is the higher one over 27% of the range of false positive rates and the balance-only curve over 11%. The AUC averages over all of them, so the classifier with the larger AUC can still be the worse one over the false positive rates that matter for a given application.

Student Strata

Default rate by student status

Student	Customers	Defaulters	Default rate	Median income
No	7,056	206	2.92%	$39,893
Yes	2,944	127	4.31%	$17,994

Students default more often and have much lower median income

Optional. 4.31% against 2.92%. The median income gap was noted in the LDA lecture; this additional analysis asks whether separate models for students and non-students classify better.

Income and balance by student

In both strata the defaulters sit at larger balances

Optional. The students’ incomes occupy a lower, narrower band, from $772 to $33,003, against $8,018 to $73,554 for non-students.

Boundaries within each stratum

LDA fit within each stratum (solid), interacted logistic regression (dashed)

LDA with balance and income fit separately to non-students and students, so priors, class means and pooled covariance all differ by stratum. The logistic regression interacts the intercept and both slopes with student status, which gives the same fitted probabilities as two separate logistic regressions. Each customer is scored by the model for its own stratum, and because each stratum’s priors are its own default rates, the posteriors from the two strata are on one scale.

Stratified and single fits

Model	Error rate at \(t = 0.5\)	AUC
LDA, single fit	0.0276	0.94907
LDA, by student stratum	0.0273	0.94955
Logistic regression, additive	0.0263	0.94905
Logistic regression, interacted with student	0.0268	0.94958

Barely changes the classifications

Training rates: stratified models estimate twice as many parameters

The extra flexibility barely changes the classifications: stratifying LDA moves the training error rate from 2.76% to 2.73% and its AUC from 0.9491 to 0.9495, and interacting the logistic regression moves its error rate from 2.63% to 2.68%. The stratified models estimate 16 parameters against 8 for LDA and 6 against 3 for logistic regression, so their training error rates tend to understate test error, and their training AUCs overstate test AUC, by more. Whether stratifying helps on new customers is a question for test data.

Multiple classes

Confusion matrix

Truth	Predicted 1	Predicted 2	\(\cdots\)	Predicted \(K\)
1	\(n_{11}\)	\(n_{12}\)	\(\cdots\)	\(n_{1K}\)
2	\(n_{21}\)	\(n_{22}\)	\(\cdots\)	\(n_{2K}\)
\(\vdots\)	\(\vdots\)	\(\vdots\)	\(\ddots\)	\(\vdots\)
\(K\)	\(n_{K1}\)	\(n_{K2}\)	\(\cdots\)	\(n_{KK}\)

\(n_{kj}\) — true class \(k\), predicted class \(j\)

Error rate: \(1 - \sum_{k=1}^K n_{kk} / n\)

Optional. With K classes the confusion matrix is K by K, true classes in the rows and predicted classes in the columns; the diagonal entries are the correct classifications.

Multiple classes, one versus rest

\[\text{Sensitivity}_k = \frac{n_{kk}}{\sum_{j=1}^K n_{kj}}, \qquad \text{Specificity}_k = \frac{\sum_{l \ne k} \sum_{j \ne k} n_{lj}}{\sum_{l \ne k} \sum_{j=1}^K n_{lj}}\]

One versus the rest: class \(k\) positive, the other \(K - 1\) classes together negative

Optional. Collapsing to two-by-two gives class k’s rates. The threshold t = 0.5 generalizes to the class with the largest posterior probability, and the two rules coincide when K = 2. Other thresholds, the ROC curve and the AUC are two-class tools.

Conclusion

• Confusion matrix
• Sensitivity and specificity
• Threshold
• ROC curve
• AUC
• Stratified models

Optional. The lecture may end on the Stratified and single fits slide (or the last Multiple classes slide), with the next lecture’s preview said aloud: quadratic discriminant analysis, naive Bayes and KNN, compared on a held-out test set.