4 Simple Linear Regression
After this lecture you should be able to
- write the simple linear regression model and interpret \(\beta_0\), \(\beta_1\), and \(\sigma^2\), including after a log transformation of the response, the explanatory variable, or both,
- derive the least squares estimators as the minimizers of the residual sum of squares, and explain why \(\hat\sigma^2 = RSS/(n-2)\) uses \(n-2\) degrees of freedom,
- write the likelihood under normal errors and show that the maximum likelihood estimators of \(\beta_0\) and \(\beta_1\) coincide with the least squares estimators,
- construct and interpret t-tests and confidence intervals for the regression coefficients,
- distinguish a confidence interval for the mean response at \(x_0\) from a prediction interval for a new observation at \(x_0\), and explain from their standard errors why the latter is wider, and
- evaluate model fit using \(R^2\) and interpret it as a proportion of variability explained.
Recall our non-linear regression model: \[Y = f(X) + \epsilon\]
where \(f\) is an unknown function and \(\epsilon\) is a random error term with mean 0 and variance \(\sigma^2\). When \(X\) is a scalar, one model that could be used to approximate \(f\) is a simple linear regression model.
4.1 Model
The simple linear regression model is
\[Y_i = \beta_0 + \beta_1 X_i + \epsilon_i\]
where \(\epsilon_i\) is a random error term with \(E[\epsilon_i] = 0\) and \(Var[\epsilon_i] = \sigma^2\).
4.2 Interpretation
The interpretation of the parameters is as follows:
- \(\beta_0\) is the expected response when the explanatory variable is zero, \(E[Y_i|X_i = 0]\).
- \(\beta_1\) is the expected change in \(Y\) for a one-unit increase in \(X\), \(E[Y_i|X_i = x+1] - E[Y_i|X_i = x]\).
A common transformation is to take the logarithm of the response variable, the explanatory variable, or both. The interpretations of the parameters change accordingly:
- If we take the logarithm of the response variable, then \(e^{\beta_1}\) represents the multiplicative change in the median response for a one-unit increase in the explanatory variable.
- If we take the logarithm of the explanatory variable, then \(\beta_1\log(2)\) represents the expected change in the response for a doubling of the explanatory variable.
- If we take the logarithm of both the response and explanatory variables, then \(2^{\beta_1}\) represents the multiplicative change in the median response for a doubling of the explanatory variable.
4.3 Estimation
The parameters \(\beta_0\) and \(\beta_1\) are estimated using the method of least squares, which minimizes the sum of squared residuals:
\[\hat{\beta}_0, \hat{\beta}_1 = \arg\min_{\beta_0, \beta_1} \sum_{i=1}^n (Y_i - \beta_0 - \beta_1 X_i)^2.\]
These are commonly referred to as the ordinary least squares (OLS) estimators.
The residual sum of squares (RSS) are
\[RSS = \sum_{i=1}^n \left(Y_i - \hat\beta_0 - \hat\beta_1 X_i\right)^2\]
We can use this to estimate the error variance \(\sigma^2\) using
\[\hat{\sigma}^2 = \frac{RSS}{n-2}\]
where \(n-2\) is the degrees of freedom, since we have estimated two parameters (\(\beta_0\) and \(\beta_1\)).
4.4 Inference
For inference, we need to assume normality of the errors, i.e. \[\epsilon_i \stackrel{ind}{\sim} N(0, \sigma^2).\] This results in the model \[Y_i = \beta_0 + \beta_1 X_i + \epsilon_i\] or, all together, \[Y_i \stackrel{ind}{\sim} N(\beta_0 + \beta_1 X_i, \sigma^2).\]
With this assumption, we can write the likelihood function:
\[\begin{array}{rl} L(\beta_0, \beta_1, \sigma^2) &= \prod_{i=1}^n \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{1}{2\sigma^2}[Y_i - \beta_0 - \beta_1 X_i]^2\right) \\ &= (2\pi\sigma^2)^{-n/2} \exp\left(-\frac{1}{2\sigma^2} \sum_{i=1}^n [Y_i - \beta_0 - \beta_1 X_i]^2\right) \end{array}\]
the estimates for \(\beta_0\) and \(\beta_1\) that maximize this likelihood are the same as those obtained from ordinary least squares. These are referred to as the maximum likelihood estimators (MLEs) of \(\beta_0\) and \(\beta_1\).
4.4.1 t-tests
For testing hypotheses about the regression coefficients, we use t-tests. The test statistic for \(\beta_j\) is
\[t_{n-2} = \frac{\hat{\beta}_j - \beta_j^0}{\text{SE}(\hat{\beta}_j)}\]
which, under the null hypothesis \(H_0: \beta_j = \beta_j^0\), follows a t-distribution with \(n-2\) degrees of freedom.
The standard error of \(\hat{\beta}_0\) is given by \[\text{SE}\left(\hat{\beta}_0\right) = \hat{\sigma}\sqrt{ \left( \frac{1}{n} + \frac{\bar{x}^2}{\sum_{i=1}^n (x_i - \bar{x})^2} \right)}.\]
The standard error of \(\hat{\beta}_1\) is given by \[\text{SE}\left(\hat{\beta}_1\right) = \hat{\sigma}\sqrt{\frac{1}{\sum_{i=1}^n (x_i - \bar{x})^2}}.\]
4.4.2 Confidence intervals
A \((1-\alpha)100\%\) confidence interval for \(\beta_j\) is given by
\[\hat{\beta}_j \pm t_{n-2, 1-\alpha/2} \cdot \text{SE}\left(\hat{\beta}_j\right)\]
where \(t_{n-2, 1-\alpha/2}\) is the \((1-\alpha/2)\) quantile of the t-distribution with \(n-2\) degrees of freedom.
4.4.3 Confidence intervals at \(x\)
A \((1-\alpha)100\%\) confidence interval for the mean response at a specific value \(x_0\) is given by
\[\hat{f}(x_0) \pm t_{n-2, 1-\alpha/2} \cdot \text{SE}\left(\hat{f}(x_0)\right)\]
where \(\hat{f}(x_0) = \hat{\beta}_0 + \hat{\beta}_1 x_0\) and \(\text{SE}\left(\hat{f}(x_0)\right)\) is the standard error of the predicted mean response. This standard error is calculated as:
\[\text{SE}(\hat{f}(x_0)) = \hat\sigma\sqrt{ \left( \frac{1}{n} + \frac{(x_0 - \bar{x})^2}{\sum_{i=1}^n (x_i - \bar{x})^2} \right)}\]
This is the uncertainty in the line at a given point \(x_0\).
4.5 Prediction
A \((1-\alpha)100\%\) prediction interval for a new observation at a specific value \(x_0\) is given by \[\hat{Y}_0 \pm t_{n-2, 1-\alpha/2} \cdot \text{SE}_{\text{pred}}(\hat{Y}_0)\]
where \(\hat{Y}_0 = \hat{f}(x_0) = \hat{\beta}_0 + \hat{\beta}_1 x_0\) and \(\text{SE}_{\text{pred}}(\hat{Y}_0)\) is the standard error of the predicted value, which accounts for both the uncertainty in the mean response and the variability of individual observations. This standard error is calculated as: \[\text{SE}_{\text{pred}}\left(\hat{Y}_0\right) = \hat\sigma\sqrt{ \left( 1 + \frac{1}{n} + \frac{(x_0 - \bar{x})^2}{\sum_{i=1}^n (x_i - \bar{x})^2} \right)}\]
The key here is that the prediction interval is wider than the confidence interval for the mean response, reflecting the additional uncertainty in predicting a single new observation due to the residual error \(\epsilon\).
4.6 Model Assessment
4.6.1 R-squared
The coefficient of determination, \(R^2\), measures the proportion of the variance in the response variable that is predictable from the explanatory variable(s). It is defined as \[R^2 = 1 - \frac{RSS}{TSS}\]
where \(RSS\) is the residual sum of squares and \(TSS\) is the total sum of squares, i.e.
\[TSS = \sum_{i=1}^n (y_i - \bar{y})^2.\]
Thus, \(R^2\) can be interpreted as the proportion of the total variability in the response variable that is explained by the model.
4.7 Conclusion
Simple linear regression approximates the unknown \(f\) in \(Y = f(X) + \epsilon\) by a straight line, \(f(X) = \beta_0 + \beta_1 X\). We estimate the two coefficients by least squares, minimizing the residual sum of squares, and the error variance by \(\hat\sigma^2 = RSS/(n-2)\), where the \(n-2\) pays for the two coefficients already estimated from the same data. Assuming the errors are independent and normal leaves those point estimates alone — the maximum likelihood estimators of \(\beta_0\) and \(\beta_1\) are the least squares estimators — and supplies the sampling distributions that inference runs on.
We read the coefficients in order. \(\beta_0\) is the expected response when the explanatory variable is zero, which is only a meaningful quantity when zero is a value the explanatory variable can actually take; \(\beta_1\) is the expected change in the response for a one-unit increase in the explanatory variable. Taking logarithms replaces those additive statements with multiplicative ones: a logged response makes \(e^{\beta_1}\) a multiplicative change in the median response, a logged explanatory variable makes \(\beta_1\log(2)\) the change per doubling, and logging both combines the two.
Different questions get different intervals. A t-test and a confidence interval for \(\beta_1\) ask whether, and by how much, the mean response changes with \(X\). A confidence interval for the mean response at \(x_0\) asks where the regression line is at that point. A prediction interval at \(x_0\) asks where a new observation will fall, and is wider because its standard error carries the error variance of that new observation on top of the uncertainty in the line. Across the whole fit, \(R^2 = 1 - RSS/TSS\) reports the proportion of the variability in the response that the line explains.
Next lecture puts this model to work on financial data. The capital asset pricing model rearranges into exactly a simple linear regression of an asset’s excess return on the market’s, so \(\beta_0\), \(\beta_1\), and \(R^2\) all acquire financial meanings — and the theory predicts a specific value for the intercept, which we can test.