4 Simple Linear Regression
After this lecture you should be able to
- write the simple linear regression model and interpret \(\beta_0\), \(\beta_1\), and \(\sigma^2\), including after a log transformation of the response, the explanatory variable, or both,
- derive the least squares estimators as the minimizers of the residual sum of squares, and explain why \(\hat\sigma^2 = RSS/(n-2)\) uses \(n-2\) degrees of freedom,
- write the likelihood under normal errors and show that the maximum likelihood estimators of \(\beta_0\) and \(\beta_1\) coincide with the least squares estimators,
- construct and interpret t-tests and confidence intervals for the regression coefficients,
- distinguish a confidence interval for the mean response at \(x_0\) from a prediction interval for a new observation at \(x_0\), and explain from their standard errors why the latter is wider, and
- evaluate model fit using \(R^2\) and interpret it as a proportion of variability explained.
Recall our non-linear regression model: \[Y = f(X) + \epsilon\]
where \(f\) is an unknown function and \(\epsilon\) is a random error term with mean 0 and variance \(\sigma^2\). When \(X\) is a scalar, one model that could be used to approximate \(f\) is a simple linear regression model.
4.1 Model
The simple linear regression model is
\[Y_i = \beta_0 + \beta_1 X_i + \epsilon_i\]
where \(\epsilon_i\) is a random error term with \(E[\epsilon_i] = 0\) and \(Var[\epsilon_i] = \sigma^2\).
4.2 Interpretation
The interpretation of the parameters is as follows:
- \(\beta_0\) is the expected response when the explanatory variable is zero, \(E[Y_i|X_i = 0]\).
- \(\beta_1\) is the expected change in \(Y\) for a one-unit increase in \(X\), \(E[Y_i|X_i = x+1] - E[Y_i|X_i = x]\).
A common transformation is to take the logarithm of the response variable, the explanatory variable, or both. The interpretations of the parameters change accordingly:
- If we take the logarithm of the response variable, then \(e^{\beta_1}\) represents the multiplicative change in the median response for a one-unit increase in the explanatory variable.
- If we take the logarithm of the explanatory variable, then \(\beta_1\log(2)\) represents the expected change in the response for a doubling of the explanatory variable.
- If we take the logarithm of both the response and explanatory variables, then \(2^{\beta_1}\) represents the multiplicative change in the median response for a doubling of the explanatory variable.
4.3 Estimation
The parameters \(\beta_0\) and \(\beta_1\) are estimated using the method of least squares, which minimizes the sum of squared residuals:
\[\hat{\beta}_0, \hat{\beta}_1 = \arg\min_{\beta_0, \beta_1} \sum_{i=1}^n (Y_i - \beta_0 - \beta_1 X_i)^2.\]
These are commonly referred to as the ordinary least squares (OLS) estimators.
The residual sum of squares (RSS) are
\[RSS = \sum_{i=1}^n \left(Y_i - \hat\beta_0 - \hat\beta_1 X_i\right)^2\]
We can use this to estimate the error variance \(\sigma^2\) using
\[\hat{\sigma}^2 = \frac{RSS}{n-2}\]
where \(n-2\) is the degrees of freedom, since we have estimated two parameters (\(\beta_0\) and \(\beta_1\)).
4.4 Inference
For inference, we need to assume normality of the errors, i.e. \[\epsilon_i \stackrel{ind}{\sim} N(0, \sigma^2).\] This results in the model \[Y_i = \beta_0 + \beta_1 X_i + \epsilon_i\] or, all together, \[Y_i \stackrel{ind}{\sim} N(\beta_0 + \beta_1 X_i, \sigma^2).\]
With this assumption, we can write the likelihood function:
\[\begin{array}{rl} L(\beta_0, \beta_1, \sigma^2) &= \prod_{i=1}^n \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{1}{2\sigma^2}[Y_i - \beta_0 - \beta_1 X_i]^2\right) \\ &= (2\pi\sigma^2)^{-n/2} \exp\left(-\frac{1}{2\sigma^2} \sum_{i=1}^n [Y_i - \beta_0 - \beta_1 X_i]^2\right) \end{array}\]
the estimates for \(\beta_0\) and \(\beta_1\) that maximize this likelihood are the same as those obtained from ordinary least squares. These are referred to as the maximum likelihood estimators (MLEs) of \(\beta_0\) and \(\beta_1\).
4.4.1 t-tests
For testing hypotheses about the regression coefficients, we use t-tests. The test statistic for \(\beta_j\) is
\[t_{n-2} = \frac{\hat{\beta}_j - \beta_j^0}{\text{SE}(\hat{\beta}_j)}\]
which, under the null hypothesis \(H_0: \beta_j = \beta_j^0\), follows a t-distribution with \(n-2\) degrees of freedom.
The standard error of \(\hat{\beta}_0\) is given by \[\text{SE}\left(\hat{\beta}_0\right) = \hat{\sigma}\sqrt{ \left( \frac{1}{n} + \frac{\bar{x}^2}{\sum_{i=1}^n (x_i - \bar{x})^2} \right)}.\]
The standard error of \(\hat{\beta}_1\) is given by \[\text{SE}\left(\hat{\beta}_1\right) = \hat{\sigma}\sqrt{\frac{1}{\sum_{i=1}^n (x_i - \bar{x})^2}}.\]
4.4.2 Confidence intervals
A \((1-\alpha)100\%\) confidence interval for \(\beta_j\) is given by
\[\hat{\beta}_j \pm t_{n-2, 1-\alpha/2} \cdot \text{SE}\left(\hat{\beta}_j\right)\]
where \(t_{n-2, 1-\alpha/2}\) is the \((1-\alpha/2)\) quantile of the t-distribution with \(n-2\) degrees of freedom.
4.4.3 Confidence intervals at \(x\)
A \((1-\alpha)100\%\) confidence interval for the mean response at a specific value \(x_0\) is given by
\[\hat{f}(x_0) \pm t_{n-2, 1-\alpha/2} \cdot \text{SE}\left(\hat{f}(x_0)\right)\]
where \(\hat{f}(x_0) = \hat{\beta}_0 + \hat{\beta}_1 x_0\) and \(\text{SE}\left(\hat{f}(x_0)\right)\) is the standard error of the predicted mean response. This standard error is calculated as:
\[\text{SE}(\hat{f}(x_0)) = \hat\sigma\sqrt{ \left( \frac{1}{n} + \frac{(x_0 - \bar{x})^2}{\sum_{i=1}^n (x_i - \bar{x})^2} \right)}\]
This is the uncertainty in the line at a given point \(x_0\).
4.5 Prediction
A \((1-\alpha)100\%\) prediction interval for a new observation at a specific value \(x_0\) is given by \[\hat{Y}_0 \pm t_{n-2, 1-\alpha/2} \cdot \text{SE}_{\text{pred}}(\hat{Y}_0)\]
where \(\hat{Y}_0 = \hat{f}(x_0) = \hat{\beta}_0 + \hat{\beta}_1 x_0\) and \(\text{SE}_{\text{pred}}(\hat{Y}_0)\) is the standard error of the predicted value, which accounts for both the uncertainty in the mean response and the variability of individual observations. This standard error is calculated as: \[\text{SE}_{\text{pred}}\left(\hat{Y}_0\right) = \hat\sigma\sqrt{ \left( 1 + \frac{1}{n} + \frac{(x_0 - \bar{x})^2}{\sum_{i=1}^n (x_i - \bar{x})^2} \right)}\]
The key here is that the prediction interval is wider than the confidence interval for the mean response, reflecting the additional uncertainty in predicting a single new observation due to the residual error \(\epsilon\).
4.6 Model Assessment
4.6.1 R-squared
The coefficient of determination, \(R^2\), measures the proportion of the variance in the response variable that is predictable from the explanatory variable(s). It is defined as \[R^2 = 1 - \frac{RSS}{TSS}\]
where \(RSS\) is the residual sum of squares and \(TSS\) is the total sum of squares, i.e.
\[TSS = \sum_{i=1}^n (y_i - \bar{y})^2.\]
Thus, \(R^2\) can be interpreted as the proportion of the total variability in the response variable that is explained by the model.