Linear Regression

Abstract

This chapter is about linear regression, a very simple approach for supervised learning. In particular, linear regression is a useful tool for predicting a quantitative response. Linear regression has been around for a long time and is the topic of innumerable textbooks. Though it may seem somewhat dull compared to some of the more modern statistical learning approaches described in later chapters of this book, linear regression is still a useful and widely used statistical learning method.

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This chapter is about linear regression, a very simple approach for supervised learning. In particular, linear regression is a useful tool for predicting a quantitative response. Linear regression has been around for a long time and is the topic of innumerable textbooks. Though it may seem somewhat dull compared to some of the more modern statistical learning approaches described in later chapters of this book, linear regression is still a useful and widely used statistical learning method. Moreover, it serves as a good jumping-off point for newer approaches: as we will see in later chapters, many fancy statistical learning approaches can be seen as generalizations or extensions of linear regression. Consequently, the importance of having a good understanding of linear regression before studying more complex learning methods cannot be overstated. In this chapter, we review some of the key ideas underlying the linear regression model, as well as the least squares approach that is most commonly used to fit this model.

Recall the Advertising data from Chapter 2. Figure 2.1 displays sales (in thousands of units) for a particular product as a function of advertising budgets (in thousands of dollars) for TV, radio, and newspaper media. Suppose that in our role as statistical consultants we are asked to suggest, on the basis of this data, a marketing plan for next year that will result in high product sales. What information would be useful in order to provide such a recommendation? Here are a few important questions that we might seek to address:

• 1. Is there a relationship between advertising budget and sales?

  Our first goal should be to determine whether the data provide evidence of an association between advertising expenditure and sales. If the evidence is weak, then one might argue that no money should be spent on advertising!

• 2. How strong is the relationship between advertising budget and sales?

  Assuming that there is a relationship between advertising and sales, we would like to know the strength of this relationship. In other words, given a certain advertising budget, can we predict sales with a high level of accuracy? This would be a strong relationship. Or is a prediction of sales based on advertising expenditure only slightly better than a random guess? This would be a weak relationship.

• 3. Which media contribute to sales?

  Do all three media—TV, radio, and newspaper—contribute to sales, or do just one or two of the media contribute? To answer this question, we must find a way to separate out the individual effects of each medium when we have spent money on all three media.

• 4. How accurately can we estimate the effect of each medium on sales?

  For every dollar spent on advertising in a particular medium, by what amount will sales increase? How accurately can we predict this amount of increase?

• 5. How accurately can we predict future sales?

  For any given level of television, radio, or newspaper advertising, what is our prediction for sales, and what is the accuracy of this prediction?

• 6. Is the relationship linear?

  If there is approximately a straight-line relationship between advertising expenditure in the various media and sales, then linear regression is an appropriate tool. If not, then it may still be possible to transform the predictor or the response so that linear regression can be used.

• 7. Is there synergy among the advertising media?

  Perhaps spending $50, 000 on television advertising and $50, 000 on radio advertising results in more sales than allocating $100, 000 to either television or radio individually. In marketing, this is known as a synergy effect, while in statistics it is called an interaction effect.

synergy

interaction

It turns out that linear regression can be used to answer each of these questions. We will first discuss all of these questions in a general context, and then return to them in this specific context in Section 3.4.

3.1 Simple Linear Regression

Simple linear regression lives up to its name: it is a very straightforward approach for predicting a quantitative response Y on the basis of a single predictor variable X. It assumes that there is approximately a linear relationship between X and Y . Mathematically, we can write this linear relationship as

$$\displaystyle{ Y \approx \beta _{0} +\beta _{1}X. }$$

(3.1)

You might read “ ≈ ” as “is approximately modeled as”. We will sometimes describe (3.1) by saying that we are regressing Y on X (or YontoX). For example, X may represent TV advertising and Y may represent sales. Then we can regress sales onto TV by fitting the model

$$\displaystyle{\mathtt{sales} \approx \beta _{0} +\beta _{1} \times \mathtt{TV}.}$$

simple linear regression

In Equation 3.1, β ₀ and β ₁ are two unknown constants that represent the intercept and slope terms in the linear model. Together, β ₀ and β ₁ are known as the model coefficients or parameters. Once we have used our training data to produce estimates $\hat{\beta }_{0}$ and $\hat{\beta }_{1}$ for the model coefficients, we can predict future sales on the basis of a particular value of TV advertising by computing

$$\displaystyle{ \hat{y} =\hat{\beta } _{0} +\hat{\beta } _{1}x, }$$

(3.2)

where $\hat{y}$ indicates a prediction of Y on the basis of X = x. Here we use a hat symbol,  $\hat{}$  , to denote the estimated value for an unknown parameter or coefficient, or to denote the predicted value of the response.

intercept

slope

coefficient

parameter

3.1.1 Estimating the Coefficients

In practice, β ₀ and β ₁ are unknown. So before we can use (3.1) to make predictions, we must use data to estimate the coefficients. Let

$$\displaystyle{(x_{1},y_{1}),\;(x_{2},y_{2}),\ldots ,\;(x_{n},y_{n})}$$

represent n observation pairs, each of which consists of a measurement of X and a measurement of Y . In the Advertising example, this data set consists of the TV advertising budget and product sales in n = 200 different markets. (Recall that the data are displayed in Figure 2.1.) Our goal is to obtain coefficient estimates $\hat{\beta }_{0}$ and $\hat{\beta }_{1}$ such that the linear model (3.1) fits the available data well—that is, so that $y_{i} \approx \hat{\beta }_{0} +\hat{\beta } _{1}x_{i}$ for i = 1, …, n. In other words, we want to find an intercept $\hat{\beta }_{0}$ and a slope $\hat{\beta }_{1}$ such that the resulting line is as close as possible to the n = 200 data points. There are a number of ways of measuring closeness. However, by far the most common approach involves minimizing the least squares criterion, and we take that approach in this chapter. Alternative approaches will be considered in Chapter 6.

least squares

Let $\hat{y}_{i} =\hat{\beta } _{0} +\hat{\beta } _{1}x_{i}$ be the prediction for Y based on the ith value of X. Then $e_{i} = y_{i} -\hat{ y}_{i}$ represents the ith residual —this is the difference between the ith observed response value and the ith response value that is predicted by our linear model. We define the residual sum of squares (RSS) as

$$\displaystyle{\mbox{ RSS} = e_{1}^{2} + e_{ 2}^{2} + \cdots + e_{ n}^{2},}$$

or equivalently as

$$\displaystyle{ \mbox{ RSS} = {(y_{1} -\hat{\beta }_{0} -\hat{\beta }_{1}x_{1})}^{2} + {(y_{ 2} -\hat{\beta }_{0} -\hat{\beta }_{1}x_{2})}^{2} +\ldots +{(y_{ n} -\hat{\beta }_{0} -\hat{\beta }_{1}x_{n})}^{2}. }$$

(3.3)

The least squares approach chooses $\hat{\beta }_{0}$ and $\hat{\beta }_{1}$ to minimize the RSS. Using some calculus, one can show that the minimizers are

$$\displaystyle\begin{array}{rcl} \hat{\beta }_{1}& =& \frac{\sum _{i=1}^{n}(x_{i} -\bar{ x})(y_{i} -\bar{ y})} {\sum _{i=1}^{n}{(x_{i} -\bar{ x})}^{2}} , \\ \hat{\beta }_{0}& =& \bar{y} -\hat{\beta }_{1}\bar{x}, {}\end{array}$$

(3.4)

where $\bar{y} \equiv \frac{1} {n}\sum _{i=1}^{n}y_{ i}$ and $\bar{x} \equiv \frac{1} {n}\sum _{i=1}^{n}x_{ i}$ are the sample means. In other words, (3.4) defines the least squares coefficient estimates for simple linear regression.

residual

residual sum of squares

Figure 3.1 displays the simple linear regression fit to the Advertising data, where $\hat{\beta }_{0} = 7.03$ and $\hat{\beta }_{1} = 0.0475$. In other words, according to this approximation, an additional $1, 000 spent on TV advertising is associated with selling approximately 47. 5 additional units of the product. In Figure 3.2, we have computed RSS for a number of values of β ₀ and β ₁, using the advertising data with sales as the response and TV as the predictor. In each plot, the red dot represents the pair of least squares estimates $(\hat{\beta }_{0},\hat{\beta }_{1})$ given by (3.4). These values clearly minimize the RSS.

Fig. 3.1

For the Advertising data, the least squares fit for the regression of sales  onto TV  is shown. The fit is found by minimizing the sum of squared errors. Each grey line segment represents an error, and the fit makes a compromise by averaging their squares. In this case a linear fit captures the essence of the relationship, although it is somewhat deficient in the left of the plot.

Fig. 3.2

Contour and three-dimensional plots of the RSS on the Advertising data, using sales as the response and TV as the predictor. The red dots correspond to the least squares estimates $\hat{\beta }_{0}$ and $\hat{\beta }_{1}$ , given by ( 3.4 ).

3.1.2 Assessing the Accuracy of the Coefficient Estimates

Recall from (2.1) that we assume that the true relationship between X and Y takes the form $Y = f(X)+\epsilon$ for some unknown function f, where ε is a mean-zero random error term. If f is to be approximated by a linear function, then we can write this relationship as

$$\displaystyle{ Y =\beta _{0} +\beta _{1}X +\epsilon . }$$

(3.5)

Here β ₀ is the intercept term—that is, the expected value of Y when X = 0, and β ₁ is the slope—the average increase in Y associated with a one-unit increase in X. The error term is a catch-all for what we miss with this simple model: the true relationship is probably not linear, there may be other variables that cause variation in Y , and there may be measurement error. We typically assume that the error term is independent of X.

The model given by (3.5) defines the population regression line, which is the best linear approximation to the true relationship between X and Y .¹ The least squares regression coefficient estimates (3.4) characterize the least squares line (3.2). The left-hand panel of Figure 3.3 displays these two lines in a simple simulated example. We created 100 random Xs, and generated 100 corresponding Y s from the model

$$\displaystyle{ Y = 2 + 3X+\epsilon , }$$

(3.6)

where ε was generated from a normal distribution with mean zero. The red line in the left-hand panel of Figure 3.3 displays the true relationship, $f(X) = 2 + 3X$, while the blue line is the least squares estimate based on the observed data. The true relationship is generally not known for real data, but the least squares line can always be computed using the coefficient estimates given in (3.4). In other words, in real applications, we have access to a set of observations from which we can compute the least squares line; however, the population regression line is unobserved. In the right-hand panel of Figure 3.3 we have generated ten different data sets from the model given by (3.6) and plotted the corresponding ten least squares lines. Notice that different data sets generated from the same true model result in slightly different least squares lines, but the unobserved population regression line does not change.

Fig. 3.3

A simulated data set. Left: The red line represents the true relationship, $f(X) = 2 + 3X$ , which is known as the population regression line. The blue line is the least squares line; it is the least squares estimate for f(X) based on the observed data, shown in black. Right: The population regression line is again shown in red, and the least squares line in dark blue. In light blue, ten least squares lines are shown, each computed on the basis of a separate random set of observations. Each least squares line is different, but on average, the least squares lines are quite close to the population regression line.

population regression line

least squares line

At first glance, the difference between the population regression line and the least squares line may seem subtle and confusing. We only have one data set, and so what does it mean that two different lines describe the relationship between the predictor and the response? Fundamentally, the concept of these two lines is a natural extension of the standard statistical approach of using information from a sample to estimate characteristics of a large population. For example, suppose that we are interested in knowing the population mean μ of some random variable Y . Unfortunately, μ is unknown, but we do have access to n observations from Y , which we can write as y ₁, …, y _n , and which we can use to estimate μ. A reasonable estimate is $\hat{\mu }=\bar{ y}$, where $\bar{y} = \frac{1} {n}\sum _{i=1}^{n}y_{ i}$ is the sample mean. The sample mean and the population mean are different, but in general the sample mean will provide a good estimate of the population mean. In the same way, the unknown coefficients β ₀ and β ₁ in linear regression define the population regression line. We seek to estimate these unknown coefficients using $\hat{\beta }_{0}$ and $\hat{\beta }_{1}$ given in (3.4). These coefficient estimates define the least squares line.

The analogy between linear regression and estimation of the mean of a random variable is an apt one based on the concept of bias. If we use the sample mean $\hat{\mu }$ to estimate μ, this estimate is unbiased, in the sense that on average, we expect $\hat{\mu }$ to equal μ. What exactly does this mean? It means that on the basis of one particular set of observations y ₁, …, y _n , $\hat{\mu }$ might overestimate μ, and on the basis of another set of observations, $\hat{\mu }$ might underestimate μ. But if we could average a huge number of estimates of μ obtained from a huge number of sets of observations, then this average would exactly equal μ. Hence, an unbiased estimator does not systematically over- or under-estimate the true parameter. The property of unbiasedness holds for the least squares coefficient estimates given by (3.4) as well: if we estimate β ₀ and β ₁ on the basis of a particular data set, then our estimates won’t be exactly equal to β ₀ and β ₁. But if we could average the estimates obtained over a huge number of data sets, then the average of these estimates would be spot on! In fact, we can see from the right-hand panel of Figure 3.3 that the average of many least squares lines, each estimated from a separate data set, is pretty close to the true population regression line.

bias

unbiased

We continue the analogy with the estimation of the population mean μ of a random variable Y . A natural question is as follows: how accurate is the sample mean $\hat{\mu }$ as an estimate of μ? We have established that the average of $\hat{\mu }$’s over many data sets will be very close to μ, but that a single estimate $\hat{\mu }$ may be a substantial underestimate or overestimate of μ. How far off will that single estimate of $\hat{\mu }$ be? In general, we answer this question by computing the standard error of $\hat{\mu }$, written as $\mbox{ SE}(\hat{\mu })$. We have the well-known formula

$$\displaystyle{ \mbox{ Var}(\hat{\mu }) ={ \mbox{ SE}(\hat{\mu })}^{2} = \frac{{\sigma }^{2}} {n}, }$$

(3.7)

standard error

where σ is the standard deviation of each of the realizations y _i of Y .² Roughly speaking, the standard error tells us the average amount that this estimate $\hat{\mu }$ differs from the actual value of μ. Equation 3.7 also tells us how this deviation shrinks with n—the more observations we have, the smaller the standard error of $\hat{\mu }$. In a similar vein, we can wonder how close $\hat{\beta }_{0}$ and $\hat{\beta }_{1}$ are to the true values β ₀ and β ₁. To compute the standard errors associated with $\hat{\beta }_{0}$ and $\hat{\beta }_{1}$, we use the following formulas:

$$\displaystyle{{ \mbox{ SE}(\hat{\beta }_{0})}^{2} ={\sigma }^{2}\left [ \frac{1} {n} + \frac{{\bar{x}}^{2}} {\sum _{i=1}^{n}{(x_{i} -\bar{ x})}^{2}}\right ],\quad {\mbox{ SE}(\hat{\beta }_{1})}^{2} = \frac{{\sigma }^{2}} {\sum _{i=1}^{n}{(x_{i} -\bar{ x})}^{2}}, }$$

(3.8)

where σ ² = Var(ε). For these formulas to be strictly valid, we need to assume that the errors ε _i for each observation are uncorrelated with common variance σ ². This is clearly not true in Figure 3.1, but the formula still turns out to be a good approximation. Notice in the formula that $\mbox{ SE}(\hat{\beta }_{1})$ is smaller when the x _i are more spread out; intuitively we have more leverage to estimate a slope when this is the case. We also see that $\mbox{ SE}(\hat{\beta }_{0})$ would be the same as $\mbox{ SE}(\hat{\mu })$ if $\bar{x}$ were zero (in which case $\hat{\beta }_{0}$ would be equal to $\bar{y}$). In general, σ ² is not known, but can be estimated from the data. This estimate of σ is known as the residual standard error, and is given by the formula $\mbox{ RSE} = \sqrt{\mbox{ RSS} /(n - 2)}$. Strictly speaking, when σ ² is estimated from the data we should write $\widehat{\mbox{ SE}}(\hat{\beta }_{1})$ to indicate that an estimate has been made, but for simplicity of notation we will drop this extra “hat”.

residual standard error

Standard errors can be used to compute confidence intervals. A 95 % confidence interval is defined as a range of values such that with 95 % probability, the range will contain the true unknown value of the parameter. The range is defined in terms of lower and upper limits computed from the sample of data. For linear regression, the 95 % confidence interval for β ₁ approximately takes the form

$$\displaystyle{ \hat{\beta }_{1} \pm 2 \cdot \mbox{ SE}(\hat{\beta }_{1}). }$$

(3.9)

That is, there is approximately a 95 % chance that the interval

$$\displaystyle{ \left [\hat{\beta }_{1} - 2 \cdot \mbox{ SE}(\hat{\beta }_{1}),\;\hat{\beta }_{1} + 2 \cdot \mbox{ SE}(\hat{\beta }_{1})\right ] }$$

(3.10)

will contain the true value of β ₁.³ Similarly, a confidence interval for β ₀ approximately takes the form

$$\displaystyle{ \hat{\beta }_{0} \pm 2 \cdot \mbox{ SE}(\hat{\beta }_{0}). }$$

(3.11)

confidence interval

In the case of the advertising data, the 95 % confidence interval for β ₀ is [6. 130, 7. 935] and the 95 % confidence interval for β ₁ is [0. 042, 0. 053]. Therefore, we can conclude that in the absence of any advertising, sales will, on average, fall somewhere between 6, 130 and 7, 940 units. Furthermore, for each $1, 000 increase in television advertising, there will be an average increase in sales of between 42 and 53 units.

Standard errors can also be used to perform hypothesis tests on the coefficients. The most common hypothesis test involves testing the null hypothesis of

$$\displaystyle{ H_{0} : \mbox{ There is no relationship between $X$ and $Y $} }$$

(3.12)

versus the alternative hypothesis

$$\displaystyle{ H_{a} : \mbox{ There is some relationship between $X$ and $Y $}. }$$

(3.13)

Mathematically, this corresponds to testing

$$\displaystyle{H_{0} :\beta _{1} = 0}$$

versus

$$\displaystyle{H_{a} :\beta _{1}\neq 0,}$$

since if β ₁ = 0 then the model (3.5) reduces to $Y =\beta _{0}+\epsilon$, and X is not associated with Y . To test the null hypothesis, we need to determine whether $\hat{\beta }_{1}$, our estimate for β ₁, is sufficiently far from zero that we can be confident that β ₁ is non-zero. How far is far enough? This of course depends on the accuracy of $\hat{\beta }_{1}$—that is, it depends on $\mbox{ SE}(\hat{\beta }_{1})$. If $\mbox{ SE}(\hat{\beta }_{1})$ is small, then even relatively small values of $\hat{\beta }_{1}$ may provide strong evidence that β ₁ ≠ 0, and hence that there is a relationship between X and Y . In contrast, if $\mbox{ SE}(\hat{\beta }_{1})$ is large, then $\hat{\beta }_{1}$ must be large in absolute value in order for us to reject the null hypothesis. In practice, we compute a t-statistic, given by

$$\displaystyle{ t = \frac{\hat{\beta }_{1} - 0} {\mbox{ SE}(\hat{\beta }_{1})}, }$$

(3.14)

which measures the number of standard deviations that $\hat{\beta }_{1}$ is away from 0. If there really is no relationship between X and Y , then we expect that (3.14) will have a t-distribution with n − 2 degrees of freedom. The t-distribution has a bell shape and for values of n greater than approximately 30 it is quite similar to the normal distribution. Consequently, it is a simple matter to compute the probability of observing any value equal to | t | or larger, assuming β ₁ = 0. We call this probability the p-value. Roughly speaking, we interpret the p-value as follows: a small p-value indicates that it is unlikely to observe such a substantial association between the predictor and the response due to chance, in the absence of any real association between the predictor and the response. Hence, if we see a small p-value, then we can infer that there is an association between the predictor and the response. We reject the null hypothesis—that is, we declare a relationship to exist between X and Y —if the p-value is small enough. Typical p-value cutoffs for rejecting the null hypothesis are 5 or 1 %. When n = 30, these correspond to t-statistics (3.14) of around 2 and 2.75, respectively.

hypothesis test

null hypothesis

alternative hypothesis

t-statistic

p-value

Table 3.1 provides details of the least squares model for the regression of number of units sold on TV advertising budget for the Advertising data. Notice that the coefficients for $\hat{\beta }_{0}$ and $\hat{\beta }_{1}$ are very large relative to their standard errors, so the t-statistics are also large; the probabilities of seeing such values if H ₀ is true are virtually zero. Hence we can conclude that β ₀ ≠ 0 and β ₁ ≠ 0.⁴

Table 3.1 For the Advertising data, coefficients of the least squares model for the regression of number of units sold on TV advertising budget. An increase of $1,000 in the TV advertising budget is associated with an increase in sales by around 50 units (Recall that the sales variable is in thousands of units, and the TV variable is in thousands of dollars).

3.1.3 Assessing the Accuracy of the Model

Once we have rejected the null hypothesis (3.12) in favor of the alternative hypothesis (3.13), it is natural to want to quantify the extent to which the model fits the data. The quality of a linear regression fit is typically assessed using two related quantities: the residual standard error (RSE) and the R ² statistic.

R ²

Table 3.2 displays the RSE, the R ² statistic, and the F-statistic (to be described in Section 3.2.2) for the linear regression of number of units sold on TV advertising budget.

Table 3.2 For the Advertising data, more information about the least squares model for the regression of number of units sold on TV advertising budget.

3.1.3.1 Residual Standard Error

Recall from the model (3.5) that associated with each observation is an error term ε. Due to the presence of these error terms, even if we knew the true regression line (i.e. even if β ₀ and β ₁ were known), we would not be able to perfectly predict Y from X. The RSE is an estimate of the standard deviation of ε. Roughly speaking, it is the average amount that the response will deviate from the true regression line. It is computed using the formula

$$\displaystyle{ \mbox{ RSE} = \sqrt{ \frac{1} {n - 2}\mbox{ RSS}} = \sqrt{ \frac{1} {n - 2}\sum _{i=1}^{n}{(y_{i} -\hat{ y}_{i})}^{2}}. }$$

(3.15)

Note that RSS was defined in Section 3.1.1, and is given by the formula

$$\displaystyle{ \mbox{ RSS} =\sum _{ i=1}^{n}{(y_{ i} -\hat{ y}_{i})}^{2}. }$$

(3.16)

In the case of the advertising data, we see from the linear regression output in Table 3.2 that the RSE is 3. 26. In other words, actual sales in each market deviate from the true regression line by approximately 3, 260 units, on average. Another way to think about this is that even if the model were correct and the true values of the unknown coefficients β ₀ and β ₁ were known exactly, any prediction of sales on the basis of TV advertising would still be off by about 3, 260 units on average. Of course, whether or not 3, 260 units is an acceptable prediction error depends on the problem context. In the advertising data set, the mean value of sales over all markets is approximately 14, 000 units, and so the percentage error is $3,260/14,000 = 23$ %.

The RSE is considered a measure of the lack of fit of the model (3.5) to the data. If the predictions obtained using the model are very close to the true outcome values—that is, if $\hat{y}_{i} \approx y_{i}$ for i = 1, …, n—then (3.15) will be small, and we can conclude that the model fits the data very well. On the other hand, if $\hat{y}_{i}$ is very far from y _i for one or more observations, then the RSE may be quite large, indicating that the model doesn’t fit the data well.

3.1.3.2 R ² Statistic

The RSE provides an absolute measure of lack of fit of the model (3.5) to the data. But since it is measured in the units of Y , it is not always clear what constitutes a good RSE. The R ² statistic provides an alternative measure of fit. It takes the form of a proportion—the proportion of variance explained—and so it always takes on a value between 0 and 1, and is independent of the scale of Y .

To calculate R ², we use the formula

$$\displaystyle{ {R}^{2} = \frac{\mbox{ TSS} -\mbox{ RSS}} {\mbox{ TSS}} = 1 -\frac{\mbox{ RSS}} {\mbox{ TSS}} }$$

(3.17)

where $\mbox{ TSS} =\sum {(y_{i} -\bar{ y})}^{2}$ is the total sum of squares, and RSS is defined in (3.16). TSS measures the total variance in the response Y , and can be thought of as the amount of variability inherent in the response before the regression is performed. In contrast, RSS measures the amount of variability that is left unexplained after performing the regression. Hence, TSS − RSS measures the amount of variability in the response that is explained (or removed) by performing the regression, and R ² measures the proportion of variability in Y that can be explained using X. An R ² statistic that is close to 1 indicates that a large proportion of the variability in the response has been explained by the regression. A number near 0 indicates that the regression did not explain much of the variability in the response; this might occur because the linear model is wrong, or the inherent error σ ² is high, or both. In Table 3.2, the R ² was 0. 61, and so just under two-thirds of the variability in sales is explained by a linear regression on TV.

total sum of squares

The R ² statistic (3.17) has an interpretational advantage over the RSE (3.15), since unlike the RSE, it always lies between 0 and 1. However, it can still be challenging to determine what is a goodR ² value, and in general, this will depend on the application. For instance, in certain problems in physics, we may know that the data truly comes from a linear model with a small residual error. In this case, we would expect to see an R ² value that is extremely close to 1, and a substantially smaller R ² value might indicate a serious problem with the experiment in which the data were generated. On the other hand, in typical applications in biology, psychology, marketing, and other domains, the linear model (3.5) is at best an extremely rough approximation to the data, and residual errors due to other unmeasured factors are often very large. In this setting, we would expect only a very small proportion of the variance in the response to be explained by the predictor, and an R ² value well below 0. 1 might be more realistic!

The R ² statistic is a measure of the linear relationship between X and Y . Recall that correlation, defined as

$$\displaystyle{ \mbox{ Cor}(X,Y ) = \frac{\sum _{i=1}^{n}(x_{i} -\overline{x})(y_{i} -\overline{y})} {\sqrt{\sum _{i=1 }^{n }{(x_{i } - \overline{x } )}^{2}}\sqrt{\sum _{i=1 }^{n }{(y_{i } - \overline{y } )}^{2}}}, }$$

(3.18)

is also a measure of the linear relationship between X and Y .⁵ This suggests that we might be able to use r = Cor(X, Y ) instead of R ² in order to assess the fit of the linear model. In fact, it can be shown that in the simple linear regression setting, R ² = r ². In other words, the squared correlation and the R ² statistic are identical. However, in the next section we will discuss the multiple linear regression problem, in which we use several predictors simultaneously to predict the response. The concept of correlation between the predictors and the response does not extend automatically to this setting, since correlation quantifies the association between a single pair of variables rather than between a larger number of variables. We will see that R ² fills this role.

correlation

3.2 Multiple Linear Regression

Simple linear regression is a useful approach for predicting a response on the basis of a single predictor variable. However, in practice we often have more than one predictor. For example, in the Advertising data, we have examined the relationship between sales and TV advertising. We also have data for the amount of money spent advertising on the radio and in newspapers, and we may want to know whether either of these two media is associated with sales. How can we extend our analysis of the advertising data in order to accommodate these two additional predictors?

One option is to run three separate simple linear regressions, each of which uses a different advertising medium as a predictor. For instance, we can fit a simple linear regression to predict sales on the basis of the amount spent on radio advertisements. Results are shown in Table 3.3 (top table). We find that a $1, 000 increase in spending on radio advertising is associated with an increase in sales by around 203 units. Table 3.3 (bottom table) contains the least squares coefficients for a simple linear regression of sales onto newspaper advertising budget. A $1, 000 increase in newspaper advertising budget is associated with an increase in sales by approximately 55 units.

Table 3.3 More simple linear regression models for the Advertising data. Coefficients of the simple linear regression model for number of units sold on Top: radio advertising budget and Bottom: newspaper advertising budget. A $1,000 increase in spending on radio advertising is associated with an average increase in sales by around 203 units, while the same increase in spending on newspaper advertising is associated with an average increase in sales by around 55 units (Note that the sales variable is in thousands of units, and the radio and newspaper variables are in thousands of dollars).

However, the approach of fitting a separate simple linear regression model for each predictor is not entirely satisfactory. First of all, it is unclear how to make a single prediction of sales given levels of the three advertising media budgets, since each of the budgets is associated with a separate regression equation. Second, each of the three regression equations ignores the other two media in forming estimates for the regression coefficients. We will see shortly that if the media budgets are correlated with each other in the 200 markets that constitute our data set, then this can lead to very misleading estimates of the individual media effects on sales.

Instead of fitting a separate simple linear regression model for each predictor, a better approach is to extend the simple linear regression model (3.5) so that it can directly accommodate multiple predictors. We can do this by giving each predictor a separate slope coefficient in a single model. In general, suppose that we have p distinct predictors. Then the multiple linear regression model takes the form

$$\displaystyle{ Y =\beta _{0} +\beta _{1}X_{1} +\beta _{2}X_{2} + \cdots +\beta _{p}X_{p}+\epsilon , }$$

(3.19)

where X _j represents the jth predictor and β _j quantifies the association between that variable and the response. We interpret β _j as the average effect on Y of a one unit increase in X _j , holding all other predictors fixed. In the advertising example, (3.19) becomes

$$\displaystyle{ \mathtt{sales} =\beta _{0} +\beta _{1} \times \mathtt{TV} +\beta _{2} \times \mathtt{radio} +\beta _{3} \times \mathtt{newspaper} +\epsilon . }$$

(3.20)

3.2.1 Estimating the Regression Coefficients

As was the case in the simple linear regression setting, the regression coefficients β ₀, β ₁, …, β _p in (3.19) are unknown, and must be estimated. Given estimates $\hat{\beta }_{0},\hat{\beta }_{1},\ldots ,\hat{\beta }_{p}$, we can make predictions using the formula

$$\displaystyle{ \hat{y} =\hat{\beta } _{0} +\hat{\beta } _{1}x_{1} +\hat{\beta } _{2}x_{2} + \cdots +\hat{\beta } _{p}x_{p}. }$$

(3.21)

The parameters are estimated using the same least squares approach that we saw in the context of simple linear regression. We choose β ₀, β ₁, …, β _p to minimize the sum of squared residuals

$$\displaystyle\begin{array}{rcl} \mbox{ RSS}& =& \sum _{i=1}^{n}{(y_{ i} -\hat{ y}_{i})}^{2} \\ & =& \sum _{i=1}^{n}{(y_{ i} -\hat{\beta }_{0} -\hat{\beta }_{1}x_{i1} -\hat{\beta }_{2}x_{i2} -\cdots -\hat{\beta }_{p}x_{ip})}^{2}.{}\end{array}$$

(3.22)

The values $\hat{\beta }_{0},\hat{\beta }_{1},\ldots ,\hat{\beta }_{p}$ that minimize (3.22) are the multiple least squares regression coefficient estimates. Unlike the simple linear regression estimates given in (3.4), the multiple regression coefficient estimates have somewhat complicated forms that are most easily represented using matrix algebra. For this reason, we do not provide them here. Any statistical software package can be used to compute these coefficient estimates, and later in this chapter we will show how this can be done in R. Figure 3.4 illustrates an example of the least squares fit to a toy data set with p = 2 predictors.

Table 3.4 displays the multiple regression coefficient estimates when TV,radio, and newspaper advertising budgets are used to predict product sales using the Advertising data. We interpret these results as follows: for a given amount of TV and newspaper advertising, spending an additional $1, 000 on radio advertising leads to an increase in sales by approximately 189 units. Comparing these coefficient estimates to those displayed in Tables 3.1 and 3.3, we notice that the multiple regression coefficient estimates for TV and radio are pretty similar to the simple linear regression coefficient estimates. However, while the newspaper regression coefficient estimate in Table 3.3 was significantly non-zero, the coefficient estimate for newspaper in the multiple regression model is close to zero, and the corresponding p-value is no longer significant, with a value around 0. 86. This illustrates that the simple and multiple regression coefficients can be quite different. This difference stems from the fact that in the simple regression case, the slope term represents the average effect of a $1, 000 increase in newspaper advertising, ignoring other predictors such as TV and radio. In contrast, in the multiple regression setting, the coefficient for newspaper represents the average effect of increasing newspaper spending by $1, 000 while holding TV and radio fixed.

Fig. 3.4

In a three-dimensional setting, with two predictors and one response, the least squares regression line becomes a plane. The plane is chosen to minimize the sum of the squared vertical distances between each observation (shown in red) and the plane.

Table 3.4 For the Advertising data, least squares coefficient estimates of the multiple linear regression of number of units sold on radio, TV, and newspaper advertising budgets.

Does it make sense for the multiple regression to suggest no relationship between sales and newspaper while the simple linear regression implies the opposite? In fact it does. Consider the correlation matrix for the three predictor variables and response variable, displayed in Table 3.5. Notice that the correlation between radio and newspaper is 0. 35. This reveals a tendency to spend more on newspaper advertising in markets where more is spent on radio advertising. Now suppose that the multiple regression is correct and newspaper advertising has no direct impact on sales, but radio advertising does increase sales. Then in markets where we spend more on radio our sales will tend to be higher, and as our correlation matrix shows, we also tend to spend more on newspaper advertising in those same markets. Hence, in a simple linear regression which only examines sales versus newspaper, we will observe that higher values of newspaper tend to be associated with higher values of sales, even though newspaper advertising does not actually affect sales. So newspaper sales are a surrogate for radio advertising; newspaper gets “credit” for the effect of radio on sales.

Table 3.5 Correlation matrix for TV, radio, newspaper , and sales for the Advertising data.

This slightly counterintuitive result is very common in many real life situations. Consider an absurd example to illustrate the point. Running a regression of shark attacks versus ice cream sales for data collected at a given beach community over a period of time would show a positive relationship, similar to that seen between sales and newspaper. Of course no one (yet) has suggested that ice creams should be banned at beaches to reduce shark attacks. In reality, higher temperatures cause more people to visit the beach, which in turn results in more ice cream sales and more shark attacks. A multiple regression of attacks versus ice cream sales and temperature reveals that, as intuition implies, the former predictor is no longer significant after adjusting for temperature.

3.2.2 Some Important Questions

When we perform multiple linear regression, we usually are interested in answering a few important questions.

• 1. Is at least one of the predictors X ₁ ,X ₂ ,…,X _p useful in predicting the response?

• 2. Do all the predictors help to explain Y , or is only a subset of the predictors useful?

• 3. How well does the model fit the data?

• 4. Given a set of predictor values, what response value should we predict, and how accurate is our prediction?

We now address each of these questions in turn.

3.2.2.1 One: Is There a Relationship Between the Response and Predictors?

Recall that in the simple linear regression setting, in order to determine whether there is a relationship between the response and the predictor we can simply check whether β ₁ = 0. In the multiple regression setting with p predictors, we need to ask whether all of the regression coefficients are zero, i.e. whether $\beta _{1} =\beta _{2} = \cdots =\beta _{p} = 0$. As in the simple linear regression setting, we use a hypothesis test to answer this question. We test the null hypothesis,

$$\displaystyle{H_{0} :\beta _{1} =\beta _{2} = \cdots =\beta _{p} = 0}$$

versus the alternative

$$\displaystyle{H_{a} : \mbox{ at least one $\beta _{j}$ is non-zero.}}$$

This hypothesis test is performed by computing the F-statistic,

$$\displaystyle{ F = \frac{(\mbox{ TSS} -\mbox{ RSS})/p} {\mbox{ RSS}/(n - p - 1)} , }$$

(3.23)

where, as with simple linear regression, $\mbox{ TSS} =\sum {(y_{i} -\bar{ y})}^{2}$ and $\mbox{ RSS} =\sum {(y_{i} -\hat{ y}_{i})}^{2}$. If the linear model assumptions are correct, one can show that

$$\displaystyle{E\{\mbox{ RSS}/(n - p - 1)\} {=\sigma }^{2}}$$

and that, provided H ₀ is true,

$$\displaystyle{E\{(\mbox{ TSS} -\mbox{ RSS})/p\} {=\sigma }^{2}.}$$

Hence, when there is no relationship between the response and predictors, one would expect the F-statistic to take on a value close to 1. On the other hand, if H _a is true, then $E\{(\mbox{ TSS} -\mbox{ RSS})/p\} {>\sigma }^{2}$, so we expect F to be greater than 1.

F-statistic

The F-statistic for the multiple linear regression model obtained by regressing sales onto radio, TV, and newspaper is shown in Table 3.6. In this example the F-statistic is 570. Since this is far larger than 1, it provides compelling evidence against the null hypothesis H ₀. In other words, the large F-statistic suggests that at least one of the advertising media must be related to sales. However, what if the F-statistic had been closer to 1? How large does the F-statistic need to be before we can reject H ₀ and conclude that there is a relationship? It turns out that the answer depends on the values of n and p. When n is large, an F-statistic that is just a little larger than 1 might still provide evidence against H ₀. In contrast, a larger F-statistic is needed to reject H ₀ if n is small. When H ₀ is true and the errors ε _i have a normal distribution, the F-statistic follows an F-distribution.⁶ For any given value of n and p, any statistical software package can be used to compute the p-value associated with the F-statistic using this distribution. Based on this p-value, we can determine whether or not to reject H ₀. For the advertising data, the p-value associated with the F-statistic in Table 3.6 is essentially zero, so we have extremely strong evidence that at least one of the media is associated with increased sales.

Table 3.6 More information about the least squares model for the regression of number of units sold on TV, newspaper, and radio advertising budgets in the Advertising data. Other information about this model was displayed in Table  3.4 .

In (3.23) we are testing H ₀ that all the coefficients are zero. Sometimes we want to test that a particular subset of q of the coefficients are zero. This corresponds to a null hypothesis

$$\displaystyle{H_{0} : \quad \beta _{p-q+1} =\beta _{p-q+2} =\ldots =\beta _{p} = 0,}$$

where for convenience we have put the variables chosen for omission at the end of the list. In this case we fit a second model that uses all the variables except those last q. Suppose that the residual sum of squares for that model is RSS₀. Then the appropriate F-statistic is

$$\displaystyle{ F = \frac{(\mbox{ RSS}_{0} -\mbox{ RSS})/q} {\mbox{ RSS}/(n - p - 1)} . }$$

(3.24)

Notice that in Table 3.4, for each individual predictor a t-statistic and a p-value were reported. These provide information about whether each individual predictor is related to the response, after adjusting for the other predictors. It turns out that each of these are exactly equivalent⁷ to the F-test that omits that single variable from the model, leaving all the others in—i.e. q=1 in (3.24). So it reports the partial effect of adding that variable to the model. For instance, as we discussed earlier, these p-values indicate that TV and radio are related to sales, but that there is no evidence that newspaper is associated with sales, in the presence of these two.

Given these individual p-values for each variable, why do we need to look at the overall F-statistic? After all, it seems likely that if any one of the p-values for the individual variables is very small, then at least one of the predictors is related to the response. However, this logic is flawed, especially when the number of predictors p is large.

For instance, consider an example in which p = 100 and $H_{0} :\beta _{1} =\beta _{2} =\ldots =\beta _{p} = 0$ is true, so no variable is truly associated with the response. In this situation, about 5 % of the p-values associated with each variable (of the type shown in Table 3.4) will be below 0. 05 by chance. In other words, we expect to see approximately five small p-values even in the absence of any true association between the predictors and the response. In fact, we are almost guaranteed that we will observe at least one p-value below 0. 05 by chance! Hence, if we use the individual t-statistics and associated p-values in order to decide whether or not there is any association between the variables and the response, there is a very high chance that we will incorrectly conclude that there is a relationship. However, the F-statistic does not suffer from this problem because it adjusts for the number of predictors. Hence, if H ₀ is true, there is only a 5 % chance that the F-statistic will result in a p-value below 0. 05, regardless of the number of predictors or the number of observations.

The approach of using an F-statistic to test for any association between the predictors and the response works when p is relatively small, and certainly small compared to n. However, sometimes we have a very large number of variables. If p > n then there are more coefficients β _j to estimate than observations from which to estimate them. In this case we cannot even fit the multiple linear regression model using least squares, so the F-statistic cannot be used, and neither can most of the other concepts that we have seen so far in this chapter. When p is large, some of the approaches discussed in the next section, such as forward selection, can be used. This high-dimensional setting is discussed in greater detail in Chapter 6.

high-dimensional

3.2.2.2 Two: Deciding on Important Variables

As discussed in the previous section, the first step in a multiple regression analysis is to compute the F-statistic and to examine the associated p-value. If we conclude on the basis of that p-value that at least one of the predictors is related to the response, then it is natural to wonder which are the guilty ones! We could look at the individual p-values as in Table 3.4, but as discussed, if p is large we are likely to make some false discoveries.

It is possible that all of the predictors are associated with the response, but it is more often the case that the response is only related to a subset of the predictors. The task of determining which predictors are associated with the response, in order to fit a single model involving only those predictors, is referred to as variable selection. The variable selection problem is studied extensively in Chapter 6, and so here we will provide only a brief outline of some classical approaches.

variable selection

Ideally, we would like to perform variable selection by trying out a lot of different models, each containing a different subset of the predictors. For instance, if p = 2, then we can consider four models: (1) a model containing no variables, (2) a model containing X ₁ only, (3) a model containing X ₂ only, and (4) a model containing both X ₁ and X ₂. We can then select the best model out of all of the models that we have considered. How do we determine which model is best? Various statistics can be used to judge the quality of a model. These include Mallow’s C _p, Akaike information criterion (AIC), Bayesian information criterion (BIC), and adjusted R ². These are discussed in more detail in Chapter 6. We can also determine which model is best by plotting various model outputs, such as the residuals, in order to search for patterns.

Mallow’s C _p

Akaike information criterion

Bayesian information criterion

adjusted R ²

Unfortunately, there are a total of 2^p models that contain subsets of p variables. This means that even for moderate p, trying out every possible subset of the predictors is infeasible. For instance, we saw that if p = 2, then there are 2² = 4 models to consider. But if p = 30, then we must consider 2³⁰ = 1, 073, 741, 824 models! This is not practical. Therefore, unless p is very small, we cannot consider all 2^p models, and instead we need an automated and efficient approach to choose a smaller set of models to consider. There are three classical approaches for this task:

• Forward selection. We begin with the null model —a model that contains an intercept but no predictors. We then fit p simple linear regressions and add to the null model the variable that results in the lowest RSS. We then add to that model the variable that results in the lowest RSS for the new two-variable model. This approach is continued until some stopping rule is satisfied.

• Backward selection. We start with all variables in the model, and remove the variable with the largest p-value—that is, the variable that is the least statistically significant. The new (p − 1)-variable model is fit, and the variable with the largest p-value is removed. This procedure continues until a stopping rule is reached. For instance, we may stop when all remaining variables have a p-value below some threshold.

• Mixed selection. This is a combination of forward and backward selection. We start with no variables in the model, and as with forward selection, we add the variable that provides the best fit. We continue to add variables one-by-one. Of course, as we noted with the Advertising example, the p-values for variables can become larger as new predictors are added to the model. Hence, if at any point the p-value for one of the variables in the model rises above a certain threshold, then we remove that variable from the model. We continue to perform these forward and backward steps until all variables in the model have a sufficiently low p-value, and all variables outside the model would have a large p-value if added to the model.

forward selection

null model

backward selection

mixed selection

Backward selection cannot be used if p > n, while forward selection can always be used. Forward selection is a greedy approach, and might include variables early that later become redundant. Mixed selection can remedy this.

3.2.2.3 Three: Model Fit

Two of the most common numerical measures of model fit are the RSE and R ², the fraction of variance explained. These quantities are computed and interpreted in the same fashion as for simple linear regression.

Recall that in simple regression, R ² is the square of the correlation of the response and the variable. In multiple linear regression, it turns out that it equals $\mbox{ Cor}{(Y,\hat{Y })}^{2}$, the square of the correlation between the response and the fitted linear model; in fact one property of the fitted linear model is that it maximizes this correlation among all possible linear models.

An R ² value close to 1 indicates that the model explains a large portion of the variance in the response variable. As an example, we saw in Table 3.6 that for the Advertising data, the model that uses all three advertising media to predict sales has an R ² of 0. 8972. On the other hand, the model that uses only TV and radio to predict sales has an R ² value of 0. 89719. In other words, there is a small increase in R ² if we include newspaper advertising in the model that already contains TV and radio advertising, even though we saw earlier that the p-value for newspaper advertising in Table 3.4 is not significant. It turns out that R ² will always increase when more variables are added to the model, even if those variables are only weakly associated with the response. This is due to the fact that adding another variable to the least squares equations must allow us to fit the training data (though not necessarily the testing data) more accurately. Thus, the R ² statistic, which is also computed on the training data, must increase. The fact that adding newspaper advertising to the model containing only TV and radio advertising leads to just a tiny increase in R ² provides additional evidence that newspaper can be dropped from the model. Essentially, newspaper provides no real improvement in the model fit to the training samples, and its inclusion will likely lead to poor results on independent test samples due to overfitting.

In contrast, the model containing only TV as a predictor had an R ² of 0. 61 (Table 3.2). Adding radio to the model leads to a substantial improvement in R ². This implies that a model that uses TV and radio expenditures to predict sales is substantially better than one that uses only TV advertising. We could further quantify this improvement by looking at the p-value for the radio coefficient in a model that contains only TV and radio as predictors.

The model that contains only TV and radio as predictors has an RSE of 1.681, and the model that also contains newspaper as a predictor has an RSE of 1.686 (Table 3.6). In contrast, the model that contains only TV has an RSE of 3. 26 (Table 3.2). This corroborates our previous conclusion that a model that uses TV and radio expenditures to predict sales is much more accurate (on the training data) than one that only uses TV spending. Furthermore, given that TV and radio expenditures are used as predictors, there is no point in also using newspaper spending as a predictor in the model. The observant reader may wonder how RSE can increase when newspaper is added to the model given that RSS must decrease. In general RSE is defined as

$$\displaystyle{ \mbox{ RSE} = \sqrt{ \frac{1} {n - p - 1}\mbox{ RSS}}, }$$

(3.25)

which simplifies to (3.15) for a simple linear regression. Thus, models with more variables can have higher RSE if the decrease in RSS is small relative to the increase in p.

In addition to looking at the RSE and R ² statistics just discussed, it can be useful to plot the data. Graphical summaries can reveal problems with a model that are not visible from numerical statistics. For example, Figure 3.5 displays a three-dimensional plot of TV and radio versus sales. We see that some observations lie above and some observations lie below the least squares regression plane. Notice that there is a clear pattern of negative residuals, followed by positive residuals, followed by negative residuals. In particular, the linear model seems to overestimate sales for instances in which most of the advertising money was spent exclusively on either TV or radio. It underestimates sales for instances where the budget was split between the two media. This pronounced non-linear pattern cannot be modeled accurately using linear regression. It suggests a synergy or interaction effect between the advertising media, whereby combining the media together results in a bigger boost to sales than using any single medium. In Section 3.3.2, we will discuss extending the linear model to accommodate such synergistic effects through the use of interaction terms.  

Fig. 3.5

For the Advertising data, a linear regression fit to sales using TV and radio as predictors. From the pattern of the residuals, we can see that there is a pronounced non-linear relationship in the data.

3.2.2.4 Four: Predictions

Once we have fit the multiple regression model, it is straightforward to apply (3.21) in order to predict the response Y on the basis of a set of values for the predictors X ₁, X ₂, …, X _p . However, there are three sorts of uncertainty associated with this prediction.

• 1. The coefficient estimates $\hat{\beta }_{0},\hat{\beta }_{1},\ldots ,\hat{\beta }_{p}$ are estimates for β ₀, β ₁, …, β _p . That is, the least squares plane

  $$\displaystyle{\hat{Y } =\hat{\beta } _{0} +\hat{\beta } _{1}X_{1} + \cdots +\hat{\beta } _{p}X_{p}}$$

  is only an estimate for the true population regression plane

  $$\displaystyle{f(X) =\beta _{0} +\beta _{1}X_{1} + \cdots +\beta _{p}X_{p}.}$$

  The inaccuracy in the coefficient estimates is related to the reducible error from Chapter 2. We can compute a confidence interval in order to determine how close $\hat{Y }$ will be to f(X).

• 2. Of course, in practice assuming a linear model for f(X) is almost always an approximation of reality, so there is an additional source of potentially reducible error which we call model bias. So when we use a linear model, we are in fact estimating the best linear approximation to the true surface. However, here we will ignore this discrepancy, and operate as if the linear model were correct.

• 3. Even if we knew f(X)—that is, even if we knew the true values for β ₀, β ₁, …, β _p —the response value cannot be predicted perfectly because of the random error ε in the model (3.21). In Chapter 2, we referred to this as the irreducible error. How much will Y vary from $\hat{Y }$? We use prediction intervals to answer this question. Prediction intervals are always wider than confidence intervals, because they incorporate both the error in the estimate for f(X) (the reducible error) and the uncertainty as to how much an individual point will differ from the population regression plane (the irreducible error).

We use a confidence interval to quantify the uncertainty surrounding the average sales over a large number of cities. For example, given that $100, 000 is spent on TV advertising and $20, 000 is spent on radio advertising in each city, the 95 % confidence interval is [10, 985,  11, 528]. We interpret this to mean that 95 % of intervals of this form will contain the true value of f(X).⁸ On the other hand, a prediction interval can be used to quantify the uncertainty surrounding sales for a particular city. Given that $100, 000 is spent on TV advertising and $20, 000 is spent on radio advertising in that city the 95 % prediction interval is [7, 930,  14, 580]. We interpret this to mean that 95 % of intervals of this form will contain the true value of Y for this city. Note that both intervals are centered at 11, 256, but that the prediction interval is substantially wider than the confidence interval, reflecting the increased uncertainty about sales for a given city in comparison to the average sales over many locations.

confidence interval

prediction interval

3.3 Other Considerations in the Regression Model

3.3.1 Qualitative Predictors

In our discussion so far, we have assumed that all variables in our linear regression model are quantitative. But in practice, this is not necessarily the case; often some predictors are qualitative.

For example, the Credit data set displayed in Figure 3.6 records balance (average credit card debt for a number of individuals) as well as several quantitative predictors: age, cards (number of credit cards), education (years of education), income (in thousands of dollars), limit (credit limit), and rating (credit rating). Each panel of Figure 3.6 is a scatterplot for a pair of variables whose identities are given by the corresponding row and column labels. For example, the scatterplot directly to the right of the word “Balance” depicts balance versus age, while the plot directly to the right of “Age” corresponds to age versus cards. In addition to these quantitative variables, we also have four qualitative variables: gender, student (student status), status (marital status), and ethnicity (Caucasian, African American or Asian).

Fig. 3.6

The Credit data set contains information about balance, age, cards, education, income, limit , and rating for a number of potential customers.

3.3.1.1 Predictors with Only Two Levels

Suppose that we wish to investigate differences in credit card balance between males and females, ignoring the other variables for the moment. If a qualitative predictor (also known as a factor) only has two levels, or possible values, then incorporating it into a regression model is very simple. We simply create an indicator or dummy variable that takes on two possible numerical values. For example, based on the gender variable, we can create a new variable that takes the form

$$\displaystyle{ x_{i} = \left \{\begin{array}{@{}l@{\quad }l@{}} 1\quad &\;\;\;\text{if}\ i\text{th person is female}\\ 0\quad &\;\;\;\text{if} \ i\text{th person is male} , \end{array} \right . }$$

(3.26)

and use this variable as a predictor in the regression equation. This results in the model

$$\displaystyle{ y_{i} =\beta _{0}+\beta _{1}x_{i}+\epsilon _{i} = \left \{\begin{array}{@{}l@{\quad }l@{}} \beta _{0} +\beta _{1} +\epsilon _{i}\quad &\;\;\;\mbox{ if $i$th person is female} \\ \beta _{0} +\epsilon _{i} \quad & \;\;\;\mbox{if $i$th person is male}. \end{array} \right . }$$

(3.27)

Now β ₀ can be interpreted as the average credit card balance among males, β ₀ + β ₁ as the average credit card balance among females, and β ₁ as the average difference in credit card balance between females and males.

factor

level

dummy variable

Table 3.7 displays the coefficient estimates and other information associated with the model (3.27). The average credit card debt for males is estimated to be $509. 80, whereas females are estimated to carry $19. 73 in additional debt for a total of $509. 80 + $19. 73 = $529. 53. However, we notice that the p-value for the dummy variable is very high. This indicates that there is no statistical evidence of a difference in average credit card balance between the genders.

Table 3.7 Least squares coefficient estimates associated with the regression of balance onto gender in the Credit data set. The linear model is given in (3.27) . That is, gender is encoded as a dummy variable, as in (3.26) .

The decision to code females as 1 and males as 0 in (3.27) is arbitrary, and has no effect on the regression fit, but does alter the interpretation of the coefficients. If we had coded males as 1 and females as 0, then the estimates for β ₀ and β ₁ would have been 529. 53 and − 19. 73, respectively, leading once again to a prediction of credit card debt of $529. 53 − $19. 73 = $509. 80 for males and a prediction of $529. 53 for females. Alternatively, instead of a 0 ∕ 1 coding scheme, we could create a dummy variable

$$\displaystyle{x_{i} = \left \{\begin{array}{@{}l@{\quad }l@{}} 1 \quad & \;\;\; \mbox{if $i$th person is female}\\ -1\quad & \;\;\;\mbox{if $i$th person is male} \end{array} \right .}$$

and use this variable in the regression equation. This results in the model

$$\displaystyle{y_{i} =\beta _{0}+\beta _{1}x_{i}+\epsilon _{i} = \left \{\begin{array}{@{}l@{\quad }l@{}} \beta _{0} +\beta _{1} +\epsilon _{i}\quad & \;\;\;\mbox{if $i$th person is female} \\ \beta _{0} -\beta _{1} +\epsilon _{i}\quad & \;\;\;\mbox{if $i$th person is male.} \end{array} \right .}$$

Now β ₀ can be interpreted as the overall average credit card balance (ignoring the gender effect), and β ₁ is the amount that females are above the average and males are below the average. In this example, the estimate for β ₀ would be $519. 665, halfway between the male and female averages of $509. 80 and $529. 53. The estimate for β ₁ would be $9. 865, which is half of $19. 73, the average difference between females and males. It is important to note that the final predictions for the credit balances of males and females will be identical regardless of the coding scheme used. The only difference is in the way that the coefficients are interpreted.

3.3.1.2 Qualitative Predictors with More than Two Levels

When a qualitative predictor has more than two levels, a single dummy variable cannot represent all possible values. In this situation, we can create additional dummy variables. For example, for the ethnicity variable we create two dummy variables. The first could be

$$\displaystyle{ x_{i1} = \left \{\begin{array}{@{}l@{\quad }l@{}} 1\quad & \;\;\;\mbox{if $i$th person is Asian}\\ 0\quad & \;\;\;\mbox{if $i$th person is not Asian}, \end{array} \right . }$$

(3.28)

and the second could be

$$\displaystyle{ x_{i2} = \left \{\begin{array}{@{}l@{\quad }l@{}} 1\quad & \;\;\; \mbox{if $i$th person is Caucasian}\\ 0\quad & \;\;\;\mbox{if $i$th person is not Caucasian}. \end{array} \right . }$$

(3.29)

Then both of these variables can be used in the regression equation, in order to obtain the model

$$\displaystyle{ y_{i} =\beta _{0}+\beta _{1}x_{i1}+\beta _{2}x_{i2}+\epsilon _{i} = \left \{\begin{array}{@{}l@{\quad }l@{}} \beta _{0}+\beta _{1}+\epsilon _{i}\quad &\mbox{ if $i$th person is Asian} \\ \beta _{0}+\beta _{2}+\epsilon _{i}\quad &\mbox{ if $i$th person is Caucasian} \\ \beta _{0}+\epsilon _{i} \quad &\mbox{ if $i$th person is African American}. \end{array} \right . }$$

(3.30)

Now β ₀ can be interpreted as the average credit card balance for African Americans, β ₁ can be interpreted as the difference in the average balance between the Asian and African American categories, and β ₂ can be interpreted as the difference in the average balance between the Caucasian and African American categories. There will always be one fewer dummy variable than the number of levels. The level with no dummy variable—African American in this example—is known as the baseline.

baseline

From Table 3.8, we see that the estimated balance for the baseline, African American, is $531. 00. It is estimated that the Asian category will have $18. 69 less debt than the African American category, and that the Caucasian category will have $12. 50 less debt than the African American category. However, the p-values associated with the coefficient estimates for the two dummy variables are very large, suggesting no statistical evidence of a real difference in credit card balance between the ethnicities. Once again, the level selected as the baseline category is arbitrary, and the final predictions for each group will be the same regardless of this choice. However, the coefficients and their p-values do depend on the choice of dummy variable coding. Rather than rely on the individual coefficients, we can use an F-test to test $H_{0} :\beta _{1} =\beta _{2} = 0$; this does not depend on the coding. This F-test has a p-value of 0. 96, indicating that we cannot reject the null hypothesis that there is no relationship between balance and ethnicity.

Table 3.8 Least squares coefficient estimates associated with the regression of balance onto ethnicity in the Credit data set. The linear model is given in ( 3.30 ). That is, ethnicity is encoded via two dummy variables ( 3.28 ) and (3.29).

Using this dummy variable approach presents no difficulties when incorporating both quantitative and qualitative predictors. For example, to regress balance on both a quantitative variable such as income and a qualitative variable such as student, we must simply create a dummy variable for student and then fit a multiple regression model using income and the dummy variable as predictors for credit card balance.

There are many different ways of coding qualitative variables besides the dummy variable approach taken here. All of these approaches lead to equivalent model fits, but the coefficients are different and have different interpretations, and are designed to measure particular contrasts. This topic is beyond the scope of the book, and so we will not pursue it further.

contrast

3.3.2 Extensions of the Linear Model

The standard linear regression model (3.19) provides interpretable results and works quite well on many real-world problems. However, it makes several highly restrictive assumptions that are often violated in practice. Two of the most important assumptions state that the relationship between the predictors and response are additive and linear. The additive assumption means that the effect of changes in a predictor X _j on the response Y is independent of the values of the other predictors. The linear assumption states that the change in the response Y due to a one-unit change in X _j is constant, regardless of the value of X _j . In this book, we examine a number of sophisticated methods that relax these two assumptions. Here, we briefly examine some common classical approaches for extending the linear model.

additive

linear

3.3.2.1 Removing the Additive Assumption

In our previous analysis of the Advertising data, we concluded that both TV and radio seem to be associated with sales. The linear models that formed the basis for this conclusion assumed that the effect on sales of increasing one advertising medium is independent of the amount spent on the other media. For example, the linear model (3.20) states that the average effect on sales of a one-unit increase in TV is always β ₁, regardless of the amount spent on radio.

However, this simple model may be incorrect. Suppose that spending money on radio advertising actually increases the effectiveness of TV advertising, so that the slope term for TV should increase as radio increases. In this situation, given a fixed budget of $100, 000, spending half on radio and half on TV may increase sales more than allocating the entire amount to either TV or to radio. In marketing, this is known as a synergy effect, and in statistics it is referred to as an interaction effect. Figure 3.5 suggests that such an effect may be present in the advertising data. Notice that when levels of either TV or radio are low, then the true sales are lower than predicted by the linear model. But when advertising is split between the two media, then the model tends to underestimate sales.

Consider the standard linear regression model with two variables,

$$\displaystyle{Y =\beta _{0} +\beta _{1}X_{1} +\beta _{2}X_{2} +\epsilon .}$$

According to this model, if we increase X ₁ by one unit, then Y will increase by an average of β ₁ units. Notice that the presence of X ₂ does not alter this statement—that is, regardless of the value of X ₂, a one-unit increase in X ₁ will lead to a β ₁-unit increase in Y . One way of extending this model to allow for interaction effects is to include a third predictor, called an interaction term, which is constructed by computing the product of X ₁ and X ₂. This results in the model

$$\displaystyle{ Y =\beta _{0} +\beta _{1}X_{1} +\beta _{2}X_{2} +\beta _{3}X_{1}X_{2} +\epsilon . }$$

(3.31)

How does inclusion of this interaction term relax the additive assumption? Notice that (3.31) can be rewritten as

$$\displaystyle\begin{array}{rcl} Y & =& \beta _{0} + (\beta _{1} +\beta _{3}X_{2})X_{1} +\beta _{2}X_{2} +\epsilon \\ & =& \beta _{0} +\tilde{\beta } _{1}X_{1} +\beta _{2}X_{2} +\epsilon {}\end{array}$$

(3.32)

where $\tilde{\beta }_{1} =\beta _{1} +\beta _{3}X_{2}$. Since $\tilde{\beta }_{1}$ changes with X ₂, the effect of X ₁ on Y is no longer constant: adjusting X ₂ will change the impact of X ₁ on Y .

For example, suppose that we are interested in studying the productivity of a factory. We wish to predict the number of units produced on the basis of the number of production lines and the total number of workers. It seems likely that the effect of increasing the number of production lines will depend on the number of workers, since if no workers are available to operate the lines, then increasing the number of lines will not increase production. This suggests that it would be appropriate to include an interaction term between lines and workers in a linear model to predict units. Suppose that when we fit the model, we obtain

$$\displaystyle\begin{array}{rcl} \mathtt{units}& \approx & 1.2 + 3.4 \times \mathtt{lines}\ + 0.22 \times \mathtt{workers}\ + 1.4 \times (\mathtt{lines}\ \times \mathtt{workers}\ ) {}\\ & =& 1.2 + (3.4 + 1.4 \times \mathtt{workers}\ ) \times \mathtt{lines}\ + 0.22 \times \mathtt{workers}\ . {}\\ \end{array}$$

In other words, adding an additional line will increase the number of units produced by 3. 4 + 1. 4 ×workers . Hence the more workers we have, the stronger will be the effect of lines.

We now return to the Advertising example. A linear model that uses radio, TV, and an interaction between the two to predict sales takes the form

$$\displaystyle\begin{array}{rcl} \mathtt{sales}\ & =& \beta _{0} +\beta _{1} \times \mathtt{TV}\ +\beta _{2} \times \mathtt{radio}\ +\beta _{3} \times (\mathtt{radio}\ \times \mathtt{TV}\ ) +\epsilon \\ & =& \beta _{0} + (\beta _{1} +\beta _{3} \times \mathtt{radio}\ ) \times \mathtt{TV}\ +\beta _{2} \times \mathtt{radio}\ +\epsilon . {}\end{array}$$

(3.33)

We can interpret β ₃ as the increase in the effectiveness of TV advertising for a one unit increase in radio advertising (or vice-versa). The coefficients that result from fitting the model (3.33) are given in Table 3.9.

Table 3.9 For the Advertising data, least squares coefficient estimates associated with the regression of sales onto TV and radio , with an interaction term, as in ( 3.33 ).

The results in Table 3.9 strongly suggest that the model that includes the interaction term is superior to the model that contains only main effects. The p-value for the interaction term, TV ×radio, is extremely low, indicating that there is strong evidence for H _a : β ₃ ≠ 0. In other words, it is clear that the true relationship is not additive. The R ² for the model (3.33) is 96.8 %, compared to only 89.7 % for the model that predicts sales using TV and radio without an interaction term. This means that $(96.8 - 89.7)/(100 - 89.7) = 69$ % of the variability in sales that remains after fitting the additive model has been explained by the interaction term. The coefficient estimates in Table 3.9 suggest that an increase in TV advertising of $1, 000 is associated with increased sales of $(\hat{\beta }_{1} +\hat{\beta } _{3} \times \mathtt{radio}\ ) \times 1,000 = 19 + 1.1 \times \mathtt{radio}$ units. And an increase in radio advertising of $1, 000 will be associated with an increase in sales of $(\hat{\beta }_{2} +\hat{\beta } _{3} \times \mathtt{TV}\ ) \times 1,000 = 29 + 1.1 \times \mathtt{TV}$ units.

main effect

In this example, the p-values associated with TV, radio, and the interaction term all are statistically significant (Table 3.9), and so it is obvious that all three variables should be included in the model. However, it is sometimes the case that an interaction term has a very small p-value, but the associated main effects (in this case, TV and radio) do not. The hierarchical principle states that if we include an interaction in a model, we should also include the main effects, even if the p-values associated with their coefficients are not significant. In other words, if the interaction between X ₁ and X ₂ seems important, then we should include both X ₁ and X ₂ in the model even if their coefficient estimates have large p-values. The rationale for this principle is that if X ₁ ×X ₂ is related to the response, then whether or not the coefficients of X ₁ or X ₂ are exactly zero is of little interest. Also X ₁ ×X ₂ is typically correlated with X ₁ and X ₂, and so leaving them out tends to alter the meaning of the interaction.

hierarchical principle

In the previous example, we considered an interaction between TV and radio, both of which are quantitative variables. However, the concept of interactions applies just as well to qualitative variables, or to a combination of quantitative and qualitative variables. In fact, an interaction between a qualitative variable and a quantitative variable has a particularly nice interpretation. Consider the Credit data set from Section 3.3.1, and suppose that we wish to predict balance using the income (quantitative) and student (qualitative) variables. In the absence of an interaction term, the model takes the form

$$\displaystyle\begin{array}{rcl} \mathtt{balance}\ _{i}& \approx & \beta _{0} +\beta _{1} \times \mathtt{income}\ _{i} + \left \{\begin{array}{@{}l@{\quad }l@{}} \beta _{2}\quad & \;\; \mbox{if $i$th person is a student} \\ 0\quad & \;\;\;\mbox{if $i$th person is not a student} \end{array} \right . \\ & =& \beta _{1} \times \mathtt{income}\ _{i} + \left \{\begin{array}{@{}l@{\quad }l@{}} \beta _{0} +\beta _{2}\quad & \;\;\;\mbox{if $i$th person is a student} \\ \beta _{0} \quad & \;\;\;\mbox{if $i$th person is not a student}. \end{array} \right .{}\end{array}$$

(3.34)

Notice that this amounts to fitting two parallel lines to the data, one for students and one for non-students. The lines for students and non-students have different intercepts, β ₀ + β ₂ versus β ₀, but the same slope, β ₁. This is illustrated in the left-hand panel of Figure 3.7. The fact that the lines are parallel means that the average effect on balance of a one-unit increase in income does not depend on whether or not the individual is a student. This represents a potentially serious limitation of the model, since in fact a change in income may have a very different effect on the credit card balance of a student versus a non-student.

Fig. 3.7

For the Credit data, the least squares lines are shown for prediction of balance from income for students and non-students. Left: The model ( 3.34 ) was fit. There is no interaction between income and student . Right: The model ( 3.35 ) was fit. There is an interaction term between income and student .

This limitation can be addressed by adding an interaction variable, created by multiplying income with the dummy variable for student. Our model now becomes

$$\displaystyle\begin{array}{rcl} \mathtt{balance}\ _{i}& \approx & \beta _{0} +\beta _{1} \times \mathtt{income}\ _{i} + \left \{\begin{array}{@{}l@{\quad }l@{}} \beta _{2} +\beta _{3} \times \mathtt{income}\ _{i}\quad &\mbox{ if student}\\ 0 \quad &\mbox{ if not student} \end{array} \right . \\ & =& \left \{\begin{array}{@{}l@{\quad }l@{}} (\beta _{0} +\beta _{2}) + (\beta _{1} +\beta _{3}) \times \mathtt{income}\ _{i}\quad &\mbox{ if student} \\ \beta _{0} +\beta _{1} \times \mathtt{income}\ _{i} \quad &\mbox{ if not student} \end{array} \right . {}\end{array}$$

(3.35)

Once again, we have two different regression lines for the students and the non-students. But now those regression lines have different intercepts, β ₀ + β ₂ versus β ₀, as well as different slopes, β ₁ + β ₃ versus β ₁. This allows for the possibility that changes in income may affect the credit card balances of students and non-students differently. The right-hand panel of Figure 3.7 shows the estimated relationships between income and balance for students and non-students in the model (3.35). We note that the slope for students is lower than the slope for non-students. This suggests that increases in income are associated with smaller increases in credit card balance among students as compared to non-students.

3.3.2.2 Non-linear Relationships

As discussed previously, the linear regression model (3.19) assumes a linear relationship between the response and predictors. But in some cases, the true relationship between the response and the predictors may be non-linear. Here we present a very simple way to directly extend the linear model to accommodate non-linear relationships, using polynomial regression. In later chapters, we will present more complex approaches for performing non-linear fits in more general settings.

polynomial regression

quadratic

Consider Figure 3.8, in which the mpg (gas mileage in miles per gallon) versus horsepower is shown for a number of cars in the Auto data set. The orange line represents the linear regression fit. There is a pronounced relationship between mpg and horsepower, but it seems clear that this relationship is in fact non-linear: the data suggest a curved relationship. A simple approach for incorporating non-linear associations in a linear model is to include transformed versions of the predictors in the model. For example, the points in Figure 3.8 seem to have a quadratic shape, suggesting that a model of the form

$$\displaystyle{ \mathtt{mpg}\ =\beta _{0} +\beta _{1} \times \mathtt{horsepower}\ +\beta _{2} \times {\mathtt{horsepowerp}\ }^{2}+\epsilon }$$

(3.36)

may provide a better fit. Equation 3.36 involves predicting mpg using a non-linear function of horsepower. But it is still a linear model! That is, (3.36) is simply a multiple linear regression model with X ₁ = horsepower and X ₂ = horsepower2. So we can use standard linear regression software to estimate β ₀, β ₁, and β ₂ in order to produce a non-linear fit. The blue curve in Figure 3.8 shows the resulting quadratic fit to the data. The quadratic fit appears to be substantially better than the fit obtained when just the linear term is included. The R ² of the quadratic fit is 0. 688, compared to 0. 606 for the linear fit, and the p-value in Table 3.10 for the quadratic term is highly significant.

Fig. 3.8

The Auto data set. For a number of cars, mpg and horsepower are shown. The linear regression fit is shown in orange. The linear regression fit for a model that includes horsepower2 is shown as a blue curve. The linear regression fit for a model that includes all polynomials of horsepower up to fifth-degree is shown in green.

Table 3.10 For the Auto data set, least squares coefficient estimates associated with the regression of mpg onto horsepower  and horsepower2 .

If including horsepower2 led to such a big improvement in the model, why not include horsepower3, horsepower4, or even horsepower5? The green curve in Figure 3.8 displays the fit that results from including all polynomials up to fifth degree in the model (3.36). The resulting fit seems unnecessarily wiggly—that is, it is unclear that including the additional terms really has led to a better fit to the data.

The approach that we have just described for extending the linear model to accommodate non-linear relationships is known as polynomial regression, since we have included polynomial functions of the predictors in the regression model. We further explore this approach and other non-linear extensions of the linear model in Chapter 7.

3.3.3 Potential Problems

When we fit a linear regression model to a particular data set, many problems may occur. Most common among these are the following:

• 1. Non-linearity of the response-predictor relationships.

• 2. Correlation of error terms.

• 3. Non-constant variance of error terms.

• 4. Outliers.

• 5. High-leverage points.

• 6. Collinearity.

In practice, identifying and overcoming these problems is as much an art as a science. Many pages in countless books have been written on this topic. Since the linear regression model is not our primary focus here, we will provide only a brief summary of some key points.

3.3.3 1. Non-linearity of the Data

The linear regression model assumes that there is a straight-line relationship between the predictors and the response. If the true relationship is far from linear, then virtually all of the conclusions that we draw from the fit are suspect. In addition, the prediction accuracy of the model can be significantly reduced.

Residual plots are a useful graphical tool for identifying non-linearity. Given a simple linear regression model, we can plot the residuals, $e_{i} = y_{i} -\hat{ y}_{i}$, versus the predictor x _i . In the case of a multiple regression model, since there are multiple predictors, we instead plot the residuals versus the predicted (or fitted) values $\hat{y}_{i}$. Ideally, the residual plot will show no discernible pattern. The presence of a pattern may indicate a problem with some aspect of the linear model.

residual plot

fitted

The left panel of Figure 3.9 displays a residual plot from the linear regression of mpg onto horsepower on the Auto data set that was illustrated in Figure 3.8. The red line is a smooth fit to the residuals, which is displayed in order to make it easier to identify any trends. The residuals exhibit a clear U-shape, which provides a strong indication of non-linearity in the data. In contrast, the right-hand panel of Figure 3.9 displays the residual plot that results from the model (3.36), which contains a quadratic term. There appears to be little pattern in the residuals, suggesting that the quadratic term improves the fit to the data.

Fig. 3.9

Plots of residuals versus predicted (or fitted) values for the Auto data set. In each plot, the red line is a smooth fit to the residuals, intended to make it easier to identify a trend. Left: A linear regression of mpg on horsepower . A strong pattern in the residuals indicates non-linearity in the data. Right: A linear regression of mpg on horsepower and horsepower2 . There is little pattern in the residuals.

If the residual plot indicates that there are non-linear associations in the data, then a simple approach is to use non-linear transformations of the predictors, such as logX, $\sqrt{ X}$, and X ², in the regression model. In the later chapters of this book, we will discuss other more advanced non-linear approaches for addressing this issue.

3.3.3 2. Correlation of Error Terms

An important assumption of the linear regression model is that the error terms, ε ₁, ε ₂, …, ε _n , are uncorrelated. What does this mean? For instance, if the errors are uncorrelated, then the fact that ε _i is positive provides little or no information about the sign of ε _i + 1. The standard errors that are computed for the estimated regression coefficients or the fitted values are based on the assumption of uncorrelated error terms. If in fact there is correlation among the error terms, then the estimated standard errors will tend to underestimate the true standard errors. As a result, confidence and prediction intervals will be narrower than they should be. For example, a 95 % confidence interval may in reality have a much lower probability than 0. 95 of containing the true value of the parameter. In addition, p-values associated with the model will be lower than they should be; this could cause us to erroneously conclude that a parameter is statistically significant. In short, if the error terms are correlated, we may have an unwarranted sense of confidence in our model.

As an extreme example, suppose we accidentally doubled our data, leading to observations and error terms identical in pairs. If we ignored this, our standard error calculations would be as if we had a sample of size 2n, when in fact we have only n samples. Our estimated parameters would be the same for the 2n samples as for the n samples, but the confidence intervals would be narrower by a factor of $\sqrt{ 2}$!

Why might correlations among the error terms occur? Such correlations frequently occur in the context of time series data, which consists of observations for which measurements are obtained at discrete points in time. In many cases, observations that are obtained at adjacent time points will have positively correlated errors. In order to determine if this is the case for a given data set, we can plot the residuals from our model as a function of time. If the errors are uncorrelated, then there should be no discernible pattern. On the other hand, if the error terms are positively correlated, then we may see tracking in the residuals—that is, adjacent residuals may have similar values. Figure 3.10 provides an illustration. In the top panel, we see the residuals from a linear regression fit to data generated with uncorrelated errors. There is no evidence of a time-related trend in the residuals. In contrast, the residuals in the bottom panel are from a data set in which adjacent errors had a correlation of 0. 9. Now there is a clear pattern in the residuals—adjacent residuals tend to take on similar values. Finally, the center panel illustrates a more moderate case in which the residuals had a correlation of 0. 5. There is still evidence of tracking, but the pattern is less clear.

Fig. 3.10

Plots of residuals from simulated time series data sets generated with differing levels of correlation ρ between error terms for adjacent time points.

time series

tracking

Many methods have been developed to properly take account of correlations in the error terms in time series data. Correlation among the error terms can also occur outside of time series data. For instance, consider a study in which individuals’ heights are predicted from their weights. The assumption of uncorrelated errors could be violated if some of the individuals in the study are members of the same family, or eat the same diet, or have been exposed to the same environmental factors. In general, the assumption of uncorrelated errors is extremely important for linear regression as well as for other statistical methods, and good experimental design is crucial in order to mitigate the risk of such correlations.

3.3.3 3. Non-constant Variance of Error Terms

Another important assumption of the linear regression model is that the error terms have a constant variance, Var(ε _i ) = σ ². The standard errors, confidence intervals, and hypothesis tests associated with the linear model rely upon this assumption.

Unfortunately, it is often the case that the variances of the error terms are non-constant. For instance, the variances of the error terms may increase with the value of the response. One can identify non-constant variances in the errors, or heteroscedasticity, from the presence of a funnel shape in the residual plot. An example is shown in the left-hand panel of Figure 3.11, in which the magnitude of the residuals tends to increase with the fitted values. When faced with this problem, one possible solution is to transform the response Y using a concave function such as logY or $\sqrt{Y}$. Such a transformation results in a greater amount of shrinkage of the larger responses, leading to a reduction in heteroscedasticity. The right-hand panel of Figure 3.11 displays the residual plot after transforming the response using logY . The residuals now appear to have constant variance, though there is some evidence of a slight non-linear relationship in the data.

Fig. 3.11

Residual plots. In each plot, the red line is a smooth fit to the residuals, intended to make it easier to identify a trend. The blue lines track the outer quantiles of the residuals, and emphasize patterns. Left: The funnel shape indicates heteroscedasticity. Right: The response has been log transformed, and there is now no evidence of heteroscedasticity.

heteroscedasticity

Sometimes we have a good idea of the variance of each response. For example, the ith response could be an average of n _i raw observations. If each of these raw observations is uncorrelated with variance σ ², then their average has variance $\sigma _{i}^{2} {=\sigma }^{2}/n_{i}$. In this case a simple remedy is to fit our model by weighted least squares, with weights proportional to the inverse variances—i.e. w _i  = n _i in this case. Most linear regression software allows for observation weights.

weighted least squares

3.3.3 4. Outliers

An outlier is a point for which y _i is far from the value predicted by the model. Outliers can arise for a variety of reasons, such as incorrect recording of an observation during data collection.

outlier

The red point (observation 20) in the left-hand panel of Figure 3.12 illustrates a typical outlier. The red solid line is the least squares regression fit, while the blue dashed line is the least squares fit after removal of the outlier. In this case, removing the outlier has little effect on the least squares line: it leads to almost no change in the slope, and a miniscule reduction in the intercept. It is typical for an outlier that does not have an unusual predictor value to have little effect on the least squares fit. However, even if an outlier does not have much effect on the least squares fit, it can cause other problems. For instance, in this example, the RSE is 1. 09 when the outlier is included in the regression, but it is only 0. 77 when the outlier is removed. Since the RSE is used to compute all confidence intervals and p-values, such a dramatic increase caused by a single data point can have implications for the interpretation of the fit. Similarly, inclusion of the outlier causes the R ² to decline from 0. 892 to 0. 805.

Fig. 3.12

Left: The least squares regression line is shown in red, and the regression line after removing the outlier is shown in blue. Center: The residual plot clearly identifies the outlier. Right: The outlier has a studentized residual of 6; typically we expect values between − 3 and 3.

Residual plots can be used to identify outliers. In this example, the outlier is clearly visible in the residual plot illustrated in the center panel of Figure 3.12. But in practice, it can be difficult to decide how large a residual needs to be before we consider the point to be an outlier. To address this problem, instead of plotting the residuals, we can plot the studentized residuals, computed by dividing each residual e _i by its estimated standard error. Observations whose studentized residuals are greater than 3 in absolute value are possible outliers. In the right-hand panel of Figure 3.12, the outlier’s studentized residual exceeds 6, while all other observations have studentized residuals between − 2 and 2.

studentized residual

If we believe that an outlier has occurred due to an error in data collection or recording, then one solution is to simply remove the observation. However, care should be taken, since an outlier may instead indicate a deficiency with the model, such as a missing predictor.

3.3.3 5. High Leverage Points

We just saw that outliers are observations for which the response y _i is unusual given the predictor x _i . In contrast, observations with high leverage have an unusual value for x _i . For example, observation 41 in the left-hand panel of Figure 3.13 has high leverage, in that the predictor value for this observation is large relative to the other observations. (Note that the data displayed in Figure 3.13 are the same as the data displayed in Figure 3.12, but with the addition of a single high leverage observation.) The red solid line is the least squares fit to the data, while the blue dashed line is the fit produced when observation 41 is removed. Comparing the left-hand panels of Figures 3.12 and 3.13, we observe that removing the high leverage observation has a much more substantial impact on the least squares line than removing the outlier. In fact, high leverage observations tend to have a sizable impact on the estimated regression line. It is cause for concern if the least squares line is heavily affected by just a couple of observations, because any problems with these points may invalidate the entire fit. For this reason, it is important to identify high leverage observations.

Fig. 3.13

Left: Observation 41 is a high leverage point, while 20 is not. The red line is the fit to all the data, and the blue line is the fit with observation 41 removed. Center: The red observation is not unusual in terms of its X ₁ value or its X ₂ value, but still falls outside the bulk of the data, and hence has high leverage. Right: Observation 41 has a high leverage and a high residual.

high leverage

In a simple linear regression, high leverage observations are fairly easy to identify, since we can simply look for observations for which the predictor value is outside of the normal range of the observations. But in a multiple linear regression with many predictors, it is possible to have an observation that is well within the range of each individual predictor’s values, but that is unusual in terms of the full set of predictors. An example is shown in the center panel of Figure 3.13, for a data set with two predictors, X ₁ and X ₂. Most of the observations’ predictor values fall within the blue dashed ellipse, but the red observation is well outside of this range. But neither its value for X ₁ nor its value for X ₂ is unusual. So if we examine just X ₁ or just X ₂, we will fail to notice this high leverage point. This problem is more pronounced in multiple regression settings with more than two predictors, because then there is no simple way to plot all dimensions of the data simultaneously.

In order to quantify an observation’s leverage, we compute the leverage statistic. A large value of this statistic indicates an observation with high leverage. For a simple linear regression,

$$\displaystyle{ h_{i} = \frac{1} {n} + \frac{{(x_{i} -\bar{ x})}^{2}} {\sum _{i^\prime=1}^{n}{(x_{i^\prime} -\bar{ x})}^{2}}. }$$

(3.37)

It is clear from this equation that h _i increases with the distance of x _i from $\bar{x}$. There is a simple extension of h _i to the case of multiple predictors, though we do not provide the formula here. The leverage statistic h _i is always between 1 ∕ n and 1, and the average leverage for all the observations is always equal to $(p + 1)/n$. So if a given observation has a leverage statistic that greatly exceeds $(p + 1)/n$, then we may suspect that the corresponding point has high leverage.

leverage statistic

The right-hand panel of Figure 3.13 provides a plot of the studentized residuals versus h _i for the data in the left-hand panel of Figure 3.13. Observation 41 stands out as having a very high leverage statistic as well as a high studentized residual. In other words, it is an outlier as well as a high leverage observation. This is a particularly dangerous combination! This plot also reveals the reason that observation 20 had relatively little effect on the least squares fit in Figure 3.12: it has low leverage.

3.3.3 6. Collinearity

Collinearity refers to the situation in which two or more predictor variables are closely related to one another. The concept of collinearity is illustrated in Figure 3.14 using the Credit data set. In the left-hand panel of Figure 3.14, the two predictors limit and age appear to have no obvious relationship. In contrast, in the right-hand panel of Figure 3.14, the predictors limit and rating are very highly correlated with each other, and we say that they are collinear. The presence of collinearity can pose problems in the regression context, since it can be difficult to separate out the individual effects of collinear variables on the response. In other words, since limit and rating tend to increase or decrease together, it can be difficult to determine how each one separately is associated with the response, balance.

Fig. 3.14

Scatterplots of the observations from the Credit data set. Left: A plot of age versus limit . These two variables are not collinear. Right: A plot of rating versus limit . There is high collinearity.

collinearity

Figure 3.15 illustrates some of the difficulties that can result from collinearity. The left-hand panel of Figure 3.15 is a contour plot of the RSS (3.22) associated with different possible coefficient estimates for the regression of balance on limit and age. Each ellipse represents a set of coefficients that correspond to the same RSS, with ellipses nearest to the center taking on the lowest values of RSS. The black dots and associated dashed lines represent the coefficient estimates that result in the smallest possible RSS—in other words, these are the least squares estimates. The axes for limit and age have been scaled so that the plot includes possible coefficient estimates that are up to four standard errors on either side of the least squares estimates. Thus the plot includes all plausible values for the coefficients. For example, we see that the true limit coefficient is almost certainly somewhere between 0. 15 and 0. 20.

Fig. 3.15

Contour plots for the RSS values as a function of the parameters β for various regressions involving the Credit data set. In each plot, the black dots represent the coefficient values corresponding to the minimum RSS. Left: A contour plot of RSS for the regression of balance onto age and limit . The minimum value is well defined. Right: A contour plot of RSS for the regression of balance onto rating and limit . Because of the collinearity, there are many pairs (β _Limit ,β _Rating ) with a similar value for RSS.

In contrast, the right-hand panel of Figure 3.15 displays contour plots of the RSS associated with possible coefficient estimates for the regression of balance onto limit and rating, which we know to be highly collinear. Now the contours run along a narrow valley; there is a broad range of values for the coefficient estimates that result in equal values for RSS. Hence a small change in the data could cause the pair of coefficient values that yield the smallest RSS—that is, the least squares estimates—to move anywhere along this valley. This results in a great deal of uncertainty in the coefficient estimates. Notice that the scale for the limit coefficient now runs from roughly − 0. 2 to 0. 2; this is an eight-fold increase over the plausible range of the limit coefficient in the regression with age. Interestingly, even though the limit and rating coefficients now have much more individual uncertainty, they will almost certainly lie somewhere in this contour valley. For example, we would not expect the true value of the limit and rating coefficients to be − 0. 1 and 1 respectively, even though such a value is plausible for each coefficient individually.

Since collinearity reduces the accuracy of the estimates of the regression coefficients, it causes the standard error for $\hat{\beta }_{j}$ to grow. Recall that the t-statistic for each predictor is calculated by dividing $\hat{\beta }_{j}$ by its standard error. Consequently, collinearity results in a decline in the t-statistic. As a result, in the presence of collinearity, we may fail to reject H ₀ : β _j  = 0. This means that the power of the hypothesis test—the probability of correctly detecting a non-zero coefficient—is reduced by collinearity.

power

Table 3.11 compares the coefficient estimates obtained from two separate multiple regression models. The first is a regression of balance on age and limit, and the second is a regression of balance on rating and limit. In the first regression, both age and limit are highly significant with very small p-values. In the second, the collinearity between limit and rating has caused the standard error for the limit coefficient estimate to increase by a factor of 12 and the p-value to increase to 0. 701. In other words, the importance of the limit variable has been masked due to the presence of collinearity. To avoid such a situation, it is desirable to identify and address potential collinearity problems while fitting the model.

Table 3.11 The results for two multiple regression models involving the Credit data set are shown. Model 1 is a regression of balance on age and limit , and Model 2 a regression of balance on rating and limit . The standard error of $\hat{\beta }_{\mathsf{limit}}$ increases 12-fold in the second regression, due to collinearity.

A simple way to detect collinearity is to look at the correlation matrix of the predictors. An element of this matrix that is large in absolute value indicates a pair of highly correlated variables, and therefore a collinearity problem in the data. Unfortunately, not all collinearity problems can be detected by inspection of the correlation matrix: it is possible for collinearity to exist between three or more variables even if no pair of variables has a particularly high correlation. We call this situation multicollinearity. Instead of inspecting the correlation matrix, a better way to assess multicollinearity is to compute the variance inflation factor (VIF). The VIF is the ratio of the variance of $\hat{\beta }_{j}$ when fitting the full model divided by the variance of $\hat{\beta }_{j}$ if fit on its own. The smallest possible value for VIF is 1, which indicates the complete absence of collinearity. Typically in practice there is a small amount of collinearity among the predictors. As a rule of thumb, a VIF value that exceeds 5 or 10 indicates a problematic amount of collinearity. The VIF for each variable can be computed using the formula

$$\displaystyle{\rm {VIF}(\hat{\beta }_{j}) = \frac{1} {1 - R_{X_{j}\vert X_{-j}}^{2}},}$$

where $R_{X_{j}\vert X_{-j}}^{2}$ is the R ² from a regression of X _j onto all of the other predictors. If $R_{X_{j}\vert X_{-j}}^{2}$ is close to one, then collinearity is present, and so the VIF will be large.

multicollinearity

variance inflation factor

In the Credit data, a regression of balance on age, rating, and limit indicates that the predictors have VIF values of 1.01, 160.67, and 160.59. As we suspected, there is considerable collinearity in the data!

When faced with the problem of collinearity, there are two simple solutions. The first is to drop one of the problematic variables from the regression. This can usually be done without much compromise to the regression fit, since the presence of collinearity implies that the information that this variable provides about the response is redundant in the presence of the other variables. For instance, if we regress balance onto age and limit, without the rating predictor, then the resulting VIF values are close to the minimum possible value of 1, and the R ² drops from 0. 754 to 0. 75. So dropping rating from the set of predictors has effectively solved the collinearity problem without compromising the fit. The second solution is to combine the collinear variables together into a single predictor. For instance, we might take the average of standardized versions of limit and rating in order to create a new variable that measures credit worthiness.

3.4 The Marketing Plan

We now briefly return to the seven questions about the Advertising data that we set out to answer at the beginning of this chapter.

• 1. Is there a relationship between advertising sales and budget?

  This question can be answered by fitting a multiple regression model of sales onto TV, radio, and newspaper, as in (3.20), and testing the hypothesis $H_{0} :\beta _{\mathtt{TV}\ } =\beta _{\mathtt{radio}\ } =\beta _{\mathtt{newspaper}\ } = 0$. In Section 3.2.2, we showed that the F-statistic can be used to determine whether or not we should reject this null hypothesis. In this case the p-value corresponding to the F-statistic in Table 3.6 is very low, indicating clear evidence of a relationship between advertising and sales.

• 2. How strong is the relationship?

  We discussed two measures of model accuracy in Section 3.1.3. First, the RSE estimates the standard deviation of the response from the population regression line. For the Advertising data, the RSE is 1, 681 units while the mean value for the response is 14, 022, indicating a percentage error of roughly 12 %. Second, the R ² statistic records the percentage of variability in the response that is explained by the predictors. The predictors explain almost 90 % of the variance in sales. The RSE and R ² statistics are displayed in Table 3.6.

• 3. Which media contribute to sales?

  To answer this question, we can examine the p-values associated with each predictor’s t-statistic (Section 3.1.2). In the multiple linear regression displayed in Table 3.4, the p-values for TV and radio are low, but the p-value for newspaper is not. This suggests that only TV and radio are related to sales. In Chapter 6 we explore this question in greater detail.

• 4. How large is the effect of each medium on sales?

  We saw in Section 3.1.2 that the standard error of $\hat{\beta }_{j}$ can be used to construct confidence intervals for β _j . For the Advertising data, the 95 % confidence intervals are as follows: (0. 043, 0. 049) for TV, (0. 172, 0. 206) for radio, and ( − 0. 013, 0. 011) for newspaper. The confidence intervals for TV and radio are narrow and far from zero, providing evidence that these media are related to sales. But the interval for newspaper includes zero, indicating that the variable is not statistically significant given the values of TV and radio.

  We saw in Section 3.3.3 that collinearity can result in very wide standard errors. Could collinearity be the reason that the confidence interval associated with newspaper is so wide? The VIF scores are 1. 005, 1. 145, and 1. 145 for TV, radio, and newspaper, suggesting no evidence of collinearity.

  In order to assess the association of each medium individually on sales, we can perform three separate simple linear regressions. Results are shown in Tables 3.1 and 3.3. There is evidence of an extremely strong association between TV and sales and between radio and sales. There is evidence of a mild association between newspaper and sales, when the values of TV and radio are ignored.

• 5. How accurately can we predict future sales?

  The response can be predicted using (3.21). The accuracy associated with this estimate depends on whether we wish to predict an individual response, $Y = f(X)+\epsilon$, or the average response, f(X) (Section 3.2.2). If the former, we use a prediction interval, and if the latter, we use a confidence interval. Prediction intervals will always be wider than confidence intervals because they account for the uncertainty associated with ε, the irreducible error.

• 6. Is the relationship linear?

  In Section 3.3.3, we saw that residual plots can be used in order to identify non-linearity. If the relationships are linear, then the residual plots should display no pattern. In the case of the Advertising data, we observe a non-linear effect in Figure 3.5, though this effect could also be observed in a residual plot. In Section 3.3.2, we discussed the inclusion of transformations of the predictors in the linear regression model in order to accommodate non-linear relationships.

• 7. Is there synergy among the advertising media?

  The standard linear regression model assumes an additive relationship between the predictors and the response. An additive model is easy to interpret because the effect of each predictor on the response is unrelated to the values of the other predictors. However, the additive assumption may be unrealistic for certain data sets. In Section 3.3.2, we showed how to include an interaction term in the regression model in order to accommodate non-additive relationships. A small p-value associated with the interaction term indicates the presence of such relationships. Figure 3.5 suggested that the Advertising data may not be additive. Including an interaction term in the model results in a substantial increase in R ², from around 90 % to almost 97 %.

3.5 Comparison of Linear Regression with K-Nearest Neighbors

As discussed in Chapter 2, linear regression is an example of a parametric approach because it assumes a linear functional form for f(X). Parametric methods have several advantages. They are often easy to fit, because one need estimate only a small number of coefficients. In the case of linear regression, the coefficients have simple interpretations, and tests of statistical significance can be easily performed. But parametric methods do have a disadvantage: by construction, they make strong assumptions about the form of f(X). If the specified functional form is far from the truth, and prediction accuracy is our goal, then the parametric method will perform poorly. For instance, if we assume a linear relationship between X and Y but the true relationship is far from linear, then the resulting model will provide a poor fit to the data, and any conclusions drawn from it will be suspect.

In contrast, non-parametric methods do not explicitly assume a parametric form for f(X), and thereby provide an alternative and more flexible approach for performing regression. We discuss various non-parametric methods in this book. Here we consider one of the simplest and best-known non-parametric methods, K-nearest neighbors regression (KNN regression). The KNN regression method is closely related to the KNN classifier discussed in Chapter 2. Given a value for K and a prediction point x ₀, KNN regression first identifies the K training observations that are closest to x ₀, represented by $\mathcal{N}_{0}$. It then estimates f(x ₀) using the average of all the training responses in $\mathcal{N}_{0}$. In other words,

$$\displaystyle{\hat{f}(x_{0}) = \frac{1} {K}\sum _{x_{i}\in \mathcal{N}_{0}}y_{i}.}$$

K-nearest neighbors regression

Figure 3.16 illustrates two KNN fits on a data set with p = 2 predictors. The fit with K = 1 is shown in the left-hand panel, while the right-hand panel corresponds to K = 9. We see that when K = 1, the KNN fit perfectly interpolates the training observations, and consequently takes the form of a step function. When K = 9, the KNN fit still is a step function, but averaging over nine observations results in much smaller regions of constant prediction, and consequently a smoother fit. In general, the optimal value for K will depend on the bias-variance tradeoff, which we introduced in Chapter 2. A small value for K provides the most flexible fit, which will have low bias but high variance. This variance is due to the fact that the prediction in a given region is entirely dependent on just one observation. In contrast, larger values of K provide a smoother and less variable fit; the prediction in a region is an average of several points, and so changing one observation has a smaller effect. However, the smoothing may cause bias by masking some of the structure in f(X). In Chapter 5, we introduce several approaches for estimating test error rates. These methods can be used to identify the optimal value of K in KNN regression.

Fig. 3.16

Plots of $\hat{f}(X)$ using KNN regression on a two-dimensional data set with 64 observations (orange dots). Left: K = 1 results in a rough step function fit. Right: K = 9 produces a much smoother fit.

In what setting will a parametric approach such as least squares linear regression outperform a non-parametric approach such as KNN regression? The answer is simple: the parametric approach will outperform the non-parametric approach if the parametric form that has been selected is close to the true form of f. Figure 3.17 provides an example with data generated from a one-dimensional linear regression model. The black solid lines represent f(X), while the blue curves correspond to the KNN fits using K = 1 and K = 9. In this case, the K = 1 predictions are far too variable, while the smoother K = 9 fit is much closer to f(X). However, since the true relationship is linear, it is hard for a non-parametric approach to compete with linear regression: a non-parametric approach incurs a cost in variance that is not offset by a reduction in bias. The blue dashed line in the left-hand panel of Figure 3.18 represents the linear regression fit to the same data. It is almost perfect. The right-hand panel of Figure 3.18 reveals that linear regression outperforms KNN for this data. The green solid line, plotted as a function of 1 ∕ K, represents the test set mean squared error (MSE) for KNN. The KNN errors are well above the black dashed line, which is the test MSE for linear regression. When the value of K is large, then KNN performs only a little worse than least squares regression in terms of MSE. It performs far worse when K is small.

Fig. 3.17

Plots of $\hat{f}(X)$ using KNN regression on a one-dimensional data set with 100 observations. The true relationship is given by the black solid line. Left: The blue curve corresponds to K = 1 and interpolates (i.e. passes directly through) the training data. Right: The blue curve corresponds to K = 9, and represents a smoother fit.

Fig. 3.18

The same data set shown in Figure  3.17 is investigated further. Left: The blue dashed line is the least squares fit to the data. Since f(X) is in fact linear (displayed as the black line), the least squares regression line provides a very good estimate of f(X). Right: The dashed horizontal line represents the least squares test set MSE, while the green solid line corresponds to the MSE for KNN as a function of 1∕K (on the log scale). Linear regression achieves a lower test MSE than does KNN regression, since f(X) is in fact linear. For KNN regression, the best results occur with a very large value of K, corresponding to a small value of 1∕K.

In practice, the true relationship between X and Y is rarely exactly linear. Figure 3.19 examines the relative performances of least squares regression and KNN under increasing levels of non-linearity in the relationship between X and Y . In the top row, the true relationship is nearly linear. In this case we see that the test MSE for linear regression is still superior to that of KNN for low values of K. However, for K ≥ 4, KNN outperforms linear regression. The second row illustrates a more substantial deviation from linearity. In this situation, KNN substantially outperforms linear regression for all values of K. Note that as the extent of non-linearity increases, there is little change in the test set MSE for the non-parametric KNN method, but there is a large increase in the test set MSE of linear regression.

Fig. 3.19

Top Left: In a setting with a slightly non-linear relationship between X and Y (solid black line), the KNN fits with K = 1 (blue) and K = 9 (red) are displayed. Top Right: For the slightly non-linear data, the test set MSE for least squares regression (horizontal black) and KNN with various values of 1∕K (green) are displayed. Bottom Left and Bottom Right: As in the top panel, but with a strongly non-linear relationship between X and Y .

Figures 3.18 and 3.19 display situations in which KNN performs slightly worse than linear regression when the relationship is linear, but much better than linear regression for non-linear situations. In a real life situation in which the true relationship is unknown, one might draw the conclusion that KNN should be favored over linear regression because it will at worst be slightly inferior than linear regression if the true relationship is linear, and may give substantially better results if the true relationship is non-linear. But in reality, even when the true relationship is highly non-linear, KNN may still provide inferior results to linear regression. In particular, both Figures 3.18 and 3.19 illustrate settings with p = 1 predictor. But in higher dimensions, KNN often performs worse than linear regression.

Figure 3.20 considers the same strongly non-linear situation as in the second row of Figure 3.19, except that we have added additional noise predictors that are not associated with the response. When p = 1 or p = 2, KNN outperforms linear regression. But for p = 3 the results are mixed, and for p ≥ 4 linear regression is superior to KNN. In fact, the increase in dimension has only caused a small deterioration in the linear regression test set MSE, but it has caused more than a ten-fold increase in the MSE for KNN. This decrease in performance as the dimension increases is a common problem for KNN, and results from the fact that in higher dimensions there is effectively a reduction in sample size. In this data set there are 100 training observations; when p = 1, this provides enough information to accurately estimate f(X). However, spreading 100 observations over p = 20 dimensions results in a phenomenon in which a given observation has no nearby neighbors—this is the so-called curse of dimensionality. That is, the K observations that are nearest to a given test observation x ₀ may be very far away from x ₀ in p-dimensional space when p is large, leading to a very poor prediction of f(x ₀) and hence a poor KNN fit. As a general rule, parametric methods will tend to outperform non-parametric approaches when there is a small number of observations per predictor.

Fig. 3.20

Test MSE for linear regression (black dashed lines) and KNN (green curves) as the number of variables p increases. The true function is non-linear in the first variable, as in the lower panel in Figure  3.19 , and does not depend on the additional variables. The performance of linear regression deteriorates slowly in the presence of these additional noise variables, whereas KNN’s performance degrades much more quickly as p increases.

curse of dimensionality

Even in problems in which the dimension is small, we might prefer linear regression to KNN from an interpretability standpoint. If the test MSE of KNN is only slightly lower than that of linear regression, we might be willing to forego a little bit of prediction accuracy for the sake of a simple model that can be described in terms of just a few coefficients, and for which p-values are available.

3.6 Lab: Linear Regression

3.6.1 Libraries

The library() function is used to load libraries, or groups of functions and data sets that are not included in the base R distribution. Basic functions that perform least squares linear regression and other simple analyses come standard with the base distribution, but more exotic functions require additional libraries. Here we load the MASS package, which is a very large collection of data sets and functions. We also load the ISLR package, which includes the data sets associated with this book.

library()

> library(MASS)

> library(ISLR)

If you receive an error message when loading any of these libraries, it likely indicates that the corresponding library has not yet been installed on your system. Some libraries, such as MASS, come with R and do not need to be separately installed on your computer. However, other packages, such as ISLR, must be downloaded the first time they are used. This can be done directly from within R. For example, on a Windows system, select the Install package option under the Packages tab. After you select any mirror site, a list of available packages will appear. Simply select the package you wish to install and R will automatically download the package. Alternatively, this can be done at the R command line via install.packages("ISLR"). This installation only needs to be done the first time you use a package. However, the library() function must be called each time you wish to use a given package.

3.6.2 Simple Linear Regression

The MASS library contains the Boston data set, which records medv (median house value) for 506 neighborhoods around Boston. We will seek to predict medv using 13 predictors such as rm (average number of rooms per house), age (average age of houses), and lstat (percent of households with low socioeconomic status).

> fix(Boston)

> names(Boston)

 [1] "crim"   "zn"   "indus"   "chas"    "nox"    "rm"    "age"

 [8] "dis"    "rad"  "tax"   "ptratio" "black"  "lstat"  "medv"

To find out more about the data set, we can type ?Boston.

We will start by using the lm() function to fit a simple linear regression model, with medv as the response and lstat as the predictor. The basic syntax is lm(y ∼ x,data), where y is the response, x is the predictor, and data is the data set in which these two variables are kept.

lm()

> lm.fit=lm(medv∼lstat)

Error in eval(expr, envir, enclos) : Object "medv" not found

The command causes an error because R does not know where to find the variables medv and lstat. The next line tells R that the variables are in Boston. If we attach Boston, the first line works fine because R now recognizes the variables.

> lm.fit=lm(medv∼lstat,data=Boston)

> attach(Boston)

> lm.fit=lm(medv∼lstat)

If we type lm.fit, some basic information about the model is output. For more detailed information, we use summary(lm.fit). This gives us p-values and standard errors for the coefficients, as well as the R ² statistic and F-statistic for the model.

> lm.fit

Call:

lm(formula = medv ∼ lstat)

Coefficients:

(Intercept)        lstat

      34.55        -0.95

> summary(lm.fit)

Call:

lm(formula = medv ∼ lstat)

Residuals:

   Min     1Q Median     3Q    Max

-15.17  -3.99  -1.32   2.03  24.50

Coefficients:

            Estimate Std. Error t value Pr(>|t|)

(Intercept)  34.5538     0.5626    61.4   <2e-16 ***

lstat        -0.9500     0.0387   -24.5   <2e-16 ***

---

Signif. codes:  0 *** 0.001 ** 0.01 * 0.05 . 0.1   1

Residual standard error: 6.22 on 504 degrees of freedom

Multiple R-squared: 0.544,      Adjusted R-squared: 0.543

F-statistic:  602 on 1 and 504 DF,  p-value: <2e-16

We can use the names() function in order to find out what other pieces of information are stored in lm.fit. Although we can extract these quantities by name—e.g. lm.fit$coefficients—it is safer to use the extractor functions like coef() to access them.

names()

coef()

> names(lm.fit)

 [1] "coefficients"  "residuals"     "effects"

 [4] "rank"          "fitted.values" "assign"

 [7] "qr"            "df.residual"   "xlevels"

[10] "call"          "terms"         "model"

> coef(lm.fit)

(Intercept)       lstat

      34.55       -0.95

In order to obtain a confidence interval for the coefficient estimates, we can use the confint() command.

confint()

> confint(lm.fit)

            2.5 % 97.5 %

(Intercept) 33.45 35.659

lstat       -1.03 -0.874

The predict() function can be used to produce confidence intervals and prediction intervals for the prediction of medv for a given value of lstat.

predict()

> predict(lm.fit , data.frame(lstat=c(5 ,10 ,15)) , interval="confidence")

       interval="confidence")

    fit   lwr   upr

1 29.80 29.01 30.60

2 25.05 24.47 25.63

3 20.30 19.73 20.87

> predict (lm.fit , data.frame(lstat =(c(5 ,10 ,15))), interval ="prediction")

       interval="prediction")

    fit    lwr   upr

1 29.80 17.566 42.04

2 25.05 12.828 37.28

3 20.30  8.078 32.53

For instance, the 95 % confidence interval associated with a lstat value of 10 is (24. 47, 25. 63), and the 95 % prediction interval is (12. 828, 37. 28). As expected, the confidence and prediction intervals are centered around the same point (a predicted value of 25. 05 for medv when lstat equals 10), but the latter are substantially wider.

We will now plot medv and lstat along with the least squares regression line using the plot() and abline() functions.

abline()

> plot(lstat,medv)

> abline(lm.fit)

There is some evidence for non-linearity in the relationship between lstat and medv. We will explore this issue later in this lab.

The abline() function can be used to draw any line, not just the least squares regression line. To draw a line with intercept a and slope b, we type abline(a,b). Below we experiment with some additional settings for plotting lines and points. The lwd=3 command causes the width of the regression line to be increased by a factor of 3; this works for the plot() and lines() functions also. We can also use the pch option to create different plotting symbols.

> abline(lm.fit,lwd=3)

> abline(lm.fit,lwd=3,col="red")

> plot(lstat,medv,col="red")

> plot(lstat,medv,pch=20)

> plot(lstat,medv,pch="+")

> plot(1:20,1:20,pch=1:20)

Next we examine some diagnostic plots, several of which were discussed in Section 3.3.3. Four diagnostic plots are automatically produced by applying the plot() function directly to the output from lm(). In general, this command will produce one plot at a time, and hitting Enter will generate the next plot. However, it is often convenient to view all four plots together. We can achieve this by using the par() function, which tells R to split the display screen into separate panels so that multiple plots can be viewed simultaneously. For example, par(mfrow=c(2,2)) divides the plotting region into a 2 ×2 grid of panels.

par()

> par(mfrow=c(2,2))

> plot(lm.fit)

Alternatively, we can compute the residuals from a linear regression fit using the residuals() function. The function rstudent() will return the studentized residuals, and we can use this function to plot the residuals against the fitted values.

residuals()

rstudent()

> plot(predict(lm.fit), residuals(lm.fit))

> plot(predict(lm.fit), rstudent(lm.fit))

On the basis of the residual plots, there is some evidence of non-linearity. Leverage statistics can be computed for any number of predictors using the hatvalues() function.

hatvalues()

> plot(hatvalues(lm.fit))

> which.max(hatvalues(lm.fit))

375

The which.max() function identifies the index of the largest element of a vector. In this case, it tells us which observation has the largest leverage statistic.

which.max()

3.6.3 Multiple Linear Regression

In order to fit a multiple linear regression model using least squares, we again use the lm() function. The syntax lm(y ∼ x1+x2+x3) is used to fit a model with three predictors, x1, x2, and x3. The summary() function now outputs the regression coefficients for all the predictors.

> lm.fit=lm(medv∼lstat+age,data=Boston)

> summary(lm.fit)

Call:

lm(formula = medv ∼ lstat + age, data = Boston)

Residuals:

   Min     1Q Median     3Q    Max

-15.98  -3.98  -1.28   1.97  23.16

Coefficients:

            Estimate Std. Error t value Pr(>|t|)

(Intercept)  33.2228     0.7308   45.46   <2e-16 ***

lstat        -1.0321     0.0482  -21.42   <2e-16 ***

age           0.0345     0.0122    2.83   0.0049 **

---

Signif. codes:  0 *** 0.001 ** 0.01 * 0.05 . 0.1  1

Residual standard error: 6.17 on 503 degrees of freedom

Multiple R-squared: 0.551,      Adjusted R-squared: 0.549

F-statistic:  309 on 2 and 503 DF,  p-value: <2e-16

The Boston data set contains 13 variables, and so it would be cumbersome to have to type all of these in order to perform a regression using all of the predictors. Instead, we can use the following short-hand:

> lm.fit=lm(medv∼.,data=Boston)

> summary(lm.fit)

Call:

lm(formula = medv ∼ ., data = Boston)

Residuals:

    Min      1Q  Median      3Q     Max

-15.594  -2.730  -0.518   1.777  26.199

Coefficients:

              Estimate Std. Error t value Pr(>|t|)

(Intercept)  3.646e+01  5.103e+00   7.144 3.28e-12 ***

crim        -1.080e-01  3.286e-02  -3.287 0.001087 **

zn           4.642e-02  1.373e-02   3.382 0.000778 ***

indus        2.056e-02  6.150e-02   0.334 0.738288

chas         2.687e+00  8.616e-01   3.118 0.001925 **

nox         -1.777e+01  3.820e+00  -4.651 4.25e-06 ***

rm           3.810e+00  4.179e-01   9.116  < 2e-16 ***

age          6.922e-04  1.321e-02   0.052 0.958229

dis         -1.476e+00  1.995e-01  -7.398 6.01e-13 ***

rad          3.060e-01  6.635e-02   4.613 5.07e-06 ***

tax         -1.233e-02  3.761e-03  -3.280 0.001112 **

ptratio     -9.527e-01  1.308e-01  -7.283 1.31e-12 ***

black        9.312e-03  2.686e-03   3.467 0.000573 ***

lstat       -5.248e-01  5.072e-02 -10.347  < 2e-16 ***

---

Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 4.745 on 492 degrees of freedom

Multiple R-Squared: 0.7406,     Adjusted R-squared: 0.7338

F-statistic: 108.1 on 13 and 492 DF,  p-value: < 2.2e-16

We can access the individual components of a summary object by name (type ?summary.lm to see what is available). Hence summary(lm.fit)$r.sq gives us the R ², and summary(lm.fit)$sigma gives us the RSE. The vif() function, part of the car package, can be used to compute variance inflation factors. Most VIF’s are low to moderate for this data. The car package is not part of the base R installation so it must be downloaded the first time you use it via the install.packages option in R.

vif()

> library(car)

> vif(lm.fit)

   crim      zn   indus    chas     nox      rm     age

   1.79    2.30    3.99    1.07    4.39    1.93    3.10

    dis     rad     tax ptratio   black   lstat

   3.96    7.48    9.01    1.80    1.35    2.94

What if we would like to perform a regression using all of the variables but one? For example, in the above regression output, age has a high p-value. So we may wish to run a regression excluding this predictor. The following syntax results in a regression using all predictors except age.

> lm.fit1=lm(medv∼.-age,data=Boston)

> summary(lm.fit1)

...

Alternatively, the update() function can be used.

update()

> lm.fit1=update(lm.fit, ∼.-age)

3.6.4 Interaction Terms

It is easy to include interaction terms in a linear model using the lm() function. The syntax lstat:black tells R to include an interaction term between lstat and black. The syntax lstat*age simultaneously includes lstat, age, and the interaction term lstat ×age as predictors; it is a shorthand for lstat+age+lstat:age.

> summary(lm(medv∼lstat*age,data=Boston))

Call:

lm(formula = medv ∼ lstat * age, data = Boston)

Residuals:

   Min     1Q Median     3Q    Max

-15.81  -4.04  -1.33   2.08  27.55

Coefficients:

             Estimate Std. Error t value Pr(>|t|)

(Intercept) 36.088536   1.469835   24.55  < 2e-16 ***

lstat       -1.392117   0.167456   -8.31  8.8e-16 ***

age         -0.000721   0.019879   -0.04    0.971

lstat:age    0.004156   0.001852    2.24    0.025 *

---

Signif. codes:  0 ’***’ 0.001 ’**’ 0.01 ’*’ 0.05 ’.’ 0.1 ’ ’ 1

Residual standard error: 6.15 on 502 degrees of freedom

Multiple R-squared: 0.556,  Adjusted R-squared: 0.553

F-statistic:  209 on 3 and 502 DF,  p-value: <2e-16

3.6.5 Non-linear Transformations of the Predictors

The lm() function can also accommodate non-linear transformations of the predictors. For instance, given a predictor X, we can create a predictor X ² using I(X^2). The function I() is needed since the ^ has a special meaning in a formula; wrapping as we do allows the standard usage in R, which is to raise X to the power 2. We now perform a regression of medv onto lstat and lstat2.

I()

> lm.fit2=lm(medv∼lstat+I(lstat^2))

> summary(lm.fit2)

Call:

lm(formula = medv ∼ lstat + I(lstat^2))

Residuals:

   Min     1Q Median     3Q    Max

-15.28  -3.83  -0.53   2.31  25.41

Coefficients:

            Estimate Std. Error t value Pr(>|t|)

(Intercept) 42.86201    0.87208    49.1   <2e-16 ***

lstat       -2.33282    0.12380   -18.8   <2e-16 ***

I(lstat^2)   0.04355    0.00375    11.6   <2e-16 ***

---

Signif. codes:  0 ’***’ 0.001 ’**’ 0.01 ’*’ 0.05 ’.’ 0.1 ’ ’ 1

Residual standard error: 5.52 on 503 degrees of freedom

Multiple R-squared: 0.641,  Adjusted R-squared: 0.639

F-statistic:  449 on 2 and 503 DF,  p-value: <2e-16

The near-zero p-value associated with the quadratic term suggests that it leads to an improved model. We use the anova() function to further quantify the extent to which the quadratic fit is superior to the linear fit.

anova()

> lm.fit=lm(medv∼lstat)

> anova(lm.fit,lm.fit2)

Analysis of Variance Table

Model 1: medv ∼ lstat

Model 2: medv ∼ lstat + I(lstat^2)

  Res.Df   RSS Df Sum of Sq   F Pr(>F)

1    504 19472

2    503 15347  1      4125 135 <2e-16 ***

---

Signif. codes:  0 ’***’ 0.001 ’**’ 0.01 ’*’ 0.05 ’.’ 0.1 ’ ’ 1

Here Model 1 represents the linear submodel containing only one predictor, lstat, while Model 2 corresponds to the larger quadratic model that has two predictors, lstat and lstat2. The anova() function performs a hypothesis test comparing the two models. The null hypothesis is that the two models fit the data equally well, and the alternative hypothesis is that the full model is superior. Here the F-statistic is 135 and the associated p-value is virtually zero. This provides very clear evidence that the model containing the predictors lstat and lstat2 is far superior to the model that only contains the predictor lstat. This is not surprising, since earlier we saw evidence for non-linearity in the relationship between medv and lstat. If we type

 > par(mfrow=c(2,2))

 > plot(lm.fit2)

then we see that when the lstat2 term is included in the model, there is little discernible pattern in the residuals.

In order to create a cubic fit, we can include a predictor of the form I(X^3). However, this approach can start to get cumbersome for higher-order polynomials. A better approach involves using the poly() function to create the polynomial within lm(). For example, the following command produces a fifth-order polynomial fit:

poly()

> lm.fit5=lm(medv∼poly(lstat,5))

> summary(lm.fit5)

Call:

lm(formula = medv ∼ poly(lstat, 5))

Residuals:

    Min      1Q  Median      3Q     Max

-13.543  -3.104  -0.705   2.084  27.115

Coefficients:

                Estimate Std. Error t value Pr(>|t|)

(Intercept)       22.533      0.232   97.20  < 2e-16 ***

poly(lstat, 5)1 -152.460      5.215  -29.24  < 2e-16 ***

poly(lstat, 5)2   64.227      5.215   12.32  < 2e-16 ***

poly(lstat, 5)3  -27.051      5.215   -5.19  3.1e-07 ***

poly(lstat, 5)4   25.452      5.215    4.88  1.4e-06 ***

poly(lstat, 5)5  -19.252      5.215   -3.69  0.00025 ***

---

Signif. codes:  0 ’***’ 0.001 ’**’ 0.01 ’*’ 0.05 ’.’ 0.1 ’ ’ 1

Residual standard error: 5.21 on 500 degrees of freedom

Multiple R-squared: 0.682,  Adjusted R-squared: 0.679

F-statistic:  214 on 5 and 500 DF,  p-value: <2e-16

This suggests that including additional polynomial terms, up to fifth order, leads to an improvement in the model fit! However, further investigation of the data reveals that no polynomial terms beyond fifth order have significant p-values in a regression fit.

Of course, we are in no way restricted to using polynomial transformations of the predictors. Here we try a log transformation.

> summary(lm(medv∼log(rm),data=Boston))

...

3.6.6 Qualitative Predictors

We will now examine the Carseats data, which is part of the ISLR library. We will attempt to predict Sales (child car seat sales) in 400 locations based on a number of predictors.

> fix(Carseats)

> names(Carseats)

 [1] "Sales"       "CompPrice"   "Income"      "Advertising"

 [5] "Population"  "Price"       "ShelveLoc"   "Age"

 [9] "Education"   "Urban"       "US"

The Carseats data includes qualitative predictors such as Shelveloc, an indicator of the quality of the shelving location—that is, the space within a store in which the car seat is displayed—at each location. The predictor Shelveloc takes on three possible values, Bad, Medium, and Good. Given a qualitative variable such as Shelveloc, R generates dummy variables automatically. Below we fit a multiple regression model that includes some interaction terms.

> lm.fit=lm(Sales∼.+Income:Advertising+Price:Age,data=Carseats)

> summary(lm.fit)

Call:

lm(formula = Sales ∼ . + Income:Advertising + Price:Age, data = Carseats)

Residuals:

   Min     1Q Median     3Q    Max

-2.921 -0.750  0.018  0.675  3.341

Coefficients:

                    Estimate Std. Error t value Pr(>|t|)

(Intercept)         6.575565   1.008747    6.52  2.2e-10 ***

CompPrice           0.092937   0.004118   22.57  < 2e-16 ***

Income              0.010894   0.002604    4.18  3.6e-05 ***

Advertising         0.070246   0.022609    3.11  0.00203 **

Population          0.000159   0.000368    0.43  0.66533

Price              -0.100806   0.007440  -13.55  < 2e-16 ***

ShelveLocGood       4.848676   0.152838   31.72  < 2e-16 ***

ShelveLocMedium     1.953262   0.125768   15.53  < 2e-16 ***

Age                -0.057947   0.015951   -3.63  0.00032 ***

Education          -0.020852   0.019613   -1.06  0.28836

UrbanYes            0.140160   0.112402    1.25  0.21317

USYes              -0.157557   0.148923   -1.06  0.29073

Income:Advertising  0.000751   0.000278    2.70  0.00729 **

Price:Age           0.000107   0.000133    0.80  0.42381

---

Signif. codes:  0 ’***’ 0.001 ’**’ 0.01 ’*’ 0.05 ’.’ 0.1 ’ ’ 1

Residual standard error: 1.01 on 386 degrees of freedom

Multiple R-squared: 0.876,      Adjusted R-squared: 0.872

F-statistic:  210 on 13 and 386 DF,  p-value: <2e-16

The contrasts() function returns the coding that R uses for the dummy variables.

contrasts()

> attach(Carseats)

> contrasts(ShelveLoc)

       Good Medium

Bad       0      0

Good      1      0

Medium    0      1

Use ?contrasts to learn about other contrasts, and how to set them.

R has created a ShelveLocGood dummy variable that takes on a value of 1 if the shelving location is good, and 0 otherwise. It has also created a ShelveLocMedium dummy variable that equals 1 if the shelving location is medium, and 0 otherwise. A bad shelving location corresponds to a zero for each of the two dummy variables. The fact that the coefficient for ShelveLocGood in the regression output is positive indicates that a good shelving location is associated with high sales (relative to a bad location). And ShelveLocMedium has a smaller positive coefficient, indicating that a medium shelving location leads to higher sales than a bad shelving location but lower sales than a good shelving location.

3.6.7 Writing Functions

As we have seen, R comes with many useful functions, and still more functions are available by way of R libraries. However, we will often be interested in performing an operation for which no function is available. In this setting, we may want to write our own function. For instance, below we provide a simple function that reads in the ISLR and MASS libraries, called LoadLibraries(). Before we have created the function, R returns an error if we try to call it.

> LoadLibraries

Error: object ’LoadLibraries’ not found

> LoadLibraries()

Error: could not find function "LoadLibraries"

We now create the function. Note that the + symbols are printed by R and should not be typed in. The { symbol informs R that multiple commands are about to be input. Hitting Enter after typing { will cause R to print the + symbol. We can then input as many commands as we wish, hitting Enter after each one. Finally the } symbol informs R that no further commands will be entered.

> LoadLibraries=function(){

+ library(ISLR)

+ library(MASS)

+ print("The libraries have been loaded.")

+ }

Now if we type in LoadLibraries, R will tell us what is in the function.

> LoadLibraries

function(){

library(ISLR)

library(MASS)

print("The libraries have been loaded.")

}

If we call the function, the libraries are loaded in and the print statement is output.

> LoadLibraries()

[1] "The libraries have been loaded."

3.7 Exercises

3.7.1 Conceptual

• 1. Describe the null hypotheses to which the p-values given in Table 3.4 correspond. Explain what conclusions you can draw based on these p-values. Your explanation should be phrased in terms of sales, TV, radio, and newspaper, rather than in terms of the coefficients of the linear model.

• 2. Carefully explain the differences between the KNN classifier and KNN regression methods.

• 3. Suppose we have a data set with five predictors, X ₁ = GPA, X ₂ = IQ, X ₃ = Gender (1 for Female and 0 for Male), X ₄ = Interaction between GPA and IQ, and X ₅ = Interaction between GPA and Gender. The response is starting salary after graduation (in thousands of dollars). Suppose we use least squares to fit the model, and get $\hat{\beta }_{0} = 50,\hat{\beta }_{1} = 20,\hat{\beta }_{2} = 0.07,\hat{\beta }_{3} = 35,\hat{\beta }_{4} = 0.01,\hat{\beta }_{5} = -10$.

  • (a) Which answer is correct, and why?

    • i. For a fixed value of IQ and GPA, males earn more on average than females.

    • ii. For a fixed value of IQ and GPA, females earn more on average than males.

    • iii. For a fixed value of IQ and GPA, males earn more on average than females provided that the GPA is high enough.

    • iv. For a fixed value of IQ and GPA, females earn more on average than males provided that the GPA is high enough.

  • (b) Predict the salary of a female with IQ of 110 and a GPA of 4. 0.

  • (c) True or false: Since the coefficient for the GPA/IQ interaction term is very small, there is very little evidence of an interaction effect. Justify your answer.

• 4. I collect a set of data (n = 100 observations) containing a single predictor and a quantitative response. I then fit a linear regression model to the data, as well as a separate cubic regression, i.e. $Y =\beta _{0} +\beta _{1}X +\beta _{2}{X}^{2} +\beta _{3}{X}^{3}+\epsilon$.

  • (a) Suppose that the true relationship between X and Y is linear, i.e. $Y =\beta _{0} +\beta _{1}X+\epsilon$. Consider the training residual sum of squares (RSS) for the linear regression, and also the training RSS for the cubic regression. Would we expect one to be lower than the other, would we expect them to be the same, or is there not enough information to tell? Justify your answer.

  • (b) Answer (a) using test rather than training RSS.

  • (c) Suppose that the true relationship between X and Y is not linear, but we don’t know how far it is from linear. Consider the training RSS for the linear regression, and also the training RSS for the cubic regression. Would we expect one to be lower than the other, would we expect them to be the same, or is there not enough information to tell? Justify your answer.

  • (d) Answer (c) using test rather than training RSS.

• 5. Consider the fitted values that result from performing linear regression without an intercept. In this setting, the ith fitted value takes the form

  $$\displaystyle{\hat{y}_{i} = x_{i}\hat{\beta },}$$

  where

  $$\displaystyle{ \hat{\beta }= \left (\sum _{i=1}^{n}x_{ i}y_{i}\right )/\left (\sum _{i^\prime=1}^{n}x_{ i^\prime}^{2}\right ). }$$

  (3.38)

  Show that we can write

  $$\displaystyle{\hat{y}_{i} =\sum _{ i^\prime=1}^{n}a_{ i^\prime}y_{i^\prime}.}$$

  What is a _i′ ?

  Note: We interpret this result by saying that the fitted values from linear regression are linear combinations of the response values.

• 6. Using (3.4), argue that in the case of simple linear regression, the least squares line always passes through the point $(\bar{x},\bar{y})$.

• 7. It is claimed in the text that in the case of simple linear regression of Y onto X, the R ² statistic (3.17) is equal to the square of the correlation between X and Y (3.18). Prove that this is the case. For simplicity, you may assume that $\bar{x} =\bar{ y} = 0$.

3.7 Applied

• 8. This question involves the use of simple linear regression on the Auto data set.

  • (a) Use the lm() function to perform a simple linear regression with mpg as the response and horsepower as the predictor. Use the summary() function to print the results. Comment on the output. For example:

    • i. Is there a relationship between the predictor and the response?

    • ii. How strong is the relationship between the predictor and the response?

    • iii. Is the relationship between the predictor and the response positive or negative?

    • iv. What is the predicted mpg associated with a horsepower of 98? What are the associated 95 % confidence and prediction intervals?

  • (b) Plot the response and the predictor. Use the abline() function to display the least squares regression line.

  • (c) Use the plot() function to produce diagnostic plots of the least squares regression fit. Comment on any problems you see with the fit.

• 9. This question involves the use of multiple linear regression on the Auto data set.

  • (a) Produce a scatterplot matrix which includes all of the variables in the data set.

  • (b) Compute the matrix of correlations between the variables using the function cor(). You will need to exclude the name variable, which is qualitative.

  • (c) Use the lm() function to perform a multiple linear regression with mpg as the response and all other variables except name as the predictors. Use the summary() function to print the results. Comment on the output. For instance:

    • i. Is there a relationship between the predictors and the response?

    • ii. Which predictors appear to have a statistically significant relationship to the response?

    • iii. What does the coefficient for the year variable suggest?

  • (d) Use the plot() function to produce diagnostic plots of the linear regression fit. Comment on any problems you see with the fit. Do the residual plots suggest any unusually large outliers? Does the leverage plot identify any observations with unusually high leverage?

  • (e) Use the * and : symbols to fit linear regression models with interaction effects. Do any interactions appear to be statistically significant?

  • (f) Try a few different transformations of the variables, such as log(X), $\sqrt{X}$, X ². Comment on your findings.

• 10. This question should be answered using the Carseats data set.

  • (a) Fit a multiple regression model to predict Sales using Price, Urban, and US.

  • (b) Provide an interpretation of each coefficient in the model. Be careful—some of the variables in the model are qualitative!

  • (c) Write out the model in equation form, being careful to handle the qualitative variables properly.

  • (d) For which of the predictors can you reject the null hypothesis H ₀ : β _j  = 0?

  • (e) On the basis of your response to the previous question, fit a smaller model that only uses the predictors for which there is evidence of association with the outcome.

  • (f) How well do the models in (a) and (e) fit the data?

  • (g) Using the model from (e), obtain 95 % confidence intervals for the coefficient(s).

  • (h) Is there evidence of outliers or high leverage observations in the model from (e)?

• 11. In this problem we will investigate the t-statistic for the null hypothesis H ₀ : β = 0 in simple linear regression without an intercept. To begin, we generate a predictor x and a response y as follows.

cor()

> set.seed(1)

> x=rnorm(100)

> y=2*x+rnorm(100)

• (a) Perform a simple linear regression of y onto x, without an intercept. Report the coefficient estimate $\hat{\beta }$, the standard error of this coefficient estimate, and the t-statistic and p-value associated with the null hypothesis H ₀ : β = 0. Comment on these results. (You can perform regression without an intercept using the command lm(y ∼ x+0).)

• (b) Now perform a simple linear regression of x onto y without an intercept, and report the coefficient estimate, its standard error, and the corresponding t-statistic and p-values associated with the null hypothesis H ₀ : β = 0. Comment on these results.

• (c) What is the relationship between the results obtained in (a) and (b)?

• (d) For the regression of Y onto X without an intercept, the t-statistic for H ₀ : β = 0 takes the form $\hat{\beta }/\mbox{ SE}(\hat{\beta })$, where $\hat{\beta }$ is given by (3.38), and where

  $$\displaystyle{\mbox{ SE}(\hat{\beta }) = \sqrt{ \frac{\sum _{i=1 }^{n }{(y_{i } - x_{i } \hat{\beta })}^{2 } } {(n - 1)\sum _{i^\prime=1}^{n}x_{i^\prime}^{2}}}.}$$

  (These formulas are slightly different from those given in Sections 3.1.1 and 3.1.2, since here we are performing regression without an intercept.) Show algebraically, and confirm numerically in R, that the t-statistic can be written as

  $$\displaystyle{ \frac{(\sqrt{n - 1})\sum _{i=1}^{n}x_{i}y_{i}} {\sqrt{(\sum _{i=1 }^{n }x_{i }^{2 })(\sum _{i^\prime =1 }^{n }y_{i^\prime }^{2 }) - {(\sum _{i^\prime=1 }^{n }x_{i^\prime } y_{i^\prime } )}^{2}}}.}$$

• (e) Using the results from (d), argue that the t-statistic for the regression of y onto x is the same as the t-statistic for the regression of x onto y.

• (f) In R, show that when regression is performed with an intercept, the t-statistic for H ₀ : β ₁ = 0 is the same for the regression of y onto x as it is for the regression of x onto y.

• 12. This problem involves simple linear regression without an intercept.

  • (a) Recall that the coefficient estimate $\hat{\beta }$ for the linear regression of Y onto X without an intercept is given by (3.38). Under what circumstance is the coefficient estimate for the regression of X onto Y the same as the coefficient estimate for the regression of Y onto X?

  • (b) Generate an example in R with n = 100 observations in which the coefficient estimate for the regression of X onto Y is different from the coefficient estimate for the regression of Y onto X.

  • (c) Generate an example in R with n = 100 observations in which the coefficient estimate for the regression of X onto Y is the same as the coefficient estimate for the regression of Y onto X.

• 13. In this exercise you will create some simulated data and will fit simple linear regression models to it. Make sure to use set.seed(1) prior to starting part (a) to ensure consistent results.

  • (a) Using the rnorm() function, create a vector, x, containing 100 observations drawn from a N(0, 1) distribution. This represents a feature, X.

  • (b) Using the rnorm() function, create a vector, eps, containing 100 observations drawn from a N(0, 0. 25) distribution i.e. a normal distribution with mean zero and variance 0. 25.

  • (c) Using x and eps, generate a vector y according to the model

    $$\displaystyle{ Y = -1 + 0.5X +\epsilon . }$$

    (3.39)

    What is the length of the vector y? What are the values of β ₀ and β ₁ in this linear model?

  • (d) Create a scatterplot displaying the relationship between x and y. Comment on what you observe.

  • (e) Fit a least squares linear model to predict y using x. Comment on the model obtained. How do $\hat{\beta }_{0}$ and $\hat{\beta }_{1}$ compare to β ₀ and β ₁?

  • (f) Display the least squares line on the scatterplot obtained in (d). Draw the population regression line on the plot, in a different color. Use the legend() command to create an appropriate legend.

  • (g) Now fit a polynomial regression model that predicts y using x and x ². Is there evidence that the quadratic term improves the model fit? Explain your answer.

  • (h) Repeat (a)–(f) after modifying the data generation process in such a way that there is less noise in the data. The model (3.39) should remain the same. You can do this by decreasing the variance of the normal distribution used to generate the error term ε in (b). Describe your results.

  • (i) Repeat (a)–(f) after modifying the data generation process in such a way that there is more noise in the data. The model (3.39) should remain the same. You can do this by increasing the variance of the normal distribution used to generate the error term ε in (b). Describe your results.

  • (j) What are the confidence intervals for β ₀ and β ₁ based on the original data set, the noisier data set, and the less noisy data set? Comment on your results.

• 14. This problem focuses on the collinearity problem.

  • (a) Perform the following commands in R:

    > set.seed(1)

    > x1=runif(100)

    > x2=0.5*x1+rnorm(100)/10

    > y=2+2*x1+0.3*x2+rnorm(100)

    The last line corresponds to creating a linear model in which y is a function of x1 and x2. Write out the form of the linear model. What are the regression coefficients?

• (b) What is the correlation between x1 and x2? Create a scatterplot displaying the relationship between the variables.

• (c) Using this data, fit a least squares regression to predict y using x1 and x2. Describe the results obtained. What are $\hat{\beta }_{0}$, $\hat{\beta }_{1}$, and $\hat{\beta }_{2}$? How do these relate to the true β ₀, β ₁, and β ₂? Can you reject the null hypothesis H ₀ : β ₁ = 0? How about the null hypothesis H ₀ : β ₂ = 0?

• (d) Now fit a least squares regression to predict y using only x1. Comment on your results. Can you reject the null hypothesis H ₀ : β ₁ = 0?

• (e) Now fit a least squares regression to predict y using only x2. Comment on your results. Can you reject the null hypothesis H ₀ : β ₁ = 0?

• (f) Do the results obtained in (c)–(e) contradict each other? Explain your answer.

• (g) Now suppose we obtain one additional observation, which was unfortunately mismeasured.

  > x1=c(x1, 0.1)

  > x2=c(x2, 0.8)

  > y=c(y,6)

  Re-fit the linear models from (c) to (e) using this new data. What effect does this new observation have on the each of the models? In each model, is this observation an outlier? A high-leverage point? Both? Explain your answers.

• 15. This problem involves the Boston data set, which we saw in the lab for this chapter. We will now try to predict per capita crime rate using the other variables in this data set. In other words, per capita crime rate is the response, and the other variables are the predictors.

  • (a) For each predictor, fit a simple linear regression model to predict the response. Describe your results. In which of the models is there a statistically significant association between the predictor and the response? Create some plots to back up your assertions.

  • (b) Fit a multiple regression model to predict the response using all of the predictors. Describe your results. For which predictors can we reject the null hypothesis H ₀ : β _j  = 0?

  • (c) How do your results from (a) compare to your results from (b)? Create a plot displaying the univariate regression coefficients from (a) on the x-axis, and the multiple regression coefficients from (b) on the y-axis. That is, each predictor is displayed as a single point in the plot. Its coefficient in a simple linear regression model is shown on the x-axis, and its coefficient estimate in the multiple linear regression model is shown on the y-axis.

  • (d) Is there evidence of non-linear association between any of the predictors and the response? To answer this question, for each predictor X, fit a model of the form

    $$\displaystyle{Y =\beta _{0} +\beta _{1}X +\beta _{2}{X}^{2} +\beta _{ 3}{X}^{3} +\epsilon .}$$