Introduction

Online Courses in Context

More and more college education is being delivered through online courses. In 2012, nearly 7 million college students—about one-third of all students—took at least one online course, and the number of students taking online courses has been growing at about 9–10 % per year (Allen and Seaman 2013). Although the growth rate has fallen since 2011, it still exceeds the growth rate of the college population (Allen and Seaman 2015). Online courses are an increasingly important mode of delivering education at the postsecondary level. For some students, the ability to attend class without traveling to campus, or without adhering to a particular schedule, offers an additional convenience that makes a college education easier to pursue.

The growth in online course-taking at community colleges has arisen in the midst of growing Internet connectivity and technological sophistication. Nearly all public community colleges now offer online courses (Allen and Seaman 2015). Access to online courses in college has been expanding, and this expansion shifts the curriculum-choice process and the learning experience of students in community colleges.

The rapid growth in online courses amounts to a sea change in the delivery of the college curriculum. Not surprisingly, a growing literature compares the effectiveness of online and face-to-face courses in fostering students’ learning and engagement. Most research on the effect of online courses is either descriptive or experimental (see, for example, Means et al. 2009 or Russell 2015 for a partial literature review), rather than quasi-experimental. Descriptive approaches to estimating the effect of online courses are clearly limited because selection into online courses is generally nonrandom, raising the potential for selection bias.

Experimental studies of online courses typically compare outcomes (often on a shared exam) between two variants of the same course: one online (or hybrid) and one face-to-face. These studies often attempt to control for as many variables not related to delivery format as possible. Ideally both courses are taught by the same instructors, with similar amounts of instructional and preparation time as well as similar access to materials. To give two examples of experiments on introductory microeconomics courses at selective 4-year colleges, Figlio et al. (2013) randomized students to receive either online or face-to-face versions of the lecture, holding instructors, assignments, exams, and support otherwise constant. They found that students in face-to-face instruction modestly outperformed students in online instruction, and that differences were greater for Hispanic students, males, and those with lower achievement. Joyce et al. (2015a) randomized students to a traditional course and a hybrid course with less in-person lecture time, holding instructors, access to lecture slides, exams, and support otherwise constant. They found very small differences between the delivery formats.

Experimental estimates, however, have some important limitations in this context. Because of cost considerations, experiments usually are done with small groups of students and compare specific online and offline versions of a particular course, rather than looking at the wider mix of online courses that typically are offered. If the online or offline curricula chosen to be on the treatment or control side of an experiment are atypical of most online or offline courses, as might be the case if the online course is designed to differ only in delivery format for experimental comparability, or if the setting of the experiment is atypical, then the results lack external validity (Allcott 2015). Moreover, by eliminating all selection on the part of the student, experiments cannot address the potentially vital question of how the effectiveness of the course intersects with the probability that a student will actually take it. Reasons for choosing online or offline courses include issues such as the ability to choose the timing of learning, or a preference for interpersonal contact (Roblyer 1999). These reasons may or may not relate to the ability to learn well in those courses. It is possible to design experiments that take into account this selection issue by performing the experiment twice, once randomized and once allowing choice but these experiments are expensive and are not standard practice (Joyce et al. 2015a). Standard online/face-to-face course experiments are extremely well-suited for answering the question “What will be the effect if a specific face-to-face course is replaced with an online course?” but not as well-suited as quasi-experimental approaches to answering the question “What will be the effect if students are given access to more online curriculum options?” Both questions are important matters for policy as colleges update their approaches to providing education.

In this paper, we use data from Washington State community colleges to estimate the effect of taking an online course, rather than a face-to-face version of the same course, on the probability of taking a follow-up course in the same field and on the probability of graduating with an associate’s degree (AA) or bachelor’s degree (BA). We estimate the average treatment effect (the effect of taking an online course for the average person) and the average treatment on the treated (the effect of taking an online course for the average person who actually takes an online course) by using an endogenous switching model in which students choose between online and offline courses. This model allows for the student choice that is a part of a real-world online course-taking, while addressing selection bias by using excluded variables that predict online course-taking but are not expected to affect learning in the course.

We find a negative effect of −1.7 % points of taking an online course on the probability of earning a degree. We also found evidence of a small amount of negative selection into online courses, contrary to prior literature. That is, the students who choose to take online courses are not the same students who get the most from them. The average treatment on the treated is a statistically significant −2.0 % points on the probability of graduating with a degree. Online courses also discourage students from continuing study in the same field. The average treatment effect of taking an online course on the probability of taking a follow-up course in the same field is a statistically significant −6.5 % points. There is evidence of some negative selection in this outcome as well, but the difference between the average treatment effect and the average treatment on the treated is insignificant and small. In all, we find that expanding availability of online courses at community colleges likely leads to lower persistence in a particular line of study or towards graduating with a degree. The effects on taking a follow-up course and on graduation rates are significant and meaningful, and more so for those who actually choose to take the courses.

These results, of course, come from the specific context of Washington State community colleges. The Washington State community college system has 34 campuses across the state, about half of which are clustered in the Seattle-Tacoma area. The system enrolls about 386,000 students annually. The student body is diverse, and the system has higher representation of black, Hispanic, and economically disadvantaged students relative to the state’s public four-year colleges. The system operates under a goal of increasing educational pathways, and is legally bound to “offer an open door to every citizen, regardless of his or her academic background or experiences, at a cost normally within his or her economic means.”Footnote 1 One way the system acts upon this goal is by developing and supporting online curriculum.

Online courses exhibit a large amount of heterogeneity across all applications (Bowen 2015). Due to the relatively robust support of the central community college system, Washington community colleges offer a strong test of the potential for online learning in community college settings. As early as 1998, Washington State community colleges provided student assessment informing them whether or not they are ready for online courses, student tutorials of online systems, easily accessible technical support, and faculty development support (Xu and Jaggars 2011). The community college system hosts a centralized platform for online courses, WashingtonOnline, developed through a 1999 grant from the U.S. Department of Education. The system facilitates the sharing of online instruction and enrollment across campuses in the community college system. Students may enroll in courses outside their home institution, allowing for specialization in courses across institutions. By 2008, student enrollment through WashingtonOnline comprised 30 % of total enrollment in online courses and 12 of the 34 colleges relied exclusively on the system for their online course offerings (Washington State Board for Community and Technical Colleges 2008). In 2011, the community college system began releasing open-source course materials for 81 of the most highly enrolled courses. While this effort was designed to reduce the cost of instructional materials for students in both face-to-face and online courses, the open-source nature of the project also created a pool of digital instructional materials available to students online and tailored to the most popular community college courses in Washington (Goodwin 2011).

Our results apply most directly when online classes are to be implemented in a similar manner as they have been in Washington State community colleges. Any community college policy that considers making more online courses available must take the deficits we find into account. These negative academic effects of online courses must also be weighed against the potential that current or future online offerings may improve in quality beyond what is available now (Bowen 2015), the possibility of lower tuition via cost savings or increased competition resulting from the implementation of online courses (Deming and Katz 2015), and the associated possibility that these lowered costs and increased access allow additional students to take college courses.

Literature

This paper is not the first to use quasi-experimental methods to address the effectiveness of online courses. Coates et al. (2004) examined the results of a standardized end-of-course exam for students who took an online economics course versus those who took a face-to-face course at three 4-year universities. They used students’ stated commute times, knowing someone who took an online course, and the use of supplemental Internet-provided material in a prior face-to-face course as excluded variables to predict online course-taking in an endogenous switching model. They found that students in the online course scored significantly lower than those in the face-to-face course, but also found heterogeneity in the effect across students—those who selected into the online course performed better than they would have in a face-to-face course. Gratton-Lavoie and Stanley (2009) use a similar approach, with distance and being in a business major as excluded variables in a switching model to study the effects of taking an online course as opposed to a hybrid course on grades in an economics course at a 4-year university. They find a small negative effect of the online course, with positive sorting into the online course.

Several papers focus on online courses in Washington State, the setting for our study. Xu and Jaggars (2013) examined the effect of taking an online versus a face-to-face course on grades in the course taken and the probability of taking a follow-up course in the same field. They used the same Washington State community colleges that form the sample in this paper. The driving distance between the student’s home and the college was used as an instrumental variable to correct for selection bias, and they found that students in online courses earned a grade about one-third of a point lower and were less likely to complete the course.Footnote 2 The difference was larger for certain groups of students, including Black students, younger students, and those with lower grade point averages (GPAs). Our study shares a sample with that of Xu and Jaggars (2013) but differs significantly in subsample (we did not focus only on students planning to transfer to a 4-year college) as well as through differences in identification and measured outcome variables, as discussed below. Krieg and Henson (2015) also used Washington data, from a large Washington State 4-year university, to examine the effect of taking an online course on grades in a follow-up course. Like Xu and Jaggars (2013), they used the distance from a student’s home as an instrument in their achievement analyses, and found that grades in a follow-up course are about one-twelfth of a point lower for students who took online courses.

The four studies listed above find statistically significant (and, arguably, meaningfully large) penalties to taking an online course relative to a face-to-face version of the same course. These studies also share a methodological approach: each uses the distance a student must travel to campus as an excluded variable. Living far away from campus imposes a clear travel cost on students that may not apply to an online course, and the correlation between distance and taking an online course is clearly large and significant in these data sets. There are reasons, however, to be skeptical of the use of distance as an excluded variable in these analyses. One is the issue of selective migration (see Heckman et al. 1996 for a discussion of this aspect in the context of estimating the returns to education quality): students who plan to take face-to-face courses may choose a residence that is closer to campus, and these students may also have differential levels of dedication to their studies.

Both Xu and Jaggars (2013) and Krieg and Henson (2015) performed falsification tests, regressing grades in face-to-face courses on distance to campus, and found no significant relationship.Footnote 3 However, this approach does not address the question of why a student who lives far away has endogenously chosen to take the course at that community college, rather than another one or at a fully online college. For students who live far away and still decide to take an online course at a particular college, the counterfactual is poorly defined—the appropriate comparison may not be not a face-to-face course at the same college, but rather something at a different college altogether, or no course at all. This issue can bias estimates of the effect of being in an online course without it necessarily being the case that distance to campus will predict grades in the sample directly. This approach does not assert that results from papers using distance to campus are necessarily wrong, but as no exogenous variable is ever perfect, it is worthwhile to consider other available sources of identifying variation. If results are similar regardless of the excluded variable used, they lend support to the use of either.

One contribution to the literature in this paper is our proposal of a different excluded variable for use in estimating the effect of online courses. We suggest that students are more likely to choose online courses when online access is cheaper and easier. We proxy this with regional and longitudinal variation in the number of residential high-speed Internet providers in the area where a student lives. Controlling for region by the inclusion of correlated random effects for each course and college, the number of Internet providers should not be related to student performance except via the choice of an online or face-to-face course. We did find that the number of providers is unrelated to performance in the course, conditional on being in an online or face-to-face course. We additionally estimate our results using student’s distance from campus as an excluded variable for comparability to prior literature.

The outcome variables we examine are, in contrast to the prior quasi-experimental literature, related to persistence in the field and towards a degree after the completion of the course. Persistence, especially towards a degree, is an issue of considerable importance for community colleges, where completion rates are commonly low (National Center for Education Statistics 2015 Table 326.20).

Relative to the existing literature, we introduce a new source of identifying variation, and focus on a different set of outcome variables. These contributions add to the ability of policymakers to evaluate the value of online community college education in the future.

Model

Here we present a brief model of choice between an online and offline version of the same college course at the same college. The implications of the model help to guide our empirical approach and make sense of the results, especially as they relate to positive and negative selection into online courses.

Assume that a student \(i\)’s utility of taking a course \(c\) depends on: the future discounted benefits accrued as a result of taking the course, \(Y_{ic}\); the cost of tuition, fees, and travel \(P_{ic}\); and the nonfinancial value of taking the course \(V_{ic}\). \(Y_{ic}\) depends on the amount of human capital accumulated through the student’s learning in course \(c\), which could operate through improved learning or by making the student interested in continuing his or her education. \(V_{ic}\) includes nonfinancial factors that influence the perceived value of the course, such as the enjoyment of taking the course itself, mental strain, and the time investment required for the course. For simplicity we assume that \(\alpha_{i1}\), \(\alpha_{i2}\), and \(\alpha_{i3}\) are positive and that the terms of the indirect utility function are additive:Footnote 4

$$u_{i} (c) = \alpha_{i1} Y_{ic} + \alpha_{i2} V_{ic} - \alpha_{i3} P_{ic}$$
(1)

Courses \(O\) and \(F\) are online and face-to-face versions of the same course at the same college. Student \(i\) prefers course \(O\) to \(F\) if

$$u_{i} \left( {c_{O} } \right) \ge u_{i} \left( {c_{F} } \right) \Rightarrow$$
$$\alpha_{i1} Y_{iO} + \alpha_{i2} V_{iO} - \alpha_{i3} P_{iO} \ge \alpha_{i1} Y_{iF} + \alpha_{i2} V_{iF} - \alpha_{i3} P_{iF} .$$
(2)

This decision requires a comparison of the investment returns to taking the courses, the nonfinancial determinants of perceived value of the course, and the tuition and fee costs of taking the courses. For simplicity, as both courses are at the same college and tuition costs within a college typically do not differ greatly by online/face-to-face status, we assume that \(P_{iO} = P_{iF}\).Footnote 5 This simplifies the above equation to

$$\alpha_{i1} \left( {Y_{iO} - Y_{iF} } \right) + \alpha_{i2} \left( {V_{iO} - V_{iF} } \right) \ge 0$$
(3)

\(Y_{iO} - Y_{iF}\) may vary among students because of differing learning styles that improve or diminish learning in an online environment relative to a face-to-face environment. Similarly, \(V_{iO} - V_{iF}\) may vary among students because of differing abilities to access a physical campus or a computer with high-speed Internet, differing schedule flexibility owing to work demands, or different preferences for learning environment.

We are interested in estimating some portion of \(Y_{iO} - Y_{iF}\), the difference in student outcomes associated with taking an online course over the face-to-face version. We are also interested in the relationship between \(Y_{iO} - Y_{iF}\) and the above selection equation. Are the students who choose online courses (those for whom \(u_{i} \left( {c_{0} } \right) \ge u_{i} \left( {c_{F} } \right)\)) also the students who get the most out of them in the long term (have high values of \(Y_{iO} - Y_{iF}\))?

If \(\alpha_{i1} > 0\), \(\alpha_{i2} \ge 0,\) and \(Y_{iO} - Y_{iF}\) is uncorrelated or positively correlated over students with \(V_{iO} - V_{iF}\), then on average students who learn best in online courses will be more likely to choose online courses. Although these assumptions make intuitive sense, we know of no study that gives a clear indication of what the correlation might be between \(Y_{iO} - Y_{iF}\) and \(V_{iO} - V_{iF}\). Instead, plausible reasons exist to expect a negative correlation. This would be the case, for instance, if students who highly value the convenience and experience of the online classroom format are more likely to be students who would learn more effectively in a more structured face-to-face format.

In the case that the correlation between \(Y_{iO} - Y_{iF}\) and \(V_{iO} - V_{iF}\) is negative, the relationship between how effective an online course is for a student and whether that student chooses an online course depends on the weights that students assign to human capital investment and consumption value. These weights are represented in the model by \(\alpha_{i1}\) and \(\alpha_{i2}\), respectively. If \(\alpha_{i1}\) is appreciably bigger than \(\alpha_{i2}\) (relative to the scales of \(Y_{iO} - Y_{iF}\) and \(V_{iO} - V_{iF}\)), meaning that student decisions focus on human capital investment, then we would still expect the students who choose online courses to be those who learn the most in them.

Prior research on the relative importance of human capital investment and consumption value in other contexts of educational choice has found that consumption value factors heavily in educational decisions (Alstadsæter 2011; Wiswall and Zafar 2015; Huntington-Klein 2015). If \(\alpha_{i1}\) is smaller than \(\alpha_{i2}\) and \(Y_{iO} - Y_{iF}\) and \(V_{iO} - V_{iF}\) are negatively correlated, then it is likely that students who choose online courses will be on average those who get the least from them.

An analysis of online courses should be interested not just in the average value of \(Y_{iO} - Y_{iF}\), but also how \(Y_{iO} - Y_{iF}\) varies with the propensity to choose an online course. The relationship could plausibly be positive or negative on the basis of the correlation between \(Y_{iO} - Y_{iF}\) and the unknown \(V_{iO} - V_{iF}\). The sign of the relationship is an empirical question that has important implications for policies that broaden access to online education.

Estimation

Our empirical model of later student outcomes, conditional on that student’s taking an online or face-to-face treatment course, takes the form of an endogenous switching model, in which the determinants of success may vary depending on whether the student takes an online or face-to-face course, and which uses a selection equation to address selection bias arising from the fact that students choose which course to take. This model is given by:

$$\Pr (Y_{iO} = 1) =\Phi (X_{iO} \beta_{O} )$$
(4)
$$\Pr (Y_{iF} = 1) =\Phi (X_{iF} \beta_{F} )$$
(5)
$$\Pr (Online_{i} = 1) =\Phi (X_{i} \beta_{S} + Z_{i} \gamma_{S} )$$
(6)
$$Online_{i} = I(Online_{i}^{*} \ge 0)$$
(7)

where \(\Phi ( \cdot )\) is the cumulative distribution function of the standard normal distribution. \(Y_{iO}\) is the outcome of interest, either taking a follow-up course in the same department or graduating with an associate’s or bachelor’s degree,Footnote 6 after student \(i\) takes an online (O) treatment course. Similarly, \(Y_{iF}\) is the outcome of interest observed after student \(i\) takes a face-to-face (F) treatment course. \(X_{iO}\) and \(X_{iF}\) are vectors of student background, prior achievement variables, a constant, and course-by-college correlated random effects for students who took online and face-to-face treatment courses, respectively. Random effects are implemented using a heteroskedastic probit estimator as in Wooldridge (2010), which allows for correlated random effects in a nonlinear context such as the above probit models, avoiding the incidental parameters problem common with fixed effects in nonlinear models.Footnote 7 The use of these random effects accounts for any course-specific effects or shared shocks among students, but does not account for possible sorting of students across community colleges on account of the online offerings. \(X_{i}\) is a vector of the same variables as in \(X_{iO}\) and \(X_{iF} ,\) but it does not condition on whether the treatment course was online. A student’s latent propensity \(Online_{i}^{*}\) to choose between online (\(Online_{i} = 1\)) and face-to-face (\(Online_{i} = 0\)) treatment courses is based on their observed data \(X_{i}\) and excluded variables \(Z_{i}\) which affect choice between online or face-to-face courses but do not otherwise affect performance in a follow-up course.Footnote 8

Issues of self-selection in the context of college education have long been explored in the literature (e.g., Willis and Rosen 1979). We follow Coates et al. (2004) and Gratton-Lavoie and Stanley (2009) in applying this model to the study of online education. This approach allows student selection into online or face-to-face courses to be a part of analysis, and allows student characteristics \(X_{i}\) to have differing effects on outcomes, depending on whether the treatment course was online or offline. Assuming that later performance depends on the quality of experience and the learning done in the treatment course, this allows learning styles, which may adjust more or less well to an online learning environment, to differ among groups.

Parameters of the above model are consistently estimated by using a two-step estimator (Heckman 1979; Lee 1978) which relies on the normality of error terms but does not rely on joint normality as does the maximum likelihood estimator. We make an adjustment from the basic model to allow for a second-stage probit regression. In the first stage, Eq. 6 is estimated using probit. Then Eqs. 4 and 5 are estimated as probit models with the following correction:

$$\Pr (Y_{iO} = 1) =\Phi \left( {X_{iO} \beta_{O} + \frac{{\phi \left( {X_{i} \widehat{{\beta_{S} }} + Z_{i} \widehat{{\gamma_{S} }}} \right)}}{{\Phi \left( {X_{i} \widehat{{\beta_{S} }} + Z_{i} \widehat{{\gamma_{S} }}} \right)}}\delta_{O} } \right)$$
(8)
$$\Pr (Y_{iF} = 1) =\Phi \left( {X_{iF} \beta_{F} + \frac{{ - \phi \left( {X_{i} \widehat{{\beta_{S} }} + Z_{i} \widehat{{\gamma_{S} }}} \right)}}{{1 -\Phi \left( {X_{i} \widehat{{\beta_{S} }} + Z_{i} \widehat{{\gamma_{S} }}} \right)}}\delta_{F} } \right)$$
(9)

where \(\phi ( \cdot )\) is the probability density function of the standard normal distribution. The correction term accounts for sample selection, effectively treating sample selection as a form of omitted variable bias. Using Eqs. 8 and 9, we predict the probability of success in either taking a follow-up course or of graduating, conditional on being in an online or offline course and student characteristics. The selection-corrected average effect of switching someone from a face-to-face course to an online course (the average treatment effect, or ATE) is the average difference between the predicted probability of success conditional on taking an online class and conditional on taking a face-to-face class, corrected for selection (Heckman and Edward 2007):

$$ATE = \frac{ 1}{{N_{O} + N_{F} }}\mathop \sum \limits_{{N_{O} + N_{F} }} \left[ {\Pr \left( {Y_{iO} = 1|X_{i} \widehat{\beta }_{O} } \right) - \Pr (Y_{iF} = 1|X_{i} \widehat{\beta }_{F} )} \right]$$
(10)

where \(N_{O}\) is the number of online treatment courses taken and \(N_{F}\) is the number of face-to-face treatment courses taken. Similarly, the Average Treatment on the Treated (ATT) is the average difference in predicted probability of success, but only for students who took online treatment courses, corrected for selection.

$$ATT = \frac{ 1}{{N_{O} }}\mathop \sum \limits_{{N_{O} }} \left[ {\Pr (Y_{iO} = 1|X_{iO} \widehat{\beta }_{O} ) - \Pr (Y_{iF} = 1|X_{iO} \widehat{\beta }_{F} )} \right].$$
(11)

Standard errors for both the ATE and ATT are calculated using 500 \(m\)-out-of-\(n\) bootstrap iterations. In each iteration, we randomly select without replacement a subsample of size \(n^{.9}\), which is the smallest integer greater than \(n^{.9}\), from the original data of size \(n\). The p values are determined by the proportion of the bootstrap estimates that are above or below zero.Footnote 9

Data

The Estimation Sample and Data

Student data are derived from the Washington State Board of Community & Technical Colleges (SBCTC) as provided by the Washington State Education Research & Data Center (ERDC). We used student transcript, background, and degree completion data on all students who took a course at a public Washington State community college from fall 2008 to summer 2013, with data from 35 different community colleges. For students who attended high school in Washington State since 2001, we also have students’ cumulative high school GPA from Washington State P-210 data, which is a database of high school enrollment information and was linked to the SBCTC data by ERDC.

Excluded variables \(Z_{i}\) come from two sources, both of which use the ZIP code for the student’s current address. In the primary analysis, \(Z_{i}\) is the number of Internet providers in the student’s area, as a proxy for Internet availability and price. We matched each student to the number of high-speed residential Internet providers in their area in each quarter. Data on Internet providers come from the Federal Communications Commission (FCC) website and count providers with at least one residential customer in a given area with a speed of at least 200 kbps in one direction (Federal Communications Commission 2014).Footnote 10 In our alternate analysis for the purpose of comparison to the literature, \(Z_{i}\) is the distance between the ZIP code in which the student lives and the ZIP code of the school, as calculated as the straight-line distance in miles between the centroid of each ZIP code, as well as that distance squared.Footnote 11 We discuss these excluded variables further in the next subsection.

Table 1 shows student information for both the full sample and the analytic sample used for all results, which is limited to treatment courses. A treatment course is one that is available in both online and face-to-face forms,Footnote 12 so that students have the ability to choose between them. We define a course as being a treatment course for a particular quarter if it has at least 20 students enrolled in the online version of the course as well as at least 20 students enrolled in the face-to-face version of the course.Footnote 13 We found that 13.4 % of enrollment is in treatment courses. Observations in Table 1 are at the student/course level, so the averages can be read as being student averages weighted by the number of courses taken. Included in the list of controls are gender, race, English skill, prior military service, whether or not students hold a high school degree, and whether or not they are employed while in school. “Economically disadvantaged” indicates that the student qualifies for need-based financial aid. Cumulative high school GPA is reported for students who attended high school in Washington since 2001; these students are less than one-half the sample.Footnote 14 About 35 % of the sample earned an AA or a BA during their time at the community college,Footnote 15 with an additional smaller group earning high school degrees or certificates at the community college. It should be kept in mind that not all students enter community college with the goal of earning a degree. About 10 % of all courses, and 28 % of treatment courses, are taken online; on average, each student has five or six high-speed Internet providers to choose from.

Table 1 Sample means
Full size table

It is possible that students may choose online or face-to-face courses differently at different stages in their college careers or given their academic aptitudes. As such, we include the cumulative number of credits and the cumulative GPA at the time the treatment course is taken as controls. On average, a student entering a treatment course has taken 52 prior credits (most courses are 5 credits each) and has a college GPA of about 2.5 up to that point.

While grading standards may not be directly comparable, it is also worth checking whether students might expect to get higher grades in one format of the class or another, and whether students tend to be dropping out more in one than the other. We find that the average course grade in an online treatment course is about 0.03 grade points below the mean grade in the same face-to-face course. Among all enrollments in treatment courses, 79 % of students enrolled online and 84 % of students enrolled face-to-face complete the course. The differences in grades across formats are smaller than the conditional differences found by Xu and Jaggars (2013), while the difference in persistence rates is quite similar to their findings.

There are some clear differences between the full and analytic samples. The analytic sample is much younger and much more likely to have recently gone to a Washington state high school. The analytic sample is also somewhat more White, more female, less likely to be working full time, and more likely to have graduated high school. A higher proportion of courses are taken online in the analytic sample. This is not particularly surprising, as most of the courses that are not treatment courses are face-to-face only and not available online. Our results, then, are best representative of younger local community college students who are more interested in taking online courses (among students who take at least one treatment course, 20 % of all courses are taken online). For our results to be generalizable to the full sample of community college students, we must assume that, if the excluded students were to take a treatment course, the factors which predict the choice of an online versus face-to-face version of the course would be similar to those in the sample.

The analytic sample covers a wide range of courses and students. A total of 371,625 students took at least one treatment course and are included in the sample. A total of 1203,254 observations are at the student/treatment course level, covering 9781 course/quarters and 1647 course titles.

Excluded Variables and Sample Restrictions

In the introduction, we outlined potential reasons to be skeptical about the use of distance-to-campus as an excluded variable in analyzing the effect of online courses. Our preferred variable is the number of high-speed Internet providers in a given ZIP code in a given quarter. We follow Vigdor et al. (2014) in using these data as a proxy for Internet availability.Footnote 16 Our identification strategy relies on the assumption that Internet access only affects student persistence through exposure to online courses.

Vigdor, Ladd, and Martinez point out the possibility that additional Internet access may also make students more or less effective in general. In the case of online courses, additional Internet access may make research easier or, instead, provide a distraction. To account for this possibility, in Table 2 we regress grades in treatment courses on the standard list of control variables as well as directly on the number of Internet providers, again using course-by-campus fixed effects for the linear regression. For both online and face-to-face courses, the number of Internet providers fails to predict treatment course grades directly (\(p = 0.533\) for online courses and 0.\(804\) for face-to-face).

Table 2 Instrument exogeneity checks
Full size table

Beyond any direct effects of Internet access on student achievement, we also assume that Internet availability affects students’ decisions about whether to take a course in online or face-to-face formats and not whether to take a particular course at all. If Internet access encourages students to “test out” college or departments outside their program through online courses, we may estimate negative effects on graduation and follow-up course-taking that are driven by selection into the treatment course. We test the viability of this assumption by estimating the first-stage model using measures of college plans and treatment department credits taken before students enrolled in the online course. If the availability of online courses affects selection into the treatment courses, we should find that the Internet provider’s instrument predicts students’ propensity to complete college or their number of total in-department credits prior to taking the treatment course. The first measure we use is a survey of degree completion intentions taken during students’ first term enrolled.Footnote 17 Students are asked about how long they plan to attend and whether they plan to complete a degree. Our second measure is the number of credits students have taken previously in the same department that offers the online course. Because this is a linear analysis, course-by-campus fixed effects are used instead of correlated random effects. In each case, these analyses are still at the student-treatment course level, to match the main analysis.

For the intention to pursue a degree (Table 2 column 2) and for pre-treatment departmental credits (Table 2 column 3), we find that Internet providers do significantly predict the number of credits taken in a given field before taking the treatment course. However, these effects are extremely small and positive. An additional provider leads to an increase in probability of intending to pursue a degree by only 0.2 % points, and an additional one-hundredth of a credit taken in the lead-up to the treatment course. Because we estimate negative effects of online courses on graduating with a degree and the probability of taking a follow-up course, the small positive selection we estimate suggests that a direct relationship between the number of Internet providers and our outcome variables does not explain our results.

Results

The Effect of Online Courses on Retention and Graduation

Table 3 presents the results of the first-stage selection model predicting whether a student takes an online or face-to-face treatment course as well as the second-stage models predicting whether that student takes another course in the same field, and whether the student graduates with an associate’s or bachelor’s degree. In all cases, probit coefficients are presented, and the number of Internet providers is used as an excluded variable, although results are very similar if distance is used instead. See “Appendix” for results using distance as an excluded variable.

Table 3 Predictors of online course-taking, retention, and graduation
Full size table

The first results we consider are the selection equations, in Table 3 columns 1 and 4, which describe which students are more likely to choose an online course, given that they have chosen to take a course available in both online and offline formats. We found that women are more likely to choose online courses, as are older students (the marginal effect of age becomes negative in the late 40s, above the average age of a student). Students with limited English proficiency are less likely to choose online courses, but the full-time employed are more likely to do so, consistent with groups we might expect to receive a particularly small or large amount of consumption value from an online course. Indicators of academic aptitude (high school graduation, grades in high school, and grades in college) are all positive predictors of taking an online course. First-stage results using distance as an instrument are not shown, but results are very similar. Both the number-of-providers and distance variables are highly statistically significant; the number of providers is significant with a Chi squared value of 213, and the distance variables (shown in “Appendix”) are jointly significant with a Chi squared value of 1028. Many groups are more likely to take a follow-up course having taken a face-to-face course than having taken an online course, although these comparisons are somewhat rough, as the scale of the two probit models may not match (Allison 1999). The full-time employed, veterans, and those with higher academic aptitude (as measured by high school graduation, high school grades, or college grades) are more likely, relative to their peers, of taking a follow-up course having taken a face-to-face course than an online course. Exceptions to this are Black students and those with limited English proficiency; these students do relatively better in online courses.

Many groups, such as older students, women, Whites, the English-proficient, the full-time employed, those with higher levels of academic aptitude, and, interestingly, the economically disadvantaged, are more likely than others to obtain an associate’s or bachelor’s degree during their time in Washington State community colleges. Some of these groups tend to be more likely, relative to their peers, to earn the degree having taken an online course rather than a face-to-face course. In particular, the penalty for being Black, Hispanic, a veteran, or having limited English proficiency is smaller having taken an online course than a face-to-face course. On the other hand, women, the full-time employed, older students, and more academically able students compare more favorably to their peers having taken a face-to-face course compared to an online course.

Average treatment effects (ATEs) and average treatment on the treated (ATT), as calculated using Eqs. 10 and 11, are presented in Table 4. These are changes in the probability of taking a follow-up course or graduating with an associate’s or bachelor’s degree, respectively. Students taking online courses are about 6.5 % points less likely to take a follow-up course in the same department, indicating a lack of engagement or affiliation with the field as a result of taking the online course. There is slight negative selection on observables, with the ATT more negative than the ATE by .1 % points, but this difference is not significant.

Table 4 Average effects of online courses on retention and graduation
Full size table

The effect of online courses on graduation follows a similar pattern; that is, there appears to be negative selection on observables into online courses. The ATE of taking an online course is a −1.7 % point difference in the probability of graduating with a degree, and the ATT is −2.0. This difference of .3 % points is statistically significant at the 1 % level and could be considered meaningfully large in the context of a graduation rate. The type of student who chooses to take an online course is not the same type of student who is likely to be guided towards graduation by an online course.

Robustness Checks

Treatment effect estimates for the probability of taking a follow-up course and of graduating with a degree are similar if distance is used as an excluded variable (see “Appendix”). However, the degree and direction of selection changes, with ATEs about .1 % points more negative than ATT. These differences are significant only at the 10 % level for taking a follow-up course and are not significant for graduation. A finding of positive selection when using distance as an excluded variable mimics the results of Coates et al. (2004). The direction of the selection effect is sensitive to the choice of excluded variable. In general, our finding of negative selection is only significant for graduation, and stands against what is found with distance as a source of identifying variation, both here and in the literature. A broader interpretation of selection, in concert with Coates et al. (2004), Gratton-Lavoie and Stanley (2009), and Joyce et al. (2015a, b), is that selection bias, of any direction, is probably small.

The strong negative effect of online courses on taking a follow-up course in the same field leads to the question of whether there are differences in the in-field ability of students who continue, depending on whether they took an online or face-to-face course. Are online students less likely to continue because these courses make it easy for a student to see they have low skill in the field, so the students who do take follow-up courses are likely to be stronger if they came from an online course? This does not appear to be the case. Figure 1 shows the distribution of grades in the treatment course among those who take a follow-up course in the same field and those who do not, and also between those whose treatment course was online and those whose treatment course was offline. While the distributions are statistically different between those who took online or face-to-face treatment courses (the means differ by about .1 grade points for those who took a follow-up course and .01 for those who did not, and a Kolmogorov–Smirnov test rejects equality of distribution in both cases), the distribution of treatment course grades among those who took a follow-up course and those who did not is qualitatively similar for those who took online or face-to-face treatment courses.

Fig. 1
figure 1

Grade distribution in treatment courses by online/face-to-face and whether the student took a follow-up course

Full size image

While our analysis suggests that online courses are less effective in promoting student persistence than the face-to-face format, online courses may further democratize education and expand access to additional students. Unfortunately, we do not see a plausible means within our data of estimating how large this effect might be. However, we should consider the extent to which democratization might affect our result (Rouse 1995). If some students are “democratized”—students who took online treatment courses but would never have taken the face-to-face version in the absence of the online class, and democratized students have poor outcomes, then our estimate of the effect of online courses is negatively biased.

To assess this bias, we assume a share \(S\) of students are democratized, and that democratized students succeed by our measures after taking an online course at a rate \(p\). The observed performance of students after taking an online course is a weighted average of performance for democratized and non-democratized students, and so an ATT estimated using the data will be a biased estimate of the ATT for democratized students. We calculate \(S\) such that the true ATT for non-democratized students is 0. That is, we calculate the share of democratized students necessary to claim that the results we find are entirely bias. Under the simplifying assumption that coefficients on all covariates would have been the same without democratized students in the sample, this is equivalent to solving \(\frac{S - 1}{S}ATT + p = \frac{1}{{N_{O} }}\sum\nolimits_{{N_{O} }} {\Pr (Y_{iO} = 1|X_{iO} \widehat{\beta }_{O} )}\) for \(S\), where \(ATT\) is the sample-estimated average treatment on the treated, and the right hand side is the predicted sample success rate after having taken an online course.Footnote 18 Assuming that democratized students succeed at 80 % the rate of non-democratized students, the ATT among non-democratized students is 0 when \(S = .382\) for taking a follow-up class and \(S = .220\) for graduating with a degree. While our results are sensitive to the possibility of democratization, the share of online enrollees who are democratized would have to be relatively high to eliminate all negative effects, even under what may be a pessimistic estimate of the success of democratized students.

The Effects of Online Courses by Course Type

A question of interest is not just whether online courses in general lead to positive outcomes for students, but which kinds of courses lead to which kinds of outcomes. The policy question of how and when to expand online offerings does not just depend on whether to do so or not, or (as implied by the previous section) how best to target the courses such that the students taking them are those who get the most out of them. In addition, colleges have the opportunity to expand access in particular fields which seem to have the most success in encouraging positive outcomes.

In this section, we repeat the analysis from the previous section while limiting the sample, in turn, to the five most popular department typesFootnote 19 among treatment courses: English (221,049 treatment course enrollments, 21.5 % of which are online), mathematics (207,356, 18.9 %), Psychology (112,384, 26.7 %), Sociology (52,715, 27.8 %), and Communications (41,345, 30.8 %). We present the results for all five top department types, although the English results are identified only on the basis of the nonlinearity in the model: the first stage Chi squared test statistic for the number of Internet providers is only 0.38 in English. In the other fields, the number of Internet providers is about as strong a predictor as it is in the full sample, with smaller sample sizes leading to Chi squared values of about 16 (psychology and sociology) to about 40 (mathematics and communications).

Table 5 shows ATE and ATT estimates by department type. The effect of online courses on taking follow-up courses in the same field varies widely by field. In all cases, the effect is less negative than for the full sample, suggesting that much of the negative effect is concentrated among smaller departments. The degree of selection on observables differs here compared to the full sample as well. Negative selection is strongest in sociology and psychology. There is evidence of positive selection in mathematics and the dubious English estimate. Among the top five department types, the effect on taking a follow-up course is strongest in mathematics and weakest in sociology; the ATT is slightly positive in sociology.

Table 5 Average effects of online courses on retention and graduation by treatment course type
Full size table

The effects on graduation do not vary as widely by department type as the effects on follow-up courses do. However, differences of even a few percentage points in graduation probability are meaningfully large. In each department, the main model result of a negative effect of about −2 % points holds, and the negative selection on observables holds, although it is not always significant, except in psychology and the untrustworthy English estimates. Online courses seem to do the best in mathematics, with an ATE of −1.4 % points and an ATT of −1.8, the least negative among the department types with strong first stages. Communications is on the other end of the scale, with an ATE of −3.0 and an ATT of −3.7. Communications also exhibits the strongest negative selection on observables.

Conclusion

Using regional and longitudinal variation in the number of high-speed Internet providers as a novel source of identifying variation, we estimate the effects of online courses, as compared to face-to-face courses, at Washington State community colleges. While of the degree and direction of selection on observables is sensitive to the choice of excluded variable, treatment effect estimates are robust to the use of distance to campus—as is common in the literature—or the number of high-speed Internet providers. This result both suggests a novel source of variation and addresses some potential concerns about the use of distance as an excluded variable in this context.

We found discouraging results on the effectiveness of online courses. The average student is less likely to continue in the field or earn a degree if he or she takes an online course rather than a comparable face-to-face course. Students who actually choose to take online courses also fare worse in them than would the average student. In short, some students see online courses as a preferable learning opportunity, perhaps because online courses do not require as many campus visits or offer a flexible schedule. However, this choice is not likely to keep the student in the field or on track to a degree.

The effectiveness of online courses varies by department. Evaluating the five most popular departments, we find that online mathematics courses, while they have large negative effects on taking follow-up mathematics courses, have the least-harmful effect on graduation (ATE −1.4 % points). Online Sociology courses seem to be the most successful in encouraging students to take follow-up sociology courses (ATE 1.2 % points). Differential effectiveness across departments indicate that different topics have varying success in translating their pedagogical approaches to the online environment, and suggest that selective, rather than wholesale, online adoption may be advisable.

These results have important implications for the study of online courses and policy surrounding their implementation. The study of online courses cannot ignore the issue of selection in a real-world context. While we find evidence of only a small amount of negative selection, the selection we do find will somewhat bias experimental studies of single courses with random assignment. The fact that our results differ from Coates and colleagues’ results suggests that the degree and direction of selection may not be consistent across settings and is likely sensitive to the choice of excluded variable(s). We suggest that future experimental research should attempt to identify whether particular subsets of students are best served by the experimental course on offer and also whether those same students tend to be the ones who would choose the course in a non-experimental scenario, as indicated by behavior in a non-experimental setting, or stated preference surveys before the experiment takes place. Joyce et al. (2015a) provide and execute on a useful framework for carrying out such experiment.

Community college policy concerning online courses must take into account the potential for significant losses in educational outcomes. In the setting described in the paper, the students took online courses saw worse learning outcomes and were less likely to persist in the field or to graduation than if they had selected the face-to-face version of the same course. Other quasi-experimental literature also finds negative effects on test scores, follow-up grades, and the probability of completing the course (Coates et al. 2004; Xu and Jaggars 2013; Krieg and Henson 2015). A 2 %-point decrease in the probability of graduation, from a base completion rate of 36 %, as a result of taking a single course is meaningfully large and should be considered when setting policy. The possibility that these effects may compound over multiple courses is outside the scope of the paper, but the effect of a single class alone is already large.

Given that completion rates are already a matter of concern at community colleges, it is worrying that possibly the largest recent change in the way that community colleges operate, the addition of online courses, is unlikely to improve completion figures. Our results imply that the implementation of online courses should be done carefully. Implementation should be considered on a course-by-course basis, and when considering previous empirical results on efficacy, administrators should consider whether student selection may have affected results, and whether the study setting in which analysis took place is representative (Allcott 2015).

These deficits in educational outcomes must be weighed against the possibility that some new students will take online courses who would not ever have taken a face-to-face course. Online courses offer the potential for a major increase in access to college education. However, as currently implemented in Washington State community colleges, these classes are not the equal of the standard face-to-face curriculum they may guide some students away from. Results from other contexts, as we have reviewed, back this up and routinely find worse outcomes in online settings. Online courses do not improve student learning or engagement with the field or college enough to convince students to continue in the field or towards graduation, compared to face-to-face classes. Policymakers and college administrators making moves to expand online access must balance the improvement in access that online courses offer against the result that online classes cannot simply be added with the assumption that they are as good as face-to-face courses.

Importantly, our results apply to general policies towards online courses in community colleges, rather than to any particular course. Online course offerings are ever-evolving and highly diverse, and the effectiveness of any particular course will depend on the curriculum, the teacher, and the audience. In this paper, we examine a wide swath of courses all at once. The effect we estimate is a generalized result of increasing the online offerings at Washington State community colleges, where development of online courses and support for students were strong (Xu and Jaggars 2011). We do not offer evidence that online courses cannot be the same or better than face-to-face courses; rather it seems that online courses do not appear to be better in the implementation we study and with the current available curricula. The negative results found suggest likely problems with increasing the selection of online offerings. This is of particular concern because so many colleges are taking the approach of expanding their online offerings. Community colleges should be extremely careful moving into a more heavily online curriculum, or they will risk diminishing their capacity to deliver a quality education.