Success in Community College: Do Institutions Differ?

Abstract

Community colleges are complex organizations and assessing their performance, though important, is difficult. Compared to 4-year colleges and universities, community colleges serve a more diverse population and provide a wider variety of educational programs that include continuing education and technical training for adults, and diplomas, associates degrees, and transfer credits for recent high school graduates. Focusing solely on the latter programs of North Carolina’s community colleges, we measure the success of each college along two dimensions: attainment of an applied diploma or degree; or completion of the coursework required to transfer to a 4-year college or university. We address three questions. First, how much variation is there across the institutions in these measures of student success? Second, how do these measures of success differ across institutions after we adjust for the characteristics of the enrolled students? Third, how do our measures compare to the measures of success used by the North Carolina Community College System? Although we find variation along both dimensions of success, we also find that part of this variation is attributable to differences in the kinds of students who attend various colleges. Once we correct for such differences, we find that it is not possible to distinguish most of the system’s colleges from one another along either dimension. Top-performing institutions, however, can be distinguished from the most poorly performing ones. Finally, our adjusted rates of success show little correlation either to measurable aspects of the various colleges or to the metrics used by the state.

Background

Community colleges have assumed an increasingly important role in this country’s postsecondary education since World War II, as American employers have demanded workers with enhanced technical skills. In recent decades, community colleges have added to this role in technical training a growing emphasis on providing a stepping stone to baccalaureate degrees (Dougherty and Townsend 2006). In addition, community colleges have expanded their mission to include short-term training programs designed to serve the interests of local business, courses to enhance the skills of adults, programs to allow high school dropouts to obtain a high school equivalency degree (a GED), and programs intended for recent high school graduates interested in gaining skills for a job or preparing for further education.

As a proportion of total college enrollment, public 2-year institutions have grown steadily, reaching 15 % of all enrollees in 1963 and 34 % by 2010.¹ Among the states, North Carolina, which provides the context for the current study, has been a leader in the development of and reliance on community colleges. Compared to that 34 % share for community colleges in the nation, in North Carolina they accounted for 43 % of all postsecondary enrollments in 2010.² In 2009 the Obama Administration highlighted the central role of community colleges when it announced what it termed the American Graduation Initiative, an effort to “reform and strengthen community colleges” (U.S. White House 2009). Community colleges are also the focus of reform efforts funded by private foundations, such as Achieving the Dream, a nonprofit organization “dedicated to helping more community colleges succeed.”³

Efforts to strengthen what community colleges can contribute to more effective education for the nation’s workforce also raise the central question of how to evaluate their performance. In its 2006 report, a commission put together by Secretary of Education Margaret Spellings castigated higher education for ignoring technological advances in teaching and urged the development of “new performance benchmarks designed to measure and improve productivity and efficiency” (U.S. Department of Education 2006, pp. 14, 19). Merely adopting the approach now accepted at the K-12 level—assessment regimes based on standardized tests—is widely viewed as impractical for postsecondary institutions. Yet the Spellings Commission spoke for many observers in calling for the development and use of measures that would make it possible to “weigh and rank comparative institutional performance” (U.S. Department of Education 2006, p. 20). For example, an institution’s graduation rate could be used as a measure of its ability to produce successfully-trained graduates, ready to enter the labor market or continue with further training.

Congress in effect endorsed this indicator of quality when, in the 1990 Student Right to Know Act, it mandated that postsecondary institutions report graduation rates.⁴ The Act required that institutions calculate and disclose a precisely and uniformly defined graduation rate: the percentage of first-time, full-time students who graduate within 150 % of the “normal” completion time for a degree at the institution where they first enrolled. In the case of 4-year institutions, for example, this is the percentage of first-time students who graduate within 6 years of first enrolling. When Congress was debating the bill, in 1990, Senator Edward Kennedy argued in favor of such quantitative outcome measures, stating that transparency would drive improvement, “Sunlight is the best disinfectant. Once colleges begin disclosing this vital information, those with the poorest records will be under the greatest pressure to improve.”⁵

One nagging worry about using the graduation rate as an outcome measure is that some institutions enroll students with much stronger educational backgrounds than others, giving those institutions a built-in advantage in achieving high graduation rates that might have little to do with their own effectiveness in educating students. This worry is especially acute for community colleges. Compared to 4-year colleges and universities, community colleges serve a more diverse population with many students attending part time and trying to balance school, family, and work obligations. Community colleges also provide a wider variety of educational programs than do most 4-year institutions. Not only do they offer 2-year associate’s degrees, they provide course work for students hoping to transfer to 4-year institutions. In addition, they offer a smorgasbord of course offerings ranging from specialized certificates and other vocational training to general-interest courses emphasizing avocation more than vocation.⁶ As a stepping stone to 4-year college, their role has been debated vigorously (Brint and Karabel 1991). What is clear, however, is that community college students have high dropout rates, low graduation rates, and long periods for completing degrees.

In this context, marked by both anxiety and ambition regarding realization of educational goals by way of community colleges, policy makers and researchers are struggling to develop appropriate measures of educational success. Some of the prior research on community colleges has focused on students; on their individual trajectories and the personal challenges they face in completing degrees. In contrast, our emphasis in this paper is on success at the institutional level. We ask whether there are important differences across community colleges in North Carolina that make personal success for students more likely at some colleges than at others.

It is worth noting that, for more than a decade, leaders in the North Carolina Community College System (NCCCS) have been using quantitative measures of institutional effectiveness, thus anticipating some of the contemporary interest in output metrics (Clotfelter and Charles 2012). These measures are published in an annual report, Critical Success Factors. Under pressure from the state legislature, the community college system in 1999 adopted a dozen explicit performance measures. In 2007 the list was shortened to eight “core indicators of student success”, among which is the graduation rate as required and defined by the Student Right to Know Act.⁷ Although well-intentioned, these legislative efforts to monitor community colleges provoke the worry noted above; colleges that appear to be the most successful along any of these dimensions may be the ones who enroll the best prepared students, rather than those that educate students the most effectively.

Our purpose in this paper is to refine the comparison of institutional effectiveness, the objective embodied in such assessment efforts as the federal Student Right to Know Act and North Carolina’s metrics, described above. In the present study, we focus exclusively on curriculum programs, which are intended primarily for recent high school graduates. We use measures of individual student success to create measures of institutional success for each North Carolina community college. Specifically, we base our measures of institutional success on two measures of the success of individual students in each community college: (a) the attainment of an applied diploma or degree and (b) the completion of coursework required to transfer to a 4-year college or university. We address three questions. First, how much variation is there across institutions in our measures of institutional success? Second, how much of this variation is attributable to the characteristics of the students enrolled? Third, how do our measures compare to those used by NCCCS?

Although we do find evidence of potentially important variation in success rates across colleges, particularly once we adjust for observed student characteristics, our measures of success are statistically imprecise. Conventional F tests permit us to reject the hypothesis that all observed variation across colleges is attributable to sampling error, but our test statistics are driven by a relatively small number of outlier colleges. We further find that our measures of success are poorly correlated with metrics used in official NCCCS publications. The results illustrate how difficult it is to estimate educational effectiveness in community colleges.

The “Conceptual Issues in Measuring Institutional Success of Community Colleges” section of the paper discusses the concept of institutional success with attention to the challenges of measuring it, the “Data and Methodology” section describes our data, and the “Quantifying Variation in Success Rates Across Colleges” section quantifies institutional success for most of the state’s 58 community colleges. In the “Explaining Variation in Institution-Level Success Rates” section, we ask whether any easily observed characteristics of the institutions themselves can account for the variation in success and in the “Correlations” section we examine correlations among four measures of success. The “Conclusion” section concludes the paper.

Conceptual Issues in Measuring Institutional Success of Community Colleges

In this section we briefly explore the concept of institutional success for a community college and discuss how measures of success may be operationally defined.⁸ Like all forms of education, community college represents an investment, by the student and the public taxpayers who subsidize that education, in return for future benefits that will accrue to the student or to the broader society. Some of these benefits are decidedly private; they go to the student in the form of access to higher paying, more rewarding jobs as well as the opportunity to pursue further education. They also include a host of non-pecuniary benefits, from better health to happier marriages (Oreopoulos and Salvanes 2011). The wider community gains as well, enjoying the benefits of a stronger and more flexible local economy that comes with a well-trained local labor force and potential savings in the form of lower expenditures on public services such as health care or prisons.

Ideally, one might compare the success of one community college to another based on the magnitude of the benefits each generates. Measuring these benefits is difficult, if not impossible, for a number of reasons. First, the very definition of success is ambiguous because of the varied roles that community colleges play. In particular, there is dissatisfaction with the recent policy of evaluating colleges on degree attainment because many students enter not with the goal of getting a degree, but rather of obtaining the course credits needed to transfer to 4-year institutions. Another confounding issue is that students may complete all or most of the requirements for a diploma or certificate but not actually apply for this credential if it is not required for a job. A second challenge in measuring success is the absence of good data on many of the outcomes of interest. Success in the labor market, for example, can in principle be measured through careful analysis using earnings data, but data of the required detail and quality are often not available. Third, measuring success is complicated by the problem of attribution. Even if some of the outcomes, such as higher wages, could be measured, it would be difficult to determine how much of the additional wages are attributable to the education provided by the community college, how much to the background characteristics of the student, and how much to the vitality, or lack thereof, of the local labor market.

For these reasons, analysts and policy makers have little alternative but to rely on one or more imperfect proxy measures for success, each of which has strengths and weaknesses. These proxies may be of three types: direct market outcomes, measures of student progress in the form of graduation rates or credits received, and input measures.

In its Critical Success Factors report (NCCCS 2010), the community college system uses the first of these approaches: direct measures of the employment success of community college completers. For example, the NCCCS uses as one measure, the percentage of community college completers who are employed within 1 year of last attendance. Another measure is the percentage of a sample of businesses that employ individuals trained or educated by a community college who indicated that they are satisfied with the quality of those employees as that quality is related to the training or education provided by the community college.

The advantage of such measures is that they directly reflect the types of benefits the community colleges are trying to produce. The disadvantages include: the fact that they represent only a portion of the total benefits generated; the satisfaction measure is much better suited to programs providing specific training to a well identified group of workers than to the general education programs of the community colleges; the evaluation data may be expensive to compile; and, these measures suffer from the attribution problem mentioned above.

The second approach to measuring institutional success is to look at students’ progress through their required courses of study, with a particular focus on graduation rates, as promoted by the federal government.⁹ The main advantages of such progress or graduation measures are their apparent simplicity and their parallel to the graduation rates for 4-year institutions. Using the 150 % metric accounting period alluded to earlier, the time frame would be 6 years for 4-year institutions and 3 years for 2-year associate’s degrees at community colleges. But graduation rates are flawed as a measure of community college success in a number of ways, some of which have been highlighted by an effort of six states to pilot a better approach (Achieving the Dream 2008). Among the flaws of the graduation rate is that it does not track the many part-time students enrolled in community colleges, does not take into account that a major mission of many community colleges is to give students an opportunity to transfer to a 4-year college, and allows too little time for graduation given the challenges that many community college students face in balancing school, family, and possibly work obligations.

Although the attribution problem also arises for this approach, it is easier to address than is the case for the market outcome approach, provided data are available on the background characteristics of the students. Specifically, one would want to adjust any measures of graduation rates or persistence through college for student background characteristics that are predictive of student success. If one did not do so, community colleges that served large proportions of economically disadvantaged students, students with low academic ability, or students who attended weak high schools, would typically look less successful than colleges that served more advantaged students.

The third approach is simply to measure institutional quality by the quantity and quality of its inputs relative to the number of students served. By this proxy, community colleges with similar enrollments that have more resources, more highly qualified faculty, or more student support services would be judged higher quality than those with fewer resources. But even this apparently straightforward measure would be difficult to implement. One problem is that community colleges offer differing combinations of programs with differing resource requirements, leading to inappropriate comparisons of apples and oranges. Another problem is that the focus on inputs provides no information on how effectively they are deployed toward the desired goals. Finally, any measure of this type would need to be adjusted for the types of students enrolled. Students who require remedial courses, for example, may put greater demands on the community college than other students. The one advantage of this input approach is that it avoids the attribution problem associated with a measure based on outcomes.

In the “Quantifying Variation in Success Rates Across Colleges” section, we use a variant of the second approach, evaluating student progress through required courses of study, to measure the relative success of the community colleges in North Carolina. In using this approach, we take into account the two important and distinct educational functions pursued by most community colleges with regard to the recent high school graduates. The first function is the preparation of students directly for the workplace, through applied training that leads to diplomas and certificates as well as 2-year associate’s degrees. The second, and rather different, function is preparing students for further education by way of transfer to a 4-year college or university. We devise measures of success based on each of these two functions: a measure of success in applied training, calculated in terms of applied degrees or diplomas, and a measure of success in terms of readiness for transfer, calculated in terms of associate’s degrees or transferable credits earned. We use these gauges of progress for measuring community college success, in part because we do not have access to the labor market outcome data that would be required for the first approach. Furthermore, even if we had such data, addressing the attribution problem would be a challenge. Fortunately, our ability to link community college students with their school records, as we describe in the next section, means that we have good information on the student background characteristics needed to address the attribution challenge that arises with success measures based on progress through college.

Data and Methodology

The data for this study refer exclusively to the community college system in North Carolina. As was the case across the United States, community colleges in North Carolina began springing up shortly after World War II. By 1957 the state had established two publicly funded postsecondary systems in addition to its 4-year institutions: one composed of industrial education centers and one made up of 2-year junior colleges emphasizing arts and sciences. Six years later, the two systems were consolidated into a unified Community College System that, by 1979, had grown to encompass the 58 institutions in existence today. The 58 community colleges in the North Carolina Community College System (NCCCS) offer a wide range of programs, organized under broad categories. These are defined as: continuing education, comprised primarily of non-credit courses; specialized programs, targeting economic opportunities in the community; and curriculum programs, involving courses taken for credit toward the associate’s degree, diploma, certificate, or college transfer. The current study focuses exclusively on the success of the institutions’ curriculum programs, which in 2009 accounted for approximately 37 % of community college student enrollment state-wide.

Between 1998 and 2009, enrollments in the community college system increased by 47 %, as shown in Fig. 1. By comparison, enrollment at the 16 4-year colleges and universities in the University of North Carolina system was lower but grew at about the same rate.

Fig. 1

Enrollment trends in North Carolina Community Colleges and in the University of North Carolina System. Community college numbers refer to unduplicated headcounts of students enrolled in curriculum programs during the academic period ending in the spring of each year, UNC numbers refer to headcount of degree-credit students enrolled in the fall semester of each year. Sources: Snyder and Dillow (2010); University of North Carolina General Administration (2009)

The present analysis takes advantage of our access to data that allow us to follow individual students over time as they progress from high school to community college. For that purpose, we use data from both the North Carolina public school system and the North Carolina Community College system that were merged and prepared by the North Carolina Education Research Data Center (NCERDC) at Duke University.¹⁰ We focus on students enrolled in the curriculum programs of NC community colleges between fall 2001 and spring 2009.

The analyses presented in the current paper are based on information for one cohort of students—those who were enrolled in a North Carolina public school and took the state’s 8th grade End of Grade test in math in 1999 and who first enrolled in a North Carolina community college curriculum program any time between fall 2003 and fall 2004. If these students made normal progress in subsequent years after eighth grade, they would have been in 12th grade in 2002/2003 and would have graduated from high school in 2003. Since some students are retained in grade, we include in the analyses those students who graduated from high school in either the spring of 2003 or 2004.

We restrict ourselves to a single cohort because we wish to take advantage of the linked student records while assuring that we have sufficient data to construct measures of successful community college outcomes. Of the 89,201 students who took the statewide End of Grade math exam as 8th graders in 1990, we have data on 14,936 who subsequently enrolled in one of the state’s 58 community colleges. Of that total, we excluded 1,900 students enrolled in programs not leading to a formal college-level award, such as those completing a high school degree through college coursework. In addition, we had to further limit the study cohort by excluding students who attended high school in Wake or Mecklenburg counties, or who first registered in a community college in Wake or Mecklenburg counties, because of incomplete data for those two institutions. The resulting sample contained 11,111 students.

The student-level variables included in our statistical analyses are described in Table 6, and their means and standard deviations are shown in Table 1. While the public school information is complete, and we are reasonably certain that we also have complete community college course information for our students, we are still missing some degree information related to program completion. This is true because our degree completion totals do not include diplomas for which students, even those who had completed all the requirements, did not officially apply. Also, while students may move between colleges, we do not take this movement into account; that is, we assume that students remain in the community college where they first registered. Table 7 contains descriptions of the community college institution-level variables that are included in the analyses described in “Quantifying Variation in Success Rates Across Colleges” and “Explaining Variation in Institution-Level Success Rates” sections.

Table 1 Summary statistics

Quantifying Variation in Success Rates Across Colleges

We adopt a two-dimensional measure of institutional success: “applied” success and “transfer” success. The first measure includes students who obtained, within 4 years of enrolling in their first course, a diploma or applied associate’s degree in any one of the vocational areas offered in the state’s community colleges. Examples include the diplomas in culinary arts and hospitality management, as well as the Associate’s Degree in Applied Science in fields such as criminal justice technology or early childhood education. The second measure of success includes students who attained an associate’s degree (in arts, science, fine arts, or general education) or successfully completed 10 transferable courses (approximately 30 transferable credits), also within 4 years of their initial enrollment. Because students need not complete a degree in order to transfer to a 4-year college or university, it is important to count as successful a student whose progress in courses completed made transferring a reasonable option, even if no associate’s degree was obtained. Some students, in fact, achieve success by both measures. As shown in Table 1, about 10 % of the students in our sample cohort succeeded in obtaining an applied degree or diploma within 4 years of enrolling (our definition of applied success) and 23 % succeeded in earning credit for at least 10 transferable courses (our definition of transfer success).

Figure 2 shows, for each of the state’s community colleges, the percentages of the 2003–2004 entering cohorts who were successful according to one of the two measures. The figure shows that the success rate for the applied outcomes in the state’s community colleges ranged roughly from 5 to 30 %. For our measure of transfer success, the rates ranged from about 8 to 35 %. If these measures of success are taken at face value, the graph in Fig. 2 suggests both that institutions appear to specialize and that they may also differ in their effectiveness. Specialization would be suggested by a negative slope of the points. That is, some community colleges—those near the top and left, such as college 894—have high transfer success rates but low success in applied degrees. Others, such as those on the right and bottom, were high in applied success but low in transfers. One college that stands out on the applied dimension, but is only middling on the transfer measure, is college 846. To the extent that colleges are arrayed southwest to northeast on the graph, however, there is the strong presumption of differences in effectiveness, with those to the northeast dominating those to the southwest on both criteria.

Fig. 2

Success rates for transfers and applied outcomes, N.C. Community Colleges. Success rates refer to the proportion of students in each community college who succeed by either or both of our definitions of applied and transfer success. See Table 2 note

Yet these comparisons are surely flawed as measures of institutional effectiveness in at least one regard. To the extent that the entering students attending some colleges are academically stronger or financially better off, those colleges would appear to have a natural advantage in achieving higher rates of success than colleges with less prepared or economically disadvantaged students. To remedy this measurement flaw, we control for the differences in the “inputs” at each institution using information from the detailed administrative records on the middle and high school characteristics of students that are likely to be predictive of their subsequent success in community college.¹¹

Table 2 presents two sets of logit estimates using our sample of 11,111 individual students, one for success at applied outcomes and one for success at transfer outcomes. Each employs a dichotomous variable Y_ij, indicating success (Y_ij = 1) or lack of success (Y_ij = 0) for individual i. Where X(i) is a vector of covariates described above for individual i attending community college j, the equations are of the form:

$$ {\text{Prob }}\left( {{\text{Y}}_{\text{ij}} = { 1}} \right) \, = {\text{ e}}^{{{\text{X}}\left( {\text{i}} \right){\text{b}}}} / \, \left( { 1 { } + {\text{ e}}^{{{\text{X}}\left( {\text{i}} \right){\text{b}}}} } \right) \, $$

(1)

Table 2 Regression of success outcomes on student characteristics

The estimates indicate consistent and predictable associations between student success and a number of their personal characteristics. First, a student’s 8th grade math end-of-grade test score is predictive of higher success along both dimensions. Second, females are more likely than males to be successful by either metric. Third, students whose parents graduated from college are less likely to be successful at applied outcomes, but more likely to be successful at transfer outcomes than students whose parents terminated their education after graduating from high school (the omitted category for this categorical variable). Fourth, students who were ever eligible to receive free or reduced price lunch, which is commonly used as a measure of low family income for students within the K-12 system, were less likely to be successful than those with higher family incomes. The estimated equations also account for differences by race and ethnic group. Compared to white students (the omitted category), black students were less likely to be successful by either measure, and Asian students were less successful in applied outcomes.

In the analysis below, we control for these personal characteristics to produce measures of institutional success that statistically neutralize their influence. Thus, our adjusted institutional success measures are not influenced by the relative affluence, racial or ethnic identification, or academic preparedness of a college’s student population. Using these estimated equations, we statistically control for some of the most important differences in the students who attend the various colleges by calculating the predicted probabilities that each student will achieve individual success in the applied or in the transfer realm. The difference between those predicted probabilities and the actual outcomes (measured by a 0 or 1 on both scales) for each student serves as the basis for our adjusted measures of institutional success—measures that statistically control for the measured characteristics of students who attend each college. Specifically, we calculate these residuals for every student in our cohort and then average them by community college.¹² Estimates from Eq. (1) are used to calculate, for each community college j, an adjusted success rate, which is the mean for all students of the difference between the actual success outcome (Y_i = 0 or 1) and the predicted probability, based on the estimated coefficients and that individual’s own values for the covariates:

$$ {\text{Adjusted success}}_{\text{j}} = {\text{ Mean}}_{\text{j}} \left[ {{\text{ Y}}_{\text{ij}} - {\text{ e}}^{{{\text{X}}\left( {\text{i}} \right){\text{b}}^{*}}} / \, \left( { 1 { } + {\text{ e}}^{{{\text{X}}\left( {\text{i}} \right){\text{b}}^{*}}} } \right)} \right], $$

(2)

where b* is a vector of estimated coefficients from (1). The resulting averages, calculated for all our community colleges, can be thought of as an input-adjusted index of institutional effectiveness. A college whose students over-achieve (by being successful more often than what would have been predicted based on their characteristics alone) will have a positive mean residual. A college whose students succeed less often than would be predicted will have a negative residual.

We think of these mean residual scores as adjusted college effects on student success rates, and believe they represent one reasonable indicator of institutional effectiveness. We have calculated such adjusted effects for each college and for both categories of success and have arrayed them graphically in Fig. 3. In comparison to the pattern shown by raw success rates in Fig. 2, there is less in the adjusted college effects to suggest specialization. An exception is college 846, which is at the mean in transfers but 0.15 above the mean for applied outcomes. For the most part, though, colleges simply differ by effectiveness on both fronts: some colleges appear to excel at both kinds of outcomes while other colleges look like under-achievers along both dimensions. Differences in adjusted college effects are quite large, implying ranges of up to 25 % points on both measures. Among the best on both measures are colleges 880, 806, and 896. On the other side of this coin are colleges whose adjusted effects on success rates are below-average on both scales, such as colleges 843, 870, and 844.

Fig. 3

College-specific effects on success rates, adjusted for student characteristics. Values represented by points are differences from the mean, for each community college, in applied and transfer success, after adjusting statistically for differences across colleges in characteristics of students. These differences are equivalent to the residual success rates not statistically explained by characteristics of students attending each college. See Table 3 note

Although the evidence points to differences across community colleges in the adjusted effects on student success, two qualifications are worth noting. The first is that some of the measured variation across colleges is likely due to chance rather than actual differences in effectiveness. This fact is captured in the standard errors of the estimates of the applied residuals, which are on the order of 0.05 for applied success and 0.06 for transfer success. These values imply that differences between adjusted college effects of less than about 0.10 or 0.12 (two standard deviations) are not statistically different from each other. Based on an F test, we can reject the hypothesis that the observed variation in adjusted transfer or applied effectiveness across institutions can be attributed entirely to random variation. The F statistic, however, is driven primarily by a small number of colleges with adjusted college effects at some distance from zero. The majority of colleges cannot be statistically distinguished from one another along either dimension.¹³

By way of further caveat, additional unobserved student characteristics might provide an alternative explanation for the patterns we observe, though we think this is unlikely. A more likely explanation is that the programs offered by community colleges differ in ways we have not captured in our analysis. For example, it could be that some community colleges prepare a higher proportion of their students for jobs where the actual diploma or certificate is required than is the case for other community colleges. These possibilities notwithstanding, we turn in the next section to ask whether successful community colleges, identified in this two-dimensional way, have any particular characteristics in common.

Explaining Variation in Institution-Level Success Rates

Our goal in this section is to examine the extent to which any characteristics of the community colleges themselves might be statistically associated with our estimated adjusted college effects. To do so, we regressed both of our unadjusted success measures and both of our measures of adjusted college effects on a number of college-level covariates (defined in Table 7). These efforts produced little in the way of statistical association. The regressions in Table 3 have, as dependent variables, the two raw measures of success. Those in Table 4 are based on adjusted college effects; that is, college effects on students, holding constant the characteristics of the students.

Table 3 Regression of unadjusted success rates on institutional characteristics

Table 4 Regression of adjusted success rates on institutional characteristics

The first of the institutional characteristics is the college’s proximity to a campus of the University of North Carolina. This dichotomous variable takes on the value of 1 if a UNC campus is in a county served by the community college and 0 otherwise. Since many students may well see 4-year institutions as substitutes for 2-year ones, especially when access to them is comparable, we reasoned that this measure might be associated with our measures of success. Indeed it was: both measures of success rates, unadjusted and adjusted, were negatively associated with this proximity measure. The association was statistically significantly in all but the equation for adjusted applied success. This finding is consistent with the following explanation. If a student plans at the outset to attend a 4-year institution, he or she may be likely to head there right away if the costs—both out-of-pocket and time—are not too burdensome. If the students with a 4-year degree in mind tend to head directly to the local 4-year institution rather than to the local community college, the remaining community college students will be less likely to try for transfer credits than similar students living farther away from any 4-year institution.

All but two of the remaining institution-level variables are statistically insignificant as explanatory variables in both tables. Community colleges with more than one campus probably increase access, which could make completing a course easier, but which might also encourage enrollment by marginal students. In any case, the coefficients are not significantly different from zero. The coefficients on expenditures (which refer to curriculum programs alone) per full-time equivalent student (FTE), for which we anticipated positive coefficients on the ground that more resources might mean smaller classes and better support services, are also statistically insignificant. To account for the possibility of economies of scale, we included the log of total enrollment. Again, there was no statistically significant coefficient. A fifth explanatory variable, also not significantly associated with the success measures, is the portion of students who are taking curriculum courses, as opposed to high school or adult education courses.

To capture any effects of specialized faculty training, we include the ratio of faculty and staff who participated in a state-funded professional development program to FTE enrollment.¹⁴ This variable is statistically significant for transfer success in our unadjusted measure but insignificant in our adjusted measure; that is, after we control for the characteristics of the students. Lastly, we include two indicators of customized training—the number of companies with customized training arrangements and the number of trainees per FTE.¹⁵ Only the last of these had a significant coefficient—a negative one for applied success. We have no intuition to explain this result.

Correlations

As noted above, the North Carolina Community College System has made it a practice of evaluating its constituent colleges using quantitative measures of effectiveness. It is a matter of practical importance to know how the measures of institutional success based on degree and transfer readiness developed in this paper, compare to these NCCCS measures. As a partial answer to this question, we present correlations between our measures, four in all, with two of the principal measures used in the NCCCS’s report Critical Success Factors 2010. The correlations are shown in Table 5. The first of those measures is the percentage of graduates from the community college who transferred to a UNC campus and were earning a GPA of at least 2.0 after two semesters at that UNC campus. The second is the passing rate in the community college’s math developmental courses. As is readily apparent from the table, the correlations between these two published measures and our measures, with or without adjustment for student characteristics, are very modest. The only two correlations that are statistically significant are between the math passing rate and our two unadjusted success rates.

Table 5 Correlations among alternative measures of institutional effectiveness

These generally low rates of correlation suggest that the community college system’s published measures must either be capturing very different qualities than our success measures or that some of the measures, including our own, are subject to considerable measurement error. These low correlations are sobering and support a healthy dose of humility in our efforts to come up with useful institution-wide outcome measures. That said, the efforts by the state community college system to come up with objective measures of outcomes is a commendable step in the right direction.

Conclusion

This paper asks whether it is possible to construct meaningful measures of the overall effectiveness of community colleges. We define two different measures: one based on obtaining proficiency in an applied field of study and one based on earning transferable credits. We find that the proportion of students who succeed according to either or both of these measures (within 4 years of first enrolling) varies widely across community colleges. Insofar as they are based on course completion or degree attainment, these measures have much in common with the most widely used “outcome” measure used in higher education—graduation rates. As an outcome measure, graduation rates have the virtues of being readily understood and fairly easy to compute. Like graduation rates, however, our unadjusted measures also could be quite misleading if no account were taken of students’ pre-college preparation. Therefore, to account for differences across institutions in the preparation, aptitude, and resources of their students, we statistically correct for differences in several student-level predictors of success. The unexplained residuals, analogous to institutional fixed effects, yield measures of institutional effectiveness corrected for differences across colleges in the academic readiness of students. We call these measures adjusted college effects. Although these measures also show a great deal of variation, they are measured fairly imprecisely, implying that we cannot statistically distinguish among most campuses in the state. At the extremes, however, the differences are statistically significant. Hence, state leaders may want to look carefully at the top and bottom performers on our adjusted success score to see what factors may account for these extreme outcomes. Lessons learned from those cases could inform policies across the board.

When we seek to relate our adjusted college effects to published measures of institutional characteristics, such as size, expenditures, staff training, and cooperation with industry, we find little statistical association. The only institutional characteristic that is statistically significant in our study is the presence of a nearby campus of the state university system, and that one enters with a negative coefficient. Statistically speaking, this lack of association may be attributed in part to the imprecision of our adjusted measures. More generally, however, the finding is not surprising. Similar statistical analyses for K-12 schools indicates that variation in measures as spending per student or average class size has little or no correlation with student achievement. While few observers would doubt that some minimal level of expenditures is necessary for success in schools, relatively small variations in such measures rarely are statistically significant. In the same way, our results for community colleges should not be interpreted to say that resources are not an important and necessary condition for college success, but rather that they are not crucial in determining, at the margin, which institutions have high and low success.

We also find no statistically significant correlations between our calculated unadjusted success rates and two of the quality measures published by state administrators in their Critical Success Factors report. As we have argued above, this lack of correlation should not be taken to indict either the state’s measures or ours. The more reasonable lesson to be gleaned is that success will have numerous dimensions, and there is no reason to think they will all be highly correlated. The lesson is that no single measure, or even a small set of measures, will suffice.

These findings illustrate the significant challenges in identifying meaningful variation in the performance of community colleges. Viewed in one way, these finding may support a skeptical view of any attempt to assign overall assessments of quality to institutions as a whole. From this point of view, institutions such as community colleges are much too different one from another, with far too much variation across programs, to make such aggregation meaningful. It follows from this viewpoint that one promising avenue for future research might be to explore curricular differences across institutions, particularly differences in the share of programs connected to certification in which employers demand a document that can be obtained only by finishing all requirements.

Even as we acknowledge these differences and the need to explore ways of controlling for them, we think that measurement is and will be a worthy activity and, in general, ought to be encouraged. Thus we do not view the lack of statistical correlation between our measures and those used by the state of North Carolina as a black mark on the state’s attempt to devise quantitative metrics of institutional effectiveness. As long as these measures are not overly costly to obtain and are not given undue weight in allocation decisions, we believe more information is better than less. And we believe the effort to search for outcome measures is commendable, because postsecondary institutions historically have been conspicuously reluctant to submit to any outcome measures whatsoever. Probably the best argument we can offer for persisting in efforts to compare institutions is a pragmatic one. Legislators, policy makers, and funders are already putting into practice policies that are based on comparing institutions by existing metrics. Efforts such as the current one, one hopes, can contribute to the refinement of such measures.

Appendix

See Tables 6 and 7

Table 6 Variable definitions for student-level table

Table 7 Variable definitions for institutional-level tables