The Role of Non-cognitive Variables in Identifying Community College Students in Need of Targeted Supports

Abstract

Non-academic characteristics and traits, such as academic self-efficacy and conscientiousness, have maintained the interest of higher education researchers for decades. A considerable amount of research has found that these non-cognitive variables (NCVs) are generally predictive of undergraduate success. However, most prior studies have focused on the use of NCVs in 4-year colleges, and understanding if and how these measures predict the academic trajectories of community college students has received less attention. As past work has indicated that NCVs are differentially predictive for students from different backgrounds, such an examination is needed. Drawing on data from two diverse community colleges, our study addresses this need by exploring if seven popular NCVs predict both short and longer-term student outcomes, how these measures are related to help-seeking behaviors, and their utility in identifying students at risk of serious academic failures. We find statistically significant relationships between NCVs and GPA outcomes but conclude that their practical utility appears trivial.

Introduction

College enrollment has increased dramatically over the past five decades (Bound et al. 2010), and community colleges have shouldered much of this growth (National Center for Public Policy and Higher Education 2011). These public, open-access 2-year schools were founded to serve many functions in the growing market for post-secondary education; they are among the main providers of academic and career-technical 2-year degrees and certificates, lower-division coursework for bachelor degree-seeking students who plan to transfer to 4-year schools, specific coursework for career retooling, English instruction for adult non-native speakers, and community education for older adults (Brint and Karabel 1989; Dougherty and Townsend 2006). However, community colleges must serve students with these diverse needs and goals with relatively little support; across the country, average per-full-time-equivalent-student state funding for these institutions is about half of what it is for 4-year public colleges.¹

Thus, community college counselors, advisors, and faculty members are serving students under great constraints. School staff must place students into appropriate academic courses, identify those at risk of poor academic outcomes, and provide services such as tutoring and counseling while managing large advising loads and having few resources at their disposal. Schools have traditionally relied primarily, if not exclusively, on placement tests (and more recently on high school grades) to triage students and carry out these functions (Ngo et al. 2018).

However, results from recent research, combined with increased data storage and computing capabilities, have recently led schools to experiment with the use of various other measures, such as behavioral and self-reported survey data, to identify at-risk students and to target services to students before they experience serious academic failures. For example, community colleges across the country are beginning to use measures other than test scores to place students into appropriate math and English courses (Bahr et al. 2019; Cullinan et al. 2018; Ngo and Kwon 2015). Schools hope that better data collection and analysis will allow them to function more efficiently and to better serve students.

Measuring students’ self-reported non-academic aptitudes and beliefs—skills, traits, or knowledge that are not inherently academic, such as self-efficacy, grit, and conscientiousness—is one area that has been gaining popularity over the past few decades. Researchers and practitioners have argued that academic data, placement tests in particular, may not adequately capture students’ aptitude and potential for success. Thus, the use of non-academic measures, in addition to the traditionally available academic data, might lead to better placement, more efficiently targeted support, and higher success rates (Bahr et al. 2019; Gore 2006; Gore et al. 2017; Hodara et al. 2012).

In this study, we examine if the use of self-reported non-cognitive variables² (NCVs) could improve the targeting of support in community colleges. We rely on a unique data set in which almost 3000 incoming students at two community colleges were asked a battery of questions regarding their perceptions of their own non-academic aptitudes and beliefs. We follow these students for 2 years to track their academic progress and success. In addition to information on course enrollment, success, and persistence, we also have data on the students’ use of on-campus resources. Together, these data allow us to ask and answer the following research questions:

• (1) Can the use of non-academic measures improve predictions of academic success and persistence in community college over the rich set of academic background and test score variables to which schools usually have access?

• (2) Do behavioral mediators, such as visiting a counselor or tutor, or the number of units attempted in the first term, explain the relationship between the non-academic measures and outcomes?

• (3) How helpful could these measures be in identifying students who are at risk of serious academic failures, such as being put on academic probation or dropping out before the second year? Is there evidence that such identification could benefit some groups of students more than others?

This paper offers three unique contributions over past work: (1) we focus our study on a diverse group of students enrolled in community colleges and examine to what extent a wide range of non-academic measures could improve predictions of academic failure across groups of students; (2) our detailed data allow us to examine both longer term outcomes and the more proximal outcomes that can help us understand the patterns that lead to these longer term academic successes and failures; and (3) we capitalize on a large and diverse sample and have access to the students’ full academic record from the schools (rather than self-reported outcomes).

Examining the relationship between NCVs and outcomes specifically for students in community colleges is important for at least two reasons. First, students in community colleges are more likely to be racial and ethnic minorities and come from low-income backgrounds than their peers in 4-year schools (McFarland et al. 2017; Smith and Stange 2016). Past work has shown that non-academic measures are differentially predictive of outcomes for low- and high-income students (Lundberg 2013) and that the relationships between NCVs and outcomes vary across ethnic groups, (e.g., Arbona and Novy 1990; House 1996; Tracey and Sedlacek 1987a).

Second, the structures and common enrollment patterns in community colleges raise questions as to how well past work from 4-year colleges might generalize to this setting. Students in community colleges are more likely than their peers in 4-year schools to be surrounded by less academically prepared peers, to be enrolled part-time, and to be exclusively enrolled in online courses (McFarland et al. 2017; Smith and Stange 2016). These schools are also notoriously difficult institutions to navigate, with scores of program and course choices (Scott-Clayton 2015). These differences in student and institutional characteristics between community and 4-year colleges raise questions as to how well the observed relationships between NCVs and outcomes in 4-year settings might predict the same relationship in the community college setting. The predictive validity of NCVs in community colleges has not been tested extensively (Fong et al. 2017).

Our data also give us a unique opportunity to examine these relationships and to understand the patterns we see. While most past work has examined either long-term outcomes such as graduation (e.g., Sparkman et al. 2012; Tracey and Sedlacek 1987b) or short-term outcomes such as grades in the first-term (e.g., Akos and Kretchmar 2017), in this study we examine both. By examining short- and long-term outcomes, we are able to determine if the contribution of NCVs manifest differentially at key stages in a student’s academic trajectory.

We also leverage available data on student behaviors at the college. By examining patterns of uptake for counseling and tutoring services, as well as enrollment patterns in the first term, we can test if these behaviors function as a mediator between the NCVs and college outcomes. This examination of potential mediators can be useful to college administrators seeking potential levers for increasing student success.

Conceptual Framework

Success for a community college student (with success defined broadly, such as graduating with an associate degree, earning a career or technical certificate, transferring to a 4-year college, or developing career-related skills), requires maintaining progress in a number of domains. Among other things, students must enroll in the correct classes, pass these courses, navigate bureaucratic hurdles such as applying for financial aid and correctly filling out paperwork, and manage relationships. Wang (2017) proposes a theoretical model for understanding community college student forward progress and success. Grounding her model with analogs to physical momentum, Wang (2017) describes three domains that are central for the development and success of community college students: the teaching and learning domain, the motivational domain, and the curricular domain. Students need to develop momentum in each of these areas to propel their academic and career aspirations forward. Wang argues that momentum in each of these areas can be affected by three sets of factors: (1) carry-over momentum (such as academic and non-cognitive skills students bring with them to college), (2) counter momentum friction, such as financial and institutional barriers, and (3) other unplanned forces, such as family obligations and unexpected tragedies (Wang 2017).

We draw on Wang’s model of momentum in building a conceptual framework for this study. Our study focuses on the motivational domain, which has received relatively little research attention (Wang 2017). Specifically, we test the relationship between pre-entry, non-cognitive characteristics (an example of Wang’s carry-over momentum) and the curricular and academic progression and success of community college students. Community colleges provide a particularly useful context for examining the relationship between non-cognitive carry-over momentum and student success for a few reasons: community college students are more likely than their 4-year counterparts to enter college with some degree of academic or meta-cognitive under-preparedness; community colleges are complex organizations to navigate because they offer many courses and programmatic options; and community college students often experience a great deal of counter friction.

To ground this study, we first describe relevant literature that examines community college student success and then provide a brief sketch of the history of the use of NCVs in academic research.

Literature Review

Student Success in Community Colleges

The open access policy of community colleges speaks to the democratic American value of higher education for the masses, but poor completion and success rates have brought to the fore the challenge of educating diverse groups of students with varied academic needs (Wyner 2014). National estimates suggest that only ~ 40% of community college students have earned an associate degree or certificate within 6 years (Bailey et al. 2015). In California, which has the largest community college system in the country, 48% of all first-time, degree-seeking freshmen have earned either an associate degree or certificate in 6 years. These rates are lower for underprepared students (students whose first math and/or English course was at the remedial level), who have a 41% 6-year graduation rate (California Community College Chancellor’s Office 2018).

A number of student- and institutional-level factors contribute to these low completion rates. A large body of research has examined various academic, structural, and relational frictions that uniquely influence student success in community colleges (see, for example: Bailey et al. 2015; Calcagno et al. 2007; Cho and Karp 2013; Crisp 2010; Crisp and Nora 2010; Goldrick-Rab 2010; Hagedorn 2010; Scott-Clayton 2015). In this section we focus on those frictions that are particularly relevant for this study: students’ academic preparation and experiences, the various out-of-school demands that students must juggle, the uniquely complicated curricular and bureaucratic structures of community colleges, and a potential lack of community and sense of belonging. The ability of each of these frictions to affect students’ outcomes could be strengthened or weakened by the non-cognitive carry-over momentum that students enter college with.

As compared to students in 4-year schools, students at community colleges have, on average, weaker academic preparation (Cohen et al. 2014). For instance, nationally, nearly 70% of community college students enroll in at least one remedial course (Chen 2016). As such, many non-cognitive skills and attributes, such as soliciting help, persevering in the face of adversity, or maintaining a sense of self-efficacy, may be particularly predictive of success for students in community colleges (Rowan-Kenyon et al. 2017; Wang 2017).

Community college students are more likely than their 4-year counterparts to be attending school part-time and are more likely to be non-traditionally aged. These students often view themselves not primarily as students, but as taking on many different roles (e.g., parents, employees) (Baum et al. 2011; Kim et al. 2010). These more complicated demands and flexible schedules may mean that non-cognitive skills, such as time-management, self-regulation, grit, and identity as a college student, are especially important for students in community colleges.

Community colleges are notoriously complicated institutions that are difficult to navigate. They offer myriad choices for classes and programs (Scott-Clayton 2015). In fact, Bailey et al. (2015) suggest that the key barrier to success for community college students is the inefficient architecture of major requirements, course pathways, and program structures. These complicated designs suggest that non-cognitive skills such as conscientiousness and grit may be key predictors of student success.

Key relationships in college can help shape and guide student trajectories, and interaction and interpersonal support is key for student success and persistence (Bettinger and Baker, 2014; Deil-Amen 2011). Building such community and relationships is often challenging for students in community colleges, who are more likely to commute to campus, attend part-time, and take classes exclusively online (McFarland et al. 2017; Smith and Stange 2016). Indeed, the most vital connections in community colleges typically occur during class time (Deil-Amen 2011). These factors indicate that community college students’ interactions with others and sense of belonging, both in and out of class, might be a key predictor of student success (Deil-Amen 2011; Wang 2017; Wang et al. 2018).

Each of these sources of counter friction could be offset by the academic and non-cognitive skills students bring with them to college (carry-over momentum). Indeed, recent research has begun to examine the extent to which non-academic skills and mindsets contribute to community college student success.

Measures of Non-cognitive Skills and Beliefs

Measures of non-academic aptitudes and beliefs have been gaining popularity as a fertile arena for acquiring actionable insight into student areas of weakness and strength (Sommerfeld 2011). Across sectors of education, administrators and policy makers are advocating for the use of NCVs to appropriately place students into academic courses, to make admissions decisions, and to target support services (Gore et al. 2017; Hodara et al. 2012; Thomas et al. 2007). However, an understanding of the practical import of measuring NCVs in educational settings is not a recent development. In the mid-1980s, Tracey and Sedlacek (1984) developed the Non-Cognitive Questionnaire (NCQ) in an attempt to more accurately assess the abilities of diverse and nontraditional college applicants. The NCQ—which uses raters to score students based on interviews—is comprised of eight subscales: positive self-concept, realistic self-appraisal, understanding of and ability to deal with racism, preference for long-term goals, availability of strong support person, successful leadership experience, demonstrated community service, and knowledge acquired in a field (Tracey and Sedlacek 1984). The variety and use of non-cognitive measures has grown significantly since this initial work, and most of this research has relied on student self-reports (via surveys) rather than on interviews. In the past three decades, work has focused on a long list of non-cognitive constructs that could be useful in predicting academic success and persistence, such as: motivation, self-perceptions, attributions, self-regulation, agreeableness, extraversion, neuroticism, openness to experience, conscientiousness, grit, growth mindset, identity as a college student, test anxiety, academic self-efficacy, and academic resilience (Bauer and Liang 2003; Beattie et al. 2018; Bowman and Felix 2017; Bowman et al. 2015; Cassidy 2015; Credé et al. 2017; Farruggia et al. 2016; Fong et al. 2017; Fonteyne et al. 2017; Kappe and Van Der Flier 2012; Snipes et al. 2012; Sommerfeld 2011; Wolters and Hussain 2015).

However, synthesizing and critically examining work related to NCVs is difficult for at least two reasons. First, the language in this field is not precise. These constructs have been referred to using various names—non-cognitive attributes, character skills, psycho-social factors, socio-emotional traits, and twenty-first century competencies—across disciplines (Duckworth and Yeager 2015). These differences in terminology can conceal the fact that researchers across disciplines are often studying very similar ideas (Bryk et al. 2015; Rowan-Kenyon et al. 2017). Second, many of these measures are interdependent and highly correlated (Sommerfeld 2011). For example, self-discipline (an aspect of conscientiousness) is needed to maintain effort, an aspect of grit. Similarly, being conscientious and orderly will probably make interactions with faculty easier, so conscientiousness is likely highly correlated with measures of a student’s ability to build relationships. Indeed, a number of studies have found that distinct NCVs often load onto one or two common factors (e.g., Bowman et al. 2018; Beattie et al. 2018). Recent work to organize and categorize NCVs has classified the most commonly studied variables under five broad domains: academic behaviors, academic perseverance, academic mindsets, learning strategies, and social skills (Farrington et al. 2012).

Use of Non-cognitive Measures in Higher Education

Recent work indicates that using NCVs to predict outcomes in post-secondary settings could meaningfully improve placement decisions and the targeting of support services above the kinds of data that are usually available. While past academic records, such as high school grades, likely reflect both students’ academic and non-academic skills and training, these measures do not fully predict academic progress in college; there is significant variation in college GPA among students with the same high school GPA (Bettinger et al. 2013; Dooley et al. 2012). Academic success in college (more so than in high school, probably) involves a range of skills that are not solely academic.

Indeed, for decades, many researchers have found that NCVs are generally predictive of academic outcomes in college even after controlling for a host of potentially confounding variables, such as prior academic achievement (e.g., Bean and Eaton 2000; Bowman et al. 2015; Tinto 1987; Tracey and Sedlacek 1987a). However, results have not been entirely consistent; the strength and stability of these relationships varies across the NCVs and across studies. Indeed, some research has suggested that these measures are unable to provide significantly more information than can be gleaned from students’ academic records (Bazelais et al. 2016). And even though the relationships between NCVs and outcomes are often statistically significant, their associations with college outcomes tend to be small in magnitude (Adebayo 2008; Thomas et al. 2007). Beattie et al. (2018) conclude that only “a few non-academic measures have predictive power…” (p. 181).

In this study, we add evidence to the debate regarding the utility of using NCVs to predict outcomes in college. We take a unique approach to this question by adopting a practitioner-centered approach. That is, we examine the NCVs that community college administrators are most likely to encounter in their work and we test the usefulness of incorporating these measures into school functioning. The NCVs we examine are those with the biggest presence in the popular media and those for which validated survey items are freely available online. In Online Appendix A we outline the sources for each set of questions and list the individual items for each construct.

The constructs we examine in this study include a variety of measures that have been tested extensively in higher education as well as those that hold potential but are currently lacking—or have mixed—empirical evidence to support their use. Those that have a more established research base include grit, conscientiousness and academic self-efficacy. Research examining the relationships between college identity, teamwork, social interaction, and mindset with college outcomes is less robust. Thus, our paper adds evidence to the literature concerning the utility of promising measures that have not been studied extensively. Below, we introduce and define each of the seven NCVs we use and briefly summarize the research on their relationships with post-secondary outcomes. Though there is some research in community colleges, the bulk of the literature concerning NCVs in higher education take place in 4-year university settings.

Grit is defined as having passion and perseverance for long-term goals (Duckworth et al. 2007). The grit scale measures two distinct components: interest and effort (Duckworth et al. 2007). Grit has been shown to positively predict metacognition (Arslan et al. 2013) and GPA among 4-year university students (Akos and Kretchmar 2017; Bowman et al. 2015). Additionally, Strayhorn (2014) found grit to be particularly helpful in predicting black male students’ grades. However, some work has found differences in the utility of the two grit components. For instance, Bowman et al. (2015) has shown that the interest subscale of grit is not predictive of student outcomes, a finding that was substantiated in a meta-analysis (Crede et al. 2017). Recent work has also shown that the interest subscale is less predictive of student outcomes than the perseverance of effort (Fosnacht et al. 2019). However, like most grit studies in higher education, the participants in this study were students attending a 4-year institution.

Conscientiousness, one of the Big Five personality traits, is defined as being orderly, self-disciplined, and intentional with regard to tasks and goals (McCrae and Costa 1987). The relationship between conscientiousness and college student motivation and academic outcomes is well documented (Conrad and Patry 2012; Komarraju et al. 2009; Noftle and Robins 2007). Although conscientiousness is regarded as a relevant predictor of postsecondary outcomes, most studies examining its predictive validity have taken place in university settings, and the community college context remains relatively understudied (Kappe and Van Der Flier 2012).

Academic self-efficacy (ASE) is defined as confidence in one’s academic abilities (Pajares 1996). The concept was adapted from Bandura's work (1977, 1986) and has been applied to predict educational attainment in a variety of settings (e.g., Chemers et al. 2001; Pajares 1996; Schunk and Pajares 2002). ASE has been demonstrated to significantly predict academic resilience, decrease test anxiety, and positively predict GPA in 4-year college settings (Cassidy 2015; Fonteyne et al. 2017; Pajares 1996; Wolters and Hussain 2015).

Other NCVs have shown positive associations with college outcomes as well, though with a weaker evidence base. Implicit theories of ability—or a growth mindset—are the beliefs “that one’s academic ability can improve with effort, rather than ability being something one is born with” (Nagaoka et al. 2013, p. 49). Growth mindsets have been associated with positive academic outcomes across a number of student populations (Cury et al. 2006; Dweck et al. 2014). Evidence suggests that growth mindset interventions increase end-of-year GPA for entering university freshman and hold the potential for large-scale improvements in student outcomes (Aronson et al. 2002; Rattan et al. 2015). Here again, most studies on growth mindset take place at 4-year schools, although one study found positive effects of a growth mindset intervention among community college students (Paunesku et al. 2012).

College identity is defined as “the extent to which being a student is important to one’s self-image or identity” (Bowman and Felix 2017, p. 235). Research, although scant, has found that identifying as a college student predicts students’ institutional and goal commitment (Bowman and Felix 2017). Generally, student identity is regarded as a salient factor in academic focus and achievement (Lally and Kerr 2005; Yopyk and Prentice 2005). Research on college identity is growing but here, too, most research takes place within 4-year college settings.

Teamwork refers to a student’s belief in their ability to effectively collaborate with others on shared tasks and projects. (Chemers et al. 2011). Teamwork has been shown to predict student learning, as well as positive student behaviors, such as committing to a major (Chemers et al. 2011; Williams et al. 2006). Teamwork is essential to many tasks in college and has been shown to be positively related to student development (Kuh et al. 2008). Research on teamwork has mostly been conducted in 4-year university settings.

Interaction refers to how students communicate with others and students’ perceptions on how choices influence their academic outcomes. Research has shown that social interactions are central to student learning (Farr 2012; Wang et al. 1997). Social interactions with peers and instructors represent a connectedness to the campus and work of the student, and research has shown that students who spend their time on campus with others increase their likelihood of persisting towards a degree (Kulm and Cramer 2006).

While each of these seven NCVs holds promise in being a helpful predictor of college outcomes, differences across studies make it difficult to converge on a recommendation for exactly how college leaders and administrators could use these NCVs to improve how they serve students. Variation across study designs makes it difficult to determine the practical utility of most of these measures. For example, studies vary dramatically on the other student-level factors for which they control. In addition to measures of NCVs, some studies include self-reported measures of performance (e.g., Hodge et al. 2018), some include self-reported measures of prior achievement (e.g., Wolters and Hussain 2015), and some include relatively few control variables (e.g., Strayhorn 2014). This lack of consistency across studies makes drawing an overall conclusion difficult and calls into question the practical utility of examining and including these constructs. Schools interested in using these measures would likely include these measures with the full complement of available student-level predictors; studies that do not adequately control for other important predictors (such as measures of previous achievement) might not provide much actionable information for schools. Including such important predictors is particularly salient when drawing conclusions from research in community college contexts in which there is often a large degree of variability in academic and social backgrounds and experiences.

NCVs and Mediators

Wang’s (2017) model highlights the motivational domain of momentum that includes aspirational momentum, growth mindset, perseverance (grit), and agentic momentum. These variables might lead to student behaviors, affections, and dispositions which in turn lead to continued curricular momentum. Recent work has examined the mechanisms that might explain or mediate the relationships between NCVs and student academic outcomes (e.g., Farrington et al. 2012; Nagaoka et al. 2013; Snipes et al. 2012). For example, grit is associated with co-curricular involvement, students’ sense of belonging, and general satisfaction, which explains part of the relationship between grit and academic outcomes (Bowman et al. 2018). Bowman et al. (2018) find that a composite NCV measure is strongly associated with retention, but that this is mostly an indirect effect attributable to GPA, social adjustment, and institutional commitment. Furthermore, the relationship between conscientiousness and academic achievement might be mediated by academic self-efficacy (Conrad and Patry 2012) and reduced test anxiety (Fonteyne et al. 2017). ASE has been associated with both lower test anxiety (and subsequently, higher GPAs) as well as higher academic resilience (Fonteyne et al. 2017; Cassidy 2015).

Given this body of research, understanding the relationships that mediate the associations between NCVs and student performance has the potential to inform policy decisions; examining the mechanisms through which NCVs affect more distal outcomes might give policy makers and administrators insight into potential interventions. For example, if students who rate their non-cognitive skills and beliefs as high are found to be more likely to persist into the second term, we do not know if this is because they exhibit better study habits, if they spend more time on school work, or if they avail themselves of resources on campus with more regularity. A deeper examination of these potential mechanisms, particularly traits that are malleable, might improve tailored efforts to affect specific behaviors and may lead to more positive student outcomes.

Current Study

Our study offers several unique contributions to the extensive body of educational research concerning NCVs. First, by testing the joint predictive ability of a broad set of non-cognitive measures, we focus our attention on the practical utility of these measures in identifying community college students in need of targeted support rather than on the individual implications of each NCV. Second, with information on short- and longer-term outcomes, our data allow us to examine the importance of NCVs across pivotal milestones in the community college timeline. Third, we link NCVs to help-seeking and enrollment behaviors to test the role of these behaviors in the relationships between non-academic traits and skills and student outcomes. Finally, this study is the first to investigate such a vast array of relationships between NCVs and student outcomes while controlling for a comprehensive set of academic, demographic, and behavioral measures within the increasingly important community college context. We examine the utility of these factors both in an important context and in a manner in which they would be used by schools.

Data and Methods

Colleges in Sample

We use application, matriculation, and transcript data from two southern California community colleges (CCCs). These two colleges serve very diverse student populations with a large proportion of first-time college students, under-represented minorities, and older students enrolling. The two colleges are located in a relatively affluent area of California. Table 1 presents the student demographics of the colleges in our sample as well as of all 113 California community colleges and of all community colleges nation-wide.

Table 1 Demographic comparison between colleges in sample, California Community colleges, and US Community colleges Fall 2015.

There are fewer black and African American students in our sample than there are in all California Community Colleges (2% compared to 7% state-wide), fewer Hispanic students (26% compared to 45%), and fewer students on financial aid (46% compared to 70%).³ The proportion of Hispanic students at the colleges in our sample is similar to the proportion at community colleges nation-wide. The two colleges in our sample serve more Asian students (16% compared to 13% state-wide), more traditional-aged students between 18 and 24 (64% compared to 59%), and more White students (46%) compared to the state-wide averages (27%). While the proportion of White students is considerably larger than state-wide averages, the two colleges in our sample serve a large proportion of middle eastern students who often select ‘White’ as an ethnic descriptor.⁴

Description of Variables

Data for this study were compiled by the institutional research offices at each school. Institutional records (based on student self-reported answers on application) were used to provide student demographic information on gender (female, male), ethnicity (Asian/Pacific Islander, black/African-American, Hispanic, White, other), age (dummy variable indicating students who are older than 24 years old during first semester), and first-generation status (dummy variable indicating if both the students’ parents have no college experience).

Institutional records (based on verified information) also included undocumented status (dummy variable indicating if a student qualifies for non-resident tuition under AB 540⁵), foster youth status (dummy variable indicating if student is receiving services at the college as a current or former foster youth), student disability (dummy variable indicating if student receives Disabled Student Programs and Services (DSPS) at the college), economic disadvantage (dummy variable indicating if student received services from the Extended Opportunity Programs and Services⁶ (EOPS) at the college), visits to counseling and tutoring services (counts of how many times students visited the counselor or tutor within a term), student placement test scores in math and English (math placement test scores range from 0 to 51, English placement test scores range from 0 to 141, representing the total score of correctly answered test items), student’s cumulative Grade Point Average (GPA) at the end of term (measured on a four-point scale), persistence (if student enrolled in a credit course during the following major term),⁷ and completion (if student earned a certificate or degree, or transferred to a 4-year university during the next major term).

Non-cognitive measures were obtained from spring 2015 to fall 2016 as part of the assessment process. At the time of the study, students who intended to enroll in courses to obtain a certificate, degree, or transfer to a 4-year university were required to complete matriculation steps that included taking an English and math assessment test to be placed into transfer-level or remedial courses in math and English. Prior to taking the assessment tests, students were given a 44-item battery of non-cognitive questions (please refer to Online Appendix A for more information). All items were measured on a five-point Likert scale from strongly agree to strongly disagree.

Missing Data

The full sample consisted of 5818 students who (1) completed matriculation, (2) subsequently enrolled in a credit course from fall 2015 to fall 2016, and (3) indicated that their goal was to earn a certificate, associate degree (academic or career/technical), or transfer to a 4-year university.

We limit our analytic sample to students who have full data on all predictor and outcome variables. We drop 2969 (51%) students due to this restriction for a final analytic sample of 2849 students. Of these that were dropped, the vast majority were dropped because they were missing high school GPA information.⁸ Access to high school data is based on a voluntary data sharing agreement between area high schools and the two colleges. While most high schools from surrounding school districts participated in this agreement, students from some in-state high-schools, students from most private high schools, all out-of-state students, and students who did not apply directly to the community college after high school (that is, older students) have missing information on this measure. At the point of data collection, the state-wide community college application did not collect self-reported high school GPA. Table 2 shows the effect of the exclusion of students with missing data on the demographics of the students included in our analyses. As high school GPA was the most common reason for excluding a record. The analytical sample included more traditional age students. Additionally, the analytical sample included more Hispanic students and fewer Asian and Pacific Islander students compared to the overall sample. In terms of outcomes of interest, GPA and tutoring/counselor measures were slightly lower compared to the overall sample but persistence and first term units attempted were higher. In general, the analytical sample is representative of first-time college students from local public feeder schools.

Table 2 Descriptive statistics and missingness for full sample and analytic sample

Despite the statistical implications of a reduced sample size and a potential loss of generalizability, we believe that it was important to include high school GPA in our models to control for prior academic achievement. Doing so allows us to truly test how powerful NCVs are in predicting college outcomes in a policy-relevant environment. High school grades are strong predictors of college performance and excluding this information would not provide a realistic account of how helpful NCVs could be to college administrators.⁹

Findings from this study can be reasonably generalized to traditional-aged, first-time community college students with a primary focus on academic outcomes who attended public high schools. The implications for generalizability from our focal schools and our sample restrictions are important, and we return to this in the Discussion section.

Descriptives on Analytic Sample

Table 2 shows the descriptive statistics for all covariates and outcome variables and Table 3 displays the means, standard deviations, number of items, and Cronbach’s alpha for each NCV in the study. All items were measured on a five-point scale ranging from strongly disagree (1) to strongly agree (5). The mean scores were close to one another, with interaction (M = 4.3, SD = 0.42) having the highest average score and grit having the lowest average score (M = 3.47, SD = 0.52). The NCV with the most variability in responses was college identity (M = 3.86, SD = 0.71). ASE was composed of the greatest number of items (11) and had the largest reliability score (Cronbach’s α = 0.89).

Table 3 Descriptive statistics of non-cognitive constructs

Correlations Between Non-academic Factors and Main Outcomes

Table 4 presents the correlations between non-cognitive measures and student outcome variables. Most of the NCVs are significantly and moderately correlated with each other. The largest correlation was between grit and academic self-efficacy, r = 0.63, p < 0.05. All of the NCVs were weakly correlated with student outcomes. For instance, ASE and cumulative GPA had the highest correlation among any NCV and outcome, r = 0.09, p < 0.05. Generally, these bivariate associations correspond with previous studies in terms of direction and relative strength but the associations in this study tend to be smaller. For example, Chang (2014) computed a correlation of r = 0.09 between grit and first-term GPA while we found r = 0.05. Crede et al. (2017) found that the average correlation between grit and undergraduate GPA among published studies is 0.15. However, we note that the Crede et al. study was based on samples of students from 4-year universities.

Table 4 Correlations between NCVs and student outcomes

Analytic Methods

Using Non-academic Factors to Predict Outcomes

We explore the relationships between our seven NCVs and student outcomes in two ways. We first include the seven NCVs independently in regressions predicting outcomes and then include two composite factors that we have extracted from the vector of NCVs. In determining the factors, we conducted an exploratory factor analysis (EFA) to examine the intercorrelations between the seven NCVs (ASE, conscientiousness, grit, teamwork, college identity, interaction, and mindset) and inspected eigenvalues, scree plots, and factor loadings to determine the number of factors to extract.

We use our first series of regression models, first including all seven non-academic measures and then including our two composite factors, to determine if NCVs are jointly predictive of important academic and persistence outcomes beyond predictions based on past academic achievement. We do this by predicting post-secondary outcomes (GPA, persistence, and completion) using the rich set of demographic and academic variables that are often available to college placement offices, including high school GPA and placement test scores. We then add the set of non-academic measures (or the factors, depending on the model) and observe if this new set of information adds meaningfully to our predictions.

We note that this method is not optimal for examining the relationship between particular non-academic measures and academic outcomes for two reasons. First, high school grades, and to some extent, placement test scores, reflect both academic and non-academic skills (Beattie et al. 2018). By controlling for this rich set of covariates, we absorb part of the total contribution of our non-academic measures. Second, our comprehensive set of NCVs includes measures that overlap conceptually and empirically; this multicollinearity might result in non-significant relationships in the multiple regression for relationships that show a significant correlation in bivariate analyses. But as our aim is to improve predictions of post-secondary success, and not to describe the relationships between specific NCVs and academic outcomes, this approach is appropriate. We estimate the same model for a series of outcomes:

$$CollegeOutcome_{ichs} = \beta_{0} + \beta_{1} HSGPA_{ichs} + \beta_{2} PlacementTest_{ichs} + \Delta_{ichs} \beta_{3} + NCV_{ichs} \beta_{4} + \delta_{c} + \pi_{h} + \vartheta_{s} + \varepsilon$$

where $CollegeOutcome_{ichs}$ is the college outcome [e.g., first-term GPA, persistence to second term, cumulative GPA, and persistence to fourth term (second year)] of student i in college c coming from high school h and entering in semester s. $HSGPA_{ichs}$ and $PlacementTest_{ichs}$ are her high school GPA and math and English placement test scores. $\Delta_{ichs}$ is a set of background characteristics that includes gender, first generation college student status, a dummy for if the student is older than age 24, if the student is undocumented, if the student is a former or current foster youth, DSPS and EOPS. $\delta_{c}$ is a set of college fixed effects, $\pi_{h}$ is a vector of high school fixed effects, and $\vartheta_{s}$ is a set of semester-of-entry fixed effects. The college fixed effects account for unobserved differences between the colleges such as grading procedures, quality of instruction, and student services available to students. We include high school fixed effects to account for the possibility that students use different reference points based on their peer groups, which could influence the assessments of their own non-academic abilities. These high school fixed effects also account for the fact that high school GPAs might not be comparable across schools. Semester-of-entry (cohort) fixed effects are meant to account for any secular trends in student outcomes or placement policies.

Our model includes $NCV_{ichs} {\varvec{\beta}}_{4}$, a vector of seven composite measures of non-academic skills or beliefs. The addition of this vector of non-academic variables allows us to test, for each of our four outcomes of interest, the collective predictive validity of these seven measures. For each outcome, we estimate a joint F-test that determines if the coefficients on the vector of non-academic measures (or the two factors) are jointly significant (that is, we test if any of the covariates in our vector $\beta_{4}$ is significantly different from zero). As many of our non-academic measures likely overlap conceptually and empirically, we are less interested in the individual contribution that each measure makes and more in the joint predictive power of our group of measures.

Mediation Models

While we might see that some of our non-cognitive measures are significant predictors of outcomes of interest, we do not know the specific behaviors that might explain these outcomes. Although many behaviors related to academic success are not visible to researchers, we do have access to information that can help us understand the mechanisms by which non-cognitive traits might affect student outcomes. Namely, we have records of the number of times students visited the counselor and tutor¹⁰, as well as the number of units attempted in the first-term. Though these might be imperfect measures, they are likely relatively good proxies of a student’s propensity to seek help and guidance as well as indicate a sense of engagement and effort.

To answer the question regarding help-seeking behaviors, we examine the indirect effect of NCVs on academic outcomes, controlling for our theoretically driven mediators. We predict five college outcomes: first-term GPA, persistence to the second term, cumulative GPA, persistence to the fourth term (second year), and completion (either earning a certificate, associate degree, or transferring to a 4-year institution), and test for mediation across four regression models: (1) using only the NCV factors derived from the EFA; (2) using the factors as well as covariates, (3) including counselor visits in the model, and (4) including tutoring visits in the model. By comparing the change in the magnitudes and statistical significance of the NCV coefficients across the models, we can observe if these help-seeking behaviors mediate the relationship between NCVs and college outcomes¹¹.

Similarly, we ask if first-term units attempted explains the relationship between NCVs and outcomes. Research has found that units attempted predict persistence in community college students (Nakajima et al. 2012), and students who spend more time away from school often perform poorly in comparison to their more engaged counterparts (Kuh et al. 2008). Thus, to examine if a count of units attempted in the first term acts as a mediator, we predict the same five college outcomes and test for mediation across three models: (1) using only the NCV factors derived from the EFA; (2) using the factors as well as covariates, and (3) including first-term units attempted in the model.

Improving Targeting of Support Services

While statistical relationships between self-reported NCV and college outcomes are interesting, statistically significant relationships might not translate into a meaningful improvement in schools’ abilities to serve students. Because of their diverse student bodies and constrained resources, administrators and counselors at community colleges must make wise decisions regarding which students are most in need of outreach and targeted support services. We next test the ability of NCV measures to help schools predict which students are at risk of poor academic outcomes so that they can better target resources.

We examine this question by examining the predictive validity of four models with different sets of predictors: (1) using only placement test scores, (2) using placement test scores + NCV, (3) using placement test scores + HS GPA, and (4) using placement test scores, HS GPA and NCV. Each model also includes college fixed effects, term of first enrollment fixed effects, and high school fixed effects. We test the ability of each of these models to correctly predict three academic outcomes: (a) being on academic probation in the first-term, (b) not returning for a second term, and (c) not persisting to the fourth term. In contrast to our previous methods, which test the statistical significance of relationships between NCV and outcomes, these models ask if the use of NCV would allow schools to meaningfully improve their predictions of which students will experience serious academic failures and thus better target support services.

We use a tenfold cross validation procedure to test how well each of these models predicts these outcomes. We first randomly divide our sample of 2849 students into ten even samples. We run our analysis ten times, each time leaving out one hold-out sample. Using the remaining ~ 2600 students, we create prediction models (predicting each of our three outcomes using each of our four sets of predictors) and use the saved coefficients from these models to predict the outcomes in our hold-out sample. We repeat this 10 times, once for each of our 10 hold-out samples.

Results from these predictive models allow us to evaluate how accurate each model is. That is, we can compare a student’s predicted outcome (between 0—not predicted to experience the negative outcome and 1—predicted to experience the negative outcome—which we can dichotomize into either 0 or 1) with their actual outcome (0 or 1). The goal of each of these models is to maximize the number of students who are correctly classified and minimize the number of students who are incorrectly classified. There are a number of ways to evaluate the accuracy of a predictive model. In this paper we employ four commonly used statistics: percent of students who were correctly classified, sensitivity, specificity, and the Area Under the Receiver Operator Curve (ROC).

For any outcome, the percent of students who are correctly classified includes students who are predicted to experience a negative outcome and actually do experience that outcome (True Positives) as well as all students who are predicted to not experience that outcome and actually do not experience that outcome (True Negatives). Sensitivity gives the evaluation of a model’s ability to correctly identify truly at-risk students. It is expressed as the proportion of truly at-risk students who are identified as at-risk by the model:

$$Sensitivity = \frac{\# True Positives}{{\# True Positives + \# False Negatives}}$$

Specificity refers to a model’s ability to correctly identify students who are not at risk. It is expressed as the proportion of truly not-at-risk students who are identified as not-at-risk by the model:

$$Specificity = \frac{\# True Negatives}{{\# True Negatives + \# False Positives}}$$

All three of these measures rely on the researcher selecting a certain threshold for predicted probabilities over which students are labeled as “at-risk.” For the analyses in this paper, we chose 0.5—any student who had a greater than 50% predicted probability of experiencing a negative outcome was labeled “at-risk.”

The receiver operating characteristic (ROC) curve synthesizes a model’s classification ability over a range of thresholds. ROC curves plot sensitivity (true-positive rates) on the Y-axis against one-specificity (the false positive rate) on the X-axis for different thresholds and allow us to examine which diagnostic model produces the most accurate predictions. The Area Under the ROC Curve (AUC) is a measure of the predictive performance of a given model (Attewell and Monaghan 2015). Very poor models (those that do no better than chance in predicting who will have a negative outcome) have AUCs equal to 0.5 (that is, they exactly follow the 45-degree line). The closer the AUC gets to one, the better the model is at predicting outcomes (Attewell and Monaghan 2015). These analyses thus enable us to evaluate how meaningful the use of these additional predictors may be for schools.

Results

Exploratory Factor Analysis

We conducted an EFA on the seven NCVs using our sample of 2849 students. We conducted preliminary tests to determine if our data met the requirements to continue with the EFA. The determinant of the correlation matrix was 0.08, indicating that the correlations are neither too low nor too high. The Bartlett test of sphericity, which tests if the correlation matrix is statistically different from an identity matrix, was significant. Lastly, the Kaiser–Meyer–Olkin (KMO), which checks how compact the pattern of associations are, was 0.86, indicating an adequate pattern to carry on with the EFA. The factors were rotated using orthogonal (varimax) rotation. Visual inspection of the scree plot, proportion of variance explained, as well as the size of the factor loadings, indicated two factors. According to the factor loadings, ASE, conscientiousness, and grit mostly correlate with factor 1, which we refer to as a general tenacity factor. Conversely, teamwork, college identity, mindset, and interaction mostly correlate with factor 2, which we refer to as inter-intra connections. Cronbach’s α = 0.82 for the first factor and α = 0.68 for the second factor, indicating the first factor as having good reliability and the second as showing largely acceptable reliability for research purposes (Santos 1999). We produced factor scores using the regression method and we standardized scores to have a mean of zero and a standard deviation of one. Table 5 shows the factor loadings for each non-cognitive variable.

Table 5 Factor loadings

Using Non-academic Factors to Predict Outcomes

In our first set of analyses, we examined if a robust vector of seven non-academic variables, or, in some models, two factors created from these variables, were jointly predictive of four important academic outcomes: first-term GPA, cumulative GPA, probability of persisting to the second term, probability of persisting to the fourth term (second year), and completing a degree or certificate. We conduct an F-test to examine the joint significance of the vector of non-academic variables (the second model in each panel) and the vector of NCV factors (the third model in each panel), while controlling for sociodemographic and prior achievement variables.

Tables 6 and 7 present the results from these analyses (Table 6 shows 1st-year outcomes and Table 7 presents second-year outcomes). Table 6 shows that the set of non-cognitive measures are jointly predictive of first-term GPA, F(7, 2780) = 3.56, p < 0.001, but not persistence to the second term. Similarly, the model with the two NCV factors significantly predicted first-term GPA, F(2, 2785) = 9.511, p < 0.001, but not persistence to the second term. Table 7 shows that the set of non-cognitive measures are jointly predictive of cumulative GPA, F(7, 2780) = 2.10, p < 0.05, persistence to the fourth term (last term of the second year), F(7, 2780) = 2.88, p < 0.01, and completion F(7, 2780) = 4.88, p < 0.001. The models using the two NCV factors followed a similar pattern of statistical significance except in the case of persistence to the fourth term, where adding the two NCVs was not significantly predictive. Generally, the NCVs predict GPA outcomes with more consistency than persistence; however, while the predictive power attenuates over time when predicting GPA, it increases for persistence.

Table 6 Predicting college outcomes with NCVs

Table 7 Predicting college outcomes with NCVs

In each of these cases, however, the non-cognitive factors explain a very small amount of variance in the outcomes above the comprehensive academic and demographic variables included as controls. Given that the set of NCVs only explains less than 1% of the variation in college outcomes, the practical utility of using these measures for administrative purposes is questionable, a point to which we return below. While we are hesitant to make much of the individual contributions of these seven factors, as they are highly correlated and by including them all together, we undoubtedly reduce our ability to observe the individual contributions, we point out two interesting trends: ASE, conscientiousness, and ability to work in a team are predictive of each of the GPA outcomes. Ability to work in a team, college identity, and interaction were the only NCVs significantly predicting persistence. As for the two factors, the general tenacity factor was a significant and positive predictor of the GPA outcomes while the intra-inter connections factor was a negative and significant predictor of the GPA outcomes. These results show a distinction between NCVs associated with general tenacity and those associated with intra-inter connections factor in the community college setting; the former is a positive and significant predictor and the latter is not.

Understanding the Mechanisms Behind These Findings

Table 8 and Table 9 (1st-year and second-year outcomes are presented in Tables 8 and 9, respectively) presents the regression results from the models estimating the association between the two general NCV factors and college outcomes while controlling for two of our hypothesized mediators, counseling and tutoring visits. For each of the five outcomes, we conducted four regression models with predictors including: (1) the two NCV factors; (2) the two NCV factors and the sociodemographic and prior achievement controls; (3) NCV factors, sociodemographic and prior achievement controls plus counselor visits; (4) NCV factors, sociodemographic and prior achievement controls, counselor visits and tutor visits.

Table 8 Help-seeking behaviors as mediators

Table 9 Help-seeking behaviors as mediators

We first note that only one of the theorized mediators–counselor visits–significantly predicted all five college outcomes. In these multivariate models, tutoring visits only significantly predicted persistence to the fourth term. We also see that there is no evidence that these two measures are mediating the relationship between the NCV factors and the outcomes. Overall, the changes in the coefficient size of the two NCV factors were not large when counselor or tutoring visits were added to the model. Additionally, the F-statistic for each model did not exhibit great change from model to model. Based on these analyses, there is no evidence suggesting that the relationship between the NCV factors and college outcomes is mediated by counselor or tutoring visits.

Table 10 and Table 11 (1st-year and second-year outcomes are presented in Tables 10 and 11, respectively) presents the regression results from the models estimating the association between the two NCV factors and college outcomes while controlling for our other hypothesized mediator, units attempted in the first-term. For each of the four outcomes, we conducted three regression models including: (1) the two NCV factors; (2) the two NCV factors and the sociodemographic and prior achievement controls; (3) NCV factors, sociodemographic and prior achievement controls plus first-term units attempted. First-term units attempted positively and significantly predicted all five outcomes. While the tenacity factor remained largely unchanged, the magnitude of the intra-inter connection coefficient increased by approximately 8% (− 0.0589 to (− 0.0544)/− 0.0544) when predicting first-term GPA, suggesting that the relationship between the intra-inter connections factor and first-term GPA is marginally mediated by first-term units attempted. An even larger increase in the point estimate (that is, moving further from zero) was found for the models predicting persistence to the second term (21%), cumulative GPA (12%), and persistence to the fourth term (25%). This suggests that students with higher scores on the intra-inter connections factor attempt more units in their first term than those with lower scores and that the positive relationship between higher unit loads and outcomes partly conceals a negative relationship between intra-inter connections and academic outcomes. The magnitude of the change, however, suggests that this mediation is relatively small, particularly for the two GPA outcomes.

Table 10 Units attempted as a mediator

Table 11 Units attempted as a mediator

Improving Targeting of Support Services

While there are many interesting significant (and non-significant!) findings in this paper, the question remains as to how helpful these results would be for schools. Does measuring these variables result in any meaningfully increased precision in identifying at-risk students?

Table 12 presents the four statistics that describe the accuracy of prediction for four different models across three outcomes, and Fig. 1 presents the Area Under the Curve for each of the three outcomes using each of the four models. One finding is immediately clear – while adding NCV to the prediction models produces models that are slightly more accurate (an increase in the AUC of about 0.5% across the three outcomes), adding NCV to the models does not increase the predictive validity of the models nearly as much as adding in high school GPA (which increases the AUC by about 3.6% across the three models).

Table 12 Predictive validity of models with NCV and HS GPA

Fig. 1

Predictive validity of NCVs across academic and persistence outcomes

Figures 2, 3, and 4 also allow us to examine how well the models perform for various subgroups. While the differential predictiveness of the models across subgroups of students is not the primary focus of this paper, a few findings stick out. Overall, these predictive models perform best for female students; across all three outcomes the AUC is largest for females. The models are least good at predicting persistence outcomes for Asian students (but they are best at predicting academic probation for Asian students).

Fig. 2

NCVs predicting academic probation in term 1 by subgroups

Fig. 3

NCVs predicting non-persistence to term 2 by subgroups

Fig. 4

NCVs predicting non-persistence to term 4 by subgroups

These figures also give some information for which groups the addition of NCV most improves predictions. On average, across the three outcomes, White students benefit the most when we add NCV to the prediction models that only include placement test scores (moving from the left-most point to the second-from-the-left point in Figs. 2, 3 and 4). Asian and male students experience the next largest increase in AUC. (Though across all groups the increases are relatively small).

These findings of the differential helpfulness across groups raises questions about potential effects to equity if schools were to use different sources of information for placing students into classes or targeting support services. These results indicate that the use of NCV could disproportionately help Asian and White students (who are already the highest performing subgroups at these two schools).

While these metrics provide overall indications of the ability of NCV to improve predictions of at-risk students, they might not represent the metrics that schools are most interested in. For example, some schools might want to ensure that all students who are at risk are receiving extra support. In this case, schools would want to minimize the rate of false negatives (Type II error). Online Appendix Figures B1, B2, and B3 present the different models’ false negative rates. Again, we see that while adding NCV to the models helps reduce the rates of false negatives a little, adding high school GPA to the model produces much larger decreases in rates of false negatives (particularly for a student’s probability of being on academic probation).

Discussion

This study contributes to the literature on the use of NCVs in higher education in several ways. First, it provides an analysis of the joint predictiveness of a set of non-academic traits and skills on longer- and short-term student outcomes in a multi-site community college setting. Examining the predictiveness of NCVs in the community college setting is an important development as these schools must serve a diverse student body with limited resources and a better understanding of students’ pre-entry skills and attributes might allow for more targeted provision of services. While work from other contexts, such as 4-year colleges, is potentially informative, the unique student- and institutional-level features of community colleges could affect the usefulness of these measures in this context.

The results of this study largely align with the results of similar studies examining NCVs and student outcomes. We observe most of the significant associations documented among NCVs in past research. For instance, ASE and conscientiousness are consistent predictors of positive academic outcomes, both in the short and longer term. We also find that the measures group in similar ways to past research—grit, consciousness and ASE load on one factor, while more social and self-perceptions, like teamwork, interaction, and college identity, load on another. These groupings largely align with Bowman et al. (2018).

Second, this study is the first to test behavioral mediators (counselor and tutoring visits and first-term units attempted) between NCVs and community college outcomes. While this paper fails to find any strong evidence of behavioral mediators, it does find suggestive evidence that could inform future work. Specifically, increased course taking in the first term partially masks the negative relationship between the intra-interpersonal connections factor and college outcomes.

Third, this study focuses on the usability—from an institutional perspective—of NCVs. By examining the joint predictive ability of a large set of NCVs in concert with academic and demographic measures, these results provide a greater understanding of the practical value of NCVs to the work of supporting students. While statistically significant relationships are intellectually compelling, these measures will only be of use to schools if they can meaningfully improve schools’ abilities to serve their students. We find that these measures are not practically useful; using NCVs would result in only relatively few more students being correctly identified as being at risk of negative academic outcomes. The cost of administering these surveys to students does not seem to be worth such a minimal improvement in predictions.

While many of our results align with past research, we note some key departures from past findings (e.g., Schmitt et al. 2009; Sedlacek 2004). For example, we observed non-significant relationships between grit and college outcomes and negative relationships between the inter-intra connections factor and college outcomes. While we cannot say definitively why we find these differences from earlier studies, we offer a few hypotheses.

First, the different results could result from modeling differences. We estimate more robust models than most of the past work, which could attenuate the relationship between individual measures and outcomes. Our models control for more demographic and academic background characteristics than most other studies. It is possible that, for example, grit is not a significant predictor over and above this rich set of controls. Similarly, we include a number of non-cognitive measures in our models. As we noted in the literature review section, past work has found that many of these variables are likely measuring similar underlying constructs, though they have different names within or across disciplines [an example of the jangle fallacy (Kelley 1927)]. By including measures with high shared variance, we might not find significant relationships where other researchers have, such as between grit and academic outcomes.

Second, our sample and context could be sufficiently different from the 4-year contexts studied previously. For example, students in community colleges are likely to have less well-defined goals. While almost all students share the same immediate goal (earning a bachelor’s degree), community colleges offer many potential outcomes, such as earning a certificate, earning an associate degree, or transferring. Students might strive for more than one of these goals. Despite being committed to long-term goals, students may not see their college classes as leading directly to a particular outcome. Similarly, curricular paths through community colleges are often less well defined than in 4-year schools. Certain non-cognitive traits may be less predictive of success in situations in which the next step is not immediately clear. Although several initiatives, such as Guided Pathways, are underway to streamline community college programs, the schools examined in this study are not highly structured; the curricular offerings in these schools represent the typical cafeteria model (Bailey et al. 2015; Crosta 2014). Perhaps as programs and courses of study begin to mimic the more tightly structured programs of 4-year schools, these relationships between NCVs and outcomes will be more relevant for community college students.

Our counter-intuitive finding, that teamwork and interacting with others have a negative relationship with academic outcomes, could also be explained by the different context in our study. For example, interactions in community colleges are more likely to occur exclusively in academic spaces such as classrooms (Deil-Amen 2011; Wang 2013; Wang et al. 2018). This contextual difference could indicate that the teamwork and interaction measures are picking up different forms of social capital or latent beliefs and skills than similar measures used in 4-year contexts.

Our finding that the predictiveness of NCVs changes over time could also be explained by contextual differences. We see that NCVs are predictive of persistence in the longer-term (persistence to the fourth term and completion) but not the short term. This pattern of findings may reveal that students need to draw more heavily on these traits as they progress in their career, as students may perceive that supports are not as readily available in later years as they are in students’ 1st-year. This could be examined explicitly via student survey.

Finally, the different findings between our study and past work could point to measurement issues. The accuracy of various NCV measures has not been tested extensively in the community college setting and perhaps current measures are failing to adequately reflect the manner in which such attributes manifest among students in these schools. For example, the items constituting the teamwork and interaction measures are global assessments of how students work in teams, take direction, assume leadership roles, and communicate with others. They are not specific measures that probe the interactions embedded within courses, which might be more appropriate for community college students (Deil-Amen 2011). It may be that, unlike in 4-year settings, both measures are capturing certain social dispositions that distract students from focusing on coursework rather than capturing salient interactions and teamwork occurring within courses. For any measure to be applicable in the community college setting, researchers cannot merely look at instruments that are relevant at the construct level (e.g., teamwork, etc.); but instead must develop items that reflect how these constructs manifest in this particular setting (Block 1996; Kelley 1927; Marsh 1994; Maul 2017).

Our analysis also revealed that the relationship between NCVs and college outcomes are not mediated by counselor or tutoring visits but are partially mediated by units attempted in the first term. We offer a few potential explanations to explain the lack of mediation from counselor and tutoring visits. Even though the number of counselor visits was a positively significant predictor of all five outcomes, these visits might be a noisy measure of help-seeking behavior. For example, students might face difficulty scheduling a meeting with a counselor or they may lack information about the supports offered at the counseling center. Similarly, tutoring is often only advertised and readily available for certain subjects and students might visit a tutor with specific questions related to only one course. For these reasons, we might not expect tutoring visits to contribute extensively to either GPA or persistence. Examining mediators that explain the relationship between NCVs and outcomes remains an area ripe for research.

One concerning finding from the paper is that while the use of NCVs for identifying students at risk of negative outcomes appears to be only marginally helpful overall, it is differentially helpful for various student groups. Specifically, White, Asian, and male students seem to benefit the most. This might indicate that these NCVs are picking up differential access to resources and social networks across groups and may be evidence in support of criticism suggesting that NCVs represent social or economic capital (Ris 2015; Tyler 2016).

Readers should be cautious about generalizing the findings from this study to other contexts given the setting and sample of this study. While the two community colleges studied enroll diverse student bodies, these schools do not exactly mirror the student bodies of all California Community Colleges or community colleges across the country. Moreover, because only students with high school GPAs are included in this study, the validity of these findings for students who return to college after long gaps is untested. While this study examines a much more diverse group of students than past work, future work in other contexts is needed to establish generalizability.

However, we note that it is unlikely that our results are overstating the usefulness of NCVs in predicting community college student success. In addition to the findings in our paper, past work has shown that the relationship between NCVs and academic outcomes varies across groups (e.g., is stronger for higher income students, Lundberg 2013). Most of the past research indicates that, if anything, results in other community college contexts (with more low income, non-White students) would be even weaker than what we find.

In this study we find that NCVs are significant predictors of community college outcomes but that the practical utility of these measures is questionable. Our research suggests that community colleges would not currently gain much in their efforts to more accurately assess and support student abilities by using NCVs. However, future work to examine the accuracy of measurement in this context, as well as more refined measures of potential behavioral mediators, could lead to more actionable insight.