Keywords

The focus of this volume is on evidence-based practical strategies to enhance student engagement at school and with learning. A key element of intervention, of course, is assessment. Without assessment, how would one verify that there is a problem, select an intervention that matches student needs, or determine whether or not the intervention was effective? In education, we commonly conduct assessments or collect data that are not suitable for or easily linked to intervention, despite numerous calls and suggestions for best practices (Christenson & Ysseldyke, 1989; National Association of School Psychologists [NASP], 2009; Ysseldyke et al., 2006). Student engagement, however, is ideally suited for identification of risk, linking assessment to intervention, and monitoring student progress (Christenson et al., 2008; Fredricks, Rescly, & Christenson, 2019; Reschly, Appleton, & Pohl, 2014).

Specifically, the assessment of student engagement may facilitate educators’ ability to determine which students are at-risk for poor educational outcomes and may benefit from additional intervention, as well as what types of interventions may be most effective for students. The potential of the assessment of engagement relates to findings that suggest student engagement is associated with academic performance and behavior (Christenson, Reschly, & Wylie, 2012), mental health (Reschly, Pohl, Christenson, & Appleton, 2017; Suldo, Parker, Shaunessy-Dedrick, & O’Brennan, 2019), and resilience (Finn & Rock, 1997; Finn & Zimmer, 2012). Student engagement is also predictive of future performance in terms of high school dropout and graduation (Finn & Rock, 1997; Lovelace, Reschly, & Appleton, 2017) and postsecondary enrollment and persistence (Finn, 2006; Fraysier, Reschly, & Appleton, 2019; Lawson & Masyn, 2015). Furthermore, unlike so many demographic variables associated with student outcomes, student engagement is amenable to intervention (see Chap. 2; Christenson et al., 2008; Reschly et al., 2014).

As described elsewhere within this book, student engagement is conceptualized as a meta-construct, consisting of behavioral, emotional, and cognitive components (Fredricks, Blumenfeld, & Paris, 2004). Christenson and colleagues further separated behavioral components into behavioral and academic subtypes of engagement to facilitate the link to appropriate interventions (Appleton, Christenson, & Furlong, 2008; Appleton, Christenson, Kim, & Reschly, 2006). Each of these four subtypes of engagement – academic, behavioral, cognitive, and affective – is represented in a section of this book. In Fig. 3.1, we provide example indicators of each subtype of engagement.

Fig. 3.1
A circle where left side is observable engagement. It has academic and behavioral. The right part is internal engagement with cognitive and affective.

Four subtypes of student engagement and representative indicators. (Reschly, Appleton, and Pohl (2014). Reprinted with permission)

Full size image

A number of methods have been used to measure indicators of student engagement, such as observations, school record data, and surveys. We find that indicators of academic and behavioral engagement may be directly observed (e.g., time on-task, academic engaged time) and/or are readily available in school records (e.g., school disciplinary incidents, attendance, grades). For example, schools that utilize early warning systems (EWS) often contain information on students’ course failures, attendance, and disciplinary incidents and are easy for school personnel to access (Balfanz & Byrnes, 2019). In contrast, information on students’ cognitive and affective engagement is not readily observable or as easy to acquire. To illustrate this point, we have often asked questions like, how can one tell if a student feels like they belong? Believes their teacher or peers care about them? Feels supported? Sees how their schoolwork relates to their future goals? This is of particular importance when one considers the differences between students’ and teachers’ reports of student engagement (Skinner, Kindermann, & Furrer, 2009) and students’ engagement with instruction (Appleton & Lawrenz, 2011) and the connection between these higher inference, internal forms of engagement, associations with behavioral and academic engagement, and in turn, student outcomes (Reschly & Christenson, 2006, 2012). For cognitive and affective engagement, the primary source of information is the student themselves, with possible supplements from teachers, parents, or peers.

The following sections of this chapter will explore the assessment of academic, behavioral, and cognitive/affective engagement, with specific examples of how to assess the relevant indicators for each subtype. Cognitive and affective engagement are grouped together given that, as previously described, they represent internal subtypes of engagement that typically require student self-report to understand and most rating scales assess both subtypes together rather than separately. We describe a few of the most widely used student engagement surveys used to gather information about students’ cognitive and affective engagement. One measure of engagement, the Student Engagement Instrument (SEI), will be described in greater detail. The SEI is based on the model of engagement that grew out of work with Check & Connect (Chapters 1 and 2). We conclude with practical considerations and promising areas for educators on the assessment of student engagement.

Academic Engagement

The main methodologies for examining students’ time on-task, credits earned toward graduation, homework completion, and course grades include school records, permanent products, student- or teacher-report, and standardized observation schedules. As previously described, many schools using EWS already have data on indicators of academic engagement, such as credits earned toward graduation, course failures, and grade point average (GPA). Teachers also collect permanent products of academic engagement, including homework and class assignments. However, as educators know, students may complete assignments and not be engaged (e.g., copying work from another student). Assessing students’ academic engagement is incredibly important, as one of the strongest predictors of achievement is the amount of time students spend actively engaged in learning (Gettinger & Ball, 2008). Below, a few examples of observation systems, rating scales, and a combination of the two (i.e., Direct Behavior Ratings [DBR]) are described in more detail.

Observations

Considered by many as the “gold standard” for assessing engaged time, a number of observation systems exist within the field of education. Although we highlight a standardized observation schedule here, observations are flexible and educators can develop their own observation systems to suit their needs. Student observations can provide helpful information about whether or not students were academically engaged during class by examining the time that students remained on-task. Various observation schedules define on-task behavior differently. A common definition coined by Gettinger and colleagues (Gettinger & Ball, 2008; Gettinger & Walter, 2012), known as academic engaged time (AET), is defined as time that students are actively engaged in the learning process. But how can we tell if a student’s time on-task is productive and successful? Ardoin and Sayeski (2019) argued that a complete picture of AET can be understood by combining information of on-task behavior with achievement data for this reason.

Regardless of the definition for time on-task, there are multiple types of systematic direct observations of on-task behavior, including whole interval, partial interval, and momentary time sampling recording (Alberto & Troutman, 2012). The observer (who may be a teacher, paraprofessional, school psychologist, etc.) may choose to examine individual students or groups of students on rotation. Once the length of the observation has been determined (e.g., 30 minutes), the observation window is divided into intervals (e.g., 1 minute each). Various recording strategies each have their own strengths and weaknesses. With whole interval recording, the observer notes if the student was on-task throughout the entire interval; this type of recording may underestimate the occurrence of behavior. Partial interval recording records whether the student was on-task at least once during the interval, and therefore may overestimate the occurrence of behavior. Finally, momentary time sampling (regarded as a more accurate recording method) assesses whether or not the student was on-task at the end of the interval. Overall, shorter intervals lead to greater accuracy (e.g., Zakszeski, Hojnoski, & Wood, 2017) but are less practical for teachers to implement.

There are also a number of published observation measures. Volpe, DiPerna, Hintze, and Shapiro (2005) reviewed seven paper-and-pencil observational coding schemes designed to measure classroom behavior, noting the importance of examining psychometric properties (e.g., interrater reliability) and ensuring the selected code matches the situation. In some cases, the observational form was part of a larger screening and diagnostic system (e.g., the Academic Engaged Time Code of Walker and Severson’s Systematic Screening for Behavior Disorders), whereas other coding schemes were standalone measures (e.g., the Behavioral Observation of Students in Schools [BOSS]; Shapiro, 1996). The recording methods included duration recording (i.e., total time engaged with instruction/learning), partial interval, whole interval, momentary time sampling, and Likert-scale ratings. Regardless of the system selected, educators and interventionists must ensure that observers are adequately trained, conduct checks on interrater reliability, and keep in mind the need for several observations (Volpe et al., 2005).

The BOSS, developed by Shapiro (1996), is an excellent example of an observational code for academic engagement. The BOSS system divides on-task behavior into active engaged time (e.g., writing on an assignment) and passive engaged time (e.g., looking at the teacher). Off-task behavior is also divided into different categories, including motor activity (e.g., being out of seat), verbal behavior (e.g., nonacademic talk), passive nonengagement (e.g., looking out the window; Volpe et al., 2005). Active and passive engaged time are recorded using momentary time sampling, while off-task behaviors are scored using partial interval recording (Hintze, Volpe, & Shapiro, 2002). The observation period is split into 15-second intervals. The behavior of a peer is coded every fifth interval for the purpose of comparison. There is also a code for Teacher-Directed Instruction (i.e., an estimate of the amount of time the teacher engaged in direct instruction; Hintze et al., 2002; Volpe et al., 2005). At the end of the observation, scores for active and passive engaged time and off-task behaviors of the target student and the comparison peer are calculated (Hintze et al., 2002).

Surveys and Rating Scales

Elements of academic engagement are also sometimes included in student self-report measures, which ask students to report on their on-task behavior, grades, and homework completion. In fact, most homework research uses student self-report (e.g., Cooper, Robinson, & Patall, 2006). However, caution is warranted with regard to the accuracy of students’ self-reported grades and homework completion. For example, approximately 82% of high school students and 54.3% of college students accurately report their grades; only 36.1% of Scholastic Assessment Test (SAT) scores are accurately self-reported (Kuncel, Credé, & Thomas, 2005). Thus, while associations to other constructs are similar between self-reported and actual grades, and there is always value in seeking to understand student perceptions, we recommend using the more objective school data when available. One example of how students may self-report on their academic engagement may be drawn from Skinner’s Engagement versus Disaffection with Learning scale (EvsD; Skinner et al., 2009) wherein students are asked if they agree with the statement, “When I’m in class, I listen carefully.”

Direct Behavior Rating (DBR)

A DBR combines positive features of both systematic direct observations and behavior rating scales. Observations still occur at specific times, with well-defined operational definitions of the target behavior, but responses are gathered via a rating scale format (e.g., 0 = not at all engaged, 10 = completely engaged; Briesch, Chafouleas, & Riley-Tillman, 2016), providing a much more efficient means of collecting student data. Chafouleas, Riley-Tillman, and colleagues have conducted extensive research on the psychometric properties of DBRs, including comparisons to systematic direct observations and the sensitivity of DBRs to the effects of intervention. DBR is frequently used to estimate academic engagement and disruptive behavior. The National Center on Intensive Intervention found evidence of reliability and validity for the use of DBRs to measure academic engagement (www.intensiveintervention.org). Additional information, including examples and training materials, may be found at the National Center on Intensive Intervention and the University of Connecticut (www.dbr.education.uconn.edu).

Behavioral Engagement

Significant aspects of behavioral engagement include students’ attendance, participation in extracurricular activities, and disciplinary incidents. Similar to academic engagement, many indicators of behavioral engagement are regularly collected in schools. Various indicators of behavioral engagement can also be garnered through observation schedules and teacher-, student-, or parent-report.

Attendance

Information regarding students’ attendance within EWS may include information on excused and unexcused absences and tardies, which can be used to calculate the percent of days that a student is present and on time relative to the number of days enrolled. This percentage can be used to determine if a student is chronically absent, typically defined as missing 10% or more of school days for any reason (which is approximately 18 days missed per year; Attendance Works, 2013). There may be no differentiation of why a student is absent (e.g., a medical issue versus skipping class), so understanding why some students choose to not attend classes may need to be assessed through student self-report or parent-report measures. However, educators should be concerned with absences because it reflects the amount of instruction a student is missing, not whether a student’s absences are excused or unexcused.

Behavior

Within EWS or otherwise, schools also typically collect data on disciplinary incidents/behavioral referrals. Office discipline referrals (ODRs) are defined as when a student engages in a problem behavior that breaks a school rule that is observed or identified by school staff which results in a consequence and written documentation of the event (Sugai, Sprague, Horner, & Walker, 2000). It is important to know the frequency and severity of ODRs, which can be broken down by the resulting consequence (e.g., detention, in school suspension, out-of-school suspension, or expulsion). ODRs are commonly used as indicators of behavioral engagement within schools, and there is some support for using ODRs to assess school-wide behavior climate (Irvin, Tobin, Sprague, Sugai, & Vincent, 2004) and for progress monitoring purposes (McIntosh, Frank, & Spaulding, 2010). However, some caution is warranted, given inconsistencies across how the same problem behavior may be perceived by teachers and the resulting consequences for different students (Irvin et al., 2004; Sugai et al., 2000). This is problematic, given racial disparities in ODRs from preschool through high school (U.S. Department of Education, Office for Civil Rights, 2016).

Observations may also be used to assess disruptive classroom behaviors; many use the same observation technologies as those described for academic engagement in the previous section of this chapter. Educators may wish to simply record the event count (i.e., frequency) or duration (i.e., length of time) of disruptions during a set observation window for a given student. As previously discussed, the BOSS (Shapiro, 1996) observation schedule contains an off-task or disruptive behavior component (including off-task motor, verbal, and passive behaviors) in addition to active and passive academic engagement. Although some may want to measure disruptive behavior specifically, on-task behavior is often the preferred metric. This is because students cannot simultaneously be disruptive and on-task, and on-task behavior (e.g., AET) is also the goal behavior.

In addition, students’ disruptive behaviors in class can be assessed via student self- or teacher-report. For example, “I get in trouble at school” is an item on the School Engagement Measure-MacArthur (SEM; Fredricks, Blumenfeld, Friedel, & Paris, 2003). Similar to academic engagement, there may be issues with students’ accuracy of self-reported attendance or disruptive behavior compared to school data. Described in the Academic Engagement section of this chapter, DBRs can also provide a measure of a student’s disruptive behaviors based on teacher-report.

Extracurricular Participation

Extracurricular participation is an aspect of students’ behavioral engagement that is not always collected or compiled systematically by schools; schools may track information about the number of activities available and which adult(s) facilitate these activities, but not necessarily the time students spend engaged in these activities. This information is certainly possible to obtain; for example, large research studies like the National Education Longitudinal Study (NELS) or the Education Longitudinal Study (ELS) include questions about student extracurricular involvement. Generally, greater involvement in extracurriculars is associated with positive outcomes across multiple domains (e.g., greater academic achievement, higher self-esteem, and reduced delinquent behavior; Feldman & Matjasko, 2005; Fredricks, 2012), although differences based on type of activity and hours spent have been identified (Martinez, Coker, McMahon, Cohen, & Thapa, 2016). This information could easily be collected as a part of school data systems or be included in EWS. Schools may also collect surveys from students or parents to gather information on what extracurricular activities (e.g., sports, arts, clubs) a student is involved in and the time spent engaged in these activities.

Surveys and questionnaires can also be used to determine students’ preparedness for class (e.g., bringing the proper materials), frequency of fighting, etc. Questions such as, “How often did you come to class and find yourself without these things: (a) pencil or paper; (b) books; (c) your homework done” have been used in large, national, longitudinal research studies conducted through the National Center for Education Statistics. These studies, such as the NELS and ELS, have regularly been used to study many of the engagement indicators described in this chapter (e.g., extracurricular activities, attendance, motivation, cognitive engagement). What one loses in terms of theoretically driven and comprehensiveness of measures of constructs in such large datasets, one gains in terms of representativeness of the population, numbers, and ability to follow students over many years, as well as the addition of parent- and teacher-reports.

Cognitive and Affective Engagement

Generally, students’ cognitive (e.g., self-regulation skills, feelings of about relevance of school, value of learning) and affective (e.g., feelings of belonging and school connectedness, relationships with teachers and peers) engagement are assessed via student- or teacher-report. As previously described, these subtypes of engagement are frequently assessed together as they are both internal aspects of engagement, with students or teachers indicating how much they agree or disagree with various statements about a student’s engagement. Student self-report is the most common and practical method of assessing these subtypes of engagement as students reflect on whether items describe themselves (Fredricks & McColskey, 2012). In addition, student’s perceptions of their own engagement should be of significant interest to educators. As educators we may think we are providing our students with the best supports and interventions, but if students do not feel supported, then are we really doing our job (Chap. 2)? Will the intervention even be effective? Furthermore, some scholars argue that given the highly inferential nature of cognitive and affective engagement, it is necessary to use student self-report (Appleton et al., 2006). Although student self-report may be preferable for older students who are capable of reporting their thoughts and feelings, teacher-report may be most beneficial for younger students who may not accurately self-report (Fredricks & McColskey, 2012). However, a study by Skinner et al. (2009) found that teacher- and student-reports of engagement correlated for behavioral engagement, but less so for emotional (affective) engagement. Despite the necessity of understanding our students, their self-reported engagement is generally underutilized or not even assessed in many schools.

Given that engagement is a broad, unifying construct, a number of measures exist that tap different aspects of the construct, such as belonging (Goodenow, 1993), motivation (Pintrich & De Groot, 1990), student–teacher relationships (Pianta & Nimetz, 1991; Pianta & Stuhlman, 2004), climate (National Center on Safe Supportive Learning Environments, 2019), and identification (Voelkl, 1995, 2012), among others. Researchers have, for example, used survey items on boredom as an indicator of cognitive engagement (Reschly & Christenson, 2006) or focused on interrelated constructs such as students’ interest or enjoyment in their classwork (Shernoff, Csikszentmihalyi, Shneider, & Shernoff, 2003). In the history of psychological and educational research, engagement is still a relatively new construct; the development and validation of instruments for engagement are also relatively recent. Below, we highlight a few different student-report surveys: the Student Engagement Measure (SEM) (Fredricks et al., 2003), the Motivation and Engagement Scale (MES; Martin, 2007), and the Engagement versus Disaffection with Learning-Student Report (EvsD) (Skinner et al., 2009), followed by an in-depth description of the SEI in the next section. See Fredricks et al., 2011 for a comprehensive review of measures.

Student Engagement Measure

The SEM (Fredricks et al., 2003; Fredricks, Blumenfeld, Friedel, & Paris, 2005) is a student self-report paper-and-pencil rating scale that assesses behavioral, cognitive, and emotional aspects of engagement within the school setting. Fredricks et al. (2011) describe that the SEM was developed to be used for research on motivation and cognition. English and Spanish versions were developed and used with majority urban, low-income, Black and Hispanic 3rd through 5th grade students (Fredricks et al., 2005). Five items for behavioral engagement (e.g., attention, disciplinary incidents), six items for emotional engagement (e.g., interest in schoolwork, enjoyment), and eight items for cognitive engagement (e.g., self-regulation, value of learning) are rated on a 5-point Likert-type scale (1 = never, 5 = all of the time). Scale scores can be added and averaged for each engagement subtype. Adequate internal consistency, the 3-factor structure, and predictive validity have been supported in previous research (Fredricks et al., 2003; Fredricks et al., 2005). The rating scale is available in Fredricks et al. (2003, 2005) or can be obtained by contacting one of the developers, Dr. Fredricks.

Motivation and Engagement Scale

The MES (Martin, 2007, 2009c) is an extensively researched rating scale with versions developed for elementary/middle school students, high school students, and college students. The underlying theoretical model developed by Martin (2007), the Motivation and Engagement Wheel, provides the basis for this rating scale. The MES assesses four factors of engagement with 11 subscales: adaptive cognition (self-belief, learning focus, and valuing school), impeding/maladaptive cognition (anxiety, failure avoidance, and uncertain control), adaptive behavior (persistence, planning, and study management), and maladaptive behavior (self-sabotage and disengagement). Each subscale contains four items. A 7-point Likert-type scale is used for the high school version (1 = not at all true, 7 = strongly agree) while a 5-point scale is used for the younger version. Fredricks et al. (2011) summarize that the MES has been used for research on motivation and cognition, evaluation of interventions, diagnosis, monitoring at the student level, and monitoring teachers, schools, and/or districts. Various studies support the factor structure, construct and criterion validity, and reliability (internal consistency, test-retest) of the MES (Martin, 2007, 2009b). It is available for purchase for paper-and-pencil or online administrations (https://www.lifelongachievement.com/).

Engagement Versus Disaffection with Learning

The EvsD (Skinner et al., 2009; Skinner, Wellborn, & Connell, 1990) was developed with both student- and teacher-report versions to assess behavioral and emotional engagement subtypes. The EvsD assesses components of a student motivational theory (Connell & Wellborn, 1991; Deci & Ryan, 1985) that emphasizes the ways in which contexts (e.g., teachers and classrooms) can support or thwart student self-perceptions (e.g., feelings of relatedness, competence, and autonomy), thus resulting in engagement or disengagement. Therefore, both engagement and disaffection (i.e., negative engagement) are assessed in the EvsD within the classroom setting with four subscales (Skinner et al., 2009): behavioral engagement is indicated by action initiation, effort, and persistence (with five items); behavioral disaffection includes passivity, withdrawal, and inattention (five items); emotional engagement is demonstrated through enthusiasm, interest, and enjoyment (six items); emotional disaffection includes boredom, disinterest, and frustration (nine items). The student-report scale uses a 4-point Likert-type scale (1 = not at all true, 4 = very true). With a sample of 3rd through 6th grade students in suburban rural schools, Skinner et al. (2009) used confirmatory factor analysis to support the 4-factor model, determined that the factors correlated in expected ways, and found fair internal consistency and test-retest reliability. This paper-and-pencil rating scale is available in the appendix of Skinner et al. (2009).

Student Engagement Instrument

The paper-and-pencil versions of the Student Engagement Instrument are freely available for research and applied use with registration on the Check & Connect website at the University of Minnesota: http://checkandconnect.umn.edu/research/sei_register.html. Survey authors may receive royalties from a web-based application of the SEI.

In a few places in this book, we described our work with Check & Connect and the realization that in order to successfully reengage students for school completion, we had to pay attention to more than the academic and behavioral standards of the school. Rather, successful school completion efforts required attention to what we later came to refer to as cognitive and affective engagement at school and with learning. Students’ own perceptions are the most relevant means of gathering this information and thus, require self-report. With Check & Connect, we could easily access indicators of students’ academic or behavioral engagement, which were useful for determining risk and monitoring students’ levels of engagement, but we did not have a way to gather information about students’ cognitive or affective engagement. Thus, we developed, refined, and later extended the SEI for this purpose. For these reasons, we describe the SEI as being based on the model of student engagement that grew out of Check & Connect (Chap. 1).

The SEI was developed following an extensive review of the literature for terms thought to be included in the engagement meta-construct (Fredricks et al., 2004), such as belonging, identification, and self-regulation (Appleton et al., 2006). Subsequently, items were written to represent these various dimensions of engagement. We piloted and revised items via focus groups with an ethnically diverse sample of 8th graders. The first study of the SEI employed a number of potential items (n = 56) that were completed on a 4-point Likert-type scale (1 = strongly disagree, 4 = strongly agree)Footnote 1 with a large, diverse group of 9th grade students in an urban school district in the Midwestern region of the United States. Exploratory and confirmatory factor analyses led to a 35-item survey that represented 6 factors, 3 each of cognitive and affective (psychological) engagement (Appleton et al., 2006; Table 3.1).

Table 3.1 Factors of the Student Engagement Instrument
Full size table

Studies have confirmed this factor structure of the SEI (Betts, Appleton, Reschly, Christenson, & Huebner, 2010; Reschly, Betts, & Appleton, 2014) and found (a) adequate internal consistency reliability (Appleton et al., 2006; Betts et al., 2010; Reschly et al., 2014); and (b) low-to-moderate significant correlations, in expected directions, with other indicators of school functioning and behavioral and academic engagement (Appleton et al., 2006; Reschly et al., 2014). A study by Betts et al. (2010) found evidence of measurement invariance across grades 6–12 and gender, indicating scores function similarly across grades and for boys and girls. There is also evidence of convergent and divergent validity with another measure of engagement and motivation (Reschly et al., 2014).

Several longitudinal, predictive studies provide what is probably the most compelling evidence for the importance of students’ self-report of their engagement and support the use of the SEI for this purpose. A common strategy in this type of research is to see whether the SEI (full score or scores on subscales, such as Future Goals and Aspirations) is predictive of outcomes such as high school graduation or college attendance after many other variables commonly associated with those outcomes are accounted for (e.g., socioeconomic status, achievement test scores, 8th grade math and language arts grades, attendance, disciplinary incidents). Studies have found that the SEI contributed unique variance in predicting “college ready” graduation from the 8th grade (Pearson, 2014) and high school dropout and on-time graduation from the 9th grade (Lovelace et al., 2017; Lovelace, Reschly, Appleton, & Lutz, 2014). Fraysier et al. (2019) extended predictions from SEI scores in 10th, 11th, and 12th grade cohorts relative to college enrollment and persistence through the first year. Together these studies indicate that students’ self-reported cognitive and affective engagement is predictive of important academic outcomes across several years.

The SEI has also been examined in more practical ways, particularly with an eye for school-based applications. Lovelace et al. (2014) compared SEI scores among three groups of students: those with high and low achievement; between students identified with Emotional or Behavior Disorders (a higher risk disability group for dropping out) versus those with Speech and Language Impairment; and, among students exhibiting high behavioral disengagement in terms of absences and disciplinary infractions and those who were not behaviorally disengaged. Groups differed as expected (e.g., those with high behavioral disengagement reported lower cognitive and affective engagement than those who were not behaviorally disengaged).

Much of the work on reporting and practical application of the SEI has been conducted by Appleton and colleagues in the Gwinnett County Public Schools. Processes for data management, scoring, and reporting at the school- and district-levels are described in greater detail in Appleton, 2012 and Appleton and Silberglitt (2019). One example of the inclusion of cognitive and affective data in an EWS can be found in Fig. 3.2. We also consistently find that (a) SEI scores are significantly associated with attendance, behavior, and achievement for middle and high school students; (b) students’ responses, on average, decline from fall to spring of each year; and (c) students in each subsequent grade report less engagement than those in lower grades (e.g., 10th graders are less engaged, on average, than 8th graders, who in turn are less engaged than 6th graders; Appleton & Reschly, 2019).

Fig. 3.2
A table records historical and formative data for academic, behavioral, cognitive, and affective. The low, moderate, and high risk are marked.

Inclusion of Cognitive and Affective Engagement in early warning system Example. (Appleton (2012). Reprinted with permission)

Full size image

Extensions of the SEI

Given research that indicates student engagement is relevant for students from the first days of primary school through college, it follows that scholars and educators are interested in measuring student engagement across this span of schooling. As such, we have extended the SEI downward to elementary and upward to college.

SEI-E and SEI-E2

The first downward extension of the SEI was intended for students in grades 3–5 (SEI-Elementary; SEI-E). A panel of engagement experts and the head of a district’s school counseling department modified the original SEI items to ensure wording was developmentally appropriate for younger students (e.g., “Most of what is important to know you learn in school” was changed to “School is where I learn important things”; Carter, Reschly, Lovelace, Appleton, & Thompson, 2012). The SEI-E was piloted with a large, diverse sample of students in grades 3–5 and responses submitted to exploratory and confirmatory factor analyses. Instead of the expected 5-factor model (dropping the Intrinsic Motivation factor), 4-factors (24 items) better represented student responses for the SEI-E: Teacher–Student Relationships, Peer Support for Learning, Future Goals and Aspirations, and Family Support for Learning (Table 3.2). Some of the cognitive engagement items thought to represent students’ perceptions of their control and relevance of their schoolwork did not function well with younger students; items cross-loaded with those about future goals and teacher–student relationships. We speculated that these items may be too abstract (e.g., conscientiousness, locus of control) for students of this age. Small and significant correlations were found between the SEI factors and other indicators of school functioning (e.g., attendance, behavior).

Table 3.2 Factor structure of the SEI-E and the SEI-E2
Full size table

A similar process to the extension of the SEI to the SEI-E was followed for the downward extension of the SEI-E to grades 1 and 2 (SEI-Elementary 2; SEI-E2). Items were examined and language was modified, as needed, to ensure appropriateness for students of this age. The 24-item survey was piloted with a large, diverse group of students; responses were subjected to both exploratory and confirmatory factor analyses (Wright, Reschly, Hyson, & Appleton, 2019). Two response options were explored. For 1st grade students, a 3-point Likert-type scale option was used (1 = no, 2 = maybe, 3 = yes). Half of the 2nd grade sample completed the SEI-E2 on this 3-point Likert-type scale, the other half completed it with the 5-point Likert-type scale (1 = strongly disagree, 5 = strongly agree). Recommendations from this pilot study were that the 3-point Likert-type scale functioned well for 1st graders; the 5-point Likert-type scale worked best for 2nd graders. Results indicated the same factor structure among the SEI-E and the SEI-E2 (Table 3.2). In addition, there were some small, significant correlations with measures of school functioning and engagement (e.g., attendance); results also indicated lower income students reported lower levels of cognitive and affective engagement as measured by the SEI-E2. In comparison to the SEI, less research has been conducted with the SEI-E and SEI-E2. Further research that examines measurement invariance of the SEI-E and SEI-E2, associations between stability of earlier and later student engagement, as well as longitudinal predictive validity of the measures is needed.

SEI-C

Given the interest in and importance of the engagement construct at the college level, the SEI has also been adapted for use with college students (Student Engagement Instrument - College; SEI-C). Only minor changes in wording from the SEI were required (e.g., “teacher” to “professor”; Grier-Reed, Appleton, Rodriguez, Ganuza, & Reschly, 2012; Waldrop, Reschly, Fraysier, & Appleton, 2019). Research with the SEI-C has suggested both a 4-factor (Grier-Reed et al., 2012) and 5-factor model (Waldrop et al., 2019). Grier-Reed et al. removed the Control and Relevance of Schoolwork factor, similar to the SEI-E and the SEI-E2, whereas Waldrop et al. (2019) removed 6 items from the whole survey (for a total of 27) to maintain a good model fit with the 5-factors of the SEI. There is evidence of adequate to good internal consistency reliability for both models (Grier-Reed et al., 2012; Waldrop et al., 2019). In addition, Waldrop et al. (2019) found evidence of measurement invariance across online and paper-and-pencil surveys, suggesting that the SEI-C could be given either way. Waldrop et al. (2019) also found evidence of convergent and divergent validity with another measure of engagement and motivation for college students (MES-UC; Martin, 2009a). In addition, the SEI-C was associated with college GPA and career perceptions (Grier-Reed et al., 2012).

Summary

Thus, there is evidence to suggest that four subscales (Teacher–Student Relationships, Peer Support for Learning, Future Goals and Aspirations, and Family Support for Learning) and the total score on the SEI can be used to measure cognitive and affective engagement from 2nd grade through college. Our preliminary results indicated that for 1st grade students, the total score is more reliable than subscale scores. The Control and Relevance of Schoolwork subscale remains in the SEI for grades 6–12 and may also work with college-age students (Waldrop et al., 2019).

SEI-B

One driver of interest in measuring student engagement is the link between measurement and intervention. Some indicators of student engagement, such as attendance, time on-task, homework completion, participation in class, may be used for both determination of risk and progress monitoring (e.g., homework completion rate may identify students at-risk for poor class performance and/or skill acquisition difficulties and be used to determine whether a selected intervention is working with a student or group of students). Because cognitive and affective engagement are internal, highly inferential, and typically measured via survey methodology, which may be more time-consuming given the number of items and designed to be given less frequently (e.g., once or twice per year), the question of sensitivity to change or use for progress monitoring assumes greater importance. For this purpose, we piloted a somewhat shorter version of the SEI (Student Engagement Instrument-Brief; SEI-B, 27 items) that aligned with the 5-factor model (27 items, excluding the Intrinsic Motivation items) with high school students. Results supported the factor structure of the reduced scale and provided evidence of longitudinal measurement invariance, which is necessary to interpret changes in engagement across time and provides support that changes in scores on the SEI-B represent changes in their engagement (Pinzone, Appleton, & Reschly, 2019).

Considerations and Promising Areas

Multiple considerations regarding the assessment of student engagement warrant discussion. As described throughout this volume, there are various interventions that target different subtypes of student engagement. Relatedly, educators should consider the importance of examining multiple subtypes of engagement. Studies indicate that subtypes of engagement are generally connected for students, but some may experience varied engagement levels based on subtype (Li & Lerner, 2011; Wang & Peck, 2013). For example, a student may demonstrate greater behavioral engagement than cognitive engagement. Understanding these differences through effective measurement should also facilitate the identification of interventions that may be most effective given a student’s profile of engagement (Fredricks, Ye, Wang, & Brauer, 2019). That is, assessment indicating low affective engagement should result in the implementation of an intervention designed to improve affective engagement. Educators should also consider the comprehensiveness of interventions, as they may be more effective if they address engagement holistically rather than single subtypes (Chap. 2). Altogether, interventions explored throughout this handbook should be aligned with assessments of engagement.

Similarly, some research describes engagement and disengagement as separate spectra rather than along one continuum (Reschly & Christenson, 2012). This idea is supported by research findings that students may demonstrate high engagement but also experience high levels of burnout (Salmela-Aro, Moeller, Schneider, Spicer, & Lavonen, 2016). Specifically, Salmela-Aro and colleagues identified four profiles in their person-centered analysis of high school students: engaged, engaged-exhausted, moderately/at-risk for burnout, and burned out. Although engagement and burnout were overall negatively correlated, as one might expect, students who were simultaneously engaged and burnt out were at a higher risk for mental health difficulties (e.g., depressive symptoms). This is a group which may not be identified as needing support by teachers as they may attain good grades, attend classes, etc. Assessing aspects of burnout or disaffection may help educators catch these students and provide them with much needed support. Educators should consider the potential utility of directly measuring both engagement and disengagement with their students. Although some measures of student engagement include aspects of disaffection (Skinner et al., 2009) or disengagement (Martin, 2007), more research and development of these types of measures is needed.

Another important consideration is determining the appropriateness of student engagement measures for different populations. As described within this chapter, some measures of student engagement lend themselves to certain age groups. This is particularly important given evidence of the developmental differences of engagement across grade levels, with engagement generally declining from elementary school to high school (Appleton & Reschly, 2019; Martin, 2009b; NRC, 2004). In addition, the required level of student engagement increases over time (e.g., more homework, more opportunities to participate in extracurricular activities). Thus, certain subtypes of engagement may be more or less important given a student’s age. This was also demonstrated in the finding that aspects of cognitive engagement on the SEI did not function well with younger students as compared to older students (Carter et al., 2012); therefore, these higher-level or more complex aspects of engagement may be more important to assess with students in middle and high school compared to elementary students.

Various measures of student engagement also lend themselves to the individual, class, or school level. That is, educators must consider whether they are interested in assessing the engagement of a particular student (e.g., as part of an individual evaluation for special education) versus screening throughout a school. Some types of assessment are better for a given purpose; for example, direct observations are more time intensive and may be best used to evaluate individuals or classrooms rather than at the school level. School-wide affective/cognitive engagement is easier to capture through teacher-report or student self-report and would be more efficient for screening purposes, in addition to data already collected by schools on grades, behavior, and attendance such as through EWS.

Regarding screening, assessing student engagement holds implications for universal screening as a part of Multi-Tiered System of Support/Positive Behavioral Interventions and Supports (MTSS/PBIS) schools and schools that use EWS. Although many schools already measure behavioral and academic student engagement variables, adding measures of cognitive and affective engagement to these systems may be beneficial. Some researchers argue that cognitive and affective engagement may precede academic and behavioral engagement (Reschly & Christenson, 2006, 2012), indicating that assessing and intervening with these internal aspects of engagement may be particularly important. In addition, longitudinal studies of student engagement suggest the importance of measuring student engagement at multiple time points; while many students experience fairly stable levels of student engagement over time, many experience variable levels of engagement across schooling (Janosz, Archambault, Morizot, & Pagani, 2008; O’Donnell, Lovelace, Reschly, & Appleton, 2019). These variable trajectories of engagement indicate that different students may benefit from intervention at different points in time and encourage regular screening (e.g., once to twice per year).

Conclusion

We hope that through this chapter, you have learned about the importance of measuring student engagement and the variety of ways indicators of academic, behavioral, and cognitive/affective engagement can be assessed. Although we highlight examples of specific observation schedules and rating scales, we encourage you to consider which types of assessments best fit the needs of your setting. As we described, most schools already collect some data on student engagement; it is just a matter of how these data are conceptualized and utilized in efforts to identify at-risk students in need of intervention. In the following chapters of this handbook, you will explore many different interventions that aim to improve students’ engagement with school. Along with selecting and implementing interventions, consider their use in conjunction with assessment, whether to screen, monitor progress, and/or evaluate classrooms and schools.