Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Researchers, educators, and policymakers are increasingly focused on student engagement as the key to addressing problems of low achievement, high levels of student boredom, alienation, and high dropout rates (Fredricks, Blumenfeld, & Paris, 2004). Students become more disengaged as they progress from elementary to middle school, with some estimates that 25–40% of youth are showing signs of disengagement (i.e., uninvolved, apathetic, not trying very hard, and not paying attention) (Steinberg, Brown, & Dornbush, 1996; Yazzie-Mintz, 2007). The consequences of disengagement for middle and high school youth from disadvantaged backgrounds are especially severe; these youth are less likely to graduate from high school and face limited employment prospects, increasing their risk for poverty, poorer health, and involvement in the criminal justice system (National Research Council and the Institute of Medicine, 2004).

Although there is growing interest in student engagement, there has been considerable variation in how this construct has been conceptualized over time (Appleton, Christenson, & Furlong, 2008; Fredricks et al., 2004; Jimerson, Campos, & Grief, 2003). Scholars have used a broad range of terms including student engagement, school engagement, student engagement in school, academic engagement, engagement in class, and engagement in schoolwork. In addition, there has been variation in the number of subcomponents of engagement including different conceptuali­zations. Some scholars have proposed a two-dimensional model of engagement which includes behavior (e.g., participation, effort, and positive conduct) and emotion (e.g., interest, belonging, value, and positive emotions) (Finn, 1989; Marks, 2000; Skinner, Kindermann, & Furrer, 2009b). More recently, others have outlined a three-component model of engagement that includes behavior, emotion, and a cognitive dimension (i.e., self-regulation, investment in learning, and strategy use) (e.g., Archaumbault, 2009; Fredricks et al., 2004; Jimerson et al., 2003; Wigfield et al., 2008). Finally, Christenson and her colleagues (Appleton, Christenson, Kim, & Reschly, 2006; Reschly & Christenson, 2006) conceptualized engagement as having four dimensions: academic, behavioral, cognitive, and psychological (subsequently referred to as affective) engagement. In this model, aspects of behavior are separated into two different components: academics, which includes time on task, credits earned, and homework completion, and behavior, which includes attendance, class participation, and extracurricular participation. One commonality across the myriad of conceptualizations is that engagement is multidimensional. However, further theoretical and empirical work is needed to determine the extent to which these different dimensions are unique constructs and whether a three or four component model more accurately describes the construct of student engagement.

Even when scholars have similar conceptualizations of engagement, there has been considerable variability in the content of items used in instruments. This has made it challenging to compare findings from different studies. This chapter expands on our understanding of the measurement of student engagement in three ways. First, the strengths and limitations of different methods for assessing student engagement are described. Second, 11 self-report survey measures of student engagement that have been used in prior research are compared and contrasted on several dimensions (i.e., what is measured, purposes and uses, samples, and psychometric properties). Finally, we discuss limitations with current approaches to measurement.

What is Student Engagement

We define student engagement as a meta-construct that includes behavioral, emotional, and cognitive engagement (Fredricks et al., 2004). Although there are large individual bodies of literature on behavioral (i.e., time on task), emotional (i.e., interest and value), and cognitive engagement (i.e., self-regulation and learning strategies), what makes engagement unique is its potential as a multidimensional or “meta”-construct that includes these three dimensions. Behavioral engagement draws on the idea of participation and includes involvement in academic, social, or extracurricular activities and is considered crucial for achieving positive academic outcomes and preventing dropping out (Connell & Wellborn, 1991; Finn, 1989). Other scholars define behavioral engagement in terms of positive conduct, such as following the rules, adhering to classroom norms, and the absence of disruptive behavior such as skipping school or getting into trouble (Finn, Pannozzo, & Voelkl, 1995; Finn & Rock, 1997). Emotional engagement focuses on the extent of positive (and negative) reactions to teachers, classmates, academics, or school. Others conceptualize emotional engagement as identification with the school, which includes belonging, or a feeling of being important to the school, and valuing, or an appreciation of success in school-related outcomes (Finn, 1989; Voelkl, 1997). Positive emotional engagement is presumed to create student ties to the institution and influence their willingness to do the work (Connell & Wellborn, 1991; Finn, 1989). Finally, cognitive engagement is defined as student’s level of investment in learning. It includes being thoughtful, strategic, and willing to exert the necessary effort for comprehension of complex ideas or mastery of difficult skills (Corno & Mandinach, 1983; Fredricks et al., 2004; Meece, Blumenfeld, & Hoyle, 1988).

An important question is how engagement differs from motivation. Although the terms are used interchangeably by some, they are different and the distinctions between them are important. Motivation refers to the underlying reasons for a given behavior and can be conceptualized in terms of the direction, intensity, quality, and persistence of one’s energies (Maehr & Meyer, 1997). A proliferation of motivational constructs (e.g., intrinsic motivation, goal theory, and expectancy-value models) have been developed to answer two broad questions “Can I do this task” and “Do I want to do this task and why?” (Eccles, Wigfield, & Schiefele, 1998). One commonality across these different motivational constructs is an emphasis on individual differences and underlying psychological processes. In contrast, engagement tends to be thought of in terms of action, or the behavioral, emotional, and cognitive manifestations of motivation (Skinner, Kindermann, Connell, & Wellborn, 2009a). An additional difference is that engagement reflects an individual’s interaction with context (Fredricks et al., 2004; Russell, Ainsley, & Frydenberg, 2005). In other words, an individual is engaged in something (i.e., task, activity, and relationship), and their engagement cannot be separated from their environment. This means that engagement is malleable and is responsive to variations in the context that schools can target in interventions (Fredricks et al., 2004; Newmann, Wehlage, & Lamborn, 1992).

The self-system model of motivational development (Connell, 1990; Connell & Wellborn, 1991; Deci & Ryan, 1985) provides one theoretical model for studying motivation and engagement. This model is based on the assumption that individuals have three fundamental motivational needs: autonomy, competence, and relatedness. If schools provide children with opportunities to meet these three needs, students will be more engaged. Students’ need for relatedness is more likely to occur in classrooms where teachers and peers create a caring and supportive environment; their need for autonomy is met when they feel like they have a choice and when they are motivated by internal rather than external factors; and their need for competence is met when they experience the classroom as optimal in structure and feel like they can achieve desired ends (Fredricks et al., 2004). In contrast, if students experience schools as uncaring, coercive, and unfair, they will become disengaged or disaffected (Skinner et al., 2009a, 2009b). This model assumes that motivation is a necessary but not sufficient precursor to engagement (Appleton et al., 2008; Connell & Wellborn, 1991).

Methods for Studying Engagement

Student Self-report

Self-report survey measures are the most common method for assessing student engagement. In this methodology, students are provided items reflecting various aspects of engagement and select the response that best describes them. The majority of these self-report engagement measures are general and not subject specific, though there are some examples of measures that assess engagement in a specific domain like math (Kong, Wong, & Lam, 2003) or reading (Wigfield et al., 2008). One of the arguments for using self-report methods is that it is critical to collect data on students’ subjective perceptions, as opposed to just collecting objective data on behavioral indicators such as attendance or homework completion rates, which are already commonly collected by schools (Appleton et al., 2006; Garcia & Pintrich, 1996). Self-report methods are particularly useful for assessing emotional and cognitive engagement which are not directly observable and need to be inferred from behaviors. In fact, Appleton et al. (2006) argue that self-report methods should only be used to assess emotional and cognitive engagement because collecting data on these subtypes through other methods, such as observations and teacher rating scales, is highly inferential.

Self-report methods are widely used because they are often the most practical and easy to administer in classroom settings. They can be given to large and diverse samples of children at a relatively low cost, making it possible to gather data over several waves and compare results across schools. However, one concern with self-report measures is that students may not answer honestly under some conditions (e.g., if administered by their teacher with no anonymity provided), and thus, self-reports may not reflect their actual behaviors or strategy use (Appleton et al., 2006; Garcia & Pintrich, 1996). Furthermore, these measures generally contain items that are worded broadly (e.g., I work hard in school) rather than worded to reflect engagement in particular tasks and situations. For researchers interested in studying how much engagement varies as a function of contextual factors, the general items may not be appropriate.

Experience Sampling

Experience sampling (ESM) is another technique that has been used to assess student engagement in the classroom (Shernoff, Csikszentmihalyi, Schneider, & Shernoff, 2003; Shernoff & Schmidt, 2008; Uekawa, Borman, & Lee, 2007; Yair, 2000). ESM methods grew out of research on “flow,” a high level of engagement where individuals are so deeply absorbed in a task that they lose awareness of time and space (Csikszentmihalyi, 1990). In this methodology, individuals carry electronic pagers or alarm watches for a set time period. In response to ESM signals, students fill out a self-report questionnaire with a series of questions about their location, activities, and cognitive and affective responses (see Hektner, Schmidt, & Csikszentmihalyi, 2007, for more description of ESM methods). This methodology allows researchers to collect detailed data on engagement in the moment rather than retrospectively (as with student self-report), which reduces problems with recall failure and the desire to answer in socially desirable ways (Hektner et al., 2007). This technique can be used to collect information on variations in engagement across time and situations. However, this methodology also has some limitations. ESM methods require a large time investment for respondents, and the success of the method depends largely on participants’ ability and willingness to comply. In addition, engagement is a multifaceted construct and may not be adequately captured by the small number of items included in ESM studies.

Teacher Ratings of Students

Another method for assessing student engagement is teacher checklists or rating scales. Teacher ratings of individual students’ engagement, when averaged across students in their classrooms, offer an alternative perspective on student engagement from that reported by the students themselves. Some teacher rating scales include items assessing both behavioral and emotional engagement (Skinner & Belmont, 1993), and others reflect a multidimensional model of engagement (i.e., behavioral, emotional, and cognitive) (Wigfield et al., 2008). Researchers have also developed teacher ratings of student participation as indicative of behavioral engagement (Finn, Folger, & Cox, 1991; Finn et al., 1995), and teacher ratings of adjustment to school, as indicative of engagement (Birch & Ladd, 1997; Buhs & Ladd, 2001). This methodology can be particularly useful for studies with younger children who have more difficulty completing self-report instruments due to the reading demands and limited literacy skills. Some studies have included both teacher ratings and students’ self-reports of engagement in order to examine the correspondence between the two measurement techniques (Skinner, Marchand, Furrer, & Kindermann, 2008; Skinner et al., 2009b). These studies show a stronger correlation between teacher and student reports of behavioral engagement than teacher and student reports of emotional engagement. This finding is not surprising as behavioral indicators are directly observable. In contrast, emotional indicators need to be inferred from behavior, and it is possible that some students have learned to mask their emotions (Skinner et al., 2008).

Interviews

A few studies have used interview techniques to assess engagement in school (Blumenfeld et al., 2005; Conchas, 2001; Locke Davidson, 1996). Interviews fall on a continuum from structured and semistructured interviews with predesignated questions to interviews where participants are asked to tell their stories in more open-ended and unstructured ways (Turner & Meyer, 2000). One benefit of interview methods is they can provide insight into the reasons for variability in levels of engagement to help understand why some students do engage while others begin to withdraw from school. Interviews can provide a detailed descriptive account of how students construct meaning about their school experiences, which contextual factors are most salient, and how these experiences relate to engagement (Blumenfeld et al.). However, interviews are not without problems. The knowledge, skills, and biases of the interviewer can all impact on the quality, depth, and type of responses. There are also questions about the reliability (stability and consistency) and validity of interview findings (McCaslin & Good, 1996). Finally, concerns about social desirability are an issue with interview techniques.

Observations

Observational methods at both the individual and classroom level have also been used to measure engagement. At the individual level, observational measures have been developed to assess individual students’ on and off task behavior as an indicator of academic engagement (Volpe, DiPerna, Hintze, & Shapiro, 2005). Academic engagement refers to a composite of academic behaviors such as reading aloud, writing, answering questions, participating in classroom tasks, and talking about academics (Greenwood, Horton, & Utley, 2002). These measures use a form of momentary time sampling, in which an observer records whether a predetermined category of behavior is present or absent for an individual student during a defined time interval (Salvia & Ysseldyke, 2004). In addition to use in research studies, these techniques have been used by school psychologists to screen individual children in both typical and special needs populations, especially those at risk for disengagement and academic failure (Shapiro, 2004).

One concern with these types of observations is that they can be time consuming to administer, and observers may need to collect data across various types of academic settings (i.e., group work, seatwork) to get an accurate picture of student behavior. There are also concerns about the reliability of observational methods without proper training. Finally, another potential problem with individual observational measures is they provide limited information on the quality of effort, participation, or thinking (Fredricks et al., 2004; Peterson, Swing, Su, & Wass, 1984). For example, Peterson and colleagues found that some students judged to be on-task by observers reported in subsequent interviews that they were not thinking about the material while being observed. In contrast, many of the students who appeared to be off-task reported actually being very highly cognitively engaged.

Rather than assessing engagement with prespecified coding categories, other studies have used narrative and descriptive techniques to measure this construct. For example, Nystrand and colleagues (Gamoran & Nystrand, 1992; Nystrand & Gamoran, 1991; Nystrand, Wu, Gamaron, Zeiser, & Long, 2001) assessed the quality of instructional discourse in the classroom as an indicator of substantive engagement, defined as a sustained commitment to the content of schooling. In these studies, the frequency of high-level evaluation questions, authentic questions, and uptake (i.e., evidence that teachers incorporate students’ answers into subsequent questions) was observed as indicative of substantive engagement. That is, these teacher behaviors were assumed to involve active student engagement. Furthermore, Helme and Clarke (2001) observed math classes for indicators of cognitive engagement such as self-monitoring, exchanging ideas, giving directions, and justifying answers. Finally, Lee and her colleagues used observational techniques to examine the quality of students’ task engagement when involved in science activities (Lee & Anderson, 1993; Lee & Brophy, 1996). In these studies, they noted behaviors such as relating the task to prior knowledge, requesting clarification, and using analogies as measures of cognitive engagement.

The prime advantage of using observation techniques to study engagement is that they can provide detailed and descriptive accounts of the contextual factors occurring with higher or lower engagement levels. These descriptions enhance our understanding of unfolding processes within contexts. Observational methods also can be used to verify information about engagement collected from survey and interview techniques. The major disadvantages of observations are that they are labor intensive, and they usually involve only a small number of students and contexts. This raises concerns about the generalizability to other settings. Finally, the quality of descriptive observations depends heavily on the skills of the observer and on his or her ability to capture and make sense of what was observed (Turner & Meyer, 2000).

Comparison of Self-report Measures

In the next section, we describe survey measures that have been developed and used in prior research on student engagement and compare these surveys on several dimensions. This chapter builds on a literature review conducted to identify measures of student engagement available for use in the upper elementary through high school years (Fredricks & McColskey, 2010). We focus on student self-report measures because this is the most common method for assessing engagement and most likely to be of interest to researchers. As a first step toward identifying student engagement instruments, a literature search was conducted by members of the research team using terms that were broad enough to capture both subject-specific and general measures of student engagement. The search was restricted to studies published between 1979 (which was selected to predate the earliest emergence of engagement studies in the early 1980s) and May 2009 and resulted in 1,314 citations.

The research team systematically reviewed the 1,314 citations to identify named instruments used to measure student engagement. A total of 156 instruments were identified from the citations. From this initial list of 156 instruments, we excluded measures for a variety of reasons including (1) developed and used only with college age samples, (2) used only with special education populations, (3) measured a construct other than engagement (e.g., school bonding, attitudes toward school), (4) based on items from a larger national dataset [e.g., National Education Longitudinal Study (NELS), National Survey of American’s Families (NSAF)], (5) did not have enough published information on the measure, (6) adapted from other instruments already included in the list, or (7) developed for use in nonacademic subject areas (e.g., physical education). This resulted in a total of 21 measures (14 self-report, 3 teacher report, and 4 observation methods) which had been used with upper elementary to high school years.

By way of describing the substantial variation that exists across engagement measures, in this chapter, we describe 11 of these self-report measures. The 11 self-report measures in this chapter are for illustrative purposes and should not be considered an exhaustive list but rather are included to show the types of self-report instruments available. We compared these 11 self-report surveys on several dimensions including: definition of engagement, usage, samples, and psychometric information. The self-report measures ranged in length from a 4-item scale [School Engagement Scale Questionnaire (SEQ)] to the High School Survey of Student Engagement (HSSSE), a broad 121-item questionnaire. In some cases, the engagement items are a subset of a longer self-report instrument that assesses constructs other than student engagement.

Table 37.1 lists the names of the 11 self-report measures and the availability of the measure (i.e., journal article, website, and contact person). Eight measures are either available in a published source, can be accessed online, or are available by contacting the developer. Three of the instruments have commercially available services for purchase (School Success Profile [SSP], High School Survey of Student Engagement [HSSSE], and the Motivation and Engagement in Schools Scale [MES]). This cost covers questionnaire materials, administration of surveys, data preparation, individual and school reports, and other technical assistance related to the use of the information.

Table 37.1 Overview of 11 instruments
Full size table

What Is Measured

Table 37.2 lists the student self-report measures, the subscales/domains measured, and sample items for each of the subscales. Some of these survey instruments were explicitly designed to assess engagement, while other measures were designed to assess constructs such as identification with school, motivation, and self-regulation and strategy use, but have been used in subsequent studies as measures of engagement. For example, the Motivated Strategies for Learning Questionnaire (MSLQ) was initially designed to measure self-regulation and strategy use but has been used in some studies as an indicator of cognitive engagement (Pintrich & DeGroot, 1990). Similarly, the Identification with School questionnaire has been used in some studies as a measure of student identification with school and in other studies as a measure of emotional engagement.

Table 37.2 Self-report subscales with sample items
Full size table

There are a variety of ways to compare these measures. First, the surveys differ in terms of whether they focus on general engagement or subject- or class-specific engagement. Seven of the measures have items worded to reflect general engagement in school, while 4 of the self-report instruments are worded for use at the class level, in particular classes, or in particular skill areas [Attitudes Toward Math (ATM), Engagement vs. Disaffection with Learning (EvsD), Motivated Strategies for Learning Questionnaire (MSLQ), and School Engagement Questionnaire (SEQ)]. The self-report measures also differ in whether and how they conceptualize disengagement. Some of the measures include subscales that assess the opposite of engagement, which has been referred to as disengagement, disaffection, and alienation (Skinner et al., 2009a, 2009b). For example, three instruments have subscales measuring the extent of negative engagement (disengagement in the MES, trouble avoidance in the SSP, and behavioral disaffection and emotional disaffection in Engagement vs. Disaffection with Learning). Other measures imply that negative engagement is simply a low engagement score indicating a lack of engagement (Appleton et al., 2006). Finally, some of the measures blur the lines between engagement and contextual precursors (e.g., quality of students’ social relationships). For example, the three Student Engagement Instrument (SEI) “psychological engagement” subscales include items about students’ relationships with teachers and peers and support for learning from families that are not direct measures of engagement but indirect measures. Other self-report measures include separate scales for the aspects of classroom or school context that are assumed to influence or be related to engagement (e.g., Research Assessment Package for Schools).

Another way to compare the self-report survey measures is in terms of the extent to which they represent the multidimensional nature of engagement. Table 37.3 shows the various self-report measures in terms of whether they reflect behavioral, emotional, or cognitive aspects of engagement. In addition to differences in scale names used by developers (see Tables 37.2 and 37.4), there were differences in how the developers aligned similar items within the behavioral, emotional, and cognitive engagement constructs. For example, class participation was used as an indicator of both behavioral and cognitive engagement, and students’ valuing of school was used as an indicator of both emotional and cognitive engagement. Below we describe the subscales and items found across the 11 instruments by behavioral, emotional, and cognitive engagement.

Table 37.3 Dimensions of engagement assessed by instruments
Full size table
Table 37.4 Scales and subscales by each engagement dimensiona
Full size table

Behavioral Engagement

Eight measures have scales that seem to reflect (either by the subscale name or the sample items) aspects of behavioral engagement (see Tables 37.2, 37.3 and 37.4). Two of the behavioral subscales (behavioral disaffection and trouble avoidance) assess the extent of negative behavioral engagement (pretending to work, not turning in homework, and cutting class). A third subscale (disengagement subscale of the MES) includes items assessing both behavioral disengagement (e.g., each day I try less and less) and emotional disengagement (I have given up being interested in school). Across the various behavioral engagement scales/subscales, individual items ask students to report on their attention, attendance, time on homework, preparation for class, class participation, concentration, participation in school-based activities, effort, adherence to classroom rules, and risk behaviors.

Emotional Engagement

Eight subscales, either by subscale name or items, appear to reflect aspects of emotional engagement. Some subscales assess emotional reaction to class or school, while others assess the quality of students’ relationships with peers and teachers as an indicator of emotional engagement. Two subscales (emotional disaffection and disengagement) include items assessing the extent of negative emotions (discouragement when working on something, given up being interested in school). Overall, emotional engagement scales include questions about a myriad of topics related to emotional reactions to school such as being happy or anxious; expressing interest and enjoyment; reporting fun and excitement; reacting to failure and challenge; feeling safe; having supportive or positive relationships with teachers and peers; having family support for learning; expressing feelings of belonging; and perceiving school as valuable.

Cognitive Engagement Subscales

Six surveys include subscales measuring cognitive engagement, though there is large variation in how this is defined and measured. Cognitive engagement is used as a broad umbrella term for (1) beliefs about the importance or value of schooling, learning goals, and future aspirations; (2) ­cognitive strategy use (how deeply students study material); (3) self-regulatory or meta-­cognitive strategies (how students manage the learning processes such as planning and seeking information); and (4) doing extra work and going beyond the requirements of school. These measures of cognitive engagement incorporate aspects of motivation, self-regulated learning, and strategy use.

Purposes and Uses

The measures included in this chapter were developed from a range of disciplinary perspectives and for a variety of purposes. A number of the measures were developed by psychologists studying motivation, cognition, and engagement. For example, one widely used measure, the Engage­ment versus Disaffection with Learning scale, was part of a larger instrument that was initially developed to test the self-system model of student engagement. According to this model, the relation between classroom context (i.e., structure, autonomy support, and involvement) and patterns of action (cognitive, behavioral, and emotional engagement) is mediated through self-system processes (competence, autonomy, and relatedness) (Connell, 1990; Connell & Wellborn, 1991). The Engagement versus Disaffection scale has been most recently used in research by Skinner and her colleagues (see Furrer & Skinner, 2003; Skinner et al., 2008, Skinner et al., 2009b, for examples). In 1998, Connell and ­others at the Institute for Research and Reform in Education (www.irre.org) revised the original instruments to provide a shorter set of instruments (RAPS) for use in evaluating school reform efforts based on the same theoretical framework. Two of the survey measures identified in the review (Attitudes Toward Mathematics Survey [ATM] and the Motivated Strategies for Learning Questionnaire [MSLQ]) were developed as part of research exploring the relationships between students’ self-regulation, cognitive strategy use, and achievement outcomes. Research in this area examines the use of cognitive, meta-cognitive, and self-regulatory strategies that foster active cognitive engagement in learning (Corno & Mandinach, 1983; Meece et al., 1988).

Other measures were developed by researchers studying the relationship between context and engagement. For example, the Student Engage­ment Measure (SEM) – MacArthur was developed for a longitudinal study of the relationship between classroom context and engagement in urban minority youth in the upper elementary grades (Fredricks et al., 2005). In addition, the School Engagement Scale/Questionnaire (SEQ) was developed as part of a large study in nine high schools that reported on ways that parents, peers, and communities influence students’ commitment to, or engagement with, school (Steinberg et al., 1996). This scale has subsequently been used by researchers trying to understand factors that explain differences in vocational attitudes and career development behaviors among subgroups of high school students (Perry, 2008; Wettersten et al., 2005).

Increasing student engagement is the primary goal of many interventions to reduce dropout rates (Appleton et al., 2008; Finn, 1989). Two measures were developed in the context of this work on dropout prevention [Identification with School Questionnaire (ISQ) and Student Engagement Instrument (SEI)]. For example, the Student Engagement Instrument (SEI) was developed to measure affective (formerly psychological) and cognitive engagement and to expand on the behavioral and academic indicators that were collected as part of Check & Connect, an intervention model designed to improve student engagement at school, reduce dropouts, and increase school completion (Anderson, Christ­enson, Sinclair, & Lehr, 2004). The Student Engagement Instrument (SEI) is currently being used to evaluate the effectiveness of district initiatives to improve student engagement in the Gwinnett County Public Schools (this volume). The Identification with School questionnaire was developed to assess the extent to which students identify with or disengage from school, and was based on the theory that school identification is a crucial factor in the prevention of school dropouts (Finn, 1989).

Other measures have been developed to help schools and districts monitor engagement and to assist schools in identifying areas in need of improvement. For example, the High School Survey of Student Engagement (HSSSE) was developed to provide descriptive and comparative data on high school students’ views about their schoolwork, the school learning environment, and interactions with the school community, relative to the responses of other schools (Yazzie-Mintz, 2007). Each school that participates receives a customized report that compares the students’ responses to that of other schools. Similarly, the School Success Profile (SSP) was developed to provide insight into how students perceive themselves and their environments and to compare school scores relative to a national sample (Bowen, Rose, & Bowen, 2005).

Finally, one survey measure [the Motivation and Engagement Survey (MES)] was developed to diagnose and identify students who are struggling or at risk for disengagement and academic failure. The MES creates profiles for individual students based on responses to 11 different ­subscales reflecting a multidimensional model of motivation and engagement. This measure has been used to diagnose students with low motivation and engagement, in studies evaluating the effectiveness of interventions and in studies examining demographic differences in engagement and motivation (Fredricks & McColskey, 2010).

Samples

The surveys included in this chapter have been used with students from the upper elementary school years (third to fifth grades) through the high school years. Two of the measures were initially developed for use with upper elementary school populations [Engagement vs. Disaffection with Learning and MacArthur (SEM)]. On the other end of the spectrum, the Motivated Strategies for Learning Questionnaire (MSLQ) was originally developed for use with college samples, but a version was adapted for use with middle school students. In addition, the High School Survey of Student Engagement (HSSSE) was modeled after the National Survey of Student Engagement (NSSE), a widely used measure of student engagement at the college level.

Table 37.5 shows that the majority of measures have been used with ethnically and economically diverse samples. In addition, four of the measures have been translated into other languages [MSLQ, MacArthur measure, SSP, and SEI]. For example, the MSLQ has been translated into multiple languages and has been used in English-speaking and non-English-speaking countries all over the world (Garcia-Duncan & McKeachie, 2005). The SSP and the MacArthur measure have been translated into Spanish. Sections of the SSP have also been translated into Hebrew, Lithuanian, Romanian, and Portuguese (Fredricks & McColskey, 2010). Finally, the SEI has been translated into Portuguese and Mandarin (Moreira, Vaz, Dias, & Petracchi, 2009).

Table 37.5 Samples
Full size table

Psychometric Information

Technical information on reliability and validity was found on all but one self-report measure, though there were variations in the amount and types of technical information available (see Fredricks & McColskey, 2010, for more detailed information on psychometric properties). The one exception was the High School Survey of Student Engagement (HSSSE) which currently has no published information on the psychometric property of this measure. However, developers of this measure have indicated that a reliability and validity study is currently underway. They also make reference to the reliability and validity reported on the National Survey of School Engagement, a widely used measure of engagement at the college level, from which the HSSSE was adapted.

Reliability

Internal consistency is the extent to which individuals who respond in one way to items tend to respond the same way to other items intended to measure the same construct. A Cronbach’s alpha of.70 or higher for a set of items is considered acceptable (Leary, 2004). Cronbach’s alpha of the engagement scales/subscales was reported for all but one measure. The HSSSE was originally reported on an item-by-item basis, but recently the developers began grouping the 121 items in the questionnaire by the three aspects of engagement. However, no information on the internal consistency of these subscales is currently available.

The reliabilities of these scales reported by both the developers and other users of the measure ranged from.54 to.93, with most scales in the range of the.70 to.80 (see Table 37.6). Because of the variation in alphas across measures and subscales, it is important to examine the information on reliability more closely in light of the particular sample and intended use.

Table 37.6 Reliability information
Full size table

In addition, three of the measures [Motivation and Engagement Survey (MES), Engagement vs. Disaffection with Learning (EvsD), and the Student Engagement Instrument (SEI)] reported information on test-retest reliability, or the extent to which two different administrations of the measure give the same results.

Validity

In this chapter, we summarize the information available on these measures under the broad umbrella of construct validity (see Fredricks & McColskey, 2010, for more information on ­validity). One way to investigate construct ­validity is to examine whether the correlations between the engagement scales and the other related constructs are in the hypothesized direction based on theory and prior empirical work. The following three examples from the surveys illustrate these relations. First, the engagement scales in the School Success Profile (SSP) were positively correlated with teacher, parent, and peer support variables (Bowen, Rose, Powers, & Glennie, 2008). Additionally, the three engagement subscales (i.e., behavioral, emotional, and cognitive) in the MacArthur measure (SEM) were moderately correlated with students’ perceptions of aspects of the academic and social context, school value, and school attachment (Fredricks et al., 2005). Finally, the cognitive strategy use and self-regulation scales of the MSLQ were positively correlated with students’ self-reports of interest, efficacy, and task value (Pintrich, 1999; Pintrich & DeGroot, 1990).

We also found evidence of criterion-related validity, or the extent to which a measure is associated with a key behavior or outcome (Leary, 2004) on the majority of measures. Eight out of the 11 measures reported positive correlations between engagement and indicators of academic performance. For example, several studies using the MSLQ have documented that cognitive strategy use and self-regulation scales are positively related to course assignments, exams, and grades (Pintrich & DeGroot, 1990; Wolters & Pintrich, 1998; Wolters, Yu, & Pintrich, 1996). Similarly, the two engagement scales of the RAPS were positively correlated with indicators of performance (Institute for Research and Reform in Education, 1998). In addition, Appleton, Reschly, and Martin (under review) documented significant differences between affective and cognitive engagement data and academic performance; students with the lowest reports of engagement had the lowest scores on state tests and the lowest graduation rates. Finally, three of the measures (SSE, ISQ, and SEM) reported correlations between engagement and indicators of participation (i.e., attendance, teacher ratings of participation) (Fredricks & McColskey, 2010).

Another way to assess construct validity is to use exploratory or confirmatory factor analyses techniques to examine how survey items load onto the engagement constructs. Seven of the instruments reported results from either exploratory or confirmatory factor analyses. However, because of large differences in both the number and types of items, it is challenging to compare the resulting scales from these analyses. The following three examples illustrate this variability. Voelkl (1997) used confirmatory factor analysis with 16 items on the Identification with School Questionnaire (ISQ) on a sample of 3,539 urban eighth graders. These analyses confirmed two subscales: belonging and value. Martin (2008, 2009a, 2009b) used confirmatory factor analyses on 44 items of the Motivation and Engagement Survey with large samples of Australian middle and high school students. These analyses resulted in 11 subscales (self-belief, learning focus, valuing of school, persistence, planning, study management, disengagement, self-sabotage, anxiety, failure avoidance, and uncertain control). Finally, Appleton et al. (2006) used confirmatory factor analysis with 56 items from the Student Engagement Instrument on a sample of 1,931 ninth graders. These analyses resulted in six subscales (teacher-student relationships, peer support, family support, control relevance of schoolwork, future aspirations, and extrinsic motivation). A more recent confirmatory factor analysis of the Student Engagement Instrument (SEI) showed evidence of the validity of five subscales, dropping extrinsic motivation as a subscale (Betts, Appleton, Reschly, Christenson, & Huebner, 2010).

Finally, construct validity can be assessed by examining the correlations between engagement measured by different methodological approaches. Three of the measures (EvsD, RAPS, and SEM) reported correlations of scores from student self-report measures with other techniques to assess engagement (teacher ratings, external observers, and interviews). For example, the student and teacher versions of the Engagement versus Disaffection with Learning scale were moderately correlated with each other. In addition, teacher reports of behavioral and emotional engagement correlated with external observations of on and off task behavior, but student self-reports did not correlate with external observations of engagement (Skinner et al., 2008). Similarly, the developers of the SEM-MacArthur measure correlated students’ self-reports with teachers’ reports of student behavior. In addition, students’ responses on the survey were compared to interviews about engagement with the same sample of students. They reported a positive correlation between the three subscales of engagement (behavioral, emotional, and cognitive engagement) and numerical ratings given to interview responses (Blumenfeld et al., 2005; Fredricks et al., 2005).

Overall, the psychometric information on these measures suggests that student engagement can be reliably measured through self-report methods. In addition, the measures of engagement relate to both contextual variables and outcome variables as expected. Moreover, the fact that engagement has been shown to positively correlate with achievement indicates that it could serve as a worthwhile intermediate outcome to monitor. Finally, the results of exploratory and confirmatory factor analyses demonstrate the variability in the different conceptualizations of engagement and challenges in comparing across different survey measures.

Conclusions and Future Directions

As evident from this chapter, there are a variety of methods for assessing engagement, each with strengths and limitations and useful for particular purposes. However, even when researchers use the same methodology (i.e., self-report surveys), there is variation in how engagement is defined and measured. For example, some of the surveys in this chapter focus primarily on behaviors such as effort, homework, and attendance. In contrast, other surveys include items related to emotional dimensions such as relationships with teachers and cognitive dimensions such as strategy use. Below we outline some of the key concerns related to measurement.

Operationalization of Engagement

As outlined in prior reviews (Appleton et al., 2008; Fredricks et al., 2004; Jimerson et al., 2003), there is considerable variation in the definitions of engagement used across studies. Although scholars have used a broad range of terms for engagement, the two most common are student engagement and school engagement. Differences between the terms were raised in a prior review of the literature (see Appleton et al., 2006, for more discussion). We echo Appleton et al. (2006) point that greater attention needs to be paid to the use of the terms student engagement and school engagement in future work and potential differences in the meaning of these constructs. Another concern is that many of the definitions of engagement overlap with other educational constructs (i.e., school bonding, belonging, and school climate). It is important that researchers acknowledge this overlap with ­earlier literatures, many of which have stronger bodies of literature supporting the construct, and be clearer in terms of both research and practice about the “value added” from studying engagement (Fredricks et al., 2004; Jimerson et al., 2003).

Although there is some agreement that engagement is a multidimensional construct, there is variation in both the number (i.e., 2–4) and types (academic, behavioral, emotional, and cognitive) of engagement dimensions. As can be seen in this chapter, different conceptualizations of engagement have resulted in a variation in the content of items used in instruments. Moreover, even within the same dataset, researchers sometimes use different variables to operationalize engagement, often without a strong theoretical or conceptual framework guiding the choice of indicators. For example, researchers have selected different items from large nationally representative datasets like the National Education Longitudinal Study (NELS:88) to create different scales of engagement (Glanville & Wildhasen, 2007). This makes it difficult to compare findings concerning both the predictors and outcomes of engagement. An additional problem is that similar items have sometimes been used to assess different dimensions of engagement. For example, student effort is used by some to describe the degree of psychological investment in learning (i.e., cognitive engagement) and by others to reflect basic compliance with schoolwork (i.e., behavioral engagement). In addition, students’ valuing of school has been used as part of both emotional and cognitive engagement scales.

Given the variations in the definitions and measures of student engagement, one of the first steps to improving measurement is for researchers to more clearly describe their particular definition of engagement. It will also be important as a field to come to a stronger consensus on the operationalization of engagement (Appleton et al., 2008; Fredricks et al., 2004). Currently, we believe that the strongest empirical and theoretical support exists for a tripartite conceptualization of student engagement which includes a behavioral, emotional, and cognitive subcomponent. However, further empirical research is needed to determine what are the best indicators of each subtype and the extent to which behavioral, emotional, and cognitive engagement are separate constructs.

Assessing Malleability of Engagement

Several scholars have argued that one of the strengths of engagement is that it represents a shift from the focus on individual characteristics toward an investigation of potentially malleable contextual factors that can be targeted in interventions (Appleton et al., 2008; Fredricks et al., 2004; Sinclair, Christenson, Lehr, & Anderson, 2003). Appleton (Chap. 35) presents an example of how the Student Engagement Instrument (SEI) is being used to guide intervention efforts aimed at improving student engagement and identifying students who are at risk. Unfortunately, many of the current measures make it difficult to test questions of malleability. The majority of engagement measures tend to be general (i.e., I like school), though there are a few examples of domain-specific measures (i.e., Kong et al., 2003; Wigfield et al., 2008). Furthermore, measures are rarely worded to reflect specific situations or tasks, making it difficult to examine the extent to which engagement varies across contexts. In addition, most current survey measures do not adequately address qualitative differences in each of the dimensions of engagement. For example, behavioral engagement can range from basic compliance with school rules to doing more than is required (Finn, 1989). Emotional engagement can range from liking school to a deeper attachment and identification with the institution (Fredricks et al.). Cognitive engagement can range from the use of shallow rote strategies to the use of deep processing strategies that promote deep understanding (Greene, Miller, Crowson, Duke, & Akey, 2004). Future research should explore qualitative differences in engagement across different contexts (i.e., teacher directed as compared to small group work).

Developmental Differences

Another important research area concerns developmental differences in the measurement of engagement. There may be different indicators of engagement depending on the age of the child, and these different types of engagement may change and evolve over time. For example, students might not be cognitively engaged in learning until they are able to self-regulate and become intentional learners (Fredricks et al., 2004). There is a critical need for research that uses confirmatory factor analytic techniques to validate surveys at different ages. One example of this is research using the Motivated Learning and Strategy Use Questionnaire (MSLQ). Exploratory and confirmatory factor analyses with this measure demonstrated different factor structures in college and middle school classrooms. In college samples, analyses resulted in four cognitive strategy factors (rehearsal, elaboration, organization, and meta-cognitive strategy use) (Pintrich, Smith, Garcia, & McKeachie, 1993). In contrast, factor analyses with younger students resulted in one general cognitive strategy use scale and one meta-cognitive strategy use scale, suggesting that younger students do not make as fine of distinctions between types of strategy use as older students (Pintrich & DeGroot, 1990; Wolters, Pintrich, & Karbenick, 2005).

Variation Across Groups

Another important question is whether engagement can be measured similarly for all groups of students. If measures of engagement behave differently by race, SES, gender, and grade, and these differences are not taken into account, comparisons in the level of engagement or effects across groups are invalid (Glanville & Wildhagen, 2007). For example, Glanville and Wildhagen used confirmatory factor analyses to create a measurement model for school engagement using NELS:88 across White, African American, Latino, and Asian youth. They found that this measurement model was invariant across ethnic groups, and it was therefore appropriate to compare the effects of disengagement across these groups. In addition, Betts et al. (2010) used confirmatory factor analysis to test model invariance across gender and grade. They found that the Student Engagement Instrument (SEI) had a similar factor structure across gender and grade-level. Further research should use confirmatory factor analytic techniques to validate existing instruments and factor structures across different groups of students (i.e., age, gender, race, and SES).

Use of Multiple Methods

Finally, we recommend researchers use multiple methods to assess engagement. Qualitative methods can help to supplement our understanding of the contextual factors that are associated with engagement. In depth descriptions of context and engagement are critical for knowing how and where to intervene. Moreover, the measurement of engagement is often related to the affordances in the environment, and it may be difficult to assess cognitive engagement in classrooms where tasks involve only superficial strategy use. Observational methods can be used to better understand variations in engagement across different contexts and how this variation may relate to affordances within the context. Qualitative methods also are a useful method for describing how the different types of engagement evolve and develop to help understand why some youth begin to disengage from school (Fredricks et al., 2004). Finally, most current methods do not adequately capture the dynamic and interactive nature of engagement. One promising approach to assessing the dynamic nature of engagement is experience sampling methods which can track fluctuations in engagement over time.

In sum, although the construct of student engagement has considerable promise, measurement issues should continue to be explored in order to fully realize this promise (Glanville & Wildhagen, 2007). We believe that a more systematic and thoughtful attention to the measurement of student engagement is one of the most pressing and imperative directions for future research. First, it is important that researchers are clearer about their definitions of student engagement and how their conceptualizations of this construct relate to both other scholars’ operationalization of student engagement and to other related educational constructs. The field will benefit if researchers spent less time generating slight variations on this construct and spent more time on theory development and integration of the different conceptualizations of engagement. Greater consistency in the use of measures across studies will also make it easier to compare findings about the outcomes and precursors of student engagement. However, as our review has also highlighted, there is large variation in the extent of psychometric evidence available on current measures. Future research testing the psychometric properties of these measures is critical. Finally, we strongly support the use of a wide range of methods to assess engagement including observations, interviews, and experience sampling techniques.