Keywords

Success in algebra during the middle grades is widely recognized to be a critical gatekeeper that constrains students’ decisions about whether to pursue further educational opportunities in Science, Technology, Engineering, and Mathematics (STEM) fields (Adelman, 2006). Unfortunately, during this developmental period many students show significant declines in motivation toward school in general and mathematics in particular (e.g., Archambault, Eccles, & Vida, 2010; Blackwell, Trzesniewski, & Dweck, 2007). One way that researchers have sought to spark students’ interests and build their sense of competence in mathematics is through the use of various technological media. These technologies have ranged in complexity and cost from the simple and inexpensive, such as repurposing television programs, to the more complicated and expensive, such as specially designed mathematical experiences based on immersive virtual environments and computer games. We refer to the collection of these various types of technology media that aim to improve learning and motivation in school settings as game-based technology activities.

Despite the widely accepted notion that all game-based technology activities are inherently engaging, the evidence regarding their motivational effectiveness is mixed (Moos & Marroquin, 2010). Part of the reason may be that many different types of technologies are available, and each can be designed well or poorly to leverage various aspects of motivation (e.g., engagement, self-efficacy, tenacity) in different ways. As a step toward improving our understanding of the potential impact of game-based technology activities on students’ motivation in mathematics, the goal of this project was to investigate the relationship between (a) specific game-based technology activities that exemplify various motivational constructs, (b) students’ engagement in mathematics and perceived competence in pursuing STEM careers, and (c) students’ mathematics learning from a short algebra lesson.

Our research questions were as follows. First, what is the impact of the 4-day intervention on students’ motivation in mathematics, including interest in pursuing STEM careers? Second, to what extent is this impact influenced by factors such as the type of game-based technology activity the students received and/or students’ demographic and academic characteristics (e.g., gender, race/ethnicity, prior achievement)? Third, to what extent is this impact influenced by teacher-level factors such as credentialing in mathematics education, undergraduate major, years of experience, and teachers’ beliefs (e.g., teaching self-efficacy)?

We begin by reviewing evidence on how and why game-based technology activities might impact students’ motivation in STEM fields.

Motivating Students to Learn STEM

As the National Academy of Sciences (2011) indicated, certain key ingredients are relevant for students who want to pursue STEM careers. These ingredients include a robust confidence in math and science capability (self-efficacy), the ability to see one’s abilities in STEM as able to improve over time (implicit theories of ability), and the ability to develop a passion or sustained interest in becoming a scientist or engineer (value beliefs). We discuss each in turn.

Capable students plagued by a loss of confidence about their capacity to succeed in math and science typically avoid careers that require a strong background in those subjects (Lent et al., 2005). Decades of research have shown that students’ self-efficacy, defined by Bandura (1997) as “the belief in one’s capabilities to organize and execute courses of action required to produce given attainments” (p. 3), is a powerful influence on motivation and achievement. Bandura (1997) hypothesized several sources of self-efficacy, including mastery experience (the interpreted results of one’s past performance), vicarious experience (observations of others’ activities, particularly individuals perceived as similar to oneself), and physiological and affective states (anxiety, stress, and fatigue)—each of which has been linked to performance in math and science, including students’ persistence in STEM fields and choice of STEM majors (e.g., Britner & Pajares, 2001; Gwilliam & Betz, 2001; Lau & Roeser, 2002; Lent, Brown, & Larkin, 1984).

Like self-efficacy, implicit theory of ability (defined as a belief about the nature of intellectual ability (Dweck & Leggett, 1988)) plays an important role in motivation. Some individuals believe that their abilities are a fixed characteristic, and that nothing can be done to change that (i.e., “I’m not smart in math, and there isn’t anything I can do about it”). This is referred to as a fixed theory of ability. On the other hand, other individuals believe that, with sufficient effort and the proper strategies, one can become more able (i.e., “If I work hard in my math class, I can get smarter in math”). This is known as an incremental theory of ability. A large body of research has shown that implicit theory of ability plays a key role in students’ academic motivation, achievement, and career choices (e.g., Blackwell et al., 2007; Good, Rattan, & Dweck, 2012; Grant & Dweck, 2003; Hong, Chiu, Dweck, Lin, & Wan, 1999).

In addition to the self-efficacy and implicit theories of ability, value beliefs are also a significant determinant in students’ motivation and achievement (Eccles et al., 1983). Value beliefs in mathematics and science deal with the question, “Do I want to pursue more opportunities in mathematics and science?” Eccles et al. defined values as being composed of several distinct constructs. First, students’ interest or intrinsic value can affect the activities they pursue—activities that are more enjoyable are more likely to be pursued than are activities that are perceived to be lackluster. Second, students’ perceptions of the utility of an activity refer to how valuable students perceive an activity to be. If an activity is perceived to be a steppingstone toward students’ desired future endeavors, then students are more likely to pursue it. Finally, doing well in mathematics and science may influence students’ identity or feelings of self-worth. This attainment value describes how important doing well in mathematics and science is to students’ identity or feelings of self-worth. Numerous studies have found that interest value predicts STEM career choice (Lent, Lopez, Lopez, & Sheu, 2008; Lent, Paixão, da Silva, & Leitão, 2010), as well as choice in taking STEM courses (Eccles, Midgley, & Adler, 1984; Watt, Eccles, & Durik, 2006).

Motivation and Game-Based Technology Activities

How can the constructs described above be targeted through game-based technology activities to support the motivation of students in mathematics and science? Although the literature on technology and motivation is quite large, relatively few of these studies employ frameworks that are grounded in well-studied psychological theories of motivation (Moos & Marroquin, 2010). Moos and Marroquin noted that the results about the effectiveness of game-based technology activities as a motivational tool are mixed.

With regard to self-efficacy, there is some evidence that engagement with innovative game-based technology activities in academic settings can positively impact self-efficacy toward STEM. For example, Ketelhut and colleagues (Ketelhut, 2007; Ketelhut, Nelson, Clarke, & Dede, 2010) found that students’ self-efficacy for scientific inquiry before using a Multi-User Virtual Environment (MUVE) called River City was related to their behaviors within the virtual world. In particular, less self-efficacious students manifested a self-efficacy boost through mastery experiences gained through engagement in the activities of the MUVE (see also Liu, Hsieh, Cho, & Schallert, 2006).

Game-based technology activities also seem to be a promising avenue for impacting implicit theory of ability. In particular, Dweck and her colleagues have developed a web-enabled intervention, Brainology®, which is designed to enhance implicit theory of ability. Students are introduced to two cartoon characters who guide them through the web-based environment, where they learn about the functions of the brain, including that the brain is like a muscle—with conditioning, it can get stronger—an attitude which is linked to an incremental view. Donohoe, Topping, and Hannah (2012) conducted a quasi-experimental study on 33 adolescents (ages 13–14) and found that Brainology® led to a significant increase in students’ incremental view of ability.

With respect to value beliefs, researchers have argued that well-designed game-based technology activities can be used to target students’ interest value beliefs by making learning goals relevant and meaningful, and by allowing students to identify with characters within the technology environment (Gee, 2003; Squire, 2003). For example, Hickey, Moore, and Pellegrino (2001) showed that the use of The Adventures of Jasper Woodbury videodisc activity led to gains in students’ mathematics interest, although these gains appeared to result both from the game-based technology activities as well as from teachers’ beliefs and instructional practices.

Context of the Present Study

To investigate the potential impact of game-based technology activities on students’ mathematics motivation, we designed three different types of game-based technology activities (or “inductions”). The inductions differed along two main dimensions. First, the design of each induction was based on a different motivational construct; in other words, the theory of change underlying each induction differed. Second, the inductions differed in the expense and technical sophistication that were required for their creation and implementation, ranging from the very expensive-to-produce and technically advanced to the modest and inexpensive. Below we describe each induction in more depth.

Induction 1: Virtual Environment

At the core of Induction 1 was an Immersive Virtual Environment (IVE)—a game-based technology activity we designed to introduce students to the mathematical concepts that were to follow in a subsequent lesson. The IVE was professionally produced such that it was similar in look and feel to video games that students may have had experience playing.

For the story line of the IVE, students were provided with the opportunity to explore an outer space environment in the context of a space rescue mission. Various mathematical puzzles were encountered as students moved around the planet; all puzzles related to the generation of and identification of mathematical patterns, similar to what would subsequently be discussed in a mathematics lesson. The initial puzzle was designed to be relatively easy; in later stages of the experience, mathematically related, more complex puzzles were broken down into many smaller steps to scaffold students’ progress and to reduce the likelihood that students would be overly frustrated. Similarly, hints were also provided by the IVE for students who requested help in completing any of the puzzles.

Prior to beginning the IVE, each student viewed a short (5-min) video clip of a young STEM professional who talked about the nature of the work they do (e.g., designing astronaut space suits), the difficulties they had encountered in their K-12 math and science classes, and how they were able to overcome these difficulties. Students were provided with a selection of several of these videos, which varied according to the demographic attributes of the STEM professionals (e.g., gender, ethnicity).

Motivationally, Induction 1 was designed to primarily impact students’ self-efficacy. In particular, the IVE experience supported mastery experiences by allowing students to experience incrementally more difficult mathematical challenges, and by providing the scaffolds necessary for students to succeed when they were met with obstacles. Vicarious experiences were included in Induction 1 by including real-life, young, STEM professionals who discussed their jobs and the types of obstacles that they faced (and overcame) as they pursued a STEM career. Finally, emotional and physiological states were addressed by ensuring that students felt comfortable and relaxed about solving the mathematical challenges in the IVE. For example, we made the design decision not to include a timer that gently reminded students to work more quickly if they were taking too long because such a timer would likely cause a good deal of anxiety—a common experience for many students in mathematics.

Induction 2: Brainology® Web-Based Activity

For the second induction, we used a commercially available series of web-based modules designed to teach students about an incremental view of ability in a game-based manner. These modules are based on the work of Dweck and colleagues and have been shown to be successful at influencing students’ motivation and achievement (e.g., Blackwell et al., 2007). Students assigned to Induction 2 were given access to an abridged version of the Mindset Works® StudentKit—Brainology® program (www.mindsetworks.com) described above. The intervention that students experienced was relatively short compared to the entire Brainology® program, which contains over 2 h of online instruction and up to 10 h of additional activities to do over a recommended period of 5–16 weeks.

With respect to motivation, the Brainology® program is explicitly designed to impact students’ implicit theory of ability. As noted above, Dweck and her colleagues (Blackwell et al., 2007; Dweck & Leggett, 1988) have shown students possess particular “mindsets” that can influence their motivational and developmental trajectories through the course of school (e.g., fixed theory of ability vs. incremental theory of ability). The Brainology® program activities have been found to encourage students toward an incremental view of ability.

Induction 3: Video on Mathematical Patterns

Induction 3 was intended to provide an off-the-shelf experience for students related to some of the mathematical ideas that were to come in the mathematics lesson. We selected a commercially available PBS NOVA video on fractals because of its engaging story line and graphics, its focus on mathematical patterns, and the accessibility of the content to our target population of students in grades 5–8. The 2009 video, Fractals: Hunting the Hidden Dimension, is 56 min long and includes visually appealing animations, interviews with mathematicians, and accessible explanations of the mathematics of fractals and their applications to everyday life, such as building smartphone antennas and generating visual effects in movies. However, note that (to contrast with the other two inductions), viewing the video on mathematical patterns was intentionally intended not to be game-like.

In terms of motivation, movies have long been used by educators to motivate and engage students in the classroom. Although this movie did not specifically target a particular motivation construct, movies are often used in educational settings as an inexpensive, simple means that teachers can employ to help students see connections between what they are learning and real-world applications.

Mathematics Content Focus

Within the general landscape of STEM, we chose to situate the present study in the content area of algebra. Algebra is widely recognized as a crucial peg in the trajectory of mathematical learning because of the conceptual and procedural groundwork it lays for accessing higher mathematics and because it presents a shift in how students are expected to think mathematically (Kieran, 1992). Algebra is often the first time students are introduced to some of the most important and useful ideas in the field of mathematics, such as the concept of a “variable” or the generalization of patterns in generated data (Star & Rittle-Johnson, 2009). Within the larger landscape of algebra, we focus here on an aspect of algebra that many mathematics educators refer to as algebraic reasoning (e.g., Kaput, 1999), which includes using arithmetic for generalizing, working with patterns to describe functional relationships, and modeling as a way to formalizing generalizations.

Hypotheses

We hypothesized that Inductions 1 and 2 would have the strongest effect on the motivational constructs that they were designed to influence. In particular, we hypothesized that Induction 1 would have the strongest impact on students’ self-efficacy and that Induction 2 would have the strongest impact on students’ implicit theory of math ability. Because Induction 3 was not designed with a particular theory of motivation in mind, it did not intentionally target any particular motivation variable. However, because of the content in the movie, we hypothesized that this third induction would have an impact on students’ value beliefs. Finally, with respect to developmental issues in motivation, the literature is clear that there is a general decline in motivation as students progress through school (Archambault et al., 2010; Eccles et al., 1984). Because the structure of schooling for students in middle school (Grades 6–8) is different from that of elementary school students (Grade 5) and because students conceive of competence differently based on age (Dweck, 1986), we expected the first two inductions to have differential impacts on students depending on their age.

Method

Sample

Data come from all fifth, sixth, seventh, and eighth grade students and their teachers in the Chesterfield County Public School district in Virginia. A total of 18,628 students participated in the study, along with their 476 teachers, from 38 elementary and 12 middle schools.

A number of teachers in our original teacher pool were assistant, ESL, or special education teachers who did not have their own classroom. We removed these teachers from our sample, ending up with 339 teachers in our active teacher sample who participated in random assignment. In the elementary schools, the 163 fifth grade teachers, who taught all subjects to the same group of students each day, implemented the intervention with their homeroom students. In the middle schools, the 60 sixth, 57 seventh, and 59 eighth grade teachers were all math specialists and implemented the intervention in each mathematics classes that they taught.

We removed students who did not have parental consent to be a part of the study, which left us with 16,879 students. In addition, we had to exclude the 8979 students (and their 113 teachers from five schools) who were missing pretest or posttest data used in our analyses, as a result of a miscommunication between the research team and the district relating to the student identification numbers that students were instructed to use at pretest.Footnote 1 After removing those students with missing data, we report on the 7900 students and 226 teachers from 44 schools who remained in our analyses. These students were approximately equally divided across grade levels (see Table 1 for demographic information about the sample). We also collected students’ most recent scores on the state standardized test in mathematics, the Virginia Standards of Learning (VA-SOL) test; this test is given annually to students in grades 3–8.

Table 1 Student demographic information by condition
Full size table

Design and Procedure

We used a pretest/posttestFootnote 2 experimental design. Prior to the start of the intervention, students and teachers were administered a pretest. After pretest administration, teachers were randomly assigned to one of three inductions described above. Participation in the main part of the intervention occurred over a period of 4 consecutive days. On Day 1, students worked on the induction to which they were assigned. On Days 2 and 3, teachers taught the 2-day mathematics lesson. On Day 4, students again worked on the induction to which they were assigned.

For students in Induction 1, Day 1 of the intervention was spent in the school’s computer lab. Each student sat at his/her own computer, with headphones, and watched the short interview of a STEM professional and then played the IVE game for approximately 30 min. On Day 4, students returned to the computer lab and restarted the game-based technology activity, including watching a video of a STEM professional and restarting the IVE game from the beginning—again playing for about 30 min. Similarly, for students in Induction 2, Days 1 and 4 were spent in the school’s computer lab, with one student at each computer with headphones, playing the Brainology® program. Finally, Induction 3 students watched the first half of the Fractals: Hunting the Hidden Dimension video (about 28 min) on Day 1; on Day 4, these students watched the second half of the video.

Professional Development

All teachers were provided with a 1-day (6.5 h) professional development (PD) workshop, administered within 1 week of the start of the intervention. The PD workshop was designed and implemented by project staff. Most of the PD (approximately 4 h) was devoted to introducing teachers to the 2-day mathematics lesson. Teachers were provided with detailed lesson plans as well as visual aids, handouts, and manipulatives that accompanied the lesson. For the remainder of the PD, we provided teachers with induction-specific training.

Measures

All assessments were administered to teachers and students online, via a password-protected website.

Student motivational measures. All students were administered a pre- and postassessment, in a proctored computer lab in each school, during the regular school day. The pretest, taken between 1 and 3 weeks prior to the start of the intervention, targeted students’ motivation, with measures corresponding to the three motivational constructs that were related to the inductions—self-efficacy, implicit theories of ability, and value (see Table 2 for descriptive information on student variables; see Table 3 for sample items and alphas). The posttest was administered on Day 4, after the implementation was completed.

Table 2 Descriptive statistics on student motivation and learning variables
Full size table
Table 3 Motivational measures
Full size table

The motivational items on the posttest were identical to the pretest. We assessed self-efficacy students with a 13-item measure that was drawn from Bandura’s (2006). The degree to which students endorsed an incremental view of ability (as opposed to a fixed view of ability) was assessed using a 6-item instrument that was adapted from Dweck (1999). Finally, interest, attainment, and utility value beliefs concerning their mathematics class were assessed using scales taken from the Michigan Study on Adolescent Life Transitions (MSALT), which has been used extensively in the past (e.g., Eccles, Barber, Stone, & Hunt, 2003).

Student mathematics learning measure. Assessing students’ mathematics learning was not a major focus of the present study, mainly because of the absence of a priori hypotheses related to the differential impact of the three technology inductions on student learning and also the short duration of the math lesson. However, as a manipulative check, we included a short five-item assessment on mathematics learning on both the pre- and posttests. These five items were on algebraic reasoning as related to the 2-day mathematics lesson, specifically data organization, pattern identification, and the ability to make generalizations. The reliability of the math learning measure was low (α = 0.30 and 0.40 for the pre- and posttest); as a consequence, the results from this measure must be interpreted with caution.

Teacher measures. Teachers were given a pretest immediately prior to the start of the professional development workshop. The pretest collected background and demographic information about teachers, such as number of years teaching, undergraduate major, advanced degrees held, and national board certification status. In addition, the teacher pretest included items that tapped teachers’ own teaching self-efficacy for instruction and student engagement (22 items), technology use (7 items), and mathematics (12 items). Items were drawn or adapted from Bandura (2006). Teachers were also administered a 6-item measure of implicit theory of ability that was adapted from Dweck (1999). See Table 3 for sample items and alphas.

Data Analysis

Given that many students had the same teacher and many teachers were in the same school, we used multilevel modeling (Raudenbush & Bryk, 2002) to account for this nesting of students within teachers and teachers within schools. The first level of the model, the student level, included students’ prior knowledge (VA-SOL) scores, pretest math learning scores, pretest self-efficacy scores, pretest implicit theory of ability scores, pretest value scores, and demographic information, including ELL status, grade, gender (male coded as 1 and female coded as 0), and ethnicity.

The second level of the model, the teacher level, measured the effect of experimental condition, teachers’ self-efficacy for student engagement and instruction, teachers’ self-efficacy for technology use, teachers’ mathematics self-efficacy, and teachers’ implicit theory of math ability. We specified Induction 1 (the immersive virtual environment) as the referent condition to compare it to the other two inductions. This resulted in the effect of condition being captured by two variables. One variable indicated the difference between Induction 1 and Induction 2, and the other variable indicated the difference between Induction 1 and Induction 3. To test the difference between Inductions 2 and 3, a Wald test (similar to an incremental F test) was used to examine whether the parameter estimates for these conditions were significantly different from one another.

The third level of the model, the school level, measured the percentage of students receiving free or reduced lunch in each school. Finally, we also included two cross-level interactions to test for possible interactions between induction and grade, as well as two cross-level interactions to test for possible interactions between induction and prior math knowledge (VA-SOL). We ran these models to evaluate our four posttest student outcomes: math learning, self efficacy, implicit theory of ability, and value.

Results

We begin by overviewing students’ scores on the motivational variables at pretest and posttest and then reporting the effects of condition at posttest.

Student and Teacher Pretest Scores

To begin, we measured whether there were any differences between the inductions on our outcome measures at pretest and on demographic variables (see Table 2). When controlling for other independent variables in the model, there were no significant differences (p > .05) between inductions on any of the pretest or demographic variables, with the exception of prior knowledge (VA-SOL). Students in Induction 2 had lower prior knowledge than students in Induction 1, β = −15.76, p = .003, and Induction 3, χ 2(2) = 13.63, p = .001. Students in Induction 3 also had slightly lower prior knowledge than students in Induction 1, β = −15.69, p = .001. Prior knowledge was included in all subsequent models, so we controlled for these differences between conditions.

Pre-/Post Gains

Before examining the effects of condition, we first consider whether the intervention generally led to gains in students’ motivation (see Table 2). Overall, students did not have statistically significant gains on our measure of self-efficacy (M pre = 4.54, M post = 4.55, t = −1.16, p = .246, d = −0.01). For implicit theory of ability, students’ incremental view of math ability decreased after the intervention, although this was a small effect (M pre = 4.22, M post = 4.16, t = −6.93, p < .001, d = −0.07). For value, students’ scores generally decreased after the intervention as well, although the effect was again small (M pre = 4.24, M post = 4.19, t = −8.71, p < .001, d = −0.06). For math learning, the intervention led to an average gain on students’ scores on the five-item mathematics learning assessment of 10 % points, and this was a moderate effect (M pre  = 0.60, M post = 0.70, t = 28.60, p < .001, d = 0.40).

Effects of Condition at Posttest

At posttest, there were significant effects of condition on several of our outcome variables (see Table 4).

Table 4 Parameter estimates for student outcomes
Full size table

Math learning. Comparing Inductions 1 and 2, students in Induction 2 earned similar math learning scores to students in Induction 1, β = 0.003, p = .872. There was also no significant interaction between Induction 2 and grade, β = 0.01, p = .129. Comparing Inductions 1 and 3, students in Induction 3 had similar math learning scores to students in Induction 1, β = −0.01, p = .409. However, there was a significant interaction between Induction 3 and grade. In particular, students in lower grades benefited more from Induction 1 than from Induction 3. Then as grade increased, Induction 3 became more effective, β = 0.02, p = .013. Thus, for students in grade 5, being in Induction 1 led to higher scores on average. For students in grades 6, 7, and 8, being in Induction 3 led to higher scores on average. Finally, post hoc Wald tests comparing Inductions 2 and 3 suggested that there were no significant differences between Inductions 2 and 3 (χ 2(2) = 1.06, p = .589); however, there was a significant interaction when considering grade (χ 2(2) = 6.22, p = .045). Essentially, Induction 2 was more effective for lower grades, and as grade increased, Induction 3 became more effective. There were no significant interactions between induction and prior knowledge (VA-SOL) (p’s > .532).

Self-efficacy. There were no significant differences between any of the inductions on the student self-efficacy variable, nor were there any significant interactions between inductions and grade or inductions and prior knowledge (p’s > .128).

Implicit theory of ability. Comparing Inductions 1 and 2, students in Induction 2 had higher implicit view of math ability scores than students in Induction 1, β = 0.09, p = .039, meaning that being in Induction 2 led to an implicit theory of math ability score that was 0.09 standard deviations higher than being in Induction 1. There was also a significant interaction between Induction 2 and grade. In particular, students in lower grades had similar implicit view of math ability scores in Induction 2 and Induction 1. Then as grade increased, Induction 2 led to higher implicit view of math ability scores than Induction 1, β = 0.12, p < .001. In addition, there was a significant interaction between Induction 2 and prior knowledge (VA-SOL), β = 0.001, p = .018; however, as the coefficient indicates, this was a very small interaction. Students with lower prior knowledge had slightly higher implicit view of math ability scores in Induction 1 than Induction 2. Comparing Inductions 1 and 3, students in Induction 3 had similar scores to students in Induction 1, β = 0.05, p = .243. There was also not a significant interaction between Induction 3 and grade, β = 0.03, p = .271, nor between Induction 3 and prior knowledge (VA-SOL), β < 0.001, p = .371. A post hoc Wald test indicated that overall students in Induction 3 had similar implicit theory of ability scores to those in Induction 2 (χ 2(2) = 4.34, p = .114). However, there was a significant interaction when considering grade (χ 2(2) = 23.62, p < .001). In lower grades, students in Induction 3 had similar implicit view of math ability scores as students in Induction 2, but as grade increased, students in Induction 2 tended to have higher scores than students in Induction 3. When comparing Inductions 2 and 3, there was also a marginally significant interaction between Induction and prior knowledge (VA-SOL) (χ 2(2) = 5.75, p = .057).

Value. For value, in comparing Inductions 1 and 2, overall students in Induction 2 had similar value scores to students in Induction 1, β = 0.02, p = .668. There was also no significant interaction between Induction 2 and grade, β = −0.01, p = .520. When comparing Inductions 1 and 3, students in Induction 3 had similar value scores to students in Induction 1, β = 0.01, p = .795. There was a significant interaction between Induction 3 and grade. In particular, students in lower grades had similar value scores in Induction 3 and Induction 1. Then as grade increased, Induction 1 led to higher value scores, β = −0.04, p = .036. Post hoc Wald tests suggested that there was no significant difference between Inductions 2 and 3 (χ 2(2) = 0.19, p = .910). There was also no significant interaction when considering grade (χ 2(2) = 4.76, p = .093). Finally, there were no significant interactions between condition and prior knowledge (VA-SOL) (p’s > .069).

Discussion

Perhaps not surprisingly given the size and complexity of the present study, our results are informative, modest, and not definitive.

RQ1: Impact on Students’ Motivation

Our first research question concerned the general impact of the 4-day intervention on students’ motivation in mathematics, particularly self-efficacy, implicit theory of ability, and value. Overall, results from the 4-day intervention were mixed. No gains were found in self-efficacy; for implicit theory of ability, a lower incremental view of ability was found; we found modest declines in value beliefs. With respect to math learning, students in all three inductions had modest improvements in their scores on the math learning measure.

RQ2: Influences of Induction Type and Student Characteristics

Second, we were interested in whether the impact of the intervention was influenced by the type of induction that student received and other student-level demographic or academic characteristics. No effects related to self-efficacy were found, and effects related to value were very minor. For implicit theory of ability, there were indications that Induction 2 was more successful than Inductions 1 and 3 in impacting students’ views, especially for older students. Induction 2 led to higher incremental views of math ability for students, particularly for students in grades 7 and 8. Induction type also appeared to have a small impact on value, with some evidence that Induction 3 had the strongest impact on utility and attainment value for the younger students, as compared to the other two inductions.

Despite the complexity of these results for our second research question, three clear patterns did emerge.

Absence of effects on self-efficacy. First, Induction 1 did not have the hypothesized impact on students’ self-efficacy. Despite the fact that the IVE was designed specifically to foster changes in self-efficacy, there is no evidence that Induction 1 improved self-efficacy any more than the other inductions. There are several possible explanations for this finding. First, given the relatively short intervention, the fact that students in any induction did not experience dramatic gains in a construct as fundamental and multidimensional as self-efficacy is not surprising. Second, Induction 1 was the most complex in terms of cognitive and temporal “overhead” required for students to enact the experience. We hypothesize that, had a longer time period been available for students to shift their focus from learning to enact Induction 1 to reflecting on the content of the experience, effects on self-efficacy would have been greater.

Recall that the three inductions also differed on the expense and technical sophistication required to create and implement them. Does the present finding about Induction 1 and self-efficacy suggest that use of virtual worlds is not worth the trouble and expense? Particularly when inculcating sophisticated knowledge and skills, a substantial body of research suggests that this is not the case (National Research Council, 2011; U.S. Department of Education, 2010). We interpret our results as indicating that this type of complex game-based technology activity with high cognitive overhead may require more instructional “dosage” than short duration provided in the present intervention. Thus, well-designed virtual worlds, which are expensive and technically demanding, can realize their power for engagement and learning only when a sufficient investment of classroom time is made.

Effects linked to students’ age. A second pattern that emerges from the complex results of our second research question is that the effects of each induction on students’ motivation were influenced by students’ age, as evidenced by the frequency of significant induction type by grade interactions. These grade-level interactions held while controlling for prior mathematics knowledge (VA-SOL scores), indicating that the differential impact of the inductions was developmental and not merely the result of differing mathematics ability. Because the structure of schooling for students in middle school (Grades 6–8) is different from that of elementary school students (Grade 5), and because students conceive of competence differently based on age (Dweck, 1986), these findings indicating differential impacts on students depending on their age are confirmatory of prior work and reinforce the importance for practitioners and policy makers of tailoring such interventions to students’ developmental level.

Absence of effects for student demographics. Finally, we did not find interactions between induction type and other student demographic variables such as free and reduced lunch, ethnicity, and gender. From a curricular perspective, this is a positive outcome indicating that, in contrast to many educational experiences, these types of intervention may narrow—not widen—troubling achievement gaps. That good design can produce motivational learning experiences effective across the full spectrum of students is very encouraging.

RQ3: Influences of Teacher-Level Factors

Our third research question asked about impact of teacher-level factors on students’ motivation, including credentialing in mathematics education, undergraduate major, years of experience, and teachers’ beliefs. Based on the extant literature, we had hypothesized that these factors might influence students’ motivation. However, teacher-level factors were not significant predictors of student outcomes. Viewing the intervention from a curricular perspective, this is a positive finding suggesting that our design and implementation ensured that all students received a roughly equivalent instructional experience.

With respect to the absence of a relationship between teachers’ beliefs and student motivation, although there is good theoretical and empirical evidence to suggest that these variables could predict student outcomes, it is also true that linking teacher-level beliefs to student outcomes is not a clear and straight path (Holzberger, Philipp, & Kunter, 2013; Klassen, Tze, Betts, & Gordon, 2011). In fact, Klassen et al. (2011) noted that there is a lack of evidence that links teachers’ self-efficacy to student outcomes, despite the commonly held belief by researchers that this relationship exists. Their review of the literature noted that correlations between teachers’ self-efficacy and student achievement were low to modest. Our findings confirm this perspective.

Limitations

There were several limitations to the present study that suggest caution in the interpretation of our results. First and foremost, as noted above, there was a very large amount of missing data—53 % of students were missing demographic, pre-, and/or posttest data. Second, it is important to note that the length of the intervention was relatively short, both in terms of the game-based technology activities, the professional development, and the mathematics lesson. Although we were able to find some influence of the intervention on students’ motivation, these effects were quite modest. Further, although a delayed posttest was administered, results were not interpretable; thus, we are not able to report whether or not the effects at posttest were sustained after the end of the intervention. Third, recall that the five-item math assessment had low reliability. Taken together, all of these results raise questions about any attempt to generalize our findings. Future studies—both additional large-scale studies of longer duration, as well as shorter-term studies that afford opportunities for more qualitative exploration—can attempt to address these limitations and continuing moving toward improving our understanding of the relationship between technology, motivation, and STEM learning.

Conclusion

Investigating along a developmental span the relationship between game-based technology activities and student interest in STEM careers is important because much potential talent in STEM is now lost. Our research interweaved alternative motivational activities with effective and authentic mathematics learning, in order to take initial steps toward developing insights about the added value of game-based technology activities for building confidence in math and science capability, seeing one’s abilities in STEM as able to improve over time, and developing a passion or sustained interest in becoming a scientist or engineer. Further, we studied the impacts of media with substantially different production costs, providing the basis for a cost-benefit analysis and for articulating contrasting conditions for success.

Our findings highlight the importance of tailoring motivational experiences to students’ developmental level. Our results are also encouraging about developers’ ability to create instructional interventions and professional development that can be effective when experienced by a wide range of students and teachers. Further research is needed to determine the degree, duration of, and type of instructional intervention necessary to substantially impact multidimensional, deep-rooted motivational constructs, such as self-efficacy.