Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Self-regulated learning (SRL) is a complex process in which learners have been described to “personally activate and sustain cognitions, affects, and behaviors that are systematically oriented toward the attainment of personal goals” (Zimmerman & Schunk, 2011, p. 1). Multiple theories of SRL (Pintrich, 2000; Winne & Hadwin, 1998; Winne, 2011; Zimmerman, 2000, 2011) describe this process and each ­acknowledges that motivational constructs play an influential role. The methodologies used to research SRL have primarily focused on behaviors that can be related to cognitive and metacognitive processes such as think aloud protocols in which learners verbalize their thoughts (Azevedo, Moos, Johnson, & Chauncey, 2010; Ericsson & Simon, 1984; Greene, Robertson, & Costa, 2011) and log analyses in which the educational software logs learners’ interactions with the system (Aleven, Roll, McLaren, & Koedinger, 2010). However, little work has used these methods to assess motivation because less is known about how such behaviors relate to various motivational constructs.

Prior research that has examined learners’ motivational states has primarily relied on construct-, context-, and task-specific questionnaires administered before and after learning activities or classes. This method has yielded interesting results, but is problematic for two reasons. First, recent research suggests that pre-/post-assessment may be insufficient to accurately capture the dynamics of various motivational constructs that vary over the course of a learning activity or unit (e.g., achievement goals; Fyer & Elliot, 2007; Muis & Edwards, 2009). Second, a critical component of SRL theory is that the underlying processes are interactive suggesting that motivational states, like cognitive and metacognitive processes, should interact with each other in real time to affect learning outcomes. Measuring a motivational state prior to and separate from this process eliminates the opportunity to observe both dynamic changes in the construct and its influence within the SRL process. As a result, the use of pre-/post-measurement can lead one to draw conclusions about the role of motivation in SRL that are, at best, insufficient to capture the dynamic complexity of SRL and, at worst, inaccurate.

In this chapter, we describe a project in which we take the first step towards developing fine-grained assessments of motivational constructs in an SRL context. We pose questions to students at varying points during learning to capture self-reports of their motivational state with respect to the domain and the unit or problem they have just completed. Our focus is on motivational constructs that are hypothesized to vary much more rapidly on the order of minutes to hours (e.g., self-efficacy for specific math problems; Pajares, 1997), although we also acknowledge that learners likely have some stable motivational characteristics such as domain-level achievement goals that change relatively slowly over the course of months to years (Ames, 1992). For this reason, we investigate motivation at multiple grain sizes, examining the variability of a learner’s motivational state when construed with respect to both the domain and at finer-grained levels such as the unit or problem. By repeatedly evaluating one’s motivational state, we can observe variation or stability of specific constructs (e.g., self-efficacy), examine the task variables (e.g., unit or problem difficulty) that might affect such a state, and identify the associated learning behaviors. Our theoretical approach is analogous to Mischel (1968, 1973), Cervone (Cervone & Shoda, 1999) and others who, in questioning the stability of personality constructs, conducted productive programs of research examining the dynamics of personality and developed a deeper understanding of those constructs; one’s personality is both coherent across situations and influenced by situational factors. The methodology we describe, especially when combined with online traces of behavior, can enrich our understanding of SRL as well as improve our ability to predict (and promote desirable) learning outcomes.

While our approach is not an online method like the log-files, eye tracking, or verbal protocols that continuously collect data, our repeated sampling of self-reports are considerably finer than traditional methods that only measure motivation at pre- or posttest. So, while our data do not provide a continuous measure of learners’ motivational states, the frequency of our sampling (as often as every 1–2 min) does provide a series of snapshots of motivational variables that can be used to examine the relationship between motivation, learning behaviors, and learning outcomes, all of which are specific to a particular learning context. Furthermore, these finer-grained snapshots can be used more productively than simple pre-/posttest assessments when relating motivational data to other streams of trace data such as those mentioned above.

In this chapter, we first describe the Cognitive Tutor, the intelligent tutoring system with which we conduct our research, and then summarize two theoretical frameworks that describe the SRL process, paying special attention to the motivational components included in each. We next describe the methods others have used to capture traces of SRL and the questionnaire methods typically used to assess motivation. In light of this review, we argue that motivation needs to be assessed using finer-grained methods that are more sensitive to the changes in motivation that occur during learning. The remainder of the chapter describes a microgenetic approach to assess motivation in SRL and illustrate the benefits and challenges of this approach with both hypothetical examples and empirical data.

First, however, we describe our interest in motivation as it relates to metacognition. We agree with the perspective of Veenman (Veenman, Bernadette, Hout-Wolters, & Afflerbach, 2006) who suggests that metacognition cannot be studied in “splendid isolation” (p. 10). In the inaugural issue of Metacognition and Learning, Veenman states that “we need to know more about how individual differences and contextual factors interact with metacognition and its components” (p. 4). Motivational constructs can operate as individual difference variables or can be influenced by contextual factors and should be examined concurrently with metacognitive processes as components of the dynamic models SRL theorists propose.

While there are dozens of motivational constructs that we might examine, we focus our work on achievement goals and self-efficacy for three reasons. First, each of these factors is explicitly referenced in one or more of the central theories of SRL (see Fig. 41.1). Changes in these factors are theorized to influence metacognitive processes. Second, these constructs have been associated with particular patterns of learning behavior in empirical studies, some of which are metacognitive in nature. Third, prior research has illustrated that learners’ level of self-efficacy (Pajares, 1997) and achievement goals (Muis & Edwards, 2009) change during learning. In order to observe these changes and to investigate their influence on learners’ metacognitive monitoring and acts of metacognitive control, we propose an approach that supports analysis of fine-grained behavioral data. In order to establish the context in which we conduct our research, we present an intelligent tutoring system that traces learners’ behaviors, the Cognitive Tutor, which we use to assess students’ motivational state while learning mathematics.

Fig. 41.1
figure 00411

Motivational constructs embedded in process models of self-regulated learning

Full size image

The Cognitive Tutor

Cognitive tutors are a family of intelligent tutoring systems (c.f. Koedinger & Aleven, 2007) that combine the disciplines of cognitive psychology and artificial intelligence to construct computational cognitive models of learners’ knowledge (Koedinger & Corbett, 2006). Cognitive tutors are unique in that they monitor student’s performance and learning by model tracing and knowledge tracing. That is, the cognitive tutor runs step-by-step through a hypothetical cognitive model (which represents the current state of the learner’s knowledge) as the learner progresses through a unit. This allows the tutor to provide real-time feedback and context-specific advice. Learning in the tutor is defined as the acquisition of knowledge components, which are the mental structures that learners use, alone or in combination with other knowledge components, to accomplish steps in a problem.

The cognitive tutor combines a series of structured learning tasks (i.e., math problems) along with opportunities for self-regulation of learning. In most cognitive tutor environments, learners are given access to a unit which includes an introductory text and a problem set, as well as tools they can choose to access to support their learning. These include a hint button that provides context-specific hints, a glossary of terms relevant to the content and, at times, a worked example of a problem similar to those they are to complete. Students can also assess their own progress towards mastery (as assessed by the tutor) by clicking on the skillometer, a menu that presents skill bars indicative of the progress towards mastery for each skill in the unit. Bars are green in color and increase when steps are completed accurately; they turn gold when mastery is met.

While the tutor chooses the problems, the learner chooses how long to spend on the introductory reading, when to begin the problem set, as well as whether or not to request hints, access the glossary, check the skill bars, review the introductory text or view worked examples and in general, how deliberately to approach to tutor problem (versus superficial processing or guessing strategies). The inclusion of these resources creates the opportunity for the learner to self-regulate learning. For example, a learner, who while completing a problem encounters an unfamiliar term, can access a glossary to obtain a definition. Another learner who begins a problem set and does not understand a step can request a hint that provides directions on that step. Students’ metacognitive monitoring is supported by the provision of the skillometer, as well as feedback about the accuracy of answers submitted per step. When tutor feedback indicates that a step is incorrect, a student might try to self-explain why that step is incorrect. After reading the hints, the student could try to reconstruct for himself/herself the line of reasoning presented in the hint (typically, a principle-based explanation of what to do next and how, and perhaps why). Access to the worked examples, glossary and introductory text also provide opportunities for metacognitive monitoring; learners can click on these features to make metacognitive judgments about their understanding of the concepts or mastery of a skill. This metacognitive monitoring may be aided by tutor feedback. For instance, the decision to ask for a hint may be based on self-assessment of whether a step is familiar (Aleven et al., 2010).

In addition to providing instruction and opportunities for self-regulation, the cognitive tutor collects fine-grained behavioral data of students’ interactions with the tutor. This data is logged at the transaction level, whenever a learner attempts a step in a tutor problem, requests a hint, accesses a glossary item, etc. The tutor records this data as log-files that serve as a database for conducting microgenetic analyses of learning, using an open repository called DataShop (Koedinger, Baker, Cunningham, Skogsholm, Leber, & Stamper, 2010). Microgenetic approaches (c.f. Siegler & Crowley, 1991) involve the logging of frequent observations of individuals’ behavior and allow for examination of change at a fine-grained level (e.g., eye-tracking, verbal protocols, log-files, etc.).

Analyses of transaction data have made it possible for researchers to identify when learners seek help in tutoring environments (Aleven & Koedinger, 2000) and when they abuse help features (Baker, Corbett, Koedinger, & Wagner, 2004). As a result of these investigations, researchers have attempted to scaffold help seeking by modifying the cognitive tutor design and creating a Help Tutor (Aleven, McLaren, Roll, & Koedinger, 2006). Transaction level data also allows for examination of behaviors as they relate to specific learning outcomes (e.g., Ritter, Anderson, Koedinger, & Corbett, 2007; Koedinger et al., 2010). In their present form, cognitive tutors provide a useful environment for studying facets of students’ SRL behaviors like help seeking.

Building on the basic functionality of the Cognitive Tutor, we have implemented an additional component that, when added to the tutor, allows for the consideration of motivational constructs as they affect learning. Before we outline how this questionnaire component is integrated to capture self-reports of learner motivation, we first define the motivational constructs on which we focus, learners’ achievement goals and perceived self-efficacy for mathematics, and highlight their role in SRL theories.

Motivational Factors: Achievement Goals and Self-Efficacy

Achievement Goal Orientation

Elliot (Elliot & McGregor, 2001; Elliot & Murayama, 2008) posits a 2  ×  2 framework describing one’s achievement goals in terms of definition (mastery vs. performance) and valence (approach vs. avoidance). Those with mastery approach goals engage in a task with the purpose of developing competence and define success with respect to intrapersonal standards of improvement over previous levels of competence, or as focused on meeting a self-imposed criterion of task-mastery (Ames, 1992; Elliot, 1999). Performance approach oriented learners define success interpersonally by measuring competence normatively against the competence of peers and aim to demonstrate their competence by outperforming peers. Mastery avoidance goals denote an orientation towards avoiding failure as defined by “avoiding self-referential or task-referential incompetence” (Elliot, 1999, p. 181). A performance avoidance oriented learner engages in a task to demonstrate that they are not any less competent than their peers.

Research on achievement goal theory has shown that individuals’ goal orientations are related to learning behaviors and performances. Mastery-oriented individuals employ effective problem-solving practices (Elliott & Dweck, 1988), are more likely to expend effort, persist in the face of failure, and engage in deep processing (Elliot, McGregor, & Gable, 1999). Performance approach goals have also been positively related to effort (Harackiewicz, Barron, Pintrich, Elliot, & Thrash, 2002) but their processing tends to be more superficial (Elliot et al., 1999). Research has shown that performance avoidance goals are positive predictors of surface processing and negative predictors of deep processing (Elliot et al., 1999), and performance avoidant learners demonstrate disorganization and low interest (Elliot & Harackiewicz, 1996; Elliot & Church, 1997). One’s goal orientation has also been associated with employment of SRL processes. Research has shown that mastery approach goals predict increased cognitive engagement and performance (Greene & Miller, 1996; Greene, Miller, Crowson, Duke, & Akey, 2004) and that performance goals have been found to predict study strategies (Archer, 1994) and metacognitive strategy use (Bouffard et al., 1995; Meece, Blumenfeld, & Hoyle, 1988).

With respect to performance, both mastery and performance approach goals have been found to relate positively to achievement (Linnenbrink-Garcia, Tyson, & Patall, 2008) and students who pursue mastery goals show evidence of transferring past learning experiences to new tasks (Belenky & Nokes, 2012). Performance avoidance goals have consistently been shown to predict poor performance (Elliot & Church, 1997; Elliot et al., 1999; Harackiewicz et al., 2002).

In sum, we are interested in assessing achievement goals with fine-grained measures and examining their relations to behaviors in the tutoring system. We do so in order to further explore how achievement goals are influenced by task context (as theorized by Ames, 1992 and demonstrated by Horvath, Herleman, & McKie, 2006), how these changes in goals might alter the behaviors learners employ, and whether these context-specific factors might explain some of the conflicting results summarized by Linnenbrink-Garcia et al. (2008) who found that mastery and performance goals are only predictive of achievement in some cases.

Self-Efficacy

Bandura (1994) defined perceived self-efficacy as the belief about one’s ability to perform at a particular level on a task. Self-efficacy is theorized to influence cognitive, metacognitive, motivational, and affective processes. Learners with high levels of self-efficacy are willing to engage in difficult tasks, set challenging goals, and maintain strong commitments to achieving their goals. High self-efficacy is theorized to support effort regulation and to influence one’s attribution of failure (Bandura, 1991; Weiner, 1986). When individuals do fail to achieve, those high in self-efficacy are more likely to attribute their failure to insufficient effort, knowledge or skills and reengage to correct this insufficiency. In addition to influencing attribution to self or environmental factors, self-efficacy influences persistence and performance in learning tasks (Bandura, 1997). This association suggests that simultaneous examination of learners’ efficacy and learning behaviors might be an important methodological approach to further our understanding of the influence self-efficacy has on other components of SRL.

Role of Motivation in Theories of Self-Regulated Learning

We describe our method in relation to the two most prominent theories of SRL, both of which depict SRL as a cyclical process involving cognitive, metacognitive and motivational components. We summarize each below and draw particular attention to Zimmerman’s (2011) recent focus on motivational processes as they occur at each phase of the SRL process.

Winne and Hadwin’s COPES Model

Winne and Hadwin (1998; Winne, 2011) offer a description of SRL as an event-based phenomenon that occurs in weakly sequenced phases. Learners, when self-regulating their learning (1) define the task, (2) set goals they would like to attain and develop a plan for their attainment, (3) enact tactics, and (4) monitor their progress towards goals against a preconceived set of internal standards. Within each phase, self-regulatory behaviors are governed by both cognitive and situative factors in which learners generate behaviors that are evaluated in light of their self-imposed standards. In this framework, motivation governs SRL processes beginning with the assessment of task conditions. We illustrate this relationship (see Fig. 41.1) using achievement goals and self-efficacy as examples, given their prominence in models of SRL.

The conditions that affect how students engage in a learning task include environmental conditions (e.g., time limits, environmental affordances) and learner characteristics such as cognitive and metacognitive capacities like prior knowledge, domain knowledge, metacognitive knowledge of tactics that could be employed, and motivational conditions including interest and goal orientation (see Fig. 41.1). These factors influence the type of goals learners set, the tactics they enact, and the standards by which they judge their learning and performance. Winne and Hadwin (1998) provide the following example:

For example, students with a performance motivational orientation that view tasks as just jobs to complete may judge that the goal they understood their teacher set is at too high a level or requires too much effort. Therefore, they adjust or alter standards for summarizing the science chapter to levels where ‘just getting by’ is adequate. In light of this re-framed goal, the student now builds a plan to approach it. This student will probably plan simplistic tactics, such as paraphrasing headings and monitoring surface features of typography to insure that every bold phrase and every scientist’s name (standards) is reproduced in the finished product (p. 281)

Similarly, one’s level of self-efficacy influences the goal setting and evaluation processes. Efficacious learners are theorized to have greater expectations for what they can achieve in a task and set goals accordingly (Bandura, 1991). Their level of efficacy for carrying out the plan to achieve the goal can influence their persistence, and recurring efficacy judgments will influence their strategies during learning. This process has implications for future cycles through learning phases. The evaluation of learning leads to adaptations of learning processes. These adaptations are based on one’s sensitivity to feedback, which can be internally or externally generated, and which is interpreted in the light of one’s goals. One’s level of self-efficacy affects the way such feedback is interpreted. Negative feedback can be a useful tool for a highly efficacious learner who uses the feedback as a cue that his or her performance is insufficient and continued or greater effort is required, while a learner with low self-efficacy may interpret negative feedback as indicative of deficits that cannot be overcome, and lead to disengagement or frustration.

Zimmerman’s Social Cognitive Model

Zimmerman (2000) defines self-regulation as referring to “self-generated thoughts, feelings and actions that are planned and cyclically adapted to the attainment of personal goals” (p. 14). Individuals are theorized to engage in planning (i.e., forethought), volitional control, and self-reflection (see Fig. 41.1). This process occurs within a larger self-regulatory context in which learners regulate their behaviors, adjust performance processes, and adapt to their environment by managing the environmental factors that might inhibit goal attainment. By monitoring the success of their strategies and using feedback about potential barriers to goal attainment, learners can adapt to changing environments and regulate processes en route to attaining their goals.

Focusing on the cycle described in Zimmerman’s framework, individuals first plan in which they analyze the task in order to identify the desired goal and develop a strategy to obtain this goal. This plan is then evaluated for its potential success, which Zimmerman (2000) describes as being mediated by one’s self-motivational beliefs, including “self-efficacy and goal orientation” (p. 17). In the forethought stage, self-regulation can break down if an individual cannot clearly determine a goal, or cannot develop a strategy for reaching it. It can also stagnate if the individual cannot motivate himself or herself to seek such a goal or carry out the selected strategy. Once a goal has been identified and the individual intends to carry out a strategy to attain the goal, the individual acts. This stage is referred to as the performance or volitional control phase. Here, individuals critique their own strategy use in an attempt to maximize the efficiency of their efforts while carrying out a chosen strategy. After having completed an action and monitored the process and outcome, an individual engages in self-reflection by evaluating the performance and attributing the success or failure of the performance to causal factors.

In a more recent conceptualization of his sociocultural model, Zimmerman (2011) provides an elaborated description of motivation as catalyst at each SRL stage. During forethought, a learner’s goal orientation dictates a goal to increase his competence, which may involve greater persistence in a difficult task, or a goal to perform well, which may involve avoiding challenges. Additionally, his perceived self-efficacy for a task will dictate the strategies he chooses to employ. In the performance phase, self-efficacy beliefs motivate his time management and self-monitoring practices (Bandura, 1997).

Zimmerman (2011) underscores the importance of assessing motivation not as “person measures” (p. 60) at pre- and posttest due to subjects’ inaccurate recall and poor calibration, and praises event-based measurement of cognitive and metacognitive processes by way of trace and think-aloud methodologies. We share Zimmerman’s views that a learner’s motivation is an active and dynamic part of the self-regulatory process and that data collected during learning is necessary to capture the “dynamic interactive relations among these variables during successive SRL cycles” (p. 60). We next summarize the methods that have been used to capture evidence of cognitive, metacognitive, and motivational components of SRL, then describe our method for administering prompts to elicit self-reports of learners’ motivational state during the learning tasks in which metacognitive and cognitive processes are traced.

Measurement of Self-Regulated Learning with Learning Technologies

Increasingly, research conducted in technology enhanced learning environments reflects the process view of SRL espoused by the most prominent theorists. Data is collected online and the analysis that ensues is conducted under the assumption that the learning process is iterative and learners’ actions are dependent upon learning that has taken place earlier in the task or during prior learning tasks. At present, however, learning technologies that capture SRL data conduct no online measurement of motivational constructs. Instead, motivational constructs tend to be assessed before or after the learning task.

With only two data points, this method can only detect linear change. For example, when measuring learners’ self-efficacy before and after the task, we cannot determine the point at which a learner’s self-efficacy began to change or how it changed (linear, stepwise, etc.) over the course of a learning task. We can only measure whether it rose, fell or stayed the same from pre- to posttest. This limits our understanding of self-efficacy to coarse-grained associative relationships with learning behaviors. In contrast, if learners respond to efficacy prompts repeatedly throughout a unit, we can examine fine-grained changes from one data point to the next, concurrent with changes in behavior and identify patterns in log-files where reports of efficacy trend higher or lower, or when they follow an initiation of a behavior or a change in performance. Next we summarize measures and procedures typically employed to assess achievement goals, as well as recent evidence outlining elements of stability and change in achievement goals. This evidence demonstrates a need to employ more frequent, fine-grained assessment than is typical.

Assessment of Motivational Constructs

Instruments and Methods

When researchers aim to assess achievement goals, they employ questionnaires that include items that gauge the learner’s endorsement of performance approach, performance avoidance, mastery approach, and mastery avoidance goals. The two most common questionnaires employed are the Achievement Goals Questionnaire (Elliot & McGregor, 2001 and a revised version, the AGQ-R; Elliot & Murayama, 2008) and the Patterns of Adaptive Learning Scale (PALS; Midgley et al., 2000). Each are composed of a series of items that pose a statement meant to reflect a specific achievement goal. For instance, an AGQ-R item reflecting a performance approach orientation reads, “My goal is to perform better than the other students.” A PALS item reflecting the same orientation reads, “It’s important to me that I look smart compared to others in my class.” Respondents select a number from a Likert scale reflecting their level of agreement with the statement and mean scores per achievement goal are derived.

These questionnaires tend to be given once prior to or after the learning task. Recent studies that have administered achievement goal questionnaires repeatedly have reported both stability, but also some change over time and with respect to task conditions. Fryer and Elliot (2007) found rank-order stability across achievement goals and that mean levels of performance approach goals were stable across three time points (reported in conjunction with exams) while mastery approach increased and performance avoidance decreased over time. Examining self-reported achievement goals for two exams and two writing assignments, Muis and Edwards (2009) found a similar pattern of results and describe the extent the changes that occurred as moderate to large. These findings conform to theories of achievement goals (Ames, 1992; Dweck, 1986; Fryer & Elliot, 2007) that suggest an individual’s achievement goal orientation is consistent to the extent that it reflects the cognitive framework the individual uses to guide behavior. At the same time, learners are theorized to set goals in light of task conditions (Pintrich, 2000; Winne & Hadwin, 1998). Because task conditions are often outside the scope of learners’ control (i.e., the content of a task is prescribed) and because they tend to change over the course of a task when learning occurs in the context of adaptive learning technologies (i.e., tutors concentrate problems requiring yet-to-be-mastered skills), learners’ task-specific goals can differ from their typical goal orientation. Muis and Edwards’ (2009) finding that endorsement of achievement goals varied from exam to exam and differed between assessment types suggests that variation in the content of a task may influence individuals’ adoption of achievement goals. A number of research studies have assigned participants to conditions in which task conditions have successfully elicited achievement goals (c.f. Linnenbrink-Garica et al., 2008, Table 2), which demonstrates the extent to which task conditions can influence achievement goals. Because learners’ achievement goals have been shown to be contingent upon task conditions, repeated measurement is necessary to understand how task conditions might influence one’s task-level achievement goals and the behaviors they motivate.

Achievement goals are not unique among motivational constructs in their capacity for change during the course of learning. Perceived self-efficacy has been shown to build upon prior efficacy judgments (Bandura, 1997) and, during the course of learning, self-efficacy judgments are adjusted in light of actual performance and feedback. Learners’ self-efficacy is theorized to influence the goals learners set, the tactics they enact and the attributions they make about feedback when judging their progress towards goals. Exploration of this dynamic relationship between motivational state and metacognitive process requires fine-grained assessment of both constructs.

Factors like achievement goals and self-efficacy represent the motivational dimension of learning that Winne and Hadwin (1998) and Zimmerman (2000) identified as germane to SRL and as influential over metacognitive processes. However, methods to capture fine-grained evidence of these and other motivational constructs have not been incorporated into educational software prior to our study. We next present our methodological approach to address this situation, followed by some preliminary results.

Fine-Grained Sampling: A Microgenetic Approach to Assessing Motivation in SRL

We have added a component to the Cognitive Tutor that collects fine-grained motivational data to concurrently examine the dynamic and interactive metacognitive and motivational factors that influence learning. Using the items from questionnaires traditionally used to assess motivation pre- or posttest, we embed single items as prompts after problems and small, task-specific questionnaires after units to capture more fine-grained changes in motivational states (Fig. 41.2). We employ these prompts at multiple grain sizes and repeatedly over time in order to develop a rich understanding of how factors such as learners’ goal orientation and level of self-efficacy affect SRL in specific contexts and at specific points during the use of the tutor. The following section serves as an overview of our first year-long investigation in which this microgenetic and longitudinal approach is employed.

Fig. 41.2
figure 00412

Microgenetic approach to assessment of motivational constructs in the cognitive tutor

Full size image

A Microgenetic and Longitudinal Approach to Questionnaire Use

We collected automated self-report (questionnaire) data in multiple classrooms of students via cognitive tutors for a range of variables. These students use the cognitive tutor software across the whole school year as part of their regular mathematics instruction. This effort has two components. First, we take a microgenetic approach to collect questionnaire data with a small number of prompts that are administered frequently (i.e., dense data collection over a range of time periods, providing motivational tracking from minutes to hours to weeks). These prompts are embedded in the learning software and therefore can be administered at the end of a unit and between problems. At these finer-grained levels, a small, specific set of constructs are sampled in order to limit the proportion of time students spend completing measures when engaging with the tutor. This method of data collection is applied to motivational variables that are expected to vary over the course of a semester or unit). Second, two or three times a year, we administer questionnaires focused on constructs that are theorized to be stable over time (i.e., these include domain-level achievement orientation, domain-level self-efficacy, and theory of intelligence; Dweck, 1999).

Key to the current approach is that this traditional pre-/post-data can be related to the more fine-grained prompts as well as traces of the behaviors in the tutor log data. Concurrent collection of these multiple streams of data allow for testing of theoretical assumptions that would not be testable using traditional methods of measurement. Additionally, students use the tutor for the duration of the school year (and often multiple years), making this platform uniquely suited for longitudinal data collection and evolution. In the next section, we expand on the benefits associated with employing a microgenetic approach including opportunities for (1) testing theories of SRL and (2) improving our understanding of motivational constructs. We then provide some initial results from a study of geometry learners’ achievement goals at domain and unit levels.

Benefits of a Microgenetic Approach

Testing Theories of Self-Regulated Learning

A microgenetic approach allows us to isolate a particular component of a learning theory and use transaction level data to determine if the theorized process plays out as expected when individuals engage in the learning activity. In SRL theories, metacognition is described in a fine-grained manner and many parameters are theorized to affect metacognitive processes in ways that influence learning. Each of these parameters is also theorized to change dynamically over time. Microgenetic methods allow us to focus on a specific metacognitive process and test whether it occurs as theorized, as well as whether the presence of, absence of, or change in another parameter might influence how the metacognitive process works.

For example, Winne theorizes that learners evaluate their learning against a self-set standard (Winne & Hadwin, 1998, 2008; Winne, 1997, 2011). Zimmerman (2011) suggests that such standards are influenced by a learner’s achievement goals, which have been found to vary when measured repeatedly (Fryer & Elliot, 2007; Muis & Edwards, 2009). As an illustration, consider a learner who consistently evaluates performances with respect to a standard over the course of the unit. If we find that his standard changes over time as evidenced by a change in the strength or prominence of one achievement goal over others, then we can explore the implications of this change on the learner’s behavior. To do so, we would examine log-file data prior to and after a shift in goal endorsement (i.e., when a learner who previously rated mastery approach goals as strongest now rates performance avoidance goals as strongest; Muis & Edwards) and examine the time elapsed between hint requests and the next transaction. Perhaps we notice that when his goals shift from a stronger desire to master a skill to a stronger desire to perform just well enough to complete the unit, the learner also spends less time reading hints (smaller durations of time between a hint request and the next click) and a pattern of hint abuse (i.e., rapid clicking to a final hint that provides the answer to a problem step, but where the speed of clicks suggests minimal consideration of the conceptual scaffolding provided). We would expect such behavior to produce poor learning and can test this by analyzing the students’ learning curve of various knowledge components traced by the tutor. Learning curves show the change in a performance metric (e.g., accuracy, time) over successive opportunities to apply a given skill, based on the performance of a group of students on problem steps that require that skill. The slope of the curves indicates the rate of learning. If our hypothesis is accurate, a learner who switches from a mastery approach goal to a performance avoidance goal should have a learning curve with a slope that flattens when the goal changes and a new pattern of behavior emerges (Koedinger et al., 2010).

This hypothetical example illustrates how a microgenetic approach allows us to isolate one element of a theory and determine whether a change in motivation precipitates a change in behavior. This approach opens new dimensions of investigation for testing the role of motivational constructs in SRL theories. A better specification of the dynamic role of motivation in SRL theories will further improve both the explanatory power of these models as well as improving the predictions for individuals’ learning.

Investigating Motivational Constructs at Different Grain Sizes

Collecting traditional pre/post and fine-grained prompts allows for comparison of motivational constructs at different levels of granularity. With this data, we can determine whether the influence of a motivational construct on a learning process changes when the construct is investigated at domain, unit, and problem (see Fig. 41.2). This multilevel depiction of phenomena like achievement goals and self-efficacy enables scientists to examine patterns of stability and change and improve theoretical models to account for this change.

For instance, we might seek to test Bandura’s (1997) hypothesis that learners’ level of self-efficacy is related to their level of performance on problems. We could measure this construct (self-efficacy) at pretest or at posttest and test correlation with course grades. However, it is possible that students might feel confident in their understanding of some concepts but not others and may excel on problems testing some skills and struggle on problems testing others. When learners are asked to make domain-level judgments, they must take an ‘average,’ so to speak, of their distinct self-efficacy judgments. In this self-reported averaging – or perhaps they simply use the last episode they can remember—some variation and precision is likely to be lost. Similarly, using student grades as a measure of performance can oversimplify scenarios where a learner performs well on one type of problem and poorly on another. In our method, we also prompt students to make efficacy judgments immediately after a unit in the tutor and compare them to measures of performance on the unit. At a finer grain still, we also prompt learners’ to judge self-efficacy immediately after problems that align to one of the unit’s learning objective (see problem-level assessment in Fig. 41.2). By sampling self-efficacy at this grain size, we could determine whether Bandura’s hypothesis holds at both the domain level (as evidenced by a significant correlation between domain-level self-efficacy collected as a pretest to math performance represented by grades) and at the problem level (correlation between problem-level efficacy judgments and performance on problems). If we were to find that the correlation between students’ self-efficacy and performance is lower at the problem level than at the domain level, we would have discovered that self-efficacy judgments are associated with performance at more general levels of specificity, but this relationship weakens in the context of an actual task. We could then examine what other factors might inform students’ self-efficacy judgments by looking at behaviors, performances, or motivational factors that may also predict variance in problem level self-efficacy judgments.

Investigating Associations Between Motivation and Metacognition

We can also use these fine-grained samplings of efficacy to examine the effect of an attempt at metacognitive control on a motivational variable. For example, we might test whether efficacy increases after students view a conceptual hint by identifying all learners who requested a hint and examine their efficacy judgments on problems testing a skill before and after the hint request. When a learner identifies that she does not understand a concept, she might seek help from the tutor and request a hint. This represents a cognitive judgment (i.e., that she needed help) and by isolating instances of this action and the students’ responses to self-efficacy prompts, we can test theoretical assumptions about relationships between help seeking and self-efficacy. The additional inclusion of performance data (available in the log-file data) allows us to examine how a motivational state and an action spurred by a metacognitive control process affect learning. We next provide an empirical example of our work employing embedded questionnaires to examine the dynamic nature of motivational variables when learning with the cognitive tutor.

An Empirical Example of the Dynamic Nature of Motivation and Its Effect on Learning Behaviors

An abundance of studies have demonstrated the knowledge tracing capabilities of the cognitive tutor (e.g., Ritter, Anderson, Koedinger, & Corbett, 2007), and additional studies have demonstrated that the tutor is also an effective platform for identifying learning behaviors, scaffolding those that are adaptive (e.g., help-seeking: Aleven, Roll, MacLaren, & Koedinger, 2010), and discouraging those that are maladaptive (e.g., gaming; Baker et al., 2006).

To determine whether the tutor can be adapted to assess student motivation, we examined 72 high school geometry students’ responses to domain-level questionnaires administered at the beginning and end of a semester and a series of unit-level questionnaires administered immediately after the final problem set of the unit was completed (Bernacki, Nokes-Malach, & Aleven, 2012). Our goal was to examine the relationship between domain and unit-specific motivation, the influence of task conditions on motivation, and relationship between motivation and learning behaviors in an intelligent tutoring system.

We tested the stability of achievement goals across levels of specificity by determining whether domain and unit-level achievement goals correlate (indicating stability), or if achievement goals for specific units differed from domain-level achievement goals. We also examined individuals’ self-reported achievement goals across five units to determine whether learners endorse similar achievement goals across units despite known differences in content (e.g., task difficulty and duration). Students used the software two days per week during scheduled math classes and some worked with the software as homework. Units varied in the number of problems students completed per unit (medians ranged from 20 to 40 problems), total time spent per unit (medians ranged 34–73 min). The content of these units included multiple geometry principles such as the Pythagorean theorem, calculation of area, and properties of triangles and trapezoids.

In the first analysis, students reported different achievement goals (i.e., mastery-approach, performance-approach, and performance avoidance) when they are measured at different levels of specificity (domain and unit level). In all but one case, correlations between students’ self-reported achievement goals for math versus achievement goals for the unit they just completed were nonsignificant, and in some cases, the correlation was actually negative. We take this to mean that students are pursuing different goals in the mathematics units they just completed compared with those they report when they reason abstractly about their goals in math.

When we examined the stability of achievement goals across units, unit-level achievement goals were highly correlated. Correlation coefficients across all pairs of units per construct ranged from r  =  0.30 to 0.71 (mean r  =  0.58). However, achievement goals were variable within learners. When averaging the proportion of students who report increases, decreases and no change in achievement goals across all pairs of units, we found that approximately one third of students increased in their endorsement of each achievement goal, one third decreased and the third reported no change in their goals. This within-learner variability confirms that fine-grained measurement is important, so long as these differences in achievement goals have implications for the behaviors learners conduct in the tasks.

When we examined the relationship between domain-level and unit-level achievement goals and learning behaviors (by comparing the coefficient of determination (R 2) for regression equations where learning behaviors were regressed on a set of domain-level or unit-level achievement goals in a single unit), results indicated that for some behaviors (help seeking, error rate and accuracy) domain-level achievement goals were better predictors of behavior, whereas for others (problems needed to achieve competence, seconds needed to complete problem) unit-level achievement goals were better predictors. Collectively, these findings indicate that when students self-report their domain-level and unit-level achievement goals, they reflect different aspects of learners’ motivational states, and these aspects are useful for predicting different learning behaviors.

Because achievement goals were found to vary by level of specificity and across units, and because they can be used to predict the behavior of learners, we confirmed that there are benefits to assessing achievement goals at a fine-grained level. Additional studies are underway that prompt students to endorse their achievement goals after a problem and within a unit (i.e., between math problems) so that we might be able to examine how a change in one’s achievement goals might instantiate a change in one’s approach to solving math problems. We are also examining self-reports of self-efficacy for math tasks after units and between problems to confirm that such measures can provide information about the learning behaviors common to students with particular perceptions of their efficacy.

Conclusion

Our approach takes the first step towards the development of fine-grained assessments of motivation as learners engage in SRL processes. The preliminary evidence suggested that learners’ unit-specific motivations may differ from their domain-level motivations, that learners’ motivations change along with changes in task conditions, and that motivational data collected in conjunction with a unit can better predict a set of learners’ behaviors as they engaged with the tutor. For these reasons, the approach appears to be fruitful for testing theories that posit interactions between motivation, cognition, metacognition, and learning outcomes. Despite these benefits, the approach has its limitations, and measurement challenges remain. We need to assess the reliability of students’ responses to items to determine the degree to which variation in responses can be attributed to true differences in a motivational state versus variation due to measurement error. Similarly, we need to find ways to validate these questionnaires through behavioral or observational measures. We must also be wary of the influence that interrupting students’ learning with prompts to answer questionnaire items may have on their learning. A long-term goal of this project is to validate students’ responses to questionnaire items and then use existing log-file data and questionnaire responses to develop machine learned detectors for motivational variables. If this can be accomplished, we can then move past embedded questionnaires and assess motivation using the same unobtrusive methods used to trace behaviors representing the cognitive and metacognitive processes characteristic of SRL.