Keywords

1 Introduction

Intelligent Tutoring Systems often are strongly system-controlled learning environments that adaptively select problems for students based on their knowledge level [13]. Recently, some ITSs have started to grant students control to select their own learning tasks to elicit higher motivation, which in turn may lead to better learning outcomes [6]. However, prior research has found that students are not good at making effective problem selection decisions [9]. Fully student-controlled problem-selection was found to lead to worse learning outcomes than system-selected problems [2]. Hence some ITSs created shared student/system control over problem selection (e.g., letting the system pick problem types while the students select a specific problem from that type) to prevent students from making suboptimal decisions and achieved comparable learning outcomes to those achieved with full system control [6]. It is still an open question how ITSs can be designed to foster better learning outcomes and higher motivation with shared control over problem selection, as compared to full system control. In addition, theories of SRL emphasize the important role of motivation in promoting desirable SRL behaviors [15]. Yet little work with ITSs has adopted a motivational design (i.e., design to foster motivations) approach to foster appropriate motivations that will stimulate effective problem-selection behaviors. Most of the interventions that support SRL processes in ITSs use cognitive and metacognitive tools, such as prompts and feedback [3, 11]. Furthermore, few of these studies have measured the lasting effects of the interventions when they are not in effect [1].

We tackle these open questions by applying motivational design to extend an ITS for equation solving, Lynnette, to help students learn an effective problem-selection rule, i.e., the Mastery Rule, while fostering a mastery-approach orientation [8]. The Mastery Rule specifies that students should stop practicing a problem type once it is fully mastered. An ITS that implements this rule (in a system-controlled manner) led to better learning outcomes than a fixed curriculum [5]. Our prior classroom studies and interviews with students revealed that the lack of a mastery-approach orientation might be a main challenge that keeps students from applying the Mastery Rule when they can select problems for themselves [8]. A mastery-approach orientation is a type of achievement goal that is associated with positive learning behaviors such as perseverance and willingness to learn new materials [14]. It aligns with the desirable problem selection behaviors based on the Mastery Rule. It is likely, but unproven that students with a mastery-approach orientation will apply the Mastery Rule to select problems and achieve better learning outcomes in the tutor, as compared to full system control. We therefore added features to foster a mastery-approach orientation. We refer to these features as the mastery-oriented features.

The current paper describes our classroom experiment that investigated two research questions: Research Question 1: Compared to full system control over problem selection, does shared control, supported by mastery-oriented features enhance students’ (a) problem-selection decisions in the tutor; (b) domain-level learning outcomes; (c) enjoyment and (d) knowledge of the Mastery Rule? Research Question 2: Do the mastery-oriented features enhance students’ (a) future problem-selection decisions and (b) future domain-level learning outcomes in an environment with shared control but without mastery-oriented features, as compared to full system control?

2 Methods

2.1 Experimental Design

The Learning Phase Versus the Future Learning Phase. The classroom experiment used a two-phase design, with a Learning Phase and a Future Learning Phase, so that we could investigate both immediate effects of mastery-oriented shared control (Research Question (1) and effects on future learning without the mastery-oriented features (Research Question (2). We created three variations of Lynnette for different conditions in the two phases, Lynnette-System, Lynnette-Shared and Lynnette-Shared-Mastery-Oriented. Lynnette-System implements full system control over problem selection through Bayesian Knowledge Tracing (BKT) and Cognitive Mastery [5], as in standard ITS. Both Lynnette-Shared and Lynnette-Shared-Mastery-Oriented implement shared control over problem selection. As shown in Fig. 1, students are free to select any level they want to practice and decide how much practice they want for each level. Once the student selects a level, the tutor assigns a specific problem from the chosen level. Students are able to select problems even after they have fully mastered that level in these two versions (as calculated by the tutor’s BKT and displayed by the mastery bars for each level). Only Lynnette-Shared-Mastery-Oriented has the mastery-oriented features that we describe below. All three Lynnette versions have the element badges and mastery bars for each level (as seen in Fig. 1).

Fig. 1.
figure 1

Problem selection screen in Lynnette-Shared-Mastery-Oriented in the learning phase

Full size image

The experiment started with two conditions in the Learning Phase, and only Levels 1 to 6 were unlocked in this phase. As shown in Table 1, the “Mastery Shared” condition used Lynnette-Shared-Mastery-Oriented, while the “Standard Tutor” used Lynnette-System. By comparing these two conditions, we can address Research Question 1, i.e., whether the mastery-oriented shared control leads to better problem-selection, learning and enjoyment as compared to full system control. In the Future Learning Phase, Levels 7 to 9 were also unlocked and the two conditions were split into four. Half of the participants from the “Mastery Shared” condition were assigned to use “Lynnette-Shared” and half use “Lynnette-System”. Similarly, half of the “Standard Tutor” condition switched to “Lynnette-Shared” and half continued using “Lynnette-System”. The four conditions in the second phase allowed us to investigate Research Question 2, i.e., the effects of the mastery-oriented features on students’ problem selection and learning outcomes when they are removed in new tutor units with shared control, compared to full system control.

Table 1. Conditions of the learning phase and the future learning phase
Full size table

Mastery Oriented Features in Lynnette. There are four mastery-oriented features in Lynnette-Shared-Mastery-Oriented that aim at helping students learn the Mastery Rule and foster a mastery-approach orientation [8]: (1) Tutorial: A tutorial is shown when students log in to the tutor for the first time. It introduces the concept of Mastery, the mastery bars, and how to apply the Mastery Rule to select problems. (2) Achievements and Stars: Two types of Achievements are implemented in the tutor to reward students’ good problem selection decisions and perseverance with practicing new problems, as shown on the right panel of the screen in Fig. 1. Students earn the Achievements when they select or complete 6 problems in a row. In addition, the student earns a star each time s/he selects an unmastered problem. (3) Instant Feedback Messages on Problem Selection Decisions: Each time the student selects a problem, either a positive message (e.g., “Good problem selection decision! Water is still unmastered, so you can learn new skill from it. Don’t be discouraged if you feel it is difficult. It is ok to make errors when you are learning!”) or a negative message (e.g., “You’ve picked Earth but it is already mastered. Your equation solving skill will not grow if you repeat material you’ve already mastered.”) will pop up and provide feedback on her/his choice. The language used in the messages emphasizes a mastery-approach orientation. (4) Problem Selection Recap: The problem selection recap screen (as shown in Fig. 2) is shown to the students after every 5th problem, in order to help students review and reflect on their recent problem selection decisions. The specific problem levels the student has selected are displayed with corresponding mastery bars showing the percentages of mastery at the time the student selected each level. The student also receives instant feedback on whether s/he has correctly clicked the unmastered levels. The names of the problem levels turn green or red when the student clicks. Green flags a correct click.

Fig. 2.
figure 2

The problem selection recap screen in Lynnette-Shared-Mastery-Oriented

Full size image

2.2 Procedure, Measurements and Participants

The experiment included 294 students from 5 local middle schools. The participants came from 16 classes, taught by 8 different teachers. Among the 16 classes, 4 were advanced 6th grade classes, 9 were mainstream 7th grade classes, and 3 were mainstream 8th grade classes. The participants were randomly assigned to one of the four conditions within each class before the experiment. All conditions followed the same procedure, summarized in Table 2, consisting of a Learning Phase and a Future Learning Phase. Three paper tests were given to measure different constructs before and after each phase of learning. Each equation on the three tests was graded from 0 to 1, with partial credit given where appropriate. The pre-test only had items from Levels 1 to 6. The mid-test and post-test had items that measure equation solving abilities for all 9 levels. The enjoyment questionnaire was adapted from the Enjoyment subscale of the Intrinsic Motivation Inventory (IMI). There were three check-box items on the mid-test to measure the students’ declarative knowledge of applying the Mastery Rule. The first item tested the students’ understanding of the concept of mastery. The second item described a scenario and tested whether the students would keep selecting problem levels that have been mastered. The third item also was scenario-based, and it tested whether the students were willing to challenge themselves with new problem types to learn new skills.

Table 2. Overview of the procedure and measurements of the experiment
Full size table

3 Results

200 students completed the pre-test and mid-test, and were present in all four class periods or mastered the first six levels during the Learning Phase. We refer to these 200 students as the Learning-Phase-Sample. 165 students completed the pre-test, mid-test and post-test. They were present during all 6 class periods (both the Learning and Future Learning Phases) or mastered all 9 levels. These students constitute the Future-Learning-Phase-Sample. We report Cohen’s d for effect sizes. An effect size d of .20 is typically deemed a small effect, .50 a medium effect, and .80 a large effect. For all ANCOVAs, Teacher was used as a co-variate to account for the variances that reside within different teachers’ classes.

3.1 The Learning Phase: Research Questions 1.a – 1.d

We first analyzed data from the Learning Phase, to answer Research Questions 1.a-1.d. The Learning-Phase-Sample was used for all analyses.

Problem Selection Decisions (RQ 1.a). To test the hypothesis that mastery-oriented features will help foster more consistent application of the Mastery Rule, we looked at the percentage of mastered problems the students selected in the “Mastery Shared” condition during the Learning Phase (under perfect application of the Mastery Rule, the students should not select any mastered problems). Twenty out of 102 students (19.61 %) in the “Mastery Shared” condition selected at least one mastered problem during the Learning Phase. On average 1.4 % of the problems (SD = 3.8 %) selected by each student in the condition were mastered problems, indicating good application of the Mastery Rule when the mastery-oriented features were present.

Learning Outcomes (RQ 1.b). To test the hypothesis that mastery-oriented shared control over problem selection will lead to greater learning gains than full system control, we compared the two conditions’ test performance on equation solving. As shown in Table 3, both conditions scored close to ceiling on the pre-test. An ANCOVA using the learning gain (Mid-Test-Equations1 minus Pre-Test) as the dependent variable revealed that the main effect of condition is significant (F (1, 192) = 4.486, p = .035, d = .30). In other words, The “Mastery Shared” condition learned significantly more during the Learning Phase than the “Standard Tutor” condition. However, given the ceiling effect, the students did not improve significantly from pre-test to mid-test on solving the equations.

Table 3. Means and SDs for test performance of levels 1–6 equations on pre-test and mid-test
Full size table

Given the ceiling effect on the pre-test, we split the sample based on the median of the pre-test score (median = .83) into two sub-groups: the Lower-Performing Group and the Higher-Performing Group. The Lower-Performing Group had 102 students (mean pre-test = 0.67, SD = 0.18), and the Higher-Performing Group had 98 students (mean pre-test = 0.98, SD = 0.05). ANCOVAs revealed that overall the two conditions improved significantly from pre-test to mid-test on Equations1 within the Lower Performing Group (F (1, 94) = 13.451, p < .000, d = .76). The condition effect was marginally significant (F (1, 94) = 3.490, p = .065, d = .37), with the “Mastery Shared” condition improving more than the “Standard Tutor” condition. On the other hand, there was a significant decrement of the two conditions’ performance from the pre-test to mid-test within the Higher-Performing Group (F (1, 90) = 25.704, p < .000, d = 1.07), probably representing regression to the mean. No significant condition effect was found for the learning gains within the Higher-Performing Group.

Enjoyment (RQ 1.c). To test the hypothesis that mastery-oriented shared control over problem selection will lead to higher enjoyment of using the tutor than full system control, we compared students’ enjoyment ratings on the mid-test. The “Mastery Shared” condition reported higher enjoyment (mean = 4.63, SD = 1.59) than the “Standard Tutor” (mean = 4.52, SD = 1.36). However, an ANCOVA test found the difference was not statistically significant (F (1, 192) = .450, p = .530, d = .09).

Declarative Knowledge (RQ 1.d). To test the hypothesis that the mastery-oriented features with shared control will lead to better knowledge of the Mastery Rule, compared to full system control, we analyzed the students’ responses to the three items on the mid-test. There were 12 options for all three items. The students were instructed to check all options that apply. We coded the students’ responses to each option as 0 or 1. On average those in the “Mastery Shared” condition (mean = 0.76, SD = 0.15) scored significantly higher (F (1, 184) = 8.263, p = .005, d = .59) than those in the “Standard Tutor” condition (mean = 0.69, SD = 0.17). The “Mastery Shared” condition showed significantly better declarative knowledge of the Mastery Rule on the mid-test after the Learning Phase.

3.2 The Future Learning Phase: Research Questions 2.a and 2.B

We performed analyses on students’ problem selection decisions and equation solving performance. The Future-Learning-Phase-Sample was used for all analyses.

Problem Selection Decisions (RQ 2.a). We tested the hypothesis that the students exposed to the mastery-oriented shared control over problem selection in the Learning Phase will transfer and apply the Mastery Rule during the Future Learning Phase with the shared control. Specifically, we compared students’ problem-selection decisions between the “Mastery to Shared” condition and the “Standard to Shared” condition. In the “Mastery to Shared” condition, 15 out of 49 students (30.61 %) selected at least one mastered problem during the Future Learning Phase, whereas in the “Standard to Shared” condition, 7 out of 35 (20 %) students selected at least one mastered problem. Moreover, on average 2.7 % of the problems selected by the “Mastery to Shared” condition were mastered, while 1.6 % selected by the “Standard to Shared” condition were mastered. Nevertheless, an ANCOVA test revealed that the difference between the percentages of these two conditions was not statistically significant.

Learning Outcomes (RQ 2.b). To test the hypothesis that shared control over problem selection (without mastery-oriented features) will lead to better learning outcomes in the Future Learning Phase, compared to full system control, we performed ANCOVAs to analyze students’ learning gains from the mid-test to post-test. Two independent variables were used in the ANCOVA analyses: (1) whether the students had mastery-oriented shared control or full system control over problem selection in the Learning Phase, and (2) whether they had shared versus system control during the Future Learning Phase. As shown in Table 4, the students’ performance on Equations1 did not change much from mid-test to post-test. An ANCOVA revealed no significant improvement from the mid-test to post-test for Equations1 for the four conditions. Also, no significant main effects or interaction were found for Equations1 with the two independent variables. On the other hand, overall the four conditions improved significantly on Equations2 from mid-test to post-test (F (1, 155) = 37.028, p < .000, d = .98), as well as the whole test (with Equations1 and Equations2 together, F (1, 155) = 16.839, p < .000, d = .66). However, no significant main effects or interaction were found between the conditions for Equations2 or the whole test.

Table 4. Means and SDs for mid-test and post-test equation solving items
Full size table

4 Discussion, Conclusions and Future Work

Our classroom experiment investigated whether mastery-oriented shared control over problem selection would foster the learning of an effective problem selection strategy, students’ learning outcomes and enjoyment, as well as future problem selection and future domain-level learning. We found that shared control over problem selection, while it was supported with mastery-oriented features, led to better learning outcomes as compared to full system control in an ITS. Specifically, during the Learning Phase, those in the mastery-oriented shared control condition improved significantly more than those in the system-controlled condition on equation solving. Although the two conditions overall did not improve significantly due to the ceiling effects on the pre-test. Within the lower-performing group, there were significant learning gains from pre-test to mid-test, and the condition effect was marginally significant. These results prove that shared control accompanied by mastery-oriented features can significantly benefit students’ domain level learning, especially for students with low prior knowledge. How did the mastery-oriented shared control over problem selection lead to greater learning gains? First, the students with the mastery-oriented shared control selected almost the same problems as the system control. They rarely violated the Mastery Rule, put differently, the students selected mostly unmastered problems as the Cognitive Mastery algorithm does for the system control. Therefore, we can mostly rule out the possibility that the difference in learning gains was due to differences in the problem sequences being practiced. Second, it is likely that the mastery-oriented features (tutorial, feedback, achievements and problem selection recap screens) might have encouraged the students to adopt metacognitive strategies such as reviewing, reflecting or summarizing, as a mastery-approach orientation has been found to be positively associated with use of such strategies [14]. Prior work has generally found that students with a mastery-approach orientation achieve better learning outcomes, compared to their counterparts who focused more on performance relative to others, i.e., with a performance orientation [12].

We also found that the mastery-oriented shared control resulted in significantly better declarative knowledge of the Mastery Rule, as compared to the full system control condition. It could possibly be attributed to the explicit instructions and motivational messages from the four mastery-oriented features. On the other hand, the mastery-oriented shared control did not lead to significantly higher enjoyment of using the tutor as compared to the full system-controlled tutor. It is likely that the badges, as well as the mastery bars implemented in the system-controlled condition also made it enjoyable to students. Prior work on learner control emphasizes its motivational benefits to students [4], but our finding suggests that enabling learner control does not necessarily enhance students’ enjoyment of the learning experience.

Although the mastery-oriented shared control enhanced students’ learning while it was in effect, no lasting effect on learning was found with only shared control over problem selection. For the Future Learning Phase, no significant condition effects were observed for learning gains on equation solving. In other words, there was apparently no carry into the next unit of a possible motivational effect on student learning. Additionally, the equations in this phase were more difficult than the Learning Phase, and the learning time was reduced to 2 class periods. The students might experience higher cognitive load when learning more difficult equations within a shorter period of time, making it difficult to initiate metacognitive processes such as reviewing or reflecting that relate to a mastery-approach orientation.

Lastly, with respect to problem selection decisions, students with shared control exhibited good application of the Mastery Rule in both phases. The mastery-oriented shared control condition selected only about 1 % of mastered problems during the Learning Phase. Similarly, the two shared control conditions without the mastery-oriented features in the Future Learning Phase selected around 2 % of mastered problems regardless of whether or not they came from the mastery-oriented shared control condition. The results regarding problem selection decisions were slightly surprising, given that in our prior classroom study, students selected 34 % mastered problems when no Open Learner Model was presented [8]. In other prior work, we also found students admitting that they would keep selecting easy problems if given control over problem selection [7]. There may be two reasons why students made overall good problem selection decisions in both phases: First, our informal classroom observations found that the badges and the mastery bars strongly encouraged the students to complete the levels without repeating already-mastered problems. Although these two features were designed to make the tutor more fun and reward students’ equation solving progress, not to influence problem selection, they might have motivated the students to make problem-selection decisions based on the Mastery Rule. A second reason may have been that the environments for this experiment were not entirely self-regulatory. The students were learning in their math classes and the teachers sometimes gave informal instructions such as “now you should work on the newly unlocked levels”. The students were practicing with a “goal” and supervision from their teachers, which might have influenced their problem selection decisions.

To sum up, the current experiment shows that shared control over problem selection accompanied by features that foster a mastery-approach orientation in an ITS leads to significantly better domain-level learning outcomes, as compared to full system control over problem selection, which is standard practice in ITS. This is a novel contribution to the literature on the effects of learner control on student learning, which has generally found that pure learner control leads to worse learning than system control [2, 10] and that shared control only resulted in learning outcomes that were comparable to system control [6]. On the other hand, our experiment did not establish lasting effects of the mastery-oriented features on future learning in a new tutor unit, with improvement only on declarative knowledge of applying the rule on an immediate paper test. Future work is warranted to further investigate how to design ITSs that support learning and motivation of Self-Regulated Learning processes that can transfer to new learning topics and environments.