Keywords

1 Objectives, Theoretical Framework, and Related Work

Self-regulated learning (SRL) is a hallmark of human learning and a key factor in problem solving, reasoning, and understanding complex instructional and training materials with advanced learning technologies (ALTs) such as intelligent tutoring systems (ITSs) [1, 2]. For example, when learning about complex STEM topics, research indicates that individuals can gain deep conceptual understanding through the effective use of cognitive, affective, metacognitive, and motivational (CAMM) self-regulatory processes [1, 3–6]. The successful use of cognitive and metacognitive SRL processes involves setting meaningful goals for one’s learning, planning a course of action for attaining these goals, deploying a diverse set of effective learning strategies in pursuit of the goals, continuously and accurately monitoring one’s own understanding of the material and the appropriateness of the current information, and adapting one’s goals, strategies, and navigational patterns based on the results of such monitoring processes and resulting judgments [7]. Unfortunately, there is ample interdisciplinary evidence to show that few learners engage in effective SRL [8, 9]. Although motivation and affect [10–12] play a role in determining learners’ willingness to self-regulate, we assume a lack of cognitive and metacognitive self-regulatory knowledge and skills is the main obstacle to adequate regulation and, subsequently, deficient learning gains and conceptual understanding [2].

Furthermore, learners attempting to self-regulate often face limitations in their own metacognitive knowledge and skills, which, when compounded with a lack of domain knowledge, can result in cognitive overload, negative affective reactions, and decreased interest and persistence [6, 11, 12]. One method of relieving the cognitive burden placed on learners in this situation is to provide assistance in the form of adaptive scaffolding. Similar to seminal work by Graesser and colleagues and Chi and colleagues, previous experiments conducted by Azevedo and colleagues [7] on human tutors as external regulating agents established that adaptive scaffolding provided by a human tutor leads to greater deployment of sophisticated planning processes, metacognitive monitoring processes, and learning strategies as well as larger shifts in mental models of the domain. The purpose of the current work on externally regulated learning (ERL) is to empirically test whether the adaptive scaffolding provided by multiple artificial PAs (as externally regulating agents) within a hypermedia-based ITS (i.e., MetaTutor) is also capable of producing the same, or better, learning outcomes and increased use of effective SRL processes during STEM learning. As such, this study examines the effectiveness of several PAs in externally regulating and fostering complex learning with ITSs.

2 Method

2.1 Participants

One hundred twenty college students (52 % female) from a large university in North America participated in this study in 2015. The mean age of the participants was 20.4 years and their mean GPA was 3.29. All participants were paid up to $40 for completion of the 2-day, 4-hour experiment.

2.2 Pretest and Posttest Measures

Several materials were developed for this study including self-report measures of emotions (e.g., EV, revised Agent Persona Inventory) and motivation and two versions of the pretest and posttest about the human circulatory system. For example, the pretest and posttest each included 30 four-foil multiple-choice items.

2.3 MetaTutor: Intelligent Hypermedia-Based Tutoring System for Biology

MetaTutor is an intelligent hypermedia-based tutoring system that includes 47 pages of text and static diagrams of the human circulatory system [13, 14]. During learning participants were guided by four PAs that provided timely scaffolding for each participant. Each agent, aside from Gavin the Guide, offered support on one specific component of SRL (i.e., planning, metacognition, and cognitive strategies). Gavin’s objective was to support participants as they navigated the environment. Pam the Planner supported participants by emphasizing planning, activating prior knowledge, and creating relevant subgoals. Mary the Monitor supported participants by helping them monitor various metacognitive processes and make accurate metacognitive judgments during the session. Mary recommends the use of metacognitive processes such as content evaluations (CE), feelings of knowing (FOK), judgments of learning (JOL), and monitoring progress toward goals (MPTG). Sam the Strategizer encouraged effective cognitive strategy use (i.e., coordinating informational sources, making inferences, taking notes, summarizing hypermedia science content) as participants progressed toward completing their goals.

The MetaTutor interface (see Fig. 1) was designed to support, model, and foster self-regulated learning. The center of the interface contains the text and diagrams. These are the learning materials that are used to accomplish all subgoals and the overarching goal of learning about the circulatory system. The SRL palette is located on the right pane of the interface and enables participants to engage in SRL strategies. By clicking on the elements of the palette, participants can use eight strategies: creating summaries, making inferences, taking notes, activating prior knowledge, MPTG, CE, JOL, and FOK. Participants are free to use any of these components at any time throughout the session, and the strategies can be either user- or system-initiated. One of the four PAs is located just above the palette, in the top right corner of the interface. The agent that is displayed is dependent upon the circumstances of the session (i.e., what type of scaffolding the system is deploying, what type of instruction is being offered). Only one agent is displayed at a time in this window. To the left of the agent window and at the top of the interface are the participants’ subgoals. The subgoal that the participant is currently working on is located at the top of the list as a reminder. On the left-hand pane of the interface, below the timer, is the table of contents. This allows participants to view the titles of each of the 47 pages, which are organized by section. At the bottom of the interface we find a textbox where participants are able to enter their subgoals and their prior knowledge about those subgoals. The textbox is also used throughout the session when the participant engages in certain SRL processes.

Fig. 1.
figure 1

Screenshot of the MetaTutor interface.

Full size image

2.4 Adaptive Scaffolding vs. No Scaffolding Conditions

We used two different versions of MetaTutor: one for each of the two conditions, an adaptive scaffolding (AS) condition and a control (C) condition. In the AS condition participants received timely scaffolding from the agents. This scaffolding was designed to reflect the interaction a learner would receive from a human tutor. In this condition there were both user- and system-initiated actions. User-initiated actions were made by using the SRL palette. For example, participants could click on the SRL palette to indicate they wanted to take notes or metacognitively judge the relevancy of text or diagrams to their current subgoals. System-initiated actions were brought on by a complex set of production rules that fire when certain conditions are met. For instance, when participants navigated to a page that was not relevant to their current subgoal and remained on this page for 15 s, a production rule was initiated that would fire Mary the Monitor. Mary would then prompt participants to make a CE about the relevancy of the page and image to their current subgoal. In total, MetaTutor uses 20 production rules (13 cognitive, 7 metacognitive) that are triggered by time and action thresholds. Participants in the AS condition were also afforded feedback from Pam while setting up their subgoals. She informed the participants on whether their proposed subgoal was too broad or too general and then continued to assist the participants in setting an appropriate subgoal.

In the control condition, participants were not afforded feedback or scaffolding from the agents. For example, during the subgoal setting phase, Pam only suggested the subgoal that the participant should choose. In this condition, participants were free to navigate the environment without any feedback or scaffolding from the agents. Further, they were not prompted to use any SRL strategies. However, it is important to note that the participants were still able to engage in SRL strategies on their own, if they so chose; they were afforded the same instructions and instructional videos; and they were exposed to the same multimedia learning content. Thus, the conditions were separated by the element of scaffolding and feedback, whereby the prompt and feedback group engaged in interaction with the agents, and the control condition did not. This design makes it possible to investigate the effectiveness of PAs in scaffolding participants as they engaged in conceptual learning. A complete description of the production rules governing the PAs’ behaviors is beyond the scope of this paper.

2.5 Experimental Procedure

The MetaTutor study took place over two sessions that had to take place within a 3-day span. The first session lasted approximately 30–60 min and the second session lasted up to 180 min. After consenting to the study, participants were instrumented, and the eye-tracker and Attention Tool were calibrated. A baseline was established for electrodermal activity (EDA) as well as for the facial recognition of emotion software. The participant was then presented with an overview of the study. Following the overview, participants filled out a demographic questionnaire and several self-report measures of personality, emotions, and motivation. After completing these measures, participants were administered the 30-item multiple-choice pretest.

During the second session of the study, participants were instrumented, and the eye-tracker and Attention Tool were calibrated. A baseline was established for EDA as well as for the facial recognition of emotion software. The participant was then presented with an overview of what was going to take place during the session and was allowed to begin. The session started with Gavin giving a short introduction, and then an introductory video launched to introduce the agents and give an overview of the user interface and its functionality. After the video, Gavin gave the participants their overarching goal of the session, which was to learn all they could about the circulatory system. Before starting, the participant had to complete the AGQ and EV (i.e., motivation and emotions self-report measures). Next Pam the Planner assisted the participants to set up their subgoals. This was aided by an instructional video that explained how to set subgoals. After the participants successfully set their two subgoals, they were provided with their pretest scores and were offered the opportunity to switch either or both of them with any of the other five possible subgoals. After the participants made a decision on their subgoals, they were asked to recall everything they knew about that particular subgoal. This was used to determine prior knowledge of the learning content. Next the participants were required to take several self-report measures of emotions and motivation, and viewed an informational video that explained how to efficiently use the interface at a higher level (i.e., how to use the SRL palette). At this point, the participants were ready to start learning with the system. Throughout the 90-minute learning session, instrumented participants were presented with several emotions and motivation self-report measures (presented by the system based on time thresholds, learning episodes, assessment results, and SRL activities) while rich trace data were collected for subsequent analyses. At the completion of the session, they were administered the same self-report measures and an equivalent posttest, paid for their time, and debriefed on the study.

3 Results

3.1 Question 1: Do Different Scaffolding Conditions Lead Students to Gain Significantly More Knowledge About the Human Circulatory System?

An analysis of covariance (ANCOVA) with two levels (scaffolding conditions: AS or C), using posttest as the dependent measure and pretest (M C = 18.73, SD = 3.81; M AS = 15.90, SD = 4.58) as the covariate, was performed to answer this research question. Before conducting each analysis, we ensured homogeneity of variance and significance of the covariate for each dependent variable. Results indicated that there were significant differences in posttest scores between experimental conditions while controlling for pretest score; F(1, 117) = 76.90, p < .001, \({\upeta }_{{\rm p}}^{2}=.40\) . Specifically, learners in the AS condition had significantly higher posttest scores (M = 21.12, SD = 4.25) compared to learners in the C condition (M = 19.80, SD = 3.83). Thus, results indicate that when learners were provided with prompts and feedback from the PAs, they outperformed learners who did not receive prompts and feedback from the PAs on the posttest. The maximum score on both pretest and posttest measures was 30.

3.2 Question 2: Do Different Scaffolding Conditions Impact the Duration, Frequency, and Quality of Learning and Knowledge Construction Activities, and Performance on Embedded Assessments During Learning with MetaTutor?

Adaptive scaffolding of SRL by PAs involves well-orchestrated learner-system interactions involving learning activities (e.g., learners reading relevant multimedia content on the biology topic while monitoring several aspects of their existing knowledge of the material, emerging understanding, relevancy of content, etc.) and knowledge construction activities (e.g., taking notes on relevant multimedia content and adding newly found biology content to existing notes) followed by periodic embedded assessments at both the page and subgoal levels to assess the quality of the cognitive and metacognitive SRL processes deployed during learning with MetaTutor. As such, we conducted several independent t-testsFootnote 1 on key variables from learners’ log-files to determine whether scaffolding conditions impacted the duration, frequency, and quality of learning and knowledge construction activities, and performance on embedded assessments during learning with MetaTutor.

The results show that those in the AS condition spent a significantly greater amount of time with the system during the second session (on average approx. 129 min for those in the AS condition vs. 106 min for those in the C condition; see Table 1). This result is accounted for by the amount of time learners in the AS condition were externally regulated by the four PAs while attempting to self-regulate their learning about the circulatory system. In contrast, those in the C condition spent a significantly greater proportion of time reading the science content. The acquisition and retention of the science content, based on reading, was periodically assessed by having learners perform page-level quizzes, and the results show that those in the AS condition took significantly more page-level quizzes and scored significantly better on them compared to those in the C condition. Note taking is a key knowledge construction activity, and while our results indicate no significant differences in the frequency and duration of note-taking events (including note checking) by both groups, we did find significant differences that show learners in the AS condition checked their notes more frequently and added more summaries to their existing notes compared to learners in the C condition. Lastly, learners in neither condition differed in the number and scores on subgoal quizzes (see Table 1).

Table 1. Means (and SDs) for learning and knowledge construction activities, and embedded assessments by scaffolding condition.
Full size table

3.3 Question 3: Do Different Scaffolding Conditions Impact the Frequency of Cognitive Strategies and Metacognitive Processes Deployed During Learning with MetaTutor, After Controlling for Pretest?

Understanding the impact of PAs’ external regulation of learners’ SRL requires analyses of the frequency of both learner-initiated SRL moves and system-initiated ERL moves during learning with MetaTutor. As such, for assessing the differences in the use of cognitive learning strategies between scaffolding conditions while controlling for pretest score, we ran a MANCOVA with total summaries, total note taking, and total inferences made as the three dependent variables, scaffolding condition as the independent variable, and pretest score as the covariate. Results indicated a significant MANCOVA; Wilks’ λ = .44, F(3, 115) = 49.37, p < .001, \({\upeta }_{{\rm p}}^{2}=.56\). Between-subjects effects indicated, while controlling for pretest score, significant differences in total summaries (F(1, 117) = 144.50, p < .001, \({\upeta }_{{\rm p}}^{2}=.55\)) and total note taking instances (F(1, 117) = 4.88, p = .03, \({\upeta }_{{\rm p}}^{2}=.04\)) between scaffolding conditions; however, there were no significant differences in total inferences made (F(1, 117) = 1.01, p = .32, \({\upeta }_{{\rm p}}^{2}=.01\)) between scaffolding conditions. Specifically, while controlling for pretest, learners in the AS condition made significantly more summaries (M = 9.83, SD = 6.04) and took significantly more notes (M = 12.97, SD = 12.82), compared to the total summaries (M = 0.20, SD = 0.44) and note taking instances (M = 8.37, SD = 14.21) by those in the C condition (see Fig. 2).

Fig. 2.
figure 2

Mean frequency of three cognitive strategies by scaffolding condition (Color figure online).

Full size image

Additionally, we assessed the differences in use of metacognitive processes between scaffolding conditions, and ran a second MANCOVA with total JOL, total FOK, total CE, and total MPTG as the four dependent variables, scaffolding condition as the independent variable, and pretest score as the covariate. Results indicated a significant MANCOVA; Wilks’ λ = .48, F(4, 114) = 30.74, p < .001, \({\upeta }_{{\rm p}}^{2}=.52\). Between-subjects effects revealed, while controlling for pretest score, that there were significant differences in total JOLs (F(1, 117) = 16.79, p < .001, \({\upeta }_{{\rm p}}^{2}=.13\)) and total CEs (F(1, 117) = 113.14, p < .001, \({\upeta }_{{\rm p}}^{2}=.49\)) between scaffolding conditions; however, there were no significant differences in total FOKs (F(1, 117) = 1.64, p = .20, \({\upeta }_{{\rm p}}^{2}=.01\)) or total MPTGs (F(1, 117) = .73, p = .40, \({\upeta }_{{\rm p}}^{2}=.01\)) between scaffolding conditions. Specifically, learners in the adaptive scaffolding condition made significantly more JOLs (M = 6.72, SD = 7.68) and CEs (M = 7.27, SD = 4.65) than learners who made JOLs (M = 1.72, SD = 4.59) and CEs (M = 0.30, SD = 0.70) in the control condition (see Fig. 3).

Fig. 3.
figure 3

Mean frequency of four metacognitive strategies by scaffolding condition (Color figure online).

Full size image

4 Conclusions and Future Directions

Our results indicate the adaptive scaffolding provided by PAs is effective in fostering complex learning about challenging STEM topics with ITSs such as MetaTutor in a relatively short amount of time (approx. 2 h). We demonstrated that compared to a control condition (where learners were not provided external regulation by PAs), those in the AS condition significantly improved their learning from pretest to posttest, spent disproportionately less time reading content (compared to other activities), took more page-level quizzes and scored significantly better on them, and also checked their notes more often and added summaries to them throughout the learning session. In addition, PAs’ adaptive scaffolding was effective in prompting learners to use more cognitive strategies such as creating summaries and notes about the topic as well as using key metacognitive processes such as making JOLs and CEs to enhance their learning.

Our data also revealed some interesting results that need further examination by analyzing the rich multimodal trace data collected in this study. First, despite spending more time reading content, those in the C condition did not outperform those in the AS condition. This leads us to believe that finer-level analyses of the trace data are necessary to understand the dynamics between learners’ SRL and the PAs’ ERL throughout the session that facilitated better performance on the posttest. In addition, this finding also raises the questions about quantity versus quality—that is, more reading does not directly translate into more learning because more accurate and efficient reading by using key cognitive and metacognitive processes such as JOLs and CEs in combination with cognitive strategies such as summaries and note taking is key to foster complex learning. The same reasoning applies to the duration of note taking during learning. Second, while those in the AS condition outperformed those in the C condition on page-level quizzes, further investigation is still needed as to why learners in both conditions performed equally poorly on subgoal quizzes. Why is the ERL provided by PAs in the AS condition not leading to significantly better subgoal quiz scores? Third, it is evident that some cognitive strategies, such as making inferences, are too sophisticated for low prior knowledge learners who need to spend time reading to acquire knowledge about the topic and therefore should only be prompted by PAs once they have demonstrated a certain level of content understanding. Fourth, it is also evident that several metacognitive processes such as FOKs and MPTGs are seldom used during a learning task and therefore may not need to be prompted and scaffolded as often as other key metacognitive processes. On the other hand, this may also reveal low SRL prior knowledge.

Lastly, SRL and ERL between human and artificial agents is a core issue in the ITS community [16]. Contemporary research on ITSs with multiple agents has focused on SRL while relatively little effort has been made to use externally regulated learning as a guiding theoretical framework [7–9, 15]. This oversight needs to be addressed given the complex nature that self- and other-regulatory processes play when human learners and artificial agents interact to support learners’ internalization of SRL processes. For example, learning with MetaTutor involves having a learner interact with four artificial PAs. Each agent plays different roles including modeling, prompting, and scaffolding SRL processes (e.g., planning, monitoring, and strategy use) and providing feedback regarding the appropriateness and accuracy of learners’ use of SRL processes in real time and potentially changing the ERL strategies based on its ability to monitor and reflect on the impact on the learners’ individual responses to ERL. For example, the external regulating agent may have to modify its cognitive and metacognitive scaffolding at some point during learning and include affect regulation strategies (e.g., cognitive reappraisal) due to its perception, understanding, and reflection that its scaffolding and feedback is resulting in increasingly negative affective reactions (e.g., frustration) from a learner. Lastly, our goal is to build intelligent artificial agents capable of ERL by detecting, tracking, modeling, and fostering learners’ cognitive, affective, metacognitive, and motivational (CAMM) SRL. By doing so, we will extend the human and computerized theoretical models typically used in this research area and therefore revolutionize the field of ITS by having interdisciplinary researchers address conceptual, theoretical, methodological, and analytical issues.