Keywords

1 Introduction

Research has shown that affect and metacognition play a significant role in learning. When students accomplish learning goals, they are likely to experience joy [1], while negative emotions during learning, such as frustration and confusion, can lead to disengagement with the learning material and prevent effective learning [2, 3]. To enable advanced learning technologies (ALTs) to effectively interact with students, it is important to allow ALTs to take actions to address students’ affective states, and understand the relationship between these affective states and students’ metacognitive monitoring processes [4, 5]. Developing an understanding of the relationship between students’ affect and their cognitive and metacognitive self-regulated learning (SRL) processes can contribute to the design of practical, scalable ALTs [6, 8]. This work moves toward affect-aware ALTs by using automated affect detection through facial expression recognition of emotion. Automatic affect detection has been an area of active research and builds on theoretical frameworks such as the Facial Action Coding Scheme [7] and machine learning-based affect detectors [e.g. 3, 5, 9]. While automatic affect detection has been used to accurately predict learning outcomes using lower-level action units [e.g., 10], previous research using automatically detected affect has not considered the metacognitive processes that are also influenced by student affect and integral to the self-regulated learning processes that can influence students’ performance during learning with ALTs.

2 MetaTutorIVH Study

A total of 66 students enrolled in a mid-Atlantic North American University participated in this study. Data from 12 students were removed due to calibration issues, resulting in 54 students (72% female). Students’ ages ranged from 18 to 29 (M = 20.5, SD = 2.34). An 18-item, multiple-choice question pre-test assessing prior knowledge of the biology concepts covered during learning with MetaTutorIVH indicated students had low to moderate (questions correct ranging from 6 to 14) prior knowledge (M = 11.0 [61.1%], SD = 1.46 [8.1%]).

2.1 MetaTutorIVH

MetaTutorIVH is an ALT with which students learn human biology concepts through text and diagrams while making metacognitive judgments, answering multiple-choice questions, and observing a virtual human. Students interacted with MetaTutorIVH over the course of 18 counter-balanced, randomized, self-paced trials that consisted of a complex biology question, metacognitive judgment prompts, a virtual human, and science content presented in text and diagrams. For each trial, a student was first presented with a science question and then performed an ease of learning metacognitive judgment. Then students were presented with the content page (Fig. 1) containing the science text and diagram, as well as the virtual human. Students decided when to progress from the content page to the 4-foil multiple-choice question, which was then followed by a retrospective confidence judgment (RCJ), where students evaluated their multiple-choice answer confidence. Each trial concluded with the students providing a justification and RCJ for their justification.

Fig. 1.
figure 1

Screenshot of MetaTutorIVH content page, the main interface containing the science question, text, diagram, and intelligent virtual human (IVH).

Full size image

Facial expression features were extracted automatically from a facial expression recognition system, FACET [13]. FACET extracts facial measurements from video streams that correspond to the Facial Action Coding System [11]. A discretization process filtered out subject and measurement variance to provide conservative estimates of emotion events that are stable across students. Once the evidence scores were converted to discrete events, sequences of emotion were created for each student. Students engaged in complex learning processes on the content page (Fig. 1), thus only emotion events that occur during the content page were considered. We examined page-level sequences each of which are a sequence of emotions produced by a student on a single content page, which have short sequence lengths (M = 5.97, SD = 6.47) due to the brief time students spent per content page (M = 100.5 s, SD = 47.3).

3 Results

The observed rates of occurrence for each emotion during student interaction with the content page indicate that joy (M = 0.67, SD = 0.61) and frustration (M = 0.66, SD = 0.60) were the most frequently occurring emotions, while contempt (M = 0.36, SD = 0.42) was the least frequently occurring emotion.

We calculate the likelihood metric for transitions, calculated similarly to Cohen’s Kappa (see [6] for additional details), of key learner-centric transitions averaged over page-level sequences. The likelihood of transitions between confusion and frustration were both significantly above 0 (Confusion to Frustration Average Likelihood = 0.40, SD = 0.30; Frustration to Confusion Average Likelihood = 0.19, SD = 0.20), which is not surprising considering the strong correlation between these emotions (r(54) = 0.70, p < 0.001). However, the transitions from confusion to frustration have a significantly higher likelihood measured across the 54 students than transitions from frustration to confusion (t(53) = 5.89, p < 0.001), indicating that while correlated, confusion was seen more often to precede frustration than frustration preceding confusion during learning.

Proportional to the frequencies observed on the student level, joy was observed in 51.4% of trials, frustration in 45.3%, confusion in 35.4% and a transition from confusion to frustration in 21.2% of trials. A mixed effects logistic regression model performed in R with the lme4 package [12] predicting multiple choice correctness using the presence of confusion, frustration, joy, and transition from confusion to frustration as fixed effects and random intercepts for students found no significant predictors among the fixed effects from a likelihood test against a null model using only the random intercepts for students (χ2(4) = 3.73, p = 0.44). This mixed effect model indicates there was no effect of the presence of these emotions and transition from confusion to frustration on multiple-choice performance after accounting for individual differences.

The relationship between presence of learner-centric emotions and RCJs was examined through a linear mixed effect model using fixed effects for the presence of confusion, frustration, joy, and transition between confusion to frustration and random intercepts for each test subject. A significant impact of the fixed effects of emotions was found through a nested F-test using the full model as the linear mixed effects model and reduced model being a random effects model with intercepts for students (F(4, 915) = 2.45, p = 0.045, R2 = 0.36). The fixed effects are reported in Table 1 and indicate that the presence of confusion and joy have a positive impact on student RCJs, specifically multiple-choice confidence, while the presence of frustration and a transition from confusion to frustration have a negative impact.

Table 1. Linear mixed effect model for predicting multiple-choice confidence
Full size table

Additionally, to assess the relationship between multiple choice performance (a binary measure of correctness) with multiple choice confidence (i.e., an RCJ) a Welch’s two sample t-test accounting for unequal variance among groups indicated the confidence levels of students was significantly greater (t(842) = 9.1, p < 0.001, Cohen’s d = 0.60) in trials where the multiple-choice question was answered correctly (M = 84.0, SD = 15.5) than when students answered incorrectly (M = 74.5, SD = 16.2).

4 Conclusion

Using an automatic affect detection system embedded in MetaTutorIVH, we conducted an analysis of learner-centered emotions and their influence on students’ learning and RCJs. Joy and frustration were found to be the most frequently occurring emotions when examining the absolute frequency of discrete emotions. Analysis of the affective dynamics revealed transitions between confusion and frustration to be significantly more likely than chance, with transitions specifically from confusion to frustration being especially prominent. The presence of learner-centered emotions (joy, confusion, frustration), and transitions from confusion to frustration during complex learning did not reveal any effect of learner-centered emotions on learning. Additional analyses revealed positive effects of confusion and joy on RCJs and negative effects of frustration and transitions from confusion to frustration. These results can inform the design of ALTs that assist learners in both cognitive and metacognitive processes through monitoring and intervening based on their affective expressions.