Abstract
Researches in areas such as neuroscience and psychology indicate that emotions directly impact learning. So, adapting to the learners’ affective reactions became a requirement and also a challenge for building a new generation of affect aware computing learning environments. In this paper, we present a hybrid approach for inferring learning related emotion that combines cognitive and physical data, gathered using minimal or non intrusive methods. In an initial experiment with students in a real education environment it was possible to obtain promising results when comparing some usual performance metrics with correlated works. In this study we achieved accuracy rates and Cohen’s Kappa near to 65% and 0.55, respectively. Furthermore, considering the open and expansible nature of this proposal, we believe that this results could be improved in the future by adding new data or new sensors to the model, for example.
Access provided by CONRICYT-eBooks. Download conference paper PDF
Similar content being viewed by others
Keywords
1 Introduction
One of the main limitations currently presented by educational software like Intelligent Tutoring System (ITS) is the lack of features to adapt to students’ emotional states [1, 5]. Much of nowadays ITS pay little or no attention to the emotional experience of students and because of this do not reach the full level of interactivity as humans tutors can do [3]. This limitation becomes relevant considering the inextricable relationship between emotions and learning [12].
Advances in the area known as Affective Computing have motivated a growing body of research in an effort to develop systems that could recognize and adapt to students’ affective states [7]. Despite these advances, accuracy of inferred emotion are not yet sufficient to be used as the bases for adaptation, particularly in real classroom learning environments [3]. There is also the practical challenge of deploying some of the physical affective sensors [1, 3].
In this context, this paper presents a proposal and implementation of a Hybrid Inference Model of Learning Related Emotions - ModHEmo. This proposal stand out by presenting an approach that combines physical and cognitive data, gathered with minimally or non intrusive sensors that could be easy deployed out-of-the-lab. This approach is grounded on the fact that emotions in humans are strongly related with some physical reactions, but also include a rational and cognitive process [12].
2 Hybrid Emotion Inference Model
As this proposal focus on educational software, we need to deal with emotion that impact and correlate with learning outcomes. Priors works [2,3,4, 8, 11] show that there is no consensus about a specific set of learning related emotion. So, to choose the set of emotions to infer we used as conceptual foundations the work of [14]. This work is based on two consolidated theory: the ‘circumplex model’ [13] and the ‘spiral learning model’ [9]. These theories define a two dimensional space in which the emotions of a student could dynamically move during learning. Based on these references, we decide to implement the inference process considering not a specific set of emotions, but grouping them into quadrants.
Figure 1 shows the approach used in this work to represent the learning related emotions in a two dimensional space. In this proposal the ‘Valence’(horizontal axis) and ‘Arousal’(vertical axis) dimensions are used to position the emotions in the “Q1”, “Q2”, “Q3” and “Q4” quadrants, more a “Neutral” state named “QN”. These quadrants played the role of classes in the classification processes performed by the ModHEmo that will be described in the next section.
The Fig. 2 schematically presents ModHEmo where the inference process is divided into two main components: physical and cognitive. The final inference of probable students’ affective state is made by combining the information of these two components.
The physical component inference is based on the classical Ekman’s model [6] and include the eight basic emotions, which are: anger, disgust, fear, happy, sadness, surprise, contempt and neutral. In the cognitive component were considered the eight emotions of the Ortony, Clore and Collins - OCC theory [10] that have impact on the agent itself (e.g. student) and that are triggered as valenced (positive or negative) reaction in response to events. The eight cognitive emotion are: joy, distress, disappointment, relief, hope, fear, satisfaction and fears confirmed.
The cognitive component, based on OCC theory, is responsible for handling relevant events in the computing environment. The physical component deal with observable reactions, using students’ face images gathered using a standard Webcam, after the occurrence of a relevant event. These two components returns scores for each emotions of the physical and cognitive components. Further, each one of the components perform a mapping of the emotions’ scores for the respective quadrants (see Fig. 1 based on the values of the valence and arousal dimensions.
The final fusion process is accomplished by creating a single dataset containing quadrants scores of each component. After the fusion, this dataset contains 10 attributes (5 physical + 5 cognitive) with scores of quadrants (plus Neutral) for each component. The Class obtained through the labeling process (to be described in the next section) is also part of the dataset. Based on this dataset, we train and test two classification algorithms which perform the inference of the final result of the model.
3 Experiment Design and Results
To check the ModHEmo’s performance in a real educational environment, an experiment was conducted with 15 elementary students (ages between 11 and 15 years). In the experiment, the students used a customized version of the educational software ‘Tux, of Math Command’ or TuxMathFootnote 1. TuxMath is an educational game that allows kids to exercise their mathematical reasoning.
While students used TuxMath, some of the main events of the game were monitored. When a monitored event occurs, an image of the students’ face was captured and used as the input to physical component of ModHEmo. The kind of event serves as input for cognitive component.
After completing the game, students labeled the events using a customized tool in order to build a ground truth dataset. This tool allows the student to review the game section, synchronized with a video that shows their facial reactions captured by the webcam. The tool automatically stop the video when some monitored event has occurred and ask students to select a quadrant (represented by emoticons) that best describe their affective state at that moment.
A ground truth with 935 instances of monitored events was created and the classes distribution was 141, 188, 173, 130 and 303 for Q1, Q2, Q3, Q4 and QN, respectively. These dataset was used for training and testing (10-fold cross validation) the classification algorithms RandomForest and IBK using WekaFootnote 2. These algorithms were chosen due their simplicity and performance. The algorithm RandomForest achieve accuracy rate of 64.81% and Cohen’s Kappa of 0.545. The IBK accuracy rate was 63.53% and Cohen’s Kappa 0.532.
Another useful tool to analyze classifiers performance are ROC (Receiver Operating Characteristic) curves that depict the performance of a classifier without regard to class distribution or error costs. The Fig. 3 depict the ROC curves for the five classes obtained by the RandomForest algorithm (the curves for IBK are very similar). This figure also show the Area Under Curve (AUC) computed in Weka.
In this work, we considered the premise that the combination of the physical and cognitive components could be an effective approach to improve the inference results. Thus, tests were performed to verify the impact in the inference process of using each of the components individually.
To perform this test we created two datasets: one with physical and other with cognitive attributes only. Using only cognitive attributes, the accuracy was respectively 39.25% and 40% for RandomForest and IBK. With only physical attributes the accuracy as respectively 55.29% and 52.19% for RandomForest and IBK. This result indicates that the combination of the two ModHEmo’s components was important for improving the inference accuracy.
The results achieved in the experiment with ModHEmo presented above show some improvements when compared with [2, 3, 11]. In our experiment, Cohen’s Kappa index was 0.545 and 0.532 for RandomForest and IBK, respectively and AUC value was between 0.843 and 0.888 (see Fig. 3).
4 Final Considerations, Limitations and Future Works
Inferences obtained with this model could be very useful for implementing learning environments or ITS able to appropriately recognize and respond to learners’ emotional reactions. The model described in this paper stand out by presenting a method to combine quite distinct information (physical and cognitive) that is little explored in the research community nowadays.
Even considering some limitations, the initial results obtained can be considered promising, since the results obtained are similar or superior to the state of the art. Furthermore, we believe that our hybrid approach resembles the natural process of emotions inference, thus presenting promising opportunities for future improvements by adding new data or sensors.
As future work wed intend to expand the current experiment involving more students with other age groups and also other types of educational environments. It is also intended to evaluate the result of the adding new information in the physical and cognitive components.
References
Baker, R.S., Gowda, S., Wixon, M., Kalka, J., Wagner, A., Salvi, A., Aleven, V., Kusbit, G., Ocumpaugh, J., Rossi, L.: Sensor-free automated detection of affect in a cognitive tutor for algebra. In: Educational Data Mining 2012 (2012)
Bosch, N., Chen, Y., D’Mello, S.: It’s written on your face: detecting affective states from facial expressions while learning computer programming. In: Trausan-Matu, S., Boyer, K.E., Crosby, M., Panourgia, K. (eds.) ITS 2014. LNCS, vol. 8474, pp. 39–44. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-07221-0_5
Botelho, A.F., Baker, R.S., Heffernan, N.T.: Improving sensor-free affect detection using deep learning. In: André, E., Baker, R., Hu, X., Rodrigo, M.M.T., du Boulay, B. (eds.) AIED 2017. LNCS (LNAI), vol. 10331, pp. 40–51. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-61425-0_4
Conati, C.: Combining cognitive appraisal and sensors for affect detection in a framework for modeling user affect. In: Calvo, R.A., D’Mello, S. (eds.) New Perspectives on Affect and Learning Technologies. LSIS, vol. 3, pp. 71–84. Springer, New York (2011). https://doi.org/10.1007/978-1-4419-9625-1_6
D’Mello, S., Lehman, B., Sullins, J., Daigle, R., Combs, R., Vogt, K., Perkins, L., Graesser, A.: A time for emoting: when affect-sensitivity is and isn’t effective at promoting deep learning. In: Aleven, V., Kay, J., Mostow, J. (eds.) ITS 2010. LNCS, vol. 6094, pp. 245–254. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-13388-6_29
Ekman, P.: An argument for basic emotions. Cogn. Emot. 6(3–4), 169–200 (1992)
Grafsgaard, J.F., Wiggins, J.B., Boyer, K.E., Wiebe, E.N., Lester, J.C.: Embodied affect in tutorial dialogue: student gesture and posture. In: Lane, H.C., Yacef, K., Mostow, J., Pavlik, P. (eds.) AIED 2013. LNCS (LNAI), vol. 7926, pp. 1–10. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-39112-5_1
Jaques, N., Conati, C., Harley, J.M., Azevedo, R.: Predicting affect from gaze data during interaction with an intelligent tutoring system. In: Trausan-Matu, S., Boyer, K.E., Crosby, M., Panourgia, K. (eds.) ITS 2014. LNCS, vol. 8474, pp. 29–38. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-07221-0_4
Kort, B., Reilly, R., Picard, R.W.: An affective model of interplay between emotions and learning: reengineering educational pedagogy-building a learning companion. In: 2001 Proceedings of the IEEE International Conference on Advanced Learning Technologies, pp. 43–46. IEEE (2001)
Ortony, A., Clore, G.L., Collins, A.: The Cognitive Structure of Emotions. Cambridge University Press, Cambridge (1990)
Paquette, L., Baker, R.S.J.D., Sao Pedro, M.A., Gobert, J.D., Rossi, L., Nakama, A., Kauffman-Rogoff, Z.: Sensor-free affect detection for a simulation-based science inquiry learning environment. In: Trausan-Matu, S., Boyer, K.E., Crosby, M., Panourgia, K. (eds.) ITS 2014. LNCS, vol. 8474, pp. 1–10. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-07221-0_1
Picard, R.W., Papert, S., Bender, W., Blumberg, B., Breazeal, C., Cavallo, D., Machover, T., Resnick, M., Roy, D., Strohecker, C.: Affective learning - a manifesto. BT Technol. J. 22(4), 253–269 (2004)
Russel, J.A.: A circumplex model of affect. J. Pers. Soc. Psychol. 39, 1161–1178 (1980)
Shen, L., Leon, E., Callaghan, V., Shen, R.: Exploratory research on an affective e-learning model. In: Proceedings of Workshop on Blended Learning, pp. 267–278 (2007)
Acknowledgments
Authors would like to thank the IFRS and UFPR for financial support of this work.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Gottardo, E., Ricardo Pimentel, A. (2018). Improving Inference of Learning Related Emotion by Combining Cognitive and Physical Information. In: Nkambou, R., Azevedo, R., Vassileva, J. (eds) Intelligent Tutoring Systems. ITS 2018. Lecture Notes in Computer Science(), vol 10858. Springer, Cham. https://doi.org/10.1007/978-3-319-91464-0_33
Download citation
DOI: https://doi.org/10.1007/978-3-319-91464-0_33
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-91463-3
Online ISBN: 978-3-319-91464-0
eBook Packages: Computer ScienceComputer Science (R0)