emoLearnAdapt: A new approach for an emotion-based adaptation in e-learning environments

Abstract

In e-learning environments, most adaptive systems do not consider the learner's emotional state when recommending activities for learning difficulties, blockages, or demotivation. In this paper, we propose a new approach of emotion-based adaptation in e-learning environments. The system will allow recommendation resources/activities to motivate and support the learner in learning. Our first contribution is modeling the learner's emotion by exploiting the facial expressions generated during the pedagogical activities. For this purpose, a probability-based emotion quantification algorithm has been proposed. To recommend support resources, we presented our adaptation approach that leverages a set of adopted adaptation criteria, where the weighting of these criteria differs from one support resource to another. Five experiments aimed at validating the approach were conducted on two groups of students (test and control groups). The results show our approach's impact on improving the learner's cognitive level, engagement time, and motivation.

1 Introduction

Computers are increasingly used in education. The term "Computer Environment for Human Learning" (CEHL), which appeared in the early 2000s to describe a "computer environment designed to support human learning, i.e., the construction of knowledge in a learner," and was coined as a result of these advances (Tchounikine, 2002). The needs of each learner can be met, and they can always access updated content through these environments.

With CEHL, it is possible to present adaptive content and activities and consider the knowledge learners have acquired during the learning process. This brings us to adaptive learning systems, which is of more universal interest (Mustafa & Sharif, 2011). These systems aim to enhance the individual learning process (De Bra et al., 2004; Henze & Nejdl, 2004; Weber, 1999) and provide the conditions for all learners to succeed. The adaptation of resources and activities has been proposed as a solution to the problem of the diversity of learners' levels and skills; it is mainly based on the learner model, which contains a lot of information about the learner's cognitive and behavioral state (Drissi & Amirat, 2016).

However, the majority of these systems do not take into account the emotional state of the learner. Emotion can also be an essential element in determining the quality of the learning process and the achievement of desired learning outcomes, according to Goetz et al. (2013). In the face-to-face mode (face-to-face), the teacher can adapt their teaching strategy by observing learners' affective states, i.e., their emotions, facial expressions, speech emotions, and body movements (Kerkeni et al., 2017). Furthermore, studies have shown that positive emotions can promote better motivation by creating positive learning experiences; on the other hand, negative emotions create a negative psychological state in learners, disrupting the learning process and reducing the academic performance (demotivation).

This importance of emotion is reinforced especially in online learning environments, where they can also have negative impacts such as isolation caused by the physical and temporal separation of the learner and the teacher, difficulties in encouraging learners, frustration, and doubt. These elements will decrease the learner's desire to learn, demotivate them, and negatively impact their emotional (psychological) state during the learning session. Thus, interaction with learners is less effective than in traditional learning environments.

In learning environments, therefore, a reliable method of emotion recognition is needed (Burić et al., 2016). The latter is a challenging task, especially since it requires recognizing complex academic emotions such as pleasure in learning, frustration, anger at difficult tasks, boredom experienced during learning, etc. Therefore, a wide range of sources of emotional information has been employed in the literature, such as questionnaires, texts, speech, facial expressions, body gestures, and even from physiological sensors such as EEG, ECG, EDA, etc.

To this end, facial expression information is often used in automatic emotion recognition systems (Mao et al., 2015). Emotion recognition from facial expressions can be inspired by classroom learning. The teacher analyzes the learner's emotions, especially from the learner's face. Data sensors from this source (camera) are available to all learners learning either from a computer or mobile (m-learning), unlike other sensors (Kouahla et al., 2022).

In this paper, we consider the emotional state recognized from the learners' facial expressions when proposing pedagogical activities to the learners or recommending activities to be undertaken in the case of a learning difficulty or a blockage, whatever their origin. Also, the ethical aspect must be carefully considered when developing and implementing emotion-based learning systems.

Several research questions can therefore be asked: How can we adapt resources and activities based on his emotional characteristics in such a way that it can unlock him and help him in his learning process? What are the adequate adaptation criteria? How can we model the learner's emotion from his facial expressions? What is the nature of the support resources to be adapted? Do all adaptation criteria have the same importance for these resources? To answer these questions, we formulate the following hypotheses:

• Hypothesis 1: “Improvement in learners’ cognitive profile is achieved through the use of the support resource adaptation tool”.

• Hypothesis 2: Improvement in learners’ emotional profile is achieved through the use of the support resource adaptation tool.

This work has integrated the proposed method into a learning environment called emoLearnAdapt (Emotion Learning Adaptation). This work has three main contributions: 1) The modeling of the learner's emotions recognized from his facial expressions; 2) The proposition of a learner's model that considers the emotional information, the cognitive level, behavioral skills, activity preferences, and learning style. 3) The proposition of a support resource adaptation algorithm based on a set of adaptation criteria extracted from the learner's model.

This system must consider the following services: a main module dedicated to adapting support resources; a recommendation generator will recommend these. A module dedicated to the recognition and modeling of the learner's emotions. Module for the management of different learner profiles. Finally, a module for the management of support resources. The rest of the paper is organized as follows: Section 2 gives an overview of related work; Section 3 defines the research objectives and hypotheses that guide our work; Section 4 describes the proposed learner's emotion modeling; Section 5 discusses our design of the emotion-based adaptation system, while Section 6 describes the experimental study and presents the results obtained; Section 7 concludes the paper and highlights the prospects for future work.

2 Related works

This section provides a comprehensive overview of various studies focusing on adaptive learning environments that consider the learner’s emotional aspect. We begin by setting the context and discussing the importance of emotion in learning environments. We then delve into specific research studies, discussing their methodologies, objectives, and features. Finally, we must discuss previous studies that have addressed ethical issues in emotion-based learning systems.

2.1 Importance of emotion in learning environments

Humans inherently exhibit emotions in a variety of events, interaction contexts, and social contexts throughout their daily lives. They can be expressed through facial expressions, body language, speech, or others. There is no universal agreement on what constitutes an emotion (Cherry, 2012; James, 1884). For some authors, emotion is a fleeting feeling accompanied by high brain activity and a high degree of pleasure or hatred (Cabanac, 2002; Kerkeni et al., 2017; Schacter, 2012). “Emotion is often characterized in psychology as a complicated state of feeling that results in psychological and bodily changes.” These changes impact the way people think and act.”(Cherry, 2012; Kerkeni et al., 2017). Emotions are characterized in many ways depending on the context, but they all have one thing in common: they are temporal (unlike mood) and involve mental activity.

Emotions have an impact on learning at several levels according to Chris Drew (Drew, 2021). They impact learner motivation (motivational impact). Positive emotions can help a learner engage longer in learning because they are motivated. The emotions felt during learning also have an impact on the learner’s feelings towards education (psychological impact). If we have positive experiences, we are more likely to appreciate our schooling and develop a love of learning. Emotions can also facilitate group work (social impact). However, we must keep in mind that learning sometimes requires confusion and frustration when we learn difficult but important concepts (cognitive impact).

• 1. Psychological Impact: Positive emotions allow the learner to better understand learning. Some cognitive psychologists believe that learners who have a positive emotional, behavioral, cognitive attitude towards education also feel like they are in control of their own learning, which encourages them to put in more effort. This is an upward spiral. Learners who put in more effort feel the positive and additional results, which pushes them to make even more effort. These learners have developed what Carol Dweck (Dweck, 2017) calls a “growth mindset” towards education.

• 2. Motivational Impact: Positive emotions make the learner more motivated. Positive emotions enhance learner motivation, leading to increased engagement and long-term learning benefits. Learners with positive emotions need fewer external incentives, as they develop intrinsic motivation. Conversely, learners with negative emotions rely on extrinsic motivators, which can lead to poorer long-term outcomes. Thus, positive emotions foster a desire to learn (Drew, 2021).

• 3. Social Impact: Positive emotions improve the cohesion of learning groups. Positive emotions enhance the social cohesion of learning groups. Learners who feel good about their learning are more likely to engage with teachers and peers, contributing to group discussions. However, learners with negative school experiences may become isolated and less engaged. Those who derive positive emotions from school are more likely to actively participate and communicate with their peers (Drew, 2021).

• 4. Cognitive Impact: Negative emotions are necessary for learning difficult concepts. Negative emotions are essential for learning complex concepts. Learning involves a cycle of emotions, as described by Kort’s emotional learning spiral (Kort et al., 2001). Learners start with positive feelings about learning a new concept (stage 1), but face confusion and anxiety when confronted with difficult information (stage 2). Correcting outdated knowledge leads to frustration (stage 3), but is a necessary part of cognitive development. Finally, learners experience determination and hope as they understand the new concept (stage 4). Despite the presence of negative emotions like confusion and frustration, these stages are crucial for learning and lead to satisfaction from overcoming challenges (Drew, 2021).

2.2 Different emotion-based adaptive environments

Affective Autotutor (D’mello & Graesser, 2013) is an ITS that responds with feedback (positive or negative) about the learner's current response in the form of predefined sentences, and a corresponding facial and vocal emotional expression of the tutor agent accompanies each feedback. Forbes-Riley and Litman (2012) proposes ITSPOKE (Intelligent Tutoring SPOKEn dialog system), which adapts in real-time to learners' emotions (detached from the speech of their responses), such as disengagement and uncertainty. The system recommends feedback in the form of messages and also tasks to complete. Shen et al. (2009) propose to recommend resources (pedagogical contents, examples, music and videos) to learners according to the recognized emotional state, its evolution and the learning objectives, while exploiting a set of recommendation rules. For emotion recognition, they integrate a multimodal emotion recognition system by combining data from several physiological sensors (EDA, EEG, EVP, heart rate). In the paper (Kouahla et al., 2022), the learning system generates psychological (relaxation exercise and tutor contact) and pedagogical (resources of different learning styles) recommendations according to the emotional states of the learner. For this, emotion is recognized by combining the results of two modalities (facial expressions and speech).

Karampiperis et al. (2014) propose to recommend pedagogical resources by collaborative filtering based on learners' evaluations of the contents of these resources computed from the comments left by these learners. They use sentiment analysis techniques based on the texts. Ezaldeen et al. (2022) propose a novel framework for e-learning recommendation that uses adaptive profiling and sentiment analysis (based on CNN and NLP techniques) to provide personalized e-content for learners. This method builds the semantic learner profile based on learner behavior, term expansion, and semantic similarity using ontologies. In the paper (Chanaa & El Faddouli, 2022), the goal is to design a new recommender system for MOOCs that considers various aspects of learners’ emotions and behaviors. They analyse and extract useful information from the data. Also, they propose a novel technique called UGPFM to predict the rating of learning objects based on the learner, the learning object and the three contexts (sentiment, cognition and confusion). Vedavathi and Anil Kumar (2023) offer individualized and pertinent course suggestions to users by analyzing their preferences, ratings, and sentiments expressed in online content. They employ the ITF-IDF method for extracting semantic features from text data, and utilize ERNN for sentiment classification. Recommendations will be generated based on the similarity between user preferences and course attributes.

2.3 Synthesis and comparative analysis of emotion-based adaptive learning approaches

Table 1 presents a comparative analysis of our emotion-based adaptive learning approach and the other studies, highlighting differences and similarities in methodology, objectives, source of emotional information, emotion model, recommendation technique, and suggested recommendations.

Table 1 Comparison of emotion-based adaptation approaches in learning environments and ours

We organize our discussion around many points: types of learning environments, emotional sources, modeling emotions, divergence of challenges addressed and nature of recommendations. This thematic organization allows for a more comparative analysis between the different emotion-based adaptive learning approaches.

Types of learning environments

At a glance, we can reveal the diversity of learning environments. They can be classified into game-based ITS intelligent tutorial system (D’mello & Graesser, 2013; Salmeron-Majadas et al., 2015), social learning environments (Karampiperis et al., 2014; Vedavathi & Anil Kumar, 2023), MOOC (Chanaa & El Faddouli, 2022) and unspecified learning environments (Kouahla et al., 2022; Rodriguez et al., 2014; Santos et al., 2014; Shen et al., 2009). The majority of these works have explained the architecture, basic concepts, and problems encountered in emotion-based adaptation, despite that, only a few works (D’mello & Graesser, 2013; Karampiperis et al., 2014; Kouahla et al., 2022) have implemented the approach. Only (Karampiperis et al., 2014; Kouahla et al., 2022) have described the recommendation strategy in detail.

Emotional sources

The sources of emotional information are diverse: 1) cameras for facial expression recognition (D’mello & Graesser, 2013; Grawemeyer et al., 2015; Kouahla et al., 2022; Santos et al., 2014) and body gestures (D’mello & Graesser, 2013); 2) speech features (Grawemeyer et al., 2015; Kouahla et al., 2022; Litman & Forbes-Riley, 2014); 3) texts (Chanaa & El Faddouli, 2022; Ezaldeen et al., 2022; Karampiperis et al., 2014; Rodriguez et al., 2014; Vedavathi & Anil Kumar, 2023), questionnaires (Santos et al., 2014), and log files (D’mello & Graesser, 2013); 4) behavioral information such as mouse movements and keystrokes (D’mello & Graesser, 2013); 5) and physiological signals such as EDA, EVP, EEG, and heart rate (Shen et al., 2009). The sources that researchers have exploited in their studies are either a single source (Karampiperis et al., 2014; Litman & Forbes-Riley, 2014; Rodriguez et al., 2014) or combinations of multiple sources (D’mello & Graesser, 2013; Grawemeyer et al., 2015; Kouahla et al., 2022; Santos et al., 2014; Shen et al., 2009).

Modeling emotions

Several researchers (D’mello & Graesser, 2013; Karampiperis et al., 2014; Kouahla et al., 2022); train their emotion recognition model that automatically predicts emotion. However, the remaining works (Litman & Forbes-Riley, 2014; Rodriguez et al., 2014) did not specify how they performed emotional labeling. In most cases, researchers modeled emotion by predefined categories (categorical modeling), where the most used are confusion, frustration, joy, boredom, anxiety, surprise.

Divergence of challenges addressed

Regarding the nature of emotion-based recommendations found, they are pretty diverse. Some focus on emotional support, which is often delivered in the form of feedback containing advice (Grawemeyer et al., 2015; Litman & Forbes-Riley, 2014; Santos et al., 2014), instructions (Grawemeyer et al., 2015), motivational messages (Grawemeyer et al., 2015; Litman & Forbes-Riley, 2014) or relaxation exercise (Kouahla et al., 2022); sometimes it is accompanied by facial and vocal expressions generated by an agent (D’mello & Graesser, 2013).

Nature of recommendations

Shen et al. (2009) support learners by delivering resources such as music and videos. Others focus on the recommendation of pedagogical content, which can include learning resources (Ezaldeen et al., 2022; Karampiperis et al., 2014; Kouahla et al., 2022; Shen et al., 2009), learning activities (Litman & Forbes-Riley, 2014; Rodriguez et al., 2014), courses (Vedavathi & Anil Kumar, 2023), Learning objects (Chanaa & El Faddouli, 2022), use cases (Shen et al., 2009), etc. Others propose to suggest learners, teachers (Leony et al., 2013), and tutors (Kouahla et al., 2022). It is beneficial to receive help from someone with the necessary expertise.

2.4 Ethical considerations in emotion-based learning systems

The integration of emotion recognition technologies in eLearning environments has opened up new avenues for personalized learning experiences. However, these advancements also bring forth a set of ethical considerations that need to be addressed:

Privacy and consent

Emotion recognition technologies involve continuous monitoring of learners, potentially infringing on their privacy (Akgun & Greenhow, 2022; Katirai, 2023). This data, which can range from text inputs to physiological signals, is highly personal and sensitive (Akgun & Greenhow, 2022; Katirai, 2023). Therefore, it’s crucial to obtain informed consent from users before collecting and analyzing their data.

In particular, facial data used for emotion recognition is subject to additional ethical considerations. Continuous monitoring and analysis of facial expressions could be perceived as an invasion of privacy (Almeida et al., 2022; Tsamados et al., 2021). Users may feel uncomfortable knowing that their every facial movement is being watched and analyzed. Hence, special attention must be given to ensure the ethical handling of facial data in emotion-based learning systems (Akgun & Greenhow, 2022; Zieher et al., 2021).

Data security

The data used by these technologies, which can range from text inputs to physiological signals, is highly sensitive as it involves personal emotions (Katirai, 2023; Kolakowska, 2013). Therefore, it’s important to ensure that this data is stored securely and not susceptible to breaches (Chernyshev et al., 2018). This is particularly crucial when dealing with facial data, which is subject to additional ethical considerations (Katsanis et al., 2021; Lee & Chin, 2022). The privacy concerns of face recognition users originate from the risk of facial image information being collected and used without personal knowledge or consent or the risk of personal biometrics being transmitted or leaked (Liu et al., 2021).

Despite the rapid advancement and widespread application of different emotion-based adaptive environments, the review of the literature (describe in Section 2.2) reveals a significant gap: there is a conspicuous absence of research addressing the ethical implications of these technologies.

3 Research objectives and hypotheses

The main aim of this work is to propose and evaluate an emotion-based adaptation approach for learning environments, which can enhance the learner’s cognitive and emotional outcomes by adapting the support resources and activities based on the learner’s emotional state. To achieve this aim, we have defined the following research objectives and hypotheses:

• Research Objective 1: To model the learner’s emotion by proposing an emotion quantification algorithm. This objective involves designing and implementing an algorithm that can recognize and quantify the learner’s emotional state from their facial expressions, using a webcam as the input device. The algorithm exploits a facial expression model that classifies the learner’s emotion into one of the six basic emotions: happiness, sadness, anger, fear, surprise, and disgust. Based on this classification, the algorithm proposes a modelisation that assigns a numerical value to the intensity of the emotion, ranging from 0 to 1, where 0 means no emotion and 1 means the highest intensity of the emotion.

• Research Objective 2: To propose an adaptation algorithm based on a set of criteria in a learning environment. This objective involves designing and implementing an algorithm that can adapt the support resources and activities in the learning environment based on the learner’s model, which includes the emotional information, the cognitive level, behavioral profile, activity preferences, and learning style. The algorithm should be able to select the most appropriate and effective support resources and activities for each learner, based on a set of adaptation criteria that are derived from the learner’s model. The algorithm should also be able to generate recommendations for the learner, suggesting the best actions to take in case of a learning difficulty or a blockage.

Based on these research objectives, we formulate the following hypotheses, which reflect our expectations and predictions about the outcomes of our proposed approach:

• Hypothesis 1: “Improvement in learners’ cognitive profile is achieved through the use of the support resource adaptation tool”.“Improvement in learners’ cognitive profile is achieved through the use of the support resource adaptation tool”.

• This hypothesis aligns with existing research that highlights the potential of adaptive learning tools in enhancing cognitive outcomes (Martin et al., 2020). Adaptive learning platforms have shown mixed results in improving learning outcomes, with certain groups of students benefiting more than others. The system takes this a step further by incorporating emotion-based adaptation, which could potentially lead to more effective personalization and improved cognitive outcomes for learners (Imhof et al., 2020; Taylor et al., 2021).

• Hypothesis 2: “Improvement in learners’ emotional profile is achieved through the use of the support resource adaptation tool”. This hypothesis directly relates to the unique feature of the system. Existing literature has suggested a link between emotional states and learning outcomes. Emotions can both enhance and interfere with learning depending on which ones are driving or coloring the experience (Osika et al., 2022). The system, by adapting resources based on the learner’s emotional state, could potentially enhance the learner’s emotional profile, thereby improving their overall learning experience (Imhof et al., 2020; Taylor et al., 2021).

4 Modeling the learner's emotion

We take advantage of our facial expression recognition model proposed by (Boughida et al., 2022) to model learner emotions. The idea is to exploit the learner's facial expressions (joy, sadness, fear, disgust, anger, neutral, and surprise) generated during a pedagogical activity of learning, evaluation, or collaboration to update the learner's emotional profile.

4.1 Process of updating the learner's emotional profile

When starting a pedagogical activity (learning, collaboration, or evaluation activity), the camera starts and begins to recognize the learner's facial expression for each frame. Not throughout the entire activity but just the first five minutes. We note that the system requires user authorization to access the webcam for emotion detection. This is done to respect user privacy and ensure informed consent. The data collected is used solely for enhancing the learning experience and is handled with utmost care to ensure security.

At the end of this time, the detection stops, and the learner's profile will be updated by the emotional degree of positivity ${\varvec{e}}{\varvec{m}}{\varvec{o}}{\varvec{D}}{\varvec{e}}{\varvec{g}}{\varvec{P}}{\varvec{o}}{\varvec{s}}({\varvec{x}})$. The latter is a value between 0 and 1 that represents the rate of positive feelings of the learner during a pedagogical activity. The closer this value is to 1, the more positive emotions the learner feels during the use of this activity, and the opposite is true. This cycle will be restored for each pedagogical activity, i.e., the emotional profile is updated after each use of a pedagogical activity. Figure 1 illustrates the cycle of updating the learner's emotional profile.

Fig. 1

Cycle for updating the learner's emotional profile

Since the person's emotional state changes rapidly from one moment to the next, we can see that it is not helpful to propose a whole approach to initialize the learner's emotional profile, so we decide to initialize the learner's emotion with the neutral state. The emotional degree of positivity is thus fixed by 0.5.

4.2 Calculation of the emotional degree of positivity

To calculate the emotional degree of positivity $emoDegPos(x)$ of the learner after the use of pedagogical activity, we do not consider the number of positive facial expressions generated in relation to the total facial expressions generated. Still, we assume that each positive emotion can generate each facial expression with a certain percentage (a certain probability of positivity); this will allow us to estimate a robust idea about the positivity expressed during the pedagogical activity. That is, each emotion has a probability that it will result a positive consequence and a probability that it will result a negative consequence. To measure these values for each emotion, we will rely on the work of An et al. entitled "Two sides of emotion: exploring positivity and negativity in six basic emotions across cultures" published in the journal Frontiers in Psychology (An et al., 2017).

After normalizing the authors' results from this experiment, we extract the probability of positivity and negativity of each of the six Ekman emotions, as shown in Table 2. The probability of positivity (negativity) represents the possibility that the recognized emotion expresses a positive (negative) feeling. It should be noted that neutral emotion is not addressed in this experiment. For this, we consider that the probability of positivity is equal to the probability of negativity equal to 0.5.

Table 2 Probability of positivity and negativity of emotions

Technically, we compute the emotional degree of positivity $emoDegPos(x)$ by adding the probability of positivity of the generated facial expressions, then we average this sum (see example in appendix C.1). Algorithm 1 illustrates the function that computes the emotional degree of positivity $emoDegPos(x)$ of learner $x$ during a pedagogical activity.

Algorithm 1

Emotional degree of positivity of learner

5 Emotion-based adaptation approach

In this section, we present the adaptation approach that considers the learner's information stored in his profile, including his emotional profile. Our approach allows the system to adapt to the learner's problems and difficulties by recommending resources and activities to help the learner get out of his difficult state and motivate him to continue the activities. We call these activities/resources that will try to help the learner the support resources (SR).

A recommendation module has been integrated into a learning environment that provides communication, evaluation, collaboration, and learning activities. We name this platform "EmoLearnAdapt" (Emotion Learning Adaptation). Figure 2 shows the general architecture of the EmoLearnAdapt system.

Fig. 2

emoLearnAdapt architecture

5.1 Learner modeling

In our approach, several learner characteristics, including emotional profile, are considered for the adaptation of support resources. The domain-independent information is the learner's preferences, including learning style, behavioral skills (described in the behavioral profile), and emotional profile. We find the learner's cognitive skills described in the cognitive profile for domain-dependent information.

5.1.1 Preferences

Learners' preferences influence their learning processes (Fleming & Baume, 2006; Komarraju et al., 2011; Morgan & Baker, 2013). Many authors have developed several models for extracting learning styles and learner preferences. Our approach uses the VARK (Visual, Aural, Read/Write, and Kinaesthetic) questionnaire (Fleming & Baume, 2006), which measures the importance of the following four learning styles: visual, auditory, textual, and kinaesthetic.¹ This questionnaire is chosen because it allows us to discover learning preferences straightforwardly. We modify it so that it takes into consideration other preferences that are needed in our approach. Also, we remove the "kinaesthetic" style we do not need. We call the new preference questionnaire: EmoPref (Emotion Preferences). This questionnaire is described in Appendix 1.

Learning styles

To measure the degree of preference ${degPrefStyle}_{sty}(x)$ of a style $sty$ of learner $x$ based on a preference questionnaire emoPref (part 1 in Appendix 1), we count the responses that indicate this style against the total number of questions (12 questions). We consider the three learning styles: visual, aural, and textual.

• Visual style ($styVisual$): when information is presented in a graphic form such as arrows, charts, diagrams, symbols, etc., learners may retain the information better and probably prefer visual, emotional supports in the form of pictures and videos.

• Aural style ($auralSty$): Auditory learners prefer to listen to information presented to them vocally to work well. These learners probably prefer auditory emotional supports in music, speech, etc.

• Textual style ($textualSty$): learners interested in this style do well with information written on worksheets, presentations, and other resources of textual content. They probably prefer text-rich emotional support such as motivational messages

Activity preferences

We ask the learner if they prefer certain activities (relaxation exercises, educational games, collaboration, and communication with others). For each of these activities, the learner chooses one of five answers: 1) No, I don't prefer them at all; 2) No, I do not prefer them; 3) Neutral; 4) Yes, I prefer them; 5) Yes, I strongly prefer them. For each of these activities, a value between 0 and 1 is assigned to measure the degree of preference ${degPrefAct}_{act}(x)$ of activity $act$ of learner $x$. These preferences are summarized as follows:

• (1) Preference for learning through games ($learnGame$): asking learners if they prefer games that focus on learning skills and knowledge and motivate them to learn.

• (2) Preference for collaboration ($coll$): by asking learners if they prefer to complete the tasks by collaborating with other learners.

• (3) Preference of learner contact ($learnerContact$): by asking the learner if they prefer to communicate and discuss with other learners in case of difficulties.

• (4) Preference of teacher contact ($teacherContact$): the learner is asked if they prefer to contact the teachers in case of learning difficulties.

• (5) Preference of relaxation exercises ($relaxEx$): where we ask the learner if they prefer to practice relaxation exercises in case they are stressed.

5.1.2 Behavioral profile

This profile refers to the set of social behaviors generated by the learner. There are several approaches to studying and inferring a user's behavior. We focus on the behavioral profile of learners who interacted with the system in our study. We focused on several types of interactions, which are learner-learner and learner-teacher interactions. To do this, we used the following indicators to create the behavioral profile: 1) The total number of messages sent and received by the learner to system actors. 2) The total number of "like/dislike" ratings of resources and activities. 3) The total number of questions asked in the forum and comments.

To deduce the behavioral profile $BP(x)$ of learner $x$, we compute the total number of interactions defined below relative to the total number of interactions established by all actors in the system.

$$BP\left(x\right)= \frac{Number \space of \space interactions \space made \space by \space the \space learner \space x}{Total \space number \space of \space interactions \space in \space the \space system}$$

(1)

We classify learners according to their behavioral profiles into one of the following four classes (inspired by (Bendjebar et al., 2016)): 1) Very isolated learner$(0 \le BP(x) < 0.25)$; isolated learner $(0.25 \le BP(x) < 0.5)$; slightly dynamic learner$(0.5 \le BP(x) < 0.75)$; dynamic learner$(0.75 \le BP(x) \le 1)$.

We initialize the learner's behavioral profile using a questionnaire inspired by the online survey development company SurveyMonkey² under Social Interaction Questions. The questionnaires of this company are used by several researchers in the field of education (Back et al., 2016, 2019). The questionnaire is addressed in Appendix 2. To calculate the initial behavioral profile, the learner answers each of the questions in the questionnaire, and each answer is assigned a score (between 0 and 1; see also Appendix 2). The sum of the scores obtained on the total of the questions will be the initial value of the behavioral profile ${BP}_{init}\left(x\right)$ of learner $x$.

$${BP}_{init}\left(x\right)= \frac{sum\ of\ the\ scores\ for\ each\ answer}{Number\ of\ questions\ (i.e. 6)}$$

(2)

5.1.3 Cognitive profile

We refer to a learner's cognitive profile as mastery of domain knowledge. This profile is updated if the learner takes an evaluation test at the end of each lesson. The cognitive profile $CP(x)$ of learner $x$ will be calculated by the following formula:

$$CP\left(x\right)= \frac{1}{k+1}\sum_{i=1}^{k}{CP}_{i}\left(x\right)+ {CP}_{init}(x)$$

(3)

With:

$k$: the number of lessons validated by the learner; ${CP}_{init}(x)$: the learner's initial cognitive profile;${CP}_{i}(x)$: he cognitive profile of lesson $i$ will be calculated after taking an evaluation test (in the form of a quiz) of that chapter by the following formula:

$${CP}_{i}\left(x\right)= \frac{Number\ of\ correct\ answers}{Total\ number\ of\ questions\ in\ the\ evaluation\ test\ of\ the\ lesson\ i}$$

(4)

We classify learners according to their cognitive profiles into four classes (Mehenaoui et al., 2016): 1) low ($0 \le CP(x) < 0.4$); 2) medium ($0.4 \le CP(x) < 0.6$); good ($0.6 \le CP(x) < 0.8$); and excellent ($0.8 \le CP(x) \le 1$).

The pretest is the most applied way to measure the learner's prerequisite knowledge in the domain. We use a pretest to initialize the learner's cognitive profile for this. The following formula calculates it:

$${PC}_{init}\left(x\right)= \frac{Total\ number\ of\ correct\ answers}{Total\ number\ of\ pretest\ questions}$$

(5)

5.1.4 Emotional profile

The emotional profile refers to information about the learner's emotional characteristics. As we showed in Section 4, we extract this emotional information from the learner's facial expressions. After each use of a pedagogical activity, the learner's emotional profile will be updated by the learner's emotional state that designates the most recognized facial expression during that activity. Also, it will be updated by the emotional degree of positivity $emoDegPos(x)$ of learner $x$ (see Section 4). We measure the emotional profile $EP(x)$ of learner $x$ by this value:

$$PE(x) = degEmoPos(x)$$

(6)

As mentioned in Section 4, this value will be initialized by 0.5. (${PE}_{init}(x)=0.5$). More detail on modeling learner emotion is in Section 4.

5.2 Proposed support resources

Support resources (SR) are activities and resources that help learners overcome difficulties and become motivated. They are adapted to the learner's preferences and profiles. The degree of relevance (DR) is used to evaluate the relevance of the support resource to the learner's psychological and pedagogical requirements. Support resources are included in the course resources and activities and are offered as support resources for each lesson. These SRs are classified as psychological, pedagogical, or hybrid SRs.

5.2.1 Pedagogical support resources

The purpose of pedagogical SR is to keep learners motivated in their learning process and to help them acquire new skills and knowledge to continue their learning path. These resources also help create positive learning experiences. We find:

• (1) Collaborative activity (collAct): their goal will be to help and support learners in solving their activities by working in groups and distributing tasks.

• (2) Reinforcement activity: These activities can help learners improve their knowledge and skills to pass the evaluation test. These activities can be reinforcement courses (${\varvec{r}}{\varvec{e}}{\varvec{n}}{\varvec{f}}{\varvec{C}}{\varvec{o}}{\varvec{u}}{\varvec{r}}{\varvec{s}}{\varvec{e}}$) or reinforcement exercises (${\varvec{r}}{\varvec{e}}{\varvec{n}}{\varvec{f}}{\varvec{E}}{\varvec{x}}$).

• (3) Learner contact (learnerContact): This communication activity allows the learner to contact their peers if they encounter any difficulties.

• (4) Teacher contact (teacherContact): This communication activity allows the learner to contact his teachers, who can help him learn.

5.2.2 Psychological support resources

These SRs provide psychological support to learners, promoting emotional health, helping them cope with stress, and improving psychological well-being. They aim to regulate emotions during learning, making them more effective. We find:

• (1) Visual resource (visRes): are any kind that can entertain the learner in case of psychological difficulty or be a visual stimulus, for example, nature figures, memes, movies, video jokes, etc.

• (2) Aural resource (AudRes): they motivate the learner or reduce stress. They can be jokes in audio form, high-rhythm music, relaxing music, etc.

• (3) Textual resource (txtRes): textual content of any kind can calm or motivate the learner, for example, text messages to calm the learner, motivation messages, etc.

Relaxation exercise (relaxEx) : it aims to rebalance the nervous system, reorganize thoughts, reduce stress, increase concentration, decrease fatigue, improve mood and reduce physical pain and muscle tension (Kouahla et al., 2022).

5.2.3 Hybrid support resources

These SRs offer emotional and pedagogical support, such as educational Games (eduGame). For example, the sudoku puzzle does not contain enough numbers to solve. Each time the learner answers a question correctly, an additional number is added to the puzzle to make it easier to solve.

5.3 Adaptation criteria and their importance coefficients

Adaptation of support resources is based on a set of adaptation criteria that must be considered in order to adapt what will support the learner to cope with their difficult condition. These criteria are:

• (1) The initial degree of relevance of SR (${drInit}_{sr}$): when a new learner account is created, each support resource's initial degree of relevance will be initialized based on the learner's initial profile and preferences.

• (2) The history of the degrees of relevance of SR (${drHist}_{sr}$): taking into account only the last $k$ DRs stored in the stack of relevance degrees of the support resource for this learner.

• (3) The learner's emotional profile ((EP(x)): the learner's current emotional profile will be taken into account to measure the relevance of the SRs that this profile influences. It may be that the SRs help the learner improve their emotional profile by reducing negative emotions and improving their learning. So, this criterion will play a role in the relevance of these SRs.

• (4) The learner's cognitive profile (CP(x)): the learner's requirements may be related to this profile. The SRs may help the learner to improve his knowledge and cognitive profile and therefore support the learner in case of blockage in his learning process.

• (5) The learner's behavioral profile (BP(x)): this profile must be considered to adapt the SRs that can help the learner come out of isolation if this is the case.

Each of these criteria has an importance that differs from one SR to another. We assign an importance coefficient IC for each of these criteria for a given SR. We will explain in detail the initialization of the importance coefficients and how to weight these coefficients during the adaptation process.

5.3.1 Initialization of importance coefficients

For the initial criterion degree of relevance of SR ${drInit}_{sr}$, we initialize its importance coefficient ${ic}_{drInit,sr}$ of a supporting resource $sr$ by a low value (${ic}_{drInit,sr}$ = 1). The main importance of this criterion is the start of the system. As the learner's profiles and preferences change, this criterion is unimportant. Regarding the criterion history of the degrees of relevance of SR ${drHist}_{sr}$, we initialize it with ${ic}_{drHist,sr} = 1$ for all support resources. History is important because it measures the SR's relevance to the learner. For example, if the "educational game" support resource did not help the learner (i.e., with a low DR), but the DR history of this SR shows that the relevance is high, then there is a chance that this SR will help them soon. We only consider the last three DRs stored in the stack of degrees of relevance of the SR, so $k=3.$

Figure 3 summarizes how cognitive, behavioral, and emotional aspects affect the different forms of SR. The emotional profile is considered for each of these SRs. Determining which SRs can help the learner feel better can be difficult. Still, psychological support resources are specifically designed to encourage and amuse the student if their emotional state is unfavorable. Therefore, compared to pedagogical support resources, psychological SRs will have a higher IC for this criterion (i.e., emotional profile).

Fig. 3

The influence of learner profiles on the different types of SR

For Educational Games ($eduGame$), there may be an influence of the cognitive aspect; it may help the learner to improve his cognitive profile; for the behavioral profile, there is probably no influence. The collaborative activity ($collAct$) will probably improve his cognitive skills and make the learner interactive because he will interact with the other learners in his group. The Learner Contact ($learnerContact$) will eventually improve the learner's cognitive profile; by default, the learner's behavioral profile is neglected for this SR. The teacher's contact ($teacherContact$) will possibly support the learner to improve their cognitive skills. For the reinforcement exercises ($renfEx$), they are likely to help the learner and improve their cognitive profile, and the same for the reinforcement courses ($renfCourse$). For textual ($txtRes$), visual ($visRes$), and aural ($audRes$) resources, they are only directly related to the learner's emotional profile. They are more likely recommended if they reduce the learner's negative emotions. Finally, the relaxation exercises ($relaxEx$) allow for reducing negative emotions by reducing stress. It is in direct relation only with the emotional profile. Therefore, we can summarize the values of the initial importance coefficients estimated in Table 3.

Table 3 Initial values of the SR importance coefficients

5.3.2 Dashboard of the weighting of importance coefficients

Some adaptation criteria will likely have more or less impact on one of the SRs, while the initial importance coefficients of these resources will not be accurately reflected. To solve this problem, a dashboard prototype is proposed, where the system administrator can weight the importance coefficients of each of the existing support resources. To do so, he consults the graphs of the evolution of the learners' different emotional, behavioral, and cognitive profiles. He then analyzes the effect of resource choice on the learners' profiles before updating these coefficients. Figure 4 shows a low-fidelity prototype of this dashboard.

Fig. 4

Prototype dashboard for weighting SR importance coefficients

5.4 Description of the process of adapting support resources based on emotion

In this section, we detail the process of adapting emotion-based support resources. The goal is to offer learners the $n$-first support resources in the order of their degree of relevance. The following three steps can summarize the steps of this process:

5.4.1 Initialization of the degrees of relevance of support resources

The relevance degrees of the SRs must be initialized to handle the cold start. This initialization process is illustrated in Fig. 5. The emotional ${EP}_{init}\left(x\right)$, behavioral ${BP}_{init}\left(x\right)$, and cognitive ${CP}_{init}\left(x\right)$ Profiles are initialized to accomplish this task, as detailed in Sections 5.1.2, 5.1.3, and 5.1.4. Also, the SR preference information is included in the initial DR. We calculate the preference degrees of the different learning styles visual ${degPrefStyle}_{visualSty}(x)$, aural ${degPrefStyle}_{auralSty}\left(x\right)$ and textual ${degPrefStyle}_{textualSty}\left(x\right)$, and also, we determine the preference degrees of the activities ${degPrefAct}_{act}({\text{x}})$ (see Section 5.1.1).

Fig. 5

Initialization process of the different support resources

For each SR, we take into account the profiles that have an impact on it, as shown in Fig. 3. We consider the inverse of the profiles, i.e., if the initial cognitive profile is low, the most relevant SRs are reinforcement exercises, educational games, etc. Unlike the adaptation process, where the relevant SRs can help the learner, another time is those that increase this profile.

Also, we consider visual, textual, and aural learning styles. Therefore, SRs adapted to the learner's learning style can be suggested. For example, if the learner prefers an aural learning style, there is a strong chance that SRs, such as relaxing music, audio jokes, etc., will be recommended. Also, the preferences of the activities will be taken into account. For example, if the learner prefers to work in a group with other learners, support resources like collaborative activity will be suggested because the initial DR of this support resource will likely be high. The initial relevance degree formulas are detailed in table on appendix C.2.

We note that the emotional state is initially set to 0.5, representing a neutral state. As such, its impact on determining the initial relevance of the support resources is minimal. Instead, other factors play a more significant role in determining initial relevance. For instance, in the case of collaborative activities, the formula to calculate initial degree of relevance is ${drInit}_{collAct} = \frac{1}{4}({degPrefAct}_{coll}(x) + \left(1 - {CP}_{init}\left(x\right)\right)+(1- {EP}_{init}(x)) + (1 - {BP}_{init}(x)) \space)$

The relevant profiles are EP, CP, and BP, along with the preference. In this scenario, the values of CP, BP, and the degree of preference for collaboration will primarily determine the initial relevance of this support resource, because the value of initial EP is in a middle (0.5). A similar process applies to other resources. Therefore, it can be concluded that the impact of the initialization of learner emotions with "neutral" on system's subsequent actions is negligible.

5.4.2 Selection of support resources

In the adaptation process, only the degrees levels of the support resources selected by the learner will be updated. The proposed SRs are available to the learner at any time, and the learner must use a support resource for a predetermined duration to say that it is selected (activated). We can define this duration by the minimum activation time of SR ${minActivationTime}_{sr}$. This value will allow us to determine whether the learner has used this SR. For example, if relaxing music has a minimum usage duration of 40 s but the learner only listens 10 s $(currentDurationUse(sr) = 10)$, we cannot report that the learner used that SR and therefore we do not consider it as selected (activated). Thus, the value of ${minActivationTime}_{sr}$ depends only on this SR's content; therefore, the administrator will estimate these values when developing all SRs.

On the other hand, the emotional, behavioral, or cognitive effect of the learner's selected support resources will not last forever. Therefore, the maximum time for SR activation ${maxActivationTime}_{sr}$, i.e., the duration it is active (selected) is defined. This value depends directly on the type of SR. For example, the relaxation exercise will perhaps reduce the learner's stress, but for a short period, we cannot say that after a week, this SR has helped him, and therefore we recommend it! During this activation period, the degree of relevance of SRs will be updated. That is, as long as the current SR activation duration $currentDurationActivation(sr)$ is less than ${maxActivationTime}_{sr}$, and after this period, the SR is not active anymore. The administrator also estimates the ${maxActivationTime}_{sr}$ at the time of SR creation.

The state diagram in Fig. 6 illustrates activating and deactivating the supporting resource.

Fig. 6

State diagram of the support resource since its creation

5.4.3 Launch of activities and adaptation of SR

When the learner initiates a learning, evaluation, or collaboration activity, the emotion recognition system is launched for five minutes to update the emotional profile of this learner (as already shown in Section 4). In parallel, using these types of activities (and communication activity) generates interactions that update the learner's cognitive and behavioral profiles. The learner's cognitive profile is updated through an evaluation activity such as a quiz. Chatting with other people changes the information about the learner's behavior (behavioral profile).

When the learner completes an activity, the degree of relevance of the selected (active) support resources will be updated according to the new renewal of the different learner profiles. The following formula is used to update the degree of relevance ${dr}_{rs}$ of the supporting resource $sr$.

$${dr}_{sr}= \frac{{ic}_{EP,sr}EP(x) + {ic}_{CP,sr}CP(x) + {ic}_{BP,sr}BP(x) + {ic}_{drHist,sr}\frac{1}{k}\sum_{i=1}^{k}{drHist}_{sr}[i] + {{ic}_{drInit,sr}drInit}_{sr}}{{ic}_{EP,sr} + {ic}_{CP,sr} + {ic}_{BP,sr} + {ic}_{drHist,sr} + {ic}_{drInit,sr}}$$

(7)

With:

• $EP(x):$ The learner's current emotional profile $x$

• $CP(x)$: The learner's current cognitive profile $x$.

• $BP(x)$: The learner's current behavioral profile $x$.

• ${drHist}_{sr}:$ The history of the degrees of relevance of the support resource sr. We consider the last $k$ update of the degree of relevance of this RS. After several experiments, we set $k = 3$. If the size of the support resource's PD stack is strictly less than 3, then the value of $k$ is equal to the size of the stack.

• ${drInit}_{sr}$: The initial degree of relevance of the support resource $sr$.

• ${ic}_{EP,sr}$: The importance coefficient of the emotional profile for the support resource$sr$.

• ${ic}_{CP,sr}$: The importance coefficient of the cognitive profile for the support resource $sr$.

• ${ic}_{BP,sr}$: The importance coefficient of the behavioral profile for the support resource$sr$.

• ${ic}_{drHist,sr}$: The importance coefficient of the DR history for the support resource sr.

• ${ic}_{drInit,sr}$: The importance coefficient of the initial degree of relevance for the support resource $sr$.

Algorithm 2 shows how to update the selected support resources after launching a pedagogical activity.

Algorithm 2

Update of the degree of relevance of the selected SRs

As a result, the support resources will be adapted, with the recommendation to the learner of the most relevant support resources—the $n$-first support resources with the highest degree of relevance. The process then restarts; the learner chooses the SRs they want and will be adapted according to the updates of the different learner profiles when using the various pedagogical activities of the course. We note that once the system determines the most relevant resources, it automatically sends these resources to the students. The professor’s validation is not required for this process. The activity diagram in Fig. 7 describes adapting learner support resources. An illustrative example of how the system works (with 10 support resources) is presented in Appendix C3.

Fig. 7

Activity diagram of the emotion-based adaptation process of support resources

6 Experiment study

To validate the proposed emotion-based adaptation system emoLearnAdapt, an experimental study was done at the University of Guelma (Algeria) with first-year bachelor students in mathematics and computer science, where three tests were conducted: the first one served to verify the effect of the proposed approach on the improvement of the student's knowledge level, the second one compares the progression of the student's knowledge level with and without the adaptation tool, and the third one has a goal to demonstrate how the adaptation of the support resources will improve the students' emotional profiles.

The developed system was tested for a period of three months, corresponding to one semester, during the year 2022. Both the control and experimental groups of students started using the system at the same time, ensuring a fair and balanced comparison. The platform was hosted and made accessible to the participants at the following address: https://emoadapt.com. Participants could access the platform from either a computer or a smartphone, providing flexibility in terms of device usage.

6.1 Development of the system functionalities

Moodle is an open-source learning platform that provides learning, collaboration, communication, and evaluation activities by allowing learners to access resources of all kinds (videos, HTML pages, folders,...) and activities (quizzes, chat, forums, surveys, wiki...).

To provide the new system functionalities that cover the emotion-based adaptation approach we proposed, we developed a set of plugins under Moodle classified into three types:

• (1) Activity modules (/mod): this type of plugin provides activities in courses. We are developing two new plugins, educational games, and Collaborative activity.

• (2) Plugins blocks (/blocks): they allow additional information and functionality to be displayed in different parts of Moodle. In this type of plugin, we are developing facial Expression Recognition, support resource recommendations List, and Dashboard plugins.

• (3) Local plugins (/local): this type includes elements that do not fit into the standard plugin types. For this, we develop the plugins of initialization and the support resource manager.

In the following, we will detail these plugins' functionalities.

• (1) Educational games plugin (mod_game): like crosswords, Millionaire and Sudoku, but we are modifying the Moodle games plugin to create educational games like support resources.

• (2) Initialization plugin (local_init): it is triggered just with the platform's first use to initialize the learner's profiles, extract preferences and learning styles, and initialize the degrees of relevance of the different support resources.

• (3) Support resource manager plugin (local_supportres): it allows to add, delete or modify SRs. Also, update the degrees of relevance of the support resources after each update of the learner's profiles.

• (4) Facial expression recognition plugin (blocks_fer): When launching a pedagogical activity (learning, collaboration, or evaluation activity), this plugin starts the learner’s webcam after asking for their permission. The emotion recognition process begins immediately and continues for the first five minutes of the activity, during which it displays the recognition result in this block, as shown in Fig. 8. At the end of the recognition period, the webcam stops detecting, and the learner’s emotional profile will be updated.

• (5) Collaborative activity plugin (mod_collabora): we integrate as SR an online collaborative code editor and compiler in real time like CodeCollab.³ It also offers a chat space between group members.

• (6) Support resource recommendation plugin (blocks_rsrec): this block will be visible on all the web pages of our Moodle platform. It displays the links of the $n$-first n support resources in the decreasing order of their degrees of relevance. Moreover, the learner can access all the SRs (Fig. 9).

• (7) Dashboard Plugin (blocks_dashboard): This block embodies the prototype dashboard proposed in Section 5.3.2. This dashboard allows for the weighting of different SR relevance coefficients by observing the evaluations of learner profiles.

Fig. 8

Facial expression recognition during student learning

Fig. 9

Support Resource Recommendation Block (right)

6.2 Participants

Forty students in the first year of Computer Science and Mathematics subject Algorithms are testing the emoLearnAdapt system. The students were divided into two groups: the first group, composed of 23 students working with the proposed adaptation system (experimental group); the second group, composed of 17 students working without the approach where there is no recognition of emotions and there is no support resources recommendation (control group).

6.3 Methodology and results obtained

In this section, we detail the studies conducted with the statistical tests employed for each of them to verify the validity of the proposed emotion-based adaptation approach. We note that the hypothesizes to be proved in out experiments are the same as those stated in the Sect. 3.

6.3.1 Test 1: Verification of the effect of the approach on the improvement of the knowledge level

We divide this test into two subtests to provide a more comprehensive understanding of the approach’s impact on improvement of the knowledge level (cognitive profile).

The first subtest focuses on the intra-group improvement, examining how the knowledge level of students within the same group is affected by the proposed approach. This will provide insights into the effectiveness of the approach in enhancing the learning experience for students working together.

The second subtest involves an inter-group comparison, contrasting the improvements in knowledge levels of students in two different groups—one with the proposed approach and one without. This comparison will help determine the relative effectiveness of the approach in different learning environments.

6.3.2 Subtest 1: Verification of the effect of the approach on the improvement of the knowledge level of students who are in the same group

1. Methodology: This study aims to determine whether the Support Resource Adaptation approach improves the cognitive profiles of the 23 students in the experimental group (who used emoLearnAdapt with its functionalities); for this purpose, the students answered two questionnaires (pretest and post-test) on the programming with the C language. The pretest is given to the students in the experimental group before accessing the system to assess their cognitive levels. In contrast, the post-test assesses their cognitive levels after using the system. We define the random variable ${CP}_{pretest}$ the knowledge level (cognitive profile) of the students was measured with the pretest, and ${CP}_{post-test}$ the knowledge level (cognitive profile) of students measured with the post-test.

Figure 10 and Table 4 show some descriptive statistics about learners' cognitive profiles before and after using the system.

Fig. 10

Box plot of the two samples ${CP}_{pretest}$ and ${CP}_{post-test}$

Table 4 Descriptive statistics of the two variables ${CP}_{pretest}$ and ${CP}_{post-test}$

2. Definition of hypothesis: We now define the null hypothesis ${H}_{0}$: "the use of the support resource adaptation tool does not improve learners' emotional profile" (${{\text{CP}}}_{pretest}$ ≥ ${{\text{CP}}}_{post-test}$). For the alternative hypothesis ${H}_{a}$: "using the support resource adaptation tool improves learners' emotional profile" (${{\text{CP}}}_{pretest}$ ≤ ${{\text{CP}}}_{post-test}$).

3. Normality of the data: We have two samples (${{\text{CP}}}_{pretest}$ et ${{\text{CP}}}_{post-test}$) are dependent because we measured his cognitive profile in these two samples for the same learner. To choose the appropriate test, we have to test if the sample data (${{\text{CP}}}_{pretest }- {{\text{CP}}}_{post-test}$) are normally distributed with the Shapiro–Wilk test; we obtain $W = 0.9$, ${P}_{value} = 0.032$.. We have therefore ${P}_{value}<0.05,$ and in consequence, we accept the null hypothesis and, therefore, the data of the difference between the paired values (${{\text{CP}}}_{pretest }- {{\text{CP}}}_{post-test}$) are not normally distributed. The normality of the data can be visualized graphically by the histogram and the Q-Q plot (Fig. 11). They also show that the data are not normally distributed. As a result, we decided to work with the one-tailed Wilcoxon test for paired samples. Wilcoxon test compares the median of two paired groups and can be used when the distribution of the differences between the two groups is not assumed to be normal.

Fig. 11

Test of normality of ${{\text{CP}}}_{\mathrm{pretest }}- {{\text{CP}}}_{{\text{post}}-{\text{test}}}$. a With histogram, b with Q-Q plot

4. Application of one-tailed Wilcoxon test: We apply the Wilcoxon test with $\alpha = 0.05$, we obtain $Z = 4.02$, ${P}_{value} < 0.001$. The values of ${{\text{CP}}}_{pretest}$ had lower values ($Median = 0.63$) than the values of ${{\text{PC}}}_{post-test}$ ($Median = 0.78$). The null hypothesis that ${{\text{CP}}}_{post-test}$ has a value less than or equal to the ${{\text{CP}}}_{pretest}$ was tested using a one-tailed Wilcoxon test for paired samples.

5. Result: ${P}_{value}< 0.05$, the result is statistically significant. The null hypothesis is therefore rejected. hence, the values of ${{\text{CP}}}_{post-test}$ are higher than the values of ${{\text{CP}}}_{pretest}$ and we can say that the use of the support resource adaptation tool improves learners' knowledge level (cognitive profile).

6.3.3 Subtest 2: Comparison of improvements in knowledge levels of students in two different groups (with and without the proposed approach)

1. Methodology: In this study, we now compare the improvement of the level of knowledge (the cognitive profile) in two samples of learners: the experimental group (who used emoLearnAdapt with all its features) and the control group. The division into the experimental and control groups was done using a randomized selection process. This random assignment ensured unbiased group formation and attributed outcome differences to the use of emoLearnAdapt. We believe this method enhances our study’s transparency and validity.

The goal of the test is to see the impact of the approach on improving the cognitive profile compared to learners who did not use the features of emoLearnAdapt. To measure the cognitive profile improvement of both experimental and control groups, students’ cognitive profile is evaluated before and after using the system through a pretest and a post-test, as in the first experiment, but this time also for the control group. We compare the two samples ${diffCP}_{contr{\text{ol}}}$ and ${diffCP}_{exp}$ (with ${diffCP}_{exp} ={CP}_{exp,post-test }-{CP}_{exp,pr{\text{e}}test}$ and ${diffCP}_{contr{\text{ol}}}={CP}_{contr{\text{ol}}, post-test}-{ CP}_{contr{\text{ol}}, pr{\text{e}}test}$).

Figure 12 and Table 5 shows some descriptive statistics of the progression of the cognitive profiles of both experimental and control groups.

Fig. 12

Box plot of the two samples ${{\text{diffCP}}}_{{\text{exp}}}$ et ${{\text{diffCP}}}_{{\text{control}}}$

Table 5 Descriptive statistics of the two variables ${{\text{diffCP}}}_{{\text{exp}}}$ and ${{\text{diffCP}}}_{{\text{control}}}$

Equivalence of pre-test results in control and test groups

In experimental research, it is crucial to ensure that the control and test groups are equivalent at the outset of the study. This equivalence is typically assessed through a pre-test, which provides baseline data for each group. For that, we conducted a Shapiro–Wilk test to assess the normality of the data for both the control and test groups. The results showed a ${P}_{value}$ = 0.3938 and $W$ = 0.9458 for the control group, and a ${P}_{value}$= 0.6355 and $W = 0.9678$ for the test group. As the ${P}_{value}$ were greater than the significance level ($\alpha$), we accepted the null hypothesis, indicating that the data for both groups were normally distributed.

Following this, we performed an unpaired t-test to compare the means of the two groups. The mean of the control group was 0.6200 and that of the test group was 0.6304, resulting in a difference of -0.0104. The t-statistic was 0.1946 with 38 degrees of freedom, and ${P}_{value}$= 0.8467. As the ${P}_{value}$ was greater than 0.05, we concluded that the difference between the two groups was not statistically significant.

These results suggest that the control and test groups were equivalent at the start of the experiment, validating the experimental design and ensuring that any observed differences in the post-test results can be attributed to the experimental treatment rather than pre-existing differences between the groups. This strengthens the reliability and validity of our findings.

2. Definition of hypothesis: We now define the null hypothesis ${H}_{0}$: "the use of the support resource adaptation tool does not improve learners' cognitive profile" (${diffCP}_{exp }\le {diffCP}_{contr{\text{ol}}})$. For the alternative hypothesis ${H}_{a}$: "the use of the SR adaptation tool improves learners' cognitive profile" (${diffCP}_{exp }\ge {diffCP}_{contr{\text{ol}}})$.

3. Normality of the data: The two samples (${diffCP}_{exp}$ et ${diffCP}_{contr{\text{ol}}}$) are independent because we have two groups of different students (control group $N = 17$ and experimental group $N = 23$). We test the distribution of the two samples separately -as they are independent- with the Shapiro–Wilk test. For the control group, we find $= 0.93,$ ${P}_{value}= 0.238$. Therefore, we have ${P}_{value}> 0.05,$ and hence we can say that the data of the sample ${diffCP}_{contr{\text{ol}}}$ follow the normal distribution. The histogram in Fig. 13a also gives us an idea of the normal distribution of these data. Unlike with experimental group, we find ${P}_{value} < 0.05$ (because $W=0.9$, ${P}_{value}=0.032$), and therefore, we can say that the data of ${diffPC}_{test}$ do not follow the normal distribution. We can validate the non-normality of this sample graphically by the histogram (see Fig. 13b). Since the data from one of these independent samples do not follow the normal distribution, we work with the one-tailed Mann–Whitney U-Test for the unpaired sample. This test does not assume normality and can be used to compare two independent groups.

Fig. 13

Normality test of a ${{\text{diffCP}}}_{{\text{control}}}$ and ${{\text{diffCP}}}_{{\text{exp}}}$. a Control group, b Experimental group

4. Application of one-tailed Mann–Whitney U-Test: The results of descriptive statistics show that the sample ${diffCP}_{exp}$ had higher values ($Median = 0.1$) than the control group ($Median = 0.04$). By applying the one-tailed Mann–Whitney U-Test with $\alpha = 0.05$, we find: $U = 108, Z = -2.4$ and ${P}_{value} = 0.008$. The null hypothesis that "sample ${diffCP}_{exp}$ has a value less than or equal to that of the sample ${diffCP}_{control}$" was tested using a one-tailed Mann–Whitney U test.

5. Result: ${P}_{value}< 0.05$, the result is statistically significant. Therefore, the null hypothesis is rejected. Consequently, we accept the alternative hypothesis that "the ${diffCP}_{exp}$ values are greater than the ${diffCP}_{control}$ values". Therefore, using the SR adaptation tool improves learners' knowledge level compared to the control group.

6.3.4 Test 2: Verification of the effect of the approach on increasing the student's positive emotions

This test aims to assess the impact of the recommended support resources on the learner's emotional profile, i.e., whether these support resources increase the positive emotions of the students in the experimental group or not. 1.

1. Methodology: The idea is to consider the changes in the learner's emotion from selecting the support resource until the end of the support resource effect. For this, and for each support resource selection (total of selections equal to 499), we calculate the difference between the emotional profile at the time of selection and the end of the support resource effect. For each of the 23 students in the experimental group ($N = 23$), we associate the average of the differences in the emotional profile for all the selections made by that student. We call this sample $DiffEP$. For example, Learner 11 made five SR selections; the EP differences of these selections are: $-0.15,-0.17,-0.16,-0.26, and \space 0.09$. This learner is assigned the average of these differences, i.e.: $\frac{\left(-0.15\right)+\left(-0.17\right)+\left(-0.16\right)+\left(-0.26\right)+\left(0.09\right)}{5}=0.13$.

If this difference is positive, the support resource has increased the learner's emotional profile, and the opposite is true. But we test if this difference is big enough to say that there is an increase. It is big enough if it is higher than a value called $testvalue$ that is fixed by 0.1. Table 6 describes some descriptive statistics for this sample.

Table 6 Descriptive statistics of ${DiffEP}$

2. Definition of hypothesis: We now define the null hypothesis ${H}_{0}$: "the use of support resources does not improve the emotional profile of the learner" ($DiffEP < 0.1$). For the alternative hypothesis ${H}_{a}$: "the use of support resources improves the emotional profile of the learner" ($DiffEP < 0.1$).

3. Normality of the data: As in all previous tests, we test the normality of the $DiffEP$ data with Shapiro–Wilk, we obtain $W = 0.95$, ${P}_{value}= 0.25$. We have therefore ${P}_{value}> 0.05$, and hence the data of this sample are normally distributed. We can also test the normality of these data graphically by the diagrams shown in Fig. 14 (the histogram and the Q-Q plot), where they show the same thing. For these reasons, we test the null hypothesis with the one-sided t-test of a single sample. This test compares the mean of your sample against the hypothetical mean (test value).

Fig. 14

Normality test of $DiffEP$. a With histogram, b With Q-Q plot

4. Application of one-sided t-test: Applying the t-test on the $DiffEP$ sample, we obtain $t(df = 22) = 5.437$, ${P}_{value}= 0.99$. ${P}_{value}$ is greater than the level of significance (α = 0.05). The sample's mean ($Mean = 0.01$) is much lower than the fixed test value ($testvalue = 0.1$).

5. Result: The t-test result is insignificant for the present data, and the null hypothesis is retained. Therefore, the sample is assumed to be from a population with a mean less than or equal to 0.1. In summary, the support resource adaptation tool could not improve the learner's emotional profile.

This difference by type of support resource is illustrated in Fig. 15. We notice that for the support resources where the learner's emotion increases ($DiffEP > 0$): audRes, txtRes, visRes, learnerContact, teacherContact, and educGame, the increase is not significant (does not exceed 0.05). There is no increase in the other support resources (collAct, renfEx, and renfEx), and the decrease in emotion is insignificant. In general, this difference in the emotional profile is not remarkable; it is, on average, between -0.05 and 0.05. Through the analysis of $diffEP$ by type of support resource, we can confirm that the resources do not increase the emotional profile of the learner.

Fig. 15

Comparison of different types of resource support by the difference in EPs

6.4 Discussion of results

The results of the first subtest of the first test indicate that the support resources' emotion-based adaptation tool increased the learners' cognitive profile. The second subtest consolidates the first results by comparing the cognitive profile progression of learners who used emoLearnAdapt with those who did not. This is justified by recommending different support resources that adapt to the learner's profile information. Some of these support resources help learners to improve their knowledge levels, such as teacher contact, learner contact, collaborative activities, or even educational games. Other types of support resources help them psychologically and increase their positive emotions, which motivates them to learn and therefore improve their cognitive profile.

The second test's purpose is to test if there is an impact of the recommended support resources on increasing the learner's positive emotions. We found results that were not significant; in this case, support resources were not able to improve the emotional profile of the learner, although they were able to increase their cognitive profile. This may be explained by the complex nature of emotion and the difficulty of recognizing the learner's current emotional state. In this work, we tried to recognize learners' emotions only from their facial expressions. We considered that the latter is a mirror of the emotion (i.e., we started from the hypothesis that the facial expression recognized by the facial expression recognition system represents the learner's emotion). The use of a single source of emotional data did not allow for an accurate recognition of the emotion, which is why we sometimes see an increase in the emotion profile, sometimes the opposite, as explained by the normality of the data (all differences in emotion profile are around 0).

These results have several implications for the design of emotion-based adaptive learning systems. First, they suggest that providing learners with different types of support resources can enhance their learning outcomes and motivation. This implies that emotion-based adaptation is not only beneficial for the learner’s emotional well-being, but also for their cognitive performance and engagement. Therefore, designers of emotion-based adaptive learning systems should consider the variety and suitability of the support resources that they offer to the learners, and how they can match them with the learner’s profile information. Second, they indicate that facial expression recognition alone may not be sufficient to capture the learner’s emotional state accurately, and that other sources of emotional data, such as physiological signals, self-reports, or behavioral cues, may be needed to improve the reliability and validity of the emotion recognition process. This implies that emotion recognition is a challenging and multidimensional task, and that relying on a single modality may not capture the complexity and diversity of the learner’s emotions. Therefore, designers of emotion-based adaptive learning systems should explore the feasibility and effectiveness of using multimodal emotion recognition methods, and how they can integrate them with the emotion-based adaptation tool.

Some limitations of this work should also be acknowledged. First, the sample size of the experiments was relatively small, and the participants were mainly undergraduate students from a single institution. Therefore, the generalizability of the findings may be limited, and further studies with larger and more diverse samples are needed to confirm the results. This limitation may affect the validity and reliability of the results, as the sample may not represent the population of interest, and the results may not be replicated in different settings or groups. Second, the experiments focused on a single domain of learning, namely programming. It is possible that different domains may require different types of support resources or emotion recognition methods. Therefore, more research is needed to investigate how the proposed approach can be adapted and applied to other domains of learning. This limitation may affect the generalizability and applicability of the results, as the domain of learning may affect the learner’s cognitive and emotional processes, and the proposed approach may need to be customized and evaluated for different learning contents and objectives.

Based on the results and limitations of this work, some potential future research directions can be identified. First, future research could explore other methods of emotion recognition to supplement facial expression analysis. This could include physiological measures (such as heart rate or skin conductance), self-report measures, or even analysis of verbal and written communication. These methods could provide more information and insights into the learner’s emotional state, and help to improve the accuracy and robustness of the emotion recognition process. Second, future research could also investigate the impact of different types of support resources on both the cognitive and emotional profiles of learners. This could help to identify which resources are most effective for different types of learners, and how they can be tailored and personalized to meet the learner’s needs and preferences. Third, future research could also extend the proposed approach to other domains of learning, and evaluate its effectiveness and usability in different learning scenarios. This could help to test the generalizability and applicability of the proposed approach, and to discover new challenges and opportunities for emotion-based adaptive learning systems.

7 Conclusion

In Computer Environment for Human Learning, learners may encounter in their learning process, learning difficulties, psychological problems, and situations of demotivation or isolation, which will have an impact on their emotional state. Therefore, it is essential in these environments to consider the learner's emotions to help them continue their learning process. In this work, we have considered the learner's emotional state by adapting the resources and activities so that they can support, motivate and unblock them in difficult situations.

For this purpose, we have established adaptation criteria inspired by different qualitative aspects. The modeling of these criteria presents a big challenge, especially with the emotional profile, which is the cornerstone of this work. The first main contribution of this work was precisely the modeling of the learner's emotion from his facial expressions generated during learning, collaboration, or evaluation activities. An algorithm for quantifying the learner's emotion is proposed based on the principle that the generated facial expression can reflect positive and negative emotions.

The second main contribution of this work is the suggestion of an adaptation algorithm for a set of support resources that describes when and how to update the relevance of these support resources while exploiting the defined adaptation criteria. The importance of these criteria varies according to the type or nature of the support resource. From a dashboard, the importance coefficients of these criteria can be weighted. Finally, the most relevant support resources (the n-top) were delivered from a recommendation tool.

To validate our approach, a first experiment was conducted at the University of Guelma (Algeria) with first-year bachelor students in mathematics and computer science. The proposed support resources succeeded in improving the cognitive level of the learners. However, the proposed method did not improve the emotional profile of the learner. The problem may be recognizing the learner's emotions, which presents a big challenge. Also, perhaps because of the nature of the emotions detected (joy, sadness, fear, disgust, anger, neutrality, and surprise) which are not universal and not academic emotions that explain all kinds of psychological processes during learning.

In our future work, we aim to enhance our study by increasing the sample size, which will provide more robust and generalizable results. We also plan to incorporate qualitative research methods, such as interviews or focus groups, to gain a deeper understanding of our approach’s effects. These improvements will not only strengthen our research but also provide a more holistic view of the learning process.

Furthermore, we aim to improve the recognition of the learner’s emotion by proposing a multimodal approach, combining several sources of emotional information to increase the performance of the emotion recognition model to recognize the most complex emotions of the learner.

Exploring the introduction of a third group, where support resources are recommended without considering the emotional state, is another avenue we plan to pursue. This could help to isolate the impact of the emotional state on the effectiveness of the learning process. Additionally, we aim to further investigate the role of the learner’s emotional state in the selection and provision of resources, to enhance the uniqueness and effectiveness of our system.

We intend to focus on more feasible sources for online learning, such as text (discussions and comments generated by learners), mouse movements during learning, keyboarding dynamics (Keystroke Dynmics), etc.

In response to valuable feedback, we plan to refine our system to incorporate alerts or notifications for educators when certain emotional states are detected. This would allow educators to intervene and engage with the students to understand the underlying reasons for these emotions, providing a more holistic approach to learning. We believe this approach would enhance the effectiveness of our system and provide a more supportive and understanding learning environment for students.

Appendices

VARK 7.0 URL: https://vark-learn.com/wp-content/uploads/2014/08/The-VARK-Questionnaire-French.pdf

Appendix 1. EmoPref questionnaire (inspired by the VARK questionnaire Version 7.0)

VARK 7.0 URL: https://vark-learn.com/wp-content/uploads/2014/08/The-VARK-Questionnaire-French.pdf

1.1 How do i learn best?

This questionnaire will tell you about your preferred learning style for acquiring and using ideas and information. Choose the answer which best explains your preference and circle the letter(s) next to it. Leave blank any question that does not apply.

• Q1. Would you prefer to help a person who wants to go to the airport by:

  • (1) Giving verbal directions (A)

  • (2) Writing directions on a piece of paper (R)

  • (3) You draw a map (V)

• Q2. You prefer to get your friends' opinions on the vacation program by:

  • (1) Using a map or website to show them the sites (V)

  • (2) Giving a printed copy of the program (R)

  • (3) Contact them by phone (A)

• Q3. Do you prefer to cook a dish, especially for your family, by:

  • (1) Asking friends for suggestions (A)

  • (2) Choosing the recipe based on the illustrations (V)

  • (3) Look up a specific recipe in the cookbook (R)

• Q4. What will influence your choice to buy a phone?

  • (1) The detailed reading of its features and performance (R)

  • (2) Its modern design and look (V)

  • (3) The salesperson's opinion of the product (A)

• Q5. You learn best by:

  • (1) Listening to someone explain and asking questions (A)

  • (2) Using diagrams, charts, and pictures (V)

  • (3) Read an instruction manual (R)

• Q6. You have a knee problem. Would you prefer that the doctor:

  • (1) Give you a web address where you can find explanations (R)

  • (2) Just describe what the problem is (A)

  • (3) Show you on a diagram the problem (V)

• Q7. Would you prefer to learn a new computer game by:

  • (1) Reading the game's instruction manual (A)

  • (2) Talking to people who already know the program (A)

  • (3) Follow the diagram provided with the instruction manual (V)

• Q8. Do you like websites with:

  • (1) Pretty design and pictures (V)

  • (2) Interesting descriptions and explanations (R)

  • (3) Animations, videos, etc. (A)

• Q9. You use a website to learn how to use your new phone. You prefer:

  • (1) An opportunity to ask specific questions (A)

  • (2) Clear instructions on all its possibilities (R)

  • (3) Diagrams describing the phone, and its different parts (V)

• Q10. Do you prefer a teacher who uses it?

  • (1) Discussions with questions and answers (A)

  • (2) Books (R)

  • (3) Diagrams, graphs, figures, etc.(V)

• Q11. When choosing what to eat at a restaurant:

  • (1) You listen to the waiter's or your friends' recommendations (A)

  • (2) You choose based on the description of the dish (R)

  • (3) You look at what the customers are eating (V)

• Q12. You are required to give a speech on a special occasion:

  • (1) You make diagrams or charts (V)

  • (2) You write a few keywords and repeat your speech many times (A)

  • (3) You write your whole speech and learn it by reading it several times (R)

1.2 Questions to describe activity preferences

• Q13. Do you prefer to learn with games?

  • (1) No, I don't prefer them at all

  • (2) No, I do not prefer them

  • (3) Neutral

  • (4) Yes, I prefer them

  • (5) Yes, I strongly prefer them

• Q14. Do you prefer to collaborate with other learners in case of problems?

  • (1) No, I don't prefer them at all

  • (2) No, I do not prefer them

  • (3) Neutral

  • (4) Yes, I prefer them

  • (5) Yes, I strongly prefer them

• Q15. Do you prefer the teacher's help in case of difficulties?

  • (1) No, I don't prefer them at all

  • (2) No, I do not prefer them

  • (3) Neutral

  • (4) Yes, I prefer them

  • (5) Yes, I strongly prefer them

• Q16. Do you prefer the help of another learner in case of difficulties?

  • (1) No, I don't prefer them at all

  • (2) No, I do not prefer them

  • (3) Neutral

  • (4) Yes, I prefer them

  • (5) Yes, I strongly prefer them

• Q17. Do you prefer relaxation exercises when you are stressed?

  • (1) No, I don't prefer them at all

  • (2) No, I do not prefer them

  • (3) Neutral

  • (4) Yes, I prefer them

  • (5) Yes, I strongly prefer them

Appendix 2. Modified questionnaire « Social Interaction Questions»

• Q1. Are you currently involved in a socially active environment?

  • Yes (1 pt)

  • Maybe (0.5 pts)

  • No (0 pts)

• Q2. How often do you meet new people on average per month?

  • 0 (0 pts)

  • More than 1 (0.25 pts)

  • More than 5 (0.5 pts)

  • More than 10 (0.75 pts)

  • More than 15 (1 pt)

• Q3. How likely are you to exchange phone numbers or social media information when interacting with a new person?

  • Unlikely (0 pts)

  • Somewhat likely (0.33 pts)

  • Likely (0.66 pts)

  • Very likely (1 pts)

• Q4. How often can you get the correct contact information after an exchange?

  • I always get the right information (name, phone number, etc.) when I exchange information (1 pt)

  • I sometimes get the right information (name, phone number, etc.) when exchanging information (0.5 pts)

  • I never get the right information (name, phone number, etc.) when exchanging of information (0 pts)

• Q5. After getting to know a new person, how likely will you get an SMS or reply from them?

  • Never (0 pts)

  • Sometimes (0.5 pts)

  • Always (1 pt)

• Q6. would you be willing to use a platform that can help exchange social information easily?

  • Yes (1 pt)

  • Maybe (0.5 pts)

  • Absolutely not (0 pts)

Appendix 3. Some detailed calculations

3.1 Calculation of update of emotional profile of learner

For example, a learner $x$ generated during a learning activity 25 facial expressions distributed as follows: happiness: 5; anger: 4; surprise: 2; disgust: 3; neutral: 3; fear: 4 and sadness: 4. $emoDegPos\left(x\right)= 5 \times 0.91 + 4 \times 0.23 + 2 \times 0.64 + 3 \times 0.21 + 3 \times 0.50 + 4 \times 0.29 + 4 \times 0.25 =4.55+0.92+1.28+0.63+1.50+1.16+1 = 11.04.$ We average the sum divided by the number of facial expressions generated (i.e., out of 25), and we get $emoDegPos(x) = 11.04/25 = 0.4416$.

3.2 Initial relevance degree formulas of each type of support resource

Table 7

Table 7 Profiles, relevant preferences, and the formulas for calculating the initial DR of the various support resource

3.3 Illustrative example of the emotion-based adaptation process

Table 10 shows an example of a scenario using our approach to adapting ten support resources based on the different information in the learner model. The learner's initial profiles (i.e. cognitive ${CP}_{init}(x)$, behavioral ${BP}_{init}(x)$ and emotional ${EP}_{init}(x)$ and the degrees of preference of the different learning styles ${degPrefStyle}_{sty}\left(x\right)$ and those of activity preferences ${degPrefAct}_{sty}(x)$ are described in Table 8.

Table 8 Example of the values of the different information in the learner profile

Using the formulas presented in Table 7 (Appendix C), we calculate the initial degrees of relevance of these six support resources. Table 9 describes these SRs, the type of each and the initial degrees of relevance calculated.

Table 9 Initial degrees of relevance of the various SR in this example

Let's assume that the learner will follow the following scenario during his learning process: Initialization → SR selection (movie, joke) → collaboration → SR selection (reinforcement exercise) → evaluation → learning → SR selection (soft music) → evaluation → SR selection (game1) → evaluation → learning → SR selection (collaborative exercise) → learning. After each activity (collaboration, learning, evaluation or communication), the system updates the different learner profiles (column "Update profiles" in Table 10). Then, it updates the degrees of relevance of the support resources that are activated (selected) with formula 7 (which takes into account several adaptation criteria), and then the system updates the list of support resources that will be recommended and which are the top five ranked by DR, as shown in Table 10.

From Table 10, we can see that after initializing the ten SRs, the relaxation exercise is the most relevant SR for the learner (with a degree of relevance equal to 0.6). This is justified by the learner's preference for this activity (degree of preference equals 0.7). Although the learner prefers educational games (degree of preference 0.8), these will be the least relevant for the learner compared to the relaxation exercise, as educational games have an inverse relationship with initial cognitive and behavioral profiles that are above average, as shown in the formula of initial DR of educational game (see Table 7).

According to this scenario, the learner begins by using two SRs for a period of time: watching a film and jokes. At the end of this activity, the learner's emotional and behavioral profiles will be updated with $EP(x) = 0.9$ and $BP(x) = 0.7$ respectively (only these two profiles, the cognitive profile CP(x) only changes after an evaluation activity, see algorithm 2). After completion of this activity, the degrees of relevance of these two SR ${dr}_{movie}$ and ${dr}_{joke}$ will be increased from 0.55 to 0.8125. For example, for the support resource "movie", the set of importance coefficients for this SR (which is a visual resource) is::$({ic}_{EP,visRes}, {ic}_{CP,visRes}, {ic}_{BP,visRes}, {{ic}_{drInit,visRes}, ic}_{drHist,visRes}$) = (6, 0, 0, 1, 1). For the last k DPs of this RS, since there is only one DP update for this RS, we take k = 1 instead of the 3 we set. Concerning the initial degree of relevance for the support resource “movie”${dpInit}_{movie}$, it is equal to 0.55. We now apply the formula 7:

$$\begin{array}{c}{dr}_{movie}= \frac{{ic}_{EP,visRes}EP(x) + {ic}_{CP,visRes}CP(x) + {ic}_{BP,visRes}BP(x) + {ic}_{drHist,visRes}\frac{1}{k}\sum_{i=1}^{k}{drHist}_{movie}[i] + {{ic}_{drInit,visRes}drInit}_{movie}}{{ic}_{EP,visRes} + {ic}_{CP,visRes} + {ic}_{BP,visRes} + {ic}_{drHist,visRes} + {ic}_{drInit, visRes}}\\ {dr}_{movie}= \frac{6\times 0.9+0\times 0.6+0\times 0.7+1\times \frac{1}{1}\left(0.55\right)+1\times 0.55}{6+0+0+1+1} = \frac{6.5}{8} = 0.8125\end{array}$$

The same for the supporting resource 'joke', we obtain ${dr}_{joke}$= 0.8125. The new order of SRs is as follows (degree of relevance in parenthesis) 1) film (0.8125), 2) joke (0.8125), 3) relaxation exercise (0.6), 4) game1 (0. 51), 5) game2 (0.51), 6) collaborative exercise (0.48), 7) reinforcement exercise (0.45), 8) soft music (0.4), 9) contact teacher (0.38), 10) motivational message (0.3). The first five support resources will be recommended to the learner. We note that the SRs the learner selected became more relevant because the learner's emotional profile EP(x), which is of great importance for these two SRs (${ic}_{EP,visRes}$ = 6), was increased after using them.

Table 10 Scenario for using our adaptation approach with ten support resources

Appendix 4. Implementation and use of software for emotion-based adaptation of support resources

This appendix provides supplementary information on how to install, use and modify the software that we developed for adapting the support resources based on the learner’s emotion. The software consists of several plugins that extend the functionality of the Moodle platform version 3.11.5, which runs on MySQL 5.7 and PHP 7.4.

4.1 Installation of software

Before you can install plugins, you need to install Moodle 3.5.11 on the webserver. To do that, download and extract the Moodle zip file to the htdocs folder (or www folder, depend on webserver), then run the installation wizard from the browser. When the installation is complete, you will need to create an admin account and a site name.

After that, you can pass to install plugins as follows:

• 1. Download the zip file of the software from this link.⁵ The zip file contains several zip files, one for each plugin.

• 2. After login into Moodle as an administrator, select Site administration on the left. Select Plugins, then Install plugins. Moodle will display a page where you can install new plugins.

• 3. Drag and drop the zip file of the plugin that you want to install to the upload plugin box. Moodle will upload the plugin and ask you to confirm the installation.

• 4. Click on the install plugin button and follow the instructions. Moodle will install the plugin and display a confirmation message.

• 5. Repeat the steps for the other plugins.

4.2 Usage of software

The usage of each plugin depends on the type and purpose of the plugin. Here are some examples of how to use the plugins in a Moodle course:

• Educational games plugin (mod_game): To create a new educational game activity, the teacher must go to the course page and turn on the editing mode. Then, the teacher should click on the add an activity or resource link and select the educational game option. The teacher should give a name and a description for the game and choose the type of game from the drop-down menu (crosswords, millionaire or sudoku). The teacher can also customize the game settings, such as the difficulty level, the number of questions, the feedback options, etc. The teacher should click on the save and return to course button to create the game. We note that we are adapting the Moodle games plugin to design educational games that act as support resources (Fig. 16).

• Collaborative activity plugin (mod_collabora): To use the collaborative activity plugin, the teacher needs to create a new collaborative activity by following the same steps as for the educational game activity, but selecting the collaborative activity option instead. The teacher can also set the group size, the time limit, the evaluation criteria, etc. The plugin will create a collaborative code editor and compiler for each group and assign the learners randomly or manually. The plugin will also provide a chat space for each group to communicate and exchange ideas (Fig. 17).

• Initialization plugin (local_init): To initialize the learner's profile, the learner needs to log in to the Moodle platform and complete a questionnaire that asks about their preferences and learning styles. The questionnaire is automatically generated by the initialization plugin and can be accessed from the course page. The plugin will analyze the answers and assign the initial values for the profile attributes and the degrees of relevance of the support resources.

• Support resource manager plugin (local_supportres): To manage the support resources, the administrator needs to go Manage support resources in the left sidebar. The plugin will display a list of the existing support resources and their types. The administrator can add a new support resource by clicking on the add button and filling in the required fields. The administrator can also view, edit or delete an existing support resource by clicking on the corresponding buttons. He can also weigh the importance coefficients of SR by clinking on Config. The plugin will update the degrees of relevance of the support resources accordingly.

• Facial expression recognition plugin (blocks_fer): To use the facial expression recognition plugin, the learner needs to launch a pedagogical activity, such as a quiz, or a collaborative activity. The plugin will start the learner's webcam after asking for their permission (Figs. 18 and 19) and begin the emotion recognition process. The plugin will display the result of the recognition in a block on the side of the activity, as shown in Fig. 8. The plugin will also update the learner's emotional profile based on the detected emotion. We note that the facial expression recognition model is developed using Python 3 and uses TensorFlow.js⁶ to integrate the model in the web browser.

• Support resource recommendation plugin (blocks_rsrec): To use the support resource recommendation plugin, the learner needs to go to any page of the Moodle platform. The plugin will display a block on the side of the page that shows the links of the most relevant support resources for the learner, based on their profile and the current context. The learner can click on the links to access the support resources, such as videos, articles, games, etc. The learner can also view all the support resources by clicking on the show all button, as shown in Fig. 9.

• Dashboard plugin (blocks_dashboard): To use the dashboard plugin, the administrator needs to go to the dashboard page, which can be accessed from the left sidebar. The plugin will display a dashboard that shows the importance coefficients of the support resources, as well as the statistics and graphs of the learners' profiles. The administrator can adjust the coefficients by using the input boxes, as shown in Fig. 20.

Fig. 16

Interface of the educational game "Millionaire”

Fig. 17

Collaborative activity interface

Fig. 18

Support resource manager

Fig. 19

Webcam authorization request message

Fig. 20

Dashboard interface for weighting coefficients of importance of SR

4.2.1 Modification of the software

The modification of the plugins requires some knowledge and skills in programming, web development and machine learning. The code is written in PHP, HTML, CSS, JavaScript and Python.

The code follows the Moodle coding guidelines and the plugin development standards. The code is structured and documented according to the type and purpose of the plugin. Here is the list of files that work the same way in all the plugins we develop:

• /plugintype/xxx/version.php: Metadata about the plugin (like version number, dependencies, etc.) is defined in this file.

• /plugintype/xxx/lang/en/plugintype_xxx.php: the English character strings of the plugin are defined in this file. This mechanism allows to obtain text strings of the language to use in the user interface.

• /plugintype/xxx/lib.php: The interface between Moodle core and the plugin is defined here for most plugin types. The expected contents of the file depend on the type of plugin in question.

• /plugintype/xxx/db/install.xml: the plugin database schema (tables, fields, indexes and keys) is defined here.

• /plugintype/xxx/db/access.php: plugin capabilities are defined here. The Access API can gives functions to determine what the current user is allowed to do. It also allows modules to extend Moodle with new capabilities.

• /plugintype/xxx/db/services.php: external functions and web services provided by the plugin are described in this file.

• /plugintype/xxx/db/events.php: in this file, we define the list of all the events that the plugin wants to observe and of which it wants to be notified.

• /plugintype/xxx/styles.css: The plugin CSS is stored in this file.

• /plugintype/xxx/amd/: the plugin's Javascript modules are written in this folder in ES6 format. We have for exemple the file /Blocks/fer/amd/src/fercam.js responsible on facial expression recognition task.

With xxx is replaced by the name of the plugin and plugintype by the type of plugin (mod, blocks or local).

Please note that the code of the plugins is protected by a password and requires the authorization of the author to access and modify it. The author needs the authorization of the laboratory to share the code with other researchers or developers. Therefore, if you want to use or modify the code of the plugins, you need to contact the author first and obtain their permission.