1 INTRODUCTION

The popularization of web technologies has enabled technology-enhanced learning [1–3]. Technology-enhanced learning is the learning supported by self-motivation, communication, efficiency, and technologies [1]. These educational technology systems' goal is to provide intelligent, one-on-one, computer-based support to students [4]. At the beginning, these system’s construction was focused on the instructional and the artificial intelligence techniques used to integrate the intelligence into these systems. In recent years, these tutoring systems have evolved, analyzing and including emotional and affective factors, which are also crucial in the learning process [5]. The integration of affective factors to the learning systems allows making the learning materials more flexible, modifying educational approach according to competence, affective states, and preference of the student and the learning tasks [1]. The integration of affective factors included motivation and feedback [6].

Feedback helps reduce discrepancies, guided student’s attention, and evaluates student’s knowledge [7]. Several researchers classified feedback into affective, social, and informational, but other theories classified it into normative, formative, corrective, positive, and negative [7]. The feedback can be used as a tool to improve motivation [8–10]. Motivation is an essential element in the learning process [11]; it is defined as emotional arousal that leads to a conscious decision to act and gives rise to a period of sustained intellectual and physical effort to attain a previously set goal [12]. Keeping students motivated to learn is a crucial challenge for any learning system [8, 9, 13]. Positive affective feedback could encourage learners to keep them studying and interacting with a tutor, while that negative affective feedback could stop such interaction [14]. Several studies stated that the inclusion of affective feedback in the tutoring systems benefits the student’s motivation [8, 9, 13, 15]. However, the affective feedback perception is determined by several factors such as gender, academic performance, learning style, intellectual ability, and personality [16–18].

The literature reports that affective feedback was more beneficial for students facing learning difficulties because they need more support and motivation than the others [8]. Gender is another variable that impacts the student’s motivation to learn. Previous studies suggest that affective feedback benefited female students more than male ones [8, 15].

Personality is a factor that influences people’s behavior and their perception of the context [14]. Several models classify personality traits, but one of the most documented and used is the Five-Factor Model (also known as the Big five) [19]: Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism.

Deniss et al. [14, 20] stated a link between personality, motivation, and affective approaches. Besides, in [13], the authors analyzed how affectivity impacts extrovert students. Their findings suggest that extraversion is not related to the perception of the positive affectivity feedback on student motivation to learn. The previous work investigated only the relation between one factor of the Big Five model (extraversion) and student motivation to learn, which would depict a reduced view about that phenomenon. Thus, the present study’s main objective is to analyze the influence of affective feedback on students' motivation to use the system based on their personality factors. The research includes statistical analysis and a Naïve Bayes classification.

The rest of the paper is organized as follows. Section 2 describes state of the art in the area of affective feedback. Then, Section 3 explains the formal representation of the affective feedback approach. Next, Section 4 describes this model’s evaluation process, and Section 5 presents its results and discussion. Finally, Section 6 concludes the study and suggests future work.

2 RELATED WORK

This study considers motivation as an emotional arousal, which leads to a conscious decision to act and gives rise to a period of sustained intellectual and physical effort to attain a previously set goal [12].

Motivation in education is tough to measure [21]. It could be because the motivation to learn is challenging to describe operationally. The key to measuring motivation must be to look for behaviors indicating high motivation and low motivation [21].

There are also several theories about learner motivation. For example, the ARCS model [22] identifies four factors that facilitate motivation, namely the learner’s attention, the relevance of instruction to the learner (goals, experience, learning styles), the learner’s confidence (including self-efficacy and attribution theory), and satisfaction (establishing a positive feeling towards the learning experience) [20]. The following concepts are significant in the study of motivation.

Interest: A learner’s characteristic that influences his/her ability to learn, preferences, and motivation [23, 24].

Effort: A student’s exertion of physical or mental power for a specified purpose [22].

Persistence: A learner’s quality to be insistent for a long time in a statement, homework, activity, or request [24].

Choice: A learner’s selection that can be autonomous in their abilities to make a learning-related decision regardless of the time and location [25].

Satisfaction: A student’s confident acceptance of something as satisfactory, dependable, or true [26].

Goals: The learner’s result or achievement toward which she/he leads her/his effort [21].

2.1 Personality

Personality is defined as a person’s nature or disposition [20]. It affects all areas of our lives; it governs who we are and how we respond to life challenges. Personality is a complex trait, which has led to the development of theories of personality. Many theories identify a set of factors; the Five-Factor Model [19] is a well-respected construct that was formed as an aggregation of many other models. The features analyzed in [19] are the following five: (1) extraversion, which refers to being talkative, energetic, assertive, and social; (2) agreeableness, which characterizes a being agreeable, cooperative, and trustful; (3) conscientiousness, which covers the characteristics of being organized, disciplined, responsible and achievement-oriented; (4) neuroticism, that refers to being worried or insecure, and related to the degree of emotional stability and anxiety; and (5) openness to experience, demonstrating a high degree of intellectuality, imagination, and independent-mindedness.

Each previously described trait splits into smaller facets, enabling a most in-depth analysis [19].

2.2 Related Work

Previous studies in literature concluded that affective feedback benefited student learning outcomes [24, 27]. D’Mello et al. [27] confirmed this asseveration. They found that input based on affective phrases improves student learning outcomes. They integrated a set of positive, neutral, and negative phrases to provide affective feedback. However, they included negative feedback that could negatively affect students. Previous work shows that affective feedback improves learning to some degree [24, 27], but not in the same way for all students. Student profile plays a significant role in this situation. The student profile contains the student’s relatively constant attributes such as gender, academic performance, age, and personality traits [24]. For instance, affective feedback is more beneficial to students with low academic performance than students with high academic performance [27, 28]. A possible explanation could be because the latter has a high level of intrinsic motivation to learn, and they do not need external factors [9].

Letzring et al. [29] hypothesized that the extroversion factor is the most related to affectivity. However, in [13] analyzed the influence of extraversion on student motivation to learn and their perception of the degree to which the study time is enjoyable, and they concluded that there was no statistical evidence to support extraversion influences in these two variables.

The present study examines the complete Big Five approach, including the four remaining factors. Analyzing these factors can help explain the effect of personality on student motivation to learn.

3 AFFECTIVE APPROACH

The affective approach proposed in this study base on a tutoring system with three components: (1) a dialogue act taxonomy to classify utterances, (2) a grammar to build the phrases, and (3) a set of affective phrases proposed by students. Figure 1 shows an example of the interaction between the feedback system and a student. Firstly, the system provides a greeting phrase to a student. Then, the system monitors students' actions, and then the student asks a question to the feedback system. In response, the system explains to the student an affective phrase (constructed by the grammar). The figure also shows part of the grammar used by the system to build the phrases.

Fig. 1.
figure 1

An example of the interaction between the feedback system and a student.

Full size image

3.1 Implementation

The proposed affective feedback system (AFS) was implemented as an instant message system using an MVC pattern. The AFS is autonomous and independent of a particular learning system (http://tipoodev.appspot.com/). However, it needs the learning system’s resources such as learning materials, concepts, links, and users’ information.

The system’s architecture in Fig. 2 is from a tutoring system proposed called TIPOO [13]. The proposed AFS is an external component of TIPOO. TIPO and the affective dialogue system provide interaction messages in Spanish, [13, 30] describe its implementation and architectural details.

Fig. 2.
figure 2

System’s Architecture.

Full size image

The proposed AFS includes an auto-completion mechanism to help students ask for a concept definition of the tutoring system. The student asks the system using a chat box. Then, the system generates the responses, the greeting, positive exclamation, and the explication phrases. The system’s model records all the interactions and student information. Figure 3 shows an example of the interaction between the student and the system.

Fig. 3.
figure 3

Interaction between a student and the system.

Full size image

4 METHODOLOGY

This study based on a quasi-experimental design method in which there are pre-test and post-test stages [31].

4.1 Design and Settings

The study recruited a sample size of 30 students from a public university in Mexico enrolled in a career related to computer science. A non-probabilistic [32] sample was ascertained from students aged between 20–23 yr (M age = 21, SD = 21.23).

The experiment had a duration of three weeks. Each week had two sessions of 40 min. In the first session, the students answered a demographic information survey and a personality test to create a student profile. The demographic survey collected the following data: gender, grade point average, and ongoing semester. The participants answered a short version of the well-known IPIP-NEO personality test [19]. It has 120 items, provides scores to the five factors, and classifies it on a 3-point scale (from low to large). Before the student-AFS interaction, students answered the Motivation Assessment Instrument (the results are called later motivation prior-interaction). In the last session, the participants created an account in the TIPOO system.

In Mexico, the grade point average score ranged from 0 to 100. Students with low academic performance were those who had a grade point average lower than 80. On the other hand, high academic performance students had a grade point average equals to or greater than 81.

In the subsequent four sessions, the students interacted with TIPOO for 40 min in each session. In these sessions, the participants used TIPOO to study several topics about web programming. During the interaction, the students asked to the system some definitions about different concepts; then, the system answered the concept’s explanation followed by an affective phrase. In the final session (sixth session), students answered the MAI (the instrument results are called later motivation to learn post-interaction).

4.2 Survey Development

For extracting the critical variables to measure the student’s motivation to learn, the instrument’s design based on a literature review process. To the best of our knowledge, the literature does not report a tool for evaluating students' motivation to learn when the student’s dialogue with a virtual environment such as tutoring systems. The present study focused on user attitudes. For that reason, the instrument design follows Cohen’s [33] method.

The original Motivational Assessment Instrument composed by ten items and one theoretical factor (Motivation to learn), all the items had five response options (from 1 = Strongly disagree to 5 = Strongly agree). Then the Cronbach’s alpha reliability test was calculated to evaluate the internal consistency of the instrument. After that, four items were deleted to increase the alpha value. Table 1 shows the variables and their correspondent items.

Table 1. Variables that indicate motivation to learn in students
Full size table

The next step conducted to an Exploratory Factor Analysis (EFA). Initially, the authors examined the factorability of 6 items. The minimum amount of data factor analysis satisfied with a final sample of 30, providing a ratio of over five cases per variable. Nunnally et al. [34] propose as necessary from five to 10 cases per item to estimate the sample size; then, it is feasible to run a factor analysis. Firstly, the authors observed that all the items correlated at least 0.3 with at least one other item, suggesting reasonable factorability. Secondly, The Kaiser-Meyer-Olkin Measure of Sampling Adequacy was acceptable (KMO = 0.601), the commonly recommended value of (KMO = 0.60) [35], and Bartlett’s test of sphericity was significant (Chi2 = 135.02, df = 15, sig = 0.001). Also, the authors examined internal consistency using Cronbach’s alpha. This value was high: alpha = 0.84. The authors eliminated one variable because it did not contribute to a simple factor structure and failed to meet a minimum criterion of having a primary factor loading of 0.4 or above and no cross-loading of 0.3 or above. The item motivation to learn loaded negatively in a second factor.

For the final stage, an EFA re-run with the remaining five items, using a principal components factor analysis as the extraction method and Oblimin rotations. All the items in this analysis had primary loading over 0.6. Bartlett’s test of sphericity was significant (Chi2 = 79.77, df = 10, sig = 0.001), and the Kaiser-Meyer-Olkin Measure of Sampling Adequacy was acceptable (KMO = 0.698). The Cronbach’s alpha of the final instrument was 0.84. We deleted the items that were not in a group in the theoretical factor that we propose. The variance was 73.071%, above the recommended 50% [35]. Table 2 presents the factor loading matrix for this final solution.

Table 2. Factor loadings and commonalities based on a principal components analysis with Oblimin rotation for five items
Full size table

4.3 Variable Classification

The student’s motivation to learn is the dependent variable of this study. The Motivation Assessment Instrument helps to analyze that variable. Then, the stanine scale supported a division of the scores' distribution into three sections, from 1 to 3 [36]. Thus, a linguistic scale contributed to a better understanding of the scale: unmotivated, moderately motivated, and highly motivated. The proposed study includes an analysis of how the student’s personality moderates the motivation to learn using an affective approach.

The personality test results helped assign a score to the student profile, and these results considered the five personality factors of the OCEAN model (Openness, Contentiousness, Extraversion, Agreeableness, Neuroticism) [19]. The IPIP-NEO describes whether a person’s score indicates a low, average, or high level of each factor.

4.4 Statistical Analysis

An electronic questionnaire supported the data collection. Then, the authors applied a Shapiro-Wilk (S-W) test to the 30 constructs to determine whether these variables were normally distributed. The S-W test for normality statistics suggested that normality was an unreasonable assumption (Extraversion, S-W = 0.798, p-value = 0.001; Agreeableness, S-W = 0.774, p-value = 0.001; Conscientiousness, S-W = 0.769, p-value = 0.001; Neuroticism, S-W = 0.813, p-value = 0.001; Openness, S-W = 0.595, p-value = 0.001). Hence, the authors applied the Chi2 test for this construct.

5 RESULTS

This section presents the influence of affective feedback in students' motivation to use the system based on student’s personality.

5.1 Sample Characteristics

The sample included 30 participants, the majority were females (58.6%), and more than a half (51.7%) had low academic performance. Most participants had a low Openness score (73.3%), a low Agreeableness score (46.7%), and a low Conscientiousness score (46.7%). Similarly, the majority had an average score of Extroversion (56.7%) and Neuroticism (50%).

5.2 Descriptive Statistics

Table 3 shows the student’s responses. The majority of participants (80%) stated that the affective feedback motivates them to reach learning goals using the affective approach (MotGoals) and 20% of the participant was undecided. This 20% was a set of students with a low conscientiousness score. Moreover, 76.7% of the students agreed that the affective feedback motivated me to continue studying (MotStudy), 16.7% of students were undecided, and 6.7% of participants disagreed with this statement. This 6.7% was a set of students with a low openness score. Among the study participants, 76.7% agreed that affective feedback arouses their interest in learning (InterestStudy) whereas 3.4% of students disagreed, and 20.7% were undecided. All the students in disagreement with that affirmation had a high neuroticism score. In the item, the affective feedback motivates me to dialogue with the system (MotDialogue), 76.7% of participants agreed with the statement, 20.7% were undecided, and 3.3% disagreed. Among the participants in disagreement, 100% reported a high neuroticism score. The statement’s results the affective feedback makes friendly my study time (MakesNiceTime) says that 83.3% of the participants agreed whereas 16.7% disagreed.

Table 3. Motivational Instrument items (N = 30)
Full size table

The authors calculated the motivational level using the stanine test and the linguistic classification described in Section 4. Table 4 shows the motivational results post-interaction.

Table 4. Motivational level prior and post-interaction results (N = 30)
Full size table

5.3 Prior Motivation vs. Post Motivation

This study compared the student motivation to learn prior-interaction and post-interaction with the AFS. Table 5 shows the motivational results prior to interaction.

Table 5. Confusion matrix where a is moderately motivated and b is highly motivated
Full size table

From the study’s data (Table 4), 73.3% of students were highly motivated to learn previous the interaction and keeps the same level of motivation to learn after the interaction.

Most of these students (77.3%) had a low Openness score. The 66.6% of students incremented their motivation to learn; they were moderately motivated prior-interaction and highly motivated post-interaction. The 3% of students were moderately motivated prior-interaction and post-interaction. Finally, 6% of students were highly motivated before the interaction with the system. After the interaction, they were moderately motivated. It is essential to observe that those students who increased their motivation to learn post-interaction with the AFS had an average or low Neuroticism score. Contrary, students who do not change their motivation value and those who decreased it post interaction with the AFS, had a high Neuroticism score and a low score on the rest of the factors.

5.4 Personality Factors

The 73.3% percent of students were highly motivated using the affective feedback system, and 26.7% were moderately motivated. Among the moderately motivated students, 63% have a high Neuroticism score.

The authors calculated a Chi-square test to explore the association between the motivational level and the student’s personality factors. There was a significant association between the motivational level and the Neuroticism factor (Chi2 = 10.51, p = 0.005). The students with a high Neuroticism score were moderately motivated or decreased their motivation using the AFS.

Otherwise, the results of the Chi-Square test revealed no significant association between students' motivational level, and Openness factor (Chi2 = 0.505, ns), Conscientiousness (Chi2 = 4.11, ns), Extroversion (Chi2 = 2.71, ns), and Agreeableness (Chi2 = 3.58, ns).

Likewise, the authors implemented a Naïve Bayes classifier, using Weka, to know if the personality factors can predict student motivation. This algorithm was selected because of its characteristics. It can be used for multiclass prediction; it works well with categorical variables and can be useful when the dataset is small. We used 66.6% of the data for training, and we used 33.3% for testing. The algorithm had a precision of 90%, classifying correctly nine classes, and only one class was incorrect. The analysis reported a kappa statistic of 0.7368. The confusion matrix is displayed in the following table.

6 DISCUSSION

In this study, the affective feedback approach analyzed the influence of students' basic information, such as gender, academic performance, and personality. Previous work [15] stated that female students were more motivated when they receive affective feedback. In the same way, affective feedback increased the motivation of students facing learning difficulties [13]. These previous works are the first step in constructing a student profile that could be integrated into tutoring systems to improve student-tutor interaction.

Likewise, the related work suggested two relations between personality factors and affectivity. Firstly, the extraversion is linked to positive affectivity [14, 29]. And secondly, individuals with a high score in Neuroticism (a psychological trait that defines a tendency to experience negative emotions) layout to be more negative [14]. According to the related work, the five personality traits are related to motivation and affectivity, but the results of this study suggested that Neuroticism is a determining factor for motivation to learn and affective perception.

Figure 4 depicts the results of the analysis of motivational level. These findings support a relationship between Neuroticism and the motivation to learn using the AFS, as Denis et al. [14] stated in their work. The students with a high Neuroticism score decreased their motivational level. In contrast, the student who had a low Neuroticism score increased their motivational level according to the prior-interaction and post-interaction analysis. There are two possible explanations for this phenomenon. Firstly, affective feedback does not influence students with a high neuroticism score because they do not need to support or perceive it negatively. Secondly, the affectivity level of the feedback was not adequate for these types of students. Hence, this work’s findings show that student personality plays an essential role in the perception of affective feedback support as an instrument for promoting student motivation to learn.

Fig. 4.
figure 4

The motivational results considering the Neuroticism factor.

Full size image

6.1 Internal and External Validity

The researchers hypothesize that personality impacts the students' motivation to use the system. To test this hypothesis, the researchers randomly assign a sample of participants, and they conducted a within-subjects experiment.

The researchers ensure no systematic bias hiding to the participants that they were under observation and experimentation.

A strict study protocol was used that outlines the procedures of the study. The researchers measured the potential confounding variables, such as the tool’s previous experience (the participants have never seen the AFS before). The students did not have prior knowledge of the topics studied in the AFS. None participants dropped out of the study.

According to the [37] statistics, the student enrollment ratio in 2015–2016 had risen to 3.3 million in the tertiary school level in Mexico. According to the latest data available from the Secretaría de Educación Pública (SEP) in that country, female students accounted for 51 percent of undergraduates enrolled at universities and males the 49%. On average, undergraduate students are aged between 19 and 29 yr. The majority of the students graduate under 30 yr. The [37] statistics reveal that the characteristics of the students in México are similar to other countries such as Colombia, Costa Rica, Spain, Denmark, USA, and Brazil. The sample selected in this study had similar characteristics as the countries integrating the OECD, which would give insight into the applicability of this study’s results in another context.

6.2 Limitations

Regardless of the theoretical and practical contributions described in this study, it is necessary to specify some limitations. Firstly, the study was conducted with small sample size. However, research in this field suggested that small samples, among 12–20 participants, could be useful for homogeneous groups [38, 39]. This type of sample could provide a vision of the population behavior [40]. Although the sample is small, it has similar characteristics to geographical context. Also, it was analyzed that there are other countries with student populations similar to the student population in México.

Secondly, the results of this work are restricted by the language. Different languages have non-similar features, such as the grammatical form to give semantics to writing [41]. The present study’s contribution would be useful to similar contexts, and future work would focus on the analysis of the current proposal on different cultural settings.

7 CONCLUSIONS

This study suggests that the affective feedback support motivates more students with a low score of Neuroticism than ones with a high score. Otherwise, the statistical analysis did not reveal a significant difference between the rest of the personality factors (Openness, Conscientiousness, Extraversion, and Agreeableness) and motivation to learn.

Even though affective feedback positively influences student motivation to learn, it benefited some students more than others. A possible reason could be that the affectivity level of the feedback was the same for all students. The one-size-fits-all approach is not recommended for learning processes because each student is different and has different support needs. This study’s results could be a first step to adapt the affectivity level of an AFS to a student profile, which eventually will improve the influence of affective feedback on student motivation to learn. Even if this study’s findings related to tutoring systems, these results could be relevant in different learning scenarios.

The approach outlined in this study is a basis for future work. The results of this work can also support the analysis of the impact of verbal and non-verbal communication on student motivation to learn, i.e., integrating voice messages and an animated avatar. Furthermore, the current approach might replicate in other settings for exploring its impact in larger groups involving more participants and on different educational levels. Besides, the affective student model could complement other characteristics of the student profile and, in this way, personalizing the affective feedback to this profile.

As mentioned earlier, the absence of affective feedback in tutoring system-student interaction could negatively impact student’s motivation to learn. The integration of affective feedback in tutoring systems suggests that student motivation to learn could improve when students interact with these systems. We believe that these findings are of considerable importance since it indicates that the integration of affective support in tutoring systems, considering personality traits, would also increase the student’s acceptance of tutoring systems and, consequently, students' learning interest and learning outcomes.