Keywords

1 Introduction

Emotions play a significant part in our daily lives. These emotions range from feelings of happiness, anger, sadness, etc. Since it is natural for human beings to seek happiness, these feelings impact our overall being [1, 2].

Emotions also play a major role not only in decision -making but also in enhancing learning. Studies on human to human and human to computer intercommunication show that for effective interaction to happen, emotions must be given due importance. By including emotions in human-machine interaction, machine would be capable of detecting the emotional condition of the user and make appropriate adjustments [3]. Affective computing is a discipline that that has user emotions during computer interaction as its focal point [4] and whose primary objective is to develop systems that can recognize and respond to these emotions [5].

Developing a system that recognizes user emotion is not easy as it involves carrying out numerous tasks such as gathering and representing data and system training. Nevertheless, many methods for recognizing emotions of computer users have been developed. Some of these are based on analyses of keyboard and mouse movements [6].

Studies by Salmeron-Majadas, et al. [7] and Grimes, et al. [8] used keyboard and mouse interactions along with the Self-Assessment Manikin (SAM) because it provides an easy to understand way of describing emotions [9, 10]. SAM has been an effective tool in measuring responses to various stimuli such as pictures, images, sounds, pain, etc. [11,12,13].

Most studies on emotion recognition require some sort of special equipment to detect emotion. The current study aims to develop an emotion recognition system by collecting keyboard and mouse data from computer programming students in a non - intrusive manner. The collected data were used, together with the students’ answers to SAM questionnaires, as training set and test set for RapidMiner to generate patterns to figure out the emotions of the user. This method provides two main advantages: (1) it does not require any expensive, specialized equipment and (2) it is non -intrusive as it does not meddle with the user’s normal computer activities.

2 Related Literature and Studies

Majadas and Santos [14] listed that the relationship between mouse and keyboard use and the user’s affective state indicators could be valuable in detecting affective states in non-interfering and inexpensive manner. Ball and Breese [15] further stated in their study that high levels of activity with other applications (typing and mouse clicking rates) also are indicative of high task orientation. Thus, observation of such behaviors should be interpreted as evidence that the user is highly oriented towards task completion.

Bixler and D’Mello, for their part, stated that detecting affect plays an important part in creating applications that converse with and aid users in completing their task. Their study used analysis of keystroke to create affect detectors. This approach requires no additional hardware and presents an effective and unobtrusive method of data collection [16].

The importance of emotion detection is further supported by Zimmerman et al. [17], which stated that in intercommunication between human and computers, expressed emotions have a big role and most people interact with the technology naturally and socially. Importance in HCI research is increasing and the interaction system for emotional intelligence is used to express and respond to emotions of humans. The study is centered on another method of behavior using standard mouse and keyboard devices for measuring user affect and extracts parameters from motor-behavior using log-files of keyboard and mouse actions.

The study considered one of the methodology used for getting information from keyboard-stroke to identify if the emotional state is positive, negative, emotionless or neutral state as the user interacts with the computer. It includes the rate of mouse clicks, the rate of keystrokes as well as the keystrokes average duration as the analyzing parameters.

In the study, Zimmerman, et al., were able to know the outcome of induced effects based on the parameters of motor behavior when the user is using the computers including film clips showing, and doing online shopping. The ninety-six (96) participants were tested using the different mood states as independent variable. The created procedure was used and conducted at the laboratory utilizing the use of sensors and bands that measures respiratory attached to the participants. The data questionnaires displayed on the computer were answered, as well as self-assessment questionnaires, by the participants. The mouse and keyboard actions from the log - files are then analyzed [17].

Another research conducted by Rodrigues, et al., states that when a student attends an e-learning course there is a poorer interaction between student and teacher. There is no verbal interaction and it is difficult for the teacher to assess the feelings and attitudes of the students. The study developed a tool that will estimate the stress level of users without meddling with the user’s normal computer activity. Developing dynamic stress estimation model is the main goal of this research. For this research, data collected are click duration and accuracy, amount of movement, mouse movement, mouse clicks and keyboard strokes. It tested 10 programming students to validate the possibility of user’s stress detection and analysis of the mouse and keyboard use. It is concluded that stress has significant effect in learning capacities and learning speed [18].

3 Methodology

This study’s aim is the detection and identification of the emotions based on the use of non-invasive tools such as the ordinary mouse and keyboard. As shown on Fig. 1(a) the Flowchart for Emotional Classification based on Shukla and Solanki [19]. Data collection is done by performing feature selection and extraction and data labeling follow based on the questionnaire, fuzzy vector and label has been applied for the training of the data and from that classification has been identified. Using data preprocessing and applying the fuzzy vector, the research can display the emotional condition of the user. On the other hand, Fig. 1(b) shows the flowchart of what this research paper is trying to solve.

Fig. 1.
figure 1

(a) Flowchart for emotional classification [19] (b) Flowchart for emotional recognition

Full size image

3.1 Data Gathering Tool

A data gathering prototype program was designed and implemented as a tool used to collect the data needed. The data gathering prototype has a survey question with Self-Assessment Manikin (SAM) images that requires the students to select [19]. The program records the keyboard keystrokes together with mouse clicks and movements then save individually in the .xls (mouse and keyboard) and .txt as the backup files.

The students have also been provided with programming questions for them to answer using Java programming language.

3.2 Data Gathering Procedures

This research intends to identify and analyze the emotional condition of the computer user. During the data gathering the following actions were conducted by the researchers: (1) randomly selected six (6) participants (three (3) males and three (3) females) who have programming course from a select academic institution. Informed consent form was filled up by the participants. (2) Orientation was done about the assessment that the participants need to answer but with no bias on what data will be gathered from them. (3) Two (2) laptops were utilized facing each other with installed prototype data gathering program. (4) The participants were given twenty (20) minutes to answer the programming problem assigned to them. (5) Java Programming Language will be used by the participants in answering the problem. (6) The Prototype Data Gathering Program is already running and it captures the features on the given time. The questionnaire will pop -up every three (3) minutes. (7) The recorded keystrokes and mouse movement and clicks is saved into an excel file. It is then converted to .csv (comma delimited format) to be used as an input operator for data mining.

3.3 Feature Extraction

Feature extraction includes lessening the measure of assets required to portray a vast arrangement of information. When performing analysis in the complex data, one of the real issues comes from the quantity of factors included. So, before starting to train, before beginning to prepare a classifier, it is important to play out an element choice strategy to dispense with the parameters which don’t relate with the class names and in this manner, decrease the dimensionality [20]. In this research, the plan is to characterize few profoundly solid components indicating certain feelings angle, for example, pleasant, calm, etc. Every element will identify the emotion level of the user. Name of the participant, age and sex were also extracted together with the keyboard and mouse data. The keystrokes, transition between key press, mouse x position, mouse y position, action, delay (dwelt time per milliseconds) and execution time [19]. The chosen emotional condition of the student is then mapped every time the student uses the mouse and the keyboard and based on the answer from the Self - Assessment Manikin.

From the extracted file, this research has acquired 14, 889 rows of recorded mouse and keyboard data, which will be used for the training and testing using the best algorithm available.

3.4 Model Development

This research developed a model to be used in the prototype program that will identify and analyze the emotion of the user using keyboard and mouse. The extracted feature is then analyzed using free open -source software available in the web like RapidMiner. The researcher selected 80% from the mapped file to be used as the TrainingSet and the remaining 20% is used as the TestSet. The two files was saved as .csv (comma delimited format) to be used as an input data in the RapidMiner. RapidMiner has been chosen by this research because of its user interface that is very friendly. There is more than one approach in labeling emotions and these methods include human labeling and the other is automatic labeling [16]. There are various kinds of labels that people can select from the word labels list from the questionnaire and uses scales to present the level of responses to each question [likert]. The other one is the use of Self-Assessment Manikin (SAM) which arousal and valence are presented and expressed graphically [9, 10]. The SAM is a pictorial evaluation tool intended to know the emotional measurements specifically by methods for two arrangements of graphical image. This method has been used in different studies and application [21, 22].

3.5 Classification

For classification, since the label for the data set is nominal, the researcher used five (5) different classifiers such as Decision Tree, Random Forest, Gradient Boosted Trees, Decision Stump and Random Tree. The result of training is shows that Gradient Boosted Trees presented the most acceptable model which has an accuracy of 70.25% and Kappa of 0.612%.

3.6 Model Testing

Model testing is done to verify that the gradient boosted tree has high accuracy and kappa using the test set data which is the 20% of the gathered data from the participants. The testing shows that the accuracy rate of the test model is 79% which is 13% higher than the Training set with the kappa of 0.75% which is 23% higher than the Training set.

4 Results and Discussion

The machine learning algorithm called Gradient Boosted Trees also known in other terms as XGboost [25], has function called Huber-M Loss generates a series of decision trees for loss data to return fit for classification [21]. The comparison of XGboost to other machine learning algorithms has highest accuracy rate 70.25% and kappa rate 0.61% when used to generate model based on the pattern and classes collected. It is because however, the function Huber-M Loss therein presented allows this algorithm to fit Loss data by generating another trees to the missing values. The accuracy indicates the highest precision point of all the classes. The classified correct example compared to the number of all examples known as accuracy is then calculated as a set of random input x = {x1,…xn} using the training set sample {Yi, Xi} or known values, and the positives ratio to all as positive predicted examples which is the precision is calculated based on the number of iteration and learning rate. The ratio of positives to all actually is positive examples known as recall is calculated based on splits nonterminal node [21].

Base on XGBoost formula the highest predictions are always on left which means the milliseconds delay value between 0.212 to 0.243 whether Male or Female after more than a second that participant starting, the “PLEASED NEUTRAL” is predicted, and most efficient compare to the right thereby the rest to the right belong to the category (Male/Female) is void.

The actual interface of our prototype on the figure below consists of only one button; it designs to be user friendly and straightforward (Fig. 2).

Fig. 2.
figure 2

Sample prototype

Full size image

The prototype program automatically generates .XLS and .TXT files when it is started, just by defining its proper directory and filename. The TXT file produce the same data as XLS does but is prepared for retrieving or viewing through web.

It is possible to predict emotion using SAM questionnaire [26], keystrokes and mouse clicks detect stress [18] although this paper uses different subject and procedures but closely similar to the application of keystrokes dynamic [19]. The findings of this research not necessarily good but different where it was tested the students on programming subject, for future works should explore additional parameters such as to check program error or if the student think while doing the programming task how it will affect emotion.

5 Conclusion

Intelligent computers have important roles in society today, with its full capacity like its ability to recognize emotions that can be used to know the needs of the users. Extracting the emotional aspects of the human with the use of computers system could help the decision of the user on how would to interact with the computer. Thus, adapting to this new system that could identify and analyze the emotion of the person using the computer is a big help not only to the academe but also to the general user. The current approaches in emotion identification with the use of computer requires expensive equipment and is intrusive in nature. This paper has presented one way that could be used to identify and analyze the emotion in a non -intrusive and inexpensive way. Using the keyboard and mouse provides an alternative way of detecting the emotions of the students. Identifying these emotions and taking the appropriate actions can be of great help for teachers in assigning computer related activities to students thereby helping in the learning process.