Keywords

1 Introduction

Automatic detection and recognition of different emotional states is a salient topic in the fast growing research field of affective computing. Emotions are complex states of minds comprised of numerous psychophysiological components, such as bodily changes, cognitive reactions and thoughts. Numerous computational models and algorithms for automatic recognition of emotions have been provided by Affective computing, which integrates knowledge of computer science, physiology, artificial intelligence and biomedical engineering.

Emotions affect all aspects of our daily life and has significant influence on our health. State of depression, anxiety and anger disrupt human immune system and thus associated with many chronic diseases. Bringing into play the current advances in IOT and sensor networks [1], smart healthcare systems should be introduce to improve overall quality of life. The development of automatic emotion recognition system can be very useful in regulating self-emotions and would revolutionize applications in education, entertainment and security.

Philosophers and psychologists presented various theories of emotions. Based on these theories, numerous methods have been developed to detect and recognize different emotions using facial images [3], speech signals [4], gestures and physiological signals. Physiological signals including EEG signals are considered to be the most useful signals for human emotion recognition due to its strong correlation with emotions and independence of people’s will. Facial images based emotion recognition have a major flaw of suppressing and intentional control of emotions while physiological signal originate from Autonomous Nervous System activity, cannot be controlled intentionally. Experimental evidence shows that physiological/bio signals can be influence by the activity of Autonomic Nervous System (ANS) and can convey information regarding human emotion [5, 6]. In this paper, we proposed human emotion recognition system using wavelet energy feature along with statistical and modified wavelet energy feature in order to improve the performance of emotion recognitions systems.

This paper is organized as follows: Sect. 2 provides a review on different factor of emotion recognition system. Section 3, illustrates the detailed methodology including dataset description, procedure for feature extraction, feature selection and classification. Performance of the proposed method and concluding remarks are given in Sects. 4 and 5 respectively.

2 Related Work

EEG based emotion recognition has gain a lot of interest and different emotion classification system have been proposed by the researchers. The results of these systems highly depends on five basic factors which includes number of participants, stimulus, emotion modeling, feature extraction and classifier. Different techniques have been employed by the researchers to investigate these factors and thus these emotion classification systems cannot be compared. However a short review of the recent work done is presented in this section.

Emotion modeling: Emotion is a mental state or feeling that arises involuntarily and comprised of different components such as feelings, bodily changes, behavior and thoughts. In literature numerous emotion models have been proposed. However the two most utilized categories are Discrete Emotional Models (DEM) and Affective dimensional model (ADM). DEM deals with six basic universal categories of emotions: happiness, surprise, anger, disgust, sadness and fear [2]. ADM deals with the description of emotions in some coordinate system. It characterize emotion into two affective parameters, Arousal and Valence. The most commonly used dimensional model is Circumplex Model of Affects (CMA) [7] as given in Fig. 1.

Fig. 1.
figure 1

Circumplex model of emotions.

Full size image

Emotion elicitation: There are numerous methods of emotion elicitation. The most widely used methods for emotion induction includes images, video clips and sound clips etc. The most popular existing databases are: IAPS—the International Affective Picture System and IADS—the International Affective Digitized Sound System facilitate the task of emotion recognition. IAPS [8] and IADS [9] provides a collection of stimuli publically to researchers in the study of emotion.

Feature extraction and classification: Different characteristics of EEG signals can be captured and used as features for classification of emotions. These features can be placed in one of two domain, time domain and frequency domain. Time domain feature can be extracted from raw EEG signal and includes mean, standard deviation etc. [10]. Frequency domain features include the power of different frequency bands of EEG signals. In addition to these, several other features extraction techniques have been presented in the literature which includes High Order Crossings [10], Discrete Wavelet Transformation [11], fractal dimensions [12] and Independent Component Analysis [13]. For classification task, several machine learning algorithms have been used as classifiers. These classifiers includes Support Vector machine [14], Neural Networks [15] and Quadratic Discriminant Analysis [16].

3 Methodology

3.1 Dataset

Recent advances in emotion recognition have increased the interest of many researchers and encouraged them to create databases containing visual, speech and physiological emotion data. These databases includes MAHNOB-HCI [17] and DECAF [18] etc. In this study, we used a public available database called DEAP proposed by Koelstra et al. [19]. 32 healthy participants (16 male and 16 female), aged between 19 and 32, took part in the experiment. The EEG and peripheral physiological signals which includes electrocardiogram, galvanic skin response, respiration, skin temperature, blood volume, electromyograms (EMG) and electrooculogram (EOG) were recorded from these subjects while watching 40 different music videos. Biosemi Active Two system was used to record EEG signals over the scalp from 32 electrodes according to the international 10–20 system as shown in Fig. 2. Preprocessing of EEG signals was performed in order to denoise the heavily distorted EEG signals from motion artifacts, EOG artifacts due to eye blinking and power supply noise. Initially EEG signals were recorded with 512 Hz sampling frequency which were down sampled to 128 Hz during preprocessing. A bandpass frequency filter from 4.0–45.0 Hz was applied to filter EEG signals. The elimination of EOG artifacts was performed using blind source separation technique. In the last of experiment every subject performed self-assessment to evaluate and rate the emotional state caused by each video using Self-Assessment Manikins. Rating was performed by participant for each music video in term of levels of arousal, valence, like/dislike, dominance and familiarity. In this paper, we used the preprocessed data released by Koelstra et al. [19].

Fig. 2.
figure 2

International 10–20 system for 32 electrodes (Marked as gray)

Full size image

3.2 Feature Extraction

Statistical-Based Features

Due to the nonlinear nature of EEG signals and brain complexity, nonlinear feature like high order crossing and fractal dimension are widely employed by researchers in recent publication. However, simple features like mean, standard deviation and band power are still considered beneficial for emotion recognition system. For 32 channel EEG data provided in DEAP dataset, we extracted statistical features in combination with wavelet based feature to improve emotion recognition accuracy. These features include

  • Mean of the raw signal

$$ \mu_{X} = \frac{1}{N}\mathop {\sum }\limits_{n = 1}^{N} X\left( n \right) $$
(1)

  • The standard deviation of the raw signal

$$ \sigma_{X} = \sqrt {\frac{\text{1}}{\text{N}}\sum\nolimits_{{\text{n = 1}}}^{\text{N}} {\left( {\text{X}\left( \text{n} \right){ - \mu }_{\text{X}} } \right)^{\text{2}} } } $$
(2)

  • The mean of the absolute values of the first difference of the raw signal

$$ \delta_{X} = \frac{1}{N - 1}\mathop \sum \limits_{n = 1}^{N - 1} |X\left( {n + 1} \right) - X\left( n \right)|\varvec{ } $$
(3)

  • The mean of the absolute values of the first signal of the standardized signal

$$ \overline{{\delta_{X} }} = \frac{1}{N - 1}\mathop \sum \limits_{n = 1}^{N - 1} |\overline{X} \left( {n + 1} \right) - \overline{X} \left( n \right)| = \frac{{\delta_{X} }}{{\sigma_{X} }} $$
(4)

  • The mean of the absolute values of the second difference of the raw signal

$$ \gamma_{X} = \frac{1}{N - 2}\mathop \sum \limits_{n = 1}^{N - 2} |X\left( {n + 2} \right) - X\left( n \right)|\varvec{ } $$
(5)

  • The mean of the absolute values of the second difference of the standardized signal

$$ \overline{{\gamma_{X} }} = \frac{1}{N - 1}\mathop \sum \limits_{n = 2}^{N - 2} |\overline{X} \left( {n + 2} \right) - \overline{X} \left( n \right)| = \frac{{\gamma_{X} }}{{\sigma_{X} }}\varvec{ } $$
(6)

Wavelet-Based Features

Discrete Wavelet Transform is a powerful analytical tool for non-stationary signals and is widely used for time-frequency analysis of EEG signals due to its non-stationary nature. DWT decomposes EEG signal into different frequency bands with successive high pass and low pass filters. The high pass filter gives detail coefficients while low pass filter gives approximation coefficients. In this paper, we used Daubechies Wavelet Transform (db4) coefficients which is considered best for multiresolution analysis of EEG signals and EEG signal with 128 Hz sampling rate. Five levels Daubechies wavelet of order 4 is applied for decomposition of EEG signals into five frequency bands, delta, theta, alpha, beta and gamma as given in the Table 1.

Table 1. EEG signals decomposition into different frequency bands
Full size table

After Discrete Wavelet Transformation, we estimated wavelet energy and wavelet entropy according to Eqs. 1 and 2 respectively.

$$ E_{l} = \mathop \sum \limits_{n = 1}^{{2^{S - 1} - 1}} |C_{X} (l,n)|^{2} , N = S^{2} , 1 < l < S $$
(7)
$$ E_{l} = \mathop \sum \limits_{n = 1}^{{2^{S - 1} - 1}} |C_{X} (l,n)|^{2} \log (|C_{X} (l,n)|^{2} ) , N = S^{2} , 1 < l < S $$
(8)

Where C X (l, n) Wavelet coefficients associated with all five sub-bands are used to estimate wavelet energy and wavelet entropy. Another energy based feature set proposed by Murugappan et al. [11] is used in this paper. These features includes Recoursing Energy efficiency (REE), Logarithmic Recoursing Energy Efficiency (LREE) and Absolute Logarithmic Recoursing Energy Efficiency (ALREE). These features are estimated for gamma band as follows.

$$ REE_{gamma} = \frac{{E_{gamma} }}{{E_{total} }} $$
(9)
$$ LREE_{gamma} = \log_{10} \left[ {\frac{{E_{gamma} }}{{E_{total} }}} \right] $$
(10)
$$ ALREE_{gamma} = \left| {\log_{10} \left[ {\frac{{E_{gamma} }}{{E_{total} }}} \right]} \right| $$
(11)

3.3 Feature Selection and Classification

Feature selection is performed in order to mitigate the high dimensionality feature space problem. In this step, the most suitable subset of all derived features is selected which not only solve the problem of dimensionality but also increase the classification accuracy due to the reduction of noise caused by irrelevant features. In this paper, successfully applied maximum relevancy and minimum redundancy algorithm (mRMR) for feature selection. After selection of most relevant features, classification is performed using machine learning classifier. For this purpose a multi class Support Vector Machine is used with radial basis function.

4 Experimental Results

The EEG recordings of 32 subjects of DEAP database have been used to classify four main classes. The arousal and valence scores on the scale from 0 to 9 is mapped into two levels, high and low. The resulting four classes are, high arousal/high valence (HAHV), high arousal/low valence (HALV), low arousal/high valence (LAHV) and low arousal/low valence (LALV). For performance evaluation of the system, EEG data is divided into two portions, training data and test data. 70% of the total data is used for training purpose while the remaining 30% was used for testing. For classification, two machine learning algorithms, Support Vector Machine and Quadratic discriminant analysis are used to classify EEG data into four classes. Grid search approach was adopted for parameter optimization. In this experiment, best performance is given by SVM with overall accuracy of 49.7% using all channels data.

In order to implement a less complex and user friendly emotion recognition method, we reduced the numbers of EEG channels as much as possible. For this purpose, we selected a group of 15 EEG channels namely Fp1, Fp2, AF3, F3, F4, F7, F8, P7, O1, O2, P8, CP3, CP4, C4 and C3, that belongs to all four major lobs of the brain. Research shows that, left frontal lobe and right frontal lobe exhibit certain activity when a negative or positive emotion is experienced by a person [10]. Furthermore, the related research also reveals that different sub bands (Delta, theta, Alpha, Beta and Gamma) are activated by different emotional states in specific brain regions [20]. In this paper, we also investigated the activity of all five bands of EEG signals in selected channels for the aforementioned four classes of emotion. The classification of emotions is performed using each frequency band separately for combination of different sets of channels. EEG signal acquired from frontal lobe (FP1, FP2, F3 and F4) and temporal lobe (T7, T8) showed best performance with features extracted from Gamma band and achieved an overall classification accuracy of 48.8% for four classes of emotions which is close to the prior accuracy gained using all channels data of EGG signals (Table 2).

Table 2. Classification accuracy
Full size table

5 Conclusion

In this paper, we presented emotion recognition system using the most significant features set extracted from coefficients of Discrete Wavelet Transformation. A public available database called DEAP has been in this work. The EEG recordings of 32 participants have been utilized to extract statistical based feature and wavelet based feature. A feature selection algorithm was adopted to select the most significant and relevant features in order to mitigate the problem of dimensionality, irrelevancy and redundancy.

The proposed approach can significantly classify four classes of emotions using Support Vector Machine from which following can be concluded. First, feature extracted using Discrete Wavelet Transformation effectively represent emotional state of the users. Second, Gamma band holds rich information of all four classes of emotions. Third, we found that there is a strong correlation in frontal brain region related to Gamma band for all four classes of emotions which validates the role of frontal lobe in emotion recognition. In future, research will be conducted on fusion of EEG signals with others physiological signals for high performance.