Keywords

13.1 Introduction

It is acknowledged that emotions are the manifestation of mental and psychophysiological states. Human-computer interaction (HCI) systems reveal hidden information from the brain and control peripheral devices [1]. It is difficult to create trustworthy emotion recognition algorithms that are accurate enough and flexible enough for use in practical applications [2].

Emotional state is believed to be a complex phenomenon comprised of biological, social, and cognitive components due to its modulating effects on physiological and behavioral processes. In brain-computer interface (BCI) applications and cognitive neuroscience research, two distinct models for characterizing the emotional state have been studied: In emotion recognition application mapping, the Circumplex model is a cognitive-emotional state model. The two measurements used to depict emotions on a two-dimensional coordinate plane are valance and arousal. Valence indicates the positive and negative intensity of an emotion, whilst arousal measures the intensity of an emotion, ranging from ecstatic to tranquil [3]. The Circumplex Model of emotion serves as an example of how this model graphs all emotions to a valance-arousal graph [4]. Researchers [5, 6] have added a third dimension that accounts the attention-rejection characteristic.

Visual and auditory inputs influence the amplitude of the sensorimotor rhythm [7]. It is believed that these inputs are the two most prevalent approaches for humans to trigger different emotional states [8] in order to reveal distinctive characteristics that would be useful for accurately determining an individual’s emotion in daily life. Recent research suggests that the optimal setting for automatic emotion recognition requires both stimulants of visual and acoustic inputs [9] to evoke a particular emotional condition. Typically, audible and visual elicitation employing short video clips is used to evoke different emotional states more effectively than other techniques [2, 7, 10]. Therefore, in this work, short audio-visual video segments were used to elicited emotion.

Numerous study investigations on the identification of human emotions have been carried out over the last few decades utilizing a range of approaches, including facial expressions [11]., peripheral physiological signs for identifying individuals based on emotional shifts are obtained using many biological measurements, such as Electrocardiogram (ECG) [12] and Electroencephalogram (EEG) [13,14,15,16,17].

Throughout this study, the EEG dataset was preprocessed with conventional filters and the hybrid empirical mode decomposition with wavelet transform (EMD − WT) technique. Linear features such as cross-correlation (xCorr), coherence (Coh), and phase lage index (PLI) as well as nonlinear features such as cross fuzzy entropy (CFuzzEn) and joint permutation entropy (JPE) were computed to capture various dynamical properties from emotion-based multi-channel EEG signals. A two independent variables analysis of variance (ANOVA) of these features is evaluated to determine the significant features that enhance the accuracy of classification of nine short emotional video clips (excitement, sadness, happiness, calmness, disgust, fear, anger, amusement and surprise). k-Nearest Neighbours (kNN) classifier was used for classification in its final step.

This work has focused on the role of emotion synchronization in the brain-emotion relationship. The paper novelty is therefore twofold. This is to be the first study to use the hybrid EMD − WT technique with the aforementioned features to evaluate emotions using EEG-based functional connectivity patterns to assess changes in brain network activity. Second, it is the first study to combine linear and nonlinear features to capture the changes in connectivity between brain lobes as measured by EEGs based on emotional states. Therefore, EEG indices allow for a more comprehensive understanding of the various effects of brain interventions on human behavior.

13.2 Related Works

In the past decade, EEG has been considered as a non-surgical clinical tool capable of monitoring neuronal activity in the brain with millisecond precision and a high level of temporal resolution [18]. Using EEG, numerous channels are recorded and analyzed for neurophysiology applications [18, 19] responsible for detecting and differentiating various brain diseases, such as epilepsy and sleep disorders. EEG signals have been used to classify mental tasks and sleep stages as well as to diagnose a number of brain disorders, such as epilepsy, ADHD, Alzheimer’s disease (AD), and vascular dementia (VaD) [20, 21]. A viable indication for assessing various affective reactions may be an EEG dataset with many channels spanning different brain areas [22]. Similarly, in [23] in order to identify current affective moods based on neural feedback and personalized modification of treatment, an integrated music therapy was developed using instantaneous techniques of affect recognition based on EEG.

The most challenging part of brain activity recognition is the low accuracy of recognition because of the significant noise compared to the EEG signal. EEG signals are subject to various types of artifacts, including physiological artifacts such as pulse, blinking eyes, and muscular activity [24,25,26,27] and sweat and other non-physiological phenomena like power line interference noise [27, 28]. Artifacts have a direct effect on EEG signals due to their frequency overlap with EEG signals. Therefore, a variety of denoising methods must be used to get over these problems and retrieve valuable data from the EEG dataset. Consequently, the most popular denoising techniques are wavelet transform (WT) [29, 30], independent component analysis (ICA) [31], empirical mode decomposition (EMD) [32], Savitzky – Golay (SG) [33] which can eliminate EEG artifacts. Numerous studies [34,35,36,37] have used hybrid techniques based on the combined use of ICA and WT to denoise EEG signals.

EMD and WT approaches were often used in ECG and EEG studies, either separately or in combination, depending on the data and the analysis’s objectives [38, 39].

EMD is frequently used with biological data that are nonlinear and non-stationarity sequence, like heart rate variability (HRV), electromyography (EMG), and EEG. This is because of its characteristics and qualities. In a similar way, wavelet transforms have found extensive usage in BCI systems because to their ability to retain details in the time and frequency domain with a broad range of scale and translation functions. Both approaches were successfully used as a foundation for filtering or further feature extraction to successfully obtain higher accuracy in classification.

To distinguish between focal and unfocal EEG data taken from epileptic patients, Das and Bhuiyan [40] employed EMD followed by wavelet. In the combination EMD − DWT, they used entropy-based characteristics to achieve excellent classification accuracy. For the elimination of EEG artifacts in [41], wavelet packet transform was used before wavelet packet transform after EMD. Generally speaking, EMD performed better than WT while denoising EEG data and getting ready for feature extraction [42].

Earlier emotional states of EEG-based studies used linear features such as correlation (Corr), coherence (Coh), and phase lag indices (PLI) between pairs of EEG sensors to estimate functional brain connectivity [43]. Coherence is a metric that has worked in a number of research fields, including physiology [44] and neurological disease [45]. Phase synchronization among the contributing neurological groups is one more technique for determining the functional connectivity of the EEG between different locations of the brain. This method is based on how two signals change in phase with one another. Neurological illnesses are studied using measures of EEG synchronization in phases [46].

Even so, using linear features, the complexity of the brain structure necessitates needs highly informative methods to inform about brain connectivity; thus, EEG signals are very effective bio-indicator of brain linear and nonlinear activity [29, 47,48,49,50].

The Hurst exponent (Hur) [51] and fractal dimension (FD) [52] are a well-known nonlinear algorithm used to examine the complex dynamic data produced by the cerebral cortex and reflect complex emotive tasks [53, 54].

The integration of entropy to EEG signals have improved the study of mental and sleep states, with techniques for identifying emotion levels. In [55], for instance, the emotions elicited by short clips were analyzed using sample entropy (SampEn), approximate entropy (ApEn), and permutation entropy (PerEn), as these techniques are unaffected by noise and capable of accurately measuring time sequence complexity. In a dissimilar study, clinical assessments of EEG signals based on symbolic transfer entropy and PerEn entropy were conducted, demonstrating that the EEG entropy investigation to relate to clinical cases of countless cognitive conditions [56]. Fuzzy entropy (FuzzEn), which substitutes fuzzy membership functions for Heaviside functions [57, 58], is also suggested for EEG examination. Existing research indicates that FuzzEn mitigates the issue of entropy mutation; however, when using such entropy approaches, pertinent information is lost since they need single-scale analysis. Compared to SampEn, FuzzEn exhibits more consistency and less dependency on data length [59]., despite the fact that SampEn is somewhat faster. The present research focuses on EEG-synchronization derived markers to categorize the gender variances in EEG signals based on emotion across the brain.

Artificial neural networks (ANNs), support-vector networks (SVNs), the k − nearest neighbours (kNN) classifier, and hidden Markov models (HMM) have all been closely examined as machine learning algorithms for a gender classification system [60, 61]. In [62], a gender and age classification system based on EEG signals was created using the SVM classifier, whereas in [55], resting-state EEG data served as the foundation for a model for automatic gender detection. An automated system for estimating age and gender was developed by combining EEG sensors with wavelet transform frequency break-down as feature extraction and random forest classifier [63, 64].

Emotional reactivity is the main focus of investigations on EEG-based emotion identification [65,66,67]. emphasized the use of spectral relative powers in linear analysis. Nevertheless, many researchers have exploited nonlinear properties to study the complexity of the brain [17, 68, 69]. In the present research, we intend to estimate the synchronization based on the EEG role for recognition using both the linear and nonlinear features to classify the emotional-based signals by analyzing the behavior of coactivated and communicating brain regions during audiovisual video clips. Additionally, the particularity of emotion Functional connectivity-based EEG research have demonstrated the ability to distinguish between diverse emotional states; however, none of these investigations have directly evaluated these multiple synchronization indices within a single study. This study subject is important because various connection measures are sensitive to various EEG signal properties.

Therefore, the goal of the current work was to revisit the problem of EEG-based emotional specificity using the linear and nonlinear synchronization indices and discover which of the indices is more effective at identifying various affective states using pattern classification analyses. Correlation, cross coherence, and synchronization indices are examples of connectivity indicators that might potentially reveal important information, given that emotion processing is a complicated activity that is not limited to a small number of brain areas. Nine distinct emotional states were elicited through the use of emotive video clips.

13.3 Materials and Methods

The proposed study is depicted in Fig. 13.1 as a block diagram.

Fig. 13.1
A schematic diagram consists of E E G dataset, Conventional filters, Denoising technique, feature extraction, channel selection, and classification. It begins with the preprocessing stage, then moves on to the linear and non-linear classification stages.

Schematic representation of the proposed system

Full size image

13.3.1 Subjects and Experimental Procedure

Electroencephalogram (EEG) and electrocardiogram (ECG) signals recorded in response to audio-visual stimuli comprise the multi-modal ‘DREAMER’ dataset [70]. The emotions of 23 participants were elicited and recorded alongside the participants’ self-assessments of their affective state in terms of valence, arousal, and dominance following each stimulus. There were 11 females and 14 males, with a mean age of 26.6 ± 2.7 years and a standard deviation of 2.7 years. 16 contact-sensors with plated gold connected to the flexible arms of a wireless headset and positioned in the Universal 10 − 20 system’s locations: AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, AF4, M1 and M2. Mastoid sensor M1 served as a ground reference point for comparing the voltage of all other sensors.

The EEG dataset consisted of 18 movie scenes chosen and analyzed by Gabert-Quillen et al. [71]. These movie clips include scenes from several movies that have been demonstrated to induce a variety of emotions. The duration of the movie clips ranged from 65 to 393 seconds (M = 199 seconds), which is deemed adequate because, according to psychologists, video stimuli between 1 − 10 minutes can trigger single emotions. However, an individual’s emotional state may change over time, especially when longer video stimuli are employed. To prevent multiple emotions from contaminating data recordings, only the final thirty seconds of each movie clip were further analyzed. To prevent external influences, the experiments were conducted in an isolated setting with regulated lighting. The room was completely darkened by an electric curtain, and the video clips played back on a 45 − inch TV panel.

13.3.2 Preprocessing Stage

The majority of artifacts that emerge in EEG wave frequency bands, where there is a potential for overlap with brain processes, necessitate the removal of noise while processing EEG signals.

13.3.2.1 Conventional Filtering

In the cerebral cortex’s scalp region, 14 channels of EEG data were recorded. Every channel of the recorded EEG dataset was initially treated using standard filtering methods. A band pass filter with a frequency range of (0.5 − 64) Hz was used to isolate the band of the recorded EEG signals, and a notch filter at 50 Hz was suggested to remove power line interference noise [72].

13.3.2.2 Empirical Mode Decomposition with Wavelet (EMD − WT) Hybrid Denoising Technique

In this study, EMD and DWT were used in a two-stage filtering process. It is proposed and discussed that EMD − WT is an automatic hybrid method for combining the benefits of EMD with WT and to mitigate their weaknesses. Consequently, the EMD − WT technique has been implemented as in the following sessions. In the initial stage, EMD decomposed the EEG dataset into Intrinsic Modal Functions (IMF) levels with a residue. Each IMF consists of limited-frequency bands that permit a more accurate representation of the EEG dataset and have the following properties [73]:

  1. (i)

    Baseline crossings and Extermas are identical or differ by no more than one.

  2. (ii)

    The envelopes average formed by extremes is equal zero. EMD can represent a raw EEG signal y[n] in terms of IMFs as [73]

$$ y\left[n\right]=\sum \limits_{m=1}^M{I}_m\left[n\right]+{r}_M\left[n\right] $$
(13.1)

where M, Im[n], and rM[n] indicate number of IMFs, mth IMF, and residue, respectively. IMFs are parts of the original EEG signal, which may be either contain noise or information. Compared to informative components, it has been observed in [74] that noisy IMFs are poorly correlated with the original signal. The optimal threshold point of correlation Cth for removal of noisy components can be calculated as follows: [74],

$$ {C}_{Th}=\frac{\max \left({C}_i\right)}{10\times \max \left({C}_i\right)-3},i=1,2,\dots, M $$
(13.2)

Ci: the IMF correlation coefficient (CC) of the ith IMF with the original signal.

Max(Ci): maximum CC among M IMFs

The removal of noisy components algorithm is given as [73]:

  1. 1.

    Initialize denoised EEG signal, S[n]D = 0

  2. 2.

    for i = 1 : M do

  3. 3.

    if Ci < CTh then

  4. 4.

    IMFi is a Noisy- IMF

  5. 5.

    else

  6. 6.

    S[n]D = S[n]D + IMFi

  7. 7.

    end if

  8. 8.

    end for

  9. 9.

    Get → S[n]D

Due to the WT’s ability to resolve EEG into distinct time and frequency components, discrete (DWT) was applied to each IMF in the second stage to gain a better understanding of the WT’s advantages for high frequencies, there is an excellent time resolution but a low frequency resolution and at low frequencies there’s a good frequency resolution and a poor time resolution. DWT transform can be obtained as a set of decomposition functions of the correlation between the signal f(t) and the shifting and dilation of a particular function known as the mother wavelet function ψ(t). According to Eq. 13.1, the mother wavelet (ψ _ (a, b) (t)) is shifted along the original signal by the position parameter (b) and stretched or compressed by the frequency scaling parameter (a) [29, 75,76,77,78,79,80]:

$$ DW{T}_{m,n}(f)={a_0}^{\raisebox{1ex}{$-m$}\!\left/ \!\raisebox{-1ex}{$2$}\right.}\int f(t)\psi \left({a}_0^{-m}t-n{b}_0\right) dt $$
(13.3)

a0 and b0 values are set to 2 and 1, respectively.

$$ {\psi}_{a,b}(t)=\frac{1}{\sqrt{a}}\psi \left(\frac{t-b}{a}\right), a\epsilon {\mathbb{R}}^{+}, b\epsilon \mathit{\mathbb{R}} $$
(13.4)

This study indicated that ‘sym9’ from the Symlets family of order 9 with four levels of decomposition is suitable for automatically removing artifacts and reducing computational time and complexity. SURE threshold is an adaptive soft thresholding technique that seeks to identify the threshold limit for every level based on Stein′s unbiased risk estimation [81] and commonly employed values in [82,83,84,85]. To obtain EEG signals devoid of artifacts, the inverse DWT (IDWT) is applied at the conclusion.

13.3.3 Features Extraction Stage

In order to determine which approach is best for predicting emotions from multi-channel EEG signals evaluation, features from the temporal domain have been used with linear and nonlinear interdependence analysis to examine functional connectivity in various brain areas.

13.3.3.1 Linear Features

  1. A.

    Cross-correlation (xCorr)

    Taking into account two separate time series xn and yn, n = 1, …, N. The most often used linear synchronization method is the cross-correlation function cxy, which is defined as [86]:

$$ {C}_{xy}\left(\tau \right)=\frac{1}{N-\tau}\sum \limits_{i=1}^{N-\tau}\left(\frac{x_i-\overline{x}}{\sigma_x}\right)\left(\frac{y_{i+\tau }-\overline{y}}{\sigma_y}\right) $$
(13.5)

where \( \overline{x} \) and σx denote mean and variance, and τ is the time lag.

  1. B.

    Coherence (Coh)

    The cross − spectral density function Cxy(ω), which is derived as the Fourier transform of the cross correlation, is called magnitude squared coherence or simply coherence. The normalization of the cross spectrum is therefore defined as the coherence function (γxy) [86]:

$$ {\gamma}_{xy}\left(\omega \right)=\frac{\left|{C}_{xy}\left(\omega \right)\right|}{\sqrt{{\textrm{C}}_{xx}\left(\omega \right){C}_{yy}\left(\omega \right)}} $$
(13.6)

Cross − correlation is a time − lag − dependent measure, whereas coherence is a frequency − dependent linear measure. For each pair of electrodes, we found peaks in coherence and cross − correlation and used these peak values to calculate the two-connectivity metrics.

  1. C.

    Phase lag index (PLI)

    Even though the amplitudes of two linked nonlinear systems are not correlated, empirical tests have demonstrated that their phase can synchronize. The analytical signal for a real-valued signal x(t) is defined as [86]:

$$ {Z}_x(t)=x(t)+i{x}_H(t)={A}_x^H(t){e}^{i{\varnothing}_x^H(t)} $$
(13.7)

Where xH(t) is the Hilbert transform of x(t). Then similarly for another signal y(t), we define \( {A}_y^H \)and \( {\varnothing}_y^H \). If we let the synchronization between x(t) and y(t) is n : m, we define the (n, m) phase difference of their analytic signal as \( {\varnothing}_{xy}^H(t)=n{\varnothing}_x^H(t)-m{\varnothing}_y^H(t) \), where n and m are integers. Then the PLI is defined as,

$$ PLI=\mid {e^{i{\varnothing}_x^H(t)}}_t\mid =\sqrt{\cos {\varnothing}_x^H{(t)}_t^2+\sin {\varnothing}_x^H{(t)}_t^2} $$
(13.8)

When the phases are not synchronized, the PLI is zero, and when they are perfectly synchronized, it is one.

13.3.3.2 Nonlinear Features

  1. A.

    Cross fuzzy entropy (CFuzzyEn)

    The synchronization or similarity of patterns between two signals is measured using cross − fuzzy entropy (CFuzzyEn). CFuzzyEn is a cross- SampEn enhancement that is better suited to short time series and more noise resistant. CFuzzyEn is computed as for two times series of length N [87]:

$$ CFuzzyEn\left(m,n,r,N\right)=\ln {\varnothing}^m\left(n,r\right)-\ln {\varnothing}^{m+1}\left(n,r\right) $$
(13.9)

where \( {\varnothing}^m\left(n,r\right)=\frac{1}{N-m}\sum \limits_{i=1}^{N-m}\left(\frac{1}{N-m}\sum \limits_{j=1}^{N-m}{D}_{ij}^m\right) \) for vector \( {X}_i^m \), and \( {\varnothing}^{m+1}\left(n,r\right)=\frac{1}{N-m}\sum \limits_{i=1}^{N-m}\left(\frac{1}{N-m}\sum \limits_{j=1}^{N-m}{D}_{ij}^{m+1}\right) \) for vector \( {X}_i^{m+1} \), \( {D}_{ij}^m \) measures the synchrony or similarity degree. Each CFuzzyEn computation requires the determination of three parameters. The first parameter, m, is the length of the sequences to be compared, or the dimension of the vector to be compared. The remaining two parameters, r and n, control the width and gradient of the exponential function’s boundary, respectively. Setting the tolerance r between 0.1 and 0.3 and choosing small integers for the selection of n works well in practice.

  1. B.

    Joint permutation entropy (JPE)

    Yin et al. introduced the joint permutation entropy (JPE), which is used to measure the synchronism between two time series [88]. It is based on permutation entropy, which compares neighboring values of each point and converts them into ordinal patterns to quantify the complexity of a signal. JPE is calculated as the Shannon entropy of the d !  × d! unique motif combinations (\( {\pi}_i^{d,t} \),\( {\pi}_j^{d,t} \)) for two signals u and v [88]:

$$ JPE\left(d,t\right)=-\sum \limits_{i,j:\left({\pi}_i^{d,t},{\pi}_j^{d,t}\right)}p\left({\pi}_i^{d,t},{\pi}_j^{d,t}\right).\ln \left({\pi}_i^{d,t},{\pi}_j^{d,t}\right) $$
(13.10)

where \( p\left({\pi}_i^{d,t},.\right) \) means the probability that the first time series has the ith pattern regardless of the second, and \( p\left(.,{\pi}_j^{d,t}\right) \) denotes the chance that the second time series has the ith motif regardless of the first.

13.3.4 Features Selection Using Statistical Analysis

The emotional task channels were chosen using the ANOVA test as inputs for the classifiers. The significant p value of the EEG feature was a key factor in the selection of features for events. p values of less than 0.05 show a 95% significant difference between classes. Selected channels were determined by the Statistical Package for Social Sciences (SPSS) version 22’s ANOVA test with a significance level of (p < 0.05).

As a result, the goal of this study is to examine the linear and nonlinear characteristics’ most efficient channels. The Kolmogorov − Smirnov test was used to judge the normality of the tests. The relevant channels among the emotions (i.e. amusement, excitement, happiness, calmness, anger, disgust, fear, sadness and surprise) were chosen using two sessions of two − way ANOVA.

In the first session of ANOVA, linear features including xCorr, Coh and PLI were applied as dependent variables and the group factor (emotions and brain regions) were the independent variable. Moreover, in the second session of ANOVA, nonlinear features like CFuzzEn and JPE features and the group factor (emotions and brain regions) were the independent variable. The significance was set at p ˂ 0.05.

Then, the feature reduction method was implemented to reduce the irrelevant daughter features (pair of channels) the method is as follows:

For each symmetric functional connectivity matrix, 14 diagonal elements were removed, and the upper triangle elements of the connectivity matrix were extracted as classification features, i.e., the feature space for classification was constituted by the 14 × (14 − 1)/2 = 91 dimensional feature vectors (pair of channels or connections).

The abnormal functional connectivity patterns associated with emotional states are mainly represented by the highly discriminating functional connections, and 91 dimensional feature vectors including all the differences caused by noise. Highly discriminatory features were selected from the original 128 features space, further reducing the number of features, accelerating computation and diminishing noise.

Therefore, the feature selection method was used to reduce the dimension for classification through retaining the most discriminative functional connections and eliminating the remaining indistinctive features. The discriminative power of a feature can be quantitatively measured by the importance of its degree of relevance to classification.

Based on how dependent each feature was on its corresponding class, features were rated. The features that are most reliant on the class were chosen, and this procedure provides those features a higher rank than other features. Based on each sample, these feature frequencies are calculated.

13.3.5 Emotion Classification Stage

Linear and nonlinear features had (91 connections without reduction) which were calculated for the last 30 second of each video clip recording with an epoch length of Fs×5 = 128×5 = 640 samples. Meaning 6 epochs for each video clip (emotion) and since we have 9 emotions and 23 subject the number of instances is 23×9×6 = 1242 for each pair connection.

In this section, k nearest neighbors (kNN) was used. One of the most often used non-parametric classification methods is kNN; when k > 1, in particular to lessen the impacted noisy points in the training set, it is more resilient. The Euclidean distance was used in this work as a similarity metric to categorize each trial using kNN.

13.4 Results and Discussions

13.4.1 Results of Preprocessing Stage

Datasets of EEG signals were filtered using traditional filters and then denoised using EMD − WT hybrid technique. The frontal brain region when anger is produced is shown by the data from Channel 7 in Fig. 13.2. Artifactual signal elements (red line) in the raw EEG data were successfully eliminated during signal denoising, leaving a clear EEG signal (blue line).

Fig. 13.2
Two graphs present the outcomes of the denoising process following the preprocessing stage for subject 9's channel 7. The plotted raw and filtered signals have a downward fluctuating trend.

The denoising results after preprocessing stage for channel 7 from subject 9

Full size image

13.4.2 Results of Features Extraction Stage

To characterize the emotion synchronization of EEGs, linear features including xCorr, Coh and PLI and nonlinear features like CFuzzEn and JPE were computed for the EEGs. Features were estimated for each channel with a total length of 3,840 samples divided into 6 epochs; the length of each epoch was 640 data points (1 segment represents 5 s) of EEGs.

13.4.2.1 Results of Linear Features

The outcome demonstrates that particular brain areas have higher levels of connection during emotional reactions. These findings were revealed from linear features, which could be used to identify a synchronization diagnostic index that would be able to distinguish emotions from EEG-based signals. Statistical analysis of network parameters revealed these features from the two groups were significantly different.

  1. A.

    Results of Cross-correlation (xCorr)

The calculated comparative plot of xCorr is shown in Fig. 13.3 to discriminate among nine different emotional states based on EEG signals. The matrix presents the topographical map of the nine emotions using linear bivariate features in all channel.

Fig. 13.3.
Nine comparative plots present emotions like calmness, surprise, amusement, fear, excitement, disgust, happiness, anger, and sadness using x C o r r. There are color gradient scales, ranging from 0.2 to 1, on the right side of the plots.

Comparative plot of the functional connectivity matrices for the nine tested emotional states using linear xCorr

Full size image

Figure 13.4 shows the xCorr ranking and the connection importance score of the feature of all channels in the EEG signals. Features that are strongly influenced by the emotional category and are ranked higher than other features.

Fig. 13.4
A bar graph represents the connection importance score versus connection pair rank. The data plotted are in a downward trend, and the connection pair rank of 10 to 13 is the highest at 40. Values are approximate.

Comparative plot of the xCorr rank of channels

Full size image

Plotting the first 15 ranked pairs of xCorr electrodes in Fig. 13.5 reveals that the most efficient pairs are conjugated with the F7 and F8 electrodes in the frontal lobe.

Fig. 13.5
A connectivity plot of the electrodes organized in the paired groups for x C o r r. The labeled parts are left hemisphere electrodes, right hemisphere electrodes, and reference electrodes.

Electrodes Connectivity plot of the xCorr first 15 ranked pairs

Full size image
  1. B.

    Results of Coherence (Coh)

Figure 13.6 illustrates the comparative plot of Coh which was shown to discriminate among nine different emotional states based on EEG signals. The matrix presents the topographical map of the nine emotions using linear bivariate features in all channel.

Fig. 13.6
Nine comparative plots represent emotions like calmness, surprise, amusement, fear, excitement, disgust, happiness, anger, and sadness using C o h. There are color gradient scales on the right side of the plots.

Comparative plot of the functional connectivity matrices for the nine tested emotional states using linear Coh.

Full size image

Figure 13.7 shows the Coh ranking and the connection importance score of the feature of all channels in the EEG signals. Features that are strongly influenced by the emotional category and are ranked higher than other features.

Fig. 13.7
A bar graph presents the connection importance score versus connection pair rank. The data plotted are in a downward trend, and the connection pair rank of 1 to 12 is roughly the highest at 10.2. Value is approximate.

Comparative plot of the Coh of channels

Full size image

The most efficient pairs are conjugated with the O1 and O2 electrodes in the occipital lobe, as seen in Fig. 13.8, which plots the first 15 ranked pair of electrodes of the Coh.

Fig. 13.8
A connectivity plot of the electrodes organized in the paired groups for C o h. The labeled parts are left hemisphere electrodes, right hemisphere electrodes, and reference electrodes.

Electrodes Connectivity plot of the Coh. first 15 ranked pairs

Full size image
  1. C.

    Results of phase lag index (PLI)

Figure 13.9 illustrates the comparative plot of PLI which was illustrated to discriminate among nine different emotional states based on EEG signals. The matrix presents the topographical map of the nine emotions using linear bivariate features in all channel.

Fig. 13.9
Nine comparative plots represent emotions like calmness, surprise, amusement, fear, excitement, disgust, happiness, anger, and sadness using P L I. There are color gradient scales, ranging from 0.2 to 1, on the right side of the plots.

Comparative plot of the functional connectivity matrices for the nine tested emotional states using linear PLI.

Full size image

Figure 13.10 shows the PLI ranking and the connection importance score of the feature of all channels in the EEG signals. Features that are strongly influenced by the emotional category and are ranked higher than other features.

Fig. 13.10
A bar graph presents the connection importance score versus connection pair rank. The data plotted are in a downward trend, and the connection pair rank of 11 to 13 is the highest at 8.1. Values are approximate.

Comparative plot of the PLI of channels

Full size image

Plotting the first 15 ranked pairs of electrodes from the PLI in Fig. 13.11 reveals that the most efficient pairs are centered in the frontal right lobe.

Fig. 13.11
A connectivity plot of the electrodes organized in the paired groups for P L I. The labeled parts are left hemisphere electrodes, right hemisphere electrodes, and reference electrodes.

Electrodes Connectivity plot of the PLI first 15 ranked pairs

Full size image

13.4.2.2 Results of Nonlinear Features

The results demonstrate that emotional responses are associated with higher levels of connection in specific brain locations. These findings were disclosed using nonlinear features that could be utilized to develop a synchronization diagnostic index that could differentiate emotions from EEG-based signals, according to statistical analysis of network parameters.

  1. A.

    Results of cross-fuzzy entropy (CFuzzEn)

Figure 13.12 illustrates the comparative plot of CFuzzEn which was computed to discriminate among nine different emotional states based on EEG signals. The matrix presents the topographical map of the nine emotions using nonlinear bivariate features in all channel.

Fig. 13.12
Nine comparative plots represent emotions like calmness, surprise, amusement, fear, excitement, disgust, happiness, anger, and sadness using C F u z z E n. There are color gradient scales, ranging from 0.2 to 1, on the right side of the plots.

Comparative plot of the functional connectivity matrices for the nine tested emotional states using nonlinear CFuzzEn

Full size image

Figure 13.13 shows the CFuzzEn ranking and the connection importance score of the feature of all channels in the EEG signals. Features that are strongly influenced by the emotional category and are ranked higher than other features.

Fig. 13.13
A bar graph represents the connection importance score versus connection pair rank. The data plotted are in a downward trend, and the connection pair rank of 5 to 10 is the highest at 89. Values are approximate.

Comparative plot of the CFuzzEn of channels

Full size image

Plotting the first 15 ranked pair of electrodes of the CFuzzEn in Fig. 13.14 reveals that the Temporal lobe electrodes T7 and T8 are related to the majority of the effective pairings (features).

Fig. 13.14
A connectivity plot of the electrodes organized in the paired groups for C F u z z E n. The labeled parts are left hemisphere electrodes, right hemisphere electrodes, and reference electrodes.

Electrodes Connectivity plot of the CFuzzEn first 15 ranked pairs

Full size image
  1. B.

    Results of Joint Permutation Entropy (JPE)

Figure 13.15 illustrates the comparative plot of JPE which was considered to discriminate among nine different emotional states based on EEG signals. The matrix presents the topographical map of the nine emotions using nonlinear bivariate features in all channel.

Fig. 13.15
Nine comparative plots represent emotions like calmness, surprise, amusement, fear, excitement, disgust, happiness, anger, and sadness using J P E. There are color gradient scales on the right side of the plots.

Comparative plot of the functional connectivity matrices for the nine tested emotional states using nonlinear JPE.

Full size image

Figure 13.16 shows the JPE ranking and the connection importance score of the feature of all channels in the EEG signals. Features that are strongly influenced by the emotional category and are ranked higher than other features.

Fig. 13.16
A bar graph presents the connection importance score versus connection pair rank. The data plotted are in a downward trend, and the connection pair rank of 9 to 10 is the highest at 77. Values are approximate.

Comparative plot of the JPE of channels

Full size image

The top 15 ranked pairings of the JPE, which are conjugated with the T7 and T8 in the temporal lobe and P8 and P7 in the parietal lobe, are the most successful in the categorization, as shown in Fig. 13.17.

Fig. 13.17
A connectivity plot of the electrodes organized in the paired groups for J P E. The labeled parts are left hemisphere electrodes, right hemisphere electrodes, and reference electrodes.

Electrodes Connectivity plot of the JPE first 15 ranked pairs

Full size image

13.4.3 Results of Features Selection and Emotion Classification Stages

13.4.3.1 Classification Results of Linear Features

Table 13.1 shows the kNN classification accuracies for full feature set of 91 attributes and the accuracies after applying the feature selection method. It can be observed that the linear features were fells to classify the emotions from EEG-based signals and that’s may related to the brain complexity structure. Therefore, in the next stage this study was used nonlinear dynamical features to be compatible with the brain complexity.

Table 13.1 kNN classification accuracies of full features set and after selection of linear features
Full size table

13.4.3.2 Classification Results of Nonlinear Features

Table 13.2 shows the kNN classification accuracies for full feature set of 91 attributes and the accuracies after applying the feature selection method. It can be observed that the nonlinear have better scores particularly for the JPE compared to the linear features.

Table 13.2 Classification accuracies of full features set and after selection of nonlinear features
Full size table

Moreover, the combination of CFuzzEn and JPE nonlinear entropy features increase the classification accuracy to 84.9%. The classification results illustrated the effect of CFuzzEn and JPE features as remarkable synchronization index for investigating emotions from EEG-based data set.

Figures 13.18 and 13.19 display the confusion matrices for linear and nonlinear features computed from emotional-based EEG signals in which correct recognition is shown on the diagonal; substitution errors are off-diagonal. According to the last confusion matrix of both data sets, sadness (Class 9) had the least misclassification due to its very low valance score.

Fig. 13.18
Three confusion matrices represent a true class versus predicted class for x C o r r, C o h, and P L I. The data are plotted using linear features.

The confusion matrix calculations for emotional-based EEGs classification using linear features

Full size image
Fig. 13.19
Two confusion matrices represent a true class versus predicted class for C F u z z E N and J P E. The data are plotted using nonlinear features.

The confusion matrix configuration for emotional-based EEGs classification using nonlinear features

Full size image

Based on the studies that attempted to estimate the best features, Table 13.3 compares the proposed method with existing methodologies. However, these methods were obtained with reduction in detection accuracy due to complicated computational calculations due to the redundant features. The experimental results show that our approach achieves a high accuracy comparing to competitive state-of-the-art methods, indicating its potential in promoting future research on multi-person EEG recognition. Thus, this study presents an automatic emotion recognition model employing synchronization indices to extract features with the highest quality to enhance the ability of emotional-based EEGs to identify different emotions. The suggested technique improves the kNN classification accuracy by a fair amount. Moreover, these methods have been used to study emotional-based EEGs, but in this work, automatic emotion detection from synchronization indices utilizing emotional-based EEG is the first to be observed in order to choose the optimum quality of features that improved the accuracy of classification of emotions from amusement, excitement, happiness, calmness, anger, disgust, fear, sadness and surprise emotional-based EEGs. Additionally, the previous studies have used already existing public dataset datasets (MAHNOB, DEAP) with mainly linear features while in this investigation the EEG dataset was investigated and examined using linear and nonlinear features particularly bivariate CFuzzEn and JPE entropy features therefore, Since the EEG estimate technique has never been used to secure emotional information, emotion contrasts may be better expressed.

Table 13.3 Qualitative comparison of the proposed method to the state-of-the-art
Full size table

There are some disadvantages, nevertheless, that should be taken into account. For instance, the suggested method made use of a regularized dimensionality reduction stage rather than including small sample size, and the future work will need to use a larger data source. This work compared only 3 linear and 2 nonlinear entropy feature sets, however, wavelet entropy, dispersion entropy, and multi-scale entropy are also worth looking into to see if they have the ability to detect emotional synchronization using EEG data. Nevertheless, the suggested EEG-based method does have a limitation, in this study. The EEG datasets were examined offline and data from examinations was gathered. However, in order to corroborate the findings, further research utilizing real-time online experiments is necessary due to the differences between offline and online categorizations.

Despite these limitations, the results from this study and those of past studies, which supported the potential of EEG signals to identify the majority of emotional disparities, are in accord and those discrepancies were reflected in the EEG bands as well [94]. Hence, when coupled with nonlinear feature sets and kNN classifier, the proposed method can provide valuable insight for automatic emotion estimation using synchronization indices on the basis of key biomarkers and dependable indicators for affective-based EEG.

This study adds to the body of knowledge about the detection of affective mood by its findings. However, additional research is needed, and combining other techniques, such as EEG and MEG or EEG and fMRI, may improve the results. This method is justified by the fact that various emotions do not affect the metabolic behavior of blood in capillaries and neuron electrical activity in the same manner.

To sum up, EMD − WT hybrid denoising technique was used, xCorr, Coh and PLI linear features and CFuzzEn and JPE nonlinear features can provide information to improve the synchronization of emotions. These features can yield useful information to characterize and identify the relation between brain lobes when the emotion changes. The former features have been used for further analysis including channels selection using ANOVA test. The results have been reported to extract the valuable EEG synchronization markers associated with emotions. This result implies that the denoising technique combined mainly with nonlinear entropy features may automatically detect and provide reliable synchronization markers to identify state of emotion of individuals. Indeed, the results emphasize the crucial role played by the novel proposed framework in the EEG signal processing chain particularly in the classification results.

13.5 Conclusion

Conventional filters and WT technique were used in the preprocessing stage to denoise the Dreamer EEG datasets of 23 subjects while watching 18 short emotional video clips with (amusement, excitement, happiness, calmness, anger, disgust, fear, sadness and surprise) audio-visual stimuli. In the second stage, linear features including xCorr, Coh, PLI and nonlinear features like CFuzzEn and JPE features were computed to capture different dynamical properties from emotional-based multi-channel emotional-based EEGs. Moreover, ANOVA has been used to statistically examined the individual performance of the used features. Finally, kNN classification technique has been used for automatic emotion recognition from EEG-based dataset. These Results shows increased emotion classification accuracy for the features that included high synchronization of certain lobe with the rest of the brain regions. The nonlinear features such as JPE and CFuzzEn had one lobe (Temporal lobe) synchronized with the rest of brain lobes (electrodes) thus providing the highest classification accuracy among the other features that are in the contrarily didn’t have any specific lobe connected to the others.