Keywords

1 Introduction

Although expressing emotions is often considered a given ability, many people struggle with them on daily basis. For example, studies have shown that many individuals on autism spectrum suffer speech impairment [1,2,3]. They may also show atypical facial expressions [3, 4]. To make the matters worse, their expressions are more poorly recognized by others, whether autistic or neurotypical individuals [5]. Therefore, trying to understand their feelings using alternative methods -such as physiological signal analyses- can be significantly helpful.

One of the major problems with using bio-signals for such applications has been the complexity of measurement device setups and their cost, which can render them impractical outside laboratories [6, 7]. However, with the rise of simple wearable gadgets which are able to measure physiological signals, e.g. [8, 9], using this information in everyday applications is more feasible than ever. This motivates us to look into the relation of these signals and emotional experiences.

Although there are some works in the literature regarding the physiological signals and their correlation with emotional experiences, the accuracy of the analysis seems to be dependent on the data set [6, 10, 11]. Therefore, further body of work seems essential to expand existing knowledge. Moreover, the literature is more sparse when it comes to experiments using portable devices [12] and often the scope of these works are limited to one or two types of signals or emotions [8, 13]. Through this work, we hope to contribute to the literature by presenting our measurement results using a smart watch, namely Empatica E4, and regarding three different bio-signals; Electrodermal activity (EDA), Skin Temperature, and Heart rate. We also discuss the observed relations between these signals and four emotions: happiness, sadness, pain, and anger.

The rest of this paper is organized as the following: In the next section, we will cast a brief glance over the literature. In Sect. 3, we explain our data collection setup and present the measurement results. In Sect. 4, we discuss our findings regarding the collected data and put it into further light by comparing it to the existing knowledge in the literature. Finally, Sect. 5 concludes the paper.

2 Literature Overview

In this section, we will have a brief glance into the some of the most recent works in the literature. More specifically, those that are most similar or relevant, which inspired us in parts of our data collection and research procedures.

A common traditional approach for emotion recognition using physiological signals have been using Electrocardiography (ECG) signals. For example, in [7], the authors used an ECG to measure the changes in the heart rate variability. The participants had to sit still for 120 s after the ECG was attached to them, so their baseline could be measured. Afterwards, the participants watched a slide show of pictures and listened to music. Measuring ECG seems to be too complicated for everyday applications, however, their experimental setup was inspiring in designing our setup as well.

In [6, 12], the experimental setups were similar too. First, the participants had to sit still fore 60 s so their baseline could be measured. After that, they watched a short clip to experience some emotions. In [6], the participants had to fill out a Self Assessment Manikin (SAM) to show how strongly they felt each emotion. Both groups used an ECG, an EDA and a skin temperature sensor. In addition, the authors of [12] used an EEG sensor too.

Many groups that tried to identify emotions out of bio-signals used more than one device and often bulky devices to recognize emotions [6, 14]. Only a few groups tried to measure emotions using portable devices. In [8], the authors used wearable EDA sensor (wore on the wrist) for measurements. The EDA was measured at two positions on each hand, and they used two different devices to measure the EDA signals. The experimental setup was similar to the ones mentioned before. The participant had to sit still so that their baseline could be measured, and after that, they had to do some tasks such as pedaling for five minutes, or watching a short clip.

Last but not least, Ayzenberg and Picard [15] tried to combine a portable device and a long term measurement. The participant had to wear an EDA sensor on their wrist, with which, not only the EDA was measured, but also the skin temperature and the movement too. The participants wore the sensor for 10 days. Every time an event occurred (e.g. getting scared), the subject had do write it down. They analysed the signals for different peaks in a period of ten minutes, and thus characterized the signals. Then, they characterized the emotions using the information of peaks and the changes in the movement. The information about peaks, as it will be described in the next sections, proved to be an important distinguishing character in our collected data-set as well.

3 Data Collection

3.1 Subjects

For this work, ten subjects participated in the data collection. All of the participant were male university students, between 20 to 25 years old. Every subject was in perfect health without any medical condition. None of them reported consumption of any medication. Before the experiments, the procedure was explained to them and they were asked for their consent. They were not compensated for their participation.

3.2 Smart Watch

For the measurements, the E4 Smart Watch from Empatica was used [9]. The watch has four embedded sensors: EDA-sensor to measure the skin conductance, Photoplethysmogram (PPG) to measure heart rate, temperature sensor to measure skin temperature, and a 3-axis accelerometer to measure movements. The latter of which was not used in our experiments since the subjects were seated and had no physical activity or movement. In addition, the watch has a button to tag events, such as the beginning and the end of the application of stimuli.

3.3 Experiments Setup and Stimuli

At the beginning of the experiment, the procedure was explained to the participants. They were then seated in front of a computer and had to put on the Empatica E4 smart watch [9] on their left wrist and a headset. The procedure consisted of watching six stimulating videos between 1.5 to 3.5 min. The videos were chosen to solicit following emotions: happiness (2 videos), sadness (2 videos), anger, and fear. The videos were played in a random order. To solicit pain, the participants were pinched in the arm. Before and after each stimulus, the subjects had to sit still for one minute so that their baseline could be obtained. Beginning and end of each stimulation were tagged (shown by red starts in the figures) by pressing the button embedded in the watch for this purpose. After each stimulus, the participants had to fill a Self-Assessment Manikin (SAM) to state what emotion they experienced and how strongly they felt it.

3.4 Computations

After experiments the collected data were downloaded on a computer, then analysed and plotted in Matlab. In the Matlab code of this analysis, a “small” peak has a height between 0.05 to 0.4 \(\upmu {\text {S}}\), and a “big” peak has a height larger than 0.4 \(\upmu {\text {S}}\). Accordingly, in the rest of this paper, a “few” peaks means 2–4 peaks, whereas “some” implies 5–7 and “many” means more than 8 peaks. Moreover, for the heart rate, a change between 3 and 9 bpm was considered as a slight change. For skin temperature, on the other hand, the slight change implies a change between 0.06 and 0.2\(^\circ \)C. The signals were analysed by the aforementioned Matlab code in the window between the beginning and end of the application of stimuli (tagged during the experiments by the button on the watch and shown by red stars on the figures).

Table 1. Number of samples marked by the subjects as each of the solicited emotions.
Full size table
Fig. 1.
figure 1

Measurement results of different bio-signals as a result of various stimuli (a) SKT for Pain, (b) Skin Temperature (SKT) for happiness, (c) Heart Rate (HR) for Anger, and (d) EDA for Pain.

Full size image

3.5 Results

As found out in other works as well [11, 16], the stimuli were not unanimously successful in soliciting the intended emotions. Based on the filled SAM, from the 70 samples of emotional experiences (60 videos and 10 sample of pain), emotions were tagged as shown in Table 1. As we can see in this table, number of samples tagged with “fear” were very low and hence, this emotion was discarded from this study.

Figures 1 and 2 show a sample of the results. In Fig. 1, we see the various changes in the measured physiological signals corresponding to three different emotions. We can see that some signals show no particular change or any significant correlation to some emotion (e.g. the skin temperature during pain as shown in Fig. 1(a)). However, most other signals have some correlation; some demonstrate slow and very monotonic behavior (e.g. the skin temperature during happiness shown in Fig. 1(b)), some have somewhat larger or less monotonic changes (e.g. the heart rate during anger shown in Fig. 1(c)). On the other hand, some other signals show a sizable and faster changes (e.g. the EDA signal corresponding to the pain, Fig. 1(d)), which leads us to the next important feature we observed in some signals.

Fig. 2.
figure 2

EDA measurements; number and size of the peaks seems to be the main characterizing factor in EDA signals for different emotions.

Full size image

In Fig. 2, we see two more plots representing a different characteristics that we found distinguishing, namely peaks. This characteristic proved specifically significant in EDA signals, a few examples of which are shown in Fig. 2. The number and size of these peaks can point to different emotions, as seen in Figs. 1(d) and 2 Footnote 1. We often observe one big peak (Fig. 1(d)) during pain, whereas we see more big peaks during anger (Fig. 2(b)). We often notice many small peaks during sadness (Fig. 2(a)), although they occur sometimes during happiness too.

We have summarized our observations in Table 2, where the intensity of cell color shows the frequency of observing the respective trend in various samples. The darker the cell color, the bigger the number of samples showing the respective behavior.

Table 2. Summary of the observed trends in the collected data. The darker the cell color, the bigger the number of samples showing the respective trend.
Full size table

4 Discussion

To put our findings in perspective, compared to the existing knowledge in the literature, we summarized the characteristics we found in the literature and tabulated them in Table 3. The colored cells in this table, show where our observations differ from what we found in the literature. The lighter the cell color, the more in accordance are the findings; i.e. no cell color shows full accordance of our findings and the ones in the literature. We further remark that since the insets of Table 3 are taken from various works, they do not necessarily follow our definitions in Table 2, Sect. 3.5. As we see in Table 3, the available information is very sparse regarding the changes of the values after experiencing emotions.

Table 3. Summary of the characteristics of emotions based on our literature review. Colored cells show where our observation is different from the findings in the literature.
Full size table

We observe in Table 3, that all of our findings are in complete accordance with the ones in the literature, except for two cases. For the case of heart rate during happiness, only three out of sixteen samples had no change, whereas seven has a slight increase. Hence, the difference in the findings could be due to the dependency on the sample set or accuracy of measurement devices. In the case of the heart rate during pain, the observed ‘increase’ in [6] is significantly small which makes the discord rather negligible. On the other hand, our observation was not very confident (no cell color in Table 2). The reason being that four out of eight samples showed no change, three showed a slight decrease, and one showed a slight increase. All of which does not give a strong statistical indication as to its “normal” behaviour or tendency.

A similar observation was made for the heart rate during sadness. In this case, we had six slight increases, and one large increase (larger than 9 bpm), whereas four slight decreases and four large decreases (larger than 9 bpm). This and the aforementioned observations confirm the high variation of the results and their high dependency on instances as discussed in Sect. 1. Another example of which is that in [17], in contrast with our work and that of [15], no change was observed in the EDA value during happiness. Furthermore, it is worth mentioning, that our current observation table is different compared to our own previous observation table inserted in [18], which was based on primary measurements on only four subjects.

Last but not least, we notice that in [15], the authors do not associate the number of peaks with specific feelings and rather observe it as a potential sign for arousal, which includes various emotions such as happiness and anger. Moreover, they don’t specify what exactly many (or a “large number” as they call it) means, in contrast to what we did in Sect. 3.5. Nevertheless, since they measured the number of peaks during ten minutes, compared to our study in which we observed few peaks in approximately two minutes, observing many peaks in ten minutes is expected as a natural extrapolation.

5 Conclusion

Advancement of the technology has led to wearable devices which can measure physiological signals on-line and without interrupting daily life of the subject. This has opened up new opportunities to study the association of the bio-signals with emotional experiences and consequently to recognized them. Although there have been data collection campaigns in the literature, there is still a need for collecting further data. Especially, considering the high variation of the measured data and their strong dependency on the samples in experiments, as well as the considerable sparsity of data measured by wearable devices.

In this paper, we tried to contribute to the literature by reporting the results of our data collection campaign. In our experiments, we used Empatica E4 watch to measure Heart Rate, EDA, and Skin Temperature of the subjects while experiencing Happiness, Sadness, Anger, and Pain. Through this campaign, we could add new information to the literature, and verify the existing knowledge further. One of the key observations of our work was finding the association between EDA peaks and the four emotions. Number and size of the EDA peaks proved to be one of the most distinguishing characteristics of the four emotions subject to this study. Given the intrinsic high variation of the data, collecting more data in order to increase the confidence of the observed trends seems to be important for further enhancement of the existing knowledge in the literature and consequently applications which are developed based on this knowledge.