Keywords

1 Introduction

The biggest challenge in the study of emotions is to measure them because they are related to internal states of the individual. First, it is noted that the emotional response may be of several types, for instance, verbally, through facial expressions, body movements and/or physiological responses. Although there are several theories about emotions, there is agreement on the two dimensions for measuring emotions: Arousal and Valence (e.g., Lang et al. 1997; Russell and Barrett 1999). Arousal refers to the psychophysiological condition caused in the individual by the presence of a stimulus, product or object (Lang 1995). It may be high when the stimulus produces a high activation in the subject (e.g., see a snake) or low when the stimulus produces low activation (e.g., listening to music for relaxing). Arousal is characterized by an activation of the autonomic nervous system (e.g., fight or flight reaction), and the activation of the endocrine system (e.g., increased blood cortisol levels) which enable the individual to respond appropriately to the stimuli. Valence (Lewin 1935) concerns to the positivity (e.g., Happiness) or negativity (e.g., Sadness) that a stimulus or situation elicits in the individual.

Existing instruments for measuring emotions can be divided into two major groups: subjective measures (e.g., questionnaires) and objective measures (i.e., psychophysiological).

2 Subjective Measures

One way to measure and evaluate emotions and emotional states is using subjective measures. This type of measures is essentially based on self-reports or questionnaires. A range of different subjective measures is available which reflects the variety of ways in which emotion has been conceptualized. Next, some of the most commonly used subjective measures are described.

The subjective measure most used to measure emotions against a given product is PrEmo. This instrument was created by Desme et al. (2000) and aims to measure the pleasantness or unpleasantness that one product elicits in the user. It consists of eighteen cartoon-like images, each image representing an emotion: 9 positive emotions (inspired, appreciative, pleasantly surprised, enthusiastic, attracted, desiring, fascinated, content, and softened) and 9 negative emotions (bored, contempt, aversive, disgusted, disappointed, dissatisfied, indignant, vulnerable, and disillusioned). The cartoons represent the facial expressions and body postures of each emotion.

The application of this instrument consists of showing the products to the participants. After the interaction with the product, the participant must choose which cartoon(s) best characterize the emotion felt in the presence of the product. This instrument has a great advantage for the evaluation of emotions elicited by products. Participants do not need to verbalize the emotion felt, since they only have to identify it visually through the cartoons represented in PrEmo. This feature is very important for this type of measures since verbally expressing emotions can prove to be a very difficult task since not all individuals have the necessary vocabulary to do it (e.g., Desmet et al. 2000).

Another widely used instrument for emotion self-reporting is the SAM (Self-Assessment Manikin) proposed by Bradley and Lang (1994). SAM evaluates the valence, arousal and also the emotional dominance (subjective feeling of control regarding the emotional event). The three dimensions are represented by a pictogram rating system: the valence is evaluated on a scale ranging between negative and positive dimensions (sad, neutral and happy faces); the arousal varies between calm and active (e.g. from a small beating heart to a heart beating with an explosive/flash appearance); and the dominance varies from submissive to overwhelming dominance (e.g. a picture element that is much larger than the rest). The participants only have to indicate the option that best applies to their emotional experience with the object/product, for each dimension. It should be noted that this test only allows access to the general emotional state of the individual, so for this reason, it is advisable that this test should be applied in conjunction with other measures (e.g., physiological measurements). This test has the main advantage that it is easy to administer, answer and analyze, and, also, the fact that it is cheaper and may be applied on paper or computer. SAM has often been used in human-robot interaction (e.g., Mussakhojayeva et al. 2017).

Other examples of self-report instruments are: PANAS—Positive and Negative Affect Schedule (Watson et al. 1988) which is a self-reported questionnaire consisting of two 10-item scales to measure positive and negative affect with different versions developed for different purposes (e.g., PANAS-C, PANAS-X); STAXI—State-Trait Anger Expression Inventory (Spielberger 1988) which was developed to measure intensity of anger and the disposition to experience angry feelings (current version STAXI II, Spielberger 1994); STEM—State-Trait Emotion Measure (Levine et al. 2011) which assesses stable and current emotions at workplaces; Affective Slider (Betella and Verschure 2016) which is a digital self-reporting tool to measure arousal and pleasure on a continuous scale; among others. In fact, there are studies in human-robot interaction area which have used PANAS (e.g., Bucci et al. 2017; Mutlu et al. 2009).

Many other techniques have been used in the measurement of emotions, such as the analysis of facial expressions proposed by Ekman and Friesen (1978) and Ekman et al. (2002) called Facial Actions Coding System (FACS). FACS is a tool to measure facial expressions by dividing facial expressions into individual components of muscle movements. Some robots (e.g., Tutsoy et al. 2017) are detecting human’s emotions by using FACS technique while others (e.g., Loza et al. 2013) use FACS to define their own facial expressions.

Ten Emotion Heuristics (Lera and Garreta-Domingo 2007) is another technique which is dedicated to help measuring the affective dimension in the user evaluations. The heuristics are based on different theories such as FACS (Ekman et al. 2002) and The Maximally Discriminative Facial Moving Coding System (MAX) (Izard 1979).

Furthermore, other techniques such as analysis of the body movements (e.g., Tracy and Robins 2004) and vocal expressions/speech analysis (e.g., Juslin and Scherer 2005) can be used to assess in a more qualitative or quantitative manner.

Only recently, emotions came to be regarded as an important component of cognitive functioning (e.g., decision-making), and not just as something that negatively affects rational thought (e.g., Damásio 2003; Goleman 1995; Norman 2004), hence the recent history of emotions in the field of Design and Engineering. In this sense, one of the most significant developments of emotion in technology was to create products, objects, and machines capable of expressing, recognizing and feeling/showing emotions. That is, it became evident the importance of the individual establishing an emotional and empathetic relationship with the products, through its design, thus giving rise to emotional design.

Despite the relevance of the subjective evaluation of emotional states, it is sometimes necessary to evaluate the individuals’ emotional experience in a deeper way, i.e. accessing the internal, cognitive and physiological mechanisms that individuals are not able to report subjectively.

3 Objective Measures

The instruments previously discussed are subjective in nature as they rely on self-reporting. Consequently, what is being assessed with these instruments is the conscious recognition of one self’s emotions, i.e. emotions are to certain extends “rationalized”. As such, the unconscious emotional processes which can, in some sense, be regarded as “truther” are not being clearly evaluated. Additionally, although such self-reporting measurements are based on scores, they only provide a semi-quantitative assessment of emotions.

The use of objective physiological measures comes then to both quantify and assess the conscious and unconscious emotional processes (e.g., Ekman et al. 1983; Lang et al. 1993; Scherer and Wallbott 1994; Stemmler et al. 2001). As before, the two-dimensional model for emotions: Arousal and Valence are considered and are now correlated to quantitative physiological measures.

Understanding which physiological measures to consider, one has to understand how emotions are “generated”. This issue has not yet a definite answer, is a topic of discussion and research for generations, and often dwelling into the realms of philosophy. Scientifically though, the processes are starting to be understood and have their bases in neuroscience.

The nervous system is an intricate and complex information and communication system (see Table 9.1) (Kandel et al. 2013). It gathers information from the external environment by making use of senses (sight, hearing, touch, taste and smell) and, similarly, from the internal environment by making use of various bodily sensors such as muscle spindle sensors that provide information regarding proprioception (knowledge of body parts/movement) and pain (e.g. inner organs malfunctioning such as in a heart attack). Stimuli information from the external and internal environments is then communicated to the brain directly via nerves (e.g. optic nerve and facial nerves) or to the spinal cord via peripheral nerves. Information at the spinal cord is then forwarded to the brain (the brain and the spinal cord comprise the central nervous system) where it is processed and interpreted in complex cognitive and emotional processes, from which result in action that is communicated to other brain regions (e.g. storage of an event into memory), to the face (e.g. triggering the mimicry muscles for smiling) or to the body via peripheral nerves (e.g. informing the hand’s muscles to wave goodbye or increase heart rate during running).

Table 9.1 Simplified organization and roles of the nervous system and its components
Full size table

Unlike, cognitive processes (e.g. memory, calculus, navigation, executive functions), which are regarded as being held solely in the brain, emotional processes do have a bodily translation (e.g. when feeling anxious the palms get sweaty and the heart is pounding). In the event of an external (e.g. listening to the favorite music; touching a rose petal) stimuli or internal process (e.g. thought of the loved one), information is further processed in the brain where is emotionally interpreted and then translated into a bodily response.

The emotional communication between the brain and the body is then mediated via the spinal cord and the peripheral nervous system, which includes the somatic and the autonomous nervous systems.

The somatic nervous system communicates both conscious and unconscious brain processes, such as voluntary and involuntary movements. Emotional content is, nonetheless, more often perceived from involuntary movements such as body movements (e.g. gestures, body stances, and postures). The autonomous nervous system, on the other hand, communicates unconscious brain processes only (see Fig. 9.1).

Fig. 9.1
figure 1

Representation of physiological measures/actions and their relations to the somatic and autonomous nervous systems

Full size image

Emotional processing of stimuli in the brain often makes use of evolutionary older brain structures such as the diencephalon (hypothalamus and basal ganglia) and the limbic system but also of evolutionary newer brain structures such as the neocortex. The emotional output is then communicated to the autonomous nervous system which is comprised of sympathetic and parasympathetic systems, which have “opposite” effects. The former system is related to a “fight or flight” or stress reaction, in the sense that prepares the body for action. The latter system is related to a rest reaction as it prepares the body for “maintenance” (e.g. digestion and sleep).

The effect of sympathetic activation (or parasympathetic inactivation) results for instance in increased pupil diameter, heart rate, blood pressure, breathing rate, and sweating as the body prepares to better assess the nature of a threat (increased visual acumen) and makes resources available for action (increased blood flow delivered to skeletal muscles, heart, and head that results in increased local temperature and blushing). The effect of parasympathetic activation (or sympathetic inactivation) oppositely results in decreased pupil diameter, heart rate, blood pressure, breathing rate as the body is being set to a rest state, and also increased secretion of salivary and lacrimal glands.

The autonomous nervous system is a system that can react fast to changing environments and therefore to emotional processes. These processes also elicit the response of the endocrine system which has a much slower but long-lasting effect. For instance, in the event of chronic stress, there is a sustained increase in blood cortisol levels, a process which is mediated via the endocrine system.

Arousal is then understood as a degree of alertness/excitement, and therefore is more directly related to the activation of the autonomic nervous system and the endocrine system, but also to some extent to the somatic nervous system. Consequently, arousal can then be measured more directly in the body. Nonetheless, aspects of arousal can also be translated into facial mimicry (e.g. surprise)

Valence, on the other hand, does not have a simple correlation to the activation of the autonomic nervous system, as stimuli strongly positive or negative could provide similar autonomic reactions. Valence can be better perceived from facial mimicry (e.g. disgust) and speech and overall from the somatic nervous system body movements/posture and gestures. Also correlates of valence have been described regarding the brain activity, such as pre-frontal alpha wave asymmetry (see below).

As such, Arousal and Valence can be perceived via different physiological measures which can be assessed using different techniques. A resume is shown in Table 9.2.

Table 9.2 Emotional dimension, physiological measure and assessment technique
Full size table

Although less studied, the dimension of emotional dominance has also been correlated to physiological measurements, such as beta wave power in the midline brain, a measure of brain electrical activity.

The different techniques to assess the physiological measures are mostly optical, and electrical/electrode or sensor-based (here, chemical methods of assessment of emotions via analysis of bodily fluids such as blood and saliva are not discussed). In particularly, when the brain or body electrical activities are assessed it is referred to electrophysiological techniques, which will be described in more detail below.

Optical techniques include the use of visible or infra-red video cameras, which may have 3D depth perception, to assess facial mimicry, gestures, postures, and movements. These cameras can further be combined with markers placed on the body for increased precision.

When used for facial mimicry these cameras are combined with recognition/computer vision algorithms based on the Facial Action Coding System (FACS) method. This method codes facial expressions, depending on involved muscles and was developed by Paul Ekman and Wallace V. Friesen in 1976. Nowadays, companies such as Affectiva offer software development kits that facilitate the development and deployment of facial mimicry recognition applications. For the assessment of body movement, other algorithms are used instead. Microsoft’s Kinect is one of the first and most well-known system for body movement recognition and more recently Leap Motion developed its controller which recognizes gestures with greater accuracies.

The same Kinect or other video cameras can make use of an algorithm called Eulerian Video Magnification to magnify subtle changes of color of the skin at each heartbeat: due to increased blood flow in the face, the skin is slightly redder. In this manner, it is possible to assess heart rate in a contactless manner (Gambi et al. 2017).

Other sophisticated infrared systems are eye trackers, which have been fundamental in the Design process. These instruments allow to perceive the points in an image or video in which users focus their gaze  and also enable the measurement of pupil diameter, thus informing on the cognitive processes of exploration of an image of a product for instance and the arousal levels such product elicites in the users (Poole and Ball 2006).

The use of the eye tracker already has more than a century of history, existing records of its use in Javal (1878). However, the first eye trackers were very invasive and dangerous measuring instruments, because to measure the ocular movement, they had to be in contact with the cornea.

It is estimated that the first non-invasive eye tracker was created in 1901 by Dodge and Cline. However, this instrument was very limited (e.g., participants could not move their heads). Over the years, the eye tracker technique has been improved and used primarily in research and studies on reading.

Its evolution was such that in 1948, Hartridge and Thompson, created the first head-mounted eye tracker. Over the years and as the technology improved, eye trackers have extended their use to other areas of knowledge, such as human-computer interaction (e.g. Josephson and Holmes 2002; Reeder et al. 2001).

Eye movements allow the individual to maintain or shift the gaze between various visual areas (Goldberg and Wichansky 2003) by capturing information that is then transmitted and processed in the brain.

This set of information collected through the eye tracker may prove to be fundamental for the designer to identify the points of the object that most catch the attention of the user, and thus increase them.

Simpler optical systems comprised of light emitting diodes (LEDs) and photoresistors or other photodetectors are used to assess blood-volume pressure (BVP) signals, a technique called photoplethysmography (PPG). BVP signals translate the amount of blood flowing through a vessel and can be used to compute the heart rate and heart rate variability, but also the breathing rate, and a correlate of the blood pressure. In fact, just after a heartbeat, excess blood flow is observed in tissues in a pulsatile manner which will reflect or transmit light differently, enabling it to be detected.

Further development of these systems which include both red and infrared LEDs is oximetry. Besides the former measures discussed above, it can also provide oxygen saturation, translating increased or decreased oxygenated blood flow in the brain that may result from an emotional response. Finally, temperature can also be assessed using an infrared LED or infrared cameras, as well as with other methods.

Other physical sensors include piezoresistors, and other force and pressure sensors may be used to measure breathing rate and breathing patterns via a flexible sensor strap placed around the thorax. Pressure sensors are typically used for assessing blood pressure, or as microphones can also be used to capture speech, which is then classified by specialized algorithms.

Inertial measurement units (IMUs) typically comprise accelerometers, gyroscopes, magnetometers/compass and, also, more recently barometers. IMUs are used to assess both voluntary and involuntary movement and may be used also to assess speech, by measuring the vocal box vibrations.

Electrophysiological techniques, on the other hand, are used to measure a large variety of brain and body electrical signals (see Fig. 9.2), especially associated to emotions and the autonomous nervous system but also to cognitive processes and to the somatic nervous system.

Fig. 9.2
figure 2

Adapted from Ribeiro et al. (2014)

Example of electrophysiological signals tracings: A—Electroencephalography; B—Electromyography; C—Electrooculography; and D—Electrocardiography.

Full size image

Since it was invented by Hans Berger in 1929 (Berger 1929), electroencephalography (EEG) has been the neurophysiological technique by excellence, as it measures the electrical activity of the brain. Both EEG signals and their frequency components (delta, theta, alpha, beta and gamma) have been associated with arousal, valence and dominance (Reuderink et al. 2013):

  • Delta (0.5–4 Hz): a power increase in the posterior right hemisphere is associated with increasing arousal;

  • Theta (4–8 Hz): a power decrease in the frontal cortex is associated with increasing positive valence and to attention;

  • Alpha (8–13 Hz): a global power increase is associated with increasing arousal; additionally, a frontal left-right hemispherical asymmetry correlates with valence with a decrease in power in the left/right hemisphere for positive/negative emotional information, respectively;

  • Beta (13–30 Hz): a correlation in the midline with dominance;

  • Gamma (>30 Hz): asymmetry in the temporal cortex was found associated with valence.

Additionally, EEG signals have been shown to assess attentive/focused and meditative/resting states. In attentive/focused states an increase in beta power and a decrease in alpha power are observed (Marrufo et al. 2001). In meditative states, an increase in alpha power in the occipital cortex, and an overall increase in both theta and beta powers were observed as well (Ahani et al. 2014).

Electromyography (EMG) is a technique that translates skeletal muscle activity. It is therefore related to the somatic nervous system and can translate both conscious and unconscious cognitive processes. EMG sensors are typically placed over the facial muscles in order to detect expressions related to facial mimicry (Bartlett et al. 1996), but they can also be placed in the upper or lower limbs to perceive movement or event jitter associated to an anxious state (Haag et al. 2004). Finally, EMG sensors can also be placed over respiratory muscles in order to assess the breathing rate, which is under control by the autonomous nervous system. Fast breadths can be associated with high arousal states while slow shallow breadths can be associated with low arousal states. Here, the deepness of breadth (as assessed by the amplitude of breathing) can further discriminate between emotions, whereas fast and deep breading can be related to an excited state, which can be positive or negative in valence, fast and shallow breading can be related to tense anticipation. Also, slow and deep breading can be related to a relaxed state, whilst a slow shallow breathing may be associated with depressive syndromes or calm happiness (Haag et al. 2004).

Electrooculography (EOG) measures the electrical activity associated with eye movements. EOG may be used as a complement to EEG for instance for the detection of EEG signal artifacts but can also be used to monitor eye movements as scanning of a particular visual stimuli, and detect blinking rate (Calvo and D’Mello 2010), which may help translate a particular emotional state (e.g. anxiety, withdrawal) and also be used to detect facial mimicry around the eyes (Setz et al. 2009).

Electroglottography (EGG) measures the electrical flow across the larynx which is increased when focal cords are closed. It can be used to depict the emotional content of speech (Hui et al. 2015).

Electrocardiography (ECG) measures the electrical activity of the heart and therefore measures the activity of the autonomous nervous system. It is perhaps the most widely used technique for assessing emotions using physiological signals. These signals can be measured using electrodes over the chest or even on the fingertips using a two-electrode setup. Heart rate (HR) and heart rate variability (HRV) can be easily computed from ECG signals and can translate a state of relaxation when the HR is decreased  and the  HRV is increased  and a state of anxiety/stress when the HR is increased and the HRV is decreased in regards to baseline values (Haag et al. 2004). Finally, ECG signals can also be used for artifact removal in EEG signals.

Finally, the electrodermal activity (EDA), also known as, galvanic skin response (GSR) is also a widely used electrophysiological signal for emotion recognition. It translates skin conductance, which changes accordingly to the skin sweat, as already first studied by German physiologist Emil Heinrich Du Bois-Reymond in 1849. EDA then assesses the autonomous nervous system, especially the arousal level. As such, EDA has been found to be a useful way to assess stress and to differentiate between conflict/no-conflict situations. The main drawback of this technique is its dependence on external factors such as temperature (Haag et al. 2004).

All these signals have been used individually, but multimodal approaches have shown improved accuracies in emotion detection (Kim et al. 2004). This type of measures involves the application of sensors and electrodes in the individual, so that, although painless and harmless, these methods are more intrusive and more expensive than self-report instruments, but also much more powerful.

So far, considering what was discussed, EEG is the only technique that directly measures brain electrical activity. The other techniques mostly measure the outcome of brain processes communicated to the peripheral nervous system.

More recently, other more complex and expensive neurophysiological and neuroimaging techniques have been used to study emotional processes in the whole brain. In this regard, besides EEG, the magnetoelectroencephalography (MEG) technique is being used for the study of emotions within the millisecond time-scale (Giorgetta et al. 2013). Moreover, when both EEG and MEG are used for high-density recording their output may result in functional images, which may provide advantageous views on the data.

Other functional neuroimaging techniques include functional near-infrared spectroscopy (fNIRS), functional magnetic resonance (fMRI) and positron emission tomography.

fNRIS uses similar technology to oximetry, but sensors are spatially arranged in order to provide increased spatial information. Nonetheless, fNRIS does not provide a direct measure of brain activity, as EEG does. Instead, it assesses the local increased arterial blow flow resulting from the increased demand of local neurons for oxygen and glucose during stimuli processing or during a task (Nishitani and Shinohara 2013).

fMRI works similarly as fNIRS, as it depends also on local increased arterial blow flood. In this case, though, it makes use of the proton properties of tissues, not the optical properties. fMRI as an imaging technique is able to provide better spatial localization of the brain processes, although being a more expensive and less available technique (Phan et al. 2002).

The PET technique is the most specific of all, as radiotracers can be designed and used to address a particular metabolic route or neurotransmitter receptor. In this manner this technique as the greatest potential to enable a deeper understanding of molecular and cellular functional processes. Nonetheless, when comparing to fMRI, a number of disadvantages has to be equated, such as the invasiveness/need to inject a contrast agent, use of ionizing radiation, longer scan times, smaller temporal and spatial resolutions, smaller market availability and higher costs.

These techniques are all related to the measurement of cognitive and emotional processes. Nonetheless, it is possible to study these processes by modulating/manipulating the brain and observing the outcomes. These interventional techniques include EEG-based brain-computer interfaces (BCI) which operate via neurofeedback, exposing the user to a translated version of his/her own physiological measures and enabling the progressive control of such measures (Johnston et al. 2011).

Other interventional techniques include transcranial current stimulation (tCS) and transcranial magnetic stimulation (TMS), which both modulate tissue electrical excitability via the application of electrical currents (in the former) and magnetic fields (the latter) (Martin et al. 2018; Notzon et al. 2017). As being interventional and more recent, there is still a great investment in assessing the safety profiles and effectiveness of such techniques, but their potential, particularly in combination with the other measurement techniques, is enormous.

4 Conclusion

Emotions are ever more a topic of research and, correspondingly of application. As more and more studies put in evidence the critical role of emotions; in what makes us humans like and dislike, love and hate, and motivate, lead and decide; the more important is to have awareness of the subjective and objective tools and techniques that are available and that are being developed for measuring (and also for modulating) emotions. That is so more important in the Design field, as the designer makes use of his/her emotions to dream and create, and the user, as more than a recipient, also co-participates in the design process guided by his/her own emotions, aesthetics and purchasing decision.

In this chapter, besides subjective measures, neural-correlates of emotions, related physiological measures, and assessment techniques are described, hopefully in with the right mix of overall perspective and detail. We further hope to have given our reader, the designer (and other related professionals), additional tools to create ever-more valued products and services.