Keywords

1 Introduction

Emotion is present in virtually all uses of technology. It is, therefore, not surprising that the study of users’ emotions is a large and ever-growing subfield of human–computer interaction (HCI) research. This research has proposed multiple theoretical and empirical approaches to detecting, predicting and explaining users’ emotions during interactive tasks. However, these approaches are often agnostic to any strong theoretical commitments about what emotion is. Yet, if theoretical assumptions are not explicated in operationalisations of user emotion, this will confound the results and make argumentation imprecise. This limits the capacity of HCI research to solve emotion-related problems.

We argue that formalising the role of the human cognitive system with the emotion process can help overcome this limitation. To this end, we review the contemporary psychological research on emotion, with an emphasis on the appraisal process. The appraisal theory defines emotion as a partially cognitive process that evaluates the significance of an event and an individual’s ability to cope with it. We discuss the use of theories related to emotion in HCI and demonstrate how the appraisal on account of emotion can improve our ability to understand users’ emotions. We explore the implications for different types of studies of user emotion using examples from experimental research on visual and emotional user experience.

When users approach interactive technologies, it is necessary to understand their goals and knowledge in order to explain their behaviour. For instance, an individual may approach a ticket vending machine with the intention of buying a ticket. If the user has accomplished this goal previously, she understands the interactive steps that must be accomplished, and can, therefore, finish the task faster than a novice user with no knowledge of how the interface works. The emotions that these users have during the interaction are conditioned on their goals and knowledge. For example, if the user encounters a problem while buying the ticket, her emotional response is conditioned on her ability to overcome the problem. A more experienced user might know the solution to the problem, but feel frustrated that a bad interface design makes her task more cumbersome. Conversely, a novice user might feel distressed because she does not know how to overcome the problem, and may fail to buy the ticket (perhaps for a train that is leaving soon). In these examples, it is clear that explaining emotion requires referring to the users’ goals and knowledge, and to the event of the interaction. The theoretical task of analysing the psychology of emotion is to explain how these factors influence the users’ emotional responses.

We argue that an appropriate theoretical approach to analysing the psychology of emotion involves using the appraisal theory of emotion, which posits that emotion is a continuous appraisal process that evaluates the subjective significance of an event. The key in this evaluation is to recognise that emotions are responses to how events are evaluated within the subject’s meaning structures (Arnold 1960; Frijda 1988; Lazarus 1966; Scherer 2009). Analysing these meaning structures will help explain an individual’s emotional responses to a given event. A person’s appraisal of something as ‘good’ and approachable or desirable, instead of ‘bad’ and avoidable or undesirable, is a function of their conceptions of good and bad. This in turn depends on what they know about the situation and its circumstances, as well as their personal preferences and goals.

The appraisal theory analyses emotion as a partially cognitive phenomenon, in which information from multiple sources is processed during the appraisal (Smith and Kirby 2001). It also integrates multiple components of emotion, such as physiological responses, motivation, categorisation and labelling (Scherer 2009). This is important for employing the theory in HCI research, which has a long methodological tradition of collecting users’ subjective and physiological responses to the use of technologies (Jokinen 2015b). In this context, appraisal theory has the potential to integrate the currently theoretically disparate strands of emotion research in HCI. Moreover, because the theory presents emotion as a causal system from an initial stimulus to a corresponding emotion and its behavioural effects, it provides a way to assess counterfactual scenarios. The ability to predict human responses in different types of ‘what if’ scenarios is paramount for any interface and task design that claims to be based on scientific thinking.

2 Background

Although emotion has always been the source of philosophical debate, it has only recently been examined scientifically. Even after the birth of modern psychology as the science of the human mental life in the nineteenth century, research was mostly focused on human cognition; emotion received less detailed and thorough analysis (Baddeley 2007). The theoretical foundations of all current major psychological theories of emotion were laid in the second half of the twentieth century (Arnold 1960; Ekman and Friesen 1971; Russell 1980), but detailed models and rich empirical data were not developed until decades later (Izard 2007; Russell 2009; Scherer 2009; Jokinen 2015a). Although we focus on the appraisal theory of emotion, we briefly describe other major theories for comparative purposes.

2.1 Basic Emotions Theory

To study the hypothesis that emotion is based on evolutionary development and is, therefore, universal in humans, Ekman and Friesen (1971) researched two Oceanic Neolithic cultures. Both cultures were isolated from any Western influence, and were, therefore, good subjects for the study of the universality of emotion. The researchers told the study participants stories that involved emotions (happiness, anger, sadness, disgust, surprise and fear), and asked them to identify which picture of a facial expression best fits with each story. The participants largely identified the intended emotions, which supported the universality of emotion hypothesis.

Theories that state that emotion has a psychobiologically universal pattern are called discrete or basic emotion theories (Ekman 1992). These theories maintain that each set of discrete emotions has a distinct pattern of bodily change, a physiological response and antecedent events. By emphasising the word ‘basic’, the proponents of these theories claim that (1) there is a fixed number of ‘basic emotions’ that differ from each other and (2) these emotions are ‘basic’ because each of them has adaptive value for human life (Ekman 1999). Researchers have different conceptions of which emotions are ‘basic’, but they generally agree that there are relatively few such emotions.

Discrete emotion theories are often used in the subfield of HCI called affective computing, in which researchers seek to establish a connection between a user’s physiological responses and emotional states (Picard 1997). Experimental research in affective computing often reports correlations between self-reported or inferred emotions and physiological measurements, such as heart rate (Drachen et al. 2010), galvanic skin response (Mandryk et al. 2006) or pupil size (Partala and Surakka 2003). However, there are still no universal solutions for the original challenge posed for affective computing—developing an automated emotion-detection machine. Indeed, meta-analyses of the psychophysiology of emotion have yielded mixed results with regard to our ability to detect discrete emotions using neuroscientific instrumentation (Barrett 2006; Cacioppo et al. 1997, 2000; Shiota et al. 2017).

2.2 Core Affect Theory

Russell (1980) was interested in finding latent structures in the ways that people categorise emotions, and asked participants to group emotions by similarity. By analysing the emerging latent structures, Russell demonstrated how people often represent emotions using two dimensions, valence (pleasantness) and arousal (activation). This can be called the circumplex model of affect; it places each emotion in a circle according to its valence and arousal. For instance, sadness and pleasure are different due to valence, and sleepiness and excitement are different more due to arousal (although in both cases, there is also a noticeable difference in the other dimension).

Using the circumplex model of emotion, Russell and Barrett (1999) studied how the relationship between the physiological emotion process and affective experience. They proposed that the human emotional system can be considered a fundamental element of emotion, which they called core affect. Core affect is the simplest part of the emotion process that is accessible to the human consciousness (Russell and Barrett 1999). It combines the component values of valence and arousal, and can be identified as a single point in the circumplex—but not one of the emotion words that can be used to describe that emotion and located on the circumplex, as it is an elementary concept. Therefore being in love, for example, can be located on the pleasant side (“feeling good”) of the horizontal axis of the circumplex, but love is not a core affect itself. Part of being in love is feeling good and at least somewhat aroused, and this is the core affect; however, love also entails other components (for example, it is usually directed at someone).

The core affect approach to emotion is very popular in HCI, because it provides a framework for classifying emotions that is easy to use and understand. One useful and often applied method is the self-assessment manikin (SAM) scale (Bradley and Lang 1994), which lets participants rate their emotional state using pictorial scales for valance and arousal (e.g. Zimmermann et al. 2006). Thanks to the fairly simple two-dimensional physiologically interpretable construct, core affect is also often used in affective computing. Affective computing research has identified physiological correlates for both arousal and valence (Lichtenstein et al. 2008), although as noted above, these correlations are often not robust across task domains and experiments (Cacioppo et al. 1997, 2000).

2.3 Appraisal Theory

The multiple theories about the emotional appraisal process all assume that emotion is related to the evaluation of an event (in the environment or within the individual), which implies it is a complex organised system (Smith and Kirby 2001; Scherer 2009). Further, appraisal is assumed to relate to the desirability or avertability of the appraised stimulus, which assigns an adaptive function to emotion (Arnold 1960). Emotions are elicited in situations that have adaptive significance to the individual; emotions prepare and motivate the individual, and help acquire and filter relevant information (Scherer 2009).

The emotion process can be categorised into four distinct, causally related, components: motivational changes, physiological response patterns, a central representation and verbalisation (Scherer 2009). It might be tempting to conceptualise emotion as the outcome of the appraisal process, but it is better to define it as the appraisal process itself, since it is present in all of the four components. For example, fear might be associated with a motivation to flee and an increase in heart rate, as well as with a conscious experience of being afraid. These components interact with each other, for example, the experience of fear may not only increase a subject’s heart rate; the subject may perceive this increase, which changes the experience of the situation and may activate some coping mechanisms—and subsequently alter his or her physiological response. This means that emotion is a multilevel process, not a state. The confusion between these two arises from conflating subjective feeling (which is a single part of the emotion process) with the entire emotion process.

The appraisal theory is not necessarily in conflict with the theories of basic emotion and core affect described above. Rather, these can be subsumed into appraisal theory. The physiological component of appraisal theory can account for the propositions of basic emotion theories, and the core affect theory describes one way in which emotions can be represented, categorised and verbalised. Methodologically, one can operate within the assumptions of appraisal theory and still conduct research on emotions in HCI using psychophysiological methods, such as in affective computing, or collecting subjective emotion data using the SAM scale (Jokinen 2015b). Importantly, appraisal theory also allows the researcher to make hypotheses about the causal evaluative processes of the observed results. The next section gives examples of how this can be accomplished.

3 The Appraisal Theory in HCI: Case Examples

3.1 Visual Experience

The scope of HCI research has broadened in recent years. The focus is no longer merely on sensemaking processes including research on functionality and usability. An increasing amount of research is seeking a holistic understanding of HCI, covering issues related to the aesthetics of interaction (e.g. Hassenzahl 2004; Hassenzahl and Monk 2010; Hekkert 2006; Moshagen and Thielsch 2010; Thüring and Mahlke 2007), emotions in technology experiences (e.g. Bødker 2006; Hassenzahl and Tractinsky 2006; Norman 2004) and meaning-making processes in visual technology experiences (e.g. Desmet and Hekkert 2007; Krippendorff 2006; Silvennoinen et al. 2017).

This change in scope has also diversified the variance and complexity of results related to the affective dimensions of HCI. In particular, differences in discussing the underlying dynamics of visual experience in HCI vary greatly. Many hypotheses have been presented to examine the cognitive–affective operations associated with how we make sense of and experience the visual interfaces of technological artefacts. Differing results have been presented, for example, regarding the relationship between aesthetic evaluations and perceived usability. Aesthetically pleasing user interfaces can make us more tolerant of inconsistencies between system properties, affect our attitudes towards technology and positively increase performance (e.g. Norman 2004; Moshagen et al. 2009), but can also have negative effects (e.g. Sonderegger and Sauer 2010). Many of the inconsistencies have emerged due to unsolid theoretical grounds.

Tight industry relationships and a lack of meta-research and replication studies (e.g. Liu et al. 2014) leave aside crucial developments in theory development. This chapter presents an appraisal-theory-based understanding of visual technology experience (Jokinen et al. 2015, 2018; Silvennoinen and Jokinen 2016; Silvennoinen 2017) to clarify theoretical approaches to examining emotional user experience and present methodical possibilities for examining emotions as cognitive processes within HCI.

Affective appraisals are at the core of visual experiences. Technological artefacts are meaningful to the people interacting with them due to the mentally represented qualities people attribute to them (Silvennoinen 2017). The appraisal theory has been used to examine emotional responses in product experiences due to its ability to explain emotion as a process (Demir 2009). Thus, the theory can explain the relationship between visual experience and a design artefact in how a subjective experience emerges from the appraisal process to encounter a design artefact. This further allows the design of such experiments, where the details of the appraisal process can be manipulated to examine the relationship between design artefacts and experiences.

Jokinen et al. (2015) examined the visual experience of shapes using a primed product comparison method. The stimuli for the experiment were pictures of drinking glasses. As the three information sources of the appraisal process have different computational demands (Smith and Kirby 2001), the authors were able to conduct an experiment that examined how these three sources are involved in experiencing product shapes. The primed product comparison method is based on reaction times and preference scores. A participant is given a prime or a cue (i.e. a word) and a stimulus pair (here, two images of drinking glasses). All the different combinations of cues and stimuli are evaluated. The participant is asked to quickly choose between the images based on the word supplied. The speed at which they were required to make the choice enables the detection of different appraisal levels to the time it takes to make a judgment. Experimentally, time is of the essence, as it enables the examination of culturally and linguistically complex elements in conscious experience.

This method can be used to examine the cognitive appraisal process in which subjective experience occurs in considerable detail. Jokinen et al. (2015) identified different levels of the appraisal process in experiencing product shapes, and connected different cues to certain shapes. Some of the appraisal criteria were judged more quickly, indicating that a shorter information processing time is associated with certain appraisal information sources. Stimulus pairs that are dissimilar to each other were also judged faster than pairs with similar shapes. Faster judgments were performed when the appraising cues depicted physical characteristics of the artefacts (e.g. durable or light), which require less associative processing and reasoning than more complex cues (e.g. traditional or timeless).

Silvennoinen and Jokinen (2016) examined appraisals of icons from different design eras. They used the primed product comparison method to examine the process of experiencing icons. Preferences and their processing times were analysed in terms of perceived visual usability and the aesthetic appeal of icons from four different design eras. These two characteristics are underlying dynamics of an overall visual experience (Silvennoinen 2017), which can be further examined in detail with the appraisal theory. Perceived visual usability was operationalised as semantic distance, i.e. the closeness of the icon’s pictorial representation to its intended function (e.g. Silvennoinen et al. 2017). The primes or cues were the icons’ intended functions (save, print and search). Aesthetic appeal was operationalised using concepts from traditional accounts of aesthetics emphasising positive engagement, intrinsic value and design-era dependency (beautiful, old-fashioned and familiar).

In addition to the differing research results regarding the relationship between aesthetic appeal and perceived usability, ease of interpretation has been reported to enhance aesthetic appeal (Reber et al. 2004; Reppa and McDougall 2015) via cognitive processing fluency; it is reportedly enhanced by stimulus familiarity (Isherwood et al. 2007). Experiencing a visual representation as familiar requires an evaluative appraisal framework. Although an icon would be familiar and easily interpreted, cognitive processing fluency determines whether it is experienced as aesthetically pleasing.

An appraisal-theory-based understanding of visual experience was utilised to explicate the connection between processing fluency, familiarity and aesthetic appeal. Aesthetic appeal and perceived visual usability preferences of the four icons varied, which allowed the icon experience to be examined as a process. Experiencing a stimulus as appealing results from cognitive information processing. Examining the influence of design eras on visual experience, perceived visual usability functions as an underlying factor of aesthetic appeal, and can increase the possibility that an icon will be experienced as pleasing. In addition, judgments regarding visual usability are more unanimous than those related to aesthetic appeal.

Since familiarity increases the fluency of cognitive information processing, familiar stimuli are experienced faster than non-familiar ones. Familiar stimuli have lower activation thresholds for relevant long-term memory nodes, which enables more automated reasoning. However, speed and ease of interpretation do not determine aesthetic appeal. Ease of interpretation is desirable in itself (especially in HCI, where the goals are often efficiency related), but should not be extended to explicate the affective qualities of visual experiences. Visual experience is more complex, as the icon experiment indicates. We are able to interpret familiar stimuli quickly as understandable, but also as unappealing.

Designing for certain targeted experiences is a difficult task. How we experience visual representations varies, and is influenced by numerous factors. There are no predetermined relationships between design elements and how these are experienced. Jokinen et al. (2018) employed appraisal theory and predictive brain theory (Clark 2013) to examine the visual experiences of user interfaces for targeted emotional outcomes. Predictive brain theory integrates recent advancements in cognitive science, indicating that brains constantly match sensory inputs with top-down expectations that support perception and action (Clark 2013).

The primed product comparison method has been combined with eye-tracking measurements to connect certain visual user interface design decisions with targeted experience goals (Jokinen et al. 2018). Participants appraised website designs with affective experiential adjectives, such as modern, civilised and beautiful. This study described the relationship between the three appraisal information sources not as distinct and separate entities, but as interwoven in a complex process. In the appraisal process, bottom-up perceptual stimuli are integrated with top-down associative information and reasoning. Thus, appraisal theory and predictive processing theory can be used to examine the complex relationship between design artefacts and human experience, as they predict how experience occurs in the design appraisal process. The details of the appraisal process can be experimentally manipulated to examine the relationship between design and experience. The methodological approach that combines appraisal theory and predictive brain theory with visual experience can be further utilised to empirically test predictions, for example, concerning cognitive processing fluency and the relationship between visual designs and visual user experience.

3.2 Emotional User Experience

In most HCI tasks, the users can be assumed to have goals that they attempt to achieve during the interaction. Given the goal-oriented nature of interactive behaviour, users’ emotions during the interaction can be analysed in terms of goal congruency (Jokinen 2015a). Under appraisal theory, the user evaluates the significance of the interaction events to his or her goals. Generally, if the event is appraised as goal congruent, the resulting emotion is positive, whereas goal-incongruent events cause negative emotions. This analysis can be likened to the original intuition of appraisal as an evaluation of an event as either approachable (the event facilitates goal attainment) or aversible (the event obstructs goal attainment).

This level of analysis is still, however, quite trivial, as it does not explain the variety of emotions—positive and negative—that users may experience when interacting with technology. A goal-incongruent event may result in being either frustrated or sad, for instance, and it is the task of the appraisal theory to explain what conditions explain these contingencies: the goal of the appraisal-theory-based analysis of users’ emotions is to give a causal account of their emotional responses. For example, Jokinen (2015a) and Saariluoma and Jokinen (2014, 2015) investigated users’ subjective emotional responses to task events in computerised tasks. Using a questionnaire with a number of emotion-related items to probe the emotions, and different experiment manipulations, they were able to make causal inferences about the appraisal process. The authors used a competence–frustration model of emotion, where they defined competence as the positive emotion resulting from goal-congruent events such that the user perceives that these events were a result of his or her own actions. Frustration is defined as resulting from goal-incongruent events that obstruct the user from reaching attainable goals.

Saariluoma and Jokinen (2015) tested the goal-oriented nature of competence and frustration. They separated participants into two groups, which either looked at screenshots of online shops or conducted ordinary tasks on them. Then they reported their experiences using an emotion questionnaire. Contrary to the authors’ initial expectations, the two groups did not differ in levels of self-reported competence. This means that the participants in the group without a goal-oriented task reported feeling competent. Initially, this seems a strange result due to the definition of competence as the appraisal of task-congruent events that result from applying one’s skills. If a user passively watches screenshots without any tasks that manipulate the states of the interactive system, they should not feel competent. However, the authors had also asked the participants about their emotions at the start of the experiment before showing them any stimuli. For the group that did not have interactive tasks, there was a correlation between pre-and post-test questionnaire responses, meaning that their feelings of competence and frustration after watching the screenshots were mainly predetermined, and not affected by the experiment tasks. Conversely, the group that conducted tasks did not have this correlation: their competence and frustration depended on task performance.

Saariluoma and Jokinen (2015) concluded that only events that occur in goal-directed interactions impact emotions, whereas those in non-directed, passive interactions do not. Jokinen (2015a) investigated the connection between task events and emotions in more detail. First, he observed the expected correlation between task performance and emotional responses. He also investigated the impact of pre-task self-reports of emotion, and found that the impact of task performance on emotions depended on prior, pre-task emotions. For instance, Jokinen (2015a) probed the participants’ self-confidence at the start of the experiment to test whether self-confidence translates into competence during the experiment. Indeed, participants who reported low self-confidence started the experiment tasks with a below-average sense of competence. However, low self-confidence participants who were successful in the tasks reported higher levels of competence than those who started the experiment with more self-confidence. In other words, self-confidence has a negative moderating effect on the relationship between task performance and self-reported competence. This is in line with the appraisal theory’s statement that the same event can result in different emotional responses, depending on the subject.

Jokinen (2015a) also investigated the second stage of the appraisal process—coping—in more depth. He explored two coping strategies, problem-solving oriented and emotional coping. He hypothesised that feelings of competence should result from problem-solving-oriented coping alongside goal-congruent events (that are the result of problem-solving), and that feelings of frustration should be moderated by the emotional coping strategy when there are incongruent events. Jokinen (2015a) treated these coping strategies as individual traits that may vary between users. To measure this trait he used a coping questionnaire, which he adapted to the context of technology use. In the experiment, the participants accomplished tasks in different software environments, such as text editing or image manipulation, and reported their emotions. Jokinen (2015a) observed the expected effect of task performance on competence and frustration. He also confirmed the coping trait hypothesis: self-reported coping traits impacted self-reported emotions, demonstrating that emotion is an individual phenomenon that cannot be understood without referring to the subject’s internal processes, such as coping.

4 Discussion

The appraisal theory of emotion explicates explains emotion as a process. We here demonstrated the plausibility of applying this theory in an examination of technology experiences and an explication of how experience occurs in HCI. The appraisal as a process operates via cognitively evaluating the subjective significance of an event, and occurs on multiple levels, including physiological level, motivational level and subjective experience. We used multiple example studies, grouped into two main categories, to demonstrate how the appraisal theory can be applied in the study of the human visual experience, and in the study of emotions in technology interaction. The examples showcase the theory’s usefulness of the appraisal theory in HCI research, particularly as. As observed, the benefit of appraisal theory in this regard is that it defines emotion in terms of events of the task environment, and in terms of as well as a subject’s knowledge and goals of a subject. Thus, the cognitive appraisal process integrates these factors and enables the explication of a variety of subjective emotional experiences in its variety, which are conditioned (and, therefore, explainable) by the circumstances in which the emotion was elicited.

For example, the appraisal theory analyses emotion as a partially cognitive phenomenon, in which information from multiple sources is processed during the appraisal (Smith and Kirby 2001). We exemplified how these information sources of subjective experience are intertwined in top-down and bottom-up processes, leading to testable hypotheses regarding how visual experience occurs and dynamically changes during interactions. The same applies to the appraisal theory-based understandings of visual experience, wherein bottom-up and top-down sources of information dynamically influence the subjective experience.

Different appraisal information sources in experience can proceed from the identification of physical qualities associated with abstract meanings involving higher level cognitive reasoning. For example, appraising a material as warm involves a relatively direct process of temperature recognition and touch perception, but appraising a material as timeless entails association and reasoning. The appraisal theory of emotion is not dependent on sensory modalities. Its logic can explicate technology encounters that are induced by and experienced with different sensory modalities (e.g. Silvennoinen et al. 2015). In addition, the primed product comparison method can be utilised to examine experience as a process pertaining to other senses, such as hearing. The primes can thus be sounds, for example. Using this method, different evaluation times between the stimuli and the primes indicated differing mental processes in visual experience, as predicted by appraisal theory.

The methodology presented in this chapter provides grounds for HCI researchers to examine subjective experiences in HCI. Designers can also use the primed product comparison method to analyse how well their visual designs (and the intended experience goals) correspond to users’ experiences.