4.1 Introduction

In this chapter, we are going to discuss modeling aspects in human-computer interaction. To be more precise, we focus on a specific scenario, namely, information search. The possibility to acquire new information is an important functionality of Companion-Systems. If the potential user requires explanation of some previously unknown facts and this information is not available in the knowledge base of a Companion-System directly, then the Companion-System can gather it from various sources like the Web. Through a dialog with a user the Companion-System clarifies the user’s intention, eliminates any ambiguities and guides the user towards successful information gain.

Search systems are an integral part of our lives. Most common known search systems come in the form of web search engines with an audience of hundreds of millions of people all over the world. This is a very wide and heterogeneous target group with different backgrounds, knowledge, experiences, etc. Therefore, researchers suggest providing a customized solution to cover the needs of individual users, e.g. [15]. Nowadays, solutions in personalization and adaptation of backend algorithms have been proposed in order to support the search of an individual user. These solutions include query adaptation, adaptive retrieval, adaptive result composition and presentation [38, 39]. But the front end, i.e. the search user interface (SUI), is usually designed and optimized for a certain user group and does not support personalization.

Common search engines allow the personalization of a SUI in a limited way: Users can choose a color scheme or change the settings of the browser to influence some parameters like font size. Some search engines also detect the type of device the user is currently using—e.g. a desktop computer or a mobile phone—and present an adequate user interface (UI). Current research concentrates on designing SUIs for specific user groups, e.g. for children [11, 15, 22] or elderly people [3, 4]. These SUIs are optimized and adapted to general user group characteristics. However, especially young and elderly users undergo fast changes in cognitive, fine motor and other abilities. Thus, design requirements change rapidly as well and a flexible modification of the SUI is needed.

Another important, currently under-emphasized aspect is the adaptation of the interaction dialog with the user. Different users require different dialog characteristics, e.g. a simple language should be used in dialogs with children. Moreover, the same user can have different cognitive and emotional states during the interaction with the system. Thus, modeling the user’s emotional development during an interaction with a system could be a first step towards a representation of the user’s inner mental state (cf. [19]) and therefore a step towards an adaptive interaction. This provides the possibility to evolve a user interaction model and gives the opportunity to predict the continuous development of the interaction from the system’s perspective. As stated in [27], moods reflect medium-term affects, generally not related to a concrete event. They last longer and are more stable than emotions and influence the user’s cognitive functions directly. Furthermore, the user’s mood can be influenced by emotional experiences. These affective reactions can be measured by a technical system. In this case, the mood can be technically seen as a long-time integration over the occurring emotional events, to dampen their strength. User emotions and dispositions were considered in information retrieval before, however, mostly only for relevance assessment [2] or as additional features, for instance, in collaborative filtering [28, 29].

To have a customized solution to cover the needs of individual users, we suggest an users with an evolving user interface that adapts to providing individual user’s characteristics and not only allows for changes in properties of UI elements but also influences the UI elements themselves and the human computer interaction dialog.

4.2 Model of an Evolving User Interface

We exploit a generic model of an adaptive system based on [39] and propose the model of an evolving user interface (EUI) as follows (see Fig. 4.1): In general, we suggest designing a mapping function and adapting the UI using it, instead of building a UI for a specific user group. We have a set of user characteristics or skills and emotional states on one side. We suggest considering cognitive skills, information processing rates, fine motor skills, different kinds of perception, the knowledge base, emotional state, and reading and writing skills. In the ideal case, the system detects the characteristics automatically, e.g. based on the user’s interaction with the system. In information retrieval we can, for example, use the user’s queries and selected results for detection. The user’s age can also be utilized as a fuzzy indicator of his or her skills. On the other side, there is a set of options to adapt the UI and the interaction dialog. In information retrieval, we can adapt the UI using different UI elements for querying or visualization of results. In between, an adaptation component contains a set of logic rules to map the user characteristics to the specific UI elements of the evolving user interface.

Fig. 4.1
figure 1

Model of an evolving user interface based on [39]. Users’ mood and characteristics are determined based on the information about the user as well as their interactions with search input and result output (see dashed lines)

Full size image

When designing an EUI, we first have to define the components of a UI that should be adapted. As different systems may have different UI designs, we will further discuss the user interface of a search engine (SUI). An overview of different SUI elements is given in [20]. Here, we consider three main components. The first component is the search input, i.e. UI elements which allow a user to transform his information need into a textual format. This component is traditionally represented by an input field and a search button. Other variants are a menu with different categories or voice input. The second component is the result output of an information retrieval (IR) system. The output consists of UI elements that provide an overview of retrieved search results. This component is traditionally represented by a vertical list of results. The third is the management component. Management covers UI elements that support users in information processing and retaining. Examples of management UI elements are bookmark management components or history mechanisms like breadcrumb trail.Footnote 1 Historically, UI elements for management are not part of a SUI. But recent research [15] shows that young users are highly motivated to use elements such as storage for favorite results. Besides these main components, there also exist general properties of UI elements which might affect all the three categories, e.g. font size or color.

Ideally, an evolving search user interface (ESUI) should be continuously adaptable. Unfortunately, this cannot be done with a high level of granularity for all elements. Some UI elements are continuously adaptable (e.g. font size, button size, space required for UI elements), whereas others are only discretely adaptable (e.g. type of results visualization). Not only are SUI properties, but the complexity of search results is also continuously adaptable and can be used as a personalization mechanism for users of all age groups.

4.3 From Customization to Adaptation: Experimental Design

In order to demonstrate the idea of an ESUI, we developed a search engine for children that is age-adaptable. The search engine was developed in two steps.

4.3.1 Step 1: Customization

During the first stage we designed a search user interface, Knowledge Journey, that can be customized to the user’s needs [15–17]. To achieve a coherent design between different variations of the ESUI, we suggest fixing the positions of different SUI parts. Figure 4.2 depicts the general structure of the developed ESUI. It consists of five groups of elements: a help section, a storage for bookmarked results, visualization of results, elements for keyword search and a menu for navigation. The search input consists of elements for text search and a menu for navigation to provide different possibilities for children to formulate their information need. A menu supports children who have problems with query formulation [15]. We implement parameters having some advantages for children, for example, having a browsing menu, coverflow result visualization, graphical separation of search results, etc. The summary of adaptable elements, their properties and the options that we implemented are given in Table 4.1. In our implementation, we also offer some parameters that, based on the state of the art, are considered to be unsuitable for kids, e.g. having no categories [21]. This makes the changes stand out by switching between different SUIs. Furthermore, the set of options that would be the best for a particular age group is unknown due to the lack of comparative studies. In order to customize the SUI, we implemented a configuration unit that allows users to manipulate the SUI directly. The adaptable elements were ordered according to their decreasing influence, which we have determined, on the entire SUI from theme to audio. As the backend, the Bing Search API Footnote 2 with the safe search option turned on is used.

Table 4.1 Adaptable elements of the implemented ESUI, their parameters and their options [17]
Full size table
Fig. 4.2
figure 2

General structure of the developed SUI with search elements, menu, help, storage for bookmarked pages and result visualization (modified from [17])

Full size image

We conducted a user study in order to derive the mapping function between users with different abilities and the UI elements of a search engine. Adults were chosen as a reference group as their abilities are different [14]. In particular, our hypothesis was that users from different age groups would prefer other UI elements and different general UI properties. We will apply our findings to offer default SUI settings.

The general evaluation procedure was as follows. A pre-interview was conducted to obtain the participants’ demographic data and their Internet experience. Afterwards the general structure of the developed SUI was explained. Then each participant was asked to try the system out and to perform a free search. However, initially child-unfriendly settings were used in the SUI. There was no menu and no theme, impact font of size 10 pt was selected, no picture was provided in the surrogate, etc. We also visualized results as a list with no separation of items. A search result was opened in a new window. We chose these settings in order to increase the participants’ motivation to configure the SUI and also for changes to be more striking. During this stage, participants also got more familiar with the system.

In the next step of the evaluation, all the configuration options were introduced. To provide a better overview for some options, like font type or result visualization, we prepared a printed sheet where all the options for each UI element or property could be seen at once. This made it easier for participants to be aware of all options. Using the configuration unit, each participant went through all adaptable elements, starting from those that had the strongest influence on the whole SUI, like theme, then selecting the result set visualization and customizing a surrogate,Footnote 3 etc. At the end, the participant was able to select whether to turn on the voice and, if so, to customize voice gender and the number of times to repeat the voice explanation.

After a participant customized his own SUI to his preferences, a search task was given. This step was designed for subjects actually to use their own created SUI. Afterwards, they were given the possibility to change SUI settings. Our search task was gender-independent, and it could be solved in a reasonable amount of time using a menu or a keyword search. We asked the participants to find out how many moons the planet Uranus has. In the last step, a post-interview was conducted to gather the user’s opinion about the proposed ESUI. Each test session lasted about 30 min.

In order to conduct a user study with children, we collaborated with a trilingual international primary school in Magdeburg, Germany. Our evaluation was done using 17” displays which were kindly provided by the school director. Adults were recruited from an academic context and tested the SUI in a lab. Forty-four subjects participated in the study, 27 children and 17 adults. The children were between 8 and 10 years old (8.9 on average), 19 girls and 8 boys from third (18 subjects) and fourth (9 subjects) grades. The adults were between 22 and 53 years old (29.2 on average), 5 women and 12 men.

To summarize, we found differences in preferences between adults and young users. Table 4.2 depicts the main finding of the user study. Pupils chose a coverflow result view, whereas adults preferred to use a tiles view. Both children (78%) and adults (100%) chose to add the URL to a document surrogate. Eighty-two percent of the adults and 41% of the children preferred to open the results in a new tab. Those children were already familiar with tab functionality. Thirty-seven percent of the children preferred to open the results in the same window and 22% in a new window. All the pupils and 82% of the adults wished to have a menu in addition to the text input. However, there was no clear choice regarding the menu type made by children. Ninety-three percent of the adult users chose adult topics for the menu. Ninety-two percent of the children chose topics meant for them. However, many adults wished to select the menu topics by themselves. Both children and adults wanted to have as many menu categories as possible. Sixty-seven percent of the children chose to have the audio option on, whereas only 6% of the adults found this option to be useful. There was a clear tendency regarding font and font size. Fifty-six percent of young participants chose Comic Sans MS and 76% of the adults preferred Arial. The majority of pupils selected a 12 pt (37%) or 14 pt (48%) font size. Adults preferred font size of 10 or 12 pt. They said that they could read text in 10 pt. However, they chose 12 pt as there was enough place within the UI for larger texts. Based on the findings of the user study, initial settings for the ESUI can be determined.

Table 4.2 Default children and adults settings for an ESUI found in the user study[17]
Full size table

4.3.2 Step 2: Adaptation

During the second stage we developed the Knowledge Journey Exhibit (KJE) [18] that implements the Knowledge Journey as an information terminal device and has an age-adaptable SUI. The previous version had a configuration window where a user was able to customize the SUI. However, we considered this window to be too difficult for children to operate without supervision and in a public place. Therefore, a decision was made to replace the configuration window with a slider where each point on the slider corresponded to a SUI configuration for a specific age starting with configuration for young children and ending with a setting for young adults (cf. Fig. 4.3). We use the age parameter to adapt the SUI. At the beginning a user is asked to input his or her age. Then, the user is forwarded to the corresponding search user interface, where he can explore other settings of the SUI using the slider. The settings for the slider were derived based on the results from the user study described in Table 4.2. The settings for young children are a pirate theme, coverflow result visualization, and large font size in Comic Sans MS. The settings for young adults are no theme, tiles result visualization, and smaller font size in Arial. The search results for adults contain twice as much text in summaries and smaller thumbnails. Each point on the slider changes one of the setting parameters, e.g. the font.

Fig. 4.3
figure 3

User interface of Knowledge Journey Exhibit. An example query for Pinguine (German for penguins) and the corresponding results are shown. Two of the shown search results marked with a yellow star are bookmarked. The user can adapt the SUI using the slider on the left side

Full size image

The system provides spelling correction after the query is submitted and suggestions for the term the user is currently typing. In addition, users can bookmark the relevant search results using the storage functionality. We used a star symbol that was added to the result surrogate to indicate if the search result is already bookmarked. Users can click directly on the star symbol to bookmark or unbookmark the result or they can place the search result in the storage using drag-and-drop. They can review the stored results, which are grouped by the issued query, in order to provide more context information.

Furthermore, we use information about the web page complexity that is calculated using the Flesch-Reading-Ease (FRE) readability index for German language [1]. We applied a traffic light metaphor and visualized each search result that is easy to understand in a green frame, while a search result that is hard to understand is visualized in a red frame, with varying levels of color in between. The traffic light metaphor is also applied to the slider.

So far, we have developed a search engine prototype with adaptable elements of the search user interface. Another aspect of the interface is the interaction dialog. It is also beneficial to have a means of adaptation for the interaction dialog between the system and the user. An important parameter for the dialog adaptation is the user’s emotional state, because it makes it possible to predict the probability of the user’s success during the interaction with the system. In the case of a low probability, the system can adapt the dialog with the user in order to get the user on the right track. To make this adaptation possible, approaches to continuously monitor the user’s emotional state are required. Therefore, in the following we present our research about mechanisms to detect fluctuations caused by the user’s emotional experiences.

4.4 Tracking Temporal Development of Users’ Emotional Reactions

The mood specifies the actual feeling of the user and is influenced by the user’s emotional experiences. As an important fact for human-computer-interaction (HCI), moods influence the user’s cognitive functions, behavior and judgements, and the individual (creative) problem solving ability (cf. [27, 30]). Thus, knowledge about the user’s mood could support the technical system to decide whether additional assistance is necessary, for instance.

Modeling the user’s emotional development during an interaction could be a first step towards a representation of the user’s inner mental state. This provides the possibility to evolve a user interaction model and gives the opportunity to predict the continuous development of the interaction from the system’s perspective. However, it is not known how a user’s mood can be deduced directly without utilizing labelling methods based on questionnaires, for instance “Self Assessment Manikins” or “Positive and Negative Affect Schedule” (cf. [10, 26]). Hence, the mood has to be modeled either based on observations or using computational models. The modeling described in this section is based of the following publications: [33, 35, 36].

To date, only limited research deals with the problem of mood modeling for human-computer-interaction. In [13], the Ortony, Clore and Collins’s model of emotions is implemented (cf. [31]), outputting several co-existing emotions, where the computed emotions are afterwards mapped into the PAD-space. The mood is derived by using a mood change function in dependence of the computed emotion center of all active emotions and their averaged intensity. The direction of the mood change is defined by a vector pointing from the PAD-space origin to the computed emotion center. The strength of change is defined by the averaged intensity. Additionally, the authors utilize a time-span defining the amount of time the mood change function needs to move a current mood from one mood octant center to another. A mood simulation presented in [5] also relies on precomputed emotional objects located in the PAD-space. In contrast to [13], this model used the valence dimension to change the mood value. Thus, this model does not locate the mood within the PAD-space. Furthermore, in this model a computed emotional valence value results in a pulled mood adjusted by a factor indicating the “temperament” of an agent. A spring is then used to simulate the reset force to decrease steadily until neutrality is reached. Both mood models are used to equip virtual humans with realistic moods to produce a more human-like behavior.

4.4.1 Mood Model Implementation

Starting from the observation of the mood as a quite inert object within the pleasure-arousal-dominance (PAD)-space, a mood modeling algorithm is presented enabling technical systems to model mood by using emotional observations as input values. Therefore, the following behavior will be modeled:

  • mood transitions in the PAD-space are caused by emotions

  • single emotional observations do not directly change the mood’s position

  • repeated similar emotional observations facilitate a mood transition in the direction of the emotional observation

  • repeated similar emotional observations hinder a mood transition in the opposite direction

  • incorporation of the personality trait extraversion to adjust the emotional observation

For the approach presented in this chapter, the mood modeling is independent of the source of emotional assessment. It can be derived either from a computational model based on appraisals, as it is described here, or implicitly from observed emotional reactions of the user.

Both the observation of single short-term affects (the emotions) as well as the recognized mood will be located within the PAD-space in the range of [−1, 1] for each dimension. This abstract definition of the modeled location by using the PAD-space allows the model to be independent of the chosen observed modality and to have the same representation for the emotional input values.

To illustrate the impact of recognized emotions on the mood, the observed emotion e t at time t is modeled as the force F t with the global weighting factor κ 0 (Eq. (4.1)). Furthermore, the emotions e t are modeled for each dimension in the PAD-space separately. Thus, the calculation of the mood is conducted component-wise [35, 36]. The force F t is used to update the mood M for that dimension by calculating a mood shift ΔL M (Eq. (4.2)) utilizing the damping D t , which is updated by using the current emotion force F t and the previous damping D t−1. This modeling technique is loosely based on a mechanical spring model: The emotional observation performs a force on the mood. This force is attenuated by a damping term, which is modified after each pulling.

$$\displaystyle\begin{array}{rcl} F_{t}& =& \kappa _{0} \cdot e_{t}{}\end{array}$$
(4.1)
$$\displaystyle\begin{array}{rcl} \varDelta L_{M}& =& \frac{F_{t}} {D_{t}}{}\end{array}$$
(4.2)
$$\displaystyle\begin{array}{rcl} M_{t}& =& M_{t-1} +\varDelta L_{M}{}\end{array}$$
(4.3)
$$\displaystyle\begin{array}{rcl} D_{t}& =& f(F_{t},D_{t-1},\mu _{1},\mu _{2}){}\end{array}$$
(4.4)

The main aspect of this model is the modifiable damping D t . It is calculated according to Eqs. (4.5) and (4.6). The damping is changed in each step by calculating ΔD t , which is influenced by the observed emotion force F t . The underlying function has the behavior of a tanh-function. It has two parameters. The parameter μ 1 changes the oscillation behavior of the function and the parameter μ 2 adjusts the range of values towards the maximum damping.

$$\displaystyle\begin{array}{rcl} D_{t}& =& D_{t-1} -\varDelta D_{t}{}\end{array}$$
(4.5)
$$\displaystyle\begin{array}{rcl} \varDelta D_{t}& =& \mu _{2} \cdot \tanh (F_{t} \cdot \mu _{1}){}\end{array}$$
(4.6)

The block scheme for the mood modeling is illustrated in Fig. 4.4.

Fig. 4.4
figure 4

Block scheme of the presented mood model. The red rounded box (e t pad) is an observed emotion, the blue rounded box (M t pad) represents the modeled mood, all other grey rounded boxes are inner model values, and white boxes are calculations. For simplification the combined components are used; internally the calculation is done for each dimension separately

Full size image

The mood model consists of two calculation paths. The diagonal one calculates the actual mood M t pad. The vertical one updates the inner model parameter D t pad. As prerequisite, the emotional force F t pad is compiled from the observed emotion e t pad.

The mood is the result of the influence of the user’s emotions over time with respect to the previous mood. The impact of the user’s emotions on his mood depends also on the user’s personality (cf. [33]). The observed (external) affect needs not be felt (internally) with the same strength. For this, the observed emotion have to be translated into an adequate emotional force with respect to the known differences in the external and internal representations. It is known from the literature (cf. [8, 24, 40]) that an external and an internal assessment of the emotional cause lead to different interpretations. Hence, this must be considered in the development of the mood model. For this, the presented model focusses on the emotional intensity as an adjustment factor to determine the difference between an external observation and an internal feeling of the user’s emotion. For this case, the personality trait extraversion is used to adjust the emotional force (F t pad).

4.4.2 Generate Continuous Emotional Assessments

The presented modeling technique needs sequences of emotion values to allow a mood prediction. Since this type of data is hardly obtainable, we utilize a method described in detail in [23] to obtain emotional sequences. This method is based on the appraisal theory (cf. [32]) and implements a computational model enriched with fuzzy sets to obtain a continuous emotional evaluation.

In appraisal theory (cf. [32]), it is supposed that the subjective significance of an event is evaluated against a number of variables. The important aspect of the appraisal theory is that it takes into account individual variances of emotional reactions to the same event. This results in the following sequence: event, evaluation, emotional body reaction. The body reactions then result in specific emotions.

Appraisal theories attempt to specify the nature of criteria used in evaluation in terms of different appraisal variables or dimensions. Examples of these dimensions are “goal significance”, representing the goals and needs that are high in priority at the moment (e.g. the goals of survival, maintaining social relationships or winning a game), or “urgency”, representing the need for an action. In [12] it is suggested that the given appraisal variables allow us to deduce an emotion as the most probable emotional reaction to a certain event. For example, joy or happiness will occur if the appraisal values for the variable “goal significance” are high, while the value for “urgency” is low, etc. To generate a continuous emotional assessment the evaluation process maps from appraisal theory to a fuzzy model in order to derive the emotions of a user in a specific situation or event.

In our implementation ten different appraisal variables are used (cf. Table 29.2 in [12]). To operationalize the appraisal variables and their linguistic values, each appraisal variable is modeled as a fuzzy set (cf. [23]). The fuzzy sets consist of several fuzzy variables, depending on the number of postulated values of the appraisal variable. For example, the appraisal variable “urgency” has five different values (“very low”, “low”, “medium”, “high”, “very high”) and thus is modeled as a fuzzy set with five corresponding fuzzy variables. Each appraisal variable is mapped to a uniform number in the range from 0 to 100; cf. Fig. 4.5. The corresponding fuzzy variables are distributed uniformly in this range. For simplicity we use a triangle function to model a fuzzy variable. Using all ten appraisal variables, we are able to model both the emotional state of the user and a possible evaluation of an arising event in terms of appraisal variables.

Fig. 4.5
figure 5

Fuzzification of the appraisal variable “urgency” using a triangle function

Full size image

In Fig. 4.6 a schematic overview of the framework is given.

Fig. 4.6
figure 6

Conceptual block scheme of our fuzzified appraisal-based framework

Full size image

According to appraisal theory, the emergence of an event triggers an evaluation process influencing the user’s appraisal variables. To model this process, the values of the fuzzyfied appraisal variables of an event are estimated and used to adapt the corresponding variables in the user state. For example, an event with high “goal significance” should positively affect the user’s condition for “goal significance”. In which manner the events affect the user’s state is not trivial because humans interpret events and their relevance differently. In the simplest way, the appraisal variables of an event affect the user’s emotional state directly by overwriting any old value with the actual one (cf. left-hand side of Fig. 4.6). As stated above, the appraisal variables can be used to deduce an emotion as the most probable emotional reaction to a certain event. Each emotion is described by appraisal variables with specific values. From this specification a rule base is generated for each emotion (cf. right-hand side of Fig. 4.6). To derive an emotion, the postulated emotion profile and the user state are used to calculate the fulfillment of the emotion rule base. This framework can then be used to gain a continuous emotional assessment. Therefore, we rely on EmoRec-Woz I, a subset of the EmoRec corpus (cf. [41]). It was generated within the SFB/TRR 62 during a Wizard-of-Oz experiment containing audio, video, and bio-physiological data. The users had to play games of concentration (Memory) and each experiment was divided into two rounds with several experimental sequences (ESS); cf. Table 4.3.

Table 4.3 Sequence of ES and expected PAD positions
Full size table

The experiment was designed in such a way that different emotional states were induced through positive and negative feedback, wizard responses, and game difficulty levels. The ESS with their expected PAD octants are shown in Table 4.3. The octants are indicated by their extreme points. A session has an average length of about 18 min. In the following, the output of the model is exemplary, shown using the experiment of one participant. For more details concerning the data set (exemplarily cf. [41]). To apply the model, all possible events (like a hit or negative feedback) were extracted and all possible evaluations regarding the corresponding appraisal variables were designed using a rule-based approach. For example, turning over a pair of matching cards is considered as “goal significant” while receiving negative feedback is considered as unpleasant. For simplicity it is assumed that an event affects the emotional state of the user directly. Since the appraisal process has a short duration, a decay process can be used, e.g. via stepwise convergence to a “neutral” state (cf. “user state” in Fig. 4.6). At this point it is assumed that a “neutral” user state will be reached if all appraisal variables receive the value “medium”.

In Fig. 4.7 the results for the ES1 up to ES6 of one test person are illustrated. In this experiment we only use the rule base to derive pleasure and arousal. The values in the diagram represent the fulfillment of the two emotion rules, calculated by the model, based on the events that occurred in the sessions. In comparison to ES5 the values in ES2 for pleasure are higher and arousal occurs only slightly. In ES5 the values for pleasure are reduced and arousal is increased, which corresponds to the expected pleasure development; cf. Table 4.3.

Fig. 4.7
figure 7

Calculated arousal and valence values of our appraisal based computational model accumulating the output of all fuzzy sets

Full size image

4.4.3 Experimental Model Evaluation

For the experimental model evaluation, we also rely on EmoRec-Woz I. To model the mood development, we limited the investigation to pleasure, as this dimension is already an object of investigation (cf. [6, 7, 41]), which simplifies the comparison of our mood modeling technique with other results. After performing some pre-tests to ensure that the mood remains within the limits of [−1, 1] of the PAD-space, we defined the initial model parameters as given in Table 4.4.

Table 4.4 Initial values for mood model
Full size table

To gather the emotion data, we rely on the computation of the emotional course based on the appraisal modeling described in Sect. 4.4.2. The same session of one experiment is used for illustrations. This emotional computation serves as input value for the mood modeling. By doing so, it makes it possible to form a mood development over a whole experiment and compare the calculated model with the experimental descriptions, as ground truth, of the complete experiment (cf. Table 4.3). The whole mood development and the division into the single ES are shown in Fig. 4.8. We concentrated on the pleasure-dimension, as, for this, secured studies on the EmoRec corpus are available (cf. [6, 41]) showing that the experiment induces an “emotional feeling” that is measurable. Investigations with emotion recognizers using prosodic, facial, and bio-physiological features and the comparison to the experimental design could support the fact that the participants experienced ES2 as mostly positive and ES5 as mostly negative (cf. [41]). The underlying experimental design—the ground truth for the presented mood model—is described as follows: in ES1, ES2, ES3, and ES6 mostly positive emotions; in ES4 the emotional inducement goes back to a neutral degree; In ES5 mostly negative emotions were induced.

Fig. 4.8
figure 8

Course of the mood model using the whole experimental session of one participant. The subject’s pleasure value is additionally depicted

Full size image

Using the computed emotional labels as input data for the modeling, we were able to demonstrate that the mood follows the prediction for the pleasure dimension of the given ES in the experiment (cf. Fig. 4.8). The advantage of this modeling is that the entire sequence can be represented in one course. Furthermore, the influence of a preceding ES on the actual ES is included in the mood modeling.

The resulting mood development is as follows: in the beginning of ES1 the mood rests in its neutral position and it takes some time, until the mood starts to shift towards the positive region. In ES2 and ES3 the mood continues to rise. During ES4 the mood reaches its highest value of 0. 204 at 11:33 min. As in this ES the inducement of negative emotions has started, the mood is decreasing afterwards. The previous induced positive emotions lead to a quite high damping in the direction of a negative mood; thus the mood falls at the end of ES4. In ES5, when more negative emotions are induced due to negative feedback and time pressure, the mood is decreasing quite fast. Shortly after the end of ES5 the mood reaches its lowest value with − 0. 100 at 15:21 min. Here, it should be noted that negative emotional reactions are observed already in the very beginning of ES5; otherwise the strong decreasing of the mood could not have been observed. The mood remains quite low in the beginning of ES6. During the course of ES6, where many positive emotions were induced, the mood rises again and reaches 0. 04 at the end of the experiment (17:07 min).

When including personality traits into the mood model, the following must be given: (1) The personalities of the participants and (2) their subjective feelings. The first prerequisite is fulfilled by EmoRec-Woz I, as the Big Five personality traits for each participant were captured with the NEO-FFI questionnaire (cf. [9]). The personality trait extraversion is particularly useful to divide subjects into the groups of users “showing” emotions and users “hiding” emotions (cf. [25]). Additionally, users with high extraversion are more stable on positive affects. These considerations lead to a sign-dependent factor to distinguish between positive and negative values for emotional dimensions. Thus, we expand the adjustment factor κ η (cf. Fig. 4.9).

Fig. 4.9
figure 9

Mood development for different settings of κ η , but not differing between κ pos and κ neg

Full size image

For this case, κ η can be modified by choosing different values for κ. We will depict results for values in the range of 0. 05–0. 4. These values reproduce the strength of how an observed emotion is experienced by the observed person itself. An example of the different values for κ η is given in Fig. 4.9. For this experiment, emotional traces based on computed emotions for ES2 are used. It can be seen that values higher than 0. 3 led to the mood rising too fast. This causes implausible moods, since the upper boundary of 1 for the PAD-space is violated. Hence, a κ η > 0. 3 should be avoided.

In contrast, for very small values (κ η < 0. 1) the mood becomes insensitive to emotional changes. Therefore, we suggest using values in the range from 0.1 to 0.3, as they seem to provide comprehensible mood courses. In Fig. 4.10 it is depicted how the difference between κ pos and κ neg influences the mood development.

Fig. 4.10
figure 10

Mood development for different settings of κ pos and κ neg

Full size image

According to the phenomenon described earlier, namely that persons with a high extraversion are more stable on positive affects, we tested different settings for the difference between κ pos and κ neg (cf. Fig. 4.10). Here, we basically distinguish between two different settings. First, a very reinforcing setting, where positive observations are emphasized and negative observations are suppressed. Secondly, a very suppressive setting where the mood development behaves the other way around, positive values are suppressed and negative ones emphasized. Again, the annotated emotional traces of ES2 are used and the previous considerations on the κ η -values are included to choose only values between 0.1 and 0.3. More details on annotation can be found in [34, 37]. By varying these values, we could change the behavior of the model to match the different settings of emotional stability. Although the input data remains the same, the emotional influence of positive observations on the mood can be either very suppressive or very reinforcing, depending only on the adjustment factor κ η , as seen in Fig. 4.10.

The subject’s extraversion can be obtained from the NEO-FFI questionnaire. The values for extraversion gathered from the questionnaires are normalized in the range of [0, 1]. Thus, a high extraversion is denoted by values above 0. 5 and a low extraversion by values below 0. 5. To obtain a mood model that reproduces the expected behavior, the values for the parameter-pair κ pos and κ neg have to be chosen adequately. In Table 4.5, we present suggestions for plausible adjustment values based on the extraversion gathered from questionnaires.

Table 4.5 Suggested κ pos and κ neg values based on the extraversion personality trait
Full size table

4.5 Summary

In this chapter, we described our vision of an evolving search user interface. An evolving search user interface should adapt itself (both, user interface and interaction dialog) to the specific characteristics of an individual user. Here, we concentrated on the front end of the search engine and developed two prototypes, one which can be customized to the user’s needs and one that takes the user’s age as a parameter for adaptation. However, in order to have an adaptive system, backend algorithms to detect a user’s abilities are required. Moreover, in this work, age was used in order to approximate the user’s skill space. However, the age is only a fuzzy indicator of what the user’s skills are. The design of a more fine-grained mapping function between the user’s skill space and the options to adapt the UI elements of the SUI is a great challenge for future work. To develop an ESUI, user studies with users of gradually different skills should be conducted.

In order to adapt the interaction dialog, we propose to monitor the user’s emotional state. This makes an early detection of users’ problems with the system possible and allows us to adapt the dialog to get the user on the right path. Therefore, we investigated methods to detect changes in the user’s emotional state.

We furthermore proposed a mood modeling from a technical perspective that is able to incorporate several psychological observations. After describing the desired mood development, we presented a technical mood implementation. This implementation has the advantage of only three internal parameters (D 0, μ 1 and μ 2) and one user-specific parameter-pair, κ pos and κ neg. The mood development is based on a mechanical spring model.

Using the EmoRec dataset, we were able to evaluate the principal function of the proposed model on two different databases. Especially on the EmoRec Woz corpus, we were able to show that the generated mood course matched the experimental setting. By utilizing the user-specific parameter-pair κ pos and κ neg the personality trait extraversion was integrated. This trait is supposed to regulate the individual emotional experiences.

The presented results are based on locating both emotions and calculated mood in the PAD-space according to their value. A problem that has to be addressed in future work is the need for smooth transitions of emotion values over time. To date, the emotional assessments are processed without regarding the gap between them, but this cannot always be guaranteed, especially when using automatically recognized emotion values. Here a further extension of the proposed model is needed by, for example, a temporal averaging or weighting technique.

In the future, we are going to use the mood model for the search scenario and incorporate it in the search engine prototype we developed. User emotions can be detected by analyzing the facial expression of a user. It is also possible to implement a voice-controlled search interface which allows a voice interaction with the search engine and to use the user’s voice for emotion detection.