Keywords

1 Introduction

Emotions are one of the central aspects in the life of human beings because they influence daily choices, social relationships and the general approach to life itself. A lot of studies have shown that a high level of emotional skills, i.e. the ability to manage and control own and others’ emotions (Emotional Intelligence), contributes to improve personal well-being, to enhance job performance and life satisfaction [1, 2].

An important factor that influences the emotional state of a person is the surrounding environment. The impact of the environment on the emotional state of humans has been widely studied by using the environmental psychology approach [3]. A great deal of experimental research demonstrate that there is an effect of the emotional context on memory, moral judgment and behavioral reaction [4, 5]. According to the results of such studies the design of the daily life contexts as work-office, cities, supermarket have radically changed in order to make them more pleasant. The major limitation in the environmental psychology approach is the study of the effect of the environment in an ecological way.

Virtual Reality (VR) can be a helpful tool as it recreates realistic scenarios in which social dynamics develop. VR, unlike conventional methods like exposure to real situations or to photographs, simulates the user’s presence in a digital environment and permits to control the events that happen within. Because of the ecological validity and control-based environments, VR has become a useful methodology for the experimental psychology research, especially to treat different kinds of mental health disease [6, 7].

Moreover, it has been shown that participants in VR-experiences perceive and feel strong emotions both during the human-avatar and the human-environment interaction. Several studies have been conducted to investigate the emotional stimuli elicited by VR by using both 3D avatars [8, 9] and affective environments [10, 11]. This research showed that participants in VR experiences perceived specific emotions corresponding to the valence of VR or Avatar Mood. Banos et al. [12] have also investigated the relationship between the perception of emotion and the sense of presence in VR experience. They found that the affective content of Virtual Environments (VEs) has impact on the sense of presence especially for the target of engagement which results higher in emotional environment respect to the neutral one. Due to these features, VEs with emotional content can be useful tools to investigate mental disorder and body image distortion [13].

All the research investigating the emotional aspect of VEs do not pay attention to the social dynamics developed in the virtual context with the risk of playing down the ecological validity of VR. According to the Narrative Theories approach [14] dynamic and storytelling VR increases the sense of being physically located within the VE and the sense of interaction with the environment. For this reason, this kind of VR applications are widely used in education [15], VR games for training [16, 17], to acquire ethical skills [18] and for dissemination of culture [19, 20].

In our study we have combined the emotional content of virtual scenarios with specific social narratives in order to replicate a daily life situations able to elicit an emotional affective state in an ecological way. The aim of this research is to introduce a novel paradigm to study environmental emotional perception by using the Emotional Narrative Virtual Environment (ENVE) approach. For this purpose we have created three different VEs endowed with disgust, fear and happy emotional contents and animated with specific social narratives. A preliminary research, to evaluate the ability of the ENVE to elicit specific emotions, was carried out on 36 subjects. In this research we have used VR-immersive technology (Oculus-Rift) to recreate a strong sense of immersion in the virtual space and a high sense of presence in the emotional scene [12]. By measuring the similarity between the users response to real and virtual situation, it has been showed that VR-immersive technology can enhance emotional and psychological responses arousing the feeling to be present in a real situation rather than virtual [21]. Moreover, immersive technologies increase the sense of being physical located in virtual space with own body [22].

Based on the results of our study, ENVE will be improved and used to further investigate the effect of the emotional content on the user experience, especially related to the avatar interaction in the virtual emotional situation.

In this paper the social scenario and social narratives of each ENVE will be presented with a brief description of the technology used to create them. Then the experimental design is described focusing on the participants, procedure and material used in the experiment. The results are presented and discussed in the last part of the manuscript. A conclusion will provide the guidelines for further investigation of emotion perception in VR.

2 Emotional Narrative Virtual Environment

The purpose of this research is the creation of Emotional Narrative Virtual Environments endowed with a specific emotional content. The main characteristics of this kind of environments are the following:

  • emotional: the environment has specific emotional content and the virtual scenario must elicit in the participant the corresponding emotion

  • narrative: the scene is animated with a social narratives similar to what could happen in a real situation, enhancing the emotional content of the scenario.

In this study we have created three ENVEs, happy, disgust and fear, each one characterized by an emotional scene. The virtual environments and scenarios are described in the following paragraphs.

Fig. 1.
figure 1

In the left panel the three virtual scenarios visualized by participants in the experiment. From top to bottom: the Happy Environment (virtual playground), the Disgust Environment (dark alley) and the Fear Environment (mountain scenario). In the right panel the three virtual scenarios with the animations. From top to bottom: the children playing volleyball in the playground, the man which loose balance walking on the Tibetan Bridge and the mice turn all around in the alley.

Full size image

2.1 Scenario

In order to evoke in the participants specific emotions we have created three different virtual scenarios starting from the photographs of the International Affective Picture System (IAPS) [23]. Figure 1 show the three different kinds of virtual scenario with emotional valence.

The three virtual environment are designed as followed:

  1. 1.

    Happy Environment (HE) is a green area with two swings, a slide, a big playground and an hot-air balloon that flies in a blue sky. Is placed in a big open garden with some trees and a field of grass that waves softly to the wind. In order to increase the joyfulness of the scenario a background sound of birds chirping and slight rustling was added.

  2. 2.

    Disgust Environment (DE) is a dark alley surrounded by buildings. All over the muddy ground lie bins completely full of garbage and a lot of trash outside. In the sky, a lot of moving clouds colour the entire scenario gray. To increase the realism of DE a murmur background and cars noise were added.

  3. 3.

    Fear Environment (FE) is a mountain scenario in which a Tibetan Bridge is located. The bridge connect two mountains that are divided by a lake and a man is walking on it. The sky is cloudy and there is a lot of mist. In the background sounds of thunders and strong wind enhance the frightening appearance of the scenario.

2.2 Social Narratives

In order to mimic daily-life situations in a realistic way each VE was animated with a specific dynamic scene that should enhance the emotional content of the scenario. The aim was to introduce a social narratives coherent with the valence of the context. In the virtual environment an event transforms the scenario into a social narrative by developing a dynamic scene. The social narratives are three, one for each environment and they develop in the following way:

  1. 1.

    HE: two children start playing with a ball in the playground. The sound of children voices drives the scene ;

  2. 2.

    DE: a lot of small mice start to move all around the virtual space and two big rats move around the garbage. The squeaking sound of several mice was used to accompany the development of the scene;

  3. 3.

    FE: an earthquake produces a strong oscillation of all the environment and of the bridge. The man walking on the bridge loses balance. A strong sound of explosion accompanies the earthquake.

3 Experimental Design

The goal of this study was to investigate the ability of ENVE to elicit specific emotion in the user experience. The ENVE approach is based on the creation of emotional environment and emotional scene in which participants are completely immersed in a real life situation. An experiment to evaluate the effectiveness of ENVE to affect emotional state of users was carried out.

In this section the experimental set up, participants and procedure are detailed. The measures used in the evaluation of ENVE are also followed explained.

3.1 Apparatus for ENVE and Technological Set up

The virtual environments were modeled by using the game engine Unreal Engine 4. Small animations for the static part of the exploration, as the wind through the grass and trees and the hot-air balloon movement according to the clouds, were added through UE4s visual scripting-language Blueprint. In order to create the social narratives the same UE4s visual scripting-language Blueprint was used. The avatars were taken from Rocketbox library (www.rocketbox-libraries.com) animated through motion capture data from The Carnegie Mellon University Motion Capture Database (http://mocap.cs.cmu.edu/) retargeted, mounted and looped in order to fit our purposes. The same language was also used to spawn an editable number of mice and to move them along a number of arbitrary points. The sound in the environments was used to increases the isolation from the outside world and feel the noises as they were real (located in the ambient in correspondence of the sources). In order to provide a strong sense of immersion in the environments a Head Mounted Display was used in the experiment. The application that runs the ENVE and the animations was realized through the game engine Unreal Engine 4 on a PC with Oculus Rift DK2. UE4 facilitates the design of complex custom environments using low poly models and complex materials through a visual editing interface and a visual scripting-language. Oculus Rift integration is native to UE4 and provides binocular vision into the virtual scenarios via HDMI input. It provides also rotational head tracking with 6dof and a front-face tracking that allows the user to look around the virtual environment and to move in it (in a short range). The dark foam surrounding the goggles blocks out most of the external light and enhance the isolation from the outside world and the related immersion in the VE.

3.2 Participants

A sample of 38 subjects were recruited to participate in the experiment. In the preliminary phase they read and signed a consent form in which the experiment was briefly explained. Prior to start the experiment a questionnaire to rate the level of depression of each subject, Beck Depression Inventory questionnaire (BDI), was used as exclusion criterion. Based on the results of BDI 2 subjects were excluded in the study because they rated a score equal or higher than 14. A total of 36 participants (28 male and 8 female) performed the experiment. They were healthy subjects with no history of mental disease, aged \(30.22\pm 6.66\) y.o. with a level of education equal or higher then high school diploma. They self-rated in a 5-point Likert scale their level in the use of personal computer (average \(40.80 \pm 0.58\)), videogames (average \(3.33\pm 1.41\)) and virtual reality (average \({3.19\pm 1.45}\)).

3.3 Measures

For each subjects tree different types of questionnaire were used to assess their emotional state and the sense of presence in the ENVE: i- The pre-experimental questionnaire to assess generic and demographic information; ii- the self-report measure to rate if participants have a level of depression prior to start the experiment (exclusion criteria) and the sense of presence at the end of the experience; iii- the self-rate questionnaire to assess the emotional state and sense of presence elicited by the ENVE.

i- A generic questionnaire was used to collect demographic information of participant and exclude from the study subjects with mental disease and history of depression. Moreover, a self-rated questionnaire about the use of videogames, virtual reality and personal computer was given.

ii- BDI questionnaire was used as a selection criterion in pre-experimental phase. It is a self-report questionnaire to assess the cognitive aspect of depression and measure the depressive symptom. It consist in 21-questions multiple-choice relating to symptoms of depression. Participants had to chose the statement that better described his/her mood state.

Selected questions of Reality Judgment and Presence Questionnaire (RJPQ) were used as self-measure to assess the sense of presence and reality judgment of participants [24]. A 10-point Likert scale was used to answer all items. 17 of these questions were used to assess Reality Judgment (Q2, Q6, Q7, Q12, Q13, Q16), sense of Presence (Q9, Q10, Q11, Q15, Q17), Emotional Involvement (Q8, Q4), Control, an important characteristic of immersion, (Q14), Realism, both perceptual and interactive, including perceptual clarity and natural modes of interaction (Q1), Expectations and the possibility of Anticipation and prediction in the virtual environment (Q3, Q5). Two additional Questions were added in order to assess the emotional involvement of the environment and the emotional involvement of the social narrative: (-To what extent did the virtual scenario was emotional?; -To what extent did the virtual scene viewed in the virtual environment was emotional?).

iii- The emotional effect of the ENVE experience was evaluated by rating on a 5-point Likert scale to what extent the experience induces in a subject a sense of fear-happiness-disgust. Also the sense of Presence elicited by each environment and social narrative was evaluated by rating on a 5-point Likert scale the feeling experienced in the virtual scene (Did you feel like you are in the alley-in the virtual play area-in mountain scenario?) and the feeling to be involved in the social narrative (Do you feel like the mice were actually rotating around the garbage-the man on the bridge were really losing balance-the children were really play in the play area?)

3.4 Procedure

36 subjects participated to the study. None of them had a history of depression and completed a Back Depression Inventory (BDI) as exclusion criteria. 2 subjects were excluded in preselected phase as they scored less than 14 in BDI Questionnaire. The aim of the experiment is to evaluate the emotional content of ENVEs, and if whether ENVEs induce in the participant the emotional state corresponding to the emotional content of the environment. The experiment will be divided in three phases.

In the Baseline phase participants were instructed about the entire procedure of the experiment. Prior to start the ENVEs visualization they had to complete the BDI questionnaire. Then the experimenter explained them to dress the helmet, to wear stereo headphones and how to explore the environments.

In the Experimental phase the participants were completely immersed in the Virtual Scenario for a period of 50 s. During this period they could explore the environment, looking all around and “let themselves go” to what they see in the virtual scenario without performing any specific task. After this time the environment “comes alive” with a dynamic scene having a duration of 20 s. The three environments are presented to each participant for one time following 6 different sequences: HE-DE-FE; HE-FE-DE; DE-HE-FE; DE-FE-HA; FE-DE-HE; FE-HE-DE. In order to reset the emotional state elicited by an ENVE a rest time of 120 s between each scenario was given to participants during which they answered to a questionnaire composed by 5 questions related to the emotional state and the sense of presence perceived within the ENVE. At the end of the three experiences the Self-Experience evaluation phase started. First the participants had to answer to a final Questionnaire about the sense of presence and emotional valence of the environment (RJPQ). Secondly a brief interview were conducted by the experimenters concerning the overall impressions and comments about the experience.

Table 1. Results of Repeated Measure Anova.
Full size table
Table 2. Results of Post-Hoc Test.
Full size table

4 Data Analysis and Results

Data was collected by considering two experimental factors: the emotional content of ENVE (\(E_{content}\)) and the emotion elicited in the participants (\(E_{elicited}\), Happiness, Disgust, Fear). A repeated measure ANOVA with ENVE as a within-group factors was conducted for each \(E_{elicited}\). It is worth underlying that Repeated Measures ANOVA has been demonstrated to be an appropriate statistical test even for ranked variables as those based on Likert Scales are [25]. The aim was to investigate if the \(E_{content}\) elicited specific emotion in participants. A Greenhouse-Geisser Correction was used for the \(E_{elicited}\) fear and disgust ANOVA, as sphericity could not be assumed (as resulting from Mauchly’s test, [26]). The congruence effect was also investigated between each \(E_{content}\) of ENVE and the corresponding \(E_{elicited}\) in participants. A repeated measure ANOVA was also conducted by considering Sense of Presence in Virtual Environment (\(SoPinVE_{elicited}\)) and Sense of Presence in Social Narratives (\(SoPinSN_{elicited}\)) as dependent variable. Results of ANOVA are reported in Table 1. A significant effect of \(E_{elicited}\) for the \(E_{content}\) of each ENVE was found in all of emotions elicited (p \(< 0.001\)). We also found a significant effect of congruence effect between \(E_{content}\) and the corresponding \(E_{elicited}\) (p \(< 0.001\)). For those ANOVA yielding a significant ENVE effect, differences between each couple of ENVE were estimated using Post-hoc tests with Sidak correction for multiple comparisons [27]. For each Post-hoc, the confidence interval on mean difference was estimated performing 1000 bootstraps. Results are presented in Table 2 and observed means with indicator of significantly different conditions are reported in Fig. 2. Significant differences in the comparisons of HE-DE and HE-FE were found for the Happiness effect \(E_{elicited}\), for the HE-DE and DE-FE when considering the Disgust effect \(E_{elicited}\), (\(p<0.001\) for both). Regarding Fear \(E_{elicited}\) a significant effect was found in the HE-DE (\(p<0.001\)) and HE-FE (\(p<0.001\)) comparisons but not for DE-FE. As shown in Fig. 2c, very similar score were obtained for Fear in FE (\(2,36\pm 0.9\)) and in DE (\(2,19\pm 1,06\)). A significant difference (\(p< 0.001\)) was found between the unexpected high mean score obtained by Fear in DE (\(2,19\pm 1,06\)) and that obtained in HE (\(1,14\pm 0,42\)). The Post-hoc tests on congruence effect between specific \(E_{elicited}\) in corresponding \(E_{content}\) yielded a significant difference (\(p<0.001\)) only for the couple HE-FE with a significantly higher happiness score in HE (\(3,15\pm 0,9\)) when compared with fear in FE (\(2,36\pm 1,07\)). A high mean score of SoP in both SN and VE was observed for all the ENVEs; no statistical difference was found either for the \(E_{content}\) of ENVE, p \(<0.114\), or for \(SoPinVE_{elicited}\), p \(<0.373\).

Fig. 2.
figure 2

Mean Score obtained for Happiness (a), Disgust (b), Fear (c), in each Emotional Narrative Virtual Environment. The bars represent the standard deviation and the stars show the significant difference found in pairwise Post-hoc test.

Full size image

5 Discussion

In the present research a novel methodology was presented to investigate the effect of the environment in the emotional perception by using an ecological approach based on Emotional Narrative Virtual Environments. The aim of the study was to create and evaluate three different scenarios animated with emotional scenes able to elicit in VR-users specific emotions. For this purpose an experiment was carried out on 36 participants who have made experience of happy, fear and disgust ENVEs and they rate their level of happiness, fear and disgust for each environment. The mean score obtained for each answer (\(E_{elicited}\) in participants) was considered as dependent variable to investigate if \(E_{content}\) of ENVE elicited specific emotion in participants. Results showed no statistical difference for the mean score of \(E_{elicited}\) in the ENVE. A Post-hoc test was carried out to investigate if \(E_{elicited}\) shows a significant different in the pairwise comparison between \(E_{content}\). We found a relevant difference between HE-DE and HE-FE for happiness. This results corroborate our hypothesis that HE elicit “more happiness” than DE and FE. As we supposed, similar results was found also for the \(E_{elicited}\) Disgust, because it obtained a relevant major mean score in DE respect to HE and FE. A comparable result was obtained by Fear in the comparison between FE and HE, but not for FE-DE. The mean score of Fear in FE \(2,36\pm 1,07\) was similar to mean score obtained by Fear in DE \(2,19\pm 1,06\). This result supports the idea that FE do not elicit a fear emotion more than DE. This is not completely in line with previous research that supports the idea that specific VEs induce more corresponding target emotion than other kind of emotion [11]. However, Banos et al. [11] do not consider the correlation between Fear and Disgust Environments in the ability to elicit the corresponding emotions. We suppose that disgust is an emotion that in some cases is strictly related with fear. In fact several participants in the brief interview after experiment stated that they have fear of mice. Moreover, some of them told us that they perceive a feeling of fear to be in a enclosed environment as the alley. In future investigations we will take into account the difference between enclosed and open environments in order to investigate their influence on the emotional perception. The analysis of congruence in which the mean of each \(E_{elicited}\) was compared with corresponding \(E_{content}\) of ENVE found a significant different for both HE-FE (p-value\(< 0.001\)) and for DE-FE (p-value\(< 0.050\)). This means that the HE was able to elicit “more happiness” than the FE was to elicit fear. At the same time DE elicit disgust more than FE elicit fear. In conclusion the congruence effect for FE (\(2,36\pm 1,07\) mean score) was significantly lower than the DE (\(2,78\pm 1,2\) mean score) and HE (\(3,25\pm 0,9\) mean score). We argue that there are two main reasons to explain this result: the design of the environment and the third person perspective. FE is a mountain open scenario with a man walking on a Tibetan Bridge. This kind of scenario is usually associated with an idea of “game experience” instead of serious activity. Secondly, participants observed the scene and they didn’t feel a sense of fear toward the man on the bridge because they couldn’t interact with him. In the after experience interview a lot of participants said that they wanted to go on the bridge to save the man, but they couldn’t do it and for this reason the experience was like a movie. Conversely, in DE the participants were completely immersed in the scene in first person perspective and the mice turned all around them. However the third person perspective in HE did not influence the emotional feeling elicited because the environment itself was joyful. Almost all participants said that it was pleasant to stay in the park and hear the chirping of birds.

The sense of presence felt during the experiences of Virtual Scene and Virtual Scenario did not statistically differ for the \(E_{content}\) of environment (see Table 1). The mean score for \(SoPinVE_{elicited}\) and \(SoPinSN_{elicited}\) was \(> 3\) in all ENVEs. This is in line with the trend of median score obtained in the RJPQ Questionnaire (see Fig. 3). In particular the high score obtained by Q4 and Q9 (To what extent did things in the virtual world have impact on you? and To what extent did you feel like you went into the virtual world, and you almost forgot about the world outside?) corroborate the suggestion that participants felt a sense of presence in ENVE. Moreover additional question Q18 (To what extent did the virtual scenario was emotional?) and Q19 (To what extent did the virtual scene viewed in the virtual environment was emotional?) are in line with the mean score results of \(E_{elicited}\) in virtual environments and demonstrate that VEs are able to elicit emotion in participants. A high score obtained in the Q5 answer (To what extent what you experienced in the virtual world fitted your expectations about what could happen in a real world?) is in line with the idea that the animation of the scene is similar to what could happen in a real life situation and that SN enhance the valence of the scenario.

Fig. 3.
figure 3

Answers to the Reality Judgment and Presence Questionnaire and to two additional questions (AQ18 AQ19) about the sense of presence in the environment and in the social narratives. The height of each bar represents the median value, while the upper and lower whiskers are the 25th and 75th percentiles respectively

Full size image

6 Conclusion and Future Work

The overall results of this research showed that the ENVE was able to elicit specific emotions related to the emotional content of the virtual environment. We believe that the reason of these results depends on the features of the Emotional Narratives Virtual Environment, i.e. the social narratives combined with specific content of the emotional environment. These findings it is corroborate the idea that the ENVE approach can be used in the field of environmental psychology to study the effect of the surrounding environment in a ecological way. However this study present two limitations. The first one is the small samples and the difference between the number of male and female that could limit the possibility of a solid statistical analysis. The second one is that during the ENVE experience participants did not interact actively with the object and people in the environment. In order to strengthen the value of our results the experiment will be tested with a wider group of subjects and the number of male and female will be conformed. Moreover, in the future the ENVEs will be improved including both human-avatars and human-objects interaction in order to create condition more similar to daily life activities.