Keywords

1 Introduction

A key component of games is the player experience (PE). When the PE is not enjoyable, it has an impact on the graphics and the gameplay [1]. The entertainment of the player, created by the interactivity of the game, is frequently seen as the definition of PE [2]. It is suggested that without players, games have little or no value [3]. PE testing is constantly evolving since the creative media industry is competitive. The use of biometric measurements in the gaming industry is one of these evolutions. Interest has been shown in implementing biometric measurement in the gaming industry [4]. The integration of physiological measurements is to evaluate the emotional engagement of the player and has been demonstrated as a suitable method using sensors for the game industry [4]. The measurements could be acceleration, eye tracking, heart rate, skin connectivity and skin temperature [5]. The gathered data could be analysed and combined to create datasets such as the arousal or valence of the player [6].

Biometric measurements could be used to track the PE through a controller. An Xbox [7] 360 controller was altered to include biometric sensors [8]. Heart rate and movements are examples of biometric measurements performed by this altered controller. Sony [9] patented biometric measurements for the PlayStation 5 [10] controllers [11].

In the game industry, biometric measurements are mainly used in research to test game events [12], features [13] or social interactions [14]. The use of biometric measurements to manipulate the game experience in real-time is still in development [15]. Biometric measurements could be used to control characters in Flappy bird [16], Snake [17], World of Warcraft [18], Portal 2 [19] and racing games [20]. It is possible to modify game elements dynamically in a dynamic difficulty system. Here the skill level of the player dictates the difficulty of the game and could be linked to the biometric measurements of the players [21, 22]. Applying biometrics in games can also be done differently by creating a framework [6]. The overall impact of these design changes on the PE has not been assessed. In this framework, Electrocardiogram sensors, electrodermal activity sensors and an accelerometer were used. Four different factors were adjusted based on the measurements. A threshold in the biometric measurements was set for the shooting mechanic, were the player would attack once the threshold was exceeded. The attack of the enemies would become more powerful after the player crosses an arousal level. The emotions of the player impacted the mood of the game [23], creating positive feelings for a bright environment and negative feelings for a dark environment. The difference between the framework [6] and the presented study is that the study focused more on the relationship between the design change and the PE. No research was done on the impact of the framework. The framework is more detailed than the study since the impact of changing one aspect of the game was researched.

In the study, the PE is measured using different indicators. The first indicator is the emotions and feelings of the participants since the PE is often described as the amount of fun of the player [2]. The competence of the participant can also be an indicator for the PE. The enjoyment of the art style impacts the player experience because different types of visual styles are pleasing to different people [24]. Another indicator of the PE is the immersion of the participants which keeps the interests to suspend disbelief. The final indicator used in this study is the experience with virtual reality (VR). To create a pleasant PE, the participant should help with the immersion without motion sickness from VR.

For this study, a VR corn maze was created with two different versions, one (the adapted version) included weather effects that were manipulated when emotional arousal peaks occurred. The other version (the control version) was a set experience without any manipulation of the weather effects. When the arousal levels of the participants were above a set threshold in the adapted version, the weather got worse.

The study focused on improving the PE by applying mood changes based on real-time biometric measurements in a virtual reality corn maze game. The assumption was that manipulating the game world using real-time biometric measurements would impact the PE.

2 Method

Two versions of the VR corn maze were developed for the study. The first version of the maze had a constant experience that did not change due to the biometric measurements. The weather and lighting for the second version of the maze were changed due to the biometric measurements.

2.1 Research Design

The environment needed to create a visible arousal spike in the biometric measurements of the participant. A questionnaire was made to understand the emotions evoked by different environments using 15 different mood boards. For each mood board, the participant had to rate the pleasantness from one to five, which can be seen in Fig. 1 which shows the emotions evoked by the pictures.

Fig. 1.
figure 1

The pleasantness of the mood boards

Full size image

To trigger the mood changing multiple times, the VR environment should evoke both negative and positive feelings. The cornfield mood board was the only one that seems to evoke a range of different emotions. The horror genre influenced the participants’ view on a cornfield, making it chilling when it is dark and soothing when it is light. To make the experience enjoyable, a gameplay element was added by making it a corn maze.

All mazes were completed by two participants who did not participate in the final experiment, which gave an indication of the time it would take to complete the maze. Figure 2 displays the mazes and the completion times of two participants for each maze. The experience was intended to take a couple of minutes, which meant that maze 1, 4, and 6 were too short. With the changing mood and extra decorations, it was expected that the completion time would increase by a couple of minutes. The fifth maze was chosen to keep the experience relatively short. This maze was not the longest, but long enough for the biometric measurements to impact the PE.

Fig. 2.
figure 2

Six different mazes and their completion times (minutes)

Full size image

A closed area is the start of the VR environment where the maze is the next part of the environment and at the end is another closed-off part. The environment is decorated to resemble a farm. The Empatica E4 [25], the biometric measurement device used for the study, needed some time to obtain a baseline for the Galvanic Skin Response (GSR) and the skin temperature of the participant. The gates to the maze would open once the calibration was done.

Three different lighting sets were created (Fig. 3). Two different skyboxes were used, a bright skybox and a dark skybox. The sound system includes changing the background music, rain getting louder, and thunder starting to play when an arousal peak was detected. The weather can change and can become more or less cloudy, rainy and windy. At the last weather stage, thunder starts as well.

Fig. 3.
figure 3

The different moods and weather states

Full size image

Ten zones were created in the VR environment (Fig. 4). In the starting and ending zones and zone 0, calm music played. Halfway between calm and stormy, transitional music would play, when the mood had entirely changed, the stormy scary music would play. Zone 1 works the same as zone 0. As a crow flies past the participant in zone 2, the intensity of the light source placed inside the scarecrows increased, and the transitional music played unless the mood had changed to a full storm. Zone 3 and 4 were almost identical to zone two; the only difference is that there is no crow. In zone 5, light rain started and stayed for all the zones after this and scary music would be playing continuously. A crow crawl sound would play occasionally and the intensity of the light in the scarecrows increased. Zone 6 was almost the same as zone 5; another crow flies past the participant in this zone. Zone 7 was identical to zone 5. Zone 8 and 9 were the same; the music changed back to the transitional music when the mood was not fully stormy. The intensity of the light inside the scarecrow would decrease.

Fig. 4.
figure 4

Different zones of the maze

Full size image

The Empatica E4 [25] did not record the biometric measurements; it was more important for the plug-in to function in Unity [26]. A Python plug-in was used five requirements to discover the presence of an arousal peak [27]. According to rule number 1, the period between the stimulus and response had to be between one and five seconds. Rule number 2 stated that there had to be a drop in the skin temperature after the peak. The latency affected the rising time according to rule number 3. Rule number 4 stated that a steeper slope to the peak indicated a more intense arousal peak. According to the rule 5, the recovery time had to be between one to 10 s. When an arousal peak was determined, the plug-in would send a signal to Unity which could change the weather and lighting in the environment [26]. It took the participants roughly eight minutes to complete the maze.

2.2 Materials

The hardware for the biometric measurements was the Empatica E4 wristband [25]. The GSR data and skin temperature were used based on factors that showed an arousal peak [28]. The Python [27] plug-in [29] and the Patterns of Basic Emotions [23] were used as supporting theory.

The Unity Engine [26] was chosen because it was familiar to the participants. The assets for the corn maze were gathered from different places, and two different VR head-mounted displays (HMD) were used.

The assets were selected because they looked like they belonged on a farm. A bright and a dark skybox were used. A couple of different weather particle effects were added in the Unity scene. The sound effects of a crow, rain, thunder, wind and background music were gathered from Quixel Bridge [30], the Unity asset store [31], websites like TurboSquid [32] or a universal sound FX library [33].

There were two HMD, the HTC Vive Pro and HTC Cosmos [34]. Three different VR set-ups were used of which two of them were used for testing the research while the other was for the development. The Experience lab [35], a research group at Breda University of Applied Sciences that works on interactive experiences, was used as the main set-up for the study. A private set-up was used for the participants that could not travel to the campus in Breda (Fig. 5).

Fig. 5.
figure 5

Experience lab VR set-up and private VR set-up

Full size image

3 Method

A playtest session was held to discover if the player experience can be enhanced by using real-time biometric data to adapt the mood of a VR game environment. The biometric measurements of the Empatica E4 [2] were used to change the mood in the maze.

Data about the emotions and player experience of the participants before and after the playtest session was gathered from the pre-experience and post-experience questionnaires which gave insights into the feelings of the participants before and after the maze, their experience while playing the maze and their PE. The pre-experience questionnaire included questions about general information such as participant number, time and research group. Participant information such as age, gender, nationality and level of education was gathered if the participant stresses easily, or was a gamer, or had experienced VR before, and their favourite game genre. The last part of this questionnaire measured the arousal, valence and current emotions of the participants [36, 37].

The post-experience questionnaire started with whether or not the participants completed the maze and the time they took. The same questions as in the pre-experience questionnaire were used for the measurement of the arousal, valence and current emotions to compare the participants’ emotions before and after the experience. To gather information about the PE, data was gathered on the feelings during the experience, skill of the participant, visuals, engagement and VR. The last part of the questionnaire focused on how recommendable the experience was and the overall grade. A combination of the Game Experience Questionnaire and Player Experience of Need Satisfaction [38, 39] was used. Both questionnaires used the Likert scale from extremely agree to extremely disagree, apart from open questions and emotions. The data analysis was done in SPSS [40].

Thirty-four participants were recruited through convenience sampling. Seventeen participants each were randomly assigned to the control group and the adapted group for the Empatica E4 [25]. The adapted group had real-time biometric measurement mood adaptations, and in the control group, the biometric measurements did not influence the mood. Since the experiment was done in the Netherlands at a university, the majority of the participants were Western European and between the ages of 20 and 30. The experience took place in two locations in the Netherlands, Breda and Zutphen in March 2021.

Fourteen participants were male and 20 were female. The mean age of the participants was 27.5 years, with the youngest being 12 years and the oldest being 69 years old. The 12 year old participant had played with permission from both the parents. One parent participated before the child and the other parent watched the spouse. The study had 28 Dutch participants. Out of the other six participants, four were Bulgarians. The remaining two participants were from Germany and Zimbabwe. Twelve participants had not used any type of VR over the past 12 months. Ten of the remaining participants had used VR once. The last two participants had more experience with VR and had used it 20–30 and 50+ times respectively.

4 Results

The control group all had the same experience. The mood was sunny until zone 5, when the rain started. There were five different outcomes for the adapted group (Fig. 6). The biometric measurements of the participant could stay low without fluctuation, implying the participant was calm, due to only sunny weather. With the second outcome, the biometric measurements could stay low with minor fluctuations, implying that the participant was reasonably calm due to the mostly sunny experience. The biometric measurements could continuously stay high, suggesting that the participant was emotionally aroused. The experience quickly went from sunny to stormy. The biometric measurements could stay high and still fluctuate, implying reasonably aroused participants, which lead to a mood that fluctuated between a light storm and a heavy storm. Alternatively, the biometric measurements could fluctuate, insinuating that the participant got emotionally aroused at certain places, which caused an experience where the mood was constantly shifting. The participants were not aware of other participants having a different experience.

Fig. 6.
figure 6

Experience of the adapted group

Full size image

The means and standard deviations of six dimensions of the adapted and controlled experiments are shown in Table 1 below.

Table 1. Means and standard deviations
Full size table

5 Discussion

Improving the PE by applying real-time biometric mood changes to a game was the intention of the study. Both study groups rated the overall experience pleasant and worth recommending. A minor yet visible dissimilarity could be seen in the data of the control and adapted groups. The participants felt better after the experience than before and the control group felt better than the adapted group. The maze was more difficult for the adapted group due to the worsening weather, which explained why the participants felt less skilled. The visuals, immersion and VR experience were also rated lower by the adapted group. Thus, the PE was not enhanced but rather reduced.

It is possible that arousal changes during the game play do not make for a better PE [41]. The worsening weather may have created a negative feedback loop which would have been enhanced by the negative connotations of rain and storm [42]. It had a negative effect on the feelings of the participants and impacted the sight of the participants. It was taxing for the eyes, the lower-rated visual style, VR experience and competence which could be explained by the connection between real-time biometric mood changes and the PE.

The reduction of PE does not mean that the PE could not be enhanced using real-time biometric data. If a positive feedback loop was created instead, the PE possibly could have been enhanced. The possibility of combining real-time biometric measurements with VR games is relatively new. There has not been much academic research in this field due to which comparing data in the study has proven to be a challenge.

5.1 Limitations

Some improvements could be made to the VR corn maze. The resolution and frame rate in VR was sufficient but not optimal. The corn maze walls could have been made denser. Places in the VR environment were not properly lit and some signposts were not readable. Every time the mood changed, the crops reset to their original position. These issues were known but could not be resolved in time due to the scope of the project.

The most important part of the VR environment was the changing of the mood which could have been handled differently. It was used to suit the scope of the project. The lack of an arousal peak could be used as a trigger for the changing of the mood.

The plug-in worked adequately but could have been tweaked more. Arousal peaks were detected occasionally, giving a null or a maximum value. This issue could not be resolved due to the limited time.

Due to the COVID-19 pandemic, gathering participants to play the game was difficult. The extent of their VR experience was therefore limited. The lack of participants resulted in two different game locations.

5.2 Future Indications

A continuation of the present study is suggested to inspect the negative relation between the mood changes and the PE. A different implementation of the real-time biometric mood changes could create a positive relationship between the mood changes and the PE, and therefore, further study into this positive relationship is advised. Increasing the number of participants and using a more diverse sample could create a more accurate assessment.

The possible uses for changing a game based on biometrics could be personalization, a dynamic difficulty system and physiological horror elements. A game experience using real-time biometric measurements to customize sound, character or environment design could improve the PE [43]. Real-time biometric measurements could be used for real-time adjustable difficulty settings which could prevent players from getting frustrated due to difficulty of the game and would improve the PE [44].

A different type of implementation could use real-time biometric measurements to increase psychological horror elements in games. If the measurement would pass a set threshold, the actual horror event could happen. If the threshold is not reached, the tension could increase [6, 45].