Keywords

1 Introduction

Various real-life stress can induce homeostatic changes in human behavior, brain, and body. For instance, associations have been reported between achievement on tests of memory and attention and the experience of everyday minor stressful events [11, 14]. To modify the amount of stress that a person has, researchers have been relying on manipulating the various factors of their subjects’ lives, such as their sleep quality and workload. However, it is unrealistic to control every aspect of the subjects’ lives, and they may always be more or less stressed than the researchers think they are, varying the data from the experiment. Furthermore, stress research to date centers on simplified stimulus-response paradigms conducted in highly controlled environments that do not resemble authentic settings, where real-life stress typically takes place. We propose developing a controlled virtual-reality (VR) environment that can allow for greater control over the subjects’ stress and performance when completing tasks [2]. These types of games are known as games with a purpose (GWAP). They are tools for helping scientists perform experiments and collect data [8].

This study modified a popular VR game, Keep Talking and Nobody Explodes [6], to test participants in a controlled environment. Keep Talking Nobody Explodes is a VR game where one player disarms a bomb while communicating with another player, a bomb-defusing expert, who has the bomb defusing instructions. The modules or puzzles for disarming the bomb include certain wires that must be cut, buttons that must be pressed at certain times, and keypads that must be pressed in a certain order. The player will need a VR headset (e.g. Oculus or HTC Vive) and controllers to play the game. Only the player can see the puzzles and bomb in VR, while the bomb-defusing expert can only check a manual to find instructions to solve the puzzles. The player and the bomb-defusing expert must communicate effectively to defuse the bomb.

2 Methodology

To create a controlled environment where researchers can measure the performance of a task and manipulate how well the subject performs, we designed a VR game based on the game Keep Talking and Nobody Explodes [6], which measures the subjects’ reaction times, communication times, and precision and recall of tasks. There are multiple levels of difficulty expressed in more puzzles (modules) and less time to solve them. There are many ways to measure stress and among those one of the most widely studied methods is to measure heart rate (HR) and Heart-Rate Variability (HRV) derived from ECG recordings [7]. We developed a simple framework where we can record the ECG signals while subjects playing the game under varying stress conditions, which are synchronized with the start and end times of game modules or events within the game. The event markers are created by the Lab Streaming Layer (LSL), a recording and annotation software [15] that synchronizes the recorded signals with events during the game.

In our version of the game, we implemented the following five types of game modules: Wires, Keypad, Simon Says, Button and Venting Gas, and each with a timer. The timer was implemented using the Unity time.time function. For the Wire module (shown in the top left, top right, and two in the middle right of Fig. 1), the player is required to cut a certain wire based on the color of the wires and how many there are. For the Keypad module (shown at the bottom right), the player must press the keypad in a certain order depending on the symbols displayed on the keypad. For the Simon Says module (shown in the top and bottom middle), the player must press the buttons in a certain order based on which button is blinking. For the Button module (shown in the leftmost middle row), the player must press the button at a certain time to solve it. The Venting Gas module, in the bottom left, is a “needy module” that captures the player’s attention and causes more stress. The entire description of the game, the design of the modules, and the implementation of the tools such as the lab stream layer details can be found in the full paper at [9].

Fig. 1.
figure 1

An image of the bomb in the VR game. The top left, right and middle portions of the bomb are the wire modules. The top middle and bottom middle modules are the Simon Says modules. The module in the bottom right is the keypad module, the module in the middle left is the button module, and the module in the bottom left is the venting gas module.

Full size image

2.1 Recording and Analyzing Electrocardiogram Data

The goal of this pilot study was to explore how the physiological responses of healthy individuals relate to the variability in time stress. To this end, we recorded the event markers, the players’ ECG, and their behavioral data under three conditions: stressful (hard), non-stressful (easy), and resting conditions. The stressful (hard) condition required the player to solve a puzzle module in twenty seconds, while the non-stressful (easy) condition allowed the player to solve a puzzle module in two minutes, and the resting condition allowed the player to take a break for one minute between two modules. In each experiment, the participants played four hard sessions, four easy sessions, and eight resting sessions. The easy and hard sessions were played in random sequence in the experiment. Two healthy individuals participated in this pilot study, resulting in the ECG signal data for a total of eight hard, eight easy, and 16 resting sessions. This study used a wearable ECG device called Heartypatch and its recording software [5] to record the ECG data during game play. Figure 2 shows the ECG data synchronized with event markers from the game, revealing the various levels of ECG signals depending on how stressful the player was at key points in the game. This study focuses on two physiological responses of our participants: HR and HRV. HRV is the fluctuation in the time intervals between adjacent heartbeats [12]. To calculate HRV, we first find the time intervals between the consecutive heartbeats using the findpeaks function in MATLAB.

Fig. 2.
figure 2

We recorded the ECG of the player using the HeartyPatch sensor while the player was playing the game, and overlaid it with the event markers. The total recording is four minutes and thirty seconds. For the first minute, the player is resting to normalize their heartbeat. Afterward, for two minutes the player is playing the non-stressful version of the game (easy mode). The player then takes a minute rest and starts the stressful version of the game for twenty seconds (hard mode).

Full size image

3 Experimental Results

After collecting the data through LSL from our experiment, we split the data using the event markers from the game. We identified which sections of the ECG data correspond to easy, hard, or resting sessions. We then analyzed the data from the separated and categorized (easy, hard, or resting) sessions to find the average heart rate and HRV values for the easy, hard, or resting conditions using MATLAB. Figure 3 shows the average heart-rate values under the three (resting, hard, and easy) conditions. The hard (stressful) sessions have the highest average heart rate of 96 bpm. The resting and easy (non-stressful) sessions have average heart rates of 84 and 82 bpm, respectively. The difference between the average heart rates of easy and resting sessions is small, but the error bar for the easy sessions is larger, indicating that the variation of heart rates in the easy sessions is larger.

Fig. 3.
figure 3

This figure shows the average heart rates in beats per minute for two subjects for the 16 resting sessions, eight easy (non-stressful) sessions, and eight hard (stressful) sessions. The error bars denote the variations in the heart rates within the resting, easy and hard sessions.

Full size image

Figure 4 shows the average HRV in milliseconds, along with error bars, under the three (resting, hard, and easy) conditions. As shown in the bar graph, the average HRV value under the stressful (hard) condition is around 30 milliseconds while those under the non-stressful (easy) and resting conditions are around 40 milliseconds and 50 milliseconds, respectively. Thus the difference between the average HRV values of the resting and easy conditions is around ten milliseconds, and the difference between those under the easy and hard conditions is also ten milliseconds, which makes the difference between the average HRV values under hard and resting conditions to be about 20 milliseconds. We also note that the average HRV value under the stressful (hard) condition is the lowest.

We used statistical tests to examine if the differences in HRV between conditions are statistically significant. To calculate statistical significance, we determined the null hypothesis and an alternate hypothesis. Since the level of statistical significance is often expressed as a p-value between zero and one, we calculated p-value to show that our results are statistically significant. Typically, a p-value less than 0.05 is considered statistically significant. We thus used a one-tailed hypothesis with a significance level of 0.05.

To calculate the p-values, we first calculated the standard deviation for both sets of data under the statistical comparison. Then, we calculated the t-score and the degrees of freedom using the number of samples in each data set [13]. Finally, we used a p-value calculator to find the p-value using the t-score and the degrees of freedom [3].

For example, to find the p-value between hard (group 1) and easy (group 2) sessions, we used the following steps:

First, we determined the null hypothesis and the alternative hypothesis as follows: The null hypothesis: “There is no significant difference in the data sets (HRV values) of group 1 (hard sessions) and group 2 (easy sessions).” The alternative hypothesis: “There is a significant difference in the data sets (HRV values) of group 1 (hard sessions) and group 2 (easy sessions).”

Second, we calculated the standard deviation \(\sigma \) of the HRV values (data sets) for the hard sessions and easy sessions using the following formula:

(1)

where \(x_i\) is an individual data value in the given group (hard or easy), \(\mu \) is the mean of the data for the given group (hard or easy), and N is the total sample size of the given group (hard or easy).

Third, we calculated the standard error s between group 1 (hard) and group 2 (easy) sessions using the following formula:

$$\begin{aligned} s = \sqrt{(\sigma _1/N_1) + (\sigma _2/N_2)}, \end{aligned}$$
(2)

where \(\sigma _1\) is the standard deviation for the data in group 1 (hard), \(N_1\) is the sample size of group 1 (8 for hard), \(\sigma _2\) is the standard deviation for the data in group 2 (easy), and \(N_2\) is the sample size of group 2 (8 for easy).

Fourth, we calculated the t-score using the following formula:

$$\begin{aligned} t = |\mu _1-\mu _2|/s, \end{aligned}$$
(3)

where \(\mu _1\) is the mean of the data for group 1 (hard), \(\mu _2\) is the mean of the data for group 2 (easy), and s is the standard error between group 1 and group 2.

Next, we calculated the degrees of freedom d using the following formula:

$$\begin{aligned} d = N_1 + N_2 - 2, \end{aligned}$$
(4)

where \(N_1\) is the sample size of group 1 (8 for hard), and \(N_2\) is the sample size of group 2 (8 for easy).

Finally, we used the p-value calculator [3] to find the p-value based on the t-score and the degrees of freedom d.

Fig. 4.
figure 4

This figure shows the average HRV values under three conditions for two subjects. The statistical testings show that the HRV differences in the resting vs hard, resting vs easy, and easy vs hard are all significant.

Full size image

As shown in Fig. 4, the p-value for the null hypothesis that there is no significant difference between the average HRV values in hard and easy sessions is \(p=0.00244\), which is less than the significance level of 0.05, indicating the difference between these values is statistically significant and unlikely to happen by chance. The HRV differences between resting and hard groups and between resting and easy groups are also statistically significant since their p-value is much less than 0.05 (\(p < 0.00001\)).

3.1 Comparison with Other Results on HRV Under Stressful Conditions

We compared our findings of the lower HRV values under stress to prior work such as [1], which analyzed the influences on HRV Values in athletes, and [7], which analyzed the relationship between HRV and regional cerebral blood flow. Both studies showed that stress in athletes and higher blood flow was correlated with lower HRV values.

In the study [10], researchers evaluated the effect of real-life stress on the cardiac response of the subjects. They concluded that stress increases arterial pressure and impairs cardiovascular homeostasis.

Furthermore, in another study [16], participants performed tasks that either had a physical, mental, or combined load, while their HRV was measured. The study concluded that HRV is affected by changes in physical or mental states, and they were also able to differentiate between resting, physical and mental conditions through the characteristics of HRV.

Researchers surveyed London-based civil servants (aged from 35 to 55) [4] to evaluate the stress levels due to their work. They reported correlations between high work stress, low physical activity, poor diet, and most importantly low HRV.

4 Conclusions and Next Steps

In conclusion, we created a VR GWAP framework based on the game “Keep Talking and Nobody Explodes” intending to relate physiological data and stress levels. The framework includes event markers so that the physiological, behavioral, and audio data streams and events in the game are totally synchronized. We used ECG to show how this GWAP can be used to assess the physiological correlates of time stress. The empirical results were consistent with those reported in prior studies that the HRV values decrease with increasing levels of stress [4, 7, 12, 16].

Our game is meant to be a framework for experimentation so that other researchers can add more tools or sensors. We used ECG to measure HR AND HRV as they are known to be correlated with stress. However, this framework can be extended to other modalities such as electroencephalogram (EEG) to measure brain activities that are correlated with stress, or eye gaze/pupillometry to find other physiological correlates of stress. This framework also allows a thorough investigation into the speech characteristics of players under stress, for example, to extract features such as pitch or loudness. We believe that this framework can facilitate further research on the associations between physiological signals and stress (and other mental and cognitive states).