Keywords

1 Introduction

Enjoyment is an important aspect of the game experience, as it is one of the factors mostly contributing in motivating users to play. Newly available technologies are reshaping the way we experience video games and in particular immersion and social interaction. Players can customize their experience, choosing the most familiar and effective forms of communication by using the new interaction technologies. The spreading of devices interfacing users with computers by using the innate human means of communication (e.g. voice and gestures), has empowered paradigms of natural interaction, namely Natural User Interfaces (NUIs), which relevantly impact both on communication and immersion. Moreover, the availability of cheap highly immersive visualization systems, such as low-cost head mounted displays (HMDs), is bridging the gap between video games and immersive Virtual Reality applications. Spatial presence and flow are considered key concepts [16] to explain how such immersive experiences commonly lead to better performance and enjoyment. The role of flow in gaming, and in particular in Serious Gaming (SG) [1], has been increasingly recognized as one of the most important factors contributing to a pleasant and effective user experience. Moreover, immersive VR is able to foster acquiring knowledge in a non-symbolic world very similar to the real one, thus increasing information retention [10] which is typically one of the most important learning goals of SGs [9].

This paper aims at assessing a shared immersive system, featuring natural interaction, with the objective of evaluating its use as a platform for SGs. After presenting the relevant literature related to the use of NUIs in Serious Games, the paper introduces the adopted technological setup and the experiments designed for its assessment. Results are finally presented and discussed.

2 Literature Review

The recent availability of low-cost devices able to acquire physical properties of users (body gestures, speech, etc.) has made possible the design and developments of NUIs, i.e. transparent interfaces allowing users to execute complex interactions. This is particularly true in games and in Virtual Reality, in which the operation to be performed are commonly simulations of corresponding real tasks. NUIs give users the opportunity of performing such tasks exactly the same way they would do in real life.

The use of NUIs in mainstream video games have been recently strongly fostered by the appearance of devices (such as the Nintendo WiiMote or the Microsoft Kinect) allowing tracking users’ motion and, therefore, enabling intuitive interaction paradigms. The same has happened in the field of Serious Games, where the use of NUIs has been proven to increase the appeal and intuitiveness of SGs for rehabilitation [11] and for exercises in the elderly [4]. The real-time detection and analysis of body motion provides feedbacks able to engage users in achieving better results [12]. In training, NUIs make it possible to develop SGs that educate how to properly follow procedures and also how to physically behave to perform the required operations by accurately monitoring body position and posture [17]. The same applies in every sector of learning where physical actions are relevant.

Brondi et al. [3] have shown how setups combining immersive display with a NUI lead to a substantial increase in enjoyment with respect to standard desktop setups in Collaborative VEs. While performances seem not to fully benefit from such platforms, although a careful and dedicated interface design could overcome this issue. Sajjadi et al. [13] investigated whether the choice of interaction mode/controller has an impact on the game experience testing a collaborative game using an Oculus Rift. They observed that almost all participants using the HMD looked for alternative ways of communication trying to use gestures to interact with the partner even if not enabled by the technological setup. Lindley et al. [8] focus on the impact of the new interfaces involving body movements on player engagement and social behaviour claiming that the amount of social interaction is higher when using input devices which allow body movements.

3 Method

In the following, the technological setup and the interaction metaphors under evaluation are first introduced, then two experiments aimed at evaluating the system are presented. The first one has aimed at evaluating system usability, awareness and embodiment using a qualitative analysis through questionnaires. The last experiment has aimed at comparing the natural versus the classical (using Keyboard and Mouse) interactions in a playful context. In the latter experiment user engagement, social presence, performances, satisfaction and awareness have been compared using questionnaire results.

3.1 Technological Setup and Interaction Metaphors

In order to provide a high immersion to the user and to enable a natural interaction with the VE, a setup including HMD displays and depth cameras has been used [14, 15].

The two experiments have been conducted using two different versions of the system. The first prototype uses an Oculus Rift DK1 and an Optitrack tracking setup made up of 8 cameras. Subsequently, the system has been upgraded using the Oculus DK2 and exploiting the included positional tracker.

The HMD connected to the workstation provides visual feedback. A depth camera mounted on top of the helmet, and integral to it, is used both for the real-time 3D capture of the user body and for fingers tracking. The reconstructed 3D mesh of the body is coherently co-located in the virtual environment. The user is able to naturally interact with the VE by grabbing and moving virtual objects in his/her peripersonal space.

In the first version, three reflective markers placed on the HMD enable Optitrack positional-rotational tracking. The orientation data coming from the inertial unit built-in the Oculus are fused with the data coming from the optical tracker. The Optitrack system allows to track a wider space than the DK2’s positional tracking module used in the second version.

We will refer to this setup as “OU” from here on.

3.2 First Experiment

A pilot test has been performed in order to evaluate the first prototype and the implemented NUI, by observing how the users interacted with the system and how they felt while navigating the virtual environment and manipulating objects using their own bodies.

The experiment has consisted in a single-player immersive game, where participants have had to perform an assembly task as quickly as possible. Each user wearing the system has been free to walk in a 4 \(\times \) 4 m virtual room partitioned by walls. Six objects acting as landmarks have been placed in the room: a table, a refrigerator, a sofa, a painting, a chair and a TV Set. Eighteen floating coloured toy boxes have been spread across the room. The painting depicting a specified layout of seventeen boxes has been placed on a wall. The aim of the task has been to recreate the layout by grabbing the floating boxes and placing them on the table.

Participants. A total of 14 volunteers took part to the study, 7 male and 7 female, aged between 24 and 57 years (\(32.71\pm 9.12\)). Before the experiment they have filled the informed consent to participate in the experiment and an entry questionnaire to collect demographic and background informations. Previous experiences with 3D gameplay (\( 1.86 \pm 1.1 \)) and HMD (\( 2.28 \pm 1.32 \)) have been assessed on a 5-point Likert Scale from 1 to 5.

Procedure. First the users have played an entry session to get familiar with the system and the NUI. The average duration of this stage has been \(158.3 \pm 52.7\) s. Players were free to decide when to stop. Once the familiarization session has ended, the assembly task has started.

The toy blocks have been spread across the room. They have been arranged in three groups close to the landmarks in order to force navigating the whole environment and making landmarks noticeable. The room and furniture arrangement has been designed to enforce obstacle avoidance in order to test navigation ability and spatial awareness.

Metrics. Upon completion of the assembly task, participants have been asked to complete a questionnaire aimed at collecting subjective measures about awareness, embodiment and ease of interaction. Measurements have been assessed on a 5-point Likert scale from 1 to 5. Furthermore they have been asked to produce a sketch map of the VE on which they had to locate the landmarks. Similarly to Huang and Alem [5], a quantitative assessment of the mental representation of the virtual space has been evaluated using a score ranging from 0 to 6. The two experimenters have assigned a score to the sketches based on the number of remembered landmarks. The map evaluation for each user has been given by averaging the independent estimations done by experimenters.

Task completion time and user’s position over the time have been recorded.

3.3 Second Experiment

The experiment has aimed at comparing the OU system with a traditional Keyboard & Mouse gaming interface (referred as “KM”) as interaction devices. The study has focused on the impact of the different technological setups and interaction metaphors on user engagement, social presence, awareness and performances.

The experiment has adopted a within-subjects design and has been based on a collaborative multi-player jigsaw puzzle game purposely developed. Players have had to work together in order to solve the puzzle before the time (7 min) expires. Two players physically located in different rooms have played the collaborative game sharing the same VE using two identical networked setups. The actions performed by each player have been immediately visible to the partner. During KM sessions the mouse pointer of the partner has been visible to the player. During OU sessions the RGBD captures of the two players bodies have been streamed between the two setups. A proxy for each player, made by a textured mesh reconstructed from the RGBD data and a virtual head replicating the user movements, has been shown in the VE. The participants have been able to communicate both verbally — through headsets — and using their bodies (e.g. using gestures).

Participants. A total of 24 subjects, 15 males and 9 females healthy subjects, aged between 23 and 50 (\(32.04 \pm 6.84\)) took part to the experiment. Each of them filled an entry questionnaire, based on a 5 points Likert Scale from 1 to 5, to collect demographic and background information: experience with the use of computers (average \(3.88 \pm 0.85\)), videogames (average \(3.12 \pm 1.33\)), immersive virtual displays (average \(2.54\pm 1.21\)), puzzle games (average \(3.04 \pm 1.2\)) and online puzzle games (average \(1.83 \pm 1.05\)).

Procedure. During the recruitment, participants have been asked to play a single player version of the puzzle game using the KM setup. The pre-experiment has aimed at assessing the skills of each participant in solving a puzzle in order to form twelve pairs with similar dexterities.

The players have been hence divided on the two identical networked setups. The subjects, spatially not co-located, were able to communicate by using only the communication channels provided by the setup. Before starting the experimental sessions, each user has performed a 5 min trial session to get familiar with the OU setup playing a simplified single player version of the puzzle game.

Each pair have therefore played two game sessions, one for each interaction metaphor. The order in which the two game sessions have been presented to different dyads has been randomized.

Metrics. At the end of each game session, players have answered a subset of the Game Engagement Questionnaire (GEQ) [2] (competence, flow, tension-annoyance, challenge, negative and positive affect), a subset of the Social Presence in Gaming Questionnaire (SPGQ) [7] (empathy and behavioural involvement), awareness and satisfaction questions. An exit questionnaire has been presented to both players in order to collect their preferences and motivations, friend relationship, general impressions and suggestions. Finally an informal debriefing session between the experimenters and both players has been conducted to further register impressions and anecdotes.

Objective measurements recorded have included: (1) completion time and score, (2) frame-rate and network latency, (3) outcome and tiles positions, (4) positions and headings of player head. Experimenters attended all the sessions taking notes of noteworthy events.

4 Results

The outcomes of the first experiment have been analysed in order to assess the liking of the OU system when navigating in a VE and accomplishing a simple task. The Mean and Standard Error of the Mean (SEM) are reported for the questionnaire’s answers. Results of the second experiment have been used to compare the different technological setups and interaction metaphors. A Wilcoxon signed-rank test has been used to statistically compare questionnaires results and performances for the two conditions as the distribution of the data was not Gaussian.

Fig. 1.
figure 1

Results of the first experiment

Full size image

4.1 System and NUI Assessments

During the first experiment the developed interface has been tested in order to obtain a first indication of the levels of embodiment and awareness reachable. Furthermore the overall complexity of the interface using a basic interaction with the VE — pick and drag — has been also evaluated. The average time needed by the participants to accomplish the assembly session has been \(455.35 \pm 113.04\) s.

Awareness. Figure 1a reports the level of spatial awareness (\(4.21 \pm 0.26\)) and self awareness (\(4.28 \pm 0.19\)) reached during the experiment. The \(78.57\,\%\) of the participants have been able to correctly estimate the size of the VE. The subjects have been able to remember almost all the landmarks encountered (\(4.71 \pm 0.91\)).

Embodiment. As shown in Fig. 1b, the subjects have had a strong feeling of embodiment. They have perceived the virtual proxy as a real representation of themselves (\(4.43 \pm 0.17\)) and they have been convinced to be in a real physical place (\(4.14 \pm 0.18\)). Furthermore the participants have strongly perceived both the interaction with the virtual objects (\(4.0 \pm 0.21\)) and the navigation in the VE (\(3.86 \pm 0.18\)) as a real physical tasks.

Interaction. As reported in Fig. 1c, participants have found the interaction with the virtual objects easy (\(4.0 \pm 0.23\)) and natural (\(4.28 \pm 0.19\)). They have also found easy to navigate the VE (\(4.57 \pm 0.17\)).

4.2 Systems and Interactions Comparisons

During the second experiment, the classic KM system has been compared with the OU in order to evaluate their impact on performances, game engagement, social presence, awareness and satisfaction.

Fig. 2.
figure 2

Results of the second experiment (*\(p \le 0.05\); **\(p \le 0.01\)). Bars reports 25th and 75th percentiles. (Color figure online)

Full size image

Performances. Figure 2a shows participants’ performances during the first and the second half of the game for each session. Results report a significant increment (\(W(23) = 45.0, p = 0.025\)) in the number of correctly positioned tiles in the KM session during the second half of the game.

Only \(41.7\,\%\) of the pairs have won the OU game session while the \(75\,\%\) have completed the puzzle in the KM session. As reported in Fig. 2b, players have completed the puzzle significantly faster (\(W(23)=21.0, p=0.0002\)) during KM sessions.

GEQ. The results of GEQ questionnaire indicate an overall positive evaluation of both setups (see Fig. 2d). Players have reported a higher level of flow (\(W(23)=35.0, p = 0.009\)) as well as challenge (\(W(23)=22.5, p=0.0105\)) during OU sessions.

SPGQ. Participants have high-rated both social components, Empathy and Behavioural Involvement. No relevant differences have been found (see Fig. 2d).

Awareness and Satisfaction. As reported in Fig. 2c, players have had a good awareness of the other’s actions, locations and intentions in both setups.

All the participants have rated both experiences as very satisfying. When asking “Which kind of user interface do you prefer?”, 16 players (\(66.7\,\%\)) have answered the natural one.

5 Discussion

Almost all the players have enjoyed the proposed immersive system. Participants have highly rated usability and immediacy of the NUI during the first experiment. The majority of the subjects (\(66.7\,\%\)) of the second experiment have found the NUI preferable to the classic KM interface even if it resulted to be more challenging. The OU experience has been perceived as more engaging and entertaining. Almost all the participants who preferred the KM metaphor appreciated the lower complexity of the interface, which resulted more familiar and comfortable for people who daily use computers.

A high level of embodiment has been registered during the first experiment. Players have believed to be involved in a real experience thanks to the high immersion and sense of presence induced by the system. Allowing users to manipulate virtual objects using their own hands and to navigate in the virtual system using their own body has greatly improved their embodiment. Participants during OU session have perceived their own proxy and the partner’s one more as a physical presence rather than virtual. Experimenters have indeed observed that when navigating in the VE the subjects have tended to avoid collisions with physical objects — like walls — as they do in real life. Only three of them have intentionally crossed the virtual walls — breaking the embodiment illusion — in order to accomplish the task more quickly. Nonetheless the first time they have tried to cross the virtual wall they have been extremely careful because they have felt like hitting a real one. During the second experiment, several collisions have happened between the proxies of the participants. The absence of any physical feedbacks when passing through the partner’s representation has been perceived by some players odd and sometimes a bit annoying, while cheerful by others.

Both games have been designed in order to stimulate participant’s movement. In both experiments the subjects have reported high levels of awareness. During all the time they have been conscious of what was happening and how to reach their goal. During the second experiment high levels of awareness have been registered for both the modalities. The essential KM interface has resulted to be more functional to the task but, as observed by the experimenters and highlighted in the open questions, less fun and more impersonal.

Flow experience is one of the key factor to make a game engaging. The mental state reached in this condition makes the players completely engrossed in the game. During the second experiment, even if players have reported a high level of flow in both setups, the psychological absorption has been significantly greater in the OU session than in KM (see Fig. 2d). The subjects have felt completely disconnected from the real world. As a consequence of the deep absorption, the break between the virtual and the real world has been described by participants as considerably sharper during the OU sessions in both the experiments, as also reported in [6].

In the GEQ results of the second experiment, the OU system has been rated more challenging than the KM. However only one player has found the OU metaphor not enjoyable and too complex to be used. The majority of players (\(75\,\%\)) have considered the KM metaphor more immediate and faster. Nonetheless, most participants (\(71\,\%\)) during the final debriefing have asserted to appreciate more the OU metaphor (“I felt like I was really playing with him a real puzzle!”).

6 Conclusion

The developed system has proven to be an interesting alternative to the classic desktop interface. The NUI have beaten the classic keyboard & mouse setup in terms of flow reached during the game. The deep immersion provided by the system has led to a greater absorption in the game. A natural interaction with the environment allows users to act and communicate like they do in real-life. Games can be played without the mediation of any interface and interacting with the VE using the own body. This turns out to be extremely important to improve embodiment, engagement and awareness in the subject: “It was like playing a real puzzle”. The adoption of a photo-realistic representation of user’s body in a coherently rendered virtual scenario has induced a strong feeling of embodiment without the need of a virtual avatar as a proxy. The user’s ability to grasp and manipulate virtual objects using their own hands has not only provided an intuitive user interaction experience, but it has also improved user engagement.

Even if the natural interaction has been described as more intriguing and enjoying, it has resulted to be also more challenging; better performances have been achieved using the classic Keyboard & Mouse setup. Nonetheless, in the second experiment during KM sessions, experimenters have observed that many players have tried to use body language and gestures to interact with each other, even if these forms of communication were not enabled, as similarly reported in [13]. Hence providing a natural interaction seems to be important during social activities but further investigations have to be carried out to ease the NUI and filling the performance gap.

Given these results, allowing users to manipulate virtual objects with their own physical hands and to navigate the virtual space using their physical body has the potential to not only improve users interaction but also spatial understanding and self perception. Considering these characteristics, we believe that this system can be proficiently used for designing effective Serious Games.