Keywords

FormalPara Overview

Serious Games and Virtual Reality (VR) have been accelerating in their quality and ubiquity within healthcare simulation, and the variety of technological innovations is outpacing the healthcare research community’s ability to evaluate their effects as an intervention or their utility in simulating an environment for research. This chapter seeks to highlight unique advantages and challenges when using serious games or VR for healthcare research that are different than those encountered with other simulation modalities such as manikins, simulated/standardized patients etc. First, we define the terminology surrounding the concept of serious games and VR, including the advantages and disadvantages for their utility in answering important healthcare research questions. Second, we provide insight into optimal models of research that are suited for serious games or VR. Finally, we describe the development process for researchers to integrate research methodologies during the development phase.

FormalPara Practice Points

  • Screen-based Simulation (SBS) consists of any digital simulation using a computer or mobile device screen or a virtual reality headset.

  • Serious Games and VR have distinct advantages and disadvantages over manikin-based simulation for both development, implementation, and data collection for research.

  • Data collection in serious games and VR must be built-in during the development process for the software.

Introduction

While serious games can be defined in a variety of ways, they can be best described as games that educate, train, and inform, for purposes other than mere entertainment [1]. Serious games can be applied to a broad spectrum of application areas , (e.g., military, government, educational, corporate, and healthcare). Many attempts have been made at defining what constitutes games – to understand how they work to facilitate learning. Specific attributes define a simulation as a serious game, which include a taxonomy of concepts described by Bedwell et al.: assessment, conflict, control, environment, rules, goals, fantasy, and immersion [2]. Not all serious games require a screen or electricity, as board and card games that facilitate learning can also be a form of serious game.

Virtual Reality (VR) is an artificial reality which is experienced through sensory stimuli, such as sight and sound, provided by a computer in which one’s actions determines what happens next in the environment. VR is constantly changing with the exponential growth of technology. In the past, VR described an environment or situation through the eyes of a computerized avatar such as a first-person video game. However, as hardware technology improved, types and opportunities for healthcare VR have also advanced. Of note, VR differs from Augmented Reality in which digital imagery, text, or characters are superimposed onto a display of an individual’s real environment. This contrasts with VR’s ability to shut out the real environment entirely to allow a completely immersive experience. Augmented Reality will not be discussed in the chapter.

Screen-based simulation (SBS) is a form of simulation in which one or more scenarios are presented through a digital screen surface [3]. This can include virtual patients, virtual worlds, and virtual trainers. The user interacts with a game selecting the next step or test from selection menus. As with other forms of simulation, SBS provides the user with a safe place for experiential learning and assessment. SBS includes serious games and VR, but not all SBS require game elements or game mechanics. Examples are shown in Figs. 5.1, 5.2 and 5.3.

Fig. 5.1
figure 1

Vital signs . (Screenshots courtesy of Dr. Todd P Chang, MD MAcM, and BreakAway Games, Ltd., with permission)

Full size image
Fig. 5.2
figure 2

(a, b) Pediatric resuscitation simulator . (Screenshots courtesy of Dr. James Gerard, MD, and BreakAway Games, Ltd., with permission)

Full size image
Fig. 5.3
figure 3

(a, b) Stanford heart project . (Screenshots courtesy of Dr. David Axelrod, MD, and the Lighthaus Inc., with funding support from The Betty Irene Moore Children’s Heart Center at Stanford and Oculus from FaceBook, with permission)

Full size image

3D VR or Head mounted VR refers to the use of a goggle/headset type device such as the Oculus Rift (Oculus VR, LLC, Menlo Park, CA), HTC Vive (HTC Corporation, Xindian City, Taipei), Gear VR (Samsung, Ridgefield Park, NJ), or Google Cardboard (Google, Mountainview, CA) to create a fully immersive 360° environment that substitutes one’s audiovisual reality with a virtual environment. Many of these definitions can refer to the same product. A serious game may use a VR headset, though not all VR experiences are games. Examples of these VR simulations are shown in Figs. 5.3 and 5.4.

Fig. 5.4
figure 4

(a, b) Oculus CHLA virtual reality project . (Screenshots courtesy of Dr. Joshua Sherman, MD, Dr. Todd P Chang, MD MAcM, a.i.Solve, Ltd., BioFlightVR, and Oculus from FaceBook, with permission)

Full size image

Advantages and Disadvantages

Screen-based simulation (SBS) has five main advantages over other forms of simulation; all of which can be useful to healthcare researchers. These advantages as noted in the literature are: standardization, portability, distribution, asynchrony, and data tracking [4, 5]. Because SBS is basically a predetermined computer algorithm; by definition, it is standardized for each user. Although modifications can be made to accommodate different levels of player expertise, each user at the same level will experience the same simulation with the same options. Portability and distribution are similar concepts. SBS can use mobile devices, tablets, laptops, or VR headsets, which are common items. Portability refers to the ability to move the hardware or proprietary devices easily across healthcare arenas or institutions, resulting in reduced equipment and travel. Similarly, distribution is the ability of the software to be replicated or copied across hardware (such as a flash drive) and online, whether through a proprietary network or the world wide web. The combination of portability and distribution reduces many of the barriers in conducting multi-center trials, for example, when compared to manikin-based simulation. Both portability and distribution allow for asynchrony , which is the use of the simulation without a facilitator or instructor immediately present. As an example, manikin-based simulation (MBS) requires a technician, confederates, and/or a facilitator for debriefing at the time of the simulation, which would be considered synchronous. While facilitator-led debriefing is standard and common in MBS, there is no such standard with SBS, because it can be completed asynchronously on one’s own. There may be a benefit to having a facilitator or briefer/debriefer available synchronously (live) either physically next to the subject, or communicating to the subject remotely. Alternatively, the debriefing can happen at a different time and location, so the users can practice and improve at their own pace and when most convenient for them. Of note, a fundamental question for SBS, serious games, and VR, is the optimal structure and format for debriefing in this relatively new modality of simulation .

SBS also has built-in data tracking ; all user actions - whether input through a keyboard, mouse, controller, or VR head movement - are documented by the software with very precise timestamps. The variety of data tracked can be massive, and the researcher is advised to pick out the most meaningful data to answer their research questions, rather than to request all data. These performance data can easily and objectively be tracked and stored for either real-time or future review and assessment.

3D head mounted VR has the same advantages as SBS, with the addition of full 360° immersion, and shows promise for the removal of potentially distracting selection menus due to the nature of its interactivity. Most SBS use drop-down menus for item selection and scenario advancement. In VR, these selections and interactions can be done with realistic movements, such as pointing, grabbing, or simply staring at an item of choice. For example, a virtual crash cart can have drawers that the player physically pulls open to reveal medical choices. In other words, a well-crafted VR environment allows the user to select an item that is in their virtual environment without needing a drop-down menu. In addition, 3D head mounted VR can track gaze patterns automatically within the hardware. Researchers wishing to incorporate gaze tracking as a measure of situational awareness or of attention and focus, can do so more readily within a 3D head mounted VR environment.

Disadvantages

The major disadvantages of SBS relate to inherent technological limitations and the concept of selective fidelity. Selective fidelity refers to the limitations of SBS in maximizing different facets of fidelity: physical, functional, and psychological [6]. VR and SBS can have incredible visual and even auditory realism, but haptic realism is still in its infancy. In other words, SBS technology is limited in portions of physical fidelity it can provide. Providing realistic sensations of touch of healthcare instruments, and particularly the human body, is still a formidable challenge. The potential lack of haptics in SBS and 3D head mounted VR is a significant limitation compared to MBS, currently making it less ideal for procedural training, practice and assessment. But with the advancement of technology that could very well be mitigated in the relatively near future. The limitation of using a screen in SBS and serious games that are screen-based simulations may limit the degree of immersion and thus affect psychological fidelity. With 3D head mounted VR, the level of immersion has improved significantly given the 360° and interactive nature of the VR experience. Finally, the use of multiple menus, dropdowns, and computer-based interactions may also affect functional fidelity, such that the interactions within the SBS feel artificial or tedious.

Development of SBS and 3D head mounted VR both come with a high front-end cost and long development time, which can often be a rate-limiting step in a research study. The subject matter experts and researchers must work together with the developers and coders, which is costly and time-consuming. Contrast this with manikin-based simulation: once the manikin is purchased the researcher can immediately begin scenarios without engineering skill. Even though space and human resources is a cost and a concern for manikins, the skilled simulationist can work around it. With SBS, if there is no module or product, there is nothing to work with at all [3, 4]. In essence, the quality of the research depends wholly on the developers; even when contractual agreements are present (a necessity when doing research in serious games or VR), and the final product may be different than that which the research envisioned because of funding or timeline limitations.

As with most technology , SBS and 3D head mounted VR are subject to technological problems such as glitches, slowing, and even complete blackout. While manikins also rely on computerized parts and connections, a shutdown may be mitigated with other manikins, retooling, or modifications to the scenarios. Internet connectivity is also an issue, as a serious game or VR that relies on wi-fi will be completely useless without. Multi-player games or SBS require particularly robust connectivity. With traditional SBS prior to 3D head mounted VR, there has been a concern over the limit of functional fidelity given the 2D nature of the simulation and lack of full immersion.

Models of Research

As with other methods of simulation, there are two main types of simulation-based research related to games and VR: (1) research that assesses the efficacy of the VR simulation as a training methodology (e.g., simulation as the subject), and (2) research where the VR simulation is used as an investigative methodology (e.g., simulation as the environment) [7].

Once developed, initial studies should be conducted to collect evidence that supports the validity of the game or VR when used as an assessment and/or training tool by the target audience. Non-comparative studies are useful for assessing factors such as the content, internal structure, and discriminant ability of the game and game scores [8]. Further details are explored in Chap. 26. We draw a distinction between evaluating the technology itself (as a validity trial), which is different than evaluating the educational efficacy of the serious game or VR inserted within a system of learning. The latter is an example of simulation as the subject of research.

Simulation as the Subject

The intended goal of studying a game or VR is often to determine the educational effectiveness of the game. A number of factors should be considered when designing a study for this purpose. Though, in general, there is good evidence to support the educational effectiveness of simulation as a training method , the educational value of a particular game or VR tool cannot be assumed, particularly for higher order outcomes such as behavior change or patient outcome changes. Producing a high-fidelity VR simulation is challenging and often affected by factors such as budgetary constraints and technological limitations. These factors may reduce the educational impact of the game or VR simulation.

Cook and Beckman highlight the strengths of the randomized posttest only design which is well-suited for this type of game research, assuming sufficient numbers and randomization [9]. With a smaller or single-center cohort, a randomized pre-post study may be more appropriate. Taking advantage of portability, distribution, and asynchrony allows for larger sample sizes and multi-center participation to fulfill this requirement. Comparative studies should be conducted to assess how learning from the game compares to more traditional methods of training . Such studies may also help to inform how to best incorporate the game into existing training curricula.

Simulation as the Environment

In this type of analysis, the simulated environment is used as an experimental model to study factors affecting human and systems performance [7]. Perhaps more than any other type of simulation, games and VR allow for standardized and reproducible scenarios and could thus be beneficial for studying a wide range of performance shaping factors including individual and team performance, environmental effects, as well as technological, systems, and patient factors. Examples from the world of manikin-based simulation include comparisons of intubation devices tested on standardized airways [10], or documentation of variations in care between different facilities [11]. Because of selective fidelity, validation is critical for these platforms to make clinically relevant conclusions applicable to the real world. Studies that use serious games and VR as an environment to examine professional behavior or safety threats are rare in the literature. For example, validity evidence is emerging for a serious game on disaster triage management [12] and for pediatric resuscitation management [13].

Because simulation as the environment requires a high level of fidelity to generalize findings to the real world, serious games and VR development can be particularly costly and time-consuming to manufacture the perfect clinical environment. Although this is a limitation of designing high-fidelity, multi-player games to simulate clinical environments, there are growing resources to prevent starting from scratch. There are open source and purchasable resources for available human anatomy, hospital architecture, and even assets such as equipment, healthcare staff models, and programmed behavior, vocabulary, or movements. Examples include Applied Research Associates’ BioGears (www.biogearsengine.com/) and Kitware pulse physiology engine (physiology.kitware.com/).

Unique Variables for Games/VR

Serious games can capture precise data including actions performed within the game and time to actions. Web-based games can also provide researchers with system-wide data including information such as Internet Protocol (IP) addresses, user IDs, and login and logout times. For multiplayer games , interactions and the timing of interactions between players can be tracked. By recording actions and paths taken by players during game play, investigators can better appraise gamers’ decision-making process and reaction times; this process can serve as the basis for assessment of learning in serious games. Researchers may be able to better understand what goes on in the minds of the learners through the players’ actions and choices. A theoretical construct used outside of healthcare to describe this type of data collection is termed the information trail [14]. Loh et al. describe a deliberate data tracking framework that reveals not just the completion of objectives, but the process and the movements learners used to get there. It tends to answer questions about what, when, where, and how, but not necessarily why, which would require debriefing [14].

Data Collection Methods for Games/VR

The collection methods for data used for game/VR analytics can be separated into two categories: in-situ and ex-situ. In-situ collection occurs in the game itself (e.g., logging game-play events), whereas ex-situ is data collected outside of the game play (e.g., post-play surveys).

In-situ Data Collection : Most game engines have or can be adapted to interact with a Data Collection Engine (DCE) that allows for easy acquisition of in-game events (e.g., assessments and treatments clicked on by the player, doses of medications and fluids selected by the player through user interfaces, etc.). DCEs can provide both detailed and summary data that can be utilized by players, educators, and researchers after game play.

In some circumstances, game researchers may wish to view the actual game play either remotely or post-game play. Several options exist for screen recording. A number of software programs designed for recording computer screen videos exist. Researchers should be aware, however, that the simultaneous use of a screen recorder may slow down game speed unless run on a computer with high processing speed and graphics capacity. An alternative method for recording game play is the use of a High-Definition Multimedia Interface (HDMI)-cloner box. These devices can capture screen video and audio and transmit them to a remote monitor or storage device without slowing game speed .

Ex-situ data collection : During initial development and beta testing of a new game/VR, developers will often want to assess players’ satisfaction with the game. A number of survey-based tools have been developed for usability testing. These include the System Usability Scale (SUS) [15], the Software Usability Measurement Inventory (SUMI) [16], and the Questionnaire for User Interaction Satisfaction (QUIS) [17].

Practical Aspects: Development Phase

Subject matter experts (SMEs) must work very closely with the development team. Whether the serious game or VR is used as an educational intervention or as an investigative methodology [7], development must focus on the educational or assessment objectives. Developers often emphasize physical fidelity, rather than the functional fidelity, despite evidence within educational simulations emphasizing the latter (particularly in situations with significant budgetary constraints) [18]. With the advent of widely available physiology engines (ARA BioGears, Kitware PULSE, and HumMOD, etc., described above), the functional fidelity of physiological reactions can be maximized at lowered programming costs. However, SMEs must pay particular attention in anticipating the wide range of actions that high-performing and low-performing subjects may do in the simulated setting. These also include aligning the timing of physiological and treatment changes both within the game time and to real time in a thoughtful manner. Development teams benefit from aligning their work with frequent inputs from SMEs, as each team are likely to make assumptions about user behavior.

We emphasize that the research data plan must be ready prior to the development work. This also includes plans for how data will be transmitted to the researchers and must take into account whether performance data in the game requires secure data transfers (e.g., where will the data reside and where can they be copied?). Institutional Ethics and Review Boards may have additional restrictions on how data are stored, particularly if storage is cloud-based and protected healthcare information (PHI) is included in the dataset. Data that may inform how a user may perform in their workplace are also subject to additional privacy and confidentiality concerns.

Data collection and filtering must be integrated into all games and VR from the beginning, as substantial memory and processing is used to save and process granular data . The manner in which data are curated must be agreed upon. For example, when measuring time durations (e.g., time to chest compression) that are common in simulated medical activities, developers will need clear guidelines on when timing begins and ends, particularly if the scenario uses a strong branch-chain logic and conditional events. Most developers will be able to provide raw data using a ∗.csv file format common to spreadsheet-type data, but often the data will need to be summarized or cleaned prior to analysis or even displayed to the user. A plan that clearly specifies the research outcome variables and how they will be analyzed will assist the developers in appropriate data acquisition. As another example, capturing gaze data during VR is possible, but requires additional programming and substantial processor power to record during gameplay .

Data collection within VR or games depends on the interactivity and the hardware involved. Standalone VR devices (Oculus Go, Samsung Gear VR, etc.) can record positioning in 3 degrees of freedom but no other positional data. Full VR devices at the time of this publication (Oculus Rift S, HTC Vive Pro) can record the subject’s position in all 6 degrees of freedom and potentially gaze pattern. VR or serious games that use their own controller can record the timing and pattern of actions, including hesitancy, inaction, or even urgency if a key or button was pressed repeatedly very quickly. Developers typically use these types of in real time data to further the simulation or game, but recording these data for later use is memory- and processor-intensive, and should be planned in advance. It is not possible for the developer – without knowledge of the research question nor outcome variables desired – to prioritize which data to keep and export without SME and research expert input.

Implementation Phase

Conducting research using games or VR is different than simply asking participants to use the software, and several implementation considerations are recommended. Because games or VR requires participants to learn new skills immediately, which includes game mechanics that they may not be familiar with (e.g., commands, buttons, rules), there is an inherent concern for construct-irrelevant variance [19]. That is, their performance within the game or VR (and even their frequency of use) may be influenced in part by their facility and skill in the platform. Construct-irrelevant variance is a known entity in K-12 games [20], but is infrequently addressed in medical simulation. Sources of construct-irrelevant variance include typing speed and skill, equipment quality (e.g., poor quality speakers vs. headphones), familiarity with control pads, familiarity with common game mechanics, or even vertigo with fast-moving VR.

To account for construct-irrelevant variance in serious games and VR research, we strongly recommend the construction of a tutorial that immerses the user with the specific controls and game mechanics necessary for optimal performance, preferably with no hint of the content that is introduced in the proper game or VR. We also recommend collecting tutorial performance data, both to quantify the level of familiarity in the environment as a covariate in data analysis, and to document improvements in successive tutorials as evidence that construct-irrelevant variance is actively minimized in the research. To that end, if the research study requires multiple playthroughs of the game or VR it is possible that simply playing the game will improve performance, as their facility with the controls and environment will gradually improve as a form of maturation bias known as the carryover effect. There are statistical ways to measure and account for carryover effect [21]; however, if the order of gameplay content can be randomized among a larger sample as in a crossover study, that can also attenuate carryover bias.

Because serious games and VR allow research activities to be done in remote areas, including participants’ own homes, the physical environment in which the research activities occur may be varied, adding another source of construct-irrelevant variance. The physical environment includes phenomena like floor space (particularly for VR), distractors (additional people, other electronics, pets), screen size, and processing speed for their own machines. Internet speed connections may also influence game performance. It may be necessary to standardize the physical environment by requiring the study to be completed in a more controlled and consistent setting.

Analysis and Dissemination Phase

Outcome variables common to many healthcare game and VR research studies can include all levels of evaluation, such as satisfaction, knowledge, behavior, and even patient-related outcomes. The allure of data collection using games resides in the large amount of behavioral and performance data available, including time-to-action, choices or selections made, and even pauses or inactive time, which could denote inaction, hesitation, or indecision. Just like any simulation-based research, the research question(s) and methodology must be declared well before the development and implementation of the final product.

Careful attention must be made to the interpretation of game performance. Depending on the game mechanics, navigation of a long menu screen may compound a time-to-critical action variable, for example. Alternatively, a particular branch chain logic that ‘ends’ a game early may not allow a participant to demonstrate all of their accrued knowledge or performance if the Game Over screen appears early. Establishing some correlation of game performance with clinical performance provides validity evidence for the use of the game or VR, and is often the sentinel research plan with a developed game or VR.

Validity evidence for the content and use of a healthcare game or VR is of interest to a variety of parties. Game developers and hardware developers often lack data on non-entertainment products, and any validity evidence within the healthcare organizations can distinguish their products from their competitors. Healthcare educators would also be interested in validity evidence before implementing games or VR into an already busy curriculum. Healthcare networks and patient safety advocates would value validity evidence of games and VR similarly to the way simulation can be used to uncover latent safety threats. Finally, funders and organizations sponsoring the monetary investment in the development of these systems should also recognize the value proposition for these games, as healthcare games and VR do not have the same profitability potential as as games intended for entertainment.

Closing

Serious games and VR are powerful tools that have distinct advantages and disadvantages when used to conduct simulation-based healthcare research. Researchers are advised to select the modality of the simulation (e.g., serious game vs. VR vs. manikin) appropriate to the fidelity of the simulation and the outcomes being investigated, either for simulation as intervention or simulation as environment. Unique elements of performance data capture include developing an information trail for in situ data capture, asynchronous debriefing, and specific user interface surveys already validated in the non-healthcare literature. Because both serious games and VR requires significant upfront development, SMEs and researchers should work very closely with developers to facilitate successful data capture and analyses.