Clinical medical education over the past century has been based on the apprentice system characterized by student–patient encounters in which trainees observe, or sometimes participate in, the care of “private” patients in the practices of clinical instructors or learn by examining and treating “public” patients assigned to them for care under the supervision of clinical instructors. With the development of simulation concepts and technologies, situated or experiential learning in simulation environments is becoming an important component of medical training. This type of training takes the form of learning in a variety of simulated environments ranging from encounters with simulated patients to participation in real or virtual team-based simulated medical activities [1].

Anesthesia crisis resource management

The widely adopted, simulation-based learning environment for anesthesia training is a high-fidelity human simulator, an instrumented manikin running a computational model of physiology and pharmacology representing the patient, and a team of health care personnel managing the simulated medical case—all in a physical space representing an operating room or other dynamic critical care environment. Initial reports of full-size manikins designed for anesthesia [2, 3] and cardiology [4] began to appear during the late 1980s.

The instructional model of anesthesia crisis resource management (ACRM) was based on an adaptation of cockpit resource management (CRM) used in aviation [5]. The training starts with the simulated experience, which is video-taped and annotated by an instructor/facilitator. After the simulation experience, the instructor/facilitator leads a debriefing discussion with the trainees to identify the strengths and weakness of their performance. Most evaluations of the effectiveness of the manikin-based simulation training method have been conducted with resident physicians in anesthesia [6].

Critical care environments beyond anesthesia

The positive experience of team training in anesthesia with ACRM was followed by adaptations to the simulation-based training method in neonatal resuscitation [7] and emergency medicine [8], the latter leading to the emergency medicine crisis resource management (EMCRM) principles. Validation of these simulation systems as effective training methods has focused on both resident [6] and practicing physicians as well as nursing personnel. In a recent extension into obstetric training in the delivery suite, Croft et al. demonstrated in 2005 that manikin-based training of midwives for an uncommon delivery procedure (shoulder dystocia) improves technical performance [9]. Furthermore, using manikin-based simulation in a benchmark transfer study, Draycott et al. [10] documented reduced adverse perinatal outcomes (cranioencephalopathy) in the southeastern region of the United Kingdom, near Bristol. The latter work is unique because it demonstrates the valued, predictive validity for a procedure simulator; patient outcomes were improved in eight hospitals in the district by nearly 50% after the training. A more recent report by Maslovitz et al. confirmed the value of simulation-based training for identifying management mistakes and improving performances in obstetrics [11].

Emergency medicine crisis resource management

Stanford’s EMCRM training program was modeled after ACRM. This 3-year curriculum for emergency medicine residents emphasizes teamwork in the preparation and management of clinical crises. Following an introduction on the concepts of human error and crisis management, the trainees learn the core concepts of EMCRM (Fig. 1). Thereafter, trainees practice their teamwork and patient management skills with a high-fidelity, manikin-based human patient simulator. Crises are introduced at various points in a scenario and may include medical crises (e.g., patient deterioration, cardiac arrest) or situational crises (e.g., ethical dilemmas, environmental emergencies, difficult personal interactions). As crises arise, participants in the “hot seat” are expected to call in a second learner, or “first responder,” to assist with the management of the patient(s) and the crisis. Communication skills and delegation of duties are important components of the “hot seat’s” responsibilities. Instructors in a separate room with a one-way mirror into the “emergency department” control the flow of cases and the responses of the manikins to therapeutic interventions. Instructors are also responsible for capturing the scenario on a video recording and annotating the recording to mark the interactions and decisions that are key learning points. The remaining learners in the group, generally three to five or more individuals, observe the scenario from an observation room using live video. These observers take notes, using the 10 principles as a guideline, which will be used for discussion during the debriefing. Each scenario runs for approximately 20 to 30 minutes and is followed by an instructor-led debriefing. Each learner has the opportunity to be in the “hot seat” and be a “first responder” during each session. Scenarios increase in difficulty as residents are promoted from their first to second to third year of residency, as does the complexity of the crisis attached to the case.

Fig. 1
figure 1

Emergency medicine crisis resource management (EMCRM) rating scale for behavioral performance evaluation

Full size image

Principles of team training

Salas and colleagues have suggested an underlying set of principles concerning teamwork training by [12, 13]. A critical component of these principles is that training in teamwork should include guided hands-on practice with feedback by knowledgeable instructors. Participants must have the opportunity to “deploy” the skills being taught, not just talk about them or see others do them, as with videos. Such “practice” might be accomplished using several types of activity, perhaps some as simple as verbal exercises or role-playing. However, role-playing out of context is often “sterile” and fails to capture the complexity of teamwork situations in the real world. The central strength of simulation is “environments” realistic enough to allow trainees to suspend disbelief.

What are virtual worlds?

These new learning technologies are based on massive, multiplayer, online gaming. They are three-dimensional (3D) virtual environments in which many participants can be electronically connected so they can role-play, communicate, and interact in real time via the Internet. This technology has been used in the gaming industry for several years and is currently being adapted for “serious games” in military and health care training [14]. In virtual worlds, manikins are replaced by computer-modeled virtual patients and environments (Fig. 2) where team members play the roles of learners using their individual computers (Fig. 3).

Fig. 2
figure 2

Screen shot of a virtual emergency department (ED) presenting an injured bicycle rider on a gurney. Time and vital signs appear on the monitor in the upper left corner (after placement of a pulse oximeter and sphygmomanometer). (From http://simtech.stanford.edu)

Full size image
Fig. 3
figure 3

Communication is conducted with a head set linked to the virtual world (VW) operating on a tabletop or laptop computer

Full size image

Virtual worlds for learning and practicing critical care medicine

Virtual reality simulator systems have been developed for anesthesia [15] and trauma [16, 17] management. These early systems enabled individuals to learn and practice problem identification and remediation in CD-ROM computer-assisted learning. Schwid et al. later implemented the Anesoft system as an online software program [15]. This system, designed for crisis management across different disciplines and with trauma and nerve toxin exposures, is supported by a complex set of mathematic algorithms for the pathophysiologic responses to injury and treatment. Similarly, the Research Triangle Institute (RTI) model implemented by Kizakevich et al. is based on the mathematic algorithms developed in The Body Simulation software by Ty Smith [17].

Case studies of training and assessing teams using virtual worlds

At SUMMIT, we have a decade of experience developing and piloting games for learning, including the use of virtual worlds for individual and team training and assessment. Three of these projects are described below.

  • Training emergency department (ED) teams (physicians and nurses) to manage individual trauma cases (Virtual ED I)

  • Prehospital and in-hospital disaster preparedness training for emergency medical technicians (EMTs) and ED staff

  • Training ED and hospital staff (physicians and nurses) to implement a “code triage” (Virtual ED II)

Our aims in these research projects were to:

  • Explore the feasibility of using 3D virtual world technologies for training and assessment of health care teams working in high-stress critical-care areas such as the emergency department

  • Create prototype scenarios using multiplayer online simulation (MOS) and implement the scenarios with representative members of the target audience of learners/trainees

  • Evaluate the MOS exercises to determine their usefulness for training and assessing clinical and teamwork skills

Training ED teams to manage individual trauma cases with virtual ED I

The aims of the study of training ED teams to manage trauma cases with Virtual ED I were as follows.

  • To develop a 3D virtual world MOS for training medical students and emergency medicine (EM) residents on the initial assessment and management of trauma cases

  • To create 10 trauma cases for training and assessing medical students and EM residents using the human patient simulator, a manikin-based system, or the 3D virtual world MOS system

  • To compare learning outcomes and usability ratings of students/residents who were trained with the human patient simulator versus the 3D virtual world simulation systems

Virtual ED I

The software development team created the 3D virtual reality learning environment—an ER acute-care suite with a virtual patient on a gurney (Fig. 2)—using Atmosphere (Adobe Systems). The four ED physician and nurse team member “avatars” were created using Poser software. Finally, the actual trauma patients were created using a low-fidelity, Java-scripted physiologic model so the condition of the patient deteriorates unless the ED team members initiate appropriate treatment. Each team member “avatar” interacts with a patient by selecting actions from a pull-down menu. Once selected, the actions are carried out by the avatar, which makes gestures to indicate activity. In addition, we used Talker, a voice over Internet protocol (VoIP) communication software program, to allow each member of the team to use a headset and microphone to speak to and hear others online. This virtual ED was designed so multiple trainees could take the role of multiple ED team members, with their corresponding “avatars,” and work together using effective team leadership, team monitoring, and communication skills to assess and manage trauma patients received from the EMT. Once the virtual ED was operational, we selected six of the trauma cases to use in the experiment described below.

Trauma cases

The trauma cases were initially created for training with the human patient simulator system by colleagues at the Center for Advanced Simulation at the Karolinska Institutet in Stockholm, Sweden [18]. Six of the available cases were then adapted for the virtual world environment and provided a Java-scripted, low-fidelity pathophysiology model that responded to a few treatments so the team training with each simulation system would be at a comparable level of difficulty.

Comparison of learning outcomes and usability

Thirty students/interns volunteered to participate in the Virtual ED I study. The trainees were either graduating or final-year medical students (finishing their clinical clerkships) or postgraduate year 1 (PGY1) residents just beginning their internship. Using random assignments, 14 were assigned to the human patient simulator (HPS) group and 16 to the virtual worlds (VW) group. The trainees arrived and worked in pairs for each of the training sessions, except when being assessed, as described below.

The training sessions for both the HPS group and the VW ED group were conducted in seven (HPS) or eight (VW ED) half-day (4-hour) sessions in which pairs of students/interns worked with the instructors and members of the research team. Several days prior to the study, participants were sent online instructional materials on the EMCRM principles of teamwork and team leadership and the basics of trauma management and were instructed to review them prior to the study date. Each session began with an initial 15-minute “hands-on” session for trainees to learn to operate the simulation system. During the training cases, pairs of trainees alternated being the team leader or a team member of a four-person ED team. When being the leader, the trainee gave instructions to his or her fellow trainees as well as two “standardized” members of the ED team played by members of the research team. Each patient case scenario lasted approximately 10 minutes and was followed by a 15-minute instructor-led debriefing in which the team received both instructor and peer feedback on their performance.

In addition to the four training cases, trainees were given pretest and posttest cases. For these test cases, each trainee played the role of team leader, and research team members played the other three roles.

During the pretest and posttest cases, three instructor/raters, who were experienced clinicians, observed the trainee’s performance and assigned a score. They used an 11-item rating scale, the EMCRM scale (Fig. 1), to assess the trainees’ behavior as the leader of the ED team. For each item on the scale, the instructor/rater selected the way in which the behavior was performed using the 5-point Likert scale (1 = low; 5 = high). All instructors/raters had been trained to use the EMCRM scale prior to implementation of the study.

After completing the pretest case, all four training cases with follow-up debriefings, and the posttest case, each group of trainees completed the exit questionnaire. They then participated in a focus group discussion to elaborate on their reactions to each of the simulation systems.

Results of the virtual ED I study

Usability and satisfaction

The trainees found both simulation systems easy to use and to have adequate realism for them to “suspend disbelief” and take the exercise seriously. We observed, anecdotally, that trainees were quite anxious prior to beginning each case, as they did not want to lose a patient. Although both groups indicated that they experienced some technical difficulties, most reported that they felt highly “immersed” in the scenarios, as if they were actually there. Most also reported that the study had changed their feelings/perceptions about working as a member or leader of an ED team; their comments indicated a positive change in attitudes. They further indicated that their confidence in their ability to lead an ED team had increased as a result of the training session by at least 1 point on the 5-point Likert scale. They also reported that they thought the simulation exercises would be extremely useful for learning to perform the initial assessment and then to manage trauma patients, as well as for learning to work as a member of an ED team [19].

Team leader performance ratings

The EMCRM ratings for both the HPS and the Virtual ED groups show a significant difference (p = 0.01) between the pretest and the posttest cases in each of these groups (Fig. 4). Moreover, analysis of the difference of the scores between the two groups on each item of the 10-point EMCRM scale plus a global rating show no significant differences in the gain scores between the two groups (Fig. 4).

Fig. 4
figure 4

Results of the Virtual ED I study

Full size image

Lessons learned with the virtual ED I

The results of this study show that team training and assessment are feasible using virtual world technologies, which are more cost-effective than many other methods and can be accomplished easily even when participants are geographically remote from each other and from the instructor. They further show that ED team leadership training using virtual world simulation may be as effective as that done using the human patient simulator [20].

Training prehospital and in-hospital teams to conduct triage after a CBRNE incident

The aims of the second study were as follows.

  • To create two 3D virtual world simulation exercises for training to participate in rapid response to a chemical, biologic, radiologic, nuclear, or highly explosive (CBRNE) incident—one of a local bank where there has been an explosion of a “dirty bomb” (radioactive) and another of the entrance to the hospital where patients arrive by ambulance, by car, or on foot after the release of Sarin, a chemical nerve toxin, on a commuter train

  • To pilot test the two simulation exercises with members of the target audience of learners—EMTs for the prehospital scenario (“dirty bomb” explosion) and hospital physicians and nurses for the in-hospital scenario (Sarin exposure)

Peninsula city

We collaborated with experienced game developers at Forterra Systems (http://www.forterrainc.com) to design the above-mentioned virtual world MOS exercises. Using the online interactive virtual environment (OLIVE) platform, they created part of a city (“Peninsula City”) where an explosion had just occurred at the bank . Avatars were created for both the injured citizens and the EMT team members responding to the call. A triage area was set up nearby with signs for immediate-care, delayed-care, and minor-care victims. The EMT avatars were able to put victims on gurneys, take them through a decontamination tent, classify them by assessing their clinical condition for assigning triage status, take them to the appropriate triage areas, put them in ambulances, and drive them to the hospital. The EMT team member trainees maintained voice communication with each other, using the headset with VoIP.

For the in-hospital simulation exercise, the Forterra group created a photo-realistic version of the emergency entrance, waiting room, and parking lot of Stanford Hospital. They also created a variety of physician and nurse avatars for the simulation who could meet the EMT personnel in the parking lot as they arrived with victims by ambulance, review the victim’s triage status, reclassify them as necessary, and take them through a decontamination tent just outside the entrance to the emergency department (Fig. 5).

Fig. 5
figure 5

Decontamination of chemical, biologic, radiologic, nuclear, or explosive (CBRNE) incident victims in a live disaster drill and in the VW simulation

Full size image

Pilot study

Eight experienced paramedics and EMT team members and eight experienced ED physicians (n = 4) and nurses (n = 4) from Stanford University Hospital were recruited to participate in the pilot study. After learning to use the virtual worlds software, the individuals in each group took the role of an avatar and participated in the simulation exercise. An experienced trainer then facilitated the debriefing session to discuss how well they worked together as a team. At the end of the session, all participants completed a brief user satisfaction survey, and one of the researchers led a focus group discussion to learn what the trainees thought of the new method of training.

Results

In all, 62% thought multiplayer game-based training was as effective as or more effective than traditional methods; 56% said the game environment would be useful for initial training; and 75% said the game environment would be useful for refresher training.

Training ED and hospital staff (physicians and nurses) to implement a “code triage” for mass casualties (virtual ED II)

The aims of the third study were as follows.

  • To create a 3D virtual world replica of the Stanford emergency department, including three treatment areas in the hospital—triage, immediate care, delayed care—for training hospital staff to assess and manage mass casualties after a CBRNE incident (Virtual ED II)

  • To create patient physiology models for 10 Sarin and 10 “dirty bomb” trauma cases for implementation in the Virtual ED II

  • To develop interactive, virtual world MOS scenarios for training hospital staff to implement a “code triage”—assess and manage mass casualties after a CBRNE incident

  • To evaluate the effectiveness of the MOS (Virtual ED II) by conducting a pilot study with trainees from the target audience

Virtual ED II

We continued our development of the first virtual ED to create a more complete emergency department for the new training goals (see below). Using the OLIVE platform, the Forterra Systems group created a photo-realistic replica of the Stanford Hospital emergency driveway and entrance, waiting area, five-bed acute-care suite, delayed treatment area, hospital beds, and equipment (Virtual ED II), as well as a variety of both hospital staff and patient avatars. The floor plan of this learning environment matched the Stanford Hospital ED floor plan, which made it easy for trainees to navigate the learning space.

Patient models

Clinicians on the research team created virtual patient models for both scenarios: exposure to Sarin and trauma injuries from the explosion of a “dirty” (radioactive) bomb blast. Critically ill or injured victim-patients of either type of event—10 with a nerve toxin or 10 with blast trauma, a total of 20 victims—arrive nearly simultaneously for triage and care. The vital signs of these virtual victims, related to the dose of the toxin or the severity of the blast injuries as well as to age, sex, and co-morbid conditions, deteriorate according to pathophysiologic models of asphyxia or hypovolemic shock. The vital signs are also responsive to fluid, blood, and drug therapy and appear in real time on monitors “in-world,” allowing trainees to make decisions about clinical management. A pop-up interface for interacting with each of these high-fidelity human simulator patient models, written in C++ as an application on OLIVE, presents text answers to queries, sketches of examination results, and reports from diagnostic procedures (http://summit.stanford.edu/research/VEDII.html).

Training and assessment with the virtual ED II

The learning goals for the expanded emergency department are summarized below. Physicians and nurses from the emergency department and elsewhere in the hospital will learn to:

  • Perform the roles of team leader and team member in the following areas: triage and immediate- and delayed-treatment areas

  • Implement the procedures described in the disaster action guidelines that have been created for a “code triage,” the hospital’s code for a mass casualty incident

  • Assess and manage patients who have been exposed to Sarin aerosol spray or who have suffered injuries from a “dirty bomb” explosion

The physician in the role of treatment areas supervisor (ED Attending) will be able to:

  • Assign roles to physicians and nurses responding to the code triage

  • Procure necessary resources for the teams

  • Communicate with other groups outside the emergency department

  • Monitor the teams in each of the areas where patients are being assessed and treated

Two basic scenarios were developed to provide practice in achieving these goals: one for the Sarin exposure incident and one for the “dirty bomb” explosion. Whether the learning goals are achieved is assessed by the instructor during the exercises, and feedback is provided to trainees during the debriefing discussion. An overall assessment is accomplished by capturing digital, “in-world” recordings of the performance of individuals and teams in each of the treatment areas during each scenario and reviewing them at a later time with performance assessment tools. In addition, the logfile data of the individual actions of each trainee were captured for review at a later time.

Virtual ED II evaluation study

Seven physicians and six nurses who were employees of Stanford University Hospital volunteered to participate in the Virtual ED II study. Two of the physicians were EM residents, and five were surgery residents. All six nurses were from the emergency department. We conducted evaluations on 2 days with two different groups of physicians and nurses on each day. Figure 6 depicts a virtual emergency department with a victim on a gurney and an avatar approaching to evaluate the virtual patient.

We followed a procedure for this pilot study similar to the one used for the previous studies. After an introduction to the study and 30 minutes of hands-on training in how to operate the software, the physicians and nurses began the first scenario—taking care of virtual patients who had been exposed to Sarin. After receiving a message that Sarin victims would be arriving momentarily, they began by selecting a leader. The leader then assigned roles—one team of a physician and nurse for the triage area and the remainder for the immediate-treatment area. These teams moved their avatars to the appropriate areas and began to receive the incoming virtual patients. The triage team assessed each victim and sent him or her to the appropriate treatment area—immediate, expectant, delayed, or minor treatment. The “immediate” team divided themselves into physician-nurse teams at each bedside where they reassessed and managed each virtual patient they received. They were able to review each patient’s condition by reading a description and seeing a graphic representation of the patient’s injuries. They were able to elicit vital signs, which appeared on the patient’s monitor, start intravenous infusions, and administer blood and drugs chosen from a formulary. Once the team had stabilized a patient, it requested a transport aide to take the patient to surgery, admitting, or the morgue.

After completing the Virtual ED II exercises, an ED instructor facilitated a debriefing discussion. We then asked the participants to conduct the exercise a second time, giving the trainees the opportunity to learn from earlier mistakes. After the second debriefing, we asked them to complete surveys and contribute to the open discussion of what they thought of the new training method.

Results

Most of the 13 volunteer subjects were not gamers (69% never played games), and most (62%) had had no prior training in responding to a mass casualty CBRNE incident. In all, 62% reported that the session changed their feelings/attitudes about working as a member or leader of an ED team. Their ratings on the exit questionnaire (5-point Likert-type scales: 1 = low; 5 = high) showed that they felt immersed (M = 3.47) and thought that the session increased their confidence in their ability to respond to a CBRNE incident (M = 2.00 before training; M = 3.08 after training). Most also thought that the simulation exercises would be useful for learning teamwork skills/behaviors (M = 3.77) as well as for learning the clinical skills necessary to treat CBRNE victims (M = 3.15). Their comments indicated that they also perceived the patient physiology models and virtual environment as realistic, although they would like the interface improved to allow them to perform a more rapid patient assessment and to get more of a “big picture” of the situation in the treatment area.

Lessons learned from the virtual ED II study

With the new Virtual ED II we were able to demonstrate that hospital staff can learn to respond appropriately to a “code triage” for mass casualties resulting from either Sarin exposure or the explosion of a “dirty bomb.” Specifically, the trainees are able to:

  • Work together in two-person teams to assess/triage incoming patients and treat both Sarin or “dirty bomb” patients

  • Work together in two-person teams in the triage area to assess/triage incoming patients who arrive by ambulance, by car, or on foot

  • Work together as a larger team, led by the treatment areas supervisor (ED Attending), to use resources wisely in the event of a mass casualty incident

  • Follow the procedures of a “code triage” as specified in the Stanford Disaster Action Guide

  • Communicate effectively with each other as members of a team

  • Coordinate duties as members of a team

In addition, we were able to see that teamwork can be assessed at several levels. With the recording and playback capability of the OLIVE platform, instructors can observe and rate individual performance, as well as the work of the team collectively, in each of the treatment areas. Instructors can review the logfile data for individuals to further analyze their actions during the Code. Finally, metrics derived from patient outcome data—e.g., the number treated appropriately; the time taken to stabilize the patient; the number who died—can be used as another measure of team effectiveness in each of the treatment areas.

Discussion

The authors recognize that these studies concern team training in emergency medicine and are not for surgical teams; however, we assert that the virtual world concepts and methods are applicable to both—indeed, probably to most critical-care areas in medicine and nursing. The key technology issue is identifying and using a software platform onto which an application can easily be built. Several single-player platforms are accessible, but few platforms exist onto which multiplayer online simulations can be developed. Our multiple virtual patients, with their pathophysiologic models operating simultaneously, placed significant performance demands even on the OLIVE platform.

Although other simulation methods—such as manikin-based surgical team training, CD-ROM patient case simulations, disaster drills, and tabletop exercises—will continue to be useful methods for learning to manage critical care medical situations, there are distinct advantages for using the virtual worlds model as a complement to those methods. With virtual worlds training:

  • The exact layout and location of resources of specific EDs can be replicated.

  • Trainees do not have to be present at the same location to play their avatar roles.

  • Drills can be conducted at any time of day or night to accommodate trainees’ schedules.

  • A variety of patient scenarios and conditions can be modeled to simulate individual patient cases or multiple victims in complex clinical situations.

  • Dangerous and/or infrequently occurring situations can be presented.

  • Scenarios can be run more than once during a short period of time, allowing trainees to learn from their mistakes.

  • Trainees’ performance during the simulation can be captured for playback and assessment after the event.

New tools for team assessment are needed

Assessment methods used for manikin-based simulation training and virtual world training studies were developed initially to assess the individual’s performance. They have now been extended to assess team performance. As multiple simultaneous teams interact in the virtual world, observer-based rating methods need to be augmented with other objective, automated measurement tools. Training in these more complex simulation environments requires new assessment tools to measure individual and team processes as well as patient outcome variables.