Keywords

1 Introduction

1.1 Awareness of Healthcare Quality

In healthcare, 54% of surgical adverse events occurring in industrialized countries are considered as avoidable events [11]. The operating room represents the highest risk for the patient, as 65% of adverse events in healthcare are related to surgery [33]. It is a complex environment [9]: (i) different disciplines, expertise and cultures coexist within the team (ii) the operators cooperate for patient care and deal with unanticipated events, (iii) the operator’s interactions are non linear and often unpredictable, (iv) humans interact with each others and with technical objects or computerised systems which deliver technical information, (v) the state of the system changes and evolves over time. A dozen of dimensions of complexity in health care are described by Carayon [7], Plesk and Greenhald [28] and Effken [9]. The composition of the team is heterogeneous and each team member has their own technical skills and responsibilities. There are multiple interactions that influence the evolution of the system but a successful operation depends on what information is dynamically exchanged [27] to understand what is going on. Different reports note that; most of the time, a communication default is the root cause or contributing factor to an adverse event [13, 16, 18, 22, 26]. Wrong surgery site (WSS), wrong patient events or wrong procedure are often reported [5, 31] although they appear in 1.7 to 3.6 events out of 100,000 operations [31]. In 2009, the World Alliance of Patient Safety project, launched in 2004 by the World Health Organization (WHO), published a list of recommendations and security checklists to prevent adverse events in operating theaters during surgical procedures [32]. More recent studies shows how errors result from misinformation (e.g., incorrect information obtained from other departments) and misperception (e.g. from right-left confusion when interpreting imaging results [3]). The median prevalence estimate for wrong site surgery was 0.09 events per 10,000 surgical procedures [3]. The WHO checklist displays 3 columns that represent the three phases of surgery: (i) from the patient’s arrival to the induction of anesthesia, (ii) from patient’s induction to skin incision and (iii) from skin incision until the end of the operation. Studying complex systems, Reason [30] shows that most of the time, accidents result from multiple successive failures which could not have been corrected or stopped in time. The WHO checklist aims to build different barriers to prevent certain types of errors that tend to be committed in each of these three stages [4]. Haynes et al. [14] showed that the use of the checklist significantly lowers surgical morbidity and mortality.

The recommendation imposes to identify a coordinator who is responsible to manage the safety procedure. In practice, the role of checklist manager can be attributed to anyone. It depends on clinical services and hospitals. The role of the checklist manager consists in checking information from different sources, on different topics and making cross-control. In case of doubt, they can stop the surgery process or ask for help. Fudickar et al. [2] show the effect of the WHO Surgical Checklist on communication. “The checklist should be understood not merely as a list of items to be checked off, but as an instrument for the improvement of communication, teamwork, and safety culture in the operating room, and it should be implemented accordingly”. Yet, very few specific courses exist to help professionals and students to learn risk management and non-technical skills. In many hospitals worldwide, simulation centers have been created for healthcare education. In most cases, they replicate different medical places as the operating room or the patients room. They focus mainly on technical skills and aim at reducing the gap between what students learn in textbooks and gestures they are expected to use in the real professional world.

The next section describes the learning game 3D Virtual Operating Room, which has been designed to train inter-professional teams to the specific context of the operating theater.

1.2 3D Virtual Operating Room

3D Virtual Operating Room [21] (3DVOR) is a representation of the operating theater under the shape of a collaborative virtual environment. 3DVOR is a real-time multiplayer virtual environment dedicated to train and prevent risk management inside the operating room. It is focused mainly on near-miss or standard situations that can be failed due to communication defaults. 3DVOR offers collaborative training for all operating theater professionals including anesthetists, surgeons, nurses and health managers. This research has been initiated by the scientific interest group named Serious Game Research Network and the University Hospital of Toulouse (France).

The universe of the virtual operating room is composed of avatars which represents medical staff, nurse staff, a patient and technical equipment: anesthesia machine, electric generator for the scalpel, surgical aspiration system, etc.

This environment was specifically designed to be used in a learning context. It provides features dedicated for the trainer for them to follow in real time what is going on within the virtual operating room and interfere with students strategies.

It was designed as a combination of standard game design mechanics and an innovative system of interaction metaphors to reproduce teamwork. Each student plays a different role to compose a virtual medical team. Any player can see on the game screen what he is doing and what the other team members are doing, for example controlling arterial blood pressure, placing a catheter, injecting a drug, etc. The details of the model enabling the real-time collaboration of several players within the game is described in [25].

Graphical interactions allow each player to collect, memorize, listen and broadcast information [29]. They can also ask questions and give answers thanks to information tags stored in their virtual memory. A voting system is available to debate and vote on predefined topics.

The virtual universe is represented by a set of objects as technical equipments, documents and avatars. In a point and click fashion, each player moves in a 3D scene, displays different menus of actions and selects the action he wants to do on a specific object. Each action is associated with an object in to universe. Some action reveal information that is automatically collected and stored within the player’s virtual memory. According to what is done or known by the players, the state of the environment is changing dynamically. The application monitors every actions, communication, discussion and decision making that the team is doing. In such digital virtual environment, the player seems to be free to act and communicate with their teammates. A large variety of possibilities can be explored and many paths can lead to different levels of success or to different failures. But actions are gradually unlocked as the player accomplishes tasks in the game. And some indicators are displayed to inform the team on their current grade.

Educational situations are designed to train the team to anticipate failures, identify, reduce or correct mistakes, evaluate the root causes, consider the situation and take appropriate actions. Among this educational situations, some of them focus on a particular topic: the use of the Surgical Safety Checklist.

2 Purpose and Goal

This article describes a model to design individual’s goals and team’s goals in a virtual collaborative environment. The embedded pedagogical content targets risk management education within the operating room. It is based on real adverse events where a communication default was identified as a contributing factor. As learning is a process which is constantly modified by experience [19], involving a team of learners in a virtual near-miss context to observe teamwork, professional and informational behaviors should bring real benefits. At the end, as the educational content is fully mastered by the trainer and entirely controlled by the game engine, it is supposed to identify the mistakes and evaluate the miscellaneous causes of a near-miss. Learners being confronted with a professional unpredictable situation in which they have been wronged are likely to be aware of the consequences of their actions. It should help to learn non-technical skills as leadership, situation awareness and decision making.

This paper presents how the educational objectives were designed and proposes to check if the students succeed to manage first a professional standardized situation an then an unpredictable situation. We study how each learner interacts with their virtual partners and whether they as a team find a way to reduce the risks in this unpredictable educational situation. The present study will focus on how an semi-automatic debriefing can be produced both for learners and their trainer based on a virtual adverse event within the scope of training in the management and prevention of surgical risks.

3 State of the Art

3.1 Virtual Environments for Learning

Learning environments are designed to support learning as the construction of knowledge in learners. Learning takes place by changing the settings of the simulated environment and observing the consequences of actions performed by the student. Simulation is now increasingly used in many domains (scientific, medical, marketing, etc.). Many benefits are highlighted in the literature; we can mention here the simplification of reality to facilitate understanding. For De Jong [17], the reasons for the learning attraction of simulation are increased motivation, better understanding of the phenomena, greater ability to adapt to similar problems in other contexts. McGaghie et al. [24] list thanks to a systematic review 12 features and best practices of simulation based learning in medical education. Among them, feedback, curriculum integration, outcome measurement, skill acquisition and maintenance are mentioned. Compliance with Mc Gaghie et al. practices leads us to define a model educational design in a virtual environment for evaluate a set of cognitive and behavioral objectives.

3.2 Multiple Users in Real Time

In a virtual environment, each users is represented by an avatar and, in addition to figuring where he is, each player needs to understand where are the others and what they are actually doing. Many features of a 3D scene participate to better comprehending the actions and states of mind of other users: graphical representation of the avatar’s actions, player’s natural speech, maps of the virtual scene, player’s chat conversation, avatar’s presence, gestures, or facial and body animation, emotions modelling. Capin et al. [6] list crucial functions in addition to those of single-user virtual environments:(i) perception (to see if anyone is around) (ii) localization (to see the others) (iii) identification (to recognize who i am), (iv) visualization of others’ interest focus.

To allow users having coherent dialogues, main features must be provided and they have to respect implicit conversation rules. To be coherent, conversation generally follows implicit rules as choice of a common conversation topic, choice of the listeners and turn-talking rules. In 1970’s, Grice [12] argued that people in conversation must be cooperative. Speakers must try to “make their contribution such as is required, at the stage at which it occurs, by the accepted purpose or direction of the talk exchange in which they are engaged”. Pons-Lelardeux et al. list main functions to represent dialogue and decision making in a multi-player environment. Their communication system proposes several features to allow several player’s conversing and making decision. Their system is based on the implicit rules of real conversation: (i) perception (to memorize the current contextual information) (ii) identification of the speaker (iii) topic of the conversation (iv)everyone’s opinion (v)visualization of the final leader’s decision.

3.3 Educational Design in a Virtual Environment

Allowing for joint activity in a virtual environment is not a sufficient condition for experiential learning to happen. Instructional design is the science of designing pedagogical experiences (in a virtual environment in this context) where the activity of one or several users is planned, observed and evaluated against a set of objectives defined beforehand. IMS-LD is a well-known educational modeling language [20] for designing such experiences. It is based on the metaphor of a play where roles and acts are related to users and pedagogical scenarios. It is closed to the UML modelling language universally in use in computer science. Maroto [23] has developed a module for deploying and executing a IMS-LD scenario onto the virtual environment Open Wonderland.

Learner tracking is another necessary step for controlling the experience of the user. Intelligent tutoring system (ITS) is the term used for describing a system where the actions and interactions of each user are collected for dynamically adjusting the experience or for further analysis (debriefing). Usually, an ITS is composed of many modules communicating with each other and dedicated to one function: activity module, error module, learner module, pedagogical module, etc. Eventually it takes the form of a virtual tutor embedded in the environment (like STEVE the pedagogical agent [15]) or remains invisible to the users, like an informed environment.

Finally, the evaluation takes place on the basis of the activity tracked online. The challenge, known as task tracking or plan recognition [8, 10], consists in reconstructing the meaning of the task from the low-level activity of the users so as to enable comparison with expected behaviours modelled in the scenarios. The desired outcome of this process is not only to ensure that the scenario has been traversed by the learners as expected, but also to point out the errors and misconducts along the way. Explicability is the most important criteria of the whole process, as the main leverage for learners to actually acknowledge their errors and thus facilitate the learning.

4 A Model to Design Collaborative Educational Objectives Including Communication and Action

Szyld and Rudolph [1] define debriefing in healthcare simulation as “the learning conversation that follows a simulation session. The instructors role in providing feedback and guiding reflection is critical to ensure that reflecting on the simulation experience yields learning and growth in accordance with the stated educational goals of the session”. Therefore, the trainer needs to see some cues to show an effective debriefing. The tutoring system presented here should help the trainees to understand their errors and the trainer to build the debriefing irrespective of their experience. The model described in Sect. 4 is used to present an automatic result of success or failure to the students at the end of the training session.

4.1 Different Kinds of Objectives

In real life or in training context, understanding how an adverse event has happened is crucial to improve behavior facing to a standardized or an unpredictable situation. To help practitioners and get them to commit to a rigorous approach, the National Authority for Health (HAS) recommends the ALARM (Association of Litigation And Risk Management) systemic analysis method. It proposes a systemic approach to complex systems, which includes 5 stages: (i) data collection, (ii) reconstituting the chronology of the event, (iii) identifying shortcomings in care (defined in relation to standards for good practices), (iv) identifying their causes (contributory and/or influential factors) and (v) proposing measures for improvement. Our model uses the ALARM method to collect, store and identify causes of success or failure, in order to display some recommendation to improve the team’s performance.

At the beginning of a game session, a briefing is displayed to inform the team on the patient’s pathology and the scenario’s expectations. The main objectives mentioned at the briefing present a general context but the specific risks the team has to managed are not mentioned. Therefore, some educational objectives are displayed and others are hidden in order not to affect the behavior of the trainees.

On one hand, in order to assess the performance of the students, the model embeds a set of metrics to measure how well the standard procedures are applied and how the team of students reacts when they are facing to an unpredictable situation. On the other hand, others objectives are used to divide the scenario into small steps and inform the students on their progress during the game session. As a result, different types of objectives compose a game scenario:

  • step objectives to inform on the level of progression in the scenario [visible]

  • educational objectives that are not visible to the trainees but are monitored by the game [invisible]. There are two sub-types of these:

    • objectives of success (expected outcomes) to inform on what was correct to reduce risks

    • objectives of cause of failure (predictable failures) to inform on what increased a particular risk

Educational objectives have to be designed as part of the scenario and must be checked in real time by the game. This allows to provide an automatized and personalized debriefing based on the activity during the game session. But the application needs to be able to understand what is the objective and how it can be evaluated. Most of the time, applications are not able to evaluate events which can not be listened by the game. So, all the macro-objectives have to be composed of micro elements that can be listened and captured on the GUI. Then, all the micro elements need to be associated with a particular grammar to construct macro-objectives.

Unit elements (Table 1) are events that can be observed and automatically captured by the game during the game session. Unit elements are actions, information acquisition or transmission, discussion and decision making. They can be considered as micro-objectives and therefore they must be associated in order to construct more meaningful game objectives.

Table 1. Element types necessary for defining an objective
Full size table

Complex objective are constructed by the tree-like association of unit elements. Operator nodes are introduced in the pedagogical description grammar in order to do so. Table 2 lists the available operators. Owing to the tree-like recursive description model, the expressiveness is potentially unlimited, although in practice only a few layers are necessary for a complex objective to be defined (see Fig. 1). The ORDER operator allows designers to define an objective with a set of basic elements that should to be done in a specific order. For example, to test ’infectious outcome’, the application needs to know if the player first washes their hands, then puts their gloves and injects drugs. If the user first injects drugs then puts their gloves, the infectious risk is very high. So the application needs to store the chronology of what happened. The order between actions and communication is also important especially for the surgical security checklist. Before ticking the checklist to confirm the patient’s identity, the checklist leader has to collect all the information about the patient’s identity from all the team members. The operators help designer to combine different objectives to build new complex objective.

Table 2. A grammar to combine objectives
Full size table

The same model is used for representing the expected outcomes of the game as well as the unexpected, yet predictable, errors. For example, the checklist manager ticks the box to confirm the patient’s identity whereas they did not make a cross-control of information on patient’s identity. They have to check from the patient their identity and check on patient record if the same identity is present on any document. The main regular error consists in confirming the patient’s identity without any cross-control. Therefore, the same actions can be bound to a success objective as well as a failure. The success objective consists in (i) reading or collecting patient’s identity information from the others on any documents in the medical record, (ii) discussing and making a decision to confirm the patient’s identity and then (iii) ticking the box on the checklist to confirm the patient’s identity. Ticking the same checkbox without any prior cross-control or collaborative discussion is considered as a failure in the procedure and therefore a failure objective.

All the objectives for a scenario (success, failure and level of progression) are likely to be presented in a large variety of tree-like structures where nodes represent objectives and leafs represent expected action, communication and decision. At the beginning, the application initializes all objectives with a Boolean value “false”. While students interact with the virtual scene, the engine listens every events and checks if some objectives are fulfilled either by the individuals or by the team. When an objective is reached, the engine converts the value of the objective to “true”. Step by step, objectives are reached or not and the risks may increase until the training session is stopped. At the end of the training session, the application is able to display the main outcomes.

This primary objective overlaps with educational objectives linked to the current damaged situation. The schema in Fig. 1 examplifies an educational macro-objective that combines different expected unit elements associated with operators.

Fig. 1.
figure 1

The educational objectives can be represented in a tree-like structure where nodes represent objectives and leafs represent action or communication.

Full size image

5 Experiment

5.1 Context

This environment was experimented at the University Hospital of Toulouse with the participation of last-year undergraduate students from the Anesthetist Nurse School of Toulouse and the Operating Nurse School of Toulouse. The experiment was separated in two steps. The first part of the experiment was dedicated to familiarize the students to the virtual environment. The second part of the experiment consisted in confronting them with an unpredictable situation.

8 teams composed of 3 students were involved in these experimental training sessions. Each student plays their role as a professional in the virtual operating room. All the students had already worked in a real operating room during a professional internship. Like any medical simulation, the experiment included a succession of three stages: (i) a briefing, during which was explained how the session would be run together with an introduction to the training objectives; (ii) the activity itself (or game stage) during which the participants played out the scenario specially designed to work to the teaching objectives relating to the established curriculum; and (iii) the debriefing.

5.2 Experimental Protocol

The experimental protocol included 4 steps: Firstly, the teacher made a briefing on the virtual environment and its main features and explained how to play the game. Natural conversation between learners was banned during the game. Everyone have to use the communication system provided by the virtual environment to exchange opinions and information. Secondly, the learners tried to familiarize themselves with the virtual environment, particularly how to talk to the patient, how to move to the computer which displays the MRI, how to discuss with the other teammates, etc. They used first a scenario presenting a standardized situation without any difficulties nor traps. Two students composed the first team in the standardized situation while the third one played the same scenario in parallel with a virtual non-playing character. At the end of this step, a debriefing was brought to recap the different features and their usage. Thirdly, the team of 3 students tried to manage a new professional situation presenting some irregularities. Fourthly, at the end of the session, an automatized and personalized debriefing was produced to support the trainer and provide feedback to students on what risks were mastered and what should have been done to reduce risks. During all training sessions, computer data, video of training sessions were recorded.

5.3 Two Training Situations

Two training situations have been designed for this experiment to train people on the patient security checklist which is supposed to be used to prevent wrong patient error and wrong site surgery. The educational context is based on the first phase of a surgery, that is the first column of the checklist: from the patient’s arrival to the induction of anesthesia. The three first checklist’s items are concerned, listed in Table 3.

Table 3. The first 3 questions of the patient safety checklist.
Full size table

The first situation is based on a standardized professional situation and the second one is a non standardized situation that contains multiple anomalies. Both present a patient who has a cerebral tumor. Depending on the size of the tumor, the patient is supposed to be able to talk or not. Students are expected to identify the anomalies, exchange on them, take appropriate decisions before they fill the checklist’s items and move the patient to the operating room. The second educational context imposes to adapt the security rules when the team is facing to non-standardized situation (i.e.: with an unpredictable anomaly). For example: the patient cannot state his name, information is missing on the medical record, etc.

The scenarios are both divided into 3 steps:

  1. 1.

    Verifying patient’s identity

  2. 2.

    Verifying patient’s operating site

  3. 3.

    Move the patient to the operating theater

And the 3 main educational objectives are: (i) Reducing the risk of patient’s identity error applying the security checklist (ii) Reducing the risk of patient’s wrong site applying the security checklist (iii) Adapt the procedure to a specific and near-miss context if they identify some anomalies.

6 Results

The question of debriefing in the context of an operating theater is a delicate one as it relates to the complexity of the system. Indeed, teamwork in the operating room cannot be summarized as the straightforward coexistence between technically competent individuals. The operating theater can be considered to be a complex system since it functions in a dynamic and uncertain environment, with the professionals concerned maintaining among themselves relations that can be both hierarchical and complementary around a shared goal of dispensing optimum care for the person being operated on. Each surgical team and each of its members have specific skills and knowledge. The system of objectives presented here tries to take into account both each member’s capabilities in carrying out their tasks and the ability of the team to ensure precise co-ordination.

6.1 Step Objectives

Among the 20 training sessions, 10 sessions focused on the same standardized situation and 10 sessions focused on the same near-miss situation. Each training session lasted in average 2 h, during which 1 h was actually spent playing the game. Among all the training sessions, most teams succeeded to reach the step objective “Move the patient to the operating theater” in the standardized situation whereas they failed to managed the near-miss situation (see Table 4).

Table 4. Synthesis of reached step objectives
Full size table

6.2 Educational Objectives

The first objective consists in evaluating if the teams managed to reduce the wrong patient error risk applying the checklist. The second one consists in evaluating if the teams managed to reduce the wrong site risk applying the checklist. Tables 5 and 6 show the number of times the team completed the main objectives.

Table 5. Educational objective: “Avoid the patient’s identity error risk” - Number of times the teams succeeded
Full size table
Table 6. Educational objective “Avoid the wrong site error risk” - Number of times the teams succeeded
Full size table

Even though most of the time teams failed to reduce the wrong site risk applying the checklist, just under half of the teams checked little more than half the micro-objectives (see Table 7).

Table 7. Success rate for the objective “Avoid the wrong site risk”. Half of the teams were not able to complete any objective whereas other teams managed to complete partially (or entirely) the game objectives.
Full size table

The most common error made by the teams playing to the near-miss situation focused on the item “Is the patient’s operating site confirmed?”. Most of the time, this item has not been checked. Table 8 lists the main errors stored by the system.

Table 8. Frequency of failures for the educational objective “Avoid the wrong site risk”.
Full size table

Among the teams that were able to fulfill a part of the objective “avoid the wrong site surgery risk”, 4 teams succeeded with the following activities: (i) the surgeon player had examined the patient’s motor function and the patient’s communication ability to identify the operating site, and (ii) the surgeon player had checked the surgery site on the MRI. Yet, only one surgeon player has sent the crucial information to the checklist manager.

7 Conclusion

The collaborative virtual environment featured in 3DVOR simulates teamwork in the operating room in different real-life professional contexts. This environment was designed to be used in a learning context. We have presented a model designed to specify, record and store different kinds of objectives: “step objectives, shown to the students for them to get their bearings in the scenario, and; “failure- or “success-objectives, hidden to the students yet allowing for identifying what was missed in the scenario. Objectives are combined in a tree-like structure with a specific grammar that helps to build, out of simple contextual actions, conversations or decisions, meaningful team objectives that enable to recap their behavior and the expected outcomes. This model was used to specify educational objectives connected to the WHO surgical checklist. The behavior of students has been recorded individually and collectively during 20 training sessions and the model has successfully been proven able to reveal their successes and failures, and therefore to evaluate their teamwork efficiency in critical situations. Future work aims to design dynamic information as blood pressure that change dynamically during the surgery to build more complex scenario and its educational objectives that take in account the temporality of events.