Keywords

1 Introduction

Serious games help to teach people about a certain subject, expand concepts, reinforce development, or assist students in skills acquisition or change their behavior. They are expected to contribute to deeper and more active education, through which students learn from their own experiences [1, 19]. Thus, serious games are essential tools for delivering a message, teaching a lesson, or gaining knowledge supported by interactivity, motivation, and engagement.

In recent years, many reports and experts point the importance to design educational environments and educational programs to reproduce with high fidelity the professional environment [6, 12, 23]. Thus, Flin et al. [9] emphasize the significance of non-technical skills that are not directly linked to anesthetist’s technical expertise. Non-technical skills are divided into interpersonal and cognitive. Interpersonal skills as communication, leadership, coordination are skills that make teamwork effective to reach a common goal [18]. Cognitive skills include task management, situation awareness, and decision making.

Hence, elaborating inter-professional simulations in the healthcare field is a perfect way to train medical specialists. Simulation is a technique that enables to replace or amplify real experiences with guided experiences that evoke or replicate substantial aspects of the real world in a fully interactive manner [11]. A simulation, in terms of healthcare training, is used to provide a safe environment for education without any real risk of accident or disease for the patient. Practically, modern simulation in healthcare education corresponds to using equipment, sometimes a computer software, a mixed reality system “which is a combination of both real and virtual" or a standardized patient for replicating a medical environment and/or a clinic situation and/or a specific pathology.

In this work, we consider dataset which was collected from game-based simulation for students from Nursing Schools, which is called Clinical Organizer Nurse Education (CLONE). The game aims at training students to organize their daily schedule, to delegate specific tasks to the nurse assistant, in order to educate professional nurses on scheduling skills, situation awareness, and decision-making [20]. The study demonstrates the analysis of data with implementation of statistical tools. We intended to find out factors that might help to predict game success or to display in-game feedback through analysis of game sessions duration, errors which students committed, and characteristics of repeated sessions.

2 Research Frameworks

2.1 Related Work

As many benefits of using games as an educational environment exist, a vast variety of applications were developed in past decades [5]. Serious games have various shapes, such as simulations or queries, and domains, such as well-being, cultural heritage, and even healthcare [17]. The importance of designing educational environments and educational program is marked by healthcare professional experts and trainers. Therefore, they intend to create an experiential learning environment, which helps students to develop inter-professional skills [6]. The majority of digital training in healthcare is designed for medical specialists and focused on technical skills and surgery [13]. Some of them are designed to train caregiving teams to improve coordination and efficiency at their operative unit [7, 16]. Others simulate an operating room to train the inter-professional team activity [14, 21].

The interaction students with virtual training environments gathers data, that might reflect important information about the learning process. Therefore, the methods of Game Learning Analytics (GLA) has been evolved in recent years. The game system usually represents data in a form of log files that determine records of events in the system. Extraction and prepossessing of meaningful data for further analysis is a challenging task related to the purpose of the research. Depending on the application purpose GLA methods study performance [2, 15], in-game players behaviors [4], student profiles [8], and motivation [10, 24]. Another challenging task is to find appropriate techniques for data analysis. In this context, GLA can be considered as an area which exploits a set of various methods such as machine learning, information retrieval, statistics, sequential mining, and visualization, in order to enhance learning experiences [3].

This work represents an analysis of data gathered from the virtual training environment CLONE, which is designed to support teachers in the Nursing Schools. We implement the methods of GLA to examine features of game sessions, which were played by students.

2.2 CLONE

The project CLONE (Clinical Organizer Nurse Education) is a Real-time Digital Virtual Environment for Training of nursing students in real-life-like professional situations. It proposes a large library of educational cases where a nurse-student plays the role of a regular nurse. This digital environment includes game mechanisms and interactive features such as a scheduling system, a task shifting, and a decision-making system.

The player chooses a game scenario from the library of educational cases, which proposes a brief description of the actual and expected situation. A scenario provides interactions that allow the players to complete the mission, it is composed of locks (educational locks or playful locks) to prevent the player to succeed. Hence, outcomes are compared to expected objectives, and results of the game are immediately displayed at the dashboard. Each proposed scenario contains a certain set of patients, unpredictable events, medical dynamic events, and the actors involved in the medical process. The player has to deal with patients’ diseases (to provide care) and to deal with the nurse assistant (to share the work). To complete the mission, the medical staff (nurse and nurse assistant) has to provide the required care for each patient according to their medical profiles. The designing process of CLONE contained three steps: (i) the domain analysis, (ii) the human activity modeling, and (iii) the scenario. A more detailed description of the designing process is exposed in [22]. The graphical user interface is presented in Fig. 1.

Fig. 1.
figure 1

The graphical user interface

Full size image

The game process contains the following steps: the briefing, the communication with the night shift, the scheduling, the activity, and the communication with the afternoon shift with debriefing. Firstly, a student reads information about the mission of the game; then at the communication step a student receives information from the night shift about the current situation when they shift at 6:30 am; afterward, a student inspects patients’ records and organizes the daily activity; at the activity step a student provides care, organizes medical examination, professional phone calls, patient discharge or arrival; and finally - informs the next staff about the current situation when they shift at 1:30 pm.

2.3 Research Questions

To investigate the factors affecting the game success, this study addresses the following research questions:

  • RQ1: Does the duration of the game session impact on gaming outcomes?

  • RQ2: Which errors do students commit during the game?

  • RQ3: Do students achieve learning progress by repeating the game?

In responding to these research questions we aim at helping domain experts to understand learners and improve the learning process by provision data-based decision-making.

3 Methodology

To explore stated above research questions empirically, we gathered data for 353 game sessions which were played by 222 students from 11 Nursing Schools during the 2018–2020 time period. Students played according to the same scenario, which includes 5 patients, who require a low-level of care. The game proposes short storytelling of what is the actual situation and what is the expected situation at the end. The current situation starts at 6:30 am and stops at 2:00 pm. The main goal aims at elaborating a care plan for all the patients, delivering care and managing potentially hazardous situations. Students should carefully inspect patients’ profiles, doctor prescriptions, and develop personalized care plan for all patients (except one, who is expected to arrive later in the morning). Then, CLONE provides interactions that allow the players to complete the mission avoiding locks (educational locks or playful locks), which prevent the player to succeed. Eventually, outcomes are compared to expected objectives, and performance indicators are immediately displayed at the end of a game session.

A list of established constraints (e.g. delivering drug, and taking blood sample) is attached to each patient. To complete a mission students have to respect to them. In case of discrepancy, the student accumulates errors: risk errors – break of soft constraints which do not have an important impact on the patient’s health, and critical errors - break of hard constraints that strongly affect the patient’s health. If the student exceeds the maximal number of allowed critical errors at least for one patient - the game is over.

4 Results

4.1 RQ1: Does the Duration of the Game Session Impact on Gaming Outcomes?

Answering RQ1, we investigated both the time of game sessions and the number of actions committed per session. The average duration of the game session according to the scenario with 5 patients is 46 min, while the average number of actions per game session is 175. Hereby, we compared these game session features depending on their results. Among 353 game sessions, 261 are lost and 92 are successful. In order to find out the difference between them we examined 2 hypotheses:

  • H1. The success of the session depends on the duration of the session.

  • H2. The success of the session depends on the number of actions, which were made by the player during the session.

To check these hypotheses, we applied non-parametric test due to a significant difference between analyzed indicators and normal distribution. According to Mann-Whitney U test for independent variables, the duration of the successful sessions is significantly higher (p < 0,0005) than the duration of lost games (Left panel in Fig. 2). Meanwhile, the number of actions for successful sessions is significantly higher (p < 0,0005) than the number of actions for lost games (Right panel in Fig. 2). Therefore, the assumed hypothesis can be accepted.

Fig. 2.
figure 2

Left panel: Boxplots of time distribution for lost/successful game sessions. Right panel: Boxplots of N of actions distribution for lost/successful game sessions. A box corresponds to the distribution of values according to 25%–75% confidence interval. The line at the center is the median. Here, L denotes lost sessions and S denotes successful sessions.

Full size image

4.2 RQ2: Which Errors Do Students Commit During the Game?

Each patient has a list of constraints that have to be respected during the game session. In the considered scenario, 7 hard constraints are attached to Patient 1 and Patient 2, 3 hard constraints are attached to Patient 3 and Patient 4, and 4 hard constraints are attached to Patient 5. In case of breaking hard constraints, student accumulates critical errors that lead to the fail of the game.

The distribution of critical errors according to their type shows that the majority of critical errors is related to omitting the delivery of medicines (e.g. a patient did not get paracetamol) as well as missing cares (e.g. a patient did not get a meal) (Left panel in Fig. 3). Meanwhile, less frequent errors were committed due to not following prescription (e.g. a patient overdosed with inappropriate medicine), missing procedures (e.g. missing an injection), and lack of monitoring parameters.

The distribution of the critical errors according to patients shows strong dependence on the number of constraints (Right panel in Fig. 3). Herein the majority of the errors corresponds to Patient 1 and Patient 2, while the minority corresponds to Patient 3, which requires less attention.

Fig. 3.
figure 3

Left panel: Distribution of critical errors by the type. Right panel: Distribution of errors by the patient. Here numbers on the bars denote number of constraints for each patient.

Full size image

4.3 RQ3: Do Students Achieve Learning Progress by Repeating the Game?

Finally, in order to investigate the playing progress, the repeated sessions were examined. Among 222 students, who played a game, 81 students played more than one session (Table 1). During the first session 94% of students failed, while the percentage of failed games during the second session is 75% and during the third session is 76%. Furthermore, with increasing session order the average number of actions made per session is decreasing as well as the average time. This can be caused by the following reasons:

  1. 1.

    When a student launches a new session and restarts a scenario, the game saves modifications of schedule from previous sessions and a student continues playing. This simplifies the game process and cuts time during the second and further sessions due to the absence of necessity to plan.

  2. 2.

    Knowledge, gained during first sessions, helps some students to achieve a better result in the next session.

Table 1. Characteristics of repeated sessions.
Full size table

5 Discussion

In our study, we analyzed data gathered from game-based simulation for students from Nursing Schools CLONE. The game aims at training nurses to organize their daily schedule, to delegate specific tasks to the nurse assistant, in order to educate professional nurses on scheduling skills, situation awareness, and decision-making. With the implementation of statistical approaches such as non-parametric criteria and distribution analysis, we examined factors that have impact on the success of the game.

Comparing duration of lost and successful games, we found out that time and number of committed actions for successful sessions are significantly higher than for lost sessions (RQ1). Along with scheduling and delegating, students may inspect additional information, such as medical reports or patients’ profiles. Likewise, they may set and modify schedule panel without any restriction. This activity requires time and numerous interactions. Consequently, students, who devoted more attention to these details, achieved better results.

Responding to the RQ2, we considered the distribution of critical errors that lead to the fail of the game. The most frequent committed errors are related to missing the delivery of medicines or cares. At the same time the less frequent errors were made by the reason of not following prescription, missing procedures, and lack of monitoring parameters. The distribution of critical errors by patients is strongly depends on the number of hard constraints, where Patient 1 and Patient 2 require more attention and Patient 3 less.

Finally, repeated sessions were examined in order to analyze learning progress during the game (RQ3). The percentage of won games is growing with the increase of session order, while the duration and number of committed actions are decreasing. This can be caused either by simplifying the game process during the second and further sessions due to the absence of necessity to work with schedule either by gaining knowledge and experience from the first sessions that help some students to improve the performance.

The obtained results point that students’ learnability and acquisition can be improved in part through involvement and repetitions in game process. Indeed, the student’s success progresses until it reaches a critical threshold of stability. It means that the students did not acquire totally the expected skills thanks to the repetitions and involvement. It highlights the need to add complementary theoretical courses to the serious game.

6 Conclusion and Future Work

A game-based simulation CLONE enables to train of professional skills of students from Nursing Schools. The analysis of collected data from the gaming sessions might help domain experts to understand learners’ strategies and behavior. It should help to improve the educational process by providing data-based decision-making. For instance, it may be useful for an indication of struggling students and maybe even offer specific remediation actions for them.

The use of GLA techniques can be applied in order to evaluate student performance, determine their in-game behavior as well as to enhance the design of other serious games. Therefore, in future work, we will focus on student profiling through analysis of action sequences that students commit during the game.