Keywords

1 The AUGGMED Concept

Funded by the European Commission’s Horizon 2020 Research and Innovation program, the AUGGMED project was established in May 2015 running for 3 years until May 2018. Throughout the project a serious gaming platform was developed alongside multiple end user agencies and was validated against unique sets of requirements established through end user/developer collaboration workshops (see Chap. 9 for details).

The multiplayer serious game training platform AUGGMED was created in a collaboration of 14 organisations and agencies distributed throughout Europe representing industry, academia, first responders and the public sector (see Fig. 6.1). Designed for law enforcement agencies (LEAs) and first responders, AUGGMED provides a multimodal training facility which enables trainees to connect remotely and train simultaneously in a single virtual environment. AUGGMED thus addresses the challenges of multi-agency training , a vital activity in the preparation for serious incidents in which different first responders will be required to work together. However, logistical issues (time, cost of travel, etc.) make it difficult to organise such training events and train different first responders at one physical location.

Fig. 6.1
figure 1

Organisations involved in the AUGGMED project

Full size image

The multimodal system utilised by AUGGMED enables a host of interaction methods including virtual reality (VR), augmented reality (AR), touchscreen mobile devices and mouse and keyboard. The AUGGMED platform thus provides users with the capability to train collaboratively using the appropriate input method for their specific learning objectives. Each of the interaction methods possesses its own benefits for training purpose and identifying the correct method for a learning objective has been found to ensure enhanced results (Ragan et al., 2015). The touchscreen capability enables trainees to train remotely whilst on the move, whereas desktop computers are generally the most familiar option for a wider audience. The latter is especially well suited for users who have played computer games in the past, as this allows users to adapt to the game controls more quickly. Both VR and AR offer a more immersive experience creating more natural interactions than using a standard desktop mouse and keyboard. Especially augmented reality, which makes use of the real-world location, can be more relatable for trainees (see Chap. 4).

The AUGGMED platform consists of three core systems: trainer tools, the automated game scenario engine and the augmented and virtual reality environments. These three components work together seamlessly to provide users with the ability to set up and run their own simulations, to customise specific parameters required to meet their specific training needs and to get a more holistic overview when observing live exercises. Trainees, trainers and observers can access the system each with their own control systems and capabilities adjusted to their role.

AUGGMED was designed from the ground up with pedagogical principles, effective learning methods and the appropriate use of technology in mind (see Chap. 2). The simulations provide a platform to enhance a first responder’s technical knowledge and decision-making skills whilst developing their emotional resilience within a psychologically stressful situation. The learning is further reinforced by choosing an appropriate interaction system, whether it be virtual reality to build emotional and psychological resilience (Wiederhold & Wiederhold, 2008) or keyboard and mouse to focus on decision-making capabilities.

The aim of the AUGGMED platform was to provide a single large-scale training solution, which could be accessed remotely allowing for multiple trainees and organisations to collaborate in large-scale training exercises. This in turn should enable the trainee organisations to significantly reduce their training costs by replacing prohibitively expensive live training exercises with virtual simulations (Allen, 1992).

Training first responders is not a new process and has been developed and iterated upon consistently throughout history. This has led to a training process, which is both refined and extremely effective for professions from firefighters to medical experts and law enforcement agencies . However, this historic training practice has always been underpinned by the capabilities of technology. As technology develops, the best practices for training strive to utilise these advancements, and AUGGMED is another step in this evolution. As technology has enabled multiple users to exist within a single virtual environment and interact using new mediums such as augmented and virtual reality, these capabilities have been identified as critical factors, which enhance existing training approaches and provide avenues for previously unachievable practices (see Chap. 10).

Recognising these factors helped identify potential solutions to this evolution in a training’s potential and eventually led to the design of the AUGGMED project itself. Developing a project through the European Commission’s (EC) Horizon 2020 funding program provided the perfect environment to research and develop on these factors as international collaboration, training and research is at the core of both the AUGGMED project and the EC’s priorities. The remainder of the chapter outlines the development process of the AUGGMED platform as a case study of a highly successful serious game development for law enforcement agencies and first responders based on tight end user integration and collaboration.

2 The AUGGMED Development Process

Large projects such as AUGGMED require regular interactions between remote teams to ensure project milestones are met reliably. The scope of the project required excellent communication between all partners, but most importantly constant contact with technical and end user partners. Without regular updates, discussions and demonstrations, the project would have risked falling into the trap, which claims many modern software development projects – creating the wrong product.

When creating software products for end users, gathering their requirements is the most important part of any project (see Chap. 7). Gathering the wrong information, misinterpreting requirements or missing changes of requirements along the course of development can stop any project in its tracks or amplify development costs exponentially.

For the AUGGMED project, initial end user requirements were gathered through numerous interviews and open discussion with participating law enforcement agencies. These requirements drove the development of the first pilot of the platform. Existing LEA training methods were reviewed to identify areas that could potentially be improved by the use of serious games as well as areas of training that worked well to ensure these aspects were retained or at least not hindered in the serious game platform . Following each pilot, feedback was gathered from the participating end users based on surveys and interviews (i.e. quantitative and qualitative data) to collect suggestions about how to further improve the platform to ensure increasing fit for purpose. The feedback gathered information on the usability of the system and features or capabilities that the end users desired to be added to the platform. After the feedback and updated requirements had been collected, the results were analysed and assessed by the project consortium to decide which suggestions would be the most beneficial for the platform as well as feasible for development. The overall approach and timeline for the development is presented in Fig. 6.2.

Fig. 6.2
figure 2

General timeline for the development of the AUGGMENT platform

Full size image

2.1 Piloting the AUGGMED Platform

Over the course of the project, regular workshops, demonstrations and pilots took place across Europe including in the UK, Spain and Greece. The pilots stress tested and validated the outputs of the project whilst providing opportunities for end users to give direct feedback and input to the platform itself. The AUGGMED piloting system thus enabled the system to be routinely tested by multiple end user agencies. The live testing and analysis provided the consortium with an immediate list of requirements, which could be implemented for the next iteration and validated at a following pilot run.

The first AUGGMED pilot in the UK validated the AUGGMED platform’s capability to be used by police officers on touchscreen and desktop devices. The platform was utilised by two Firearm Training Police Constables (PCs), an Authorised Firearms Officer, a Firearms Training Sergeant and a Police Constable. This variety of users helped ensure every stakeholder who would utilise this platform validated the system against their individual requirements and the requirements of their organisation.

The training scenario requested officers to evacuate civilians from an airport whilst identifying and neutralising multiple terrorist threats. In the scenario, the red team members (terrorists) started a large fire in the main lobby of the airport, initiating an evacuation and forcing civilians into small bottlenecks at emergency exits. Throughout the training simulation the trainees were observed and evaluated on their critical decision-making skills, threat response methods and communication skills during a high threat situation.

The results from the pilot were extremely valuable, with updated requirements gathered from end users and a number of technical problems delivering important lessons for future pilots. The importance of the mantra ‘test early and test often’ was highlighted during this first pilot, which experienced a number of technical difficulties including network problems, hardware malfunctions and software bugs. Whilst these issues did not stop the pilot from being a success, they did interfere with the reliability of the system and reduced its viability as a working training system. Evaluating these problems, the AUGGMED consortium identified a number of important lessons, which could be integrated into the second pilot. These lessons combined with the updated end user requirements and feedback gained from post pilot interviews and performance measurements fed into a new requirements document, which ensured that the technical development remained in line with the end users’ expectations.

The second AUGGMED pilot was held in Barcelona, Spain, and validated the first implementation of the virtual reality technology within the AUGGMED platform. Multiple trainees from both Ferrocarrils de la Generalitat (FGC) and Sistema d’Emergències Mèdiques (SEM) trained using post explosion and suspect package scenarios. The suspect package scenario was tailored to the learning objectives of FGC’s existing doctrine ensuring trainees experienced and responded to the simulation as they would be expected to in real life. Within this scenario, users trained in pairs utilising virtual reality to interact with civilians, find any suspect packages and respond accordingly. The trainer and observers monitored the trainees’ behaviours, communications and decisions, grading them based upon their performance and knowledge. The post explosion scenario was tailored to the needs of SEM. The main objective of this training was to prepare trainees emotionally for what they may witness in a similar event. For this purpose, trainees were exposed to some fairly graphic sights and sounds and required to carry out triage on casualties in an effective and timely manner. The use of virtual reality helped to immerse the trainees into the scenario to increase its impact on the trainee.

Over the course of this second pilot the AUGGMED consortium streamlined the VR training processes, facilitating communications between users, building user confidence in the platform and helping new users acclimatise to the new technologies, which many of the users had not seen before. This process was documented and utilised consistently following the pilot, enabling the AUGGMED platform to be used across Europe at conferences, workshops and industrial expositions including the Security and Counter Terror Expo 2017 and the 2017 Security Research, Innovation and Education Event.

The third AUGGMED pilot was hosted in Athens, Greece, showcasing the platform’s multimodal augmented and virtual reality control systems. Trainees wearing augmented reality glasses participated in simulations on-site with digital agents, non-AR trainees and red team members rendered over the real world. Remote VR trainees participated and collaborated in the same simulation as the AR users, working together to identify and neutralise threats effectively.

The third pilot validated and demonstrated the final version of the AUGGMED platform providing end users with an extensive demonstration of the platform and its capabilities to meet their training needs. Security officers from a Greek police force trained in a multi-agency simulation containing hundreds of simulated civilians. Trainees were responsible for ensuring the safety of civilians by responding to incidents involving red team members wielding firearms and explosives. In total six scenarios were played in this final pilot, including some involving tactical planned attacks from the red team players.

The results from the final pilot helped identify further benefits of using AR/VR for training including the importance of immersion compared to classroom training exercises. Trainees also highlighted motion sickness as a large barrier to the ubiquitous deployment of such a training platform and the importance of improving user comfort when using such a system for longer periods of time (cp. Chaps. 3 and 4).

3 AUGGMED Core Components

This section introduces each of the components within the AUGGMED system, discussing the motivations for their inclusion whilst presenting their roles and functionalities in the context of what the AUGGMED platform was aiming to achieve.

3.1 Core System

The AUGGMED platform is comprised of a set of core systems. The platform itself utilises the Unity® Games EngineFootnote 1 to handle the base game algorithms responsible for rendering, physics simulation, sound, networking, etc. Through the utilisation of a pre-existing game engine, the development could focus on the simulation-specific requirements of the platform, rather than the base algorithmic functionalities that exist within most game platforms. The trainer tools are built on top of this, which enables trainers and trainees to customise, observe, record, analyse and assess any training scenario as well as trainees on an individual basis. Civilian intelligence and fire and explosive simulations are processed and handled by the ExodusFootnote 2 platform, a civilian population simulation program designed for large-scale evacuation models.

The architecture of AUGGMED consists of several individual components, each responsible for a specific aspect of the platform. By dividing the platform into individual instances – each with a specific responsibility – the project can also break down the overall development challenges into manageable tasks (see Table 6.1). This method of dividing the development into succinct modules enabled the AUGGMED consortium to distribute responsibilities to partners across Europe, helping to overcome common development challenges including communication, time zones and remote development operations.

Table 6.1 Module challenge matrix
Full size table

3.2 Trainer Tools

The trainer tools are responsible for providing the trainers, trainees and observers with an interactive interface, which provides control over all integrated AUGGMED systems. This single point of interaction enables the system to be user-friendly and approachable whilst avoiding the chances of user error. The trainer tools consist of three modules: configuration interface, real-time observation and intervention interface and assessment and evaluation module.

3.2.1 Configuration Interface

The configuration interface allows the trainer to create custom scenarios. A number of parameters can be adjusted to affect the scenario. These parameters include location (airport, station, port), threat type (gun attack, explosive, car bomb, etc.), crowd population, avatars for trainees and weapons available to each trainee type. The trainer is also able to select how many trainees can join in each role. This helps to avoid errors when trainees enter the game and choose a role.

The interface is constructed through an amalgamation of iconography and text, ensuring the system is immediately accessible to users and easy to understand (see Fig. 6.3). Each interface was also optimised to ensure mouse and keyboard user as well as touchscreen users can utilise the tool effectively.

Fig. 6.3
figure 3

Configuration options in the configuration interface

Full size image

3.2.2 Real-Time Observation and Intervention Interface

The real-time observation and intervention interface provides controls for trainers and observers to interact with the system whilst a simulation is taking place. The observation controls are available for both trainers and observers throughout the duration of a live simulation. They enable users to navigate the virtual environment from a top-down perspective and to monitor teams or individuals by setting the camera to follow their movements.

In top-down view, trainers and observers can quickly identify trainees using markers, which are easily identifiable and distinct. When the trainer wishes to observe a specific trainee, the trainee can be selected, and the camera will automatically follow this trainee. These functionalities are useful to help the trainer identify and assess an individual trainee’s performance. Users can also access a ‘trainee view’ which allows them to observe a trainee’s actions through their own camera giving them immediate access to the trainee’s experience for assessment and feedback.

The intervention controls are specifically tailored to the needs of trainers enabling them to pause/play the simulation, deploy and redeploy trainees who have been injured or trigger alarms. Specific trainer intervention controls also allow deploy red team members and suspect packages to any location within the geometry. Once the game started the trainer is able to intervene in a number of ways. The trainer can set off an explosion or start a fire at predetermined locations. This will cause close-by crowds to run to the nearest exit, and the building will start to fill with smoke. The trainer will have the option at this point to turn off the visibility of the smoke in their view, so that they are still able to see trainees even though they are caught in the smoke. Trainers are also able to start an alarm, which will cause the whole building to evacuate . At any point the trainer can pause, resume or end the simulation.

3.2.3 Assessment and Evaluation Module

The assessment and evaluation module enables trainers to evaluate trainees’ performance following the completion of one or more training scenarios. Information regarding individual trainee performance is collated and summarised, enabling trainers to quickly assess trainee performance using data including weapon fire statistics, trainee movement heatmaps and environmental reviews. Information recorded and displayed within the assessment and evaluation module can be used by trainers during debriefing sessions or by trainees who wish to evaluate their own performance following a successful simulation.

3.3 Exodus

ExodusFootnote 3 is a crowd simulation environment which predicts in real time the movement and behaviour of large crowds of people within urban environments. Running in parallel with the AUGGMED platform, Exodus creates simulations of crowd movements by receiving environmental and interaction information directly from the game engine using a bespoke API. Developed by the University of Greenwich, UK, Exodus uses data collected from real-life environments to predict the movements of crowds in several scenarios, creating a scientifically accurate simulation which reinforces training with realistic civilian behaviours.

Exodus incorporates realistic fire and explosion simulations by communicating directly with the game environment, calculating injuries, civilian smoke inhalation and hazardous material propagation throughout the environment. This enables trainees to experience realistic complex and life-threatening situations, which are impossible or challenging to replicate in real life. The fire and explosion simulations are grounded in real-world ballistic and thermal data, incorporating environmental and situational data to provide a realistic solution. Exodus also simulates intelligent crowd agents within the platform, which react to trainees, the environment and the situation. Trainees and red team members can directly interact with agents using voice commands, weapons fire and/or explosives. This crowd simulation provides a realistic representation of a crowd’s behaviour in the event of an incident and enables trainers to assess a trainee’s ability to interact with civilians within the simulation.

The bi-directional communication between the AUGGMED and Exodus platforms ensures real-time simulations can be dynamic and adaptive. This enables trainers to provide unique training experiences, which do not become repetitive or predictable, testing any trainee’s ability to react to unexpected challenges.

3.4 Automated Game Scenario Engine

The game scenario engine is responsible for simulating and managing the virtual objects and behaviours, providing a realistic digital structure which replicates real-world expectations during training. The functionality of the engine exists within the Unity Games Engine, building on the existing simulation capabilities which the engine provides.

Alongside the basic physics behaviours (e.g. gravity or collisions) and AI behaviours (such as pathfinding), the AUGGMED platform built new simulation features. A bespoke functionality, which is built upon these existing aspects, was the fire-based smoke distribution system. Designed with efficiency in mind, it enabled the distribution of smoke to be rendered dynamically across large environments. This provided an effective method for displaying the effects of realistic smoke distribution within an environment whilst creating a medium which could inform a trainee’s situational awareness. In the simulation, trainees which enter a smoke cloud begin to inhale smoke and are detrimentally affected, losing speed over time and eventually falling unconscious. The smoke rendering system is only one of the simulation requirements which were necessary to provide an effective training platform, some of which were achieved within other modules of AUGGMED.

3.5 Environments

The environments in which the trainees operate in the context of the AUGGMED platform were based upon real-world environments. This ensures that the training remains grounded, reinforcing the gravity of the simulation the trainees are participating in whilst preparing them for real events.

Three environments were created for AUGGMED: a fictitious airport based in the UK, an existing metro station in Spain and an existing port in Greece. For the airport environment, the geometry required was both complex and large scale. The entire environment needed to facilitate hundreds of agents behaving realistically, including checking in bags, queueing at security, embarking and arriving on flights and collecting baggage. Due to security considerations it was decided that a fictitious airport would be constructed from pre-existing CAD DXF files utilised by the University of Greenwich for the Exodus platform.

In order to create a high sense of realism, extensive high-resolution reference images were taken and used for texture creation. The 3D models of the locations were either created by hand or by using CAD files and blueprints as a starting point. For the mixed reality (MR) mode, it was especially important to have an accurate model, as any inconsistencies in scale or positioning of objects would cause misalignments with the real world, which would result in errors when obscuring virtual objects. This would dramatically reduce the sense of immersion that the player experiences when training in MR. Another reason that the environment had to be scaled was that the 3D environment had to map precisely to the Exodus environment, which is responsible for the intelligent agents. Any inconsistencies within the environmental scale data could cause avatars to collide with environment models, causing unrealistic behaviours such as agents walking through solid objects within the virtual environment.

3.6 Multimodal Interaction Interfaces

The multimodal interaction interfaces consisted of a combination of commercially available hardware and bespoke interaction systems. Touchscreen, desktop and VR interfacing was achieved using standard commercial hardware, ensuring the system remains accessible and affordable. The requirements of the AUGGMED system meant the augmented reality and haptic vest systems were bespoke creations, which were designed to the specifications provided by end users in order to ensure they could integrate with every aspect of the system.

The touchscreen and desktop versions of the platform are designed to work with little to no technical expertise. Because of this the system requirements were specially tailored to work with most game-ready desktops and high-end touchscreen devices. However, due to technical limitations inherent in touchscreen devices, the decision was made to stop their support as the platform became more graphics intensive, requiring more than just a CPU to render the environment.

The virtual reality capabilities were achieved through the utilisation of the HTC Vive,Footnote 4 a head-mounted display which provides users with full freedom of movement within a space of up to 5m2. This enables trainees to fully explore their virtual environment, utilise cover in the event of live fire events in the simulation and search effectively when looking for suspicious packages. The controllers give VR users full control over their environment, including weapon selection and discharge, voice commands and locomotion.

The augmented reality interface required specific technical capabilities which were not readily available on the commercial market. Due to this a bespoke system was designed which modified the HTC Vive by integrating a square HD camera to the headset that streams visual data to the engine. Using a positional triangulation of the HMD user alongside positional/rotation matrices relating to agents/objects within the environment, the AUGGMED platform can overlay these digital visualisations onto the real world in a realistic manner. However, at the time of testing the system, this method was slow to update and render due to hardware limitations.

To facilitate vibrotactile and thermal feedback, the AUGGMED consortium designed and created an integrated haptic vest which can communicate directly with the AUGGMED platform. Building upon AUGGMED’s iterative development process, the haptic vest was developed through a number of iterations, each one fed by end user feedback and testing. The vest consists of three different types of feedback mediums utilising vibrotactile actuators (vibration motors) to provide tactile feedback, thermal actuators to provide heat-based feedback and impact actuators which replicate bullet impacts. Each type of feedback is responsible for conveying specific information to the wearer. The vibrotactile actuators provide physical feedback for contact with solid objects within the virtual world, be it a wall or a person. In contrast, the thermal actuators can help replicate the temperature of an environment or inform a trainee of their proximity to a heat source. Lastly, the impact of actuators can provide immediate feedback for injuries during a live fire scenario, informing the wearer of a potential injury immediately in line with the simulation.

4 AUGGMED’s Impact – Now and in the Future

The outputs of the AUGGMED project have an impact that goes beyond the platform itself. From new design considerations to validation studies and scientific papers, AUGGMED is having a continuing influence on research into serious game development, virtual reality and training.

The AUGGMED project has provided several significant lessons learned, which have influenced further serious game projects and helped shaped the content throughout this book. Subsequent chapters will discuss the developmental processes in more detail (see Chaps. 7, 8, and 9) as well as subsequent projects, which have all been influenced by the research and development carried out within AUGGMED (see Chaps. 10, 11, 12, and 13).

Alongside the lessons learned, platform demonstrations, conferences and expos have helped proliferate the idea that serious games can be used to train a wide range of skills. This has been especially evident for first responders and law enforcement agencies. The project’s use of VR and AR technologies has demonstrated their key niche in enabling trainees to experience simulations in an immersive environment without the risks involved in dangerous real-world trainings. It has also showcased the affordability of serious games as training medium, which have capabilities that go beyond live training or classroom exercises.

5 Conclusion

Over the course of 3 years, AUGGMED proved to be a highly successful research and innovation project, which had a significant impact upon both the perceptions of serious game technologies and the virtual training industry specifically in the area of security, law enforcement and first responders. The success of the platform depended on a conclusive development that ensured the software aligned with end user needs whilst being adaptable for use by multiple agencies. The iterative method of end user engagement and development ensured that platform was able to adapt and evolve throughout the project to respond to ever-changing threats, such as of terror attacks, and that each iteration could be demonstrated and validated as part of a pilot.

Following the completion of the project, the successes and failures were analysed and used to help guide future serious game projects. These lessons learned are one of the most significant intangible results of AUGGMED.