Keywords

1 Introduction

In the field of security and safety practical trainings have always been used to exercise organizational processes and to strengthen the problem solving skills of the individual, the whole team and across organizations. A recurring problem with the trainings, however, is that especially large trainings with participants from different organizations cost a lot of organizational and financial effort. Also, due to structural or economic issues, for example, the trainings often cannot be carried out everywhere where it would make sense. An example of this can be the mass casualty incident exercise carried out by the fire brigade and a number of rescue services in Southern Brandenburg (Germany) in 2017 [1]. The exercise involved about 400 people from different organizations and took almost a year to be organized. In order to minimize the number of overlaps with daily operations, the exercise held on hospital grounds was carried out on a Saturday.

But not only the field of security and safety, specifically emergency response and crisis management, could benefit greatly from taking practice to the next level. Various concepts already exist to improve the results by using modern technology and applying new methods to facilitate learning success. As such, a lot of aspects regarding planning and training in various fields like education, manufacturing and production, facility management and many more can be improved upon by utilizing a more hands-on approach that is as close to reality as possible. This is where virtual reality (VR) comes into play – literally. Pushed by recent advancements that mostly stem from the gaming industry current VR applications that make use of head-mounted displays (HMDs) and advanced tracking methods hold great potential in regards to transferring real movements into a virtual environment that offers intuitive visualization. Properly utilized this can lead to better results as the processes involved engrain themselves deeper in the trainee’s mind and muscle memory. And while areas such as sports can mostly replicate the appropriate situations during training, there are a number of fields that need to find a solution elsewhere, be it that an adaptation of realistic scenarios would require too much space, be inadequate, too costly or straight up impossible to carry out. New developments allow multiple users to dive into a virtual environment at the same time all the while using different technologies and even tracking and incorporating real world objects into a given scenario, even combining 1st person VR and desktop applications. Following this trend, the enhanced immersion from VR coupled with other approaches can lead to positive results as has been established within the scope of so called serious games.

This paper aims to further examine the potential of virtual reality applications in the given context. The use of replicating the real world within a virtual environment that offers various possibilities to practice whatever situation is required or desired has been a subject of research for several years now and modern technologies coupled with a broader availability of the necessary hardware open up plenty of opportunities for new and improved planning and training methods. Immersive and interactive scenarios can be used to increase the effectiveness of learning: trainees not only observe or listen but rather play an active part by influencing the digital environment around them. Here, VR provides the framework for a highly customizable application that is tailored towards the specific needs at hand and may go beyond traditional training methods in order to study and absorb motion sequences, build muscle memory and impart knowledge on the participants. Employing collaboration techniques and multi-user environments in this context provide a wide range of possibilities, i.e. through the use of student-teacher interaction or cooperative efforts by working on a task together. The approach established here is based on feedback of emergency response teams and institutions as well as a literature review of the related work. The collaboration aspect is implemented through different roles and rights, for example by allowing only certain participants to perform a given task or manipulate specific objects. This design enables a number of interactions and behaviors that can be utilized to carry out the designated scenario by switching between active and passive roles and views. The focus further lies on 1st person VR and interactions enhanced through real world objects as well as in-game scenario building for dynamic exercises and adaptive learning.

The approach was implemented featuring a fire drill that includes both a tactical planning phase and a training phase. Roles like fire departments chiefs and squad leaders can guide through the first phase using several tools that are only at their disposal. Decisions and changes made during briefing will influence the events within the second phase, where the assignments for those that were observers will change as they now have to translate the results into actions, i.e. by putting out fires and following other orders. This exemplary application will be implemented via available modern VR technology such as HMDs but also support other means of input like keyboard and mouse together with a regular monitor. This allows for dynamic custom-fit solutions that improve existing practices and learning concepts while enabling detachment from the aforementioned spatial, monetary and organizational constraints. The collaborative virtual environment holds great potential when the technical possibilities mentioned above are applied to the full extent to achieve an immersive and holistic approach which can increase the effectiveness of learning.

The paper is organized as follows: Sect. 2 gives a short introduction to virtual reality and an overview about the related work; Sect. 3 contains the description of the requirements; Sect. 4 exemplifies the concept and design for the collaborative virtual environment including the roles and rights management; within Sect. 5 the implementation and the scenario are explained. And finally, Sect. 6 draws the final conclusions and gives some input regarding outlook.

2 Background and Related Work

Virtual reality as a field of research is relatively new and highly correlates with the development of the necessary computer hardware. Up to now there is no uniform definition for it. On the one hand it is a technology that enables the creation of immersive and interactive environments using special hardware and on the other hand it refers to a methodology for an inclusion of users to a seeming reality [2]. Compared to conventional computer graphics VR allows a multimodal presentation including visual, acoustical and haptic components. Presentations, interactions and simulations in VR are time critical and have to be done in real time. Whereas in conventional computer graphics interactions are usually done in a two dimensional manner (e.g. keyboard and mouse) and the perspective is user-independent, VR uses 3D-interactions including movement tracking (e.g. head and hand tracking) and allows for a user-dependent perspective [2].

As described by Guillen-Nieto and Aleson-Carbonell [3] the value of VR for education and training purposes within serious games has been shown multiple times in research and in practice. Next to the effect of an increased personal motivation and satisfaction, different styles of learning are accommodated for and decision-making as well as problem-solving is being supported. Serious games and VR also have a positive impact on the effectiveness of learning outcomes as shown by the authors. Especially virtual reality-based instructions and task planning [4] resulted in a higher learning outcome. The use of VR has furthermore been proven to be effective for collaborative learning for example in e-learning environments [5] and other fields [6].

Based on the mentioned learning results, it is not surprising that VR is currently being used in many areas for learning, training and planning purposes: In [7] for example the authors implemented a virtual prototyping system to allow the collaborative planning and design of vehicles. They argue that “virtual prototyping is a logical extension of computer-aided design…” and helps to overcome the problem of having people from multiple locations working on the same project. Another example is shown in [8] where the authors among others designed the “Sky Classroom”, a VR-tool that aims to teach global engineering collaboration skills. The avatar-based platform allows students to virtually enter the building construction site and review the BIM model.

Potentially the biggest field where serious games and VR are used is the area of security and safety including medical and military trainings as well as civil safety and emergency management. An example is the modular virtual reality patient simulation system for U.S. Navy medical providers [9] that offers both, a three-dimensional VR and a two-dimensional desktop interface for teaching and training of cognitive assessment and treatment skills. Pucher et al. [10] argue that mass casualty incidents have become more common and therefore low cost, immersive VR platforms for training and assessment could represent “[…] a powerful adjunct to existing training methods […]”. They designed, implemented and evaluated a virtual clinic where multiple doctors could simultaneously manage the situations within the limits of the hospital’s resources. Sharma et al. [11] propose a collaborative virtual environment for preparedness and response training in subway evacuations. Based on Unity3D and Photon they designed a multi-user avatar-based training environment in the cloud. Through the combination of computer simulated and user controlled (using provided controls or a joystick and a VR-headset) agents, they try to analyze crowd behavior and people trust in evacuation scenarios. Passos et al. [12] provide and evaluate a collaborative virtual environment and compare it against commercial solutions and other research projects. They use Autodesk 3ds in combination with Unity3D to simulate big events and train security agents. Their research focusses mostly on the collaboration model including communication, coordination and cooperation and the scenario design tools. Like before they use avatars controlled by users with keyboard and mouse to play the serious game. Some tools for data analysis were included for a better scenario evaluation.

Next to the mentioned publications and finished research projects there are also examples for currently ongoing projects like the Auggmed project [13] or commercial solutions like the XVR platform [14] or LUDUS [15]. They all focus on scenario building and training for collaborative multi-user scenarios in security and safety. In addition to that they all include features for the evaluation and analysis of the training scenario.

The use of serious games and VR for planning and training scenarios as described in the publications and systems mentioned above is already proven to be an effective tool for learning. Especially the design of collaborative virtual environments and multi-user scenarios was shown to be of high importance for most learning areas. In addition, the implementation of these systems in a cloud- or server-based fashion is an important prerequisite as many teams do not share the same spatial location. To allow scenarios that feature for example big crowds it is appropriate to combine computer simulated avatars with avatars controlled by the trainees. To be able to evaluate the learning experience it is furthermore worthwhile to add tools for analysis in regards to the developed system. In most papers much effort goes into the scenario building and avatar design. However, so far all approaches focus on agent based trainings where the trainee uses a joystick or a keyboard plus a mouse to interact (sometimes in combination with a VR-headset for head-tracking). This results in a non-immersive training scenario and mostly 3rd person visualizations and interactions. Moreover, the number of interactions might be limited due to predefined, static scenarios and real world objects are not included in the virtual environment. What is needed is a generic virtual training environment that allows users from different organizations or acting within different roles to directly interact with their surroundings and each other in a 1st person manner and include real world objects into a dynamic scenario in order to facilitate the immersion and enhance the learning experience and outcomes.

3 Requirements for the Collaborative Virtual Environment

While there are many successful applications considering the training of individuals or teams for tasks and missions in the field, the planning aspect is often viewed separately or disregarded entirely. Thus, the interaction between user and environment as well as the collaborative aspects mostly take place in a set scenario with changes stemming mostly from user interactions that are different between repetitions while the surroundings and tasks remain the same. In order to overcome those inherently static approaches the potential of incorporating the results from planning was identified as a way to introduce variety and improve learning effects. The added value that arises from the inclusion of planning aspects for the training of emergency forces has been known for some time and is taken up in these areas by means of simulation games on stationary training boards [16]. Other examples also show that situation analysis and planning in crisis management has already established itself as a great tool and is practiced by various means, such as the use of interactive table tops [17]. The following section identifies the requirements with regard to the use of VR for planning and training in accordance with the method of requirements engineering. This method is comprised of identifying, analyzing, specifying and validating the characteristics and surrounding conditions of the software system [18] and allows researchers to work closely with experts, programmers and operators alike. Aspects of planning are incorporated with the aim to develop an immersive, interactive and holistic experience for multi-user VR based on the correlating procedures. It is of significant importance to clearly define all the requirements that have to be considered when designing a concept for a collaborative virtual environment that carries out dynamic scenarios.

The primary component that was identified in regards to a collaborative virtual environment is by definition that of multiple users in the same scenario. However, in the given context of utilizing planning to make the scenarios dynamic in nature it is no longer sufficient to simply rely on the for the most part limited scope of conventional networking solutions. While there is a plentitude of possibilities for basic operations such as establishing connection between clients or sending messages and other information, the implementation of the planning aspects and the various interactions that come with it needs to be much more refined. This includes but is not limited to the management of roles and rights for the manipulation of users and objects. For example, if multiple organizations are planning a scenario it has to be ensured that clients can only interact with things they are allowed to while maintaining a unified view for all parties involved. Due to that, the environment’s implementation has to allow for a multitude of clients with differing access rights that can be granted and revoked at runtime. In order to account for those changes, all objects relevant to planning have to be integrated in the networking infrastructure one way or another. The roles and rights management component is required to handle both a fixed hierarchical command structure that affects interactions as well as a flexible system that features things like ownership requests and has an according and intuitive representation in the environment.

While handling different levels of access and interaction in the various scripts that are responsible for objects and actors the visualization is also an important requirement in order to ensure intuitive controls and understanding. If this is not provided, such a complex system may quickly become an intrusion or a distraction when inside the virtual environment and hinder planning and training alike. The requirement is therefore set to have colors that accurately represent whichever organization the object belongs to and gives the user an idea of the possible interactions. Since this approach is limited in a couple of ways additional help should be provided, for example through the use of meaningful and easy to understand names and variables that will show up on the user interface (UI) or the information board in the meeting room during the planning phase. This extends to the avatars of the users within the virtual space. Not only should they reflect either a certain person or role, but they should also be accurate enough so that actors will know immediately whether or not they are talking to or interacting with another party. An additional requirement is thus formed to outline a character representation concept that defines the visual aspect of user avatars. Since modern HMDs like the HTC Vive or the Oculus Rift rely on the tracking of the head as well as two controllers (one for each hand) it became immediately obvious that those will also be the core components of character representation. Since full body avatars are not only highly complicated to animate in a way that does not impair user experience due to inaccurate or flat out wrong movements in combinations with factors like clipping but also have to be networked correctly within the collaborative scenario, the requirements for avatars where reduced to representing the position of head and hands. Clients receive both the position and the rotation which still allows for an adequate perception of other users and their movements and interactions. For example, one of the most basic but also most important requirements was the look direction that enables an immediate understanding of what other actors are currently doing or talking about. Coupled with tracking and networking the controllers that should not be limited to the visual representation of hands but also allow for various tools such as pointers and markers the rather abstract and simplistic approach to avatars is deemed sufficient. An additional benefit is that this requirement also prevents users from obstructing each other’s views when huddled together in the meeting room. Concluding these specifications, the environment should seamlessly provide integration for all possible interactions as well as character representations (avatars) and offer an intuitive surrounding that holds all necessary features and objects of the planning and training scenario. While a certain level of visual quality is definitely not only desired but also needed for a sufficiently immersive environment and intuitive handling of objects photorealism could never be an actual requirement. Not only are current HMDs limited in quality, e.g. due to resolution, but implementing such high quality visuals in the given context is neither time- nor cost-efficient. Furthermore, once computer graphics reach a level like that, the so called uncanny valley may lead to unpleasant side effects [19]. Thus the overall requirement for environment visualization can be described as a trade-off between quality and performance, basically stating that as long as the user is not occupied with graphical or other shortcomings of the virtual environment the specifications are adequate.

Following a similar line of not interfering with immersion and facilitating intuitive comprehension are the requirements for the focus on 1st person VR. While allowing spectators and even certain actors to use regular desktop applications the main goal is the integration of intuitive 1st person interactions through HMDs and controllers as well as real world objects that enhance the learning effects of planning and training. This includes, but is not limited to, different views and camera perspectives that improve user experience by enabling them to follow the action in a both comfortable and practical manner. Controller haptics and interfaces for different hardware and application types are of equal importance when it comes to providing a suitable VR environment. This should lead to an implementation that features easy to grasp manipulation of objects, for example in the sense of what can be grabbed and how, as well as more abstract interactions like granting or requesting permission for certain actions and working together with other users. Lastly, this requirement also entails the use of real world objects that are tracked and networked similar to the HMDs and controllers, currently available for the HTC Vive [20]. As such the specifications for 1st person integration should be able to handle different HMDs, incorporate tracking and interaction with both virtual and real world objects that is intuitive and improves the multi-user scenarios by processing all necessary information within the network. Since the scope of this requirement remains fairly wide it will be explained further in Sect. 4.

As not every interaction can be carried out through the means of what is inherently provided by the VR systems, custom tools are another critical requirement in order to make planning and training work in a collaborative virtual environment. This includes a lot of features that can be found in most VR applications like teleportation for when users run out of tracked area or want to participate while remaining seated at their desk. The requirements for movement in the virtual space also extend to aspects like limiting users to certain areas and providing tools like navigating to a given target. Together with the avatar representation described above this ascertains that users can follow others while moving fast or even teleporting. During the planning phase, the UI has to implement several context-sensitive features that pertain to manipulating the relevant objects but also administrative operations like adjusting scenario settings, managing roles and rights and finally initiating the training phase that is based on the dynamic features of planning. One example of this is the placement of so called pawns. Similar to chess pieces, those pawns represent certain aspects of the training scenario that can be moved around to place spawn points (determining where an organization or user begins the training scenario), location markers (used for pathfinding or highlighting tasks) or influence the environment by blocking pathways or diversifying tasks depending on the desired scenario. Specifically, the collaborative virtual environment should make it possible to practice the same situation in a multitude of different ways. Again, a core aspect of the requirements is to keep them as flexible as possible in order to be able to design varying components that facilitate different learning goals. Upon initiating the secondary phase, pawn placement together with all the other features and tools is transferred into the training environment, thus creating a dynamic scenario based on the results of the planning phase. It has to be considered that a lot of those tools not only have to be made available for individual users but also need to be networked properly and are potentially influenced by other actors. For example, the leader of an organization can grant and revoke certain rights for other members during the planning phase, e.g. letting them interact with specific objects, or direct and observe them during the training phase while changing environment parameters.

All aspects discussed in this section aim to support an implementation that remains as generic as possible to allow for various types of scenarios that are dynamic in nature. In order to accomplish that, the definitions above ensure that the proposed system remains independent of location and technology. This is achieved by using online networking that supports a multitude of users and organizations, different platforms as well as application types and provides tools for dynamic scenarios in various contexts. Since the possibilities of integrating such tools in a 1st person environment are basically limitless, the corresponding requirements merely outline necessary features and considerations without narrowing down the conceptual implementation which will be exemplified in the following sections.

4 Concept and Design of the Collaborative Virtual Environment

Based on the requirements outlined in the previous section, a concept for a collaborative and interactive planning and training scenario in the scope of serious games was developed. Important for this concept is a modular structure to transfer possibilities and opportunities easily in different scenarios and fields of applications like teaching, facility management or rescue training. Although the implementation should be transferable and enable individual solutions, each scenario in a distinct field also offers various outcomes through use of planning and training. In the planning room, there is the opportunity to discuss the action before training with all users and influence the scenario individually, for example by introducing new goals, starting positions and obstacles to the scene. That way the focus can be changed in reoccurring exercises or to add something new and unknown. The concept shows the usage of VR for a fully immersive scenario while retaining the option of using a regular desktop application for an easy overview without the need for an HMD. This section will mainly focus on the aspect of building a generic but dynamic multi-user environment. The first part outlines how the project-specific components like the Unity3D engine and different software development kits (SDKs) will work together to create the baseline for the collaborative planning and training. Subsection 4.2 explains the use of different objects by various users and roles.

4.1 Modular Components in Unity3D Using Photon and VRTK

In order to create a scenario which is ready for both VR and non-VR systems and usable independent of location by means of a networking solution a game engine called Unity3D [21] was used. Unity3D is an extensible and established engine via which flexible scenarios can be created based on a modular programming paradigm. An active developer group is steadily expanding the functionality and possibilities of the engine by developing SDKs or packages with objects, textures, scripts or entire modules which can be easily integrated into a project. Therefore, Unity3D is a game engine which is highly customizable and due to the structure of tailored to fit packages it allows for development of transferable scenarios by means of so called game objects. Unity3D also combines graphics and functionality through an easy import from a large number of file formats that can be implemented through the aforementioned game objects, leading to a hierarchical and accessible project structure and an extensible implementation. Since Unity3D is limited in terms of modeling, it is recommended to rely on Autodesk 3ds or similar software to develop the objects visuals or the environment as a whole and import it into Unity3D [12]. The resulting 3D models can be of various file formats such as FBX which can be exported from numerous programs, even utilizing entire building geometries available through building information modeling (BIM) software overhauled in Autodesk 3ds. The so called mesh of an object visualizes the underlying information in combination with textures to create the environment for a scenario. Within the Unity3D project the objects obtain functionality through scripts. Due to the hierarchical and modular structure game objects can be transferred to another project or scenario while retaining all features when developing a collaborative virtual environment for dynamic planning and training scenarios.

One of the required extensions is the software Photon which is a tool to implement a networking solution and automatically manage online lobbies via a master-client relationship, sending data to all participants [22]. Photon provides a free cloud-based option which is able to manage a multitude of users in one virtual room and distribute the data of the scenario. Every user has a local copy of the scenario which is linked to the network lobby. The lobby connects the users and the virtual room so that everybody can send or receive data through the lobby. Sending data is limited to 400 messages per second, meaning that it is important to control the exchanges when the number of users increases and reduce the data sent if necessary. For the intents and purposes described here however, the limits are high enough to be neglected [23]. The incoming data is processed through the use of various scripts and can be send to specific users, groups or everyone in room, directly influencing user interaction via a process that is called interest management. As such, the Photon engine is a satisfactory solution to network the collaborative scenario (see also [11]).

Another useful package is the virtual reality toolkit (VRTK) [24] which includes extensive functionality via customizable scripts to simplify a VR solution in Unity3D. With VRTK a VR user can be created that handles tracking of the HMD and the controller. Examples include scripts to enable interactions with a controller or the HMD in order to touch, grab or otherwise manipulate objects as well as interacting with an UI to control buttons, input fields or check boxes. VRTK establishes connection with Steam VR, an SDK used for most HMDs to manage the tracking of the hardware, representing and transferring tracked objects in the virtual world and transmitting information such as user inputs, e.g. a click on the trigger. Therefore, VRTK can be used to handle the HTC Vive and the Oculus Rift, but also to instantiates a simulator if no appropriate HMD is found. The simulator can be treated similar to any 1st person VR user, although interaction is achieved via keyboard and mouse only which is less intuitive and does not build muscle memory like HMD and controllers. Figure 1 shows the structure of the usage Unity3D, VRTK and Steam VR with a HTC Vive or Oculus Rift HMD. As described in the requirements it is important to enable different points of view to be able to follow the action in the scenario. This is done with a spectator view which can be controlled with common input devices like keyboard or joystick allows for the use of a regular desktop application. An additional view together with the required tools is implemented for instructors who lead the training, give feedback or hints for specific situations and problems to participants and interact as a neutral user. This allows observer and instructors alike to utilize the various views in order to analyze and judge characteristic values that are not measurable, for example the ability to work in a team.

Fig. 1.
figure 1

Implementing different view modules and hardware systems

Full size image

Due to the chosen design it does not matter if the participant is using a HTC Vive or Oculus Rift for 1st person VR, moves via “real movement” in a tracked area of 5 × 5 m, teleports to be independent of real-world boundaries, e.g. while sitting down at a desk, or uses the trackpad or joystick to walk and control without an HMD. Another alternative would be a so called locomotion platform, e.g. Virtualizer [25], to move without boundaries through the entire virtual room. With the help of Photon and VRTK an avatar is sent to the users so they can perceive position and movement of other participants no matter the view or hardware. The described procedure also employs the modular approach mentioned above and can thus be transferred to other areas and applications. A combination of VRTK and Photon can be used to create a networking script that handles the collaboration, interaction and planning in one room while the framework and the environment is handled in Unity3D. It is also possible to spectate other users depending on the chosen hardware and perspective. The resulting game objects in Unity3D may be transferred to different scenarios and can be turned on and off as required.

This further exemplifies the modular project structure explained above and enables users to individualize the scenario before training by adding, manipulating or removing objects. It is also possible to deactivate whole parts in a scene and enable them again when they are needed, distinguishing between purely visual features or full-fledged functionality. If there are no direct dependencies between the objects, they can be moved, changed and otherwise manipulated resulting in a very flexible usage in order to create dynamic scenarios. The training phase is directly influenced by the planning which can be managed in a collaborative way by means of Photon networking. This process is based on the action the user takes throughout the briefing and is transferred to the training scene after being accepted by all participants. The concept based on the modular setup of game objects and scripts is easily transferable to various applications and scenarios in different fields of training.

4.2 Managing Collaboration with Custom Roles and Rights

In order to create a dynamic scenario which can be used by various participants in different locations a collaborative room is required. Collaborative in this case means that more than one person can be in a virtual room, recognize other users and interact with them or perform interactions that are made visible to others. Therefore, it is important that each person is in the same virtual room independent of their real-world location. From a technical point of view each user has a local copy and changes, interactions as well as the other relevant information are sent to a specific interest group or all users, after which the data has to be handled. The main room is hosted by a so called master on whose side all settings and interactions in the scene are managed, independent on this user’s role. If the master leaves the room before ending the scenario the environment is reset and all information in the scene together with objects created at runtime is lost. In some cases, partial information is sent to another user who becomes the new master. This solution entails various problems for the proposed concept because users take on different roles and each role has its own responsibilities, interaction options, knowledge and rights based on its real-world counterpart (e.g. team leader). In order to prevent a user either obtaining more rights than his role allows or losing objects and interaction possibilities, roles and rights need to be thoroughly managed. As shown in Sect. 4.1 the solution is based on Unity3D and the Photon SDK automatically creating a virtual master-client room and managing all users in a flat hierarchy. While the master can manipulate the networked room or environment in various ways, other clients can do so only within their local copy, preventing this information from being distributed via the network. If a user is not the master and wants to interact with an object, the existing solution has to be expanded upon. To allow an adaptive, intuitive and collaborative approach that represents an appropriate rights distribution of a real-world scenario like a relationship between teacher and student or organization leaders and employees, this paper defines a custom roles and rights management.

All different relationships of various application areas are reflected in a three-part hierarchy. A group called ‘organization’ was created which can have different rights and interaction options while remaining on the same hierarchy level. In this concept any number of organizations may be created. An organization consists of one leader and a certain amount of members. Within an organization, the leader has more tools and possibilities at his disposal than a member. The last hierarchy level is the ‘own’ which does not influence other users and as such reflects the handling of a local copy. This hierarchy concept is shown in Fig. 2 and attached to relevant objects via a script which enables the modular approach described above. Depending on the hierarchy level, users can add objects to the scene or manipulate existing ones. Afterwards, all relevant information such as the object’s owner and the associated organization has to be transmitted via the network. The information is automatically sent to all users so that it can be checked who may interact with an object. If the transaction is valid, all connected clients receive an update view. The roles and rights management is based on real-world planning and training of emergency response and civil protection teams [16]. Their expertise was also incorporated in the modular and flexible structure which creates opportunities to adapt the concept to a user-defined scenario. In order to achieve this, the corresponding script automatically transfers all relevant information over the network via Photon and manages all important aspects regarding the usability for other users and organizations. Furthermore, it allows for easy extension to include a multitude of organizations as well as adjusting the associated hierarchy. It is also possible to send the information only to a specific group of participants that actually require it, reducing the data volume to a minimum. This information attached to objects can be displayed via UI elements or result in further action such as changing the colour depending on the ownership to visualize the roles and rights in play.

Fig. 2.
figure 2

The designed roles and rights management for object interaction

Full size image

As part of the roles and rights management an adjustable and automated request handling is integrated that represents real-world interactions required for planning and training. In such a scenario it has to be possible to use or manipulate objects, interact with them or change their state. ‘Using’ an object means to interact with its functionality, e.g. to open a door, whereas ‘manipulating’ means to alter the attributes of the object, like changing the size of the door. The custom management defines how users of a certain role may interact with an object. Since there are too many cases to manage individually this is done via an interface. Implemented by the roles and rights management it handles the underlying request necessary for user and object interaction.

The easiest case is a ‘general object’ like a door. This object is usable by all participants and is not limited to an organization or a specific role. The manipulation of a general object can either be turned on or off entirely for all users and is set when creating the object before the scenario begins. To start an interaction with an object the user only has to touch it with his VR controller. This initiates the ‘control’ component of the rights management and checks the available options depending on the user. For example, when touching an ‘organization object’ it is determined through the role and the membership of the user whether or not he is allowed to use or manipulate the object. If it is not an object of the same organization the user belongs to, a request is sent to the leader of the owning organization. If the owning organization however is the same the right management feature checks the role of said user. In case of it being the leader the right to use and manipulate the object is automatically granted. If instead the participant is on a lower hierarchy level, he may use the object but only manipulate it after sending a request to the leader (which of course has to be accepted). Lastly, if it is an ‘own object’ the user has the full rights to manipulate and use it. If another actor wants to use such an object, a request is sent to the owner while organizational aspects are omitted. The interaction options depending on the role the user has chosen and the type of object is shown in the following Table 1:

Table 1. Object interaction through use and manipulation in the roles and rights management
Full size table

The rights management automatically provides control if a participant is allowed to interact by either using or manipulating an object. The type of access granted is determined through organization, role and ownership of the object as well as the user. However, if a user wants to interact with an object of the same organization but does not have sufficient rights or wants to use an object of a different organization entirely, a request is sent to the person (in most cases the owner) responsible for said object. These requests can be answered by the owner in three different ways in order to give other members rights to the requested object. If none is selected, the request is denied after a certain time so as to not overload the network messaging stack. Figure 3 shows the underlying concept, outlining how the rights management handles the requests of the parties involved depending on their role.

Fig. 3.
figure 3

Request handling in the roles and rights management

Full size image

The owner has the option to grant the requesting person permission to use the object as if it belongs to them. In this case, the requesting user receives permanent rights until either the owner takes back control or the object is deleted. Another way is to grant temporary access to the object, so that the requesting user can perform precisely one action with the object before the permission is revoked. The last way to answer to a request is to reject it, as a result of which the associated rights remain the same and no permission is granted. Due to the modular approach of handling objects via the roles and rights management the corresponding scripts and game objects in Unity3D may easily be adjusted to implement new features. It allows for an intuitive and realistic way of dealing with objects in a given planning and training scenario. On one side the automated request handling and roles management minimizes the necessary interaction and data to handle object rights. On the other side the rights management provides the opportunity to interact with other users in the scenario individually and practice real-world dynamics and relationships, e.g. when an organization leader has to handle his team members as well as the communication and interaction with other organizations. Therefore, the implemented roles and rights management is a powerful tool to handle the objects in a fast and realistic way and a key feature of the proposed planning and training concept.

The interactions between users and objects as well as the actors themselves that are processed with this roles and rights concept are especially key when dealing with the implications of the planning phase. However, many can still be utilized when shifting towards the training phase. By handling all relevant objects in the way described above it becomes possible to create dynamic scenarios with a multitude of different tasks or ways to carry out the designated training routines. This will be demonstrated with a specific collaborative virtual environment in the following section.

5 Implementation of the Collaborative Virtual Environment

In order to validate the concept outlined in Sect. 4 together with the requirements defined in the third section, a use case was constructed that incorporates all necessary features and allows extensive testing of all aspects that were developed to facilitate the collaborative virtual environment. Using the game engine Unity3D and the multiplayer game backend Photon as described in Sect. 4.1 a safety and security scenario was devised which starts with a detailed planning phase. Once this first phase has concluded, the results of this briefing- or meeting-like session are put into action during the training phase. The setting is an emergency response scenario that takes place inside a building and offers various possibilities to practice certain aspects and procedures. This includes but is not limited to extinguishing fires, navigating the building to locate persons or objects or perform other tasks previously specified in the meeting room. All of the described features can be directly influenced during the planning phase as shown in Fig. 4, for example through varying a team’s point of entry, the general location of the fire as well as the specific objects that will burn or the building’s layout by blocking certain paths with obstacles or locking doors that could previously be opened. Due to the modular setup it is very easy to narrow down or expand the scope of the exercise or even turn specific features on and off entirely. At the same time, the scene geometry is interchangeable as the program logic can be applied to a broad range of conceivable scenarios.

Fig. 4.
figure 4

Overview of the collaborative virtual environment

Full size image

Before this can be done however, some setup is required. Upon connecting to the room, users may select from different organizations that are related to the given emergency and crisis management scenario, e.g. police, fire brigade or other first responders. They may also simply choose to spectate without taking on a particular role. This initial step also serves as a proof of concept for the roles and rights management described in Sect. 4.2. The leader and member shown in Fig. 4 are representative for various constellations of organizations as well as hierarchy levels that are possible with this implementation. For example, each organization’s leader may interact with all objects that belong to the team while the members lower in the hierarchy have to request permission. It is of course entirely possible to assign further interactions such as granting access to everyone in the room, only members of a certain organization, only to specific roles (e.g. all team leaders) or revoke permission for everyone but the object’s owner. Due to the requirements defined in Sect. 3, different levels of interaction are visualized, for instance through their color, to help users during the planning phase. Additional feedback is provided through the UI, either utilizing individual and context sensitive menus or the information board that is placed on one of the room’s walls. Once a user has selected an appropriate role and the selection was verified through the network, he may move freely inside of the designated area and interact with different objects or communicate with other participants during both phases (s. Fig. 4).

The main feature of the meeting room setup is an interactive miniature model of the building that the training will take place in. Not only does it give all participants an immediate overview of the environment that the training following the planning will be evolved around but it is also used to implement most of the dynamic aspects of that secondary phase based on the results of the first. This feature is also significant to the roles and rights management as a number of users can interact with different objects on varying levels of access. These objects appear in the form of so called pawns as mentioned in Sect. 3 and can be moved around and placed in the interactive miniature model to support the planning phase through visual feedback and provide a unified view of what is happening during the briefing. Additionally, the pawns are scaled to match the size of the model and later used to create a dynamic individual scenario as relevant features are transferred to the building model when the training is started (s. Fig. 4). To exemplify this further, three types of pawns will be discussed in detail. Firstly, each organization’s leader may place a marker that represents the spawn point for his team – if it is directly involved in the exercise. While team members can make suggestions by being granted temporary access, the leader has the final say. For now, every team has one spawn point that is symbolized via the figurine-like pawn in the organization’s color. On the other hand, location markers can be used to signify important spots, tasks or simply as a waypoint for navigation. All members of an organization can request permission to place one or more of those markers. Lastly, the fire is part of a category of objects that may only be manipulated by a select group of people. As it introduces a whole aspect (in this case fire extinguishing) to the training exercise that (depending on the desired scenario and its learning effects) could be left out entirely and is thus primarily relevant to the fire brigade and their goals this pawn is exclusively handled by those in control of the overall scenario coordination and therefore requires the highest level of access rights.

The setup of the meeting room for the planning phase is completed by a big information board as well as placeholders for different cameras, maps and overviews. In the current version, the information board has a rather generic purpose – it displays the participants currently in the room and any messages or events that are not specific to a certain user or have no place in the personalized UIs attached to the VR controllers. It retains several methods that can be used to display additional information and can fulfill many different purposes in a given scenario or planning session. The wall next to it holds various camera angles and views that can be used to analyze the scenario and function as an important tool that leaders and trainers can use to follow the action and assess trainee performance during the second phase. This could include a live overview on a map or floorplan (s. Fig. 4) that shows key aspects such as a trainee’s position or current tasks and location markers. Other possible viewing angles could include live 1st person views from other participants, cameras placed at crucial spots in the building or a map of the surroundings area to plan the approach on a greater scale. All of the features present in the meeting room are once again fully interchangeable and can be swapped in and out depending on the scenario at hand or the goal of a given planning session.

Once briefing and discussion have concluded, the results of the planning phase are carried over to the training phase in order to create and start the dynamic scenario. Only the session leader can initiate the secondary step, at which point all participants that opted to take part in the exercise are moved to the building. As explained before, their starting point is determined by their organization’s pawn object that was placed within the interactive miniature model during the planning. Depending on their role, users may immediately start with the appropriate tasks such as search and rescue or firefighting. The latter was used as an example to incorporate real world objects into the training scenario through the Vive tracking technology. To accomplish this, a tracker is attached to an actual fire extinguisher that is accurately represented in the virtual environment (s. Figs. 5 and 6). This provides the opportunity for an immersive experience that greatly improves learning effects. The same algorithm that places spawn points throughout the building is also used to transfer other results of planning into training. The tactical approach to a given scenario can be diversified by creating different tasks, blocking or opening specific paths and most importantly changing the general location of the fire as well as placing and moving flammable objects. The manner in which an object burns (and thus the way the fire behaves during the secondary phase) can be influenced via several attributes like heat resistance, ignition temperature and burn time. As such, it is possible to have a big fire spread throughout the building, limit the scenario to certain rooms or even dismiss it in favor of directing the exercise goals entirely towards search and rescue. All of this is done through the interaction with the corresponding objects during the planning phase. While instructors can observe from the planning room using the different tools mentioned above they can also directly join the training room and observe the exercise as shown in Fig. 5.

Fig. 5.
figure 5

Collaborative 1st person firefighting training and observation by an instructor (top right)

Full size image
Fig. 6.
figure 6

Multi-user setup from left to right with trainee (HTC Vive), spectator (desktop) and instructor (Oculus Rift)

Full size image

Users that are not actively taking part can however still interact with the scenario in general or specific trainees in order to sway the course of events and facilitate the desired learning effects. One example for such an interaction is the navigation feature that guides participants to a certain task or building part and enables real-time influence, e.g. by highlighting a door or room that should be approached next (s. Fig. 4). The corresponding visual components, similar to most other objects, provide intuitive feedback to the users, for example by coloring location markers in accordance to the team that placed them during planning. The objects based on the pawns described above are dynamically placed using Unity3D’s resource feature and instantiated through the Photon network at runtime. Through the flexible and modular system mentioned in Sects. 3 and 4 it is possible to adjust the level of interaction and thus further individualize the training scenarios. It also incorporates the roles and rights management explained in Sect. 4.2 to prevent invalid or confusing actions where users grab the wrong object or are assigned tasks of another team. Those role-specific interactions are exemplified by fire extinguishers being limited to members of the fire brigade as well as participants only being able to navigate to spots or tasks that where assigned to them during the planning phase. All users can interact with the designated objects during the entirety of planning and training, communicate with one another and move freely within the collaborative virtual environment. The described scenario in an emergency response setting thus showcased the validity of the concept by allowing users with different application types and VR technologies (s. Fig. 6) to learn from planning and training by means of the tasks associated with it. This includes, but is not limited to, the capacity for teamwork by working in a multi-user scenario, improving muscle memory through means of immersive 1st person interaction or practicing situations that might otherwise be difficult or even impossible to carry out. This further reinforces the great potential of such a collaborative virtual environment.

6 Conclusions and Outlook

The approach outlined in this paper was successfully demonstrated using the scenario described in Sect. 5. The fully immersive collaborative environment showed great potential when coupled with a VR application. The designed system utilizes current technologies including HMDs and controller and tracking hardware as well as modern engines for visualization, interaction and networking to establish a dynamic 1st person scenario. This allows users to practice different abilities in a way close to the real world without requiring too much space, having to disrupt daily routines by shutting down buildings or being too costly. It is even possible to go beyond the limits of the real world and tailor a given scenario towards specific needs that could otherwise not be practiced, e.g. by repeating or resetting certain steps. Due to the online networking and the flexible, modular approach the system is mostly independent of physical location or available hardware. The incorporation of real world objects like the fire extinguisher allows participants to learn new interactions that facilitate muscle memory and knowledge alike. As the dynamic training scenario changes every time based on what users came up with during the planning phase this learning effect is reinforced as it is being applied in various ways and settings.

The implementation described here aims to estimate the concept’s suitability rather than assessing individual parts of the scenario building or the planning and training in order to determine a final or optimal solution. To accomplish that, existing solutions for certain aspects of a collaborative virtual environment like networking via Photon and VR hardware integration through VRTK where coupled with highly customizable scripts in Unity3D, e.g. the roles and rights management mentioned in the previous sections. The presented concept seamlessly integrates new and existing features making the proposed solution capable of utilizing any building model, various hierarchy levels as well as a number of different interactions with objects and other participants alike. This was achieved through a prototypical implementation which is then adjusted to ensure a suitable course of events and fit one’s need regarding the desired planning and training exercise. Here, the authors combined the existing aspects with new features like the roles and rights management, networking relevant parts of the environment and ensuring proper 1st person interactions in VR resulted in an application with a broad scope and the potential to create dynamic scenarios, where specific use cases can utilize all or at the very least a subset of those custom features. The interactive miniature model for example was built from the ground up and tailored specifically to integrate all relevant aspects of planning that are then transferred to the ensuing training, all the while taking roles and rights, collaboration and different user interactions into account.

The emergency response scenario demonstrated the validity of such an approach with a successful implementation of the requirements as well as the concept and design. The exercise as described in Sect. 5 was tested by numerous people including plant fire brigade members of a big German manufacturer. Albeit the demonstrator being limited in its range of features it was well received with the multi-user planning and training as well as the integration of the real-world fire extinguisher being described as very intuitive and thus useful. The immersive interactions facilitate positive effects like learning to navigate a building under ever-changing conditions which further reinforces the need for dynamic scenarios. However, it was also determined that especially users with little to no experience in VR became quickly overwhelmed even though some of functionality was yet to be fully implemented. Therefore, it became apparent that it might be advisable to reduce the extensiveness for specific future applications and utilize the generic approach to build a collaborative virtual environment that focuses on fewer aspects at a time. Due to the flexible and modular setup all of the necessary systems can be adjusted to fulfill specific needs.

In order to easily accomplish that task it would be conceivable to include an accessible scenario builder for certain applications in the future. This way, all of the required functionality of the generic systems like roles and rights management might be incorporated in planning and training of any kind while parts specific to an exercise like the environment or real-world objects could be embedded through the scenario builder. An already possible first step is loading most of the relevant scene geometry from a building information model and afterwards adding components like the network infrastructure or roles and rights that already proved successful. The editor would allow to simply adjust the necessary settings and develop and consequently practice dynamic planning and training. Using the emergency response scenario as an example again, any office building in the world could quickly be made into a planning and training exercise that allows employees to practice things like extinguishing fires or evacuation routines without having to shut down parts of the building or compromising workplace safety. The planning phase would focus on education turning the meeting room into more of a classroom followed by opportunity for a “hands-on” approach. Instructors would be able to connect from anywhere, thus enabling such scenarios to be quickly deployed without the need for extensive on-site personnel.

Before that is possible however, some improvements are required in regards to certain VR modules. Movement and teleportation for example were not focused on during this case study. The teleporter currently retains unlimited range and speed which disrupts immersion and can at times make it difficult to follow other users within the virtual environment. Depending on the situation, the user avatars (heads and hands) can be hard to spot, e.g. behind burning objects or through thick smoke. Possibilities to overcome these problems include “freezing” users for a certain amount of time after they teleport based on the distance covered as well as introducing full body avatars or at least networking more parts that could provide a (realistic) benefit, for example spotting a person’s legs or torso during a search and rescue mission. It is also likely that given scenarios might have need for additional tools to put together a proper exercise. In this instance, an interface that seamlessly integrates new aspects into the virtual environment, the networking component as well as the roles and rights management for interactions will prove useful. While the network held up well during testing, integrating more users, objects and interactions might also entail the need for an improved interest management. Another future aim is to increase the number of features available for evaluation and debriefing. In the current implementation, the scenario concludes after the secondary training phase and provides feedback to neither trainees nor instructors. Prospective tools include but are not limited to tracking measurable indicators like time taken for a certain task, recording voice communication and making tasks or entire scenarios repeatable. This would enable instructors to provide detailed feedback, demonstrate better ways of dealing with a situation and generally help with assessing the learning effectiveness of planning and training within a given collaborative virtual environment.