US Air Force Weather Training Platform: Use of Virtual Reality to Reduce Training and Equipment Maintenance Costs Whilst Improving Operational Efficiency and Retention of US Air Force Personnel

Abstract

Myriad Global Media was founded in 1989 to develop high impact communications and training for private and public sector organisations. Over that period, Myriad Global (MG) has worked across multiple industry sectors including: Energy, Transport, Defence, Engineering, Healthcare and Education with clients as diverse as the Cabinet Office, Disney and ExxonMobil. It is headquartered in London (UK) with regional hub offices in Abu Dhabi (UAE) and Houston (Texas, USA), and a professional support network in over 30 further countries worldwide. The Founder and Group CEO is James Thompson; Dr. Ramzy Ross is Head of Innovation and Dainius Slavinskas is Head of Technology.

1 Company Description

Myriad Global Media was founded in 1989 to develop high impact communications and training for private and public sector organisations. Over that period, Myriad Global (MG) has worked across multiple industry sectors including: Energy, Transport, Defence, Engineering, Healthcare and Education with clients as diverse as the Cabinet Office, Disney and ExxonMobil. It is headquartered in London (UK) with regional hub offices in Abu Dhabi (UAE) and Houston (Texas, USA), and a professional support network in over 30 further countries worldwide. The Founder and Group CEO is James Thompson; Dr. Ramzy Ross is Head of Innovation and Dainius Slavinskas is Head of Technology.

The Innovation and Technology Solutions team are a key engine of growth for MG and are focused on research and development. They apply an approach that combines strategy, user insight and technological engineering to create bespoke solutions that deliver a return-on-investment. Key capability areas include immersive and interactive technologies involving Mixed Reality (MR), and associated digitalization and deployment, of Augmented Reality (AR), Virtual Reality (VR) and associated applications.

2 Project Summary

The US Air Force (USAF) is required to operate in varied, and often challenging, environments. The US Air Force recruits are continually being trained to become highly skilled practitioners who can readily adapt to any potential changing situation. The Air Force Agency for Modelling and Simulation (AFAMS) was established in 1996 and is tasked with enhancing, and leveraging, modelling and simulation ‘to support and facilitate integrated, realistic and efficient operational training across war-fighter domains to enhance full-spectrum readiness’ (further information can be found on https://www.afams.af.mil/About-Us/Fact-Sheets/Display/Article/429786/afams/ [Jan. 7, 2019]).

The US Air Force has been increasingly looking to immersive and synthetic solutions to support the development and learning of its teams (e.g. Bell et al., 2020). This particular use case involved the creation of a VR solution to train recruits in setting up, and operating, the TMQ-53 Weather Station and Broadband Global Area Network (BGAN) equipment. Such systems are portable and automatically take observations, in up to one-minute intervals, enabling flying missions across the globe. The data produced can be utilized by a weather observer in the field or by the Air Force Weather community using satellite communications.

A solution was needed to ensure that ‘real-world’ USAF technical training objectives could be replicated with the precision required for operational effectiveness. The use case emerged from the logistical challenges USAF was experiencing in not having enough readily available equipment to balance the requirements for individual training units and for live missions. A digital twin VR solution was seen as desirable because it would enable the capability to train individuals, at any location, without having to potentially disrupt live mission activity. For interest, further information on the digital twin approach, and related developments, has been previously covered in a recent publication (Fuller et al., 2020). The solution would also assist in reducing equipment related costs as related training would become less reliant on more costly, real-life equipment as a result of incorporating the VR approach. It was also seen as a training platform that would be embraced by trainees that came from a generation that were familiar with gaming technology and saw the implementation of immersive technology as a positive development in their training cycle.

3 Project Details

3.1 Getting Started: Aligning Hardware Specifications and Requirements

The MG development team was tasked to deliver a standalone, locally deployable VR application to enable immersive training of USAF trainees on the digital twin equivalent of the Vaisala TMQ-53 Weather Station and Hughes’ BGAN. Figure 1 shows an image of a similar portable real-life weather station set up.

Fig. 1

An example of a Vaisala TacMet® Tactical Meteorological Observation System used to develop a digital twin equivalent for the USAF VR training application

The US Air Force first Weather Group procured first edition HTC VIVE (Taoyuan City, Taiwan) VR headsets and Windows-enabled computers (Microsoft Corporation, Washington, USA) with GeForce GTX1060 graphics cards (Nvidia, California, USA). This set the baseline for the development, and design, of the virtual reality training application. From the outset the MG development team were aligned with USAF hardware specifications, and related requirements, to ensure the application would run without any performance issues whilst also aiming to achieve the highest levels of digital twin fidelity and asset quality. Unity 3D (California, USA) was used to develop the application. Autodesk Maya (California, USA) and Substance 3D (California, USA) was used to create visual 3D elements and objects.

The MG development team was tasked to deliver a standalone, locally deployable application with the following deliverable items defined:

• Real-time and hands-on experience for recruits to be able to setup, operate, teardown and troubleshoot the TMQ-53 weather station equipment.

• Mimic three weather conditions (sun, rain, and snow) that will require the user to think differently and ensure they adapt their methodology to complete the setup task effectively.

• Trainer-trainee multiuser mode.

• Training and assessment mode.

• Three different backdrops/locations, each with a different level of difficulty (area surrounded by buildings, area surrounded by the trees and a location in an open airbase).

• Tracking and collection of user actions and time taken to complete each assembly step.

3.2 Developmental Stages and Challenges

The specific weather station model was the TMQ-53—an off the shelf system manufactured by VAISALA (Vantaa, Finland) and used by USAF (Roles, 2017). The kit comes packed in five durable cases and is shipped to missions with some additional items, such as sandbags, to weigh the deployed equipment down for enhanced stability purposes. There are approximately 80 individual component parts including: tripods, wind sensor, rain sensor, antennas, modems, cables, power adapters, power supplies, various tools and a laptop computer.

As the client and original equipment manufacturer (OEM) were unable to provide computer-aided design (CAD) models of the various components, a key initial challenge related to being able to accurately model digital versions of each component. After extensive research, the MG development team collated reference images from various OEM materials alongside coordinating a procedural video demonstration showing complete setup, and teardown, procedures as demonstrated by a USAF team at an airbase.

Next steps involved the development of a framework that could handle the large number of individual components, each component’s respective snap targets, and be able to track the correct equipment assembly sequence. Due to the large number of components, an additional challenge was the availability of testing time to meet the application delivery timeline. For each of the approximately 80 component parts there was a need to test for potential bugs/errors, carry out any subsequent bug fixes, correct procedural sequences and allocate time for the addition of potential new features. All had to be considered and carefully managed.

Finally, consideration had to be given towards security. In particular, restricted access to the airbase network resulted in MG being unable to carry out any of the application deployment work. Further, due to the same network infrastructure-related restrictions, the MG proposed cloud-based multiplayer Photon Engine (Hamburg, Germany) could also not be deployed. As a result, the multi-user feature had to be subsequently removed from the scope-related deliverable items.

3.3 Training Space Set Up

The training application was designed to work with room scale user tracking allowing the user free movement within the constrained space limited by the VR headset cable length. To avoid cables tangling and being damaged, a retractable cable management system was assembled, and fixed to the ceiling, at given locations. HTC VIVE base stations were placed around the training zone perimeter to establish VR headset and hand controller tracking within the virtual space. Figure 2 provides an illustration of the training zone setup and perimeter.

Fig. 2

A USAF recruit wearing a VR headset and interacting with virtual objects with handheld wireless controllers

3.4 Virtual Reality User Experience

The devised training application formed a blend of the VR application and Computer Based Training (CBT). Some elements, such as data entry or monitoring, took place on the workstation monitor whilst training took place via the VR environment.

The training experience starts with the trainee, or instructor, entering personal identification details on the respective PC terminal. The user is then prompted to select a ‘User Mode’ by selecting either ‘Trainer’ or ‘Trainee’ mode and this is then followed by an option to select either ‘Tutorial’ or ‘Assessment’ mode. Following this, the user is prompted to select the specific weather conditions the environment should encompass—either ‘Sunny’, ‘Rainy’ or ‘Snowy’. The user is then instructed to put on the VR headset. User placement within the 3D environment is randomised but the user has the ability to change the location. Assessment of the environment is via a virtual action control user interface (UI) triggered with a hand gesture that is captured by the handheld controller. Additional actions can be triggered as Fig. 3 illustrates below.

Fig. 3

Hand gesture recognition triggers the action control UI. In this example, the user swipes the right-hand controller over the left-hand wrist to open the UI

The training mode begins with the user opening storage crates and taking out the required parts of the equipment. Each component has information tooltips identifying what part it is and where it connects—an example is shown in Fig. 4.

Fig. 4

Labelled equipment parts help the user to identify and connect each respective component to the right place

To continually guide users and aid the training experience, when a part/component is picked up, a hologram appears on the snap target which guides the user to the correct placement of the part (Fig. 5). The hologram aids and tooltips are removed if the user is in the assessment mode and all connections and part placements must be completed, in the right sequence, without any aid. To pick up objects, the user must press and hold the controller trigger button which is named ‘one button for all’. The same trigger button, and functionality, allows for the securing of bolts (Fig. 6), placement of sandbags (Fig. 6), extending parts, and collapsing parts amongst the other actions required in order to successfully complete all steps of the training application. This approach simplifies user-hardware interaction by taking away time needed to learn how to use the controllers, especially for recruits who have never used VR systems and controllers prior.

Fig. 5

Hologram identifying the precise part/component-specific snap target

Fig. 6

Trainees have to make suitable judgements for the placement of the main tripod and to ensure the equipment is firmly in place with actions including securing with bolts and placements of sandbags

The MG development team used procedural cables to visualize equipment cable connections. The user would pick up a plug, already having identified the correct markings and pin count, and plug it into the right socket. The cable is then procedurally traced from one connection point to another. When a connection is made correctly, a visual indicator (LED light) turns on to indicate power or data connectivity. In order to replicate real life scenarios, and train recruits to deal with varying conditions, a selection of power adaptors and extension blocks were introduced. As an example, there may be a situation where the given location power outlet has a UK 3 pin socket, but power supply would be provided with a US 3 pin cable, thus, users are challenged to choose the correct adaptor to complete the connection. Users are also asked to consider an appropriate setup location—for example, not to place the equipment too close to a building, or other obstacles, that might interfere with the station’s accuracy.

This approach was also extended to train users on making the right decisions depending on weather conditions. For example, if weather conditions were rainy or snowy, user should use provided concrete blocks to support the power supply off the ground and, if this step is missed, the equipment will report a fault and warn the user accordingly.

Audio effects were also introduced to enhance realism and the user experience. Each action would have a different sound loop to give the user audio-based feedback that an action has been applied. Unexpected audio sounds were also introduced to create better audio realism and ‘easter eggs’, as named by MG developers, were also featured to further enhance the user experience with noises such as those of a jet flying by randomly.

Upon the successful setting up of the weather station the user progresses to the next phase of the training which involves a similar approach but applied to the BGAN equipment.

4 Feedback from End-Users

4.1 Continual Feedback and Responsiveness

It was imperative that feedback was continually obtained throughout the training application development process to fine tune the approach and enhance the user experience. As part of this process, the MG development team released Alpha and Beta versions of the application to keep recruits engaged whilst also enabling the continual testing of the application at early stages of development. Further, this process also allowed for early engagements with recruits, many of which had never used a VR system prior, and demonstration sessions were pivotal for familiarisation purposes. Various forms of feedback were received throughout the development process—from the inception phase to delivery of the finalised application for training purposes.

4.2 Using VR as the Solution

Virtual reality systems can often involve a big learning curve, especially for new users. The Myriad Global approach resulted in recruits being able to effectively navigate and interact within the virtual space, alongside carrying out the required equipment assembly, in a time appropriate manner. Providing a VR-focused solution also enabled reductions in costs associated with the purchasing of real-life equipment, and associated maintenance costs, which was also a key deliverable. Tech Sgt, Rob Thomas, first Weather Wing Systems and Training Officer for USAF, provided the following related statements:

VR cuts down on the need to purchase additional equipment, eliminates temporary duty travel and dramatically lowers the potential for equipment breakage. The real savings come from Airmen not being taken away from their missions.

Saving costs is what everyone wants to hear, but opportunity cost is today’s enemy, particularly time. A trainee can be immersed and familiarize him or herself without ever seeing the equipment and a member already certified can use it towards refresher training.

The solution either simplified, or removed, related logistical considerations relating to equipment availability across USAF sites. It was acknowledged that the VR solution would not completely replace related training activities but would be focused on providing familiarisation as well refresher training. After 6 months of deployment, over two million USD in related costs were saved by USAF.

In terms of hardware, no major related challenges were experienced or reported. However, a number of considerations were raised for future developments of similar solutions. Firstly, the standard headset cable length (5 m) was not long enough for the purposes of the training application. Myriad Global developers had to purchase several alternative cables, carry out testing, and extend the cabling to 10 m. Given the portability requirement of the VR set up, for use in various locations, the tracking base stations were mounted on to tripods and not fixed to a wall or ceiling and so several tracking considerations arose. If the base stations were moved out of position, accidently or otherwise, the system would need to be re-calibrated and training would have to re-start from the beginning. Another consideration was related to tracking being lost if an individual stepped into the training space and blocked a base station. When this occurred, on occasions, this caused the virtual camera (i.e. the users view) to shake and cause discomfort (e.g. a feeling of the user falling or being dragged to the side with hand controllers ‘floating’ away from the user’s hands). Tracking issues were also experienced where locations were used with excessive reflective surfaces such as glass. Recruits had to ensure training took place in an appropriate space away from windows or glossy furniture.

Interestingly, a unique consideration arose when MG developers discovered one USAF recruit with an eye condition—astigmatism. Solutions, subsequently, were being considered including the use of customised lenses (Tseng et al., 2018).

4.3 Using the Training Application

Positive feedback was received in regard to the training application. In particular, several references were made in relation to the level of VR realism and attention to detail with the inclusion of small markings and tiny labels. Capt. Matthew Perkins, first WXG Science Officer for USAF, provided the following statement:

The realism of the VR was incredible. I could make out tiny labels and serial numbers on equipment and aircraft even flew overhead during the simulation. Virtual reality brings unprecedented realism to our training ability when the physical equipment is unavailable. Our deployed Airmen will have greater familiarity with these tools than ever before.

A number of considerations did arise as a result of hardware-related limitations that impacted the training application experience. One particular point was raised on the HTC VIVE headset Field of View (FOV) of 110 degrees and, on occasions, this became a limiting factor particularly when working in a large VR space. Occasionally, sourcing tools, or parts, in the large VR setting resulted in users having to carry out more excessive head movements (i.e. more than the required equivalent movement in real-life) to source the items. Where the user was unable to source an object, possibly due to objects being ‘lost’ or misplaced, there was no functionality to reset the training application from a previous step or time-point. Training would need to be stopped and re-started from the beginning. Finally, the application did not have the feature to save a session for later resumption—training had to be completed in one single take.

Further information on this work can be found on https://www.af.mil/News/Article-Display/Article/1822830/1st-wxgs-virtual-training-brings-real-benefits/ [Apr. 24, 2019]).

5 Future Outlook and Road Map

The USAF’s own assessment of this project was that it greatly enhanced training capability around the TMQ-53 and delivered a significant return on investment within the first 6 months of deployment. More broadly it has proven the use of digital twins and immersive VR, in particular, for ‘muscle memory’ training, analysis and fault finding. The first Weather Group are looking to further enhancements of the solution in the future as the TMQ-53 evolves.

More broadly, MG has been expanding the digital twin concept to address the challenges of training around aviation. Training of engineers in military aviation can be time-consuming and expensive. For example, training on General Electric’s F110 turbofan jet engine, used on F-14, F-16 and F-15E platforms, has often been restricted by the availability of engine units to train on. It requires ‘hands-on’ experience with expensive equipment which can be easily damaged by inexperienced trainees rendering the platforms non-operational.

In common with the TMQ-53 Project, MG are developing a VR application and accompanying blended e-learning solution. This consists of a photorealistic, digital twin of the F110, enabling trainees to learn at no risk to themselves or the equipment. This combined immersive training tool, and e-learning package, is designed to be used alongside the physical training curriculum whilst also supporting conventional classroom and computer-based learning. This will create a safe, standalone or integrated training platform that gives highly realistic visual preparation for engineers, and technicians, preparing them for work on complex engineering procedures on real aircraft.

There are a number of benefits of this approach over more traditional methods, including:

• Improved productivity offering training from a broader range of locations including remote sites.

• Reduction of training time.

• Improved information retention.

• Improved return on investment as a result.

In the energy sector MG are working with complex asset owners (Petrochemical, LNG plants and refineries) to develop VR digital twins again for training, work planning and familiarisation. These are designed to incorporate key elements including reality capture, 3D models, persistent data, building information modelling, and dynamic/real-time data.

Myriad Global are also working on other applications for US and UK Defence organisations, as well as the Energy, Medical and Technology sectors focusing on the deployment of VR for familiarisation, procedures and maintenance. The outlook for this application of immersive technology is positive given the challenges that many organisations face. These include finite resources around the availability of physical assets, the need to train and work remotely, and the challenge of reducing training time, improving information retention and reducing cost (e.g. Gallerati et al., 2017).

6 Conclusions

Overall, the project, and use of the VR training solution was regarded as a success by the client and users. The client’s requirements were achieved from a delivery point of view and there was a significant return on investment achieved.

Specifically, the goals of reducing training and equipment maintenance costs, whilst increasing efficiency of training and improving retention of related USAF personnel, were achieved. The USAF has quantified the cost saving at approximately two million USD in the first six months of deployment.

The approach of creating a virtual replica (or digital twin) for training purposes is growing as an effective alternative to more traditional methods. With the challenges of traditional methods only likely to continue (or become even more constrained), and immersive capabilities continuing to improve consistently, organisations across a range of sectors are reviewing their current solutions and looking towards innovation.

The ability for trainees to learn the required skills and gain the required experience, combined with the benefits of reduced deployment costs, reduced equipment wear, flexible deployment options and the reduced risk to human safety, will lead to a significant increase in the number of organisations experimenting with, and adopting, virtual training solutions into the future.