Keywords

1 Company Description

Holovis International Ltd. is a world leading innovator and solution provider of sensory and extended experiences by bringing together leading-edge emerging enabling technologies. Holovis HQ is located the Midlands UK with offices in London, Manchester, Orlando and Shanghai.

With over a hundred technologists worldwide across multiple disciplines, Holovis delivers end to end experiential solutions across industries. Within entertainment, Holovis is recognised as a global leader in themed entertainment with clients such as Merlin Entertainment, Sea World and Farah Leisure. By combining creative story telling with immersive multi-sensory technologies, Holovis creates some of the most compelling experiences, transforming some of the world’s most popular visitor attractions.

For enterprise applications, Holovis has a track record of innovation within automotive and manufacturing with a portfolio of high-profile clients, such as McLaren, Jaguar Land Rover and BAE Systems. With industrial subject matter expertise, Holovis integrates cutting-edge technologies such as VR/AR, tracking systems, computer vision and big data systems. Combined with creative and engaging user experience designs, Holovis expands the way human interacts with industrial applications from industrial/product design and training, to quality monitoring and machine maintenance.

Holovis continues to innovate. The company is engaging in a R&D initiative that combines real time data acquisition and machine learning with Virtual and Augmented Reality use interfaces to rethink the way we interact with complex data and information for both entertainment and enterprise applications.

2 Project Summary

Vehicle design and automotive electronics are becoming increasingly complex. The 1927 Ford Model T has 1481 parts, while a modern vehicle has over 30,000 parts (Japan Management Association, 1986) with some 1500 copper wires totally about 1 mile in length (Sprovieri, 2014). This is inevitable as technology progresses alongside increasing competition, regulations and customer expectations. A consequence of this complexity is the increase of service and repair costs (Gotsch, 2018). The automobile has become part of a larger service system, encompassing “dealers, authorised repair shops, rental organisations and the users themselves” (Singh, 2010).

To help mitigate escalating service and maintenance costs, a luxury vehicle OEM (Original Equipment Manufacturer) in the UK wanted to explore the adoption of Augmented Reality for the service and maintenance of its vehicles. Holovis embarked on a Proof of Concept (PoC) that seeks to tackle three technical challenges:

  • 2D, digital or printed, wiring diagrams do not adequately communicate the three-dimensional geometry and routes of a vehicle wiring harness in contextual relationship to the vehicle form;

  • access to technical documentation and information can be difficult during inspection and maintenance tasks; and

  • maintenance tasks and notes are manually written afterwards or on separate data systems requiring duplicate data entry, which is prone to error and omission.

Holovis developed an Augmented Reality maintenance system called ViXAssist. Using the Microsoft HoloLens2, the wiring harness of vehicle is superimposed onto the physical vehicle through the HoloLens’ holographic lenses, known as waveguides. This enables the engineer to see exactly where the wires and connectors are intuitively. In so doing, the engineer can access, disassemble and reassemble the vehicle without the need for external printed references which can be cumbersome.

Via the HoloLens2 hand gesture interface, service engineers can select parts of the wiring harness to access detailed technical information. Moreover, service engineers can add annotations that are associated with a specific component. Annotations can be general comments, notes or outstanding tasks. Meta data, such as date, time and engineer ID are automatically associated with each annotation, which are logged in real time onto a central data server. Data can then be disseminated into a project management and PLM systems, and can be reviewed at a later date Figure 1 illustrates how a user can interact with the AR environment, using the HoloLens 2 with data overlay and gesture recognition.

Fig. 1
A photograph of a person who interacts with the augmented reality environment, with the use of the HoloLens 2. A virtual screen displays some data and the person controls them with his hand gestures.

Augmented Reality HoloLens 2

Full size image

3 Project Detail

3.1 The Requirements

As vehicle design are becoming increasingly complex, a luxury brand automotive OEM wanted to explore the use of emerging enabling technologies to support engineers at their dealerships globally and help mitigate escalating service and maintenance costs. The OEM approached Holovis to conduct a Proof of Concept (POC) to assess the viability of different technologies.

The fundamental premise of the project is to incorporate the use of Augmented Reality (AR) to correctly and accurately align the digital twin wiring harness of the vehicle on its physical counterpart. The system must also be able to visualise the digital twin independent of the physical model.

From a UI (User Interface) perspective, the system must be hands free with audio cues to enhance interactions. The system must incorporate hotspots or Points of Interests (POI), with which the user can interact. The system must be connected to a Content Management System (CMS) for the user to retrieve up-to-date information pertaining to the POIs. Content includes documentation, in the form of PDF or HTML, and videos. To help users navigate around the complexity of the vehicle, the user needs to have the ability to toggle the visibility of components, for example, to only view the electrical components. In addition, the system must incorporate the ability to view real time 3D animations, for example, the assembly/disassembly of components.

A key requirement is to allow the user to make annotations that are attached to specific POI of the vehicle. Annotations can be both text and voice notes and are uploaded to a remote server to support the wider PLM process.

Since the technology seeks to target global dealerships, there are two additional functional requirements. Firstly, the system should be multilingual. Secondly, the system should incorporate a remote assistance feature that allows engineers at dealerships worldwide to contact specialists in the UK.

3.2 The Technology

Holovis integrates four pieces of technology for this project.

VixAssists is Holovis’ proprietary Augmented Reality (AR) system designed for manufacturing and maintenance use cases. It originated from iGuide, which is an AR vehicle owner’s manual. The ambition of iGuide was to eventually replace the printed owner’s manual. Using the AR interface on a smart phone, iGuide provides operating instructions around the vehicle centre console and dashboard. iGuide was generalised into ViXAssists, which is device agnostics to work on smart phones, tablets as well as laptop/desktop computers.

ViXCore is a sub-component of DeepSmarts, which is Holovis data compute, analytics and visualisation platform. ViXCore facilitates real time data communication between the AR device, the CMS and a remote database, which in turn is accessible from a PLM system.

The Microsoft HoloLens2 was selected over other AR devices to provide a complete hands-free experience. It has built in hand tracking, eye tracking as well as voice control (Microsoft, n.d.). According to Alex Kipman, the HoloLens 2 waveguides delivers a 52° diagonal FOV (Wired, 2019), which gives an approximate horizontal FOV of 43° and a vertical FOV of 29°. This is 43% and 65% larger than the original HoloLens horizontal and vertical FOV respectively (Heaney, 2019).

The AR environment was developed using Unity as the core rendering engine and Microsoft’s Mixed Reality ToolKit V2 was used for the AR component of the system.

3.3 The Challenges

While the project in its entirety was uneventful, we did encounter some challenges on the way. When the project commenced, Holovis was fortunate enough to have early access to the HoloLen2. This did mean that documentations, SDKs and community support were not widely available and sparse, compared to existing devices which ViXAssists was built upon. Since the MRTKv2 was new, we encountered a number of bugs, which meant that a lot of testing had to be done on the device itself.

A lot of changes were made throughout the development process to accommodate a wider range of users. Since ergonomics is one of the crucial factors of the development, a number of customisation functions were added. Although the majority of them were simple, they are not usually required in typical POC projects. A tutorial, similar to what you might find in a computer game, was introduced, allowing users to “learn and try” to achieve device familiarity and to help users get used to a new way of working.

4 Feedback from End Users

Overall, the technology integrates and operates as anticipated. HoloLens 2 worked well and was remarkably comfortable to use compared to the original HoloLens. It has a flip up visor and can accommodate users with spectacles. The improved FOV was significant to provide a much more conducive environment for service and maintenance applications.

Hand tracking and gesture recognition allows users to select components and retrieve documentation without the need for a handheld device. Using the built-in virtual keyboard, users can enter text notes and tag them to specific components, again without the need for a wand or other handheld devices, leaving their hands free to carry out physical tasks.

The POC demonstrated successfully to the OEM leadership team that the technology is viable and has reached a level of maturity for field implementation into global dealerships as a pilot for early adopters. However, a number of ergonomics issues remain to be resolved.

The virtual keyboard is still cumbersome to use, making data entry a slow process when compared to a conventional set of keyboard and mouse. As speech to text becomes increasingly accurate and common place, it is anticipated that in the near future, the keyboard may become obsolete (Brown, 2019).

While gesture recognition worked well, users need to be very deliberate in order to invoke the correct command. Since the HoloLens is a new piece of technology, the interface is still not widely intuitive for many new users. Therefore, it is imperative that the system incorporates a robust and engaging training feature to reduce its learning curve.

Users can quickly access technical documents in the form of a PDF or HTML, which are displayed as a floating window within the AR environment. However, they can be difficult to read within the AR environment. Users can move, resize and even walk up closer to it. However, this is not intuitive compared to just picking up a printed document and reading it.

An interesting observation is that, while users can retrieve technical documents by selecting a component, in reality, engineers often need to do the reverse where they need to identify the physical location of a component from the technical document. Thus, a bi-directional link is advantages between technical documentation and the AR environment.

5 Future Outlook/Road Map

Augmented Reality is undoubted a very promising technology. With the commoditization of smart phones and the continual development and miniaturisation of wearable AR devices, AR’s growth is expected to surpass that of VR (Statista, 2020). “VR will be big, AR will be bigger” (Merel, 2017). The arrival of 5G may help pave the way for miniaturisation by off-loading computation and rendering processes to the cloud (Marr, 2020).

Two obstacles remain.

Firstly, a main obstacle is the usability of deviceless interaction necessary for deskless application, such as VR and AR. While gesture and speech can be intuitive at first, accuracy and precision are difficult to achieve compared to the keyboard and mouse. Baig and Kavakli (2018) found that “efforts for performance” and “fatigue levels” for gesture and speech are much higher than those of the keyboard and mouse. “The usability of traditional input devices [keyboard and mouse] supersedes the multimodal inputs [speech and gesture].” However, Baig and Kavakli postulate that “the gap between these two has been minimized, and with advanced technology for speech and gesture recognition, it can be overcome in near future.” With sophisticated AI and/or fuzzy logics, it is possible for computer systems to make inferences, allowing users to issue complex and precise commands without or with less necessities for precision input devices, like the keyboard and mouse.

The second is the cost of content creation. Currently, the creation of digital 3D content with the necessary behaviours and triggers is time consuming and labour intensive. Content creation typically requires a team of SW developers and artists, creating an expensive bottleneck (Porter & Heppelmann, 2017). Outside of high value manufacturing, this can be cost prohibitive.

Since Holovis is not an AR hardware manufacturer, we are reliant on the likes of Microsoft and Magic Leap. But the technology is maturing quickly and is already robust enough for early adopter applications.

To help mitigate the cost of content creation, Holovis is developing a Content Generator that will allow the OEM to add POI and link technical documents to the AR model. This means that once the AR environment has been set up, the OEM has the ability to edit the models. In addition, Holovis will implement a vehicle configurator by integrating our Rule Engine Services (RES), allowing the OEM to create different variants of a vehicle platform. An automated process will allow a service and maintenance engineer to identify a specific car through its unique VIN and visualise its exact specification, configuration as well as service history.

Holovis also plans to implement a remote collaboration tool through integration with a Video Streaming Services (VSS). This allows engineers abroad to contact specialists in the UK HQ during a maintenance task. Unlike standard video conferencing, the remote user will be able to see exactly what the engineering is seeing and interact with the digital models.

Holovis is also exploring the implementation of Bluetooth, allowing the HoloLens 2 to connect to an iPad. This has three potential benefits. Firstly, technical documents can be viewed on an iPad, which is much easier to read. Secondly, the iPad can be used to add annotation in lieu of the AR virtual keyboard, which will make typing simpler. The iPad may also be able to serve as a control interface. While this contravenes the principle of deviceless interaction, this may be a suitable way to mitigate the low level of usability with today’s hand tracking and gesture recognition technology.

6 Conclusion

The POC demonstrated that AR technology has reached a level maturity ready for field implementation. While challenges remain, in particular around the cost of content creation and the ergonomics of deviceless interaction, Augmented Reality has shown to be a promising technology to support engineers and to help mitigate the escalating cost of vehicle service and maintenance. Data connectivity allows users to access data and information through the deviceless AR interfaces and supports a much more integrated PLM process.

Moving forward, a Content Generator will help reduce the cost of content creation and custom configuration. A real time video streaming feature will enable engineers around the world to collaborate effectively by allowing them to see what each other sees across different devices. iPad or tablet Bluetooth integration can serve as a stop gap to mitigate existing usability issues.