Keywords

1 Introduction

The ability to realistically visualise and intuitively manipulate an engineering assembly is key to an efficient design process [6]. When presented with a large assembly with heavy components, it is vital to cost-optimise its design and assembly process to mitigate prototyping costs. This ‘optimisation’ is known as DFMA, and improves upon areas such as material selection, machining tolerances, and most importantly the order of assembly of complex components to lower manufacturing and assembly overheads.

Existing work has explored conducting DFMA in virtual environments to reduce physical prototyping, however users are less able to do spatial correlation [6,7,8]. Spatial correlation refers to the ability to visualise, relate, and compare physical characteristics of a virtual object with real physical objects or environment. Within the context of virtual DFMA, the capability of visualising a virtual assembly within its intended environment is vital for assembly planning. Augmented Reality (AR) has also been utilised to allow virtual assemblies to be overlaid onto physical environments for visualisation, although with limited interaction [10, 13]. Mixed Reality (MR) presents a useful platform where engineers will be able to intuitively interact, conduct design reviews without prototyping, and visualise their virtual Computer-Aided Design (CAD) assemblies within their intended physical environment.

Understanding the requirements of conducting an assembly within a mixed reality environment through a requirements gathering session, the goal is to develop an intuitive and effective MR application for conducting assembly planning. Two pilot studies were conducted to quantitatively and qualitatively evaluate the usability of the proposed application and gather useful insight into its effectiveness and learnability.

2 Related Work

Existing virtual assembly planning systems allow users to be able to visualise and evaluate the assembly of their mechanical systems in various virtual environments with different tools. External controllers such as a mouse [5] or a haptic device [4] provide a physical object for the user to interact with the virtual environment, however will incur cost for setting up the tracking system for each device. VR-based systems [2, 6, 11, 12] provide a useful platform for simulating different environments, however it is limited in its ability to replicate dynamic and realistic environments. AR systems [10] addresses the lack of spatial correlation through overlaying useful information onto physical surfaces, however the virtual objects do not interact with the physical world. While conducting engineering design reviews in a virtual environment shortens the product development life-cycle and lowers manufacturing overheads [7, 9], these systems lack the spatial awareness needed to conduct feasibility tests of comparing the virtual models within a real physical environment. Mixed reality presents a platform in which virtual objects could interact with the physical world through spatial mapping and gesture interaction. Through conducting assembly planning within MR, the user will be able to visualise and iterate on their designs quickly, intuitively interact with their virtual models, and reduce manufacturing and prototyping overheads.

3 Assembly Planning Features

The goal of this work is to utilise MR as a platform to conduct engineering design reviews. A list of assembly planning features were obtained through a requirement gathering session with professional design engineers. Within the list, three primary categories of features were identified: Component Manipulation, Assembly Guidance, Spatial Awareness.

Fig. 1.
figure 1

Assembly planning features. (a) Component browsing interface, (b) ‘Bounds’ interface for anchoring

Full size image
Table 1. List of assembly planning features
Full size table

Component Manipulation. Being able to manipulate assembly component parts is critical in testing and planning out possible manoeuvres for assembly. A 3D browsing interface (Fig. 1a) enables the user to spawn and hide component objects to allow the user to bring parts into the assembly area when the user requires of it. A toggle button from the ’Hand Menu’, the user will be able to toggle between freehand bi-manual manipulation and single-axis rotation.

Spatial Awareness. Spatial awareness is critical in developing a useful MR application, especially within the context of an engineering design review. Spatial anchoring was added such that the virtual assembly model would be able to adhere to a specific location in the real physical environment to improve transfer of knowledge and correlation. This anchoring also helps in visualising the 1:1 scale of the model in real life. To provide a frame of reference for assembly orientation, the Orientation Indicator rotates along with the user in relation to the anchored assembly. Upon spawning a component, an arrow will appear around the component pointing towards the direction of its position within the anchored assembly.

Assembly Guidance.

To provide assembly guidance, a list of visual and audio prompts were introduced to assist during assembly planning. Upon manipulation of an assembly component, a translucent copy of the same component appears in the position of where the component should be within the anchored assembly. Component-To-Component collision is an effective tool to definitively validate assembly manoeuvres. To simulate this interaction, the components change colour from a base colour of white (indicating no collision) to blue (indicating collision). Audio feedback emphasises any application events such as button presses, component collisions, and manipulation.

4 Implementation

Fig. 2.
figure 2

Vehicular heavy lifter with 13 parts (13 parts). (a) View of assembly components with orientation indicator within MR view, (b) shows a completed view of the assembly, (c) shows an exploded view of the assembly

Full size image

All assembly planning features and interactions were built using gestures and interaction components within Microsoft’s Mixed Reality Toolkit (MRTK). The application was developed using Unity Engine and deployed and tested on Microsoft Hololens 2. For the CAD models, the assembly model used were designed and tested through SolidWorks and converted into a Unity-compatible format using Blender. The subject (Fig. 2) is a large assembly of a vehicular heavy lifter (Dimensions: 3.5 m \(\times \) 1.5 m \(\times \) 2 m, Actual Physical Assembly Weight: 4 tons).

Fig. 3.
figure 3

Component collision collider implementation. (b) (From Left to Right) Box collider and the generated non-convex collider. Base component with box collider. (b) (From Left to Right) Two solid assembly components in varying levels of collision (No collision, colliding)

Full size image

The user is able to carry out several basic interactions for manipulation and rotation through the attachment of ManipulationHandler script for input detection and handling. For single axis rotation, the Bounds component provides a visible interface which users could interact with to pivot the model along its axes. To simulate collision, a non-convex collider is generated and attached to each assembly component upon spawning. Upon contact between components, a collision event will be triggered which will modify the colour of the material of each colliding components. To avoid false collisions caused by overlapping and jagged colliders, a user-defined parameter was introduced to define a margin of error that the colliders could overlap before triggering a collision event Fig. 3.

5 Evaluation

To evaluate the effectiveness of the application as a platform for assembly planning, two preliminary studies were conducted. First, the relative usability and sense of flow of the application was assessed by a group of users with little to no existing expertise through a pilot quantitative study. Second, a pilot qualitative study was conducted on a group of senior engineers to evaluate the effectiveness of the application for conducting a an assembly planning.

5.1 Pilot Quantitative Study and Results

In this study, participants (n=7, Age: 20–45) who have no prior experience in mixed reality and engineering are required to assemble the vehicular lifter through the use of all available assembly guidance. At the end of the assembly session, the relative usability [1] and perceived flow [3] of the application were recorded.

The usability results from the experiment in Table 2 suggest that the interface is empirically usable (M=56.07, SD=7.89). Given the lack of prior experience in engineering or mixed reality, this could suggest that more visual instruction may be necessary for first-time users to navigate the application. Findings also show in Table 3 that the participants felt that they were clear about their goals (M=12.29, SD=1.60), had a sense of control when experiencing the application (M=11.57, SD=1.51), and their thoughts ran somewhat smoothly with minor difficulty in concentrating (M=17.00, SD=2.16). However, the participants felt that there was too much challenge in understanding the interface and that they feel that they were somewhat competent (M=13.1, SD=3.3). Despite this, most participants reported they enjoyed the experience (M=6.57, SD=1.13).

5.2 Pilot Qualitative Study and Results

A pilot think-aloud was conducted on senior design engineers (n=3, Age: 35–50) to assess the feasibility of the features developed in the application. The participants had 8 to 10 years of experience in coordinating assembly planning and conducting design reviews. The participants were required to assemble the heavy vehicular loader. A thematic analysis was conducted upon the think-aloud and semi-structured feedback by the participants.

During the walkthrough, participants mentioned that finger-pointing and pinch-tap as a tool for basic manipulation ‘feels’ natural however compromises on accuracy that is needed for more complex manoeuvres. While there were visible assembly guidance prompts, participants suggested that more guidance is needed when conducting an assembly in a mixed environment. While collision exists in real life, other physical information such as assembly tolerance and sub-assembly information could be useful. Overall, participants mentioned the suitability of the application for macro-structure assembly planning and highlighted its potential for visualising and validating possible sub-assembly interactions involved without the need for physical prototyping.

Table 2. System usability scale scores (n=7)
Full size table
Table 3. Flow state scale scores (n=7)
Full size table

5.3 Evaluation Summary

Quantitative results show that while the interface can be initially challenging, users were eventually able to find enjoyment and a sense of control when navigating. For engineering professionals, this could hint at a steeper learning curve when first introduced to concepts that require specialised knowledge in an unfamiliar environment. Qualitative results suggest that engineering professionals find part collisions and superposition of virtual assembly parts within a physical context as a useful and cost-effective tool for conducting an assembly design review. A further study could be conducted on a larger population of engineering experts which would yield greater accuracy in understanding the effectiveness of the application.

6 Conclusion

The developed application is designed to enhance the efficiency of mechanical design engineers when conducting DFMA. MR presents a compelling platform for testing and visualising complex virtual assembly models and component interactions onto physical space without the need for prototyping. Results from the preliminary studies reveal that the application is usable and accessible despite a learning curve. Feedback from design engineers highlights that they foresee a potential for using MR in improving their DFMA process. Future work could involve exploring a multi-user experience for assembly planning to correlate better with the nature of assembly planning in a group.