Keywords

1 Introduction

Additive manufacturing (AM) is a process to fabricate parts or products from a 3 dimensional (3D) digital model by depositing the material in layer by layer custom. With the aid of computer-aided-design (CAD) technology, design of a complex 3D structure can then be transformed into a digitalized data for computerized printing [1]. Manufacturing that incorporates AM aims to minimize the cost of production results from customized fabrication of tailored-made products. Progressively, the technique allows fast printing of complex structures with lower material wastage and subsequently improves the yield of production [2]. Till date, AM has gained much popularity since its first inception in the 1980s. Nowadays, the technology can be found in application such as medical [3, 4], electronics [5], mechanical assembly [6], automotive industry [7] and so on.

Despite the appealing image of AM in the era of industry 4.0, the market is consistently challenged by the continuous changes of customer demand [8]. Generally, these changes are unpredictable and varied through time. Although characterization can be performed to model the specification of customer, mapping the data to the specific production batch for customized fabrication can be very complex. Moreover, in addition to the competitive and fast-growing market, manufacturers are in short to coordinate their manufacturing platform to match these changes in real time. Hence, there is a need to standardize the manufacturing process to be responsive to changes in customer demand and ultimately ensuring customer satisfaction.

Augmented reality (AR) is an emerging technology that creates visual perception of physical object in real-world environment via computer techniques. Particularly in manufacturing, AR helps improve customer satisfaction, creates a much interactive user interface, assist in understanding complex manufacturing operations and also improve efficiency of production plant [9,10,11,12]. Recently, researchers have demonstrated a working AR to output context aware information to support maintenance and assembly process [10, 13]. The proposed AR interface is able to improve efficiency of the manufacturing process by means of adaptive knowledge transfer to the worker in real time. Equally, if the information is readily accessible and visualised to customer, uncertainties to the desired specification of the finished product can be managed prior to actual fabrication. This enables the fabrication of customized product in batch production settings and in addition reduces the expenses needed for a similar fabrication in a customized manufacturing plant. On the other hand, with the augmented visualisation of product, customer can then customize the process of additive manufacturing based on own preference and thus improve customer satisfaction.

In view of this, development of an augmented interface to visualise AM process of personalized product is both strategic and advantageous. Therefore, the objective of this research is to design and create a user-oriented augmented display to interface AM for fabrication. Specifically, firebase, a mobile development platform is integrated to the proposed AR for remote data retrieval of AM. In this paper, an implementation of cloud database integrated AR to personalize product is presented. At the same instance, the virtual reality system serves as a visual interpretation, simulation tool as well as modeling platform to characterize 3D prototyping of customized product.

The remainder of this paper is structured as follows: Sect. 2 outlines the related works; Sect. 3 describes the initial implementation of the augmented reality system on Ender 3 3D printer; Sect. 4 demonstrates an evaluation of the augmented reality system in a real-world application; Sect. 5 draws the conclusions, limitations and future improvement of the work.

2 Related Works

The term “augmented reality” was first coined by Caudell and Mizell when their developed an experimental AR system on wiring harnesses [14]. Throughout the decade, it can be observed that AR has been increasingly used for researches in manufacturing industry. In research, the group of A. Y. C. Nee and S. K. Ong has been actively committed to the development of AR system. In 2012, Zhu et al. [15] have developed an authorable context-aware AR system to aid maintenance operator. Using marker based tracking method, the AR content is generated on mobile user interface in real time. Overall, the proposed system enabled the operators to update the rendered AR content on-site and successfully implemented in a CNC milling machine a year later [16]. During the same year, Wang et al. [17] from this group have introduced an interactive AR interface to aid manual assembly of knuckle joint. To augment the assembly sequence accordingly, the group adjust the orientation of the virtual joint based on the orientation of hand movement. This omits the necessity of an auxiliary computer-aided design (CAD) information and reduce the preparation time upon actual assembly. After another year, Fang et al. [18] have presented an AR interface to plan robot path. An Euclidean distance based method has been developed to interface human command to virtual robot in AR environment. Till recently, Wang et al. [9] demonstrated an ubiquitous AR for shop floor planning. In this work, genetic algorithm has been used for dynamic task scheduling of shop floor in real time. The proposed AR system will assign maintenance task to on-site operator should the machine experience breakdown.

Aside that, AR solution has been explored to customize design for children footwear [19]. In this work, marked stocking is used for augmenting virtual footwear on the foot of customer. The environment is consistently captured by a KINECT sensor and as such enabled real time foot tracking of customer. Never the less, maintenance is still the major focus for most AR researches [20]. It is found that the augmented content helps reduce the maintenance time and hence leading to an improved efficiency. Interestingly, some researches claimed that by projecting the augmented content on large screen, the augmented instructions managed to help reduce the completion time of maintaining a motorbike engine [21]. Furthermore, some researchers even converted the maintenance instruction as standard symbol for visualisation [22]. Overall, the researches on AR are mainly augmented on a marked object and are focused on overlapping the information on a real time recorded environment background.

This paper focuses on development of an AR system to visualise additive manufacturing. As mobile devices are used in this work, sophisticated setup such as high end camera and headset is omitted. Moreover, since edge detection is employed to track the target object, physical markers that require precise settings can be excluded as well. This work presents an interactive mobile interface featuring two modes of visualisation: first mode is to augment maintenance instructions onto the virtual manufacturing station. Second mode observes a virtual simulation scene for the additive manufacturing process. The development of AR system in this work is performed using Unity 3D and Vuforia for their robust tracking capability. In addition, Firebase is integrated to the AR to augment specific content from the server onto the mobile screen. For simplicity, Ender 3 3D printer is setup as the additive manufacturing station for this work.

3 Results

The work proposes development and implementation of an augmented interface on mobile devices. The augmented virtual model should visualise maintenance, simulation of additive manufacturing process and contain the integration of cloud server for information retrieval. In this work, One Plus 5 was used to run the developed mobile application. The architecture of the proposed system is illustrated in Fig. 1.

Fig. 1
figure 1

Architecture of the proposed augmented reality interface

Full size image

3.1 Target Tracking

The first part of the work is focused on tracking or recognition of target object to be displayed onto the mobile screen. In view of the aforementioned advantages, this paper demonstrated edge detection method instead of the marker based solution. Figure 2 (left) presents the AR scene prior to target tracking. Once the edges set in the AR system matches physical outline of the machine, virtual model of the setup will overlap the actual station in real time. To create a much realistic AR scene, the edge layout is rendered invisible on the AR scene [23].

Fig. 2
figure 2

The AR interface prior to object tracking (left) and after target recognition (right)

Full size image

Secondly, the work focused on creating the virtual object of the manufacturing station. To project the virtual model of the target station, the 3D digitalized content of the machine is required. In this work, Autodesk Fusion 360 was used to design and construct the 3D content of the manufacturing station. Based on Fig. 1, Vuforia engine is integrated to enable object recognition and robust tracking onto the Unity 3D development platform. The virtual model shown in Fig. 2 (right) is the 3D content drawn using Fusion 360.

3.2 Assisted Maintenance

This section visualises the maintenance and servicing of the 3D printer. Generally, the maintenance information is augmented as text on the AR scene to suggest solution for the particular maintenance activities. The instructions provided on the maintenance activities are based on the possible causes of failure and its corresponding resolve action. In cases of breakdown, the activities shown on the augmented instructions should be conducted. Instead of manual diagnostic, the AR system illustrated graphical presentation of the augmented command for assisted maintenance. Since the time for lookup of manual is reduced, the downtime of maintenance is also decreases and hence leading to higher efficiency of additive manufacturing. To improve the interaction with user, the augmented content is presented with symbol, textual and intuitive presentation. While symbol and textual describes the maintenance information as augmented logo and instruction respectively, intuitive presentation allows better understanding of the maintenance instructions with projected graphical content.

4 System Implementation

The intuitive presentation of the maintenance instruction is realized with object transformation of the manufacturing station. To convert manufacturing station from position 1 to position 2, an object transformation matrix is applied.

$$\varvec{O = T}*\varvec{o}$$
(1)

  • O is the new coordinate of object after oriented

  • T is the transformation matrix

  • o is the original position or orientation of object in world coordinate.

The transformation matrix T consists of a 3 × 3 rotational matrix r and translational vector t.

$$\varvec{O} = \left[ {\begin{array}{*{20}c} {\varvec{r}_{11} } & {\varvec{r}_{12} } & {\varvec{r}_{13} } & {\varvec{t}_{\varvec{x}} } \\ {\varvec{r}_{21} } & {\varvec{r}_{22} } & {\varvec{r}_{23} } & {\varvec{t}_{\varvec{y}} } \\ {\varvec{r}_{31} } & {\varvec{r}_{32} } & {\varvec{r}_{33} } & {\varvec{t}_{\varvec{z}} } \\ \end{array} } \right] * \left[ {\begin{array}{*{20}c} \varvec{X} \\ \varvec{Y} \\ \varvec{Z} \\ 1 \\ \end{array} } \right]$$
(2)

The system calculates the destination coordinate of the target object based on the transformation matrix. The outcome of this stage observes the object oriented in the AR scene. This approach omits complicated mapping of marker frame and the need for calibration of high-tech camera.

4.1 Maintenance of Nozzle

The maintenance operation of the nozzle is consisting of 3 steps. Figure 3 shows the maintenance scene of the proposed AR system. The operation involved unlocking the casing and fan cover to reach the nozzle. Graphical content of the parts and servicing steps were projected onto the AR scene. Servicing of the nozzle is essential since the filament for printing often clog at the nozzle and causes breakdown of the machine. In order to validate the developed AR mobile interface, a group of engineers aged within 22–29 years old were requested to perform the maintenance task using the AR system. A survey which scaled from 1 (disagree) to 10 (agree) was given to the engineers and none of them had any experience on using AR for visualisation. Since manual lookup is omitted with the application of AR, the time on performing the task is expected to be shorten and hence increasing the efficiency of the maintenance task. Figure 4 shows the response of participants on the effectiveness of the AR system in improving efficiency of the maintenance operation.

Fig. 3
figure 3

Augmented reality presentation of the maintenance task

Full size image
Fig. 4
figure 4

Response of participants on the effectiveness of the developed AR system

Full size image

Based on the response, it is worth noticing that all of the participants agreed on using the proposed AR system in improving the efficiency of maintenance. It was also found that the participants are more confident on servicing the nozzle using the AR system. In view of this, the developed AR interface is proved to be useful especially in training new staff on the aforementioned task.

4.2 Virtual Simulation

This section presents a virtual scene for user to visualise the additive manufacturing process. Figure 5 presents the virtual simulation scene for the additive manufacturing process. The AR scene allows user to visualise the process from various angles by dragging the screen in accordance to the finger movement. For simplicity, this section simulates the product in shape of a 1000 cm3 sized cubic. The rate of filament insertion in this simulation was set at 6.67 filament/sec and the filament size as well as printing height is of 0.2 mm3.

Fig. 5
figure 5

Virtual simulation scene of the additive manufacturing process

Full size image

The virtual model can easily be dragged into the station for simulation. However, due to the technical setting of the development platform, the 3D content cannot be automatically loaded into the virtual scene, and hence the virtual model to be visualised on simulation should be integrated into the platform in advanced.

Some technical difficulties were faced during the simulation. Foremost, the AR system experiences lagging upon simulation of the virtual models onto the scene which may be caused by the low memory or processing power of the mobile device used. Thus, for smoother operation, only one model (blue cube) is simulated on the AR scene. This section is important as it provides visualisation on the additive manufacturing pathway and the number of steps required to print the product. The information is especially useful to plan and design the manufacturing process with respect to the product requirement. A similar survey which is scaled from 1 (agree) to 10 (disagree) was given to the participants to assess their experience in using the AR interface. From the survey, it can be evaluated that the mobile AR interface is easily used and comfortable for usage to user. The participants also responded that, the AR system assisted them to properly visualise the technical aspect of the fabrication and the desired product feature prior to actual fabrication. Figures 6 and 7 shows the response of participants on the feature and conveniency of the developed AR system respectively. As the AR provides virtual simulation, misunderstanding of product enquiry and wrong fabrication of unwanted product can be prevented. The AR interface is compatible for installation on mobile device and convenient for visualisation at any time. Never the less, further standardization of the AR interface and simulation on sophisticated product is still required in the future.

Fig. 6
figure 6

Response of participants on the appearance and feature of the developed AR system

Full size image
Fig. 7
figure 7

Response of participants on the conveniency of the developed AR system

Full size image

4.3 Cloud Database

In this work, the AR interface is integrated with a cloud database for online information retrieval. Figure 8 presents the context ontology for the cloud database service.

Fig. 8
figure 8

Context ontology of the cloud database service

Full size image

An internet connection is required to access the content in database. For better user experience, the internet address to retrieve the information has been preliminary hardcoded into the AR system. User is required to key in the name of the product material on the search bar in order to activate this operation. The label in the AR scene should match the tag referring to the respective information and output the information onto the mobile screen accordingly. For implementation purpose, only basic information of the product material such as melting point, tensile strength and density were programmed into the mobile application. Once a valid entrance has been keyed, the developed application will automatically retrieve the information from cloud. Figure 9 shows the coding used to retrieve information from the cloud database to the developed AR system.

Fig. 9
figure 9

Code to automatically retrieve information from the cloud database

Full size image

The above work presents an implementation of a markerless mobile AR interface to visualise additive manufacturing. There are two important contributions; (1) assisted maintenance on the 3D printer using the developed AR interface, (2) virtual simulation of the additive manufacturing process. The developed AR allows retrieval of technical information related to the manufacturing process from a dedicated cloud database. The integration is meant for implementing Industry 4.0 into the AR technology. Never the less, further development and research is still required in order to develop and standardize the use of AR in the manufacturing industry. The developed AR is able to visualise the additive manufacturing process of desired product on own try-on. Overall, credits to the mobile framework, the developed mobile AR interface omit the use of expensive camera and sophisticated maker alignment. Based on the survey, the mobile application developed is in agreement useful in terms of improving efficiency and in guiding staff especially to first-time learner. On the whole, the proposed AR system has been developed and the functions have been tested accordingly.

5 Conclusions

In the presented work, AR was implemented to visualise additive manufacturing process. The proposed approach incorporated virtual simulation for user to visualise the production. In addition, database was integrated into the AR for data transfer purposes. As there is a growing interest of manufacturing trend shifting to product customization, the developed AR system may serve as solution to interpret customer demand into product specifications. Based on the evaluation, although AR is not dominantly used in manufacturing, it may prove helpful in the field of design for additive manufacturing. Especially in the consultation stage, the use of AR can help avoid misunderstanding in the product requirement which may lead to additional costs for reproducing the product.

Future work will focus on expanding the feature of AR. Expansion to include new materials such as biomaterials, metals and alloys and their specification in additive manufacturing is anticipated. The development of AR on additive manufacturing of highly customized product for various functions and applications is also part of the research interest. By looking at the system performance, research on hardware which supports the AR is also important. The research on testing and application of AR in additive manufacturing still require further standardization.