1 Introduction

Augmented Reality (AR) enables interaction between virtual images and real objects, in order to align in real-time the virtual and the real domain [1, 2]. Differently, Virtual Reality (VR) enables users to immerse in a synthetic environment that reproduce the real world without actually integrating the interaction with the real.

Many applications of both Virtual and Augmented Reality systems, while demonstrated to be powerful tools for training humans to perform tasks, are expensive or dangerous to duplicate in the real world without extensive use of advanced computer graphics [3, 4]. Promising domains that could benefit from the use of AR technology are engineering, entertainment and education. In [5], an attempt to improve design reviews in the domain of industrial design engineering used prototyping and recording in parallel by means of the I/O Pad to enrich physical prototypes with additional information (color, features) that can be altered in situ, while all users can observe and interact. In [6], augmented reality (AR) technologies have been employed in education research integrating teaching and AR into a library’s learning environment in order to enhance student learning performance and motivation. In [7], thanks to VR simulation, gaming and scenarios, a new product design method has been validated giving different stakeholders (non-designers) such as users, production engineers, marketing managers, maintenance workers, a proactive role in the design process. This paper presents an interactive tool based on combined AR/VR technologies aimed at enabling unskilled users to efficiently complete a task, i.e. the assembly of parts.

Through AR, a closer contact between the user and the external environment is realized by adding virtual images to a real world scene and immersing humans. AR applications have hardware requirements that can be synthesized adopting a camera, a display and a marker (that can also be embodied by a referenced picture of the area in which to operate). Eventually, advanced devices like special glasses (i.e. Hololens) can be used. Synchronizing virtual objects with the external environment in a real-time and interactive way and detecting the relative position between the observer and the external environment using a marker forms the basic idea underlying AR applications. In particular, a marker can be a geometrical symbol with sizes and structures already known, and AR applications are framed to identify the relationships inside reference systems. The marker in the scene can be detected by fast algorithms that can also be used to recognize its corner and edges, and finally inner content. In this way, a calculation of the \(4\times 4\) transformation matrix between the camera reference system and the marker reference system (homogeneous coordinates) can be carried out. Finally, a 3D object to be added to the scene can be referenced in the correct position of the scene, and then the 3D object position is coherent with changes of position and angular orientation of the camera. A more detailed description of these algorithms can be found in [8]. It is worth noting that also referenced images of the external environment can be used as an alternative to a physical marker. This technique is called “Markerless AR.” Since the introduction of AR in literature, several studies have been carried out to improve the method, to find new applications, and to test it in both industrial and every-day-life environments. From an industrial perspective, the activities which can mostly benefit from AR are: maintenance [8, 9], manufacturing [10], aesthetic evaluations [11], design check for ergonomics [12], and assembly [13]. The integration of AR in these fields can help close the gap between product development and manufacturing tasks, mainly because of the ability to reuse and reproduce digital knowledge while supporting assembly workers [14]. For studies on AR assisted assembly tasks, the literature proposes several approaches. Baird et al. [15] did some early studies on how assembly tasks can benefit of the support of AR technique. Caudell and Mizell [16] proposed one of the first industrial AR applications for assisting operators in assembling aircraft wire bundles. Reiners et al. [17] developed an AR prototype to help operators with the assembly of automotive door locks. However, the work proposed requires a considerable amount of training prior to gain any perceptible benefit because of its instability and complexity. The AR technology has also been applied in assembly processes such as the sequence design, which aim to optimize the use of resources. The work by [18], highlighted the impact of including expert knowledge into computer processes that automatically produce assembly steps, define the sub-assemblies, and identify base assembly parts. Makris et al. [19] proposed an AR system to support both assembly sequence and the generation of human assembly instructions. The evaluation of application confirmed that its use can be effective for both production engineers and shop-floor workers. Similarly, Wang et al. [20] introduced a two-step decision support framework based on the use of AR devices: assembly sequences are acquired automatically by analyzing the manual process carried out by human operators, in a first step; then, in a second step, the framework acquires precedence constraints and evaluates feasible assembly sequences by analyzing the disassembly process. Another successful application of AR in assembly lines is in providing dynamic information to shop floor operators during the assembly process. [21] and [22], proposed the use of AR tools to support the virtual training process in an assembly line. In the work by [23], a Virtual Interaction Panel based on AR devices was introduced to stream assembly information to line workers during the assembly process of toys. Similarly, Kollatsch et al. [24] developed an AR system to share instructions to human operators in an assembly line. An assembly procedure design based on AR with marker and markerless has been described also in [25], while the monitoring of a whole industrial assembly line with the support of AR technique is presented in [26]. The benefits of the integration of AR with real-world assembly have also been presented by similar studies in other domains, including laser printer maintenance [27], furniture assembly [28], and medical assembly [29]. Useful indications about how to develop an AR based application to support maintenance can be found in [30]. Literature suggests that further studies and methodologies must be investigated and evaluated since developing a user-friendly and effective environment to support the assembly in AR is not a trivial task. Another problem is related to the preparation of the models to be used in AR. A trade-off between model detail and computational weight should provide realism with real time capabilities in elaborating the added scene. This paper is organized as follows. Section 2 describes the layout of the environment we developed to support assembly task in AR. Section 3 describes a case study developed to validate the methodology based on the AR assisted assembly of a scale model. Section 4 includes some brief comments on the advantages and drawbacks of the methodology followed by a conclusion.

2 Methodology description

2.1 Motivation

AR based user manuals can be developed for industrial applications, but they present some critical challenges. The assembly sequence is habitually planned by a skilled designer and can be difficult to follow by an inexpert user. The setting of the step-by-step simulations for the assembly sequence in AR can be a time consuming and painstaking activity, especially with complex groups made by a large number of parts. The assembly sequence does not consider the way in which a user grips the part to be assembled, but typically shows only the position of the part itself. A too complex assembly sequence can be boring for the final user with the risk of overlooking the virtual instructions and forging ahead based on their own experience. Due to these limitations, a new methodology to design assembly sequences and to draft a user manual has been developed and will be described in this paper.

The new methodology to design assembly sequences has the following aims and steps shown in Figs. 1 and 2.

Fig. 1
figure 1

The principal goals of the guided methodology to perform the assembly

Full size image

To reach the goals listed in Fig. 1, the work will take place in the following steps:

Fig. 2
figure 2

The steps required for the virtual user manual fine tuning

Full size image

2.2 Step 1: Hardware setup

In the first stage of the procedure, it is important to create an efficient hardware setup, including the following elements:

  1. (1)

    Camera with webcam @30fps

  2. (2)

    Personal Computer

  3. (3)

    Marker on a reference plane

The camera captures the images from the reference plane and projects them to the monitor of the Personal Computer. Images become interactive using a marker (Fig. 3).

Fig. 3
figure 3

Hardware setup

Full size image

2.3 Step 2: Software setup

After the hardware setup, it is also necessary to implement a Software Setup. In fact, for each operation to be carried out, specific software must be chosen and integrated. As shown in Fig. 4, the following assignments are considered optimal:

  1. (1)

    AR Toolkit for Camera Registration

  2. (2)

    Optical Flow Lib for Motion Tracking

  3. (3)

    Solid Edge for Cad Modeling and Assembly

Fig. 4
figure 4

Software setup

Full size image

2.4 Step 3: Alignment between virtual environment and real objects

In the third step of the procedure, it is necessary to perform the alignment between the virtual and the real object environment. For this purpose, it is necessary to combine the AR Toolkit software and the physical marker(s) put on the reference plan. This operation allows the synchronization between Real World and Virtual Coordinate SYS (Fig. 5).

Fig. 5
figure 5

Alignment between virtual environment and real objects

Full size image

2.5 Step 4: Components recognition

For the components recognition (Fig. 6), the definition of INITIAL POINTS is fundamental. Then, it is possible to identify each part related to them, for example using identification code. Figure 6 below depicts the case study example explained further below.

Fig. 6
figure 6

Components recognition

Full size image

2.6 Step 5: Motion tracking development for assembly of parts by end user

The fifth step of the procedure covers the phases of motion tracking (Fig. 7) that can be synthesized as:

  1. a.

    CAD Assembly Sequence Computing or Retrieving

  2. b.

    Visual aids and verifications

Fig. 7
figure 7

Example for motion tracking

Full size image
Fig. 8
figure 8

The methodology layout

Full size image
Fig. 9
figure 9

The methodology synthesis

Full size image

This step combines Virtual Reality procedure with typical Augmented reality process. These operations are developed to produce an AR based User Manual for Assembly (ARUMA), with some specific aspects to support the users in complex tasks. The AR manual contains written text and simple animations to be added to the video streaming of the assembly zone.

The sequence and assembly trajectories performed by the end user are recorded and compared to the designer procedures. Differences in the assembly parts sequence and trajectory are noted and can help improve the efficacy of the user manual employment. In this way, the AR user manual can be iteratively enhanced to overcome any weak points of the procedure, provide more information where needed and avoid confusing the user with trivial parts of the assembly. A Knowledge Based Engineering system could be integrated further to provide troubleshooting for the most frequently occurring errors in the assembly sequence. Once the user manual has been fine-tuned, the user can employ this interactive tool to perform the whole assembly.

This procedure immerses the user in the environment in a more realistic way than a traditional screen, and it is helpful in evaluating relative dimensions of the parts and trajectories.

The markers are placed upon the reference plan, near to the zone in which the model will be assembled. In this way, a virtual representation of the part to assemble can be viewed in real scale to check the real physical assembly in different stages. A second marker can be placed to support the assembly of subparts or groups. Moreover, by visual tags, the AR manual provides hints on how to move the parts from the boards to the growing assembly. Instead of tracking the position and orientation of the part to be assembled with markers glued to the part or pencils equipped with InfraRed sensors (Wiimote or similar), the tool we propose tracks the position of the user’s fingers through devices like the Leap motion tracker [31]. This procedure can avoid problems related to the masking of the part to assemble (or the partial hiding of markers/IR sources with the loss of tracking). Moreover, the proposed tool is more user-oriented since the hand motion is strongly correlated with the positioning and orientation of the part in space. According to literature, Optical Flow techniques [32] have been proven effective in guessing the trajectories of objects in the three-dimensional space by comparing the positions of key points in images from a video streaming and can be considered a mature technology [33] which can be integrated in the AR guided assembly methodology. Figure 8 provides a graphical representation of the methodology layout.

The methodology has been applied to some case studies to check feasibility and verify the possible implementation. One of these tests will be presented in the next paragraph to better illustrate the procedure.

2.7 Step 6: Methodology synthesis and extension

The methodology can be synthesized in the following scheme (Fig. 9). It is important to also assume that it could be extended (Fig. 10) to more complex systems.

Fig. 10
figure 10

The methodology extension

Full size image

3 The case study

The case study presented in this paper deals with the assembly of a simple scale model of a car produced by Tiger [34]. Figure 11 shows the car model features and the boards with plastic components to be assembled. As Fig. 12 depicts, all the parts have been named and coded according to the board to which they belong. Figure 13 provides an idea of the first stages of the methodology. The assembly sequence is developed within a CAD system by an expert designer and by an unskilled end user to evaluate if there are differences between the approaches to the assembly procedure and to highlight working critical phases.

Figure 13 depicts, for instance, the CAD simulation of the assembly procedure of the wheels, which are made by a rim and a tire. After the detection of the most critical phases, an AR assisted user manual is implemented as shown in Fig. 14.

With reference to Figs. 12–1, two markers are placed on a flat surface: one L- shaped to reference the assembly, and another one chessboard-shaped to support the assembly of sub-groups. A device equipped with a camera and a screen, like a tablet or a PC, frames the assembly zone and virtual writings or objects are added to the video stream. In frames (1) and (2) the user selects from the “A” board the A1 part, according to the virtual instructions overlapped to the video stream of the assembly zone. In (3), the A1 chassis is placed close to the L-shaped marker. The finger position of the user grabbing the board is tracked, so that an arrow suggesting the hand move to follow appears. In sequences (4), (5) and (6) included in Fig. 15, the AR based user manual suggests to select the front wheel rim and indicates the movement of the hand required to place it close to the chessboard-shaped marker.

Fig. 11
figure 11

Scale model car parts and box

Full size image
Fig. 12
figure 12

Boards with parts (arrows point to parts label)

Full size image
Fig. 13
figure 13

CAD simulation of the assembly procedure

Full size image
Fig. 14
figure 14

AR based user manual: operations from 1 to 3

Full size image
Fig. 15
figure 15

AR based user manual: operations from 4 to 6

Full size image

This is necessary to assemble the front wheel sub-model (coded with H1) which is made by two parts: the rim (part coded with A4) and the tire (B2). Virtual written instructions suggest the operations to follow and new ones are projected once the end user confirms the end of a phase, as frames (7), (8) and (9) suggest in Fig. 16.

Fig. 16
figure 16

AR based user manual: operations from 7 to 9

Full size image

Once the sub-group H1 has been assembled, the AR based user manual shows the procedure to assemble the H1 group onto the main assembly (P1). Frames (10), (11) and (12) of Fig. 17 show the virtual written instructions suggesting the next operations to follow and the arrows representing the movements of the end user hand, required to move the H1 assembly from the chessboard marker to the P1 assembly. The step-by-step assembly sequence goes on until the final model is achieved.

Fig. 17
figure 17

AR based user manual: operations from 10 to 12

Full size image

4 Discussion and conclusions

New digital devices have been introduced to support design and sketching [35] aimed at increasing the capability provided by traditional drawing on paper sheets. In a similar manner, the use of Augmented Reality based user manuals technology will increase in the future to support the maintenance of complex systems in a more effective and exhaustive way than paper based supports. The simple case study presented herein should be considered a test to validate a more general methodology to be applied in more complex design scenarios. The innovative tool proposed in this paper does not select the correct part to be positioned in the assembly. In fact, it would be quite complex to detect single parts of a large assembly. Instead, it provides hints on the movements of the hands assuring that a required part, lying in a known position, could be moved from a start point to a final destination, following a suggested trajectory in the correct way. Thus, it is possible to perform the assembly without a real awareness of the shape or function of the part itself. Care should be given to counter intuitive assembly steps or phases which can be confusing, and the Augmented Reality animations would require a long time to prepare in order to support in a detailed way all the building phases of the assembly. The use of more than one marker allows preparation of sub-groups or temporary “parking” of components requiring operations on the main assembly. A benefit of this approach is that there is no need for a marker directly glued on the parts to be assembled. An optical flow based strategy can be exploited to guess the position of the parts by simply tracking the position of the user’s hand in the space. The unskilled user is not aware of the selection of the correct part during the assembly. In this “lean” approach, the user knows the starting point of the part to be assembled on the board and its final position. Only the user’s hand trajectory is suggested.

This innovative method is centered on a series of steps targeted to overcome typical problems usually illustrated by current literature. The assessment in CAD environment concerning the assembly sequence developed by an expert designer and the end-user can contribute to perceiving the most critical points of the process. In order to reference the virtual model of the assembly (or sub-groups) with the real parts, markers are used. By displaying the partially assembled object in its right location, the AR environment is able to track the end-user hand movement. In this manner, the AR manual proposes a trajectory to be trialed by the hand rather than simply proposing a location or alignment of the part which can lead to counterintuitive motions. A tangible advantage of this approach lies in the possibility of hiring unskilled users to perform accurate tasks with evident savings of resources such as money and time. A case study showing some steps of the assembly of a scale model of a car is provided in order to supply the reader with knowledge of the process requirements. The method has been preliminarily implemented for a simple case, but it has the potential to be extended to more complex industrial scenarios in order to better check its efficiency and effectiveness. Promising enhancement of the method for an interactive user manual could be the elaboration of the conceptual design by means of design methods based on the pursuit of quality and innovation (QFD-TRIZ) [36]. These methods allow improving the efficacy of the product, in terms of increasing the success of the task, based on the users’ needs and oriented to an inventive problem solving process.