Keywords

1 Introduction

With a global market size estimated at 198 billion dollars by 2025 [1], Augmented Reality (AR) is expanding its application field through many areas, including marketing, education, and medicine. Furthermore, industrial training and instructional support, especially in the context of maintenance and assembly, are also among the key field of application [2]. AR applications aim to provide personnel different levels of in situ guidance either from on-site support or remote experts. This technology enables users to manipulate in real-time the virtual objects which are superimposed upon the physical world. AR training and support reduces staff cognitive load, improve performance while minimizing mistakes, therefore, efficiency is increasing. In the new era of Industry 4.0 where products and processes are highly complex and diverse, AR is particularly one of the most essential technologies [3].

“It’s the total experience that matters” is how the usability engineering pioneer Don Norman said about the importance of user experience. Evidence has shown that good user experience and engagement leads to better performance, an engaged employee delivered a better result than those who not [4]. Among various methods to enhance user experience and engagement is the idea of turning the prosaic tasks into more fun and exciting version by borrowing tools and knowledge from the game industry into other domains, so-called gamification. While the academic world is still debating on the consensus of definition and scope, hereby we agree with the most widely accepted definition from Sebastian Deterding [15]:

“Gamification is the use of game design elements in non-game contexts.”

In this work, we combine the two techniques to create a gamified AR training for an assembly task. Our main hypothesis is that users will significantly prefer the gamification version of the training to the non-gamification one. Moreover, we expect a significantly better performance in the gamification training over the non-gamification.

The paper is structured as follows: Sect. 2: we provide the related work. Section 3: we describe the design of the application as well as gamification design. Section 4: we present the experiment design. The result of the study is presented in Sect. 5, followed by a short discussion in Sect. 6. Lastly, Sect. 7 concludes the paper.

2 Related Work

Although the term “gamification” is relatively new, since around 2003, its applications have already widespread across many industrial as well as scholarly fields. Recently in the Gamification 2020 report, Gartner predicted that gamification in combination with emerging technologies will create a significant impact on several fields including the design of employee performance and customer engagement platform [13]. In this context, there are numerous examples of studies for either AR training or gamified training, yet there was hardly any work on the combination of those.

A recent survey of Seaborn et al. [19] provides a good overview of gamification from a Human-Computer-Interaction perspective in both theoretical and practical lights. The work showed that gamification is primarily practiced in the domain of education, e-learning especially. In the theoretical foundations, there was a dynamic movement towards carving the boundaries between gamification and other similar concepts. The applied research, meanwhile, painted a positive-leaning but mixed picture about the effectiveness of gamified systems. Despite usual expectation, similar gamified designs under different settings returned clashing result over user experience along with performance. The reason was believed to be highly context-specific requirements. Furthermore, learning about the effects of gamification on the human is a complicated subject. The overall effort toward this direction is still nascent.

While the gamified system was well accepted in business contexts, it is not necessarily the case in production training, left alone Augmented Reality training. Lee [18] showed that AR for education and training innovation was leaning towards the “serious game” pole while gamification was left outside of the picture. According to Lee, AR games were particularly interested in by both “educators and corporate venues.” A role-playing game for teaching history [16], for example, proved the benefit of enabling students for problem-solving, increasing collaboration and exploration via the virtual identities.

However, whether we like it or not, production training is different from traditional classroom training. When transforming the operational work into a game, a serious game, there will always be a risk of taking the focus away from the task at hand. This is when gamification comes to play as integrating gamification can provide the fun aspect while still keeping the workers’ full attention on the operative job  [17].

Probably the most well-known gamification in production is a series of works from Korn et al. [17, 20,21,22]. The center of his works is to evaluate users’ acceptance of gamification in modern production environments. Different designs, “Circles & Bars” and “Pyramid,” were proposed [17]. Both designs were used to visualize work steps as well as their sequences. Color-coded from dark green to yellow, orange and read is employed to indicate user specific time progression. Later on, they were projected into users’ working space as an assistive application for impaired individuals. The result indicated a good acceptance level for gamification designs and the “Pyramid” approach was favorable in general. While the study showed a promising outcome, it focused on user acceptance and did not measure the quantitative factor of gamification on task completion time and error rates.

3 Implementation

In this section, we present the implementation of the application under study. A process of replacing the battery for a robot arm was implemented based on the instruction manual of the Mitsubishi Industrial Robot RV-2F Series. Two prototypes were made, one with the gamification design and the other without. The designs were named Gamification AR (GAR) and Non-Gamification AR (NGAR) according to their characteristics.

The application ran on the Microsoft HoloLens (Fig. 1). Microsoft HoloLens is a standalone mixed reality device which showcases a field of view of 30 degrees by17.5 degrees. Due to Microsoft HoloLens small field of view, here we provide the user interfaces captured from Unity Editor to showcase the whole scene setup. Figs. 3 and Figure 2 illustrate the GAR and NGAR design respectively.

Fig. 1.
figure 1

HoloLens - the mixed reality head-mounted display (HMD) from Microsoft. [23]

Full size image
Fig. 2.
figure 2

The NGAR design with no gamification elements. Only text instruction was provided. [5]

Full size image

3.1 The Application

The process for changing the battery was identically built for both prototypes. There were 21 actions made up 10 steps. Disassembling the cover of the battery compartment, for example, included two steps of removing the screws and removing the cover. While removing each of the screws was counted as an action.

For navigating the process, we augmented the instruction text for each step as a head-up display which was always facing the user at the top right corner of the user view. An instruction manager was used to control the flow of text visualization. The requirement from the instruction manual specified that the steps of the process had to be performed in a fixed order that’s why only one instruction was displayed at a time. The next instruction triggered when the user carried the current step correctly.

Two main interaction types were used to simulate different interactions. Air tap [23] was used for interacting with static objects (e.g. pressing a button) while we utilized drag and drop for assembling actions (e.g. removing the screw). Similar to the real working space, disassembled objects were designed to be placed at a specific location. For instance, the screws needed to be placed inside a designated tray instead of dropped on the floor.

To simulate a sense of reality, sounds such as robot arm were running or turned off were used.

3.2 Gamification Design

The game design elements were implemented only for the GAR version. It allows to isolate and analyze the effect of gamified system on the user. This could be reflected by comparing the outcome of the two experiments.

Fig. 3.
figure 3

The GAR design with gamification elements: points, progress bar and signposting. [5]

Full size image

As a result of Korn’s investigation [17], gamification in the production environment has its own specific requirements. To avoid resistance from users or the potential of taking away their main focuses, we followed the identified requirements in designing gamified application for production settings. First, “keep the visualization of gamification simple.” This focuses mainly on avoiding animation, moving elements and using complex graphical structures. The second and third requirements come together as “avoid explicit interaction with gamification elements” and “support implicit interaction with gamification elements.” For that matter, in our designs we did not ask for any user’s effort to direct input or reach out to the gamified items.

Point System. The point system was built based on users’ actions. There was a maximum of 21 points according to 21 actions. Points were rewarded to the user when the action was done. As the first attempt to study the effect of gamification design on user engagement, we did not implement a complex point system with losing points or rewarding extra points at this stage.

Progress Bar. While the points were based on actions, progress bar visualized the steps. As stated as one of the requirements, the user interface was intentionally kept simple with only one color. Additional text was in place for indicating the percentage.

Signposting. Signposting aims to direct the user in the right direction. While users without background knowledge could be confused with the mechanical part names (e.g. Controller box), signposting highlighted the part corresponding to the currently displayed instruction. It provided the “just-in-time” hints for the trainees, especially the totally beginner one.

Sound System. Audio cues are employed to exemplify achievement. A coins collect sound effect plays simultaneously to acknowledge that users receive a point. It also indicates that users have just finished a step.

4 Experiment Design

The experiment was conducted to investigate how gamification in AR training impacts user engagement and performance. The studies for both conditions (GAR and NGAR) took place in the same room at our research laboratory. To avoid the learning effect, we employed the between-group design in which each participant randomly exposed to only one design, either GAR or NGAR.

Due to the fact that Microsoft HoloLens requires specific hand gestures for interaction, the participants were asked if they have experience with this device. In the case of none, the participant used the default HoloLens “Learn gesture” application. This was especially important because the main task could not be carried on without this step. Before the experiment, regardless of the HoloLens experience, we repeated the main information about the interactive gestures to all participants.

Once the participants were confident interacting with the device, the main experiment task proceeded. When the user hit the “Start” button at the first scene of the application, the timer for measuring task completion time was started until the last step completed.

As we focused on the user engagement we used a post-study questionnaire with the refined User Engagement Scale (UES) [24]. UES is a five-point rating scale: strongly disagree, disagree, neither disagree nor agree, agree and strongly agree, respectively from 1 to 5 point. Given the task was not complicated, the level of fatigue after that was expected not to be high so that we decided to use the UES long form (UES - LF). The UES - LF consists of 30 items covering 4 factors:

  1. 1.

    FA: Focused Attention

  2. 2.

    PU: Perceived Usability

  3. 3.

    AE: Aesthetic Appeal

  4. 4.

    RW: Reward Factor

As constructed in the guide to use of UES, all items were randomized and the indicators (e.g. AE.1) were not visible to the users.

5 Results

Most of the participants reported having little or none experience with AR technology, in particular, Microsoft HoloLens, before this experiment. So, a potential novelty effect when initially establishing interaction with new technology might influence the research result. The test population was 22 participants with 11 regarding each condition. Participants ages vary from 18 to 34 years old, 15 male and 7 female subjects. Although some unease and uncertainty were expressed at the beginning, all participants were more certain after the learning gesture phase.

Figure 4 displays that the GAR design was rated better in all sub categories. In general, it was clearly preferred to the NGAR approach. The overall Engagement score was 15.2 (SD = 1.8) in GAR and 13.3 (SD = 3.5) in NGAR. However, this did not make up a statistically significant difference between the two groups. Table 1 provides the results in more detail, looking at the average score, standard deviation and also the result of a t-test for both the overall engagement score and its factor.

Fig. 4.
figure 4

User engagement score as a bar chart with indicated standard deviations. [5]

Full size image

The standard deviation in the overall user engagement score was much lower in the GAR design (SD = 1.8), versus SD = 3.5 in NGAR, which shows that the GAR subjects more homogenously perceived the result throughout the group. This tendency, lower standard deviation, remained true for all four subfactors in the GAR design as shown in Fig. 4. On the other side, the opinions of NGAR subjects seem to be more diverse.

Looking at the training performance, the difference regarding average task completion time (in seconds) between the two study conditions is statistically significant. The t-test resulted in . The average time was 306.9 (SD = 123.2) and 439.5 (SD = 134.4) for GAR and NGAR groups respectively. This positive outcome probably directly influenced by the signposting design element.

Table 1. Comparision of user engagement score
Full size table

6 Discussion

Besides the required surveys, participants often complained about the cumbersome of the hardware. Even though Microsoft HoloLens is one of the mixed reality device market leaders, it is still heavy for constantly wearing. A missing of ergonomic design makes it difficult for the device to stably seating on user’s heads, especially those with small heads. To interact with the device, it is mandatory to learn the hand gestures. These fix hand gestures are not intuitive, as reported by participants, leads to the result that users often forget them along the way.

As a preliminary result, this work demonstrates the potential of gamified AR training for assembly tasks in improving user engagement and performance. Nevertheless, there is a need for further investigation focusing on both short-term and long-term training effectiveness. A consideration over skills and knowledge acquisition should be taken into account. To serve this goal more complex tasks should be implemented with a higher level of gamification, different training levels and challenges design for individual specific demands for example.

As we focused on the improvement of user engagement in gamified AR training, we did not take in to account the isolated effect of how each game design elements affects the user. As mentioned in the Related Work, gamification design is highly context-specific so that the next important step will be a qualitative study on how the users perceive different design elements and their impacts.

Points, Badge and Leader Board are the most common elements of gamification, one of the reason is due to their ease to implement. The use of these external rewards recently raises an ethical question in academia. When a user is exposed to external rewards for a long time, disappointment and frustration may appear once these elements are absent. Turning the research direction to investigate the use of implicit motivation is believed to provide an enduring effect on user engagement. However, how to design a meaningful and relevant user experience requires great effort and long-term design studies.

7 Conclusion

As the Literature review reveals a shortage of using gamification in the modern industrial environment, our goal is to fill the gap. In this project, we developed an assembly training system using gamified Augmented Reality. Firstly, we created two versions of the same training, with and without game design elements. The training guides novice users through a process of changing a robot arm battery. Then, we reported a study over 22 participants with none to limited knowledge of this assembly task. The study’s goal is to confirm our hypothesis if the gamified training creates better user engagement and performance over the non-gamified version.

As a result, the gamified version returned a better score on user engagement and performance over the non-gamified version. However, the difference in user engagement score is not statistically significant. Even though, the outcome of the gamified group indicates a more homogeneous effect on the users. This suggests potential in using gamification in industrial training. To obtain a more in-depth result, the study should be extended with more complex training on multiple platforms (HMD, mobile) and a bigger population.