Keywords

1 Introduction

Current production systems are characterized by a very high degree of automation [1]. However, with shorter product lifecycles and increasing need for product variations, a full automation of all production processes seems to be not feasible. The need for product variations and production strategies such as “build-to-order” and “design-to-order”, used by many companies, have actually increased the need for human operations [2]. At the same time, the advancement of technology, such as Augmented Reality technology (AR) has tremendously enhanced the perception of factory workers by providing an interface that augments the worker’s field of view [3]. Research has already shown that there is a large potential for the application of AR technology in production environment (see, e.g. [4–6] or [7]). These technologies could help to assist human operators in processing and handling large and complex systems in manufacturing contexts.

Consequently, we observed a growing need for adaptive and flexible assistance systems that support production workers to master the rising demand of customization and enable the systems for the support of manual workers to guide them through the process of manufacturing [2, 8]. These “smart” and multimodal assistance systems use sensors embedded in the production facilities and products to capture real time contextual information and provide individualized process support. Within the broad context of manufacturing, intelligent systems are applied to several areas such as assembly and dismantling, information management, training and inspection [3]. Most of the assistance systems proposed are based on augmented reality technology and often rely on sensory inputs, such as motion recognition or depth monitoring.

With this paper we seek to provide an overview of current trends and developments regarding adaptive assistance systems in manufacturing contexts. We looked for major design principles used in these systems and identified challenges they currently face. Finally, a discussion around possible future research directions for intelligent and assistance systems is provided in the end.

2 Application Areas for Assistance Systems

In a first step we reviewed the literature focusing on areas within the domain of industrial manufacturing where Assistance Systems are employed. A useful classification can be found in [3] so that we further investigated some of these areas.

2.1 Assembly and Dismantling

Assistive systems can be seen at many assembly lines where human operators receive live information from the system while working on assembly tasks. There are factories where operators make use of light barriers to identify the correct container to pick, this is known as “pick-by-light” [2]. Also, real-time assembly line visualization on computers is made possible by utilizing motion recognition technology in assistive systems. At the same time, assembly speed could also be easily calculated using the concept of “trigger areas” [2]. Another relatively new application approach has been around assembly design assisted by AR technologies. Researches have shown that AR environment could be useful even in the early design stage of assembly by improving both the workflow structure and the efficiency [9].

2.2 Information Management

With the help of AR technologies, it is possible to collect critical data directly from built-in sensors. This also allows for the worker to receive just-in-time support in terms of data visualization that is useful for the work at hand [3]. The usage of AR-based assistive system also allows the workers to be free from reading up physical manuals or guidelines while performing the task since this information could be stored in the database and strategically displayed into the workers’ field of view through AR technology [10].

2.3 Training and Inspection

AR-based assistive systems can also be seen in industrial training environments. They are adopted in order to boost productivity as well as effectiveness of training. In their work, Lin et al. [11] have described one of the early approaches and architecture frameworks for establishing an environment of production training based on AR technology. They have demonstrated the feasibility as well as effectiveness of such training environments while Stadtler and Wiedenmaier highlight that such an environment does not present the same level of feedback one would receive from non-virtual environment [12]. Assistive systems also play a role in quality assurance in manufacturing lines. In their work, Zhou et al. [10] explained how spatially augmented reality technology could enhance the effectiveness and efficiency of the spot welding inspection process in automobile manufacturing.

3 Approaches for Developing Assistance Systems

3.1 Human Factors

To ensure a high usability and performance, the assistive system also needs to be designed in accordance to human factor engineering principles [13–15]. Among others, following an ergonomic design is imperative to HMDs, as it affects the success and acceptance of these displays [14, 16, 17]. In one of the studies done by Edelmann et al., they found that although there is no short term effect on visual acuity by wearing HMDs, most of the subjects did report an impairment of well-being [18]. This might be due to three problems identified by Reuss and Menozzi [19].

  • Accommodation Problem. To avoid users incapable of accommodating when HMD’s viewing distance is different from the subject, HMDs need to have flexible or adjustable viewing distance.

  • Navigation Performance. The performance, defined as the average time to hit a target, is significantly affected when comparing HMDs and desktop configurations. Assistive systems need to evaluate their designs through benchmarking navigation performance.

  • Display Units Resolution. Display units available in the market have low resolution compared to human factor requirements and this affects the display quality as well as quality of receiving information.

As for HMDs, more ergonomic principles need to be followed in order to ensure maximum usability of the device. The device that is mounted on the user needs to be small and not obstructing user’s movement. Also, it needs to be placed securely on the user so that it withstands any physical movement without falling off the original position. Lastly, the problem that these display units are causing eye strain on the user needs to be addressed [19].

3.2 Relevant Technologies Enabling Assistive Systems

Head-Mounted Displays (HMD). Head-mounted Displays (HMD). HMD, sometimes also referred to as Head-up Display (HUD), is one of the key technology enabler of AR application. As the name suggests, such device is usually worn by the user on the head and allow the user to superimpose physical environment with virtual information [20]. According to Schreiber, HMD could be classified according to their method of image creation, namely Optical-See-Through systems and Video-See-Through systems [20].

Projection. The use of projection is also becoming widely common as an integral part of smart environments in general [21–25] and manufacturing and production settings in particular [26]. Such a system, driven by software or camera, allows users to see and interact with the information, objects or images projected onto various kinds of surfaces [26]. Zhou et al. describes a spatial augmented reality system that employs data projector to project virtual images onto physical environment [27]. In another system whereby user is able define their own tangible controls, projector is also used as the primary technology for displaying digital information onto physical spaces [5].

3.3 Strategies for Interacting with Assistive Systems

Augmented Reality (AR). AR is a relatively new and advanced human-machine interface [26]. Such an interface allows computer-generated/virtual objects to be displayed on objects of the real environment through enabling technologies [20]. User-facing display. An order picking system that is assisted by cart-mounted display is analyzed and compared with other order picking methods such as pick-by-light and pick-by-paper [28]. Overlaid display. An example of overlaid display is an order picking system that is assisted by HUD. Relevant information is displayed directly onto the field of view of the user [28]. Spatial AR (SAR). SAR is different from many attached AR technologies, it makes use of projector technology to project data information directly onto physical surfaces, and users can view and interact with information being displayed in the physical space. One of the applications is in automotive spot welding inspection [27].

Tangible User Interfaces (TUI). Prominent types of user interfaces are tangible user interfaces (TUI) [27, 29–32]. They make use of projection, motion detection and depth sensors to superimpose virtual reality information onto physical objects. In one type of TUI, user-defined TUI, the user is able to define his or her own physical control and connect it to pre-defined “digital functions” [33].

Motion Recognition. The most commonly used sensory input in a smart system setup is motion recognition technology. It is used to recognize user’s physical movement and enable natural interaction with the system [34, 35]. Motion recognition technology usually consists of hardware and software. The hardware is the actual sensor in the physical space, responsible for collecting information, whereas software is a software program pre-installed in a PC, responsible for storing and processing information collected. Pairing the hardware with motion recognition software, it is able to create a live feed of what is going on in the plant and measure the speed of each station [2]. In an augmented workplace set up by Funk et al. [33], Kinect and Leap Motion sensors were used to track user input through hand movements and gestures by understanding both 2D-touch interaction and depth changes. The Kinect system developed by Microsoft Inc. is as a technology that understands human body language and lets users interact naturally with games through hardware like depth sensor, color camera and microphone array [36].

3.4 Further UX Design Approaches

Gamification. Gamification is the process of applying modern gaming elements into other activities. These elements, such as point scoring and competition with other players are proven to be capable of improving user experience and encouraging more user engagement [2]. Over the years, gamification has been applied to many non-gaming settings such as healthcare and education [20, 37–40], whereas industrial application is still a relatively new area [26].

One of the important implications of such applications is that assistive systems need to be designed in a way as to adapt the level of challenge to the ability of the user, similar to the way in which games adapt to its players [2]. Another design decision influenced by gamification is that instantaneous visual feedback, such as solving a puzzle or climbing a pyramid, can be represented as the work progress the user is currently doing. This set up is studied by Korn et al. with regard to assembly speed and quality. Parts assembly time is shown to be significantly reduced by integrating gamification design elements in the system [2].

4 Current Challenges for Assistance Systems

4.1 Product Quality Versus Completion Speed

The meaning of product quality might not be the same for different tasks in which assistive technology is integrated. For example, in an assembly line, product quality may be defined as the quality of the finished products, i.e., defects in a batch. If AR is applied in assisting inspection tasks, product quality might be defined as the number of defective products that is approved by the inspector [10]. So far, research has focused primarily on taking advantage of AR technologies to improve manufacturing quality and completion speed. This is supported by indeed many evaluations that concluded with a great potential of AR technology applications in manufacturing environment [8, 12, 16, 41].

Although most of the researches have shown a positive implication on productivity, quality or efficiency when AR technologies are used in manufacturing processes, the degree of such implications varies. While Korn [2] found a statistically insignificant increase in production speed, Stadtler and Wiedenmaier [12] mentioned that previous research has shown an increase of 30 % in productivity with the help of AR technologies. Disparities in effectiveness of AR-based assistive systems exist mostly due to differences in the complexity of the assistive system itself [42]. The effectiveness of these systems on improving product quality and finishing speed are influenced by its usability as well as extra workload exerted on the user. This will be discussed in the sections below.

4.2 System Unpredictability Versus Worker’s Mental Load

Assistive systems are made intelligent in the sense that they could help with the performance of the manufacturing environment. However, the fact that this is essential human-machine interaction, the question of system predictability becomes a key to minimize the so-called “automation surprises” [43]. In his work, Sheridan suggests that there is a positive relation between system predictability and the amount of mental workload exerted on the user [43]. One of the incentives of developing such an assistive system is to reduce the mental load of worker in work place, this goal is difficult to achieve if the system is not reliable, or the user might often have to guess “What is the automation doing now?” [43]. Rötting also suggests that system unpredictability can happen when a user is required to deal with complicated interfaces [42]. Generally, the mental workload of a user is increased when he has to process the information presented on the interface and make decisions based on that. Research on assistive driving systems has shown that a low complexity, highly integrated interface could minimize distractions to the driver, in other words, the mental workload for the driver [42].

4.3 Production Benefit Versus Implementation Cost

Current research has focused mainly on evaluating the potential benefits that assistance systems could bring to production environments. This paper summarizes some general benefits:

  • Saving Time. By facilitating the operations, assistance systems have the potential to save cycle time in many processes, for example during design [9], production, assembly and inspection [10].

  • Saving Cost. Cost saving could be achieved by many ways in the environment where the system is employed. More efficient process could translate into higher productivity, which essentially means more output using less input. For example, Stadtler mentioned an estimated productivity gain of up to 30 % [12].

  • Improving Reliability. Production reliability could also be enhanced by intelligent adaptive systems. With the help of such systems, products could be produced with higher quality and fewer defects [10]. Another way of improving the reliability of production lines is that assistive systems lower the training difficulty of new products or new employees, which reduces the dependency on few more experienced operators and hence facilitates job rotation as well as continuous production [28].

On the other hand, we only found little research on the implementation cost of such assistive systems in production settings. Most of the papers reviewed suggested the cost of such systems to a limited extent whereby there is no research specifically focused on the cost evaluation:

  • Setup Costs. Assistance systems usually need support from both hardware (i.e., sensory input) and software (i.e., databases). Depending on the complexity of the system installed, the cost of implementing hardware and software will vary [41]. Setup costs also include the customization efforts made to tailor the system to the production environment.

  • Training Cost. Training costs include both the initial training needed for the assistance system, as well as the training for workers who work with the system.

4.4 Challenges Related to Sensory Input

As a key part of assistance systems, sensory inputs, for example from depth sensors and motion recognition devices, currently face challenges that are limiting the performance of the system. Visual input/output is usually a crucial part of the assistive system, lighting conditions largely affect the consistency of these sensors’ performance. For example, the effectiveness of certain depth sensors depend on the lighting condition of the environment to accurately measure [33]. Also, lighting condition affects the visibility of projected information in automated inspection applications, e.g., spot welding [10]. Another shortcoming of visual sensors, as mentioned by Funk, is that they are not able to distinguish two exactly same objects [33]. Many assistive systems employ leap motion sensors to register 3D movement, however, these sensors require that there must not be objects interfering the motion so that accurate detection can be achieved [33]. As for assistive systems which rely on outputting information through projection, finding a suitable and consistently visible object surface is a challenge. The object’s shape and size determine the suitability of projection, the amount and the complexity of information that can be projected on the surface [10].

5 Discussion

We have seen that most of the assistive systems we extracted from our literature research are mainly based on Virtual respectively Augmented Reality technologies, typically with sensory inputs such as depth sensors and motion recognition devices. There are also assistive systems which make use of technologies such as light barriers [6]. Further research could explore more on assistive systems which are utilizing technologies which are not VR/AR related. Besides technology-related factors, we also noted that design paradigms from other disciplines could potentially be appropriate to be applied in the design of assistance systems. The use of gamification techniques for example showed positive influences also in production environments [2, 26]. These assistance systems are developed to be integrated into manufacturing processes so that production is more efficient, safer and more reliable. While current examples of AR devices are demonstrating a great potential of increasing the capabilities of production facilities, many ergonomic issues are still unsolved and result in a lack of comfortable and effective long-term usage. Another concern is the trade-off between the level of system unpredictability and the mental workload for workers. This is caused by the fact that truly adaptive systems are more unpredictable to its users compared to traditional systems, whereas having more control over the system behavior would increase mental workload of the worker.

6 Conclusion and Outlook

In this paper, we reviewed current trends and approaches regarding assistance systems in industrial production environments. We analyzed the relevance of various interaction paradigms, such as the use of Augmented Reality (AR) technologies including Head Mounted Displays (HMD) or the use of Tangible User Interfaces (TUI). We have also identified some of the application areas of such adaptive systems in production environments and their current development progress. Sensory input, as a crucial constituent of such systems, has also been considered.

We took account of the various challenges related to the inspected systems. One common issue identified in this process involves the trade-off between product quality and completion speed, which we consider as one of the most important challenge. Yet, this problem may be overcome by putting more emphasis on error detection and auto-correction mechanisms when designing assistance systems where finishing quality is as important as completion time.

Another concern is the interaction between the user and the assistance system. Incorporating principles from human factors engineering into the design of such systems is necessary to cater for a high degree of usability and meaningful user experiences. Further research has to be performed on how to design for and evaluate the usability and user experience within the industrial domain due to its specific requirements. Criteria for these new production-relevant contexts need to be developed where safety and usability criteria are adapted. This is a problem to be fixed in the future, once production assistance systems are more established.