Keywords

1 Introduction

Industry 4.0 is an industrial concept which allows the cooperation and the communication between humans and technologies, and between technologies themselves, in real time. Thanks to dedicated tools like Internet of Things and data management, industrial staffs will soon be able to predict the future production turnovers. All along the production line, smart sensors record all information regarding the process and the products made. The staff is now sufficiently informed to adjust supply chain, measure performances and take decisions in real time. We call this kind of factory, smart factories. In a smart factory, all products have sensors, e.g. RFID chips, dedicated to send and receive data from the products on production lines through IoT in order to optimize delivery time to the customer, reduce consumed resources and learn from former production runs to predict certain events of its future operational processes.

Companies needs to be both, more flexible and improve their productivity while they facing to great challenges as “increasing market volatility, shorter product lifecycles, higher product complexity, and global supply chains” [1].

Another important point to emphasize is the mass customization needed by the consumers and considered as relevant by the companies. In article [2], a survey done on 30 Flemish companies at the beginning of 2017, highlights the mass products customization as one of the most important business challenge for industrials behind a more products diversity and lead times reduced. To make these demands real, it is important to digitize production apparatus and facilitate crowdsourcing [1] in order to help to a faster products design process.

If technological innovation is the first and obvious way to improve and make production more flexible, a second pillar, maybe the most important, is to keep Human at the centre of the production. Called social innovation [1], or social perspective [3], main idea is allow to the workforces to improve their digital skills and culture [2], by new organizations, laws, regulation and/or business models [1], in order to give up responses to societal needs, and assist ageing, disabled, or apprentice workers [3]. Both words, social and sustainable, are correlated to technologies and, more specifically to cyber-physical systems, by a collaboration between humans and machines.

Enhanced human’s knowledge, sensorial or cognitive abilities inside working environments is mandatory if they want remain competitive and meet the societal needs. Future industry worker would be able to oversee the production by controlling it through cyber-physical systems as augmented reality, being trained to new digital processes and/or workbenches on the production line via virtual reality, work more efficiently with collaborative robots on assembly lines especially if we are talking about impaired workers. Some authors [3] called operator 4.0, a human trained about digital culture and skills, “aided by machines as and if needed – by means of human cyber-physical systems, advanced human-machine interaction technologies and adaptive automation” [3] in order to be more flexible and accountable in case of change of production series, and unforeseen events on the production lines.

We previously mentioned three technologies 4.0: augmented reality, virtual reality and collaborative robots. Despite of the wide range of possibilities to integrate technologies 4.0 inside the production apparatus, it remain a large types of challenges to solve before considering a human-machine interaction.

If we focus on augmented reality we can observe that many study cases in Literature point out the increasing performances and the reducing of error rate after implementing AR on an existing processes [4, 5], and [6]. However, it never pointed out how we could improve the user experience during job in order to increase the user acceptance during the technology 4.0 implementation process. Other lack, technologies are implemented in an existing process without consideration regarding the production workshop and organizational processes [7]. It is necessary to respond to the organization in order to reorganize it with regard to the integration of new technologies in the company’s processes [8]. Finally, certain ergonomic features of the technologies can have an impact on the operator’s work (e.g. visual fatigue regarding the type of technology [9], the effects of discomfort resulting from the weight of the device [10], or the effect on the perceived image facing to panel type embedded inside your AR device [11]).

Concerning collaborative robotics, Literature brings out a lack of energy autonomy and a low social acceptance on the one hand, a weak in, bilateral communication between human and robot, voice recognition, comprehension and utilization of natural language, and perception and interpretation of human behaviors from the other hand [12]. This paper focuses on many other issues such as safety and security of work in a space shared with humans, reprogrammability of robots and user-friendly interfaces to simplify use on production lines by unskilled or cognitive impaired workers.

To avoid a failure of integration in production, it is necessary to consider industrial organization and processes. They must provide space for work debate and times for collective regulation, an enabling management in order to fight against work stress and for improve performances and innovation [13]. Compan et al. [13] recommend also to design the future workbenches with operators since the beginning of the process design in order to increase the acceptance of the technology implementation. Regarding technologies, they must be understandable by operator 4.0 and this last must be able to change process characteristics and/or information displayed through HMI. The setting up of criteria previously exposed is called Enabling Collaborative Situation (ECS) by Compan et al. We will rely on this work to build an efficient human-machine collaboration in industrial workshop.

All criteria needed to build an ECS in order to reach a successful production implementation will be described in details in the next section, Sect. 2. To explain the concept of an ECS, we describe the genesis of this paradigm through former ergonomics works.

Then, in Sect. 3.1, we present the workbench where pre-tests are carried out and where we want to improve user experience by redesign HMI in the case of using AR technology and redesign workbench in the case of collaboration between human and robot. After presentation, always in Sect. 3.1, we explain and describe in details the results of our pre-tests where a set of ten people realized assembly tasks of components on a dedicated sub-assembly.

The improvements planned to move towards an Enabling Collaboration Situation of work between human and machine is presented in Sect. 3.2 just before conclusion of this paper.

2 Enabling Collaborative Situation

Enabling Collaborative Situation (ECS) is a concept born following the work of Amartya Sen [14] who exposes the idea that an operator’s ability to act depends on the capabilities offered by the process with which he interacts. Thus, a person may possess abilities allowing to reach desired performances without being able to used them. This concept named capability approach (CA) defines a range of conversion factors, which determine the power for a person, to act on a process (Fig. 1). These factors are sorted in three categories: individual (gender, age, experience, level of education, etc.); social (work team, etc.); and environmental (technical means, work organization, etc.).

Fernagu-Oudet et al. [15] conclude on the capability approach: “Capability defines, according to this logic, a field of possibilities both for the individual who is the bearer of the capability and for the organization that can benefit from it. It is based on a set of mobilizable resources (internal and external to the individual) that will undergo conversions in order to be actualized in selected achievements or behaviors. Sen [Amartya] speaks of accomplishments or operations”. Thus, capability approach enables individual intrinsic capabilities to be taken into account and weights the opportunities that the individual has to develop his or her personal skills and knowledge. It is from this approach that the idea of the enabling environment emerges.

Fig. 1.
figure 1

The capability approach theory (adapted from [15]).

Full size image

Pierre Falzon [16] defines an enabling environment as: “an environment that allows people to develop new skills and knowledge, to broaden their possibilities of action, their degree of control over their task and the way in which they carry it out, i.e. their autonomy”. Fernagu-Oudet et al. [15] add: “[…] energizing work environments to make them empowering, consists in helping individuals to mobilize and use the resources at their disposal, not just make them available”.

The enabling environment is therefore a set of physical data and information flows arranged in such a way as to optimize the work and learning of the individual in the context of his or her function. However, because of its intrinsic nature, enabling environment is very exogenous to the individual, and the individual seems to be relegated to the background. The focus should not be on the work environment of the operator but rather on the operator himself in relation to the 4.0 technology he uses, to perform the tasks assigned to him/her.

Based on the concept of an enabling environment describe by Falzon, Compan et al. [13] bring out three following criteria:

  1. 1.

    Learn a new and more efficient way of doing things.

    Authors explain the fact it is necessary to build an ecological assessment of the performance close to real situations of technology use in order to define relevant performance criteria.

    Human needs to know that his work is done correctly and even improved by using technology. It increases the sense of utility of the technology used by him.

  2. 2.

    Increase the available possibilities and ways of doing things.

    In the face of increasing product complexity and frequent changes in production runs, operator 4.0 has to deal with a large range of situations. That improve his experience of work, and it is important to leave him the possibility to add new ways of doing things in order to improve quality and/or productivity. If necessary, process steps can to be modified if operator is seeing a better way to do things. Technology 4.0 should not prevent him from changing the procedure and should even help him to do so.

  3. 3.

    Adjust the Human-Machine couple attributes according to the evolution of situations over time.

Last criterion is based on the time during which human works with machine on production. It investigates the united couple through interactions between entities and eventually, the situation evolution during work. To do that, it is important for operator to be able to understand technology and modify it according the lived situation. The understandable characteristic is named by [13] operational transparency. The operational transparency is the minimum understanding required to anticipate the device behaviour and modify its internal characteristics in consequence. The ability for the operator to modify the characteristics of the technology is called continuous design in use [13]. This last help to modify process step if operator knows a way to do better (criterion 2) or modify graphical interface in order to display only relevant information for him (e.g. an expert/novice mode as pointed out in [4]). At the end, it helps to do a joint evolution/construction of the human-technology couple over time.

Finally, this criterion questions the industrial organization and operational processes in order to define whether they are consistent with increased operator responsibility, enhanced individual skills and knowledge, and improvement performance at the workstations. To achieve this, industrial managers have to develop spaces for debate on work, times for collective regulation and set up an enabling management.

As seen before, all these criteria have emerged from ergonomics disciplinary field. If technologies have been called up, any experiment has been made in order to confirm or refute if these criteria are relevant in an organization, and more specifically, inside an HMI. If it is impossible to verify organizational and operational processes, we could check, inside a laboratory experiment, the relevance of criteria 1 and 2, as well as operational transparency and continuous design in use of criterion 3.

To do this, we will use an industrial case study in our laboratory as it was designed and built for a past laboratory experiment [17]. This is a replica of existing French industrial training workstation (see Fig. 2(a)). Moreover, we have decided to prospect both types of following technologies: augmented reality and collaborative robots. Next section will present the workbench in details, the limitations observed during the pre-test carried out in a first step and the experimental method.

3 Workstation Presentation and Experimental Method

3.1 Workstation Presentation

How Does It Works?

The workplace is a training workbench dedicated to pick and place parts (two plastic clamps, a rubber, and a wire) on an existing sub-assembly. It uses Hololens 1 augmented reality technology to help user to pick right component and place it at the right location on sub-assembly, but also gives to the operator, the safety rules and assembly steps. The workbench allows assembly of four parts on a rear windowscreen, that we will call now, main part, used in automotive industry.

Fig. 2.
figure 2

Industrial workstation replica (a) and its CAD model (b).

Full size image

There are eight steps, from step 0, reminding the security rules, to step 7 useful for packaging the sub-assembly in the packaging bin after components assembly (Fig. 3). The assembly procedure can be broken up as follows (for confidentiality reasons, some steps are not explained but replaced by “confidential”):

  1. 1.

    Confidential;

  2. 2.

    Assembly rubber;

  3. 3.

    Confidential;

  4. 4.

    Confidential;

  5. 5.

    Setting up wire on main part;

  6. 6.

    Confidential;

  7. 7.

    Packaging sub-assay in the packaging bin.

In augmented reality, via see-through glasses, for displaying virtual information at the right place, it is mandatory to calibrate AR device with the real environment. To do that, a CAD model has been created it is necessary to positioning this virtual avatar on the real one (see Fig. 2(b)). Calibration operation is made by placing CAD model on real mock-up. For that, it was proposed to position a white cube under the rear left foot of real workstation, then, it is possible to adjust CAD model position by translational and/or rotational movements, through keyboard buttons. It is important to notify that the well positioning of every next displayed information in the User Interface regarding the real prototype is depending on the good initial calibration realized by the user.

Fig. 3.
figure 3

General process view.

Full size image

User Interface (UI) provides two types of modes: first, the training mode where all steps are explained and operator follows steps one-by-one assisted by UI with a lot of explanations during process through watching videos, reading information and/or observing pictures at each step (Fig. 4), by providing audio instructions and by observation of 3D kinematics. Videos can be paused, moved in the operator’s field of view, and operator could skip a step to go backwards or forwards. Additional information can be displayed by clicking on a yellow plus virtual button, the latter being transformed into a less yellow one. Audio instructions can be replayed if wanted by operator. 3D kinematics are displayed showing how the component must be placed on the main part. 3D model of main part is stacking on real one and 3D model of component moving from the box where it is stored to main part where it is placed with the right orientation.

Second, the validation mode. Here, operator can watch videos and kinematics and read information if he wants but the mode is built for assembling components on sub-assembly and validate his productivity and quality made. Any time counter or quality mismatch detector has been implemented in the UI for this training workbench. Normally, in validation mode, numerated steps (Fig. 3) are enough for operator to carry out his tasks.

Fig. 4.
figure 4

A process step detailed with more information and pictures.

Full size image

UI is able to recognize both following human-machine interactions: gesture recognition through Air tap, Hololens 1 dedicated gesture, and through the clicker, tool delivered with Hololens bundle, and speech recognition. However, Hololens 1 API expects user to speak in English to use this type of recognition.

UI provides also guidance cues in order to guide user’s gaze if the projected information isn’t in the device’s field of view. These cues are displayed via the form of red arrows.

Limitations.

To detect limitations on the workstation, we carried out pre-test with ten students in July 2020. These people following engineering courses in MSc or was Ph.D. students. They were not necessarily experienced in the use of an AR device especially with an HMD device. However, they all know what AR technology is. People was all young (<30 years old).

All people made both mode: training and validation in this order and learned Hololens 1 Air tap gesture during training mode. In this mode, they all watched the videos and the kinematics, read all information, listened all audio instructions and followed all needed gestures in order to assembly the components on the main part. They could ask help to do task, or the selection dedicated gesture (Air tap). In validation mode, no help was provided but if they want, they could watch the videos and the kinematics and read information if necessary.

At the end of the experiment, an interview was realized in order to know what kind of problems has been experienced and/or detected by users and what improvements could be made in UI and on the device in order to increase the user experience and the performances on the training workbench.

Limitations observed and reported by users are listed in the table below. They are sorted in three categories: User Interface issue, device issue, and other. We have added also the number of times (n) when the problem has been reported by operators (n/10 operators).

As we can see in the table, there are three types of issues observed during our experiment. First, device problems. They are resulting from the device characteristics and design. We can mention the Air tap gesture (6 people on 10 tested said that this gesture is very difficult to learn), the weight of the device, its Field of View (5/10 people complained about it), the accuracy of the selection gesture (2/10 people), and for one people, the difficulty to wear the device with big glasses. A good point detected is the clicker because it replaces the Air tap gesture (2/10 people exposed this good point) (Table 1).

Table 1. Details of issues reported by operators during experiment.
Full size table

Besides, concerning UI, a lot of people exposed problems linked to the concept of operational transparency. We can mention the selection/validation not explicitly shown, with 4 individuals complaining about that. Also, displaying issues as difficulties to see component’s orientation or problems displaying virtual elements overlaying the real environment resulting from colors, luminosity and/or virtual elements’ edges.

One person told us he experienced a problem with virtual elements rendering. Indeed, she don’t want to be fooled by the system and wanted to keep the knowledge of whether she was dealing with a virtual element or a real one. We think we could resolve this problem by using a rendering technic called cel-shading. It allows to emphasize the virtual elements’ edges and operators could see the difference between the virtual elements and the real ones. We could resolve the problem of misunderstanding between real and virtual elements and the difficulties to see the edges at the same time.

The calibration of the virtual mock-up on real prototype is considered also as difficult for 3 people. They preferred add a trihedron in order to place the virtual workbench more easily on the real one. Some people told us that accuracy of the location between the virtual elements and the real was not good. This is the result of the calibration process. These people were not aware of and did not express this cause but because we know the root cause of this problem we added these three people to the three first, and the calibration process was thus a problem for 6 people.

Three people think HMI provides too much information. They want would expect to hide or remove information from the graphical interface. We can associate this limitation to the fact that 5 people think videos aren’t useful when they have already audio instructions, assembly kinematics and information displayed on the field of view. Four people think it will be a good improvement if the final assembly was projected at the beginning of the process in order to know what the expected final result is. Adding a button in this mean seems needed in order to give to the operator the possibility to add or not this final result. Hide/removal or add this video or any other information in the graphical interface is the way to reach the concept of continuous design in use as explained by Compan et al. [13].

Last relevant issue exposed by users, the questioning about utility of AR device. It’s a good question because in [13], the first criterion (of an ECS; Learn a new and more efficient way of doing things) questions the affect, the utility and the sense-making, of the technology 4.0 implementation on already existing workbenches, in addition to the performance criteria.

3.2 Experimental Method

Outcomes Expected.

Based on these pre-tests carried out, we can imagine a future experiment where we will check the increasing of utility, performances, and users satisfaction during use of AR technology on our training workbench. To do that, we want to explore two ways: first, the same device, Hololens 1 with the former UI (the one used during pre-tests) versus the new one (improved thanks to users feedbacks). We want to see if operators can see the difference during the completion of their tasks with the help of improvements points previously exposed (e.g. are the virtual elements and their edges more visible thanks to cel-shading rendering?; Does HMI create cognitive overload with possibilities to hide/add information inside?; Operational transparency and continuous design in use are achieved during use by operators?; Finally, are the operators in an enabling collaboration situation with the technology?).

In a second time, we will test the new User interface projected inside Hololens 1 versus Hololens 2. We want to audit if the improvements of AR characteristics are increasing the sensation, for the operator, to be in an Enabling Collaborative Situation with his technology and how (e.g. is just a characteristic which allow to reach that or a set of characteristics which help to create and to live an ECS? In a same way, is just one characteristic which allow to solve an issue or is it a bundle of new characteristics which allow to reach the resolution of this issue?).

To answer these questions, we will experiment these both AR devices, Hololens 1 and Hololens 2, with our new User Interface improved thanks to relevant issues exposed by operators in pre-tests.

Moreover, in addition of improvements integrated in new UI, we need to know if performances are reached in order to confirm the usefulness of AR technology to the industrial companies. We decided to explore two types of performances indicators: time to assemble components on main part, and the error rate (the number of errors made on the number of assemblies built). These both indicators are usually the most indicators measured during experiments.

Concerning users tested, we will try to make our experiment with people with a large range of age, gender and education level. It is important for us to explore with a large panel of people in order to be as close as possible to the employee profiles present in the companies.

Regarding collaborative robotics, we need to redesign the workbench in order to create a working situation between human and robot in a shared-space. For example, we could imagine a situation where a disabled worker could be assisted by a collaborative robot in order to reach or even exceed the required industrial performances. In order to help the operator to understand robot when it runs, a static screen could be placed beside it and human could interact with robot, the latter ask to human what are its need and human could answer in consequence. Operational transparency being fulfilled by this setting up.

Concerning the continuous design in use, operator could change the robot speed in order to increase the work pace. Other example to illustrate the concept of continuous design in use, we could add a task to be made by the robot at the place of the human or the contrary, remove an action made by robot and replace it by an action did by human. We need to elaborate an UI to allow collaboration and interaction between human and robot through the static screen.

For both technologies, AR and collaborative robotics, we need to give to the operators the means to reorganize the assembly tasks. Indeed, the second principle of an ECS (Increase the available possibilities and ways of doing things) sets to search the ways to give at the user the possibilities to change the tasks order if he found a better way to doing task, or a way which improving performances. Thus, the operator must have the possibility to reorganize steps between them, in process, merely, through HMI and without computer skills.

4 Conclusion

In this paper, we have defined the industry 4.0 framework and the concept of operator 4.0, then we pointed out the problems for setting up the technologies 4.0 in the production apparatus and the intrinsic defaults of AR devices and collaborative robots that make it difficult to integrate them into the production environment because of dissatisfaction of operators and/or performances decreasing at workstations.

We decided to rely on the works of Compan et al. and the paradigm of Enabling Collaborative Situation. This latter is based on former ergonomics works such as capability approach and enabling environment. After, defined an ECS, we have exposed the experiment, based on a former industrial training workbench for automotive industry, where we have detected a lot of issues thanks to the help of users, inside the HMI and with the device used (Hololens 1). This pre-study seems to confirm the relevance of ECS’ criteria but to validate them, we must realize a new study to design and redesign workstation according to issues pointed out during pre-tests in order to move towards an ECS. Thanks to the pre-study we have been able to consider technical solutions for the problems encountered.

In the case of the assembly with the augmented reality device, we will rework the UI, by redesign interaction between Human and AR device, by rework visual rendering, and by adding new buttons/options/menus/etc. inside the UI. We will analyze also the contribution of device in the operator’s work improvement by testing Hololens 1 and Hololens 2, this latter having a more important field of view, a reduced weight, another fixations design on head, another gesture to interact with UI (which replace the Air tap) and a lot of other improvements.

Concerning the collaborative robot, we will redesign the workbench in order to create a shared-space between human and machine, create an HMI via a static screen beside the robot to allow to human to understand and interact with the robot during the production.

For both technologies, and according to the second principle of an ECS elaborates by Compan et al., we will give to operator the means to reorganize the order of assembly tasks through the created UI.

Finally, we will measure for both technologies in interaction with human, if the performances indicators are reached or exceed. These indicators will be the assembly pace (time to assemble all components on the main part) and the error rate (the number of errors made on the number of assemblies built).

All these experiments aim to make a problem/solution cartography which cannot be independent of the technologies used. The enabling criteria are key elements for this mapping. Besides, the analysis of test cases from the literature will allow to consolidate this knowledge base. In the long term, we wish to build a heuristic for the development of ECS between operators and machine in industry 4.0.