Keywords

1 Introduction

The birth of smart factories, driven by Industry 4.0 (I4.0) paradigm, shifts attention to the novel role of the human operator as a crucial element to deal with new and unpredictable behaviors in smart production systems. Human operator in I4.0 [1] should be extremely flexible and demonstrate adaptive capabilities due to the wide range of problems to solve. Nevertheless, even for flexible operators, it could be difficult to manage the big amount of information that would be available in I4.0 production plants, as well as the rapid changes in the configuration of production lines to satisfy customer requirements.

One of the functions, in an industrial plant, that will much benefit from I4.0 is that of equipment maintenance. Commercial solutions (e.g., PTC ThingWorx, REFLEKT ONE, Scope AR) for the development of AR maintenance applications are constantly increasing, as well as prototypes [2, 3]. However, research works are still needed to address specific issues, as information comprehension [4, 5], authoring [6, 7], cognitive aspects [8, 9], and so on. Other issues derive from the operating context. One of these is the difficulty for the operator to remember the location of all equipment in the plant and other useful data. This aspect is due to the rapid changing in plant layouts and the greater amount of data to manage [10] in the context of Industry 4.0. In the past, this function relied mainly on operator experience supported by technical drawings. Then, in this work, we aim to provide a tool to support operators in the information retrieval about equipment in industrial plants, allowing accessing even more data. This tool will improve maintenance tasks, to reduce the retrieval time of useful information in the maintenance procedure and, consequently, the downtime of the process. Shortly, when structured and unstructured data will become increasingly available from all points of the process, the target will move to predictive maintenance, based on fault prognosis [11].

Taking a snapshot of industrial plants, we can say that most of them are far to be ready for predictive maintenance, although lots of prototypes and theories were presented in the literature. There is still old equipment, and sometimes it is difficult to integrate them with sensors. A pure data-driven approach is far to be implemented, i.e., extracting the process information for maintenance from the records available in process databases and deriving from machine sensors. Furthermore, employees are accustomed to relying more on their know-how than on new technologies. A knowledge-based approach is still predominant, i.e., exploiting pre-existing knowledge or information about the process connections.

In this context, the introduction of I4.0 features in existing plants will be gradual. A strategy to integrate data-driven and knowledge-based approaches would be needed. The use of Augmented Reality (AR) allows showing equipment information in the form of digital contents, thus augmenting technical drawings [12].

An example of technical drawing commonly available for maintenance tasks in industrial processes is the P&ID (Piping and Instrumentation Diagram or Process and Instrumentation Diagram), a drawing showing the interconnections between the equipment of a process, the system piping and the instrumentation used to control the process itself. The P&ID does not contain additional information on equipment, such as the location of equipment in the plant, the description of the machine’s functionality or the maintenance history. It also requires constant updating because of system modifications. Maintenance operators examine this documentation directly in the plant in correspondence of equipment. Thus, they cannot use personal computers, large monitors, but just paper documents and handheld devices.

Then, in this work, we implemented an AR application, running on a handheld device, that enhances P&ID drawings with information retrieved from the plant database. We also evaluated the effectiveness of AR to improve user performance in the task of information retrieval for plant equipment, which is supposed to be the one inspected by the process operator. Providing the operator with more information, without increasing operator confusion, is important to plan a maintenance procedure in a reduced time. We then tried to answer the following research questions about this task. (1) Is AR effective to reduce times and errors respect to paper documents? (2) Is the AR application accepted by user respect to paper documents? (3) Is performance influenced by the handheld device (smartphone vs. tablet) used for the AR application?

In Sect. 2, we present the related work about solutions adopted for the introduction of a data-driven approach in industrial procedures. In Sect. 3, we describe the approach followed for designing the proposed AR application. In Sect. 4, we present an example of implementation of the application for the case study of the maintenance of a milling plant that was evaluated through a user study. The results of the user evaluation were reported in Sect. 5 and discussed in Sect. 6.

2 Related Work

In the literature, several attempts of introducing a data-driven approach in industrial procedures were made, also exploiting AR. For example, Mourtzis et al. [13] presented a condition-based preventive maintenance approach integrated into a machine monitor framework. The system gathers and processes data, related to the operation of machine tools and equipment, and calculates the expected remaining useful life of components. Then, the system provides notification to process operators and maintenance department in case of the failure events during production.

Pintzos et al. [14] proposed a framework for the use of AR goggles coupled with handheld devices to assist operators for manual assembly. In this framework, there is an application related to the monitoring of process indices, which were displayed on the AR goggles in the form of KPIs related to time, cost, quality, and energy values.

Segovia et al. [15] implemented a system that exploits AR to display KPIs gathered from measuring devices, in the corresponding of workstations inside an industrial plant. They tested the system in a machine shop department against two not-AR modalities. They report that AR provides a simplified way to access the process performance information of several workstations in a production line, then it is no longer necessary to visit each station one by one to consult their status.

Liu et al. [16] presented a new generation of the machine tool with an AR-enabled process monitoring, thus integrating the AR technology with real-time process data from the CNC controller and various sensors to provide users with an intuitive perception of the machining processes. Furthermore, prognostic and health information can also be rendered on the related components to indicate the health status and remaining life so that proactive maintenance can be realized during the machining process.

Only a few of the system presented in these and other similar works were tested in a real or simulated scenario. Most of them are just prototype; then, it is difficult to consider all the issues in the implementation of such solutions, including human factors. For example, especially in small and medium-sized enterprises (SMEs), the operators are still accustomed to the knowledge-based approach, mainly based on drawings as the P&ID.

Furthermore, many companies use P&ID in paper form, for which the recognition of the various components and their functions is often tied to the know-how of the technicians working in the company. Other works have already been presented in the literature, to improve the comprehensibility of P&ID. Many specialists have tried to develop systems that automatically transform the P&ID from a paper to a digital form, including the automatic recognition of the component. Arroyo et al. [17] presented a method based on optical recognition and semantic analysis, which is capable of automatically converting legacy engineering documents, specifically P&ID, into object-oriented plant descriptions and ultimately into qualitative plant simulation models. Tan et al. [18] proposed a novel framework for automated recognition of components in a P&ID of raster form, based on image processing techniques to make a mathematical representation of the scanned image. They further extended this method to acquire also the connectivity among the components [19].

Considering state of the art, we then presented an AR application that allows the introduction of the data-based approach in a more gradual way within industrial plants, integrating it with the knowledge-based system already present in the enterprise and mainly based on technical drawings as the P&IDs. Furthermore, we tested this application through a user study.

3 Our Approach

The application developed in this work supports the operator in the maintenance of a manufacturing plant using AR, providing information about the equipment to inspect directly on P&ID drawings.

In the system that we introduce (Fig. 1), equipment data are stored in the plant database and analyzed by a control room. It notifies the operator/s about the equipment to inspect communicating its code. Then, the maintenance operator searches the equipment to inspect on the P&ID. At this point, to accomplish the maintenance procedure, he/she needs other information about the component and/or the process. This information could be either plate technical data or real-time data coming from the process database, either numeric or in the form of other graphical visual assets. The presented application allows the operator to access all this information augmenting the P&ID drawing through these data.

Fig. 1.
figure 1

Description of information flow for maintenance task according to our approach.

Full size image

In a first prototype of the AR application (Fig. 2), virtual hotspots are displayed in correspondence of plant elements on the P&ID; the hotspots could be of assorted colors to indicate different elements: e.g., pumps, conveyors, filters, and so on. Users can filter the hotspots displayed at the same time, grouped either by category (e.g., all the pumps, all the conveyors, and so on) or by subsections of the plant. Though operators are familiar with P&ID reading, this filtering utility facilitates the location on the P&ID of plant element to be analyzed, especially in case of layout modifications.

Fig. 2.
figure 2

Visualization of additional technical information through a pie menu: a technical chart with information retrieved from a database, a navigable 3D CAD model, and a 360-degree image of a plant section.

Full size image

When users tap on the virtual hotspot on a plant element, that hotspot gets bigger, whereas the others get smaller and become not selectable, and the name of the plant component is displayed for a check. A menu appears on the screen with three selectable buttons. These 3D buttons (pie menu) are registered on the trackable as a generic virtual element, and when the user clicks a second time on the hotspot, the menu disappears.

A first button opens a technical chart of the component with all the information retrieved from the factory database where all the information associated with the plant components is stored. This information could be plate data (e.g., model number, supplier, efficiency, and so on), history data (for example, about maintenance and modifications), real-time data coming from process database.

A second button opens a navigable 3D CAD model of the selected component where, based on the available information, areas of the machine that require an inspection (e.g., bearings in a transmission shaft) can be highlighted.

A third button opens a 360-degree image of the component and its surroundings in the real plant. In this way, operators can rapidly associate the drawing to a physical location within the plant and know-how the component is connected to the rest of the plant.

The application was designed using Unity 3D and Vuforia for the AR behavior. We used the image-based tracking using the digital version of the drawing as trackable; an important remark is that all the lines in the drawing should be black to achieve the highest tracking quality. Black lines on white sheets are mostly used in technical drawings, according to the drawing standards (UNI EN ISO 128-20:2002), however for P&ID other line colors are often used because many lines may overlap and also to distinguish the fluids flowing. From “AutoCAD plant 3D” we exported a datasheet of the (X, Y) coordinates of the plant components and they were used for the positioning of the virtual hotspots in Unity 3D. In AutoCAD each block represents a machine, the machines are divided by category to filter more effectively, and each category is also assigned a different color. In Unity 3D, a C# script reads the.txt file generated by AutoCAD with information about the location, color, tag, name. Then, in the Unity 3D scene, a copy of the hotspot is created for each machine with all the previous information automatically assigned. Then, we developed a second C# script to filter the visualization of the components displayed at the same time. The behavior of the pie menu is managed linking to the buttons a new Unity 3D scene with the 3D CAD model and the 360-degree image, respectively. For the technical chart, the link opens a 2D window that was designed and added to the canvas. For the information filling in the chart, we took it from an SQLite database automatically generated from AutoCAD Plant 3D. The information can be added either in AutoCAD Plant 3D or in the database since they are synchronized.

4 User Evaluation

We designed a user study to answer our research questions. We implemented a prototype of our application for the maintenance of a milling plant. During the experiment, users were asked to retrieve information about plant components on the P&ID. The P&ID was that of the cleaning section of the milling plant.

Components on the P&ID were indicated by the experimenter. Then, users had to search and communicate aloud the following information about the component: equipment type, the floor where it is located. We limited to information already present in the paper documents, besides the specific information does not affect the results. The task was repeated for five different components, and the overall time between the indication of the first component and the communication of the information for the fifth component was measured. An experimenter supervised each test and checked for errors in real-time. Users accomplished the task in three modalities:

  • Paper: information about the component is retrieved from tables on paper sheets like those commonly used in actual practices.

  • Smartphone: information about the component is retrieved from the AR application we developed running on a smartphone OnePlus 3 (screen size 5.5”).

  • Tablet: information about the component are retrieved from the AR application we developed running on a tablet SAMSUNG Galaxy Note 10.1 (screen size 10.1”).

At the beginning of the experiment, we formulated the following hypotheses for the task of information retrieving for plant equipment:

  • (H1) AR application will significantly reduce the amount of time compared to paper documentation;

  • (H2) AR application will significantly reduce errors compared to paper documentation;

  • (H3) AR application will significantly improve usability compared to paper documentation;

  • (H4) performance in terms of time, error, and usability will be better for a tablet than for a smartphone.

A total of 39 voluntary participants (9 females) were recruited among engineering students at Polytechnic University of Bari. The average age was 24.2 (min 22, max 29, SD = 1.92). We interviewed the subjects about their frequency of usage of AR applications: 13 never, 15 rarely, 8 sometimes, 3 often, and none always used AR applications. Among users who had used at least once AR applications, the fields of use were video gaming (9), social network (8), cultural heritage (4), DIY (3), retail (2). Conversely, users had great familiarity with paper manuals/drawings: 2 sometimes, 3 often, 34 always used them. Users had no experience with plant maintenance tasks, but we designed a very basic experiment whose results are not affected by user experience and motivation. A total of 39 voluntary participants (9 females) were recruited among engineering students at Polytechnic University of Bari. The average age was 24.2 (min 22, max 29, SD = 1.92).

We interviewed the subjects about their frequency of usage of AR applications: 13 never, 15 rarely, 8 sometimes, 3 often, and none always used AR applications. Among users who had used at least once AR applications, the fields of use were video gaming (9), social network (8), cultural heritage (4), DIY (3), retail (2). Conversely, users had great familiarity with paper manuals/drawings: 2 sometimes, 3 often, 34 always used them. Users had no experience with plant maintenance tasks, but we designed a very basic experiment whose results are not affected by user experience and motivation.

We used a Latin square design of the experiment. Then, we had thirteen participants performing the task in the sequence “Paper-Smartphone-Tablet,” thirteen in the sequence “Smartphone-Tablet-Paper,” and thirteen in the sequence “Tablet-Paper-Smartphone.” They were told to complete each task as quickly and as accurately as possible. Each participant was allowed to familiarize with the three modalities for 10 min before the test. This training phase helped the participants to get accustomed to the AR user interface. At the end of the training phase, an experimenter checked that the participant was able to use the application easily. After completing the test, users were asked to respond to a SUS (System Usability Scale) questionnaire, to evaluate the usability of the three modalities.

5 Results

Our purpose was to evaluate the main effects of the execution modalities on user performance. Thus, we collected data into three samples, one for each modality. We had three types of data: completion time, error rate, and the SUS score.

To make statistical inferences, we started to enquire whether the completion time sample followed a normal distribution. We used the Shapiro-Wilk normality test, AS R94 algorithm, on all samples. All the original samples did not follow a normal distribution; thus, we applied the Box-Cox transformation with α = −0.9241. Transformed samples positively passed normality (Paper: W(39) = 0.968, p = 0.324; Smartphone: W(39) = 0.964, p = 0.241; Tablet: W(39) = 0.971, p = 0.411) and homoscedasticity test (F(2, 114) = 0.275, p = 0.760). Then, we used ANOVA to compare the samples. We found a statistically significant difference between the three samples (F(2, 114) = 63.974, p < 0.001). Tukey’s posthoc test revealed that users performed significantly better with “smartphone” than “paper” (p < 0.001) and with “tablet” than “paper” (p < 0.001), whereas there was not a statistically significant difference between “smartphone” and “tablet” (p = 0.997) modalities. Mean completion times (Fig. 3) are: 118.2 s for “paper,” 66.5 s for “tablet,” and 65.0 s for “smartphone.” These results allow us to confirm hypothesis H1 and to reject hypothesis H4.

Fig. 3.
figure 3

Results about mean time (left) and SUS score (right) of the user study for the three execution modalities.

Full size image

We used the following error rate definition:

$$ {\rm{ER}}\% = \left( {{\rm{n}}{.}{\rm{errors}}} \right)/\left( {{\rm{n}}{.}{\rm{targets}}} \right) \cdot 100 $$
(1)

The “n.errors” is the sum of all the participants’ errors observed for each task. The “n.targets” is the maximum number of errors that a user could make for each task (5), multiplied by the number of participants that performed the experiment (39). We used the method of “nx2 contingency tables” to make a statistical inference. We did not find a significant difference between error rates of the three samples (χ2(2) = 3.866, p = 0.145). Error rates are: 3.08% for “paper”, 1.54% for “tablet”, and 0.51% for “smartphone”. These results allow us to reject both hypotheses H2 and H4. Also, for SUS scores, we first checked the samples for normality. Two out of three of the original samples did not follow a normal distribution. Thus we applied the Box-Cox transformation with α = 3.0378. However, the “paper” sample did not follow normal distribution also with the transformation. Then, we used the Kruskal-Wallis test for the sample comparison. We found a statistically significant difference between the three samples (χ2(2) = 57.626, p < 0.001). Pairwise comparisons revealed that usability was significantly higher with “smartphone” than “paper” (T = −48.577, p < 0.001), as well as with “tablet” than “paper” (T = −52.077, p < 0.001), whereas there was not a statistically significant difference between “smartphone” and “tablet” (T = −3.500 p = 1.000) modalities. Mean SUS scores are (Fig. 3): 55.4 for “paper,” 84.6 for “smartphone,” and 86.9 for “tablet.” These results allow us to confirm hypothesis H3 and to reject hypothesis H4.

6 Discussion and Conclusion

We developed an AR application that augments a P&ID drawing of a plant, thus allowing operators to retrieve useful information for the maintenance procedure, as the location of equipment in the plant.

We effectively tested the application in the scenario of a milling plant through a user study. Considering the results of the user study, we found that AR is effective in the retrieval information task in term of task time reduction. This result confirms what was found in the literature in other industrial scenarios [20,21,22,23]. However, in terms of error rate, we did not detect any statistically significant difference between the use of AR and paper documentation. This could be due to low task difficulty.

The SUS test reveals that users prefer to use the AR application rather than paper documentation, although they are more accustomed to paper manuals/drawings than to AR. This result is mainly due to the minimalist, but effective design of the interface designed for this application, compliant to our vision of a gradual introduction of new approaches in industrial practices.

An interesting result, for the AR modality, is that the use of handheld devices with different screen sizes (smartphone and tablet) does not affect user performance. In a smaller screen, the density of virtual elements simultaneously displayed in the interface is higher. Then, the distance between two hotspots in our application is lower in the smartphone, thus increasing the possibility to interact with the wrong hotspot. However, we noted that the user naturally tended to bring the smartphone closer to the drawing after they understood which hotspot interact with. On the other side, having a device with a smaller screen implies an easier and more rapid interaction with touch, which can also be done with one hand.

An innovative aspect of the proposed solution is the automatic update of the virtual hotspot location on the P&ID when there is a layout modification in the plant. In this way, using our application, operators can accomplish their maintenance tasks with low mnemonic effort even in case of frequent layout modifications, a situation even more frequent in the Industry 4.0 context. Furthermore, external operators, that do not know the plant layout and processes can be employed. With this tool, technicians can easily understand the components and connections in the plants even if they do not know the coding of the symbols used in the P&ID. However, this tool does not help operators in support of decisions, since it does not provide further information, for example, for the planning of maintenance procedures. Then, a future step of this research will be the integration of other features in the framework, as the identification of the equipment to inspect on the drawing and the displaying of selected KPIs directly superimposed on the equipment. Once all these features have been implemented, a user test with experienced operators would be needed to validate the framework.