Keywords

1 Introduction

Sectors characterised by small batch production and complex products (e.g. aeronautics) need to combine high levels of flexibility with high productivity rates. In such sectors, assembly and auxiliary operations are mainly performed by humans as they bring inimitable agility to adjust to changes, as well as skills that cannot be replaced by automation. However, manual intensive activities can also present disadvantages such as potential physical or mental limitations that can restrict overall performance of the assembly system.

In a scenario with ever-changing demands, assembly systems need to put together humans and automation taking advantage of each other’s strengths to balance flexibility and productivity requirements in an easy and cost-effective way. This collaboration raises challenges that must be faced to get a successful collaborative workplace: human and robot must know about each other situation; they must be able to interact naturally; and personalised and adapted support must be provided to operators specially in new assembly processes.

In this paper, we present a generic approach for new assembly scenarios that face all these challenges based on a semantic approach. Section 2 presents the related work. Section 3 describes the semantic approach including the main components overview and a detailed description of the VAR ontology. Section 4 presents the application of the system with the corresponding ontology instantiation of two use cases. Section 5 includes the discussion of both experiences. Finally, in Sect. 6, conclusions and future work are presented.

2 Related Work

One of the main issues related to the presence of the Semantic Technologies in the manufacturing domain is the lack of generally accepted and available ontologies. Furthermore, although some proposals have been done during recent years, few of them are public and available for reuse.

On the one hand, there are ontologies aimed at covering the manufacturing domain area, such as Manufacturing’s Semantic Ontology (MASON) [1], the P-PSO Ontology [2] or the (Manufacturing Core Concepts Ontology (MCCO) [3]. On the other, there are ontologies covering a very specific area of the manufacturing domain. ExtruOnt ontology [4] aims at describing an extruder, CM-Core ontology [5] is aimed at representing the core entities of the condition, PRONTO Ontology (Product Ontology) [6] captures the core concepts to represent products, Ontology of Standard of the Exchange of Product model data (OntoSTEP) [7] aims at representing product information but focusing on their geometry, and the Manufacturing Service Description Language (MSDL) [8] ontology represents the production service capabilities. However, none of the mentioned ontologies deal with the information exchange required among the different agents in manufacturing scenarios. Although they do not ensure interoperability in a semantic level, there is a group of relevant and extended standards that have been developed for information exchange in the manufacturing domain.

Business To Manufacturing Markup Language [9] (B2MML) is an XML implementation of IEC/ISO 62264 that is an international standard for enterprise-control system integration. B2MML is meant to provide a common data definition to link enterprise resource planning (ERP) and supply chain management (SCM) systems with manufacturing systems such as Industrial Control Systems (ICS) and Manufacturing Execution Systems (MES).

AutomationML [10] aims to standardise data exchange in the engineering process of production systems. Therefore, AutomationML e.V. develops and maintains an open, neutral, XML-based, and free industry data representation standard which enables a domain and company crossing transfer of engineering data.

eCl@ss [11] has established itself internationally as the only ISO/IEC-compliant industry standard and is thus the reference data standard for the classification and unambiguous description of products and services. With the help of eCl@ss, standardised digital data transfer is enabled. As a result, classifications and product description properties can be exchanged across the value chain.

3 Semantic-Oriented Framework

The semantic-oriented framework aims to support adaptive, interactive, assistive and collaborative assembly workplaces in an ever-changing scenario by providing: (1) plug- and-produce mechanisms to enable the reconfiguration of the workplaces; (2) natural communication enhancing human-automatism collaboration; (3) adaptation of the workplaces to the dynamic conditions of the environment and (4) personalised, context-aware guidance in the execution of productive tasks [12]. Figure 1 shows the set of key generic components of the framework and implements the aforementioned mechanisms by exploiting and exchanging the information through a central semantic repository based on a core ontology named VAR.

Fig. 1
Flow chart depicts the enterprise legacy system and security. It has Interaction devices, sensors and automation mechanisms, with different components represented by different colors.

Semantic-oriented framework reference implementation for collaborative workplaces

Full size image

The green components (Mediation Services, Device Manager and the Multimodal, Multichannel Interaction Manager) collect real-time context information from operators, automatisms (i.e. such as robots, machines or smart tools) even legacy systems and executing adaptation commands. While the blue components (Event Manager, Collaborative Asset Manager and Semantic Repository) enable real-time adaptation as well as personalisation, and finally, the orange ones (Decision Support System, Collaborative Knowledge Management and the VR/AR-Based Training and Guidance) are the ones in charge of providing context-aware assistance.

The Mediation Services, Device Manager and the Multimodal, Multichannel Interaction Manager components manage and interact with all the agents in the workplaces (i.e. legacy systems, automatisms and operators) and gather all the relevant real-time information coming from them. In particular, the Mediation Services enables collection of dynamic information about the operator involved and the operation in progress from Manufacturing Execution Systems; the Device Manager supports automatisms discovery by identifying the methods and variables exposed as well as status update (e.g. regarding the automatism itself or the assembly process), and the Multimodal Interaction Manager manages the commands coming from operators. All these components must verify the exchanged information and, if it is correct, include it in the semantic repository through the CAM component.

Then the Event Manager triggers the adaption and notification commands based on the defined rules and the dynamic context information stored in the semantic repository. Finally, the Decision Support System, Collaborative Knowledge Management and the VR/AR-Based Training and Guidance components consume the commands and notifications triggered by the Event Manager and all the information gathered in the semantic repository that reflects the dynamic and realistic view of the manufacturing process. Furthermore, they aim to assist operators the best way possible, considering their profiles and the dynamic context.

Operators are provided with the required knowledge and process definition and dynamic status information through the semantic repository according to the VAR ontology. This enables the reusability of all the components in different scenarios without any modification exception for the semantic repository, which requires a new instantiation of the VAR ontology for each scenario according to the targeted assembly process.

3.1 VAR Ontology

The VAR ontology is the core element in the semantic-oriented framework, enabling the data exchange from and to diverse agents in the assembly scenarios including external sources such as legacy systems (e.g. Manufacturing Execution Systems), operators, robots, tools and so on to make possible adaptive, interactive, assistive and collaborative assembly workplaces.

The VAR ontology was developed following the well-known NeOn methodology [13]. First, a group of manufacturing experts defined the scenario requirements, which were later registered in the form of Competency Questions (CQs) in the Ontology Requirements Specification Document (ORSD). These requirements included adaptation to workplace environmental conditions, natural interaction between humans and machines and optimal automation configuration among others. From these CQs, the main ontology concepts were extracted.

The VAR ontology’s design has been based on the B2MML standard in order to enhance interoperability with external legacy systems such as ERP and MES. In the context of the SatisFactoryFootnote 1 project, this standard was translated into OWL. Following the ontology reuse best practices, a total of 18 classes and 48 properties have been reused by the VAR ontology. As for the requirements of the new assembly workplaces which were not covered in the B2MLL OWL version, a set of new resources were defined in the VAR ontology. As a result, the VAR ontology is composed by 86 classes, 97 object properties and over 70 data properties.

The VAR ontology follows a modular approach avoiding strong dependencies between modules in order to empower its module’s reuse, to support more efficient query answering and to enhance modules’ evolution [14]. Furthermore, this ontology modularisation has been undertaken from the ontology design stage to avoid performing arduous and time-consuming ontology modularisation techniques in the future.

It is worth mentioning that the VAR ontology does not contain any contradictory facts, as a Pellet reasoner has shown its logical consistency. This consistency feature is of utmost importance for the VAR ontology, as autonomous software agents may perform reasoning tasks with instantiations and come to conclusions without human supervision. Therefore, without ensuring ontology consistency, wrong conclusions could be deduced. Additionally, all the defined CQs are adequately addressed by the VAR ontology; thus, it is considered verified.

The ontology can be divided into four main modules: manufacturing assets; plug and produce; traceability and interaction. The modules are related to each other through five main properties connecting classes from different modules, as it is shown in Fig. 2.

Fig. 2
Flowchart of VAR ontology main modules, has four major components such as, Traceability, Manufacturing asset, Plug and produce, Interaction. Each component has many sub components.

VAR ontology main modules

Full size image

Manufacturing Assets Module

The manufacturing assets module contains all the relevant classes and properties for defining products produced by assembly processes. It includes physical entities (tangible assets) in assembly workplaces, such as product, material, equipment, personnel and interaction devices, as well as non-physical entities (intangible assets) like processes. A UML representation of the excerpt of the VAR ontology of the manufacturing assets is shown in Fig. 3.

Fig. 3
Flow chart of UML diagram of tangible assets represent excerpt in VAR ontology with segments like Product and Process, Specifications like Equipment segment, Personnel segment, Process segment and others like Equipment class, equipment, personnel class, person and parameter.

UML diagram of tangible assets representation excerpt in VAR ontology

Full size image

The product is represented by the ProductDefinition class that is composed of product segments (ProductSegment class). In turn, each product segment can be made of a set of product segments following a dependency flow (hasSegmentDependency). Each product segment is defined by a process segment (ProcessSegment class) that represents the personnel and equipment resources required to carry out a production step, and it can be made of a set of process segments following a dependency flow.

A person (Person class) represents a specifically identified individual with each own characteristics and capabilities and can be described by a set of properties (PersonProperty class) that can be grouped based on the personPropertyType data property in a specific PersonnelClass.

Plug and Produce Module

The plug and produce module defines all the necessary classes and properties for supporting automatisation in assembly workplaces such as OPC-UA server, methods and variables that equipment provides.

Both robots (Robot class) and smart tools (Smarttools class) are automation mechanisms involving adaptation capabilities and are represented by an OPC-UA server (OPC-UAServer class) to support the standard-based plug-and-produce approach. The OPC-UAServer class provides a binary representation of both, and it monitors variables (Variable class) linked to changes in robot/smart tool properties and execute methods (Method class). Furthermore, the methods can involve a set of parameters (MethodParameter class) as shown in Fig. 4.

Fig. 4
Flow chart of UML diagram of plug and procedure excerpt in VAR ontology with Robot, Variable, O P C – U A Server, Method, Smart tool and Method parameter.

UML diagram of plug and produce excerpt in VAR Ontology

Full size image

Traceability Module

The aim of the traceability module is to gather all the necessary trace information. For that, it includes classes like JobOrder or TraceProcessSegment. Furthermore, this module enables to have in real time actual context status control to support adaptation capabilities. The involved object properties are updated in run-time according to the real situation: some of them directly through the services provided by the CAM, and others based on semantic rules, property chains and logical inferences. The current situation is controlled by the job order (JobOrder class) which is linked to the operation (ProcessSegment class) in progress as well as the involved equipment (Equipment class) and worker(s) (Person Class) through the TraceProcessSegment and its links to the rest of the instances of the mentioned classes (like JobOrder isExecutedBy Equipment, or Person isLoggedIn Equipment.

Interaction Module

For interaction issues, the ontology includes individuals like Start, Stop, Resume and Move belonging to the class BasicAction (a subclass of TemporalThing class) that are used to determine the commands that can be used to interact with the automatisms linked to their Methods.

This module is also in charge of representing the notifications and the related channels as well as the interaction devices that supports the natural and adapted interaction through Notification, Channel and InteractionDevice classes, respectively.

4 Use Cases

To demonstrate the easy adoption of the semantic approach presented in the previous section, we have deployed such system in two assembly scenarios: (1) optimisation of the assembly and tightening of the hydraulic system on the A350 over wing panel (OWP) including automatic tool configuration, on the job guidance and traceability at Airbus and (2) the collaborative assembly of a latch valve where the system adapts itself to the operator’s characteristics and the operator interacts with the Manufacturing Execution System, an industrial assembly robot and a mobile logistic robot in a natural way (i.e. using voice and/or gestures) at Tekniker’s facilities.

In both scenarios, the VAR ontology was instantiated detailing the corresponding process step by step: including all the task dependency restrictions as well as all the parametric configurations. Furthermore, during task execution, once the corresponding automatism (smart torque wrench, dual arm robot or logistic robot, respectively) was discovered, the related OPC-UA servers is, automatically, instantiated. Figure 5 includes an RDF excerpt that shows, in the Airbus scenario, the smart tool M05 discovered and related to the specific work centre (MS40.A) where it is operative. The OPC-UA and smart tool information is provided by the smart tool itself once it is discovered, publishing it in the semantic repository through the CAM services. As for the IP and the work centre, they are dynamically updated when discovered.

Fig. 5
Coding of smart tool discovery RDF excerpt is depicted.

Smart tool discovery RDF excerpt

Full size image

In addition, during the operation, the dynamic current status, including full traceability, is updated. For instance, in Fig. 6 you can see the soi.001-Task_03_04_01-JO_soi001 operation trace individual with the real reached workbench value (23.56), the operation start and end time as well as who has participated in such an operation.

Fig. 6
Coding of dynamic status RDF excerpt is depicted.

Dynamic status RDF excerpt

Full size image

The RDF excerpt also includes a malfunction reported by the smart tool (M05-07-17T08:41:28Z) during the job order execution. All this information is exploited by the quality and metrology personnel to supervise the task execution, identify potential conflictive operations and even decide on the life of certain smart tool.

5 Results Discussion

The Airbus’s evaluation involved seven participants completing an experiment where they were trained to use the HoloLens and smart tool on a mock-up before completing a hydraulic pipe installation in the OWP of a test aircraft. The usability was explored through surveys gathering quantitative data on usability and mental workload. The results showed a good level of usability for all usability dimensions. The usability and mental workload mock-up scores were better than the scores obtained from the participants after completing the task on the OWP. Another potential benefit identified is the improvement of the productivity due to the reduction of time required to search for information and to change tool or the increased traceability as everything can be recorded and reported.

In Tekniker’s evaluation involved, twenty participants completed the assembly process and included the assessment of usability (i.e. including both the gesture and voice-based interaction), mental workload and trust in human–robot interaction. The usability, mental workload and trust scores were all positive, indicating good usability for the system. The participants’ responses indicated that they found the voice input more usable than the gesture inputs, and this may have resulted from the ability to use their natural language rather than having to remember the gestures to use. Furthermore, some potential benefits such as an increase in productivity due to a reduction of the displacements have been identified.

6 Conclusion and Future Work

This paper tackles the challenges that arise when putting humans and automation together in collaborative manufacturing scenarios, to leverage each other’s strengths to balance flexibility and productivity requirements in an easy and cost-effective way. Towards that goal, a generic semantic-oriented framework based on the VAR ontology has been developed, including modules addressing: (1) automatisms plug-and-produce mechanisms to enable dynamic reconfiguration of the workplaces; (2) natural communication enhancing human-automatism collaboration; (3) adaptation of the workplaces to the dynamic conditions of the environment and (4) personalised, context-aware guidance in the execution of productive tasks. All these modules take advantage of the real- time semantic information representation, according to the VAR ontology.

The reusability of the generic approach has been demonstrated by deploying the framework in two real scenarios, and the experimentations carried out in them show that the functionalities supported by the framework are well accepted and exploited by the users, leading to an increase in productivity even in changing environments.