Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

The Condition-based Maintenance approach (CBM) applies predictive techniques to avoid unforeseen breakdown of equipments. This kind of systems predict where, when and why the failure will occurs [1]. CBM systems are often used in equipments which breakdown impacts directly the production line. Intelligent Maintenance Systems (IMS) combines systems software and sensors for a CBM approach. This represents an evolution from the traditional systems of corrective and preventive maintenance to predictive systems [2, 3].

IMS new paradigm increases the role of sensor networks and computational forecasting systems that are constantly monitoring the equipments, providing support to repair and maintenance decisions [4].

On the other hand the maintenance technicians still holds many of the information associated with the operation of the equipment. The manner in which they perform repair activities may result in different future demands for maintenance. Human and context extrinsic (environmental) and intrinsic (skills, mood) factors are determining factors in activities of repair and diagnostic for future maintenance.

Thus the rescue of the role of the operator and the addition of context information to the monitoring, intelligent diagnosis and prediction of maintenance activities can contribute to improve the CBM approaches, enhancing the requirements of cost, time and quality of the processes, making IMS smarter.

In this context the challenge is to combine digital and numerical information from monitored equipments (virtual components) with qualitative and subjective perceptions from the users and the context (real components). What factors should be perceived? How to perceive them? How to integrate these real impressions to the system? These are open questions.

In recent years technological advances are leading to new relationships between humans and machines. New digitally supported environments arise allowing different agents (real and virtual) to interact in a way previously unimaginable. In this new scenario there are new theories, methodologies, techniques and tools from different areas related to Human-Computer Interface.

In this paradigm different objects equipped with embedded computing can interact each other. They can sense and adapt to the environment in a transparent manner, making the HCI simpler [5].

Researches on ubiquitous computing seek models for the interconnection of “things” (objects, computers, animals, people, etc.) in a network, similar to the devices today already interconnected through Internet [6, 7].

More recently a new approach has emerged, coining the term Cyber-Physical Systems (CPS). A CPS is a system that combines and coordinates physical and computational elements [8]. The CPS integrates the ability to act in the physical world and the intelligence from cyber (virtual) world, adding a new Human Computer Interface (HCI) and resources to real world systems [9]. Possible applications of CPS systems are in the medical, education and security areas [10].

Considering the perspective where the main maintenance tasks or fields are grouped into three primary categories: technical part, economic part and human part, our work is in the human part. Human factors associated with the maintenance management play an important role in improving the efficiency and effectiveness of maintenance processes. The operator teams have lots of knowledge about the system to be repaired which cannot be obtained solely through data analysis of sensors [11, 12].

We propose a HCI for IMS focusing on how CPS can (re)introduce the human and context factors in maintenance activities. The role of human factors in maintenance activities need to be clarified to ensure better understanding and integration of the individual knowledge and capabilities of the service personnel. This can lead to an IMS in which these information might be acquired and used for the planning of maintenance processes and forecasting the demand for spare parts.

As until now it is not possible to integrate the knowledge and the capabilities of the service personnel into the IMS, it was developed an user interface that is suitable for meeting the requirements for suchlike integration. The idea aims to explore the CPS concepts and how the human knowledge and capability can be integrated into the IMS in order to support the planning and the collaboration between the different service personnel so that the responsiveness and the skills of the IMS can be maximized.

Advanced Technologies in Virtual Reality and Sensor Networks was used to allow taking into account human factors in the maintenance activities. The proposed CPS for Human Computer Interface is based on TOOGLE platform that have four main sub-systems: i. Middleware and Components, ii. Editor, iii. Intelligent Decision and iv. Browser. We use the Editor to create and edit the different components (real and virtual) of the maintenance scenario, which is called hyper-environment with its components interconnected through the Middleware. The Intelligent Decision module gathers a set of applications related to the prediction maintenance. Finally the Browser allows visualization of different multi-modal information in the system.

In what follows we present a contextualization and the CBM architecture for maintenance processes (Sect. 2). Section 3 presents our system implementation to integrate both technologies (IMS and CPS). Some experiments with the proposed system (Sect. 4) and the conclusions of the paper (Sect. 5) are finally presented.

2 Condition Based Maintenance

The development of systems for Condition-based Maintenance should include the integration of a wide variety of hardware and software components.

Figure 1 shows the OSA-CBM, which provides architecture and a standard framework for implementing systems for Condition-based Maintenance [13]. This architecture has six functional blocks:

Fig. 1
figure 1

OSA-CBM architecture [13]

Full size image

  • Data Acquisition (CA) converts an output from the transducer into a digital parameter;

  • Data Manipulation (DM) performs the signal analysis, calculates significant descriptors and provides the virtual sensor readings from the measurements obtained;

  • Detection State (SD) provides normal “profiles”, search for abnormalities whenever new data is acquired, and determines to which area of abnormality, if applicable, the data belong;

  • Health Assessment (HA) provides fault diagnosis and rates of current health of the equipment or process, considering all state information;

  • Prognostic Assessment (PA) determines future health states and failure modes based on the current assessment of health and use estimation of future demand for equipment and/or process, as well as the remaining useful life.

  • Advice Generation (AG) provides practical information on maintaining.

Notice that the module Human-Computer Interface is a small module. It is proposed that maintenance routines and failure diagnosis can be improved taking in account human and context perceptions and information [14] and in this paper we intended to develop a solution for implantation of a system that can rescue and add human and context factors providing a smarter IMS.

3 Functional Requirements

In order to take into account the human factors in IMS approaches, a number of technical and social issues have to be treated. For instance, an issue is how to allocate functions among service personnel and computer systems. Another key issue is how to design a human-oriented software system to capitalize on and accommodate human skills in perception, attention, and cognition, while minimizing the opportunities for and effects of human error. Operator decision support should be designed such that the operator downloads almost all of the skill-based commands and some of the rule-based commands, while retaining the knowledge-based tasks for him.

From the IMS and HCI requirements, a set of functional requirements of our Human-Centered IMS may be identified as follows:

  • Responsiveness;

  • Sharing of knowledge and resources with other agents/users;

  • Access to existing information;

  • Dissemination of results;

  • Fostering a creative environment for generate and introduce new information to the system, from the skill and mood of service personal.

The input data of the service personnel into the HCI and provided data by the HCI is the following:

  • Location of the technical system (HCI);

  • Product ID (HCI or service personnel);

  • Bill of material (HCI);

  • Damaged part(s) identified in the bill of material and/or in the visualization of HCI via e.g. a smart mobile device (service personnel);

  • Required skills and equipment (HCI or service personnel);

  • Description of the problem (service personnel);

  • Reasons for instance due to the environment (service personnel);

  • Breakdown probability or estimated breakdown date (service personnel).

An HCI-IMS may be informally defined as a “cyberspace structure of planning, control and communication mechanisms to support human decision-making via monitoring and simulation of actual maintenance situations through modeling of all activities and resources in a physical maintenance system”. It can be used either as a design tool or as an operational tool. The former is used to prototype Virtual Manufacturing Systems (VMS) and can be called Editor HCI-IMS and the latter is used to simulate and control Manufacturing Systems through VMS and can be called Operational HCI-IMS.

Our approach will focus in advanced computational technologies associated with the Intelligent Environments concept. These are heterogeneous distributed sensors-actuators systems including multimedia presentation services, automation and control components, intelligent physical objects, wireless sensor network nodes, nomadic personal or shared devices, and many other systems and entities in a Cyber and Physical System.

We introduce the intelligent environment concept through an architecture called TOOGLE, which supports the merge between various technological elements and their virtual representation, giving to designers and developers the necessary support for the creation of intelligent environments.

4 Toogle Plataform

The Toogle platform was proposed as an implementation of a technological architecture for HCI-IMS applications. It is a CPS targeted on IMS. It uses Robot Operating System (ROS) [15] for communication between real devices and the virtual world. Moreover, it uses Blender 3D rendering software [16] as object visualization tool for virtual-real industry environment. Figure 2 shows the modules that compose the system architecture.

Fig. 2
figure 2

Toogle platform architecture

Full size image

4.1 Middleware and Components

An IMS scenario is composed by real and virtual elements that can be grouped on the following categories:

  • Physical Entities: workers, equipments, tools, etc. They can be accessed by devices (sensors, actuators and tags); and

  • Digital Entities: a set of services associated with logs, scheduling, digital applications, etc.

The Toogle Components abstract the physical and virtual entities resulting in an identifier, a set of associated properties and resources available. Physical entities to be abstracted are called Smart Objects. The Toggle Middleware is responsible for communication between the components. It is composed by the ROS system. Each Smart Object and Digital Entity is a computing process, which can run in different machines. The Middleware allows synchronous and asynchronous communication. ROS has a set of drivers for different sensors and equipments.

4.2 Toogle Editor

Toogle Editor allows building and editing cyber-physical environments, providing navigability and interaction with the information provided by smart objects and digital entities. The module allows adding and removing components (Smart Objects and Digital Entities). Moreover it is possible to edit the features of these Components, i.e., their properties, resources (devices and services) and 3D representation. For 3D representation Toogle Editor uses the Blender tool that is an open source and multiplatform system. Figure 3 show a screenshot of the Toogle Editor.

Fig. 3
figure 3

Toogle editor

Full size image

An IMS scenario is composed by a set of components and is called a hyper-environment. Digital Entities (technical reports, tutorials, datalogs, prediction, planning) and Smart Objects (workers, tools, equipments) compose an hyper-environment besides a set of Maintenance Goals (optional). Toogle uses a STRIPS-like formalism to describe proprieties, resources and goals in Hyper-environments [17].

4.3 Toogle Intelligent Decision Module

This module provides intelligent making decision for IMS. We can add different tools and services to allow smart maintenance prediction and diagnostic.

Currently we offer two distinct services:

  • Toogle Physics Simulation: it allows realism and high performance physical simulation, updating the smart object properties. It uses Bullet open source library, which uses OpenGL for real time renderingFootnote 1; It has mechanisms for collision detection and rigid body dynamics. The physical simulator makes it possible to simulate objects that can fall, roll and collide with other objects, all with a realistic appearance. Aspects of scene lighting make use of GLSL and Pixel Shaders techniques;

  • Toogle Planning: a strips planning is used to achieve the hyper-environments goals.

Figure 4 shows a screenshot of the Physical Simulator.

Fig. 4
figure 4

Physical simulator

Full size image

4.4 Toogle Browser

The Toogle browser allows the distributed and remote access and visualization of information that has been created in Toogle Editor. We can browser for Hyper-environments—their smart objects and digital entities. Toogle Browser is a multi-modal viewer with a 3D viewer scenario where a replica of a real world scenario can be seen, according to the connections created in Editor. An interface for multi projection in a CAVE visualization environment is also available. This interface allows an immersive visualization of hyper-environments. Figure 5 shows a screenshot of Browser.

Fig. 5
figure 5

Toogle browser

Full size image

TOOGLE proposal explicitly considers the possibility of co-existence of virtual and real worlds in advanced HCI, focusing on the study of a large scalability of devices crossing/interfacing between them. The model of the architecture and implementation of TOOGLE as well as their autonomy and intelligence are being tested and validated in different scenarios involving heterogeneous devices of high scalability.

We intend to combine the novel technologies and facilities being developed to exploit and augment human knowledge and capabilities of the service personnel. The service personnel will have multi-modal, immersive 3D experiences in mixed physical-virtual worlds, including interaction with large surface displays, smart mobile devices, and wearable computers. As a result, human/context interaction with IMS will become significantly more natural, intuitive, and spontaneous than it is today.

5 Testes and Results

The platform has been tested and validated in two case studies. Real and virtual components compose a hyper-environment associated with the perception and integration of human and context factors to monitoring, diagnosis and prognosis of equipment maintenance. Next we present each experiment.

5.1 A Mobile Robot Maintenance

In this experiment we adopt as equipment to be repaired a mobile robot. This device may be considered as a complex system. The equipment is fitted with several sensors and actuators. A team of several technicians repairs the robot.

5.1.1 Components and Middleware

The experiment consists of the following components (equipment, humans and context) see Fig. 6:

  • Smart objects: mobile robot, operator, toolbox and tools, tablets and computers;

  • Devices: global and local cameras, GPS, RFID; and

  • Digital entities: tracking, logs, and scheduler.

Fig. 6
figure 6

Physical entities of study case: mobile robot, operators, tools, toolbox, tablets and cameras

Full size image

5.1.2 Editing Hyper-Environment

We have used the editor to create the components (smart objects and digital entities). Each component has a set of proprieties (position, charge level, on/off state, owner, etc.) and resources. Resources describe devices (cameras, GPS and RFIDs) and services (logs, trackers and scheduling). Strips-like predicates describe the proprieties and main goals to be achieved. Strips-like actions describe the resources. Smart object have also a 3D representation.

5.1.3 Toogle Intelligent Decision Module

We have used Bullet Physical Simulator to forecast the physical behavior of smart objects. mGTP planner [18] was used to plan a set of actions to achieve maintenance goals.

5.1.4 Toogle Browser

Figure 7 shows a snapshot of hyper-environment multi-modal representation. The browser allows accessing the different real time proprieties. For instance, location of the technical system and team, objects ID, bill of material, images from damaged parts, videos and procedures from used team skills and equipment, description of the problem, maintenance forecast.

Fig. 7
figure 7

Browsing in a mobile robot maintenance hyper-environment

Full size image

5.2 Monitoring Truckers in Shipyards

We have applied the platform in another case study related to the shipping industry. The case study was associated with the monitoring and truck repair in shipyards. We used a methodology similar to that applied in the previous example.

We decide to track trucks and workers with the use of Radio Frequency Identification—RFID. We have used Toogle Editor to create our components and their properties, resources and 3D representations. Figure 8 shows a screenshot of the hyper-environment obtained. The prototype was tested and validated, generating a database with the activities performed by all operators of the equipments and the positioning of the equipments and truckers.

Fig. 8
figure 8

Shipyard hyper-environment

Full size image

In two study case real and virtual components are mixed aiming to improve the topics associated with responsiveness, sharing of knowledge and resources with other agents/users, access to existing information, dissemination of results, and fostering a creative environment for generate and introduce new information to the system.

6 Conclusions

This paper presents advanced technologies for HCI in Intelligent Maintenance Systems. We have proposed a CPS approach for rescuing/adding the operator/context in CBM.

We have presented the Hyper-environment concept, which is formalism for hybrid world maintenance scenario description. Toggle is a framework to design Hyper-environments. It is composed by Middleware and Components (Smart Objects, Digital Entities and ROS); Editor (Multi-modal editor, Blender, STRIP-like description); Intelligent Decision (Physical Simulator and Planner) and Browser (multiplatform, mobile, stereoscopy, multi-projection).

As future works we intend to do more tests on IMS scenarios, to treat usability and friendliness issues, and finally to improve the Intelligence of the system through a recognize system for complex events [19].