Keywords

1 Introduction

Inspection and diagnostics of large contemporary systems usually requires direct acting and big effort of human experts. There are many examples supporting this statement. One of them is inspection and diagnosing of an actual state of roadways in a coal mine affected by a catastrophic event such as explosion of methane or coal dust, or even exogenous fire. Here human inspectors are faced with significant risks of explosion or can be affected by dangerous components of the atmosphere such as carbon oxide. Therefore, in many situations the inspection of such a roadway may be impossible when carried out by humans. Other example is inspection of large objects such as concrete dams or cooling towers installed in power stations. In this case human inspectors have to hang on ropes while inspecting surfaces of structures. It is worth to mention also inspections of wind turbines, especially those that are located on sea shelf. Two latest examples require long-lasting work carried out in dangerous circumstances. Bearing all of them in mind, the author formulated a conception of “robotized diagnostics” as a means of freeing human inspectors by carrying out risky works, and by carefull examination of the evidence and then arriving at conclusions and preparing reports.

The author came up on the idea of the robotized diagnostics as some generalization of his experiences and works carried out in the field of technical diagnostics, mobile robotics, and applications of artificial intelligence. First formulation of this conception has been presented at the International Congress of Technical Diagnostics in 2016. This paper goes far beyond the scope of this first presentation, showing more general conception of a system of robotized diagnostics and intelligent interpretation. It is composed as follows. Section 2. deals with general model of the process of inspection and diagnostics. In Sect. 3. we introduce a conception of Mobile Diagnostic Agent (MDA). Some issues of controlling MDA are signalled in Sect. 4. Intelligent support of investigations is described in Sect. 5. Some examples of applications of Robotized Inspection and Diagnostics are given in Sect. 6. The paper ends with conclusions.

2 Process of Inspecting and Diagnosing

According to [3], inspection is “critical appraisal involving examination, measurement, testing, gauging, and comparison of materials or items. An inspection determines if the material or item is in proper quantity and condition, and if it conforms to the applicable or specified requirements”. It is assumed that the actual state of the object being inspected is not affected by the inspecting agent. The second keyword, diagnosis, is [4] “Identification of a condition, (...), or problem by systematic analysis of the background or history, examination of the signs or symptoms, evaluation of the research or test results, and investigation of the assumed or probable causes”. If one analyzes both the definitions it comes out that diagnosis includes inspection. Therefore, further on we will focus our attention on diagnosing process rather than inspecting one.

Fig. 1.
figure 1

Process of diagnosing

Full size image

The process of diagnosing is in general form presented in Fig. 1. In the following we will shortly discuss the subsequent steps of this process, basing on [7].

Planning concerns preparation of the whole process. It should take into account: identification of the goal of the process, quantities to be measured, kinds of sensors and their location, possible states of the object to be evoked, scenarios of observations, numbers of repetitions of the observations, safety requirements, and many others. To accomplish these tasks, relevant knowledge is necessary.

Running the experiment depends on whether the experiments are passive or active. The former involve simple observation, while the latter is connected with stimulation of the object by special forcing phenomena. To this end, relevant means such as specialized tools or equipment may be required.

Collecting data and observations is carried out while the experiment is run. The goal is to acquire relevant data whose sources are sensors (in wide sense – including cameras, microphones, and many others) that are applied in order to observe the object, and then to store the data into a database in some systematic manner. Furthermore, observations of experts and other personnel who are involved in controlling the process to be diagnosed are collected. This task is usually carried out by human inspectors or diagnosticians.

Analyzing data and estimating features is vital task that can be completed either in real-time, or once the experiment has been finished. This activity requires domain-specific knowledge and can be very time-consumming.

Diagnostic reasoning is necessary to arrive at conclusions. It takes advantage of one or more general approaches (deduction, abduction, induction, case-based reasoning, and many others). This task requires relevant knowledge base. Furthermore, due to noise contained in data, and also approximate and uncertain knowledge, application of approximate reasoning methods can be necessary.

Reporting is the last stage of the process of diagnosing. The goal is to prepare reports for end-users. This is a very time-consumming activity, which ought to be supported by some intelligent report generator.

The analysis done so far allows us to conclude that it is possible to aid the human expert by diagnosing, using specialized technical means and intelligent information systems.

3 Mobile Diagnostic Agent

The approach presented in the paper takes advantage of a concept of “Mobile Diagnostic Agent” (MDA) whose general idea can be based on the Intelligent Agent [13]. The environment for MDA is (or includes) the given plant where the process to be diagnosed takes place. The MDA is equipped with sensors that collect percepts (data and observations). The MDA is also equipped with effectors whose meaning is twofold. First, they allow the MDA to move to the plant and within the plant’s borders, and, finally, to return to the operator. Second kind of effectors can be used to accomplish active diagnostic experiments in which the plant is stimulated by the MDA in order to acquire the plant’s response.

Fig. 2.
figure 2

Process of diagnosing with the aid of MDA for running experiments and collecting the evidence

Full size image

It is postulated that the MDA be capable of carrying out tasks that are distinguished by having red background in Fig. 2. When required, the MDA should be equipped with tools and effectors that allow it to stimulate the plant in order to record its response.

4 Control of Mobile Diagnostic Agent

An operation of the MDA described in Section 3 is controlled by a control box (the rectangle with question mark). Two modes of operation are typically used: Remote Control carried out by an external operator, and Autonomous Operation.

4.1 Remote Control

Basing or wide bibliography research one can state that in contemporary systems this mode of control is preveiling. To allow remote control, the MDA must be equipped with a broadband communication (usually wireless) with little latency. This mode may be understood as an extension of the limbs and senses of a diagnostician.

4.2 Autonomous Operation

Nowadays several research centres carry out research concerning autonomous systems. To allow autonomous operation of the MDA, its control box must be significantly improved. The MDA must be able to localize and orient itself. It has to sense and understand the environment basing on percepts collected by its sensors. The control box must execute the plan of diagnosing, adopting it in case of perceiving any problem. It has also to decide on using effectors in order to stimulate the plant under test. The most advanced MDAs are fault-tolerant.

All the functionalities listed above require complete knowledge about the MDA itself, and of the plant, and possibly of this part of the environment the MDA must traverse in order to arrive at the plant.

4.3 Virtual Teleportation as a New Functionality

The last requirement concerning complete knowledge base is very difficult to satisfy due to uncountable number of states of the real world. Therefore, the MDA can be faced with problems that it is unable to solve by its own. An example of this may be a four-wheeled robot that cannot escape from muddy terrain. To this end, an innovative technology of Virtual Teleportation (VT) can be appliedFootnote 1.

The goal of VT is to evoke in the operator the feeling as if he/she were personally present at the place in which MDA operates. To achieve this, an innovative Human-Machine Interface (HMI) is applied, what involves high speed broadband bidirectional communication, good quality stereovision and possibly special controls. Moreover, HMI can use other human senses such as hearing or touching. Innovative techniques of Augmented Reality (AR) and Virtual Reality (VR) may be used to amplify the effect of VT of the operator. Furthermore, the operator can control the MDA by specific behaviours such as gestures or movements of body parts, particularly limbs.

5 Specialized Expert System

Although a human diagnostician can be replaced by MDA in carrying out experiments and collect evidence (see Fig. 2), there still remain several tasks that constitute the process of diagnosing, which usually are worked out by humans. Majority of them require domain-specific knowledge. The need to analyse huge amounts of data and even thousands of photos in order to estimate relevant features is an example of a knowledge-intensive task. Diagnostic reasoning is other task that can be carried out using an intelligent support. Moreover, planning of the work also needs knowledge-based work. Finally, the last task of the process – reporting – engages a lot of time of the diagnostic team. Concluding, there is a high rationality to aid diagnosticians with an intelligent tool that could take over the work. The scheme of the process presented in Fig. 3 proposes these tasks that can be taken over by a specialized diagnostic expert system [5, 12].

Fig. 3.
figure 3

Process of diagnosing with the evidence collected by MDA, and other stages carried out by an expert system

Full size image

Contemporary intelligent systems are capable of mining huge amount of data searching for subfiles of data that may carry interesting information. When one considers the essence of inspection or diagnosis, the goal is to find the evidence that can support diagnostic conclusions. These findings can be either regularities that are identified in randomly distributed data [14], or – in contrary – irregularities discovered among regular patterns. Findings identified in an automated way by the system that replaces a human can then be presented to the expert in order to interpret them and to formulate diagnostic conclusions. If the diagnostic expert system is equipped with sufficient knowledge, then the findings can be interpreted in an automated way, too.

6 Examples of Application

Robotized diagnostic systems, although very advanced and requiring innovative research and development, are ever more and more often applied. In the following some examples are presented that correspond to the research carried out by the R& D team the author belongs to. The examples concern inspection and diagnosis of coal mine roadways, inspection and diagnosing of wind turbines, and inspection of airplanes.

6.1 Inspection of Underground Galleries

If a catastrophic event happens to part of a coal mine, the roadways and galleries affected are closed by meand of a special dam that separates the area from the working part of the coal mine. The rescuers-inspectors usually are not allowed to enter the closed area of the coal mine because of risks caused by explosive and poisonous atmosphere, and also too high temperature. To this end, an innovative robotized system taking advantage of VT techniques is developed [8]. The idea of the system is presented in Fig. 4.

Fig. 4.
figure 4

Conception of the TeleRescuer inspection system [8]

Full size image

The system is composed of: a mobile robot (a case of MDA) capable of traversing obstacles it might be faced with in the roadway affected by catastrophe, a broadband communication subsystem, and an operator station (Fig. 5). The mobile robot is equipped with respective sensors, stereovision subsystem, and infrared cameras. Since the robot conforms with ATEX M1 Directive it is able to operate in explosive atmosphere. Additionally, the robot can prepare 3D maps basing on 3D laser scanning subsystem [11].

Fig. 5.
figure 5

Control system and main parts of the TeleRescuer System [2]

Full size image
Fig. 6.
figure 6

(Courtesy: Petr Novák)

Portable operator station of the TeleRescuer system

Full size image

In this application the robot (MDA) mainly plays the role of an extension of the human senses. This is possible thanks to an innovative interface that takes advantage of Virtual Teleportation (Fig. 6). To create more realistic interface and to allow easier control of the 8-DOF system, stereovision has been applied. This stereo image is supplemented by mono IR image and point clouds collected by 3D mapping system. The operator can choose between a game controller and typical controls used in airplanes. More detailed description of the system can be find in [2, 8, 11].

6.2 Inspection and Diagnosing of Wind Turbines

Wind farms become ever more and more important sources of electric energy. The future of this kind of reneval energy sources are off-shore wind farms that can group more than hundred of wind turbine generators (WTG). They have to be periodically inspected by human inspectors [6]. In most cases such inspections are carried out by highly-trained human inspectors who work on special platforms mounted on ropes that are hooked to the turbine nacelle. In case of contemporary WTGs the height of operation of inspectors can be more than one hundred metres above sea level. Blade inspection takes one day for two inspectors.

Taking this into account the team from the Silesian University of Technology developed a conception of a robot for inspecting blades of WTGs, for which an European Patent has been obtained [9]. The robot motion subsystem is capable of climbing along the blade and scanning both the surfaces of the blade by means of respective set of cameras and sensors. Figure 9 shows the scheme of the mobile platform of the robot equipped with two arms that can move pendulum-like scanning both the surfaces of the blade. Furthermore, the technique of moving along the blade is explained. The moving platform can adopt to changing width of the blade by means of additional DOFs that allow bending two parts of the platform frame (cf. Fig. 7(a), up and down).

By replacing human inspectors by the robot the duration of the inspection can be substantially decreased. The robot may perform not only passive experiments based on observations, but also active diagnostic experiments using active termovision, ultrasonic testing, or even a tap test. Moreover, small repairs can be carried out. If the goal of the inspection is to collect photos to be analysed off-line, that the experts are faced with interpreting thousands of images. The experts can be replaced by this activity by an expert system.

Fig. 7.
figure 7

Robot for inspecting blades of WTG: (a) mobile platform, (b) method of moving along the blade [9]

Full size image
Fig. 8.
figure 8

Inspection of WTGs by a specialized drone [10]

Full size image

Nowadays such inspections are carried out by drones [10] (Fig. 8). The advantage of this approach is further significant reduction of time. It is possible to inspect up to sixteen WTGs daily. However, such a kind of inspections is more depending on weather conditions. Furthermore, since energy resources of a drone are limited, no active experiments can be performed.

6.3 Inspection of Aircrafts

Nowadays drones have become a very promising tool for inspecting and diagnosing large objects and structures. Nevertheless, due to limited energy sourced on board, they can be applied to reduced variety of tasks. As one of examples may serve inspection of aircrafts at airports.

An interesting example concerns examination of the integrity of structure of passenger aircrafts [1]. The inspecting drone is controlled by a team composed of the drone operator and inspector(s). The goal of the drone is to deliver high-quality images taken from structural parts of the plane (Fig. 9(a)). The images are interpreted in real time by the inspector. Further actions of the drone can follow the advices and requirements of the inspector to allow him/her to identify possible flaws.

Fig. 9.
figure 9

Quality inspection of an aircraft by a drone: (a) drone inspection, (b) 3D digital model including information carried by images collected by the drone [1]

Full size image

A very innovative part of this system is the User Interface (Fig. 9(b) [1]). The images presented onscreen are based on information fusion from: CAD models containing information of structural parts of the plane (obtained from the producer), maintenance records, and findings collected by the drone. Such advanced 3D model, which additionally can be easily operated by the inspector, allows him/her to better understand the reasons of faults, to assess their severity and development of defects, and provisionally plan corrective actions.

7 Recapitulation and Conclusions

In the paper some fundamental issues of robotized inspection and diagnostics have been briefly discussed. Contemporary development of mobile agents such as riding robots and unmanned aerial vehicles, allow replacing human diagnosticians by hazardous and boring work. Although nowadays the remote control is most often used, peculiar attention should be paid to autonomous systems, which however can take advantage of the possibility to engage human operator if the on-board knowledge-based control system is unable to solve the given problem. To this end, Virtual Teleportation technology can help in better understanding the problem by the expert, and then by solving it remotely.

Further on, the experts can be replaced by analysing big data collected by many sensors the robots are equipped with. Specialised expert systems containing relevant knowledge can search through huge evidence in order to select reduced amount of results that require diagnosticians’ intervention, interpretation and personal assessment.

The work required for the development of such complex industrial systems is interdysciplinary, since it involves colaboration of mechatronic engineers, electronics, automatic control, informatics, and finally – domain experts. Therefore, to achieve the general goal that is implementation of such systems in modern economy, there is necessary a very efficient collaboration of many R & D centres, universities, and companies, together with experienced end-users.