Keywords

1 Background

The complexity of industrial equipment and increased requirements for reliability pose the on-line diagnostic task based on the analysis of a large number of monitored parameters. However, traditional methods aimed at measuring a limited set of signals and their subsequent analysis often do not allow to detect the rapid development of failures and emergency situations. Nowadays, measurement methods and devices allow you to record significant amounts of information about an object, for example, controlled object images in different spectral ranges: visible, infrared (IR) [1, 2] and ultraviolet light (UV) [3].

In the electronic component testing, the LOC-in Thermography is used [4]. To do this, the object under investigation is affected by low frequency thermal waves and the response is measured on the surface. LOC-in Thermography method increases on the order of the thermal imager sensitivity, making it easier to identify small areas with a slight overheating.

Another new method for electronic device inspection use Raman IR-Thermography [5, 6]. But in this case there is a problem of operational analysis and decision making, as the operator, or analyst does not have time to evaluate such a large volume of video data.

The authors develop an approach associated with intellectualization the process of analyzing information about the object technical states [7, 8]. This approach is based on the use the artificial neural networks (ANN) for processing the received information in real time. A fundamental property of ANN in this case is the ability of the neural network deep learning [9, 10]. For this, high-performance computing resources are used. Then, a trained neural network may be implemented as a program running on relatively simple computer.

The article is devoted to the study of the intellectual method of diagnosis and to the use the image neural network analyzer in various spectral ranges.

2 Problem of the Technical Object’s Diagnostics by Their Images in Various Spectral Ranges

The most widely used methods are found in infrared thermography, and they are often used in conjunction with the object visible images analysis. In industrial diagnostics also used UV control [3], which is focused primarily on the discovery the phenomena associated with the electrostatic field, creepage, corona and arc discharges in high voltage equipment. Joint processing of all three types of images was restrained by the lack of effective methods and means of automatic analysis.

The authors propose to perform the processing of a complex image as a whole, which includes three components: visible, infrared and ultraviolet.

Another problem, in most cases, is due to the absence of a representative dataset of actually measured images in different ranges. This is due to the fact that in the practice of diagnosing many objects the measurement databases were not used and the results of previous tests were not generalized. To solve this problem, we propose in the preliminary stage of neural networks learning to use mathematical models of controlled objects. On the basis of such models, simplified images set is formed that adequately represent the object.

Denote the sets of model images: \(R_{1}^{M} (x,y)\)—visible images, \(R_{2}^{M} (x,y)\)—ultraviolet images, \(R_{3}^{M} (x,y)\)—infrared images (thermograms), x, y—coordinates of the observed surface of the monitored object. Joint analysis of images by a group of experts makes it possible to form classes of the object’s technical states, corresponding to certain defects, failures or emergency conditions. For controlled object we denote the set \(D_{k} ,\,k = \overline{0,K}\) of possible states. This set includes K inoperable states, corresponding to classified failures, and one D0 state for normal operation of the equipment.

In addition, each class will correspond to a subset of model images:

$$V_{k}^{M} (x,y) = \{ \{ R_{1j}^{M} (x,y)\} ,\{ R_{2j}^{M} (x,y)\} ,\{ R_{3j}^{M} (x,y)\} \} ,j \in J_{k} ,$$

where Jk—an index set of model images corresponding to the inoperable state k.

It should be noted that in the general case the classification problem in this formulation is an incorrect inverse problem. In order to achieve one-to-one correspondence between model types of images and the status of the controlled object it is proposed to analyze the surface video images simultaneously with the set S (t) of additionally measured parameters. As a rule, it is possible to measure some subset of object parameters using built-in measurement channels.

The class Dk is determined by the set of images in different spectra and additionally calculated object parameters:

$$D_{k} :\{ (R_{1j}^{k} ,R_{2j}^{k} ,R_{3j}^{k} ,S_{i}^{k} ) \in\Omega \wedge (E_{k} = 1)\} ,j \in J_{k} ,i \in I_{k} ,$$

where

Ik:

an index set of parameters corresponding to the inoperable state k,

\(\Omega\):

domain of parameters in case of failure,

Ek:

expert opinion on the compliance of specified images and additional parameters to the k-th failure type

It is required to find the neural network operator Nk, which establishes the relationship between the vector Dk and measured vector DM.

Then the cooperative processing of complex model images and the S vector in the neural network regularizes the inverse classification problem based on information on additional parameters.

Thus, the set of classes of complex model images is formed in the database of the diagnostics system.

3 Intelligent Diagnostic Method Using the Artificial Neural Networks

The proposed intelligent method of improving the classification accuracy in the diagnosis is implemented through three basic procedures:

  1. 1.

    The complex model images are constructed, supplemented by a set of the object measured parameters to obtain an exact concordance with faults, as well as the formation a diagnostics knowledge base.

  2. 2.

    Using the neural network analyzer in the diagnostic system in the form of a two-branch neural network (2BNN) consisting of a deep convolutional network for image processing and a fully connected neural network for processing additional object parameters.

  3. 3.

    Training 2BNN with a complex model image dataset and further performing the classification of the technical states in the diagnostic process for decision making about maintenance service.

More detailed actions to be taken in the implementation of the proposed method are disclosed below:

  1. 1.

    For a certain class of controlled objects on the basis of mathematical models, the construction of two-dimensional computational images corresponding to various technical states of the object is performed. Together with the selected limited set of measured values of the object parameters, they form a set of complex model images.

  2. 2.

    On the resulting set, subsets of variable design images and measured parameters are built, covering possible deviations of the monitored object’s states from the nominal values and characterizing faults.

  3. 3.

    The diagnostic knowledge base is formed containing complex model image set.

  4. 4.

    A neural network analyzer is introduced into the diagnostic system structure. It is a two-branch multilayer neural network for processing images and a vector of additionally measured parameter values.

  5. 5.

    Network 2BNN is being trained on a complex model image dataset from the knowledge base. To verify the learning quality and the efficiency of the chosen structure of the diagnostic system, a multidimensional analysis of the object’s state classification is performed.

  6. 6.

    The diagnostic system performs the procedure of measuring real images of the surface and real additional parameters of the object under test.

  7. 7.

    The measured real multispectral image of the object is fed to the convolutional network input (the main branch in the 2BNN), and the additionally measured parameters are fed to the inputs of a fully connected network (auxiliary branch in the 2BNN).

  8. 8.

    As a result, the neural network analyzer performs the classification of object technical states and the type of failure is determined. Then the DSS makes a conclusion about the possibility of its further operation.

  9. 9.

    According to the results of images measurements, the complex model images correction can be performed, or real images are added to the dataset for retraining.

The intelligent diagnostic system structure using a neural network analyzer is shown in Fig. 1.

Fig. 1
figure 1

Intelligent diagnostic system using images analysis

Full size image

The measuring system MS registers a three-part video stream: images in the visible, ultraviolet and infrared ranges. In addition, the MS registers additional object parameters S (t). The neural network analyzer processes the time series of additionally measured parameters. Thus, the result of the object technical state classification appears at the analyzer output. The decision support system DSS generates object modes control signals and, under certain conditions, initiates the correction of mathematical models to retrain the neural network.

4 Neural Network Analyzer

The diagnostic system uses a neural network analyzer, which is a two-branch neural network 2BNN consisting of a convolutional branch and an auxiliary branch made as a fully connected network (Fig. 2).

Fig. 2
figure 2

General scheme of neural network analyzer 2BNN

Full size image

The main branch is the deep convolutional neural network ANN 1 which built according to some architectural principles proposed by LeCun [11] also combines some feature introduced by Krizhevsky in [12]. The neural network analyzer implements the sparsity of synaptic connections in the same way as proposed in [10]. In this way, a selection of different features map subsets is performed at the entrance to the second convolutional layer. As for the convolutional layers neurons, each of them is fully connected to the perceptual field input of the convolutional layer. The neural network uses batch normalization which was first introduced Joffe and Szegedy in [13].

The network ANN 1 consists of several convolutional layers combined into a feed-forward neural network. Such networks are well treated multidimensional signals. The input of the ANN 1 receives three images, which are analyzed simultaneously, as a complex multi-layered image. As proposed in the developed method, additional parameters S (t) are analyzed that are fed to the auxiliary branch ANN 2. The ANN 2 is formed as a fully connected perceptron. The output vectors Ψ1 and Ψ2 are the input signals for the last layer of the neural network analyzer. On this layer, this vectors merge, normalize and form the output vector Ψ3 = (D0, D1, …, DK), whose components are probabilities of the classified technical states.

The detailed structure of the neural network analyzer 2BNN is shown in Fig. 3. The complex image supplied to the first convolutional layer is an array of 225 × 225 × 5, where 5 is the depth of the entire array. This depth is determined from the following considerations. Three images are received at the input, while the visible image is a color RGB image with a depth of 3. The infrared image has a depth of 1, since the thermal imager transmits an array of measured surface temperature points in the format “Radiometric JPEG” with 14-bit encoding. The UV image is a monochrome image in the CCIR standard; the image depth is also equal to 1.

Fig. 3
figure 3

The detailed structure of the 2BNN

Full size image

The receptive field size for the first convolutional layer is 11 × 11. The first layer neurons extract certain “features” from the complex image. The number of features maps after the first layer is taken equal to 9. The output signal of the first convolutional layer is transmitted to the layer that performs the selection of the maximum value from the receptive field (Max pooling layer). The receptive field size of this layer is 4 × 4 pixels and the stride is 3 pixels. The output signal of this layer is transmitted to the input of another convolutional layer, the receptive field size of which is 5 × 5 pixels and the stride is 1 pixel. The number of feature map generated by this layer is 19. The second Max pooling layer with receptive field 4 × 4 prepares data for the third convolutional layer with receptive field 6 × 6. Since the input signal size of the third layer coincides in size its receptive field, the third layer is essentially fully connected and the length of its output vector is 120.

The ANN 2 branch processes data on the additionally measured parameters S (t) in parallel. To do this, the fully connected layer is used. The combination of the output signals of both branches occurs on the Y-junction layer. The output normalization is performed on the interval [0, 1] in the neurons final layer. This neural network analyzer classifies five states: operable D0 and four inoperable D1, D2, D3, D4.

5 The Training of Neural Network Analyzer

The training dataset is prepared as follows:

  • a complex model image set is being formed, possibly supplemented with real images obtained earlier;

  • experts classify the resulting of image set in accordance with recognizable technical states;

  • the process of neural network learning is carried out using the method of error back propagation.

The quality of the learning is crucial for efficient operation of the neural network analyzer and all diagnostic system in general [14, 15]. The use of model images in ANN learning instead of real images leads to a method error δM in determining the object technical states. The main factors affecting its value are:

  • the computational method error for solving the mathematical model equations, as well as the discretization and rounding errors,

  • variability of the object design parameters,

  • influence of the neural network architecture,

  • influence of the data processed by the neural network,

  • the method and neural network training parameters.

The relative method error δM is defined as:

$$\delta_{M} = \sqrt {\delta_{Str}^{2} + \delta_{D}^{2} + \delta_{Tr}^{2} } ,$$

where

δStr:

method error determined by the ANN structure,

δD:

the error determined by the noise in the input data of the neural network analyzer,

δTr:

the error determined by the neural network learning method.

Let us estimate the error of data noise. The type of data and the algorithm used to generalize them are important for machine learning. By noise we shall mean the presence of any region in the n-dimensional sign space of the dataset, where there are complex images of several classes simultaneously, inseparable by the hyperplane. The location of the dataset element in this space is determined by the attribute values to which the additionally recorded parameters correspond, as well as the features extracted from the images by the convolutional network. The feature space dimension is determined by the output vector length and is equal to 122.

Analyzing the data in the dataset, we can determine the proportion of the images that are in noisy areas. This makes it possible to estimate the accuracy threshold of an ANN trained in such a dataset, since images related to noise are likely to introduce an error into the neural network operation. To estimate the dataset and separation of individual noisy areas the Dimensionality Reduction methods have been used [16, 17]. Three dimensionality reduction techniques were used: Principal Component Analysis (PCA), Multidimensional Scaling (MDS), t-Distributed Stochastic Neighbor Embedding (t-SNE) [18]. Figure 4 shows the results of the dataset dimension reducing in the diagrams for the five technical states classes. Charts are built using the tools Skikit-learn [19].

Fig. 4
figure 4

Diagrams for five classes: a PCA chart; b MDS chart; c t-SNE chart

Full size image

6 Implementation and Results

The developed intellectualization method and neural network analyzer were used in a number of the technical diagnostic systems.

  1. 1.

    The photosensitive matrix chip diagnosis. Monitoring was carried out by chip’s surface thermograms with simultaneous measurement of a number of electrical signals. The use of a neural network analyzer for the classification of failures according to thermograms made it possible to increase accuracy up to 97.5% and reduce data processing time by 20% due to the technical states analysis in real time.

  2. 2.

    Catenary of contact wire monitoring. The control is performed in the movement of the computerized track-test car. The diagnostic system used the thermal imager Micro Epsilon TIM600 and ultraviolet camera CoroCAM 6D. The thermal imager registers the thermal field of the high voltage equipment and the ultraviolet camera registers corona and arc discharges on the catenary equipment. The classification accuracy was obtained at 98% with a reduction in data processing time up to 30%.

  3. 3.

    Rail track monitoring. In the computerized track-test car there is a system for obtaining railroad bed video images using four video cameras. The neural network analyzer was used when rail fastenings have been monitored, an example of which is shown in Fig. 5.

    Fig. 5
    figure 5

    Rail fastening images: a railway bed without fastening, b rail fastening is operable

    Full size image

Dataset was composed the rail fastenings images in good condition and the missing rail fastenings. It was analyzed by dimension reduction to identify distinguishing features. Then the input of the learned neural network received images which gradually closed black rectangle considered most characteristic for fastening elements. Similar experiments are often referred to as the “occlusion test” [20]. According to the neural network processing results can be seen, which elements of the image are making the greatest contribution in favor of a classification outcome. An example of model images for the described experiment is shown in Fig. 6.

Fig. 6
figure 6

Images of rails with closed elements on the images: a the original image

Full size image

Figure 7 shows over each image the vector of values obtained at the neural network analyzer outputs. The first element of the vector is the probability of the rails fastening presence on the image, the second element is the probability of the rails fastening absence.

Fig. 7
figure 7

Classification results of modified images

Full size image

The diagrams presented in Fig. 8 show that the classes considered in this task are separable in the dataset space.

Fig. 8
figure 8

Dimensionality reduction diagrams: a PCA; b t-SNE

Full size image

This justifies the possibility of the neural network analyzer learning to solve the classification problem. The relative number of images that fall into common areas for classes does not exceed 1%. The neural network analyzer was trained on a dataset of 50,000 images, 25,000 in each of two classes: serviceable rail fastenings and missing rail fastenings. The convoluted neural network with a learning rate of 0.00005 and a number of epochs equal to 100 achieved a classification accuracy of 90%.

7 Conclusion

Intelligent technologies allow improving the accuracy of detecting faulty states and reducing the processing time of control data. The obvious advantage is the possibility to obtain a new quality of diagnosis through the rapid analysis the large amount of information contained in the images of the different spectral ranges. The development of modern neurochips [21] opens up new possibilities for building miniature parallel computing structures for the neural networks implementation. In this case, they can be installed on air drone to monitor the extended objects, such as railways, oil and gas pipelines and high-voltage power transmission lines.