Keywords

1 Problem Statement

The number of information channels in automated process control system (APCS) increases due to the raise of the complexity in the technological process under control, which requires the application of a new paradigm when creating complex control systems, such as a cyber-physical system (CPS) [1]. This system is characterized by the use of multidisciplinary approaches in its operation as well as Big Data methods caused by the increase in APCS complexity [2,3,4].

Therefore, a distinctive feature for a number of manufactures is their long service life which leads to the use of outdated technological solutions. Their change is impossible without a general modernization of production. Modernization in its turn involves significant financial expenditure because of downtime as well.

An alternative direction of modernization is reengineering of the information support for technological process by improving the systems for collecting and processing technological information, their duration and the volume distribution in the entire physical process, which is typical for CPS, the use of modern state diagnostics and control algorithms. This procedure has less financial and time expenditures compared with equipment modernization. Thus, the direction to improve control and measuring infrastructure of APCS, based on modern achievements of information technologies, presents and actual research problem which solution can bring tangible advantages for enterprises in a short period.

2 Background and Methods

The proposed methods are based on machine learning [5]. Deep learning using convolutional neural networks (CNN) is among its methods which fined wide application in solving real problems. High results, shown by CNN when recognizing images, lead to a great spectrum of their applications in the subject fields where there is an opportunity to reformulate the initial problem for the task of images recognition. These fields include medicine [6, 7], social engineering [8], text processing [9], gesture recognition [10], automatic identification of vehicles in coating production line based on computer vision [11], vehicles identification in a tunnel surveillance control system [12], cracks recognition in concrete [13] etc. The above mentioned list shows, that CNN can be a universal tool for problem solution of data deep analysis too.

The proposed approach for developing a model of complex technological processes is based on the implementation of CNN ensemble connected to the analysis of data in different points of technological process with subsequent processing of the obtained results of the neural classification in the analytical block. In addition to spatial partitioning of industrial zones the processing is discretized according time, it helps to monitor the processes dynamics.

There is a great diversity of buildings and CNN ensembles work interpretation [14], they are widely used in medical applications, in systems of biometric data control, people activity [15,16,17,18], but there is practically no works on generalized modeling and diagnostics of technological processes.

Control technical areas, where CNN is supposed to be used, can contain data of various format data sources. In addition to places with evident presence of video streams, for example[19, in Russian], other forms of signals can be also processed, including in-plant noise control, using various methods of sound waves conversion [20].

The enlarged structure of the proposed model of a complex technological process based on the application of deep neural networks is shown in Fig. 1.

Fig. 1
figure 1

The structure for the information processing model in CPS

Full size image

The general algorithmic structure of the proposed model for technological information processing in CPS is as follows. It is supposed, that control and measuring information from technological zones (zone 1, …, d) is presented by multichannel sets of data for each zone. Through the commuter this information is fed to the output of local CNN ensemble which, for each zone, is formed taking into account the form of data representation in information channels and requirements of their further analysis which is carried out in the group of zone analyzers (analyzer 1, …, d). From the zone analyzers output information goes to neural network output (CNN_out) which carries out the estimation and forecast for the state of the entire technological process.

The algorithm for processing information is as follows. Denote the interval of discretization for information channel kzd in technological zone d by Δtzdi = tzdi–tzdi-1. At tzdi and tzdi–1 moments channel commutator kzd is closed and the image of the technological parameter under control is supplied to CNN input. The technological parameter under control is denoted by Pzd. The interval value of discretization Δtzdi is calculated by a particular analyzer with regard to Kotelnikov theorem (it is also known as Nyquist–Shannon theorem) [21].

CNN of information channel kzd recognizes an image on the introduced time step. This procedure consists in forming output vector \(V_{i}^{{k_{zd} }}\) with dimension corresponding the number of classes n_CL_kzd for each channel:

$$V_{i}^{{k_{zd} }} = \left( {V_{i,1}^{{k_{zd} }} ,V_{i,2}^{{k_{zd} }} , \ldots ,V_{{i,n\_CL\_kzd}}^{{k_{zd} }} } \right)^{\text{T}}$$
(1)

It is supposed, that the architecture of the output CNN layer is formed in the way the elements of vector (1) contain values from 0 to 1. It characterizes the degree of neural network confidence in parameter Pzd controlled according to the image belonging to a particular class at moment tzdi.

CNN application for information channel kzd is carried out through the whole technological process, but all this time can be divided into fragments with Ti. duration given as initial data for the algorithm operation. Fragments with Ti. duration are identified by the requirements for the periodicity of information flow into APCS. Thus, the matrix of results classification is processed in particular counting analyzers iT and becomes available to the moment Ti:

$$MV_{{i_{T} }}^{{k_{zd} }} = \left( {\begin{array}{*{20}c} {V_{1,1}^{{k_{zd} }} ,\quad V_{1,2}^{{k_{zd} }} ,\quad \ldots ,\quad V_{1,n\_CL\_kzd}^{{k_{zd} }} } \\ \ldots \\ {V_{{i_{T} ,1}}^{{k_{zd} }} ,\quad V_{{i_{T} ,2}}^{{k_{zd} }} ,\quad \ldots ,\quad V_{{i_{T} ,n\_CL\_kzd}}^{{k_{zd} }} } \\ \end{array} } \right)^{\text{T}}$$
(2)

The obtained set (2) contains the data about CNN confidence dynamics in classification results and can be used when making control decisions in APCS. For this purpose the relation matrix of elements increments (2) to the given discretization Δtzdi interval is:

$$DV_{{i_{T} }}^{{k_{zd} }} = \left( {\begin{array}{*{20}c} {\tfrac{{V_{2,1}^{{k_{zd} }} - V_{1,1}^{{k_{zd} }} }}{{\Delta t_{i}^{zd} }}} & \ldots & {\tfrac{{V_{{i_{T} ,1}}^{{k_{zd} }} - V_{Ti - 1,1}^{{k_{zd} }} }}{{\Delta t_{i}^{zd} }}} \\ {\tfrac{{V_{2,2}^{{k_{zd} }} - V_{1,2}^{{k_{zd} }} }}{{\Delta t_{i}^{zd} }}} & \ldots & {\tfrac{{V_{{i_{T} ,2}}^{{k_{zd} }} - V_{Ti - 1,2}^{{k_{zd} }} }}{{\Delta t_{i}^{zd} }}} \\ \ldots & \ldots & \ldots \\ {\tfrac{{V_{2,n\_CL\_kzd}^{{k_{zd} }} - V_{{1,_{n\_CL\_kzd} }}^{{k_{zd} }} }}{{\Delta t_{i}^{zd} }}} & \ldots & {\tfrac{{V_{{i_{T} ,n\_CL\_kzd}}^{{k_{zd} }} - V_{Ti - 1,n\_CL\_kzd}^{{k_{zd} }} }}{{\Delta t_{i}^{zd} }}} \\ \end{array} } \right)$$
(3)

For the purpose to simplify the notation denote matrix (3) elements as dvi,j. Their sense load can be interpreted as analogue of derivatives for continuous functions because they reflect the change of neural confidence in image belonging to a particular class.

The rate of change, in this case it is element dvi,j value, can be used for the forecast of the technological process development. For this purpose matrixes of type (3) are calculated for all measuring channels and all technological zones.

The formation of an output tensor for CNN_out, based on the fragment in iT timing, is performed by the combination of matrix (3) for all channels and zones

$$TR_{{i_{T} }} = \left( {DV_{{i_{T} }}^{{k_{z1} }} \quad DV_{{i_{T} }}^{{k_{zd} }} \quad \ldots \quad DV_{{i_{T} }}^{{k_{z2} }} } \right)$$
(4)

It should be noted, that the number of classes for various information channels and various technological zones can be different (see Fig. 1), therefore, matrix (3) dimension in a general case is different. The number of lines in (3) for all zones is equal as it is determined by the number of counting for discrete time i = 1, 2, …, iT. The number of columns for matrix TRiт is determined by the number of information channels in each zone, the number of classes identified for each channel and is equal to (kz1 · n_CL_kz1) + (kz2 · n_CL_kz2) + ··· + (kzd · n_CL_kzd).

When input tensor TRiт is formed, the multi-stage preprocessing of data ends to ensure the work of output neural network (preprocessing for local CNN was not described as it is not of special interest in this work). Then, it remains to determine the hypotheses space for CNN_out as the application of deep learning deletes the need for construction features to replace complex, contradictory and heavy conveyors with simple learning models which area usually built with the use of several tensor operations [4].

Tensor TRiт is formed on the basis of data on the transformation dynamics of class membership for information channels parameters reflected in matrix (3). Thus, when forming the space of hypotheses good results to forecast the state of a technological process, values of any parameters for a finished product can be expected with the help CNN_out.

3 Application and Results

To test the proposed model for processing information in CPS of a complex technological process control, a simulation experiment to recognize aluminum alloy ingots images was carried out to determine their aggregate state. The aggregate state is estimated according to the image of a surface observing through a viewing window fitted on the furnace door [19, in Russian]. The received image is shown in Fig. 2 in its left part. For the application under study the general scheme is presented in Fig. 1.

Fig. 2
figure 2

The structure for processing of melting zone images

Full size image

The image for a working surface of aluminum ingots forming by a video camera, has a resolution of 640 × 480 (Fig. 3a). If this resolution is left and used as an example at input of CNN, then the process of learning requires more time considering that the numbers of such examples will be several thousand. Therefore, the dimension of an initial image is programmatically reduced to 90 × 90. In addition, it is taken into account, that the melting zone is lighted by an electric arc, so the image brightness can fluctuate when the supply voltage changes in a circuit. To reduce the influence of this factor brightness was normalized (Fig. 3b).

Fig. 3
figure 3

Melting surface images

Full size image

The existing methods of visual identification for aggregate state of a substance (method of triangulation, burst mode, area method) analyze the changes of the surface image area for a melting metal in a three-dimensional space, but not the area of the surface for remelting ingots.

The proposed method for control of an aggregate state based on the deep neural networks also analyzes the three-dimensional surface of ingots. It is ensured by the fact, that the analysis of the melting process in images matrix is characterized by the changes of rises heights in histogram (Fig. 3c) which is taken into account by the neural network when forming the answer. One example from the learning set is a tensor of the second order, the totality of such examples forms the tensor of the third order which is fed to the output of CNN1,1. Temporal discretization interval of a video sequence coming from the video camera focused on the bars surfaces is taken equal to one second. Free Video to JPG Converter was used to take shots from the video. To enlarge the number of learning examples the augmentation procedure was applied. During this procedure shifts, zooming, rotations, mirror reflection were implemented for the initial image.

The time for melting of an aluminum bars lot is approximately 300 s, but it was divided into intervals corresponding different classes identified by CNN1,1 (see Table 1).

Table 1 Melting time distribution into classes
Full size table

The software model for the technological process (aluminum bars melting) under study implementing the recognition of aggregate state transition was performed in Python 3.6 language. IDE Spyder from Anaconda (version for Linux) was chosen as the development environment. Convolutional neural networks were developed using specialized Keras library, which is a superstructure above the tensor computation framework TensorFlow [22, 23].

CNN1,1 containing seven alternating convolutional layers and subsampling and one output fully connected layer with four outputs (according to the numbers of recognizable classes), was implemented in the software. The learning sample has a size of 2000 examples (400 examples are from the testing sample).

The analysis of the results for the software work, presented in the upper graph of Fig. 4, shows that within one run at CNN1,1 outputs the dynamics of classification during melting process is visible.

Fig. 4
figure 4

Image recognition results

Full size image

The final results of classification can be obtained if the choice of CNN1,1 output according to the majority principle is made at every second counting. At multiple run the number of classes, as it is shown in the graph at the bottom of Fig. 4, are correctly recognized by CNN1,1 neural network, unstable recognition is occurred only in the zone of transition from one class to the other one. This circumstance can be explained by the complexity of detection for difference in the surfaces on the junction of classes, in addition the relative length of unstable time intervals is not long.

To forecast the dynamics of the process development matrix (3) is calculated, the number of which forms the initial data for CNN_out working. The results of the obtaining images for two melting’s are presented in Fig. 5. Figure 5 shows that for different processes of aluminum bars melting the image is different, this is the evidence of possibility to use the methods of texture recognition in CNN_out block to forecast the process development. In this study the experiment to calculate the dynamics was not carried out, but it is planned for further work.

Fig. 5
figure 5

Matrix dynamics process visualization

Full size image

CNN learning was performed on GeForce GTX 1060 video card installed on Asus FX502VM notebook with CPU IntelCore i7-7700HQ. The process control was exercised with the help of TensorBoard which allows to visualize the current accuracy and learning error.

4 Conclusion

As a result of the conducted study for the possibility of using deep neural networks as a part of a cyber-physical system of complex technological objects control the following results were obtained:

  1. 1.

    The algorithmic structure for the model of processing information coming from different technological zones of the integrated multi-stage technological process based on the convolutional neural networks ensemble implementation is developed.

  2. 2.

    The results of model experiment for recognition of aluminum aggregate state based on the proposed algorithm are given. The results show that the algorithm, based on the convolutional neural networks, solves the task for classification of metal aggregate state well, excepting some zones in the transition regions from one class to the other one.

  3. 3.

    The method to follow up the dynamics in technological process based on the formation of matrix values changes at the output of the convolutional network, performing the classification for technological process conditions and its further visualization for output convolution neural network recognition is proposed.