INTRODUCTION

The factors limiting the effectiveness of the functioning of communication and control systems developed in the Brain-Computer Interface (BCI) technology are the accuracy of classification of patterns of brain activity, general stability and reliability of functioning [1, 2]. It is obvious that progress in solving these problems will determine the dynamics of the advancement of neural interface technology not only among research laboratories, but also in the mass consumer market [3, 4].

The most commonly used methods for classifying EEG patterns used to form control commands in such systems, as a rule, include linear classifiers [5] and models of neural networks that are widely used in neural control problems [6, 7]. A linear classifier is a relatively simple method, fairly easy to implement in on-line systems, however, since EEG patterns can be linearly inseparable, this type of classifier often shows a low accuracy in detecting control commands [5]. Neural networks are successfully used to solve a wide range of tasks, in particular in modeling, filtering and classifying biological signals [8]. Comparative analysis showed the superiority of nonlinear neural network algorithms over linear, primarily in terms of operating efficiency and adaptive tuning [9]. Improving the efficiency of such systems is associated with the development of hybrid methods that integrate the capabilities and benefits of various classifiers of brain activity patterns in the neural network basis [10].

The purpose of present study is to develop a new neural network approach using the example of detecting and classifying EEG patterns accompanying the voluntary execution of motor imagery. The features of the developed neural network method for classifying EEG patterns are:

(1) structure, including a committee of specialized RBF-networks;

(2) using the gradient method of tuning the components of the width vector of RBF-functions with regularization;

(3) use of the median method of determining the threshold for building dividing boundaries between classes;

(4) identification of the EEG pattern of mental movement by analyzing the duration and content of the response sequence of each RBF network of the committee;

(5) formation of new metrics of motor imagery EEG patterns based on Locally Approximated Spectral Power (LASP) indices and Locally Approximated Cross-Correlation (LACC) of the activity of different areas of the cerebral cortex.

EXPERIMENTAL DATA

The study involved 15 subjects (10 men and 5 women) aged 23 to 42 years (mean age 32.5 ± 9.5 years). In accordance with the ethical standards approved by the ethics committee of the Southern Federal University, the surveyed signed a protocol on voluntary consent to participate in the study. In the course of the electrophysiological studies, the subjects were in a comfortable chair in a light- and soundproofed chamber.

The main series was preceded by training, during which its participants adapted to the conditions of the survey, trained to perform specified movements (including motor imagery) at a convenient pace, determined their individual speed characteristics, namely, the average time for performing movements. The training was conducted without EEG registration. In accordance with the instructions, the subject was asked to perform movements with his hands and feet in a given sequence, fixing his eyes on the cross in the center of the screen (Fig. 1). For the implementation of each movement was given 2 s, the interval between movements was 10 s.

Fig. 1.
figure 1

The timing diagram of the standard research scenario for the implementation of real movements and motor imagery. Notes. Series 1 (D)—real movements; Series 2 (D + MD)—real movements and their mental equivalents; Series 3 (MD + D)—motor imagery and real movements. The time of registration of the background EEG and the presentation of the instruction is 60 s. The duration of each series is 540 s.

Full size image

The movements of the hands consisted in the compression of the hands into a fist. The movements of the legs were simultaneous bending and unbending of both feet in the vertical plane. The training procedure was divided into 3 series, each of which lasted 180 s. In the first series, the subjects performed only real movements with their hands and feet. In the second series, after each real movement, the subjects had to reproduce their mental equivalents, in the third series – motor imagery preceded the actual implementation of the corresponding movement. It was not required to report the beginning or end of the motor imagery. The experiment itself contained similar 3 series.

Considering the time for acquaintance with the instruction and registration of the background EEG, the duration of each experiment was 35 minutes, during which each movement was reproduced by the subject from 30 to 80 times. During the main experiment, the EEG was recorded continuously both when the subject underwent real movements and motor imagery, and in a state of operative rest between them, as well as at rest with eyes open and closed. The duration of each functional test, which was recorded both before and after the survey, was 60 s. The state of operational rest corresponded to the interspin intervals (pauses between movements). Registration of EEG monopolarly from 13 standard leads on the international system “10–20”, namely: F7, F8, F3, Fz, F4, C3, Cz, C4, P3, Pz, P4, O1, O2. Reference were electrodes located on the earlobes (referent combined).

The electromyogram of both hands was recorded in the superficial muscles, flexing the forearm in the elbow joint (m. Brachioradialis), and superficial flexor fingers (m. Flexor digitorum superficialis), and the legs in the anterior tibial muscle (m. Tibialis anterior). Brain potentials associated with motor imagery (motor imagery-related potentials) were isolated and accumulated relative to the initiation marks of real movements recorded on electromyogram channels (EMG). Labels were set after filtering electromyogram bandpass filter (0.1–4 Hz). The threshold level of the amplitude of the EMG, the achievement of which tags were set, was set to 10 µV. The threshold was determined at the leading (rising) front of the smoothed EMG signals. The epoch of the analysis of motor imagery potentials was 2 s after the completion of myographic activity in series 2. In the series 3, the epoch of analysis also amounted to 2 s, but preceded the label of the initiation of an arbitrary real movement.

For the detection and subsequent rejection of artifacts associated with eye movement and blinking, horizontal and vertical electrooculogram were recorded. Registration of all electrophysiological parameters was carried out using the Encephalan multichannel electroencephalograph (elite version) manufactured by Medicom (Taganrog, Russia). The sampling rate of analog signals when entering into a computer was 250 Hz for each of the channels. In real-time, the electrograms were digitally filtered with a bandwidth of 1–70 Hz, and a notch filter 50 Hz was used to remove network crosstalk. In digital form, files were recorded in the international EDF format and subjected to automated software processing. After preliminary preparation, the data were presented in the form of a three-dimensional array containing information about the analyzed classes of movements (real or motor imagery), EEG leads and the current amplitude of the native signal, as well as a numerical array including program integral event labels corresponding to the type of movement.

For the analysis, samples of EEG epochs were compiled corresponding to motor imagery performed by the subjects in two test series, as well as samples of electrograms of background brain activity recorded in the state of operative rest. In total, therefore, for each surveyed, 823 EEG epochs were selected in the first test series and 731—in the second. In total, 3108 non-artifact EEG epochs were analyzed.

In order to form reference vectors of EEG signs of mental movements, electrograms registered in the second and third series were used for computational analysis. They were subjected to a fast Fourier transform (FFT) and for which mutual correlations were calculated for all pairs of leads. The vector of characteristic signs of EEG mental movements, obtained using the FFT method, consisted of 143 components. The spectral power values were calculated in 10 frequency ranges (Hz): delta (1–4), theta (4–7), alpha-1 (7–10), alpha-2 (10–13), alpha-3/mu (13–16), beta-1 (16–19), beta-2 (19–22), beta-3 (22–25), gamma-1 (30–45), gamma-2 (55–70). The feature vectors obtained using the cross-correlation method consisted of each of the 78 components, which were cross-correlation coefficients. The spectral power averaged in each frequency range and the pairwise cross-correlation of EEG leads were calculated for each EEG era of the corresponding class of motor imagery.

CLASSIFICATION METHODS

The neural network classification method developed was based on the specialized committee of RBF functions with the definition of the class discrimination threshold by the median method and the analysis of extended response patterns of RBF networks. The developed model of the neural network EEG pattern classifier was a committee of three neural networks. Each of them specialized in detecting only one class of motor imagery. The inputs for each neural network were signs extracted from the EEG, namely, the spectral power and cross-correlation coefficients. The need for a network for each type of mental movement was due to the possibility of overlapping clusters represented by multidimensional vectors in the feature space (Fig. 2), which would result in recognition errors when using a single neural network. To account for the temporal structure of the EEG patterns, the interpreter of the neural network response sequence developed earlier [11] was used. In addition, the interpreter organized the joint work of neural networks, which was aimed at minimizing recognition errors.

Fig. 2.
figure 2

The projection of the feature space of two motor imagery classes on the plane. Sammon’s mapping.

Full size image

The basis of the neural network classifier model is a network of radial-basis functions. Each radial-basic network consisted of three layers of neurons namely input, hidden and output ones. The first layer transmitted the input signal to the hidden layer. The number of input neurons in each of the three neural networks was 13 and was equal to the number of EEG leads. The hidden layer implemented a non-linear transformation of the input signal through the activation functions of its neurons, which are Gauss functions:

$${{y}_{j}} = \exp \left( { - \mathop \sum \limits_i \frac{{{{{({{I}_{i}} - {{c}_{{ji}}})}}^{2}}}}{{{{d}_{{ji}}}}}} \right),$$
((1))

where yj is the output of the j-th hidden neuron, Ii are the components of the input vector of the network, сji are the components of the coordinate vector of the center of the j-th neuron of the network, dji are the width components of the j-th neuron of the network. All neural networks of the committee contained only one neuron in the hidden layer, since the input data of each class had a Gaussian distribution with its own characteristics of the center and width. The neurons of the output layer calculated the scalar product of the vector of the outputs of the neurons of the hidden layer y with the vector of own weight coefficients w, forming the output of the network O:

$$O = \mathop \sum \limits_j \left( {{{w}_{j}}{{y}_{j}}} \right),$$
((2))

where wj is the weight coefficient of the j-th neuron of the hidden layer with the neuron of the output layer, yj is the output of the j-th neuron of the hidden layer.

The decision of the classifier on the presence in the input signal of a pattern associated with the motor imagery performance of a voluntary movement was implemented by the interpreter in 2 stages.

At the first stage, the responses of the neurons of the output layers of the neural networks of the committee to each input vector were binarized:

$${{o}_{k}} = \left\{ \begin{gathered} 1,\,\,\,\,\,{\text{if}}\,\,{{O}_{k}} \geqslant {{\tau }_{{1k}}} \hfill \\ 0,\,\,\,\,{\text{if}}\,\,{{O}_{k}} < {{\tau }_{{1k}}}, \hfill \\ \end{gathered} \right.$$
((3))

where \({{O}_{k}}\) is the output value of the k-th neural network of the committee, k = 1, …, 3.

For each neural network, as the threshold level τ1k, the value corresponding to the median value of separation of the corresponding class from the “rest” class was selected. The single response of the neural network indicated that the input vector of the network belongs to the target class, zero – the class “rest”. Thus, each of the three neural networks carried out the classification of input vectors according to the “motor imagery/rest” scheme.

When interpreting the responses of neural networks at the second stage, the temporal structure of EEG patterns corresponding to motor imagery was taken into account. The discrete binarized responses of each of the networks were represented as a sequence of оk(t). The decision on the presence of a pattern corresponding to a specific motor imagery was taken by the committee when a subsequence appeared in one of the three sequences, which consisted of single answers and had a continuous duration of τ2. For each motor imagery class, an individual value of τ2. was determined. The search for the optimal value of τ2 for each class was carried out using the brute force method with 5-fold cross-validation.

The competitive gradient method was the basis for the training procedure of the neural network committee. The adjustable network parameters were the center (c) and the width (d) of the neurons of the hidden layer. The weights of the neurons of the output layer were fixed and equal to unity for all neural networks. The traditional solution to setting network parameters is to consistently apply a competitive algorithm to configure the centers (c) of the hidden layer neurons, and then the gradient algorithm to adjust the link weights w of the neurons of the output layer. The width (d) of the neurons of the hidden layer usually has the form of a scalar, is set in advance or is calculated during initialization and does not change in the subsequent iterative process of learning. This scheme was modified so that the competitive and gradient modes worked in parallel. Instead of the weighting coefficients of the output neurons, the width of the neurons of the hidden layer, which was represented by a vector, was adjusted by the gradient method, and the adjustment was made for each component of the width vector. modes worked in parallel. Instead of the weighting coefficients of the output neurons, the width of the neurons of the hidden layer, which was represented by a vector, was adjusted by the gradient method, and the adjustment was made for each component of the width vector.

The competitive-gradient method of training within the framework of setting the parameters of the neurons of the hidden layer implemented competitive and gradient algorithms within one iteration of forward-return-continuous computing in the neural network. At the beginning of the training of neural networks, the centers were initialized by randomly selected input vectors. The input vectors of three neural networks, each of which belonged to a separate motor imagery class, were assigned to the centers of the only hidden neurons. The width of the neurons of the hidden layer was initialized by small values.

Competitive tuning of the centers of the neurons of the hidden layer was performed at the stage of direct-flow calculations. The objective function was as follows:

$${{E}_{c}} = \frac{1}{2}\mathop \sum \limits_{i,j} {{({{I}_{i}} - {{c}_{{ji}}})}^{2}},$$
((4))

and minimized by the gradient descent method:

$$\Delta {{c}_{{ji}}} = - {{\eta }_{c}}\frac{{\partial {{E}_{c}}}}{{\partial {{c}_{{ji}}}}},$$
((5))

where \({{\eta }_{c}}\) is the coefficient of the learning rate of the center parameter.

In direct-flow computations, the output values of neurons of the hidden and output layers of the neural network were calculated in layers for the input vector I shown in accordance with formulas 1) and 2). Then the components of the vector of the center of hidden neurons were:

$${{c}_{{ji}}}\left( {n + 1} \right) = {{c}_{{ji}}}\left( n \right) + \Delta {{c}_{{ji}}}\left( n \right),$$
((6))
$$\Delta {{c}_{{ji}}}\left( n \right) = {{\eta }_{c}}\left( {{{I}_{i}} - {{c}_{{ji}}}\left( n \right)} \right),$$
((7))

where n is the number of the learning iteration.

The gradient tuning of the d-width parameter of the neurons of the hidden layer was performed in the course of reverse flow calculations in accordance with the back-propagation error method. Given the uniqueness of the output neuron in neural networks, the objective function was:

$${{E}_{d}} = \frac{1}{2}{{(O - t)}^{2}} + \gamma \frac{1}{2}\mathop \sum \limits_{i,j} d_{{ji}}^{2},$$
((8))

where t is the target output value of the neural network, γ is the regularization coefficient.

The second term is responsible for regularizing the widths, i.e., imposing a penalty for large widths. The value of the penalty was controlled by the value of the regularization coefficient γ, which was determined by the method of enumeration with 5-fold cross-validation for each of the neural networks of the committee. The objective function was minimized by the gradient descent method:

$$\Delta {{d}_{{ji}}} = - {{\eta }_{d}}\frac{{\partial {{E}_{d}}}}{{\partial {{d}_{{ji}}}}},$$
((9))

where \({{\eta }_{d}}\) is the coefficient of the learning rate of the width parameter.

Inverse-flow calculations performed the modification of the components of the vector of the width of the neurons of the hidden layer as follows:

$${{d}_{{ji}}}\left( {n + 1} \right) = {{d}_{{ji}}}\left( n \right) + \Delta {{d}_{{ji}}}\left( n \right),$$
((10))
$$\Delta {{d}_{{ji}}}\left( n \right) = {{y}_{j}}\left( {O - t} \right)\frac{{{{{({{I}_{i}} - {{c}_{{ji}}})}}^{2}}}}{{d_{{ji}}^{2}\left( n \right)}} + \gamma {{d}_{{ji}}}\left( n \right),$$
((11))

where n is the number of the learning iteration.

Each neural network has passed from 1000 to 10 000 iterative training cycles (n).

RESULTS AND DISCUSSION

Comparative analysis showed (Table 1) that the neural network classification method developed by us, based on the use of the specialized committee of LASP-RBF functions, allows us to increase the reliability and accuracy of the classification of the target EEG motor imagery patterns, compared to other classifiers, to 81–83%.

Table 1.   Test results of the conventional and developed method for the classification of EEG patterns represented by spectral power indices for three classes of motor imagery (mentally left arm (LMI), mentally right arm (RMI), mentally feet (FMI)). Model designations: RF—Random Forest, LDA—Linear Discriminant Analysis, LR—Linear Regression, LASP-RBF—Radial Basis Functions
Full size table

Computational analysis showed a certain increase in the classification accuracy of the corresponding patterns, evaluated on the basis of EEG cross-correlation coefficients, and the neural network classifier developed by us based on the LACC-RBF method also showed higher rates of pattern classification accuracy corresponding to hand movements compared to the RF and LR methods (Table 2). The increase in the reliability of the classification of target motor imagery EEG patterns was from 2 up to 7%. In addition to the combinatorial alignment of the lead vectors/frequency range of all 13 leads and in all frequency spectra, EEG leads were also chosen, the signs of which play a key role in the formation of EEG patterns of real movements and motor imagery.

Table 2.   Test results of the conventional and developed method for classifying EEG patterns, represented by cross-correlation coefficients (LACC-RBF), for three classes of motor imagery. Designations as in Table 1
Full size table

In accordance with this, for the formation of small-dimension vectors, we used leads of the frontal (F3, F4), central (C3, Cz, C4) and parietal (Pz) brain regions. The frequency ranges were selected based on known data – alpha, gamma-1 and gamma-2 [12–14].

In computer analysis it was shown (Table 3) that with a decrease in the dimension of the attribute space to 6 EEG leads and 3 frequency ranges, the classification accuracy of target motor imagery EEG patterns has increased markedly, and practically in all tested models. The specified accuracy increase for the RF method was up to 8%, LDA – 6%, LR – 5%. The LASP-RBF method developed by us demonstrated an increase in the accuracy of detection of target EEG patterns by up to 9%, making it possible to increase the classification accuracy to 83–90%.

Table 3.   Test results of the conventional and developed method of classification under conditions of decreasing dimensionality of the attribute space, represented by the EEG spectral power values (LASP-RBF). Designations as in Table 1
Full size table

The accuracy of the LACC-RBF classification of three motor imagery, whose characteristics were the cross-correlation coefficients of EEG derivations, was also evaluated in terms of a decrease in the feature space to 6 recording channels. Computational analysis showed that, in general, there were multidirectional changes in the accuracy of the work of various models of the classification of target EEG patterns (Table 4).

Table 4.   Test results of the conventional and developed method of classification under conditions of decreasing dimensionality of the attribute space, represented by the values of the cross-correlation coefficients of EEG leads (LACC-RBF). Designations as in Table 1
Full size table

There was a decrease in the classification accuracy of motor imagery patterns using the linear discriminant analysis method (Fisher classifier), the value was up to 6%. The method of radial basis functions showed higher classification accuracy compared with the methods of the Random Forest, the Fisher and Linear Regression classifier. The increase in the reliability of the classification of motor imagery patterns of small dimension for the right arm and legs was up to 3%, at the same time, it was comparable to the results obtained under the conditions of the analysis of multidimensional motor imagery patterns of the left hand.

Thus, the neural network classification method developed by us, based on the use of a specialized committee of LASP-RBF functions, is able to provide a sufficiently high accuracy (from 80 to 90%) of detection of arbitrarily generated by human control commands represented by EEG patterns, including, under conditions of minimizing the dimensionality of the characteristic space. At the same time, a higher accuracy was shown by the classification of EEG patterns based on the assessment of the attribute space, represented by the EEG spectral power values, than the cross-correlation. The results are consistent with the already available data, indicating that neural interfaces that exploit mental activity and the neural network approach to control, can provide a reliable control channel [15–17], but require more thorough user training and education [11, 18]. In this regard, there is reason to believe that the development of new patterns of recognition and classification of brain activity patterns [19–22], including on the basis of promising self-learning models of neural networks with reconfigurable (including, completion with increasing complexity of the problem being solved) architecture will significantly to increase the reliability of the channel of non-verbal and non-muscular communication within the frameworks of neural interface technology, which is the subject of the research we are currently conducting.

СONCLUSIONS

The main results of this study are as follows:

(i) a committee of three neural networks providing the classification of target EEG patterns accompanying the execution of motor imagery by the upper and lower limbs has been developed;

(ii) the method is based on Local Approximation of Spectral Power using Radial Basis Functions (LASP-RBF) and the original algorithm for interpreting the time sequence of neural network responses;

(iii) new metrics of motor imagery EEG patterns based on the spectral power indices and cross-correlation of the activity of different areas of the cerebral cortex has been created and tested;

(iv) a higher accuracy was shown by the classification of EEG patterns based on the assessment of the attribute space, represented by the EEG spectral power values, than the cross-correlation;

(v) a decrease in the dimension of the attribute space to 6 EEG leads (F3, F4, C3, Cz, C4, Pz) and 3 frequency ranges (alpha, gamma-1 and gamma-2) the classification accuracy of target motor imagery EEG patterns has increased markedly;

(vi) the classification accuracy using the developed approach is higher (up to 90%) than other classifiers (Random Forest, Linear Discriminant Analysis and Linear Regression).

Evidently development of additional algorisms such as self-learning models of neural networks with reconfigurable architecture can significantly increase the reliability of the channel of mental-based neural control in the frameworks of the brain-computer interface.

FUNDING

The work is supported by the Russian Ministry for Education and Science, project no. 2.955.2017/4.6 “Development of the hardware and software system for monitoring the attention level and psychoemotional state of pilots and dispatching personnel to improve flight safety”.

CONFLICT OF INTEREST

The authors declare that they have no conflict of interest.