Introduction

Neurorehabilitation constitutes the systematic retraining of the brain to enhance or restore impaired communication and motor functions, based on the premise that motor learning can facilitate motor recovery (Kitago and Krakauer 2013). The main objective of neurorehabilitation is to implement strategies to stimulate the brain to create and reorganize neural connections (Kleim and Jones 2008; Binks et al. 2023).

In the brain, the Primary Motor Cortex (M1), the Supplementary Motor Area (SMA), and the Pre-Motor Area (PMA) are activated during motor mental tasks. M1 is directly responsible for the coordination of voluntary movements, whereas PMA supports the movements generated by M1 of both hemispheres, allowing Motor Imagery (MI) or “Mental Practice” tasks (Padfield et al. 2022).

A stroke can affect several regions of the brain, depending on the location of the disrupted blood flow, with the symptoms being different according to the affected hemisphere. When affecting hand movements, the effects of a stroke vary between left-handed (about 10% of the population) and right-handed individuals (about 90% of the population) (de Kovel et al. 2019). Thus, if the affected person is right-handed, a stroke occurring in their brain’s right hemisphere would impact the non-dominant left side of their body. In this case, adapting to weakness on the non-dominant side is generally easier, as the dominant side can often compensate (de Kovel et al. 2019). Nonetheless, even affecting the non-dominant side, a stroke can still have a substantial impact on performing essential Activities of Daily Living (ADLs) due to the provoked hemiparesis.

Scientific evidence indicates that Brain-Computer Interfaces (BCIs) can be used successfully for neurorehabilitation in people affected by stroke (Tsiamalou et al. 2022). Specifically, BCIs act as an alternative non-muscular communication channel between the user’s brain and a computer to contribute to motor rehabilitation (Vavoulis et al. 2023; Tabernig et al. 2018). Several studies have shown that BCIs can be applied as part of a high-level control system in active orthoses, robotic exoskeletons, or robotic wheelchairs to enable training methods and interventions in neurorehabilitation. For this reason, the use of a BCI may also be an important factor for hand function recovery, which is one of the most challenging topics in the rehabilitation of post-stroke patients (Yue et al. 2017; Basteris et al. 2014).

When used for upper limb rehabilitation, BCI-based MI can enhance conventional therapy. Several BCI developments have used MI-based protocols in rehabilitation intervention (Tabernig et al. 2018; Frolov et al. 2017; Simon et al. 2021). Furthermore, the literature also reports that the combination of two mental practice techniques, such as Static Visual Stimuli (SVS) and MI can enhance the neurorehabilitation of post-stroke patients (Binks et al. 2023). In SVS, the subject observes a static image on a screen as a cue, which indicates the mental action he/she must execute. In fact, a recent study has shown that SVS combined with MI evokes neuronal reorganization that corresponds to the effects of physical practice, improving plasticity from cortical levels to the patient’s spinal circuits (Binks et al. 2023). This effect was even confirmed using functional Magnetic Resonance Imaging (fRMI) (Nagai and Tanaka 2019), which has reported a significant difference when comparing SVS+MI with only SVS. Interestingly, Taube et al. reveal that the use of SVS from one hand (termed OwnSVS) generates a stronger \(\mu \)-Event-Related Desynchronization (\(\mu \)-ERD) response in the frequency range from 8 to 12 Hz in the sensorimotor area compared to the non-SVS hand (Taube et al. 2015). In addition, our previous study mentioned the importance of using AOH in a protocol for hand opening and closing, where ERD decreased more when using the robotic glove rather than using only the hand movement performed by healthy subjects (Blanco-Díaz et al. 2023).

It is worth mentioning that the relationship between SVS and motor function has already been demonstrated in studies conducted with post-stroke patients, using the traditional technique of Mirror Therapy (MT) implemented by Occupational Therapists (de Freitas Zanona et al. 2023). MT is a technique in which the patient positions their functional limb on the reflective side of a mirror, and the limb affected by hemiplegia is hidden on the opposite side. The less affected limb is visible to the patient, whereas the hemiplegic limb is hidden from view. Hence, the mirror reflects the image of the functional limb, and consequently, the patient perceives the mirrored image as his/her hemiplegic limb (Denslow 2023). This setup enables the patient to mentally simulate movements in the hemiplegic limb while actively moving the functional limb (Taube et al. 2015). Therefore, MT-based strategies have also been considered in recent BCIs (Rungsirisilp and Wongsawat 2022).

It is known that when MI is performed, it evokes changes in ERD and Event-Related Synchronization (ERS) patterns associated with \(\mu \) and \(\beta \) rhythms. ERD/ERS are phenomena that produce decreasing/increasing power changes in specific frequencies of brain signals, such as Electroencephalography (EEG) signals (Pfurtscheller et al. 2003; Pfurtscheller 2001). It is worth mentioning that the act of imagining or executing a complex hand movement can enhance motor learning (Heena et al. 2021) and neural plasticity (Ruffino et al. 2017). This may be significant for post-stroke patients’ rehabilitation owing to imaging complex tasks requires more concentration from the patient and more cognitive control to generate discriminative patterns between classes (Guerrero-Mendez et al. 2023a). In addition, the imagination of complex hand movements can help provide greater usability, controllability, and reliability for rehabilitation systems based on BCIs.

The BCI principle for upper limb rehabilitation systems is widely based on the classification of the user’s intention of hand KMI by EEG (Wang et al. 2023). Commonly, most upper limb-based BCIs aim at the recognition of right and left-hand KMI or between resting and MI of movements (Meng et al. 2016; Wang et al. 2023). BCIs with robotic end-effectors have demonstrated promising results under this concept for the rehabilitation of people with disabilities. As a limitation, binary classification-based systems have also been associated with limited degrees of freedom, which can affect the human–machine interaction, and with this, the BCI usability (Padfield et al. 2022). Therefore, systems should be able to discriminate different hand movements from the same limb using EEG. In this context, open or closed-hand postures do not appear to be a highly complex task, as explored in Guerrero-Mendez et al. (2023a), but it continues to be a high challenge considering the different factors that affect the BCI performance, such as inter-subject variability, concentration, among others (Lee et al. 2019). Moreover, the current resolution of low-cost EEG systems poses a challenge for easily measuring deep cortical brain activity, which gives an opportunity for computational strategies to discriminate activity related to the same limb, such as hand opening and closing (Tavakolan et al. 2017; Alazrai et al. 2019).

In order to improve existing BCIs, a recent review reported that Kinesthetic Motor Imagery (KMI) produces neural signals that are easily classified through EEG (Simon et al. 2021). This occurs because of motor cortical regions are strongly activated during KMI, since, in this case, the subject is asked to imagine movements, also feeling sensations elicited by a robotic glove, rather than just visualizing these movements, as occurs with visual MI (Nagai and Tanaka 2019). In fact, studies conducted by Frolov et al. (2017) show that adding BCI control to robot-assisted physical therapy through KMI to open/close both hands improves post-stroke rehabilitation outcomes. The literature shows that the power changes of the sensorimotor rhythms can differ according to handedness, significantly affecting the performance of MI tasks (Zapała et al. 2021). Moreover, when using MI, various physiological and non-physiological artifacts impact the weak amplitude of EEG signals, complicating the recognition of this type of imagination pattern (Padfield et al. 2022), which underscores the difficulties inherent in this research.

Fig. 1
figure 1

A Location of the brain motor area: M1, SMA, and PMA; B electrode locations used in this work

Full size image

This work presents a novel protocol of EEG acquisition of open/close hands based on KMI and SVS. In this protocol, the participant is asked to observe a static image of their own hand (either open or closed) on a screen, while concurrently perceiving the movement of their hand facilitated by an AHO. Our protocol was evaluated with healthy subjects whose EEG signals were processed and classified. We believe that this protocol can be used to design BCIs for the rehabilitation of hand and finger movements of post-stroke patients. As an additional contribution of this study, it is worth noting that the protocol proposed here is different from the commonly conducted, in which the subject only imagines their left and right-hand movements.

Materials and methods

EEG preparation and acquisition

In our protocol, EEG signals are captured through a cap with 16 Ag-Cl electrodes, positioned according to the International 10/20 System. A biosignal acquisition system composed of two boards: Cython and Daisy (OpenBCI, US), acquire the subject’s EEG signals. This system is controlled through a Graphical User Interface (GUI) installed in a laptop connected only to the battery itself. The connection between the laptop and the signal acquisition boards is wireless, taking place through a dongle connected to the laptop’s USB port. For this study, we modified the original location of the OpenBCI EEG cap, so that the electrodes used in our research were grouped close to the SMA, M1, and PMA of the brain, as shown in Fig. 1A, which shows the specific cortex area responsible for hand movements (area of interest for our research).

During the MI task, the initial cognitive activity that manifests the intention to perform movement tasks originates in the frontal cortex, propagating toward the motor area (Padfield et al. 2022). Thus, based on this knowledge, the electrode locations for our research were defined as FP1, FP2, F3, F4, FC3, FCz, FC4, C5, C3, C1, C2, C4, C6, CP3, CPz, and CP4, as shown in Fig. 1B, with reference electrodes (A1 and A2) placed on the earlobes. According to the literature, this electrode positioning has demonstrated the most relevant power changes and covers the most significant sensorimotor areas to discriminate MI tasks (Alazrai et al. 2019).

The EEG signals are recorded at a sampling rate of 125 Hz, and the impedance of the electrode skin was kept below 20 k\(\Omega \). This impedance value close to 20 k\(\Omega \) allows us to have a faster process of wearing the EEG cap on the subject’s head (about 15 min), preventing post-stroke patients from giving up rehabilitation therapy, which is very common, according to our experience. However, very high impedance values affect the reliability of the EEG signals, increasing the level of outliers. It is worth commenting that in our previous work (Romero-Laiseca et al. 2020), we have conducted experiments with a lower electrode-skin impedance (10 k\(\Omega \)). However, this process required more preparation time (more than 30 min), causing discomfort and boring the subjects.

Protocol design

Figure 2 shows the timing diagram of the experiments for EEG signal acquisition using the experimental design based on KMI+SVS. The protocol includes user training and recording of the EEG signals based on their own SVS in first person and KMI with AHO, represented by a robotic glove. The actions (or classes) contemplated in our protocol are Open Hand KMI (OH-KMI) and Closed Hand KMI (CH-KMI). A GUI was developed using OpenViBE software (Inria, FR), which is compatible with Python and displays the SVS synchronized with the robotic glove to generate the protocol sequences. The stimuli are displayed on a 15.6-inch computer screen, 60 cm away from the participants.

The first phase of our protocol corresponds to 16 s of baseline, which is a black screen displayed to the participant. Subsequently, the screen shows figures representing the hand actions (two classes: OH-KMI or CH-KMI). The AHO used here is a pneumatic robotic glove (Model ML-115A from Gendoing, CN), powered by 5 V/2 A, with a pressure range from -58 kPa to 120 kPa. An electronic circuit based on a generic 6V solenoid valve, a voltage source, and a microcontroller board Arduino Mega 2560 (Arduino, IT), is used to inflate the glove pneumatic bags, opening the AHOs, according to the proposed protocol.

Each subject completed a session of ten trials for MI conduction (Fig. 2), with a total of 150 trials per class. Between each trial, the subject rested for 30 s to prevent mental fatigue. The total duration of the protocol was approximately 65 min (approximately 5 min for instructions on how to perform the protocol; 15 min for fitting the cap on the subject’s head, placing the gel, and obtaining electrode-skin impedance below 20 k\(\Omega \); and 45 min for the acquisition of EEG signals following the protocol).

Fig. 2
figure 2

Protocol for KMI paradigm based on SVS and a robotic glove. Notice C refers to close hand, while O refers to open hand

Full size image
Fig. 3
figure 3

Experimental protocol carried out by all subjects for open- and closed-hand KMI with SVS

Full size image

Protocol execution

The protocol was carried out in ten healthy subjects (seven men and one woman right-handed, and two left-handed women, 28.4 ± 11.6 years). The subjects voluntarily accepted to participate in the experiment and were fully informed of the tasks to be performed. In addition, they signed the Free and Informed Consent Form, in accordance with the Declaration of Helsinki. The protocol used in this research was approved by the Ethics Committee of the Federal University of Espírito Santo (Brazil) under the CAAE number: 39410614.6.0000.5060. In each experiment, only each subject and the researcher were present in an isolated room to avoid interference from other people. Figure 3 shows images of the ten participants performing the protocol.

Fig. 4
figure 4

Block diagram of the methodology used in this work for the recognition of hand opening/closing MI

Full size image

During the experiment (see Fig. 2), the participant was seated in a comfortable chair and asked to imagine the movement of the hand while the image of a figure representing the movement appeared on the screen in front of him/her. This was done to reinforce the MI and generate visual feedback to the subject. In addition, subjects could observe their own hand (worn with the AHO) in front of the screen. Synchronously, the AHO moved according to the image of the hand displayed on the screen. Thus, the concepts of SVS of the subject’s own hand (OwnSVS) and KMI were applied. Participants were instructed to avoid eye movements during the protocol. Before obtaining EEG signals, subjects used the AHO to familiarize themselves with both the device and the tactile sensations transmitted to the hand and its five fingers, which they had to mentally simulate during MI.

EEG processing and classification

Figure 4 shows the strategy developed to process the EEG signals and recognize the action performed: hand opening and closing MI. The complete process starts with the EEG acquisition followed by a pre-processing stage with the application of a 5th order zero-phase Butterworth filter and a Common Average Reference (CAR) filter. Then, the feature extraction process based on Power Spectal Density (PSD) and Riemannian Geometry (RG), and feature selection using Pair-Wise Feature Proximity (PWFP) are executed. Note that for the RG method, the CAR filter was not implemented, in order not to overlap spatial filters and avoid loss of EEG information. Subsequently, these features are the input of four classifiers: Linear Discriminant Analysis (LDA), Support Vector Machine (SVM), k-Nearest Neighbors (kNN), and Decision Tree (DT). Finally, two evaluation metrics (ACC and F1 score) were used to determine the algorithmic performance to classify the two classes mentioned above.

EEG signals pre-processing

Initially, the signals are segmented for each experiment to perform a better generalization of the algorithms and increase the size of the examples, avoiding under- or over-fitting. In this study, the 300 total trials per subject are split into 80% and 20%, randomly, where the first 80% of the EEG data are used to train and validate the data, whereas the remaining 20% of the trials are used to evaluate the algorithms. This strategy was implemented to fit the models in a way that can identify unknown data, which would allow a generalization of the classification strategy (Delisle-Rodriguez et al. 2019). Then, each trial is segmented into 1-s time windows with 50% overlap. Subsequently, a CAR filter was applied to reject the common noise between channels (not applied for RG). Subsequently, the Butterworth filter is implemented in the \(\mu \) and \(\beta \) centered frequency bands between 8 and 30 Hz.

Feature extraction

  • Power Spectral Density (PSD) PSD is normally used in MI-based BCIs (Blanco-Díaz et al. 2023; Padfield et al. 2022), as PSD allows mitigating fluctuations in energy values observed during the MI. PSD value is calculated by considering three filter banks: Mu (\(\mu \), 8–12 Hz), Low-Beta (Low-\(\beta \), 13–17 Hz), and High-Beta (High-\(\beta \), 18–24 Hz), using the Fast Fourier Transform (FFT) on each set of signals. This methodology is performed according to Algorithm 1.

    Algorithm 1
    figure a

    Feature extraction based on PSD.

    Full size image

    The algorithm represents the feature extraction based on PSD for the pre-processed EEG signals for each filter bank. Once PSD is calculated, data normalization is performed using \(z-score\) before the feature selection stage, considering training and evaluation data sets.

  • Riemannian Geometry (RG) RG has attracted considerable attention in recent studies reported in the literature, especially when it comes to identifying mental activities such as MI (Delisle-Rodriguez et al. 2019). RG relies primarily on unsupervised techniques, including temporal filtering and estimation of covariance. A notable advantage of this method is its ability to minimize prevalent interference while amplifying the difference in variance among EEG trials derived from various motor activities. Interested readers can explore more deeply the computational algorithms associated with RG in Delisle-Rodriguez et al. (2019). The Algorithm 2 specifically elucidates how RG extracts EEG features when used in binary classification scenarios.

    Algorithm 2
    figure b

    Feature extraction based on RG.

    Full size image

    Although the application of RG for MI has emerged as a powerful and promising technique, this technique focuses on the intrinsic structure of data spaces, allowing capture and analysis of the variability and essential features of EEG signals. A significant challenge encountered in the implementation of this methodology is the high dimensionality inherent in the resultant data (Barachant et al. 2012). This high dimensionality may be counterproductive, as it can significantly reduce the precision of identifying specific patterns in EEG signals (Barachant et al. 2012, 2013). Furthermore, the presence of irrelevant or redundant features can introduce noise into the analysis, leading to misinterpretations or the failure to detect subtle but critical patterns. Therefore, it is imperative to employ feature selection methods that allow the filtering and retention of only those attributes that are truly relevant to the task. These methods not only improve the accuracy of the analysis but also optimize computational time, allowing more efficient real-time applications (Barachant et al. 2010). In summary, while RG offers robust tools for the study of MI through EEG, it is essential to complement it with appropriate feature selection techniques to ensure optimal results, which are discussed below.

Feature selection

The PWFP method has been established as an effective tool for feature selection (Happy et al. 2017; Romero-Laiseca et al. 2020). Thus, PWFP is used after the feature extraction stage for both PSD and RG. PWFP focuses on evaluating the proximity or distance between individual features in a high-dimensional space. Usually, this proximity is measured using the Euclidean distance. For EEG, where features can have complex shapes and non-linear relationships, the appropriate choice of a distance metric is essential (Happy et al. 2017; Delisle-Rodriguez et al. 2019). However, to maintain a low computational cost, the Euclidean distance is used here as the selection metric. Once these distances are computed, PWFP selects those pairs of features that have significant proximity, indicating that they are relevant and complementary features for EEG class discrimination (Happy et al. 2017). In doing so, this method allows for reducing the dimensionality of the data by preserving only the most informative features, improving accuracy and efficiency in the subsequent analysis (Romero-Laiseca et al. 2020). The singularity of PWFP lies in its exploitation of the k-Nearest Neighbors (kNN) principle (Happy et al. 2017), as instead of only considering pairs of features, PWFP evaluates the proximity of a given feature to its kNN in feature space. The method could be applied as follows.

  • Distance Computation: The distance between two features (\(f_i\) and \(f_j\)) can be calculated using the Euclidean distance:

    $$\begin{aligned} d(f_i,f_j) = \sqrt{\sum _{i=1}^{n}(f_{i,k} - f_{j,k})^{2}}, \end{aligned}$$
    (1)

    where i the individual dimension of the features.

  • Selection based on kNN: Once the distances have been calculated, for each feature \(f_i\), its k-nearest neighbors are identified and their average distance is stored. In this case, five near neighbors are used, according to the suggestions made in Delisle-Rodriguez et al. (2023).

  • Percentile-based approach: After obtaining the average distances for each feature with respect to its k-nearest neighbors, we set a threshold based on percentiles. For example, if the 90th percentile is selected, we retain those features whose mean distances are in the top 10%, suggesting that they are the most distinctive. In this case, the 75th percentile is used as a criterion for the selection of the features.

In this way, RG and PSD were obtained after performing this step, considering a time window of 1 s with 50% overlap. All recorded EEG channels were used, with the PWFP method as the feature selector.

Classification

Four conventional classifiers were used to recognize both MI tasks (opening/closing hand), such as LDA, SVM using a polynomial kernel, kNN with a \(k=1\), and Decision Tree (DT). To derive a robust and generalized model, a 10-fold cross-trial validation methodology was employed, ensuring that the data used in the training were not present in the evaluation of the methods.

Evaluation methodology

  • Topographic representations To observe the spatial effects of power around the brain, power changes are located for each channel and recorded on topographic maps, where power changes are determined by the calculation made in the methodology. The power effects are compared for two situations: the subject using AHO with movements (i.e., an AHO moves the subject’s hand) and the subject using AHO without movements (the AHO does not move the subject’s hand) during the MI. In this study, ERD/ERS was not used for signal analysis because the objective here is to analyze the spatial power variations that occur during the cognitive process by comparing the absolute power effects when either the AHO moves the subject’s hand and when it does not move the subject’s hand.

  • Evaluation metrics Accuracy (ACC) and F1 scores were used to assess the performance of the proposed classification strategies. From the 80% EEG data split before, a 10-fold cross-validation was performed. Each implemented fold was randomized, where 90% of the data set were used to train the models, whereas the remaining 10% were used to validate the algorithms. Additionally, the two feature extraction methods (PSD and RG) were implemented with the four classifiers and also considering the feature selection stage to analyze the response with and without this stage. Subsequently, classification methods were evaluated in the remaining 20% of total data.

Fig. 5
figure 5

Topographic maps (average of all subjects) of EEG signals for both conditions: with and without AHO movements for \(\mu \) (A–D), Low-\(\beta \) (E–H), and High-\(\beta \) bands (I–L). The blue color indicates low power values, meaning that the power of the frequency band decreases during KMI, whereas Red indicates the power increase during the KMI period

Full size image

Results

General topographic maps

Figure 5 illustrates the power effects obtained from the experiments, which show the location and intensity of brain activation in the \(\mu \), Low-\(\beta \), and High-\(\beta \) bands for the mean of all subjects. All subjects followed the protocol with SVS (in which subjects observe figures of an open or closed hand on a screen) and KMI (the subject receives kinesthetic sensations and movements from the AHO, represented by the robotic glove). It is noteworthy to highlight the difference between brain activations for the open- and closed-hand tasks in AHO use and non-use. Here, low power values are represented by blue color and high values by red color. Thus, it is possible to observe that the use of AHO evokes more differentiated power changes compared to the non-use of AHO. For each of the tasks using AHO, the power increases were presented in the \(\mu \) band with ipsilateral behavior, where in the C1, C3, C5, and FCz channels the power increase is centered as shown in Fig. 5A and B. For non-use AHO, no differential power changes were presented for the \(\mu \) band, maintaining low power around the entire motor cortex of the brain. Furthermore, power changes occurred in the Low-\(\beta \) band using AHO, presenting a contralateral behavior for the opening task, which is associated with left-hand motor movements (Rungsirisilp and Wongsawat 2022; Sadaghiani and Kleinschmidt 2016). For the closing task, a decrease in power was presented in the same band; however, this was localized in the fronto-central area, as shown in Fig. 5E and F. For High-\(\beta \), contralateral behavior was present in the opening task, which is marked by a decrease in power centered on C4, and for the closing task, contralateral and ipsilateral behavior was presented, with C3 presenting greater power than C4. As mentioned above, the non-use of AHO, does not cause a specific or centered power variation in any channel, as shown in Fig. 5C, D, G, H, K, and L, where the use of AHO causes a greater power variation, which may be more associated with an ease of discriminating closing and opening hand KMI tasks.

Fig. 6
figure 6

Performance metrics of all classifiers considering all subjects. A ACC using AHO, B ACC without AHO, C F1 score using AHO, and D F1 score without AHO

Full size image

Classifier performance

Figure 6 shows the average performance of each classifier in terms of ACC and F1 score for all subjects performing the protocol with and without AHO movements. In Fig. 6A, it is possible to see that the PSD-PWFP reached the highest ACC in comparison to RG-PWFP, obtaining 81.04±3.49%, 85.36±3.14%, and 67.92±3.56% against 71.17±3.18%, 75.48±4.07%, and 64.90±4.16%, for SVM, kNN, and DT classifiers, respectively, with AHO movements. Likewise, Fig. 6B leaves evidence that PSD-PWFP obtained higher ACC compared to RG, with values of 82.62±3.42%, 84.07±3.86%, and 67.87±3.52% against 66.98±5.48%, 72.10±5.39%, and 58.45±4.21% for SVM, kNN, and DT classifiers, respectively, without AHO movements. For the LDA classifier, the RG method presents higher results compared to PSD for both with and without AHO movements.

Considering the F1 score, the results are proportional to ACC, where it is possible to observe higher values for PSD-based feature extraction compared to RG. For instance, PSD-PWFP with kNN achieved an F1 score close to 85±4% and 83±5% with AHO and without AHO movements, respectively. The RG-PWFP values were 75±3% and 71±7% for the same conditions (see Fig. 6C and D). Furthermore, these findings allow inferring that the performance of kNN outperforms LDA, SVM, and DT, considering the feature selection method. For example, it is possible to observe for PSD that the mean F1 score is 85.36% and 84.07% with PWPF, whereas without feature selection this metric is around 73.01% and 68.39% for the conditions without and with AHO movements, respectively. Similarly, the feature selection method improves the RG performance, where a F1 score close to 75.46% and 72.10% is obtained, and without using PWFP, values of 54.60% and 53.41% are obtained using kNN for with and without AHO movements, respectively.

Fig. 7
figure 7

Performance metrics per subject carrying out the protocol with glove and without glove movements, considering RG and kNN classification strategy A ACC, B F1 Score

Full size image

Analysis about the use of SVS and KMI

One of the objectives of this study corresponds to improving the identification of MI tasks of the same hand, such as opening and closing, using image-based SVS and KMI induced by AHO movements. Thus, considering that the PSD-kNN method using PWFP was the one with the highest scores, Fig. 7A and B represent the results of the performance metrics for each subject in the two situations: execution of the protocol using only SVS (without AHO movements) and SVS with AHO movements. Here, it is possible to observe that S1 represents the best performance with AHO movements (ACC = 89.81% and F1 score = 90%), while S9 shows better results without AHO movements (ACC = 89.74% and F1 score = 89%). In general, performance metrics with AHO movements reached higher values compared to those without AHO movements, highlighting an ACC of 85±3.14% over 83.10±3.86% for PSD-PWPF, and 75.48% over 72.10% for RG-PWPF, indicating an increase of approximately 3%. However, this increase was not significant, according to the application of the two-sample t-test, as a p-value equal to 0.37 was obtained for ACC, and 0.21 was obtained for F1 score.

Discussion

This work presents the cortical and performance effects of using a novel protocol employing AHO with and without movements to classify hand opening/closing KMI tasks. Regarding the classification results obtained in this research, it is worth commenting that the literature reports the use of different classifiers to recognize MI tasks, reaching ACC close to 75%, based on different signal features (Vourvopoulos et al. 2022; Guerrero-Mendez et al. 2023b). However, most studies only present results when performing MI tasks in the right and left hands, which could limit the usability and design of BCIs based on upper limbs (Padfield et al. 2022). Furthermore, considering that the cortical effects in the brain during tasks of the same hand, such as opening and closing, may be similar, the identification of these types of tasks becomes challenging (Guerrero-Mendez et al. 2023a). In this context, some authors have focused on classifying MI tasks on the same hand, where ACC values close to 73% have been reported (Alazrai et al. 2019; Tavakolan et al. 2017). However, those studies implemented more complex algorithms, such as Artificial Neural Networks (ANNs), which may increase BCI calibration times and consume more computational resources. In this way, it is important to highlight the methodology implemented in our study, which achieved an ACC close to 85% (maximum 89%) and a F1 score close to 85% (maximum 90%) using PSD-kNN with feature selection to identify hand opening/closing KMI, improving by 3% the identification of tasks compared to without AHO movements. In this context, it is also important to highlight the role of the feature extraction and selection stages, where it was possible to determine that, unlike previous studies, PSD-based methods were more discriminant than RG, which may be related to the fact that frequency information may be more useful than time domain information for the identification of upper limb tasks from the same limb (Barachant et al. 2010, 2012, 2013; Alazrai et al. 2019; Tavakolan et al. 2017).

On the other hand, the literature also presents studies with results based on public datasets, which could be a disadvantage for real-time implementations (Vourvopoulos et al. 2022), while in our study the complete dataset was developed and analyzed by ourselves. Our data set plays an important role in the field of BCI, considering that, to the best of our knowledge, the literature does not report the inclusion of SVS of the own subject’s hand (OwnSVS) and KMI together. Furthermore, studies, such as those presented by Rungsirisilp and Wongsawat (2022), indicate that the use of SVS+MI (not using KMI) may be an alternative strategy to improve the effectiveness of BCIs for post-stroke rehabilitation. However, incorporating movement as part of KMI represents a new approach to EEG-based BCI training, which may improve neuroplasticity, considering mental tasks and the sensory feedback produced by AHO (Frolov et al. 2017). For this reason, in our experimental design, the movements assisted by the AHO are present throughout the protocol, which can be pivotal in therapeutic interventions for patients experiencing hemiparesis as a result of neurological conditions.

One possible explanation for the observed better performance of the kNN classifier in our classification tasks could be related to the use of the same technique during the feature selection stage. By employing kNN in both feature selection and classification, it is plausible that the selected features are inherently optimized for this classifier, resulting in improved performance. However, it is important to note that in our study we conducted cross-validations and compared multiple methods to ensure the robustness of our findings. We evaluated the performance of several classifiers with the same selected features and observed that kNN consistently outperformed other methods. This suggests that the superior performance of kNN is not only due to the use of the same technique in feature selection but also to its suitability to the structure of our data.

Crucially, the execution of voluntary motor activities induces localized desynchronization within the upper-alpha and lower-beta frequency bands in proximity to the sensorimotor cortex. However, in post-stroke patients, ERD during MI or movement execution is commonly reduced or absent in the affected hemisphere, due to damaged neurons (Ramos-Murguialday et al. 2019; Moslehi et al. 2020). This fact needs to be considered to adapt our protocol for post-stroke patients. There are some limitations in this study due to MI instructions explained by the researchers are not always the same as that performed by the subjects. One of the reasons for this mismatch is the difficulty of MI, especially KMI, as verified in a previous study (Igasaki et al. 2018). In addition, it should be noted that among subjects there were both right-handed and left-handed. This latter group represents about 10% of humans (de Kovel et al. 2019). However, research has predominantly centered on the right-handed population (Willems et al. 2014). On the other hand, in terms of processing signals, a hyperparameter study was not performed. This way, fixed parameters were chosen to recognize opening and closing hand tasks. Therefore, innovative solutions, such as signal analysis techniques, hyperparameter selection study, and user training methodologies should be formulated with a foundation in the attributes of this specific group and to generalize the classification system.

Finally, to confirm the advantage of our protocol in terms of improving the efficacy in post-stroke rehabilitation, future works will focus on performing a hyperparameter selection to indicate the best classifier performance as well as studying the effects of our methodology in real time for the development of an AHO-based BCI controlled by MI.

Conclusion

The findings of this research may be a very useful complement for neurorehabilitation of post-stroke patients, as well as for controlling robotic devices for assistance and rehabilitation. However, such as aforementioned, ERD during MI or during movement execution is commonly reduced or absent in the affected hemisphere of post-stroke patients, due to damaged neurons, which will require the adaption of our protocol to be applied to this group of people. Four classifiers (LDA, kNN, SVM, and DT) and a PWPF-based feature selection stage were used for the recognition of open- and close-hand tasks using PSD and RG feature extraction methods. The frequency bands of \(\mu \), Low-\(\beta \), and High-\(\beta \) were extracted from subjects who carried out the protocol with or without AHO movements. As a results, the feature using PSD-PWPF achieved a maximum ACC of 89% with the kNN classifier and a mean ACC close to 85%. Furthermore, the F1 score values with PSD were considerably higher than using the RG-based method. Regarding the protocol carried out with AHO movements, the mean ACC was 3% higher compared to without AHO movements. Despite the limited number of subjects (ten), we believe that the findings of this research may be useful for the rehabilitation of hand and finger movements in post-stroke patients, and providing a contribution to the field of BCI designed for the rehabilitation of people with disabilities.