Keywords

1 Introduction

Successful implementation of neuro-robotic systems employing the EEG-BCI technology requires careful consideration of multiple factors as summarized in Fig. 6.1. In this chapter, we focus on the real-time implementation and discuss the limitations of the neuro-technology. We also provide solutions provided by our group to some of the major problems that prevent wide-scale usage of such systems.

Fig. 6.1
figure 1

Neuro-Robotics Framework: Many critical components have been defined for a successful Brain–Machine Interface (BMI) based neuro-robotics implementation. Although the application domain in this chapter is lower-body powered exoskeletons, these components also apply to other types of systems and application domains

Full size image

The selection of the compensatory/restorative robot depends on the intended use, the user’s capabilities, and the needed functionality. Exoskeletons differ on their number of actuated joints, mobility and speed, and how they are controlled by the user and/or therapist. The usability of an exoskeleton, for instance, depends on the user’s needs and capabilities and also refers to the different modes of movement, support, and movement intensity that the exoskeleton can provide. The selection of an appropriate physical interface between the robot and the user (i.e., harnesses, straps) is generally concerned with the form factor and should provide safe and effective implementation during the dynamic motion of the exoskeleton. These options are usually promoted by the exoskeleton manufacturer. However, the response time of the exoskeleton systems are often overlooked. Response time refers to the time elapsed between a given movement command and the actual observed execution. Most of the systems provide time delays that are within acceptable ranges for generic use. However, interfacing with a physiological control modality (i.e., BCI control, myoelectric control, or even motion control) the exoskeleton time delay can cause lagged response, therefore, providing altered feedback to the users.

Another consideration should be made on the measurement modality, such as EEG or any other physiological (e.g., electromyography or EMG) or mechanical (e.g., detection of body tilt using accelerometers) signal, for the detection of the initial movement intent and continuation of the movements. Each control modality has its own set of major challenges to overcome. In this chapter, we are concerned with the EEG measurements for BCI control of exoskeletons (Fig. 6.2) and the critical problems that are associated with them (for a recent review, see (He et al. 2018)). The cortical information that can be extracted from the EEG is proven to be adequate for motor movements (Bhagat et al. 2016; Venkatakrishnan et al. 2014; Bradberry et al. 2008, 2009; Contreras-Vidal et al. 2010; Kilicarslan et al. 2013). Our group demonstrated the decoding of movement intention for lower-body exoskeletons using slow cortical oscillations (delta band [0.1–4 Hz]) (Kilicarslan et al. 2013). EEG measurements, however, are considered having low signal-to-noise ratio (SNR), due to physiological and non-physiological artifacts. Any real-time implementation using EEG should consider the artifacts that have similar frequency contents compared to that of the EEG features used for control. The event-locked nature of many types of artifacts makes it exceedingly difficult to implement high-performance neural decoders. In EEG delta band, the ocular and motion artifacts, as well as impedance changes can be considered as dominant factors that adversely affect the EEG measurements on all scalp electrode locations. We will provide real-time compatible solutions to handle these two major EEG contaminants. Other considerations for a successful real-time implementation, such as the neural decoders, EEG processing pipeline, and day-to-day variability will also be discussed in this chapter.

Fig. 6.2
figure 2

General framework for a neuro-robotic application. Blue region indicates the type and closed-loop structure of the implementation discussed in this chapter. From Venkatakrishnan et al. (2014) with permission

Full size image

Understanding the neural dynamics before and after the rehabilitation paradigms using BCIs is a major topic that can have a significant contribution to the implementation of such systems. Comparing the neural source activations and assessing neural plasticity during a longitudinal use can improve the overall success of the implementation. This allows for the optimization of the decoder parameters taking into account contextual information leading to changes in internal states of the user, day-to-day changes in neural variability, as well as optimizing the number of sensors. Most of the methods that are discussed in this chapter are also applicable to other systems that are intended for EEG-based BCI systems for rehabilitation robotics (i.e., upper limb, see Chaps. 2, 5 and 13), virtual reality, and software interfacing.

The generic framework of the implementations that are discussed in this chapter are summarized in Fig. 6.2. In this closed-loop implementation, the EEG data are measured, processed, and passed into a decoder that interprets the user’s intention of movement. The decoder output is then applied to the exoskeleton as a control command. In our discussed application domain, unless an external device is configured to provide a specific type of feedback, the users experience visual, proprioceptive and kinesthetic feedback while wearing the exoskeleton. In a BCI setting, the completion of the task is usually felt by the motion of the exoskeleton or fed to the user as a computer graphic or audio cue. Following this closed-loop implementation, the clinician monitors the recovery and individually adjusts the intervention parameters according to the functional status of the user (i.e., range and speed of the movement, therapy times, and repetition rates of the given tasks).

2 Robotic Systems: Lower-Body Exoskeletons

Orthotic/prosthetic systems usually cover a single human joint and provide support or constrain the motion of the human limb. Both type of systems can be active or passive in design. An active system is one which provides actuated motion or assistance to a human joint or artificial limb via an external electromechanical system attached to it, such as electric motors, pneumatic or hydraulic actuators. A passive system does not have an energized external actuation mechanisms. Systems that have freely rotating joints fall into this category. A passive system can also be designed to apply resistance to the free motion via a dampening system or it can store energy and release it via a rotational or extension/compression springs. We define a lower-body exoskeleton as an electromechanical system that is attached to multiple human lower-limb joints to provide a coordinated activity, support or motion. These robotic devices can also be designed as active or passive systems. However, since the main purpose of an exoskeleton is to provide assist-as-needed support of coordinated movements of two or more joints, at least one is usually designed as active joint. It is also possible to implement a combination of active and passive joints in a single exoskeleton system.

Exoskeletons can be grouped by their control modalities as being assistive, resistive, corrective, or fully active. An assistive (or assist-as-needed) control of an exoskeleton is based on the measurement of the residual limb activation of the end-user (using sensors on the exoskeleton). For a given task, the measured voluntary effort of the end-user is compared against the effort that is required to complete the task. If any deviation is detected, the exoskeleton completes the action and helps the user to complete the given task. The degree of assistance that the exoskeleton applies can be adjusted by the physical therapists. A resistive exoskeleton utilizes like in assistive exoskeletons the residual limb activation of the person. In this case, the exoskeleton applies resistance to the motion, similar to an exercise equipment, to execute a strength training. A corrective exoskeleton is combining the basic characteristics of an assistive and a resistive control modalities. Given that a patient is able to complete the task, but not in an optimal way, the exoskeleton measures the limb activation and trajectory (i.e., the limb joint angles at any given time) and compares it with the optimal values that of an person without disabilities. It then corrects the deviations from the physiological task trajectories. This control modality mostly concerns with providing synchrony between individual joints. Finally, a fully supportive exoskeleton exerts the optimal path, force, and speed to the impaired limb of an end-user to accomplish the given task. This is usually the case for patients with motor complete spinal cord injury (SCI) with very little to no residual voluntary limb movement.

Lower-limb exoskeletons are considered as a promising tool in motor rehabilitation programs with the potential to improve motor and physiological functions such as bladder and bowel or cradiovacular functions, (Federici et al. 2015; Contreras-Vidal and Grossman 2013). In addition, these devices may also reduce the physical burden on clinical staff, quantitatively assess the progress of rehabilitation, and benefit from the fidelity of repetitive training (Banala et al. 2008).

2.1 User Selection, Usability, Form Factor, and Response Time

There are several key factors on selecting the appropriate robot for lower-limb rehabilitation. The area of application of the exoskeleton is depending on the ability of the exoskeleton to provide the needed assist-as-needed, supportive, resistive or corrective control modalities. The selection of one or multiple control modes depends on the end-user’s needs and capabilities. As an example, if the implementation is chosen for patients with complete SCI, the assist-as-needed control mode might not be feasible as the users will not be able to exert any voluntary residual movements. Whereas for end-users with incomplete SCI or after stroke, the assist-as-needed or corrective modes might be preferable. The control modality should be determined by the clinical expert as the implementation often needs to be individualized. This, in general, concerns the personalization of the exoskeletons for specific patients, closely followed by the patient’s needs for personalized form factors.

The form factor can be defined as the patient’s physical interaction with the exoskeleton which must be done in a safe and comfortable manner. For lower-body exoskeleton systems, the end-user’s joint locations must be matched to those of the exoskeleton. Unless the exoskeleton is personalized for a specific patient, most systems are designed to be adjustable to the upper and lower-limb segmental lengths of the users. However, since the exoskeleton joint cannot precisely mimic the physiological trajectory of human joint center of rotation (i.e., knee), some level of mismatch is unavoidable. The exoskeleton can only be adjusted to the center of the human joint for a given fixed position for the knee, meaning that only the instantaneous center of rotation can be measured, albeit within a reasonable margin. Most exoskeletons are designed to carry almost 100% of the body weight of the users and thus are comprised of powerful actuation mechanisms. Therefore, a large mismatch between the anatomical and the technical joints can cause serious injuries, especially when the end user has a pathological bone density. Exoskeletons are dynamic systems, thus any unwanted forces that are caused by the joints’ mismatch will be experienced by the end-user in a cyclic fashion during walking. The unphysiological loading pattern might cause musculoskeletal problems, depending on the end user’s physical condition. Another rather unavoidable consequence of misalignments of technical and anatomical landmarks is the rubbing of padded harnesses on the skin (harnesses that hold the upper and lower leg, i.e., at shin level). It is advisable to check for skin conditions before, during, and after each exoskeleton session while the user is in a safe position (e.g., sitting with the exoskeleton). The intensity level of the training and duration of the session, therefore, should be adjusted in a way to minimize the risk for such complications. Other usability and form factor considerations include the safe placement of the user’s feet, not to limit any physiological movements (e.g., at the ankle level) and setting an adequate level of pressure that the harnesses provide to the user’s legs as excessive pressure can negatively impact local perfusion and blood flow (He et al. 2017). One way towards optimal fit would be to 3D scan the user’s legs and providing custom, individualized harnesses as we are currently pursuing for pediatric applications (Savage 2018). Overall, it is very important to clinically assess prospective users of exoskeletons and to mitigate any potential risks of using this technology (for a recent review, please see (He et al. 2017)).

Most exoskeleton systems provide negligible time lags (decent time response for a given command to be executed). Although this might not be an issue for generic use, excessive time lag becomes a major issue when fast response to other control modalities is required (such as BCIs). Late response to BCI generated commands affects the overall performance of the human-in-the-loop-control of exoskeletons as the users might receive altered proprioceptive and kinesthetic feedback even though the command generated by their mental imagery processes are correct. Even worse, the response times can be variable, depending on where in the overall gait cycle the exoskeleton is. In such cases, the compensation of the lagged response is left to the user’s adaptation to the overall system dynamics, which is not preferable in any situation.

3 Neural Measurements and Brain–Machine Interfaces

3.1 Measurement Modalities and EEG

Engaging the patient to the rehabilitation session, thereby promoting cortical plasticity, is perhaps the most critical component in rehabilitation of patients with neurological gait disorders (Venkatakrishnan et al. 2014; Nudo 2003; Kortte et al. 2007; Blank et al. 2014). Providing the most intuitive and engaging control interface to robotic rehabilitation devices is an active research area. With their high repetition rates, sustained, precise joint activation trajectories and controllable intensities, robot-based rehabilitation at first represent a promising add-on to classical rehabilitation practices (where physical therapists manually assist with the movement of the neurologically impaired limbs). However, it was soon realized that the autonomous actuation of such devices removes the human-factor from rehabilitation and as a result the user engagement level drops significantly. In such devices, the end-users can practically be mentally and physically passive while the device moves the limbs. These factors can diminish the effectiveness of the rehabilitation as the mental component necessary for enhancement of reorganization within the central nervous system (CNS) and thus neurological and functional recovery is missing or dramtically reduced. One solution to this challenge is to harness the residual muscle activation measured by the residual EMG of the neurologically impaired limbs and assisting the patient to achieve a physiological movement accordingly (Hargrove et al. 2013). Since the electromygraphic muscle activation is a measure for the user’s voluntary intent, it could be used to estimate the end users level of mental effort. One challenge of such a control interface, however, is that the EMG activation required for a specific joint movement cannot always be accurately measured, and not all muscle activation pathways might be activated by the patient. In other words, patients might substitute physiological muscle groups’ activation pattern with unphysiolgical ones that are better suited for control of the actuated joint. This can happen consciously or unconsciously, as the goal is to make the robotic device’s activation most accurate. Avoiding this drawback is an unsolved problem, which can be minimized by very accurate placement of EMG electrodes to all subjects at all sessions and avoiding sensor shifts. It should also be mentioned that the residual muscle activation of the neurologically impaired limb is a result of the actual cortical motor intent, and thus, there is an unavoidable cortico-muscular delay between the onset of the actual intent and the muscle activation. A control interface seamlessly integrating in the body motor control sheme, therefore, require the measurement of the intent without time delays, even predicting the activation before it happens. Similar to the EMG control interfaces, some researchers measure the interaction forces between the robotic rehabilitation device and the user’s limbs as the only parameter for gait initiation. Any residual movement at a specific joint would be originating from the user’s intent, and the robot would use the detection of the voluntary generated forces to complete the full movement cycle. Although this is a good practice in terms of providing task completion for rehabilitation after the gait is initiated, the abovementioned time delay problem also applies to this type of interfaces for the actual initiation of the gait. Additionally, the residual electrical (EMG) or mechanical activation levels can be very low, and as a result, the residual movement can be increasingly hard to detect. Therefore, robust control is in some cases with very little preserved residual motor functions hard to achieve. The BCI control of such active devices, on the other hand, provides important advantages compared to the aforementioned control modalities (Venkatakrishnan et al. 2014; Wang et al. 2010; Bhagat et al. 2014). Since BCIs inherently harness one’s own thought processes and movement intentions, it ensures full user engagement to the therapy session and as a result, promote CNS reorganization and ultimately functional recovery (Luu et al. 2017). EEG-based BCI systems detect the intent of movement non-invasively and provide an intuitive control interface for exoskeleton users. In this framework, the modulation of cortical signals during user’s movement intents are detected by advanced algorithms to initiate exoskeleton movements. The individual joint angles tajectories are then executed by the exoskeleton’s internal control loop, forming an overall shared control structure.

3.2 EEG Artifacts and Information Content

The rich information content of EEG for detection of motor intent has been proven by many research efforts (Bradberry et al. 2009; Kilicarslan et al. 2013; Lotte et al. 2007). However, these analyses are mostly done offline using previously collected EEG (see Sect. 6.4.1 for a detailed discussion). The major problem with the EEG-based BCI systems is the recovery of underlying neural sources, in real time, when physiological and non-physiological artifacts are present. Here, we will review our solutions to minimaize the influence of some of the major EEG contaminants, namely ocular artifacts (eye blinks and eye movements), signal bias and drift (due to impedance changes), and motion artifacts (electrode movement). The suggested solutions for handling these physiological and non-physiological artifacts are fully real-time applicable.

Physiological artifacts can be defined as contamination of the neural source measurements by the non-neural source activations via volume conduction. As an example, eye blinking generates electrical signals due to dipoles that are located around the eyes. This activation contaminates the scalp EEG recordings where the largest contamination occurs at electrodes close to the eyes (i.e., forehead sensor locations), and propagates towards the posterior locations with changing amplitude, polarity, and phase characteristics. Non-physiological artifacts, on the other hand, can be defined as the changes in measured signal characteristics due to external causes. As an example, motion artifacts during walking are caused by the movement of the electrodes. These artifacts are manifested as oscillatory patterns with frequency harmonics, which are not present in any clean neural source activation pattern. Due to the measurement setup and subjects’ movements, these artifacts do not belong to any distinguishable statistical distribution and are highly variable even among EEG sensors, even for the same session and subject. Electrode movements also cause sudden or gradual changes in the impedance values (see Fig. 6.4b (Kilicarslan et al. 2016)). The transient behavior of these impedance changes causes the measurement to have high-peak semi-oscillatory behavior. Continuous disruption of these transients (continuous movement) result in superimposed transients and manifest themselves as artifacts with very complex dynamics. Even without the electrode movements, sensor impedances can be affected by sweating and dryed gel at the electrode/skin interface. For a successful implementation of real-time BMI applications, these highly non-linear physiological and non-physiological artifacts should be removed from all EEG sensor signals simultaneously and in real time (more details on EEG artifacts can be found in Chap. 3).

Our group has developed a real-time denoising framework for high-performance artifact removal based on the robust adaptive H∞ filtering formulation (Kilicarslan et al. 2016). We have demonstrated the effectiveness of our technique for cleaning ocular artifacts (eye blinks, eye movements), signal bias, and signal drifts, for 60 EEG locations simultaneously, in real time. Comparisons with the very-well established offline cleaning tools clearly show the improved cleaning performance accomplished by our method. One important advantage of our method is that it depends on the real-time measurement of the noise source. This might seem like a disadvantage at first due to its requirement of additional sensors for measurement; however, compared to other existing methods that depend on the definition of clean EEG segments or statistical distributions, it allows us to be very selective on what exactly is removed from the EEG measurements. Figure 6.3(top) shows before and after cleaning of EEG data, in a pseudo-real-time setting (sample by sample processing) for two subjects and frontal (FP1) electrode. Frontal electrodes are most affected by ocular artifact contamination. The yellow traces depict the stop (low) and walk (high) segments of the session while the subjects execute the tasks using a lower-body exoskeleton. The increase in longitudinal BCI decoder accuracies (for 9 sessions, spanning 3 weeks) before and after ocular artifact cleaning are shown in Fig. 6.3(bottom).

Fig. 6.3
figure 3

Sample time trace of before and after ocular artifact cleaning of frontal electrode FP1 (top), and the overall decoding accuracy increase for the same subject, for multiple sessions (bottom). From (Nudo 2003), with permission

Full size image

Artifact presence in a BCI framework has often been interpreted as a factor improving the decoding accuracies when the artifacts are also task dependent and are in the same frequency range. Both conditions are true for the ocular artifacts. However, the improved accuracies after cleaning ocular artifacts suggest that our decoder (Kilicarslan et al. 2013) is selective of the neural delta-band sources. Additionally, it suggest that the delta-band oscillations have an information-rich structure, allowing effective implementation in real time. The adaptive nature of our method allows for the real-time (sample by sample) identification of the volume conduction effects on scalp EEG measurements (Fig. 6.4). Figure 6.4c shows a snapshot of the raw EEG scalp distribution (left) and the identified artifacts (middle) when the artifacts are at their peak values. The similarities between the two indicate a high level of neural information loss due to ocular artifacts. The artifact-free scalp map (right), composed of recovered actual neural source data shows a very different amplitude distribution, demonstrating the effectiveness of our method in enhancing the information content per sample of EEG data (Kilicarslan et al. 2016).

Fig. 6.4
figure 4

(a) shows the percentage contamination of scalp areas to the characterized eye-blink (VEOG) component, where upper plot shows scalp distribution of the mean values of 15 eye blinks near the beginning of the session, and lower plot shows the mean of last 15 eye blinks for a 10 min data collection. Compared is the effect of electrode impedance changes on the scalp distribution of eye blink contamination. (b) shows the time traces of single blink components at the beginning (blue) and end (cyan) of the session for select electrodes that cover the scalp diagonally (marked white on c). Blue and cyan circles are impedance values recorded at the beginning and end of the experiment, respectively. (c) is a single sample amplitude distribution over all scalp areas when the contamination is in its peak value. The left topographical plot shows the raw EEG data amplitudes and the middle plot is only the identified contaminants’ amplitudes. High similarity between two plots suggests a high level of contamination overall scalp areas in both amplitude and scalp distribution. Right topographical scalp map shows the EEG amplitude distribution after cleaned of ocular artifacts using the real-time H∞ filter. From (Nudo 2003), with permission

Full size image

Another worsening of the signal-to-noise (SNR) ratio occurs due to impedance change of the EEG electrodes. The impedances of 8 EEG sensors were measured at the beginning and end of a 10-min data collection. Gradual reduction in impedances during an experimental session affects the amplitude characteristics of the artifacts (blue and cyan traces in Fig. 6.4b). This is an example of one artifact (impedance change) affecting another artifact (ocular) in a continuous and gradual manner. Our adaptive method was able to identify the changes in skewness and sharpness of the artifacts themselves and clear them accordingly.

The robust sample adaptive formulation and the selective nature of the method make this framework a good candidate for complex tasks as motion artifact handling, in real time. We extended our linear ocular artifact mapping technique to a non-linear mapping for the motion artifact problem (Kilicarslan and Contreras-vidal 2019). We have discovered that the gravity compensated acceleration values of the head (i.e., measured via a forehead-mounted intertial measurement unit (IMU) sensor) allow for a non-linear projection to identify artifact components in each EEG sensor separately, and thus used as a reference signal for our implementation. We have used a second order Volterra Series representation and identified the kernel weights using our H∞ formulation. To target the harmonics of the motion artifact contamination, we have used a filter banked version of the reference signal.

Figure 6.5 shows the before and after gait locked events for a subject walking on a treadmill at speeds of 2.0 and 4.0 mph. Our algorithm handled the clear gait-locked structure of the artifacts (i.e., electrodes moving relative to the head due to walking causing cable pulling/tagging, etc.) and their harmonics. Both the signal power increase (lighter colors) and suppression (darker colors) caused by the artifacts were cleaned effectively. Figure 6.5 bottom plot shows two EEG spectra of different levels of artifact contamination. The power spectrum on the left is contaminated less compared to the spectra on the right for which the harmonics of the artifacts are dominant. Our method was able to adapt to both conditions without relying on any pre-statistical knowledge or pre-measurement of EEG data.

Fig. 6.5
figure 5

(top) Event Related Spectral Perturbations (ERSPs) showing gait locked motion artifacts averaged for all gait cycles within the session, for two walking speeds of 2 and 4 [mph]. Gait phases are marked as LTO/RTO: Left/Right Toe Off, LHC/RHC: Left/Right Heel Contact. Raw and cleaned ERSPs were compared. Clean ERSPs show no sign of artifacts and their harmonics. (bottom) two sets of EEG spectra showing multiple levels of contamination. Raw EEG spectra (blue) can exhibit a few artifactual peaks (left) or severe harmonics (right). The rest EEG spectra (yellow) was also shown for comparison. After cleaning the motion artifacts, no sign of contamination was observed (red). From (Kortte et al. 2007), with permission

Full size image

The real-time compatibility and generalizability of our adaptive filtering framework allows for the effective use of non-invasive BCI systems and dramatically expands the implementation type and application domains to other types of problems where signal denoising is desirable. Combined with our previous efforts of filtering ocular artifacts, the technique allows for a comprehensive adaptive filtering framework to increase the EEG SNR. We believe the implementation will benefit all neural measurement modalities, including studies discussing neural correlates of movement and other internal states, not necessarily of BCI focus.

4 Real-Time Implementation

4.1 Neural Decoders, Processing Pipeline, and Day-to-Day Performance Variability

Similar to the real-time compatibility of artifact processing methods, other EEG analyses and feature extraction tools need to be applicable for real-time implementations. Although the necessary calculation for a single processing step might be fast enough, the combined processing time, elapsed from the raw EEG measurements to the output of the decoder must be faster than the time between successive EEG samples. The violation of this rule would cause unpredictable behavior at best. If the feature extraction is based on windowed EEG data, e.g., features found on past 1 s of data, the minimum delay expected from the system response would also be 1 s. The overall calculation time of the neural decoder is therefore detemining the overall response time of the neuro-robotic system.

Many of the decoding accuracies reported in the literature are based on offline processing and evaluation of EEG data, which are modeled using advanced machine learning and classification tools. This step is also referred to as the training or calibration session, when followed by an online implementation. For the model training stage, many repetitions of the same task are executed by the user, and the corresponding EEG and exoskeleton data are collected. The idea behind having high number of repetitions is to collect statistically rich EEG data, and thus to be able to identify and later filter out task-independent neural activations. The EEG features that are calculated per task are then mapped to the robot states that are measured for the same task (i.e., robot’s state as walking vs standing/stop). The successful mapping is called the model for that specific user. In real-time implementations, this model is then evaluated for the measured EEG data, and the output is considered to be the user’s intended command to the exoskeleton. However, it should be noted that, as it stands, the real-time implementation of these offline generated models is not expected to work well, at least initially, for two main reasons; (1) the highly complex nature of the measured cortical signals; and (2) the user training. The first concerns with the dataset shifts (Quionero-Candela et al. 2009). Even when many trials have been executed by the user and the overall statistics is considered rich, the properties (e.g., distribution, mean, variance, etc.) of the extracted EEG features are usually different from the training data. This results in the wrong interpretation of the EEG data by the model, thus overall reduced decoding accuracies. As the sessions progress and significantly more data are collected, two major improvements can be observed; the statistics can get richer for the training algorithm to achieve a better generalization in decoding; and on the measurement level, EEG with better information content (stronger task-dependent data). The second is concerned with the user’s learning and cortical plasticity. As a way of coping with the low initial decoding performance levels, often a new EEG decoder is trained for each session.

As an examplary application of this framework, Fig. 6.6 shows an offline processing pipeline (Kilicarslan et al. 2013) and real-time implementation of a neural decoder applied to the REX lower-body exoskeleton system [REX®, Rex Bionics Inc], where the user imagines his/her leg moving as a mental task. This moving intent is then decoded in real time for BCI control or offline for further analysis and quantification of the changes in neural signals (assessment of user learning, brain plasticity, etc.). The dashed lines represent the offline data acquisition and model training phases, whereas the solid lines represent the real-time implementation. The only difference between the two is that instead of training the model (which requires heavy calculations and optimization), the real-time implementation only uses the model parameters generated during the training. Substituting the calculated parameters requires very little computational load. The idea behind the presented example is to collect the EEG data and identify a model for the user per session, within 7 min, and implement the real-time decoder by evaluating the model immediately after training. For the results that will be discussed below, this process was repeated for nine sessions spanning 3–4 weeks (longitudinal testing).

Fig. 6.6
figure 6

An example of processing steps for a closed-loop BCI implementation (NeuroREX (Kilicarslan et al. 2013)). The implementation seeks for fast implementation of neural decoders and longitudinal testing to assess performance and statistical variability of the data

Full size image

The offline validated decoding accuracies for an end user with SCI are shown in Fig. 6.7. There is a clear increasing trend in decoding accuracies, which suggests the user’s learning process, thus cortical plasticity. For each day in the plot, the accuracies were calculated with ten-fold cross-validation. All training settings were kept constant, and a new model was trained for each day. This indicates that the model was able to identify stronger correlations between the EEG features and the exoskeleton states (walk vs. stop) as sessions progress, and the task-relevant EEG data becomes stronger over time.

Fig. 6.7
figure 7

Offline training accuracies per session for an SCI user executing stop vs walk tasks using the NeuroRex closed-loop BCI system. Unpublished data

Full size image

Figure 6.8 shows the accuracies for an non-disabled subject. In contrast to Fig. 6.7, this figure shows the real-time decoding accuracies. The purpose of this real-time implementation is to assess the subject’s adaptation process to two cases; first, training a new model for each session (similar to the results reported in Fig. 6.7); and second, keeping the trained decoder fixed and implementing the last trained model for the last four sessions. In the first case, the data from all previous sessions were concatenated and used for training a new model. This model is then tested in real time for 12 trials. The completion time of each trial were used as the success metric (time-weighted task completion accuracy in Fig. 6.7). For the case 1, the decoding accuracies are low due to the previously mentioned dataset shifts and subject’s adaptation. For the second case, the model was fixed at session 5 and evaluated for the remainder of the experiment. Gradually increasing decoding accuracies for this case suggests user’s adaptation to the fixed model.

Fig. 6.8
figure 8

Real-time decoding accuracies per session for a non-disabled user executing stop vs. walk tasks using the NeuroRex closed-loop BCI system. Unpublished data

Full size image

As previously mentioned, longitudinal BCI implementations are mostly based on a new model training for each session to cope with the changes in data statistics and signal shifts. This implies mathematical model’s adaptation to the newly collected EEG data. However, it also requires the user’s adaptation to the new model dynamics, creating a complex loop of adaptations to changing dynamics. The adaptation reported after session 5 (Fig. 6.8) brings the question of whether or not this practice is suitable for all subjects. This is an active area of research and the presented results are not conclusive and not all subjects have the same level of adaptation and overall control. However, there is clear evidence that the EEG information content and users’ adaptation to the BMI implementation to exoskeletons can increase over time, suggesting cortical plasticity. It should also be noted that the exoskeleton REX (Rex Bionics, Auckland, New Zealand) used in the reported sessions exhibit large and variable time delays, especially in stopping from a walk task. The delayed exoskeleton response, even when the correct command was sent, brings additional difficulties for the users to associate their thought patterns to the exoskeleton states, thus makes the adaptation to the overall system exceedingly difficult.

5 Assessing Neural Dynamics and Optimization of Parameters

The previous section underlined some of the difficulties associated with the real-time implementation of BCI systems. The subject and/or model’s adaptation to the rehabilitation session is one of the key difficulties discussed. In all BCI implementations, the most important goal is to increase the information content in the EEG and developing models using advanced tools (i.e., machine learning frameworks, classifiers, etc.) that can capture the task-dependent information. EEG artifact cleaning, longitudinal use, and statistically rich data (i.e., using many trial and session data for training) were also discussed. An additional step towards data with reliable and rich information content relates to pinpointing the spatial locations of relevant neural dynamics on EEG sensor level, as well as the cortical source level. Namely, instead of using the EEG from all scalp locations and identifying relevant data in the model building stage, a prior analysis can be done investigating which sensor locations contain rich data and what relevant features are in the data per given task. Figure 6.9 summarizes the EEG sensor space analysis to pinpoint relevant electrode sites (Zhang et al. 2017). This multiple Kernel importance weight (a form of machine learning algorithm employed to find relevant information in highly complex multi-dimensional EEG data) study investigates which electrode locations contribute most to the decoding accuracies. Panel (a) shows the analysis for a study participant with SCI executing walk vs stop tasks with the REX exoskeleton for nine sessions. The analysis is complementary to the offline analysis summarized in Fig. 6.7. The electrode information at region of interest (ROI) 4 and ROI 5 are found to be most relevant for the successful decoding. The topographical plot of the important regions for the last session (when the subject has the most experience with the exoskeleton) is shown in the left plot of panel (b). The right plot on panel (b) shows the same analysis for a non-disabled study participant.

Fig. 6.9
figure 9

Framework for determining the information-rich EEG sensor locations. Adapted, with permission, from (Hargrove et al. 2013). (a) summarizes the results for a study participant with SCI. The increasing decoding accuracies per session suggest increase in information content in the EEG, due to neural plasticity. (b) shows the topographical plots of the importance levels for each EEG sensor for session-9, for both the SCI and a non-disabled subject

Full size image

An additional analysis can be done on the source level, either by utilizing functional magnetic resonance imaging (fMRI) or source analysis using the projections from the EEG electrode data. Figure 6.10 shows the fMRI identified sources for a subject (complementary to Fig. 6.8). These scans were taken after the ninth and last session of the exoskeleton usage, while the subject executed the same mental task but this time in the scanner, watching first-person view of their exoskeleton usage. The hotter color scale points to increase activity while imagining the walk compared to stop, and the colder color scale refer to the increased activity of the lack of walking imagery compared to the existence of it. The green colored regions point to the active areas that are common to both conditions.

Fig. 6.10
figure 10

fMRI contrast analysis of a non-disabled subject executing the same mental imagery of stop vs walk as in the NeuroREX sessions. The scans were taken after the ninth session of the subject’s NeuroREX usage. Unpublished data

Full size image

The increased activations in motor and somatosensory cortices during walking imagery are visible. The increased activation in visual cortex can be explained by the subject’s fixation to a visual cue (cross) in the video which indicates the start of the stopping task. The key conclusion that can be drawn from these analyses would be projecting the source activation patterns to the EEG sensor domain and pinpointing the locations of the information-rich EEG electrodes. Combined with the results from the sensor domain analysis, we hope to be able to provide information-rich data to the classifier. This, in turn, would result in a better generalization capability of the decoder for real-time implementations and would reduce the subject’s adaptation time to the closed-loop decoder.

6 Conclusion

Despite recent advancements in robotic technologies and neural control modalities for rehabilitation, there are still many unknowns for their effective integration in clinical rehabilitation programs and high-performance usage. This multi-disciplinary integration requires careful considerations from hardware and software engineers, clinicians, and machine learning/signal processing experts. We have provided solutions and suggestions to some major difficulties that applies to many types of neuro-robotic systems. Wide-scale applications and deployment of neuro-robotics technology could benefit from further investigation of multiple aspect from all disciplines, which include:

From the engineer’s perspective

  • adapting the neural decoders per session (to account for shifts in internal states of the user and external environmental factors), and its potential adverse effects,

  • improving the robot dynamics, balance control, and time lags,

  • providing a generic interfacing data protocol and synchronization capabilities to third party systems (for multi-modal recordings),

  • identifying and adopting engineering metrics to facilitate comparison across systems.

From the clinician’s perspective

  • Inclusion/exclusion criteria for the patients

    • determining risk profiles of potential users,

    • history of health complications,

    • identifying and adopting metrics that assess clinical improvements compared to classic rehabilitation routines.

Both the clinician’s and the end-user’s perspective

  • Feedback about the feasibility and therapeutic effects and side effects

    • Usability,

    • Independence,

    • Training time,

    • Bladder and bowel function,

    • Spasm intensity,

    • Circulatory function improvements,

    • Skin conditions,

    • Overall well-being.

As discussed before, the combined effort from multiple disciplines can provide the designers and end-users a comprehensive overview of the possibilities and limitation of the current technology.

We have also discussed the user adaptation time to any neuro-technology and the fact that the information content of the neural data can dramatically increase with new data registered. Additionally, the improvements in the user’s learning and adaptation process were described. Another effective practice for neuro-robotic implementation is to provide the users prior access to the robotic system in question, before it is controlled via the neural interfaces. This process not only helps the clinicians to assess the feasibility and safety of the exoskeleton in a specific user, but also allows the user to gauge the dynamics and operating conditions of the exoskeleton. The effective use of a BCI based neuro-robotic system requires some level of focus while the user executes mental imagery of their limb movement. Early access to the robotic technology will help in this regard, as the users would become confindent in the use of the mechanical system and therefore be able to direct their focus on the mental and functional task dueing the training session.