Keywords

1 Introduction

Brain-computer interfaces (BCIs) represent an essential field in human-computer interactions. BCIs provide communication between a human brain and a computer or external device analyzing electrophysiological signals measured from the brain [1]. This technology has already proven its great potential in various assessment, communication, and therapy methods [2,3,4]. Various approaches for measuring brain activities (brain signal acquisition technologies) can be applied to gather the desired information [5]. For non-invasive measurements, Electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) are the state-of-the-art acquisition methods in BCIs [6,7,8]. EEG systems screen brain activities via measuring voltage potentials on the scalp resulting from the underlying neural activities. Due to its high temporal resolution, EEG systems come into use, especially for time crucial applications. Though, EEG has the limitation in its spatial resolution. The fNIRS uses optical densities to measure the oxygenated (HBO) and deoxygenated (HBR) hemoglobin concentrations. Active brain regions require more generous blood flow to fulfill increased energy demand, also known as neurovascular coupling. Based on this information, the cortical activity can indirectly be evaluated [9]. Using both approaches (EEG and fNIRS) as a hybrid method has enhanced performance in several BCIs [6,7,8, 10].

The identification and training in cognitive impairments (e.g., Alzheimer’s disease, dementia) is an enormously increasing area of interest [11, 12]. The potential to process information and regulate actions and thoughts depending on the current task is called cognitive control or executive control (EC) [13]. Underlying cognitive adaption and filtering information on the importance and suppression of non-task-related and irrelevant content is vital for human decision-making [14]. EC processes and sub-processes are addressed while executing the Stroop-test. This experiment aims to trigger the Stroop effect, which reflects a mental conflict (unexpected mental load for a familiar task) [13]. Cognitive experiments, such as the Stroop-test, evoke brain activity in the frontal lobe area [15, 16].

The prefrontal cortex (PFC) plays a significant role when investigating memory and decision-making behaviors [17]. However, only a few studies have shown the influence of hemodynamic response in connection with EEG examining frontal lobe activities.

This study analyzed neurophysiological findings connected to cognitive control processes using Electroencephalography and functional infrared spectroscopy. As the basis for discussion, the results of prior published data from Motor-Imagery experiments using this hybrid EEG-fNIRS approach were exploited [18].

2 Material and Methods

The Stroop-test resulted in brain activation, which was measured by the g.NAUTILUS fNIRS (g.tec medical engineering GmbH) device and subsequently classified. The experiment is a binary classification problem and aims to differentiate between rest and shuffled Stroop-test (frontal lobe activation).

2.1 Data Acquisition

The g.NAUTILUS fNIRS system includes 8 transmitters and 2 receivers, resulting in 2 × 4 fNIRS channels for measuring the hemodynamic responses. The EEG signals were acquired using 16 independent electrodes of the hardware. A detailed optode and electrode setup is depicted in Fig. 1. Overall, 6 healthy subjects without known pre-conditions participated in the experiments.

Fig. 1.
figure 1

The 16 EEG electrodes are placed over the frontal cortex. FPz, AFz, Fz, F1, F2, F3, F4, FCz, C1, C2, AF5h, AF6h, AFF7h, AFF8h, FFC1h and FFC2h. The 8 fNIRS transmitters are approximately placed at AFF4h, F6, AF8, AFp4, AFF3h, F5, AF7, and AFp3. The two fNIRS receivers at AF7 and AF8.

Full size image

2.2 Paradigm

The paradigm's main sequence can be seen in Fig. 2 (pre-rest, instruction, task, rest, and post-rest). This sequence is the same for both the calibration and evaluation phase. Each phase consists of 40 trials. The two-folded experiment is training classifiers and common spatial pattern (CSP) filters in the calibration phase and subsequently performing online classification in the evaluation phase. The algorithm utilizes both EEG and fNIRS signals.

Fig. 2.
figure 2

The sequence of the experiment used in calibration as well as in the evaluation phase.

Full size image

First, the Instruction window notified of the incoming task, which the subject then had to perform. The task itself was to read/imagine as many colors as possible within the 10 s task window. Overall a grid of 10x10 words was displayed. Instruction to relax was given as a control task. Control- and assorted Stroop-tests were sequenced in a stratified and randomized way.

2.3 EEG Processing

Signal processing was done in MATLAB (The Mathworks, Inc., Massachusetts, USA) and g.HIsys High-Speed Online Processing for Simulink (g.tec medical engineering GmbH). EEG processing involves pre-processing, CSP filtering, and feature extraction.

The EEG signals are bandpass filtered (6th order Butterworth filter, 8–30 Hz) and notch filtered (2nd order, 48–52 Hz) to set the focus on the \({\upalpha \mathord{\left/ {\vphantom {\upalpha \upmu }} \right. \kern-\nulldelimiterspace}\upmu }\) and \(\upbeta \) band and remove interferences of the power line.

Further on, to maximize the weight of the channels that contain essential information, spatial filtering was performed adopting a CSP filter [18, 19]. The two first and last filters \(\left( {Z = \left[ {w_{1} , w_{2} , w_{15} , w_{16} } \right]} \right)\) were applied to the pre-processed discrete EEG signal \(x^{{\prime }} \left[ k \right] \in {\mathbb{R}}^{4}\):

$$ x^{{\prime }} \left[ k \right] = Z^{T} x\left[ k \right], x^{{\prime }} \left[ k \right] \in {\mathbb{R}}^{4} $$
(1)

Finally, features were extracted by calculating the variance of the CSP filtered data. As this information is proportional to the band-power and the band-power is linked to cognitive activation, this is a suitable approach to gather information about the impact of the conducted Stroop-test [20]. From the four surrogate channels, the log variance was calculated and normalized over all channels:

$$ y\left[ k \right] = \log \left( {\frac{{var\left( {x_{i}^{\prime } \left[ k \right]} \right)}}{{\sum\nolimits_{i = 1}^{4} {var\left( {x_{i}^{\prime } \left[ k \right]} \right)} }}} \right) $$
(2)

2.4 fNIRS Processing

After removing artifacts based on the raw optical densities (ODs), the relative HBO and HBR concentrations from the acquired OD were determined. Signals were low pass filtered at 0.3 Hz with a 4th order Butterworth filter to remove pulsatile fluctuations. Power line noise was suppressed using a 2nd order notch filter. The concentration changes for HBO and HBR were calculated by applying a baseline correction that used the averaged signal (taken from the pre-rest phase) subtracted from the subsequent concentrations.

A rolling window \(X^{{\prime }} = \, \in {\mathbb{R}}^{NxT}\) was used for feature extraction, where N represents the number of channels and T = 30 the number of sampled points. Each sample \(x^{{\prime }} \left[ k \right]\) was consequently represented by the feature vector \(y^{{\prime }} \left[ k \right] \in {\mathbb{R}}^{{\left( {N7} \right)}}\). Two features, mean and slope, proved to be an optimal trade-off between accuracy and computational effort. The two chosen features were extracted from the pre-processed signals and passed to the classifier. The channel-wise mean is given as:

$$ mean\left( {X^{{\prime }} } \right) = \frac{1}{T} \sum\nolimits_{k = 1}^{T} {x\left[ k \right]^{T} } $$
(3)

The channel-wise slope is given as:

$$ slope\left( {X^{{\prime }} } \right) = { }\frac{1}{T} \left( {x\left[ T \right]^{T} - x\left[ 1 \right]^{T} } \right). $$
(4)

2.5 Classification

Linear discriminant analysis (LDA) with Fischer's criteria was utilized to classify the EEG and fNIRS features, respectively [21]. The final prediction was performed by a meta-classifier using the scores of the individual classifiers.

2.6 Analysis

Performance analysis was done via trial-wise checking for correct classified samples. The classification accuracy was set to be the accumulated and averaged value for each time point. All three methods (EEG, HBO, and HBR) and the combination of the three classifiers (COMBINED) were analyzed this way. Classifiers with the best performance in the offline analysis were adopted for online classification in the evaluation phase.

3 Results

3.1 CSP Analysis

A weighting for channels in the frontal lobe, as suggested by w15 and w16, can be extracted from the CSP analysis (see Fig. 3). With the available CSP weights, four surrogate channels could be calculated.

Fig. 3.
figure 3

Visualization of w1, w2, w15, and w16 from the CSP analysis.

Full size image

3.2 Offline Analysis

The time-variant 10 × 10 fold cross-validation (CV) was utilized for the offline classification performance analysis for calibration and, additionally, in retrospect, with the evaluation data. A considerable hemodynamic response (HR) was present during the Stroop-test, as seen in the offline analysis. The EEG-based classification delay was considerably lower than the one of HBO and HBR (Fig. 4). Results from the calibration and evaluation phase seemed to be comparable.

Fig. 4.
figure 4

Calibration and evaluation phase accuracies calculated from a 10x10 fold CV. Mean accuracy and standard deviation over all subjects and the real chance level (dashed line at \(\upalpha \) = 0.05) are additionally plotted.

Full size image

3.3 Online Analysis

The online performance results are the time-variant average online accuracies from the evaluation run. Table 1 shows the maximum achieved accuracies during the 10 s trial window with the relative appearance time. The maximum mean accuracy-based classification for the EEG was comparatively late with 9.3 s. As this is the global maximum, this might be misleading.

Table 1. The achieved maximum classification Accuracy (Acc) in percent with their relative timing (seconds) after the task onset from the Stroop-test. Suppose the accuracy did not exceed the chance level, the max. accuracy was considered to be 50%. Only maximum accuracies during the task window are shown.
Full size table

As seen in Fig. 5, the EEG accuracy rose clearly within the first 2 s of the task. On average, a maximum accuracy of 81.0% was achieved 6.2 s after the task onset. Further, the task window was divided into 10 slots, 1 s each. At 7–8 s after the task onset the following maximum accuracies (respective standard deviation) were achieved: Average = 83.6% (±12.3%), Class0 = 86.1% (±19.7%) and Class1 = 81.0% (±5.9%). Class1 in this case represents the Stroop activation and Class0 resting-state activity.

Fig. 5.
figure 5

Time variant accuracies from six subjects with the respective mean accuracy over all subjects and the real chance level (dashed line at α = 0.05). The blue area represents the instruction window of 2 s (4 s–6 s). The red-marked field represents the 10 s task window (6 s–16 s) while performing the Stroop-test. (Color figure online)

Full size image

4 Discussion and Conclusion

Prior research has already exploited the Stroop test's influence on an increased cortex activity, measured with fNIRS [23]. Hybrid EEG-fNIRS systems have also been employed to study brain activation during mental tasks [24, 25]. However, combining hybrid online-BCI systems using the Stroop task as a mental activation method has not been pointed out in the literature. The Stroop experiment's primary motivation was to investigate the frontal lobe activation, compared to the resting state, and to what extent this activity can be assessed via a hybrid EEG-fNIRS system. Equally, methods such as mental arithmetic or simple counting could be used to provoke frontal lobe activation.

The results of the CSP analysis emphasize the activation of the prefrontal- and motor cortex. One has to consider that eye movement artifacts during the word reading might also cause cortical activity. Compared to the one of the initially mentioned MI experiment [18], a distinct separation was less visible in the hereby present feature analysis. However, considering the temporal distribution, a similar onset was detectable. Overall, the results of both runs from the calibration and evaluation phases were comparable. The respective accuracy values' actual trajectories clearly showed that the EEG-based classifier had higher responsiveness than the HBO/HBR classifiers (see Fig. 4).

Using the 10 × 10 fold cross-validation for offline analysis turned out to be a suitable method for evaluating the signal quality and determining discrepancy from the different signal types (EEG, HBO and HBR). As the offline analysis has its limitations (e.g. overfitting), online analysis is crucial for the performance improvement and was seen as the assessment’s final aim.

Despite many advantages, such as good usability, fNIRS has its limitations in a delayed response. As the temporal resolution is the major advantage of EEG systems, combining these systems shows to be superior to applying only one method. In this study, investigating cognitive control processes using these combined methods shows clear benefits.