Keywords

1 Introduction

Many kinds of driving assistant system have been developed for a driver’s convenience and safety. Especially, the braking assistant technology is one of most important parts to assure a driver’s safety. There were neurophysiological studies concerned with the use of brain signals for enhancing driving assistance systems. However, most of the studies focused on measuring and detecting drivers’ physical conditions and mental states such as decreased concentration [1] or sleepiness [2–4] during a monotonous drive. Other studies focused on controlling the vehicle based on analysis of brain signals [5–11].

Footnote 1 In a recent study, upcoming emergency situations during simulated driving were detected using event-related potentials (ERPs) [12]. This study demonstrated that neurophysiological correlates of emergency braking occur about 130 ms earlier than the corresponding behavioral responses related to the actual braking. However, it is impossible to prove the feasibility of a driving assistant system based on a brain-computer interface in the real world, since the participants were exposed to a very reduced set of driving situations.

In this article, the feasibility of the driving assistant system is investigated based on the experiment with more diversified simulated driving situations than the previous study [12]. In our experiment, the participants were exposed to traffic situations including three kinds of emergency situations while they drove a virtual vehicle: the sudden stop of a leading vehicle, the sudden cutting-in of a vehicle from a neighboring lane, and the unexpected appearance of a pedestrian. In addition, there were driver’s voluntary braking situations without any kinds of emergency stimulus. Electroencephalogram (EEG) signals were measured during the experiment and the neural responses of braking intention in various kinds of traffic situations were analyzed to investigate the differences of neural signals in emergency and non-emergency situations.

Three kinds of signal components were combined and used for the detection of driver’s braking intention in this study. One signal component is the readiness potential (RP), a preparatory (i.e., pre-movement) component that indexes movement intention [13], ERPs (i.e., visual evoked potentials and P300 components), and event-related desynchronization (ERD) to distinguish diverse driving situations such as sharp braking.

The class-discriminability of univariate ERP features is investigated after the description of experimental paradigm and approach for signal analysis. After that, we assess and compare the classification performances of the proposed feature combination method and the ERP feature based on the area under the specificity-sensitivity curve (AUC). The findings and experimental results are summarized with discussions at the end of this chapter.

2 Material and Methods

The following section follows closely a prior published paper by the authors [39].

2.1 Participants and Experimental Setup

Fifteen healthy individuals (all male and right-handed, age 27.1 ± 1.7 years) participated in this study. All participants had a valid driver’s license and had driven for more than 3 years with no car-accident history. All had normal or corrected-to-normal vision. None of the participants had a previous history of psychiatric, neurological, or other diseases that might otherwise affect the experimental results. The experimental procedures were explained to each participant. Written informed consent was obtained from all participants before the experiment. The participants received a monetary reimbursement for their participation after the completion of the experiments. The participants were seated in a driving simulator cockpit (made by R.CRAFT in Korea) with fastened seat belts (the experimental apparatus is shown in Fig. 2.1).

Fig. 2.1
figure 1

The experimental apparatus and environment

Full size image

The virtual driving environment displayed on the screen was developed using the Unity 3D (Unity Technologies, USA) cross-platform game engine. This environment resembled an urban neighborhood without traffic lights and included autonomous (computer-controlled) vehicles as well as a vehicle to be steered by the participant. There was a six-lane road with many vehicles driving controlled by a simulator.

2.2 Experimental Paradigm

The participants were asked to drive a virtual vehicle using the accelerator and brake pedals and the steering wheel during one experimental session that lasted about 2 h. They were instructed to drive freely without getting into an accident, and to perform immediate braking to avoid crashes if necessary. The maximum speed of a vehicle was 120 km/h. We defined three kinds of braking situations. First, if an emergency situation occurred, the participants were instructed to press the brake pedal sharply. We defined this situation as sharp braking. Second, when the participant performed spontaneous braking to decrease a vehicle’s speed, the vehicle decelerated softly (i.e., gradually). This situation was defined as soft braking. Finally, in many situations, the participants did not need to decrease the speed of their vehicle (i.e., normal driving). For all of the stimuli, the inter-stimulus interval (ISI) was between 4 and 18 s, and drawn randomly from a uniform distribution (see Fig. 2.2).

Fig. 2.2
figure 2

Timing of the experimental paradigm

Full size image

2.2.1 Sharp Braking Condition

There were three kinds of stimuli inducing sharp braking (emergency) situations. 1) The sharp braking by brake light condition: when the vehicle in front of the participant (lead vehicle) abruptly decelerated. A leading vehicle’s brake light flashing was defined as the stimulus onset in this condition (see Fig. 2.3a). 2) The sharp braking by cutting-in condition: when a vehicle on the neighboring lane (side vehicle) abruptly cut in front of the participant’s vehicle. The moment when the side vehicle came across the lane was defined as the stimulus onset (see Fig. 2.3b). 3) The sharp braking in the pedestrian condition: a pedestrian moved quickly toward the participants’ vehicle from the side. The moment when the pedestrian left the sidewalk was defined as the stimulus onset (see Fig. 2.3c).

Fig. 2.3
figure 3

Stimuli related to emergencies (a) Sharp braking by the leading vehicle’s brake light, (b) Sharp braking by cutting-in, (c) Sharp braking by a pedestrian

Full size image

2.2.2 Soft Braking Condition

The soft braking condition was defined based on spontaneous braking in the absence of any stimulus. To slow down the vehicle, the participants spontaneously pressed the brake pedal. In this soft braking condition, the moment when the participant pressed the brake pedal was defined as the response onset (see Fig. 2.4a).

Fig. 2.4
figure 4

Stimuli not related to emergencies (a) Soft braking or normal driving (no stimulus), (b) No braking by brake light, (c) No braking with brake light of neighboring vehicle

Full size image

2.2.3 No-Braking Condition

The no-braking condition comprised three kinds of traffic situations. 1) One was normal driving: the participants just focused on driving, and no stimulus was given (see Fig. 2.4a). 2) The no braking by brake light condition: when a leading vehicle abruptly decelerated far away from a participant’s car. In this condition, the participants did not have to press the brake pedal. The stimulus onset was defined as the moment in which the lead vehicle’s brake light flashed (see Fig. 2.4b). 3) The no braking with brake light of neighboring vehicle condition: when the leading vehicle on a neighboring lane abruptly decelerated. In this condition, participants did not have to press the brake pedal. The side vehicle’s brake light flashing was defined as the stimulus onset in this condition (see Fig. 2.4c).

2.3 Data Acquisition and Feature Extraction

The EEG signals were recorded using a multi-channel EEG acquisition system from 64 scalp sites based on the modified International 10–20 system [14]. We used Ag/AgCl sensors mounted on a cap (actiCAP, Brain Products, Germany). The ground and reference electrodes were located on scalp position AFz and the nose, respectively. The sampling rate was 1000 Hz throughout the experiments. The low and high cut-off frequencies were 0.1 and 250 Hz, respectively.

Electromyogram (EMG) signals were also acquired using a unipolar montage at the tibialis anterior muscle. The impedance of the EEG and EMG electrodes were kept below 10 kΩ. The EEG and EMG data were amplified and digitized using BrainAmp hardware (Brain Products, Germany).

Brake and gas pedal deflection markers were acquired at a 50 Hz sampling rate provided by the Unity 3D software. The time points of the braking response were defined based on the first noticeable brake pedal deflection that exceeded the jitter noise level.

The EEG signals were low-pass filtered at 45 Hz, and the EMG signals were band-pass filtered between 15 and 90 Hz. To remove line noise, we further applied a second-order digital notch filter at 60 Hz to the EMG signals.

The sampling rates of the physiological (EEG and EMG data) and mechanical (brake and gas pedal data) channels were down-sampled or up-sampled to 200 Hz for synchronization.

Segmentation was performed with 1500 ms length of epoch. Therefore, each epoch consists of 301 lengths of data points. After that, three different kinds of pre-processing and feature extraction methods were considered, corresponding to three different types of features (ERP, RP, and ERD) to be extracted from each epoch (see Fig. 2.5). The selection of time intervals (not necessarily of equal length) determined heuristically for each channel [15] (Ten-time intervals for ERP features and three-time intervals for RP features). These time intervals were selected only using training data. The same intervals were used for the test data. Each feature was computed for all electrodes (640 dimension of ERP features and 192 dimensions of RP features). Moreover, there are 6 ERD features for all channels. Thus, the combined feature vector has 838 dimensions (640 for ERP, 192 for RP, and 6 for ERD). The combination features were normalized by subtracting their empirical means and dividing with their empirical standard deviations as estimated on the training sets to rescale the three kinds of features after concatenation of features. The test data sets were also normalized in the same way by subtracting the empirical means and dividing with the empirical standard deviations as estimated on the training data sets.

Fig. 2.5
figure 5

Schematic flow of the data analysis process

Full size image

2.4 Event-Related Potentials and Area Under the Curve Analysis

The arithmetic mean of the extracted epochs of all 15 participants was computed to obtain grand-average ERP signals. Additionally, the discriminability of univariate (single-timepoint single-sensor) features with respect to the three pre-defined classes was investigated using the area under the curve (AUC) [16]. This analysis was conducted separately for each pair of classes. The AUC is symmetric around 0.5, where scores greater than 0.5 indicate that a feature has higher values in class 1 than in class 2 and scores smaller than 0.5 indicate the opposite (smaller values in class 1) [16, 17]. The arithmetic mean of the AUC scores across the participants was calculated to obtain grand-average AUC scores.

2.5 Classification

As in [12], we evaluated the extent to which different feature modalities contribute to the overall decoding performance. Classifiers were trained on four kinds of modalities. These modalities were EEG (feature combination), EEG (only ERP features), EMG, and BrakePedal, which denotes the driver’s actual brake pedal inputs. The first half of the epochs were used as the training set, and the second half were used as the test set. The entire analysis process including pre-processing is shown in Fig. 2.5.

The class discriminability of optimized combinations of spatio-temporal features was investigated using the regularized linear discriminant analysis (RLDA) classifier [18–21]. For regularization, the automatic shrinkage technique [22–25] was adopted. We had three classes of driving situations. For each pair of classes, we calculated the AUC scores of the RLDA outputs on the test set.

2.6 Statistical Testing

Whether a given AUC score was significantly different from 0.5 (that is, chance level) on the population level was assessed by means of two-sided Wilcoxon signed rank test [26]. To assess whether two scores AUC1 and AUC2 were significantly different from each other, the difference AUC1-AUC2 was also tested for being nonzero using the same two-sided Wilcoxon signed rank test. Bonferroni-correction was implemented to obtain reliable p-values [27]. The correction factor was 301 (time instants) × 64 (electrodes) = 19,264 in the ERP analysis. For other channels, we used a correction factor of 301 (time instants). Features whose p-value was smaller than 0.05 are considered statistically significant.

3 Results

The following section follows closely a prior published paper by the authors [39].

3.1 ERP Analysis Related to Braking Conditions

Each of the three classes of driving situations induced a specific cascade of brain activities representing low- and high-level processing of the (visual) stimulus as well as motor preparation and execution. The same number of braking situations was induced for all braking types and subjects. However, after filtering and artifact rejection, the number of trials used in the analysis of neural correlates varied. On average, 57.1 ± 12.3 trials related to sudden stops of leading vehicles, 60.5 ± 8.1 trials related to cutting-in of leading vehicles, and 44.7 ± 5.6 trials related to sudden appearance of pedestrians were used in the data analysis. Half of the trials were used for training the classifier, and the remaining trials were used to evaluate the decoding performance.

We investigate the spatio-temporal ERP sequences reflecting the class-discriminative brain processes. Figure 2.6 shows topographical maps of grand average AUC scores in five subsequent 160 ms long time intervals. Figure 2.6a shows the AUC scores related to the difference between sharp braking and no braking. The feature value of sharp braking is higher than that of no braking (AUC > 0.5) in the time interval between 320 and 800 ms post-stimulus in parietal areas. The AUC score is maximal in the time interval between 480 and 640 ms (z > 6.8, p ≈ 0). The electrode having the highest AUC score (0.58) between 480 and 640 ms post-stimulus is Pz. On the other hand, the higher feature value of no braking than that of sharp braking (AUC < 0.5) is observed in the time interval between 160 and 320 ms in lateralized occipital areas (z < −6.8, p ≈ 0), and in the time interval between 320 and 800 ms in central areas (z < −6.8, p ≈ 0). The electrode Cz shows the lowest AUC score (0.40) between 320 and 480 ms post-stimulus.

Fig. 2.6
figure 6

Topographical maps of grand-average AUC scores calculated from average ERPs in five temporal intervals. (a) Sharp braking vs. no braking, (b) Soft braking vs. no braking, (c) Sharp braking vs. soft braking

Full size image

Figure 2.6b shows the AUC scores related to the difference between soft braking and no braking. The higher feature value of no braking than that of soft braking is observed over the entire time interval in central areas. The lowest AUC score (0.43) is observed in the electrode Cz between 320 and 480 ms post-stimulus. The electrode TP10 has the highest AUC score (0.53) between 640 and 800 ms post-stimulus. Figure 2.6c depicts the AUC scores related to the difference between sharp braking and soft braking. Here, the highest feature value of sharp braking is observed in parietal/occipital areas in the time interval between 480 and 640 ms (z > 6.8, p ≈ 0), while a highest feature value of soft braking around electrode Cz is observed in the time interval between 320 and 640 ms (z < −6.4, p ≈ 0). The electrode P7 has the lowest AUC score (0.44) between 160 and 320 ms post-stimulus while the highest AUC score (0.59) is observed in electrode Pz between 480 and 640 ms post-stimulus. Thus, sharp braking elicits stronger feature values than soft braking and is moreover characterized by the additional presence of visual-evoked potentials and a P300 component. Note that it is impossible to distinguish emergency braking situations from normal driving (e.g., no-braking events) before the stimulus. Therefore, the classification before stimulus onset must be at chance level as indicated by AUC scores of 0.5 (see Fig. 2.6).

3.2 Comparison of Classification Results Based on ERP Features and a Novel Feature Combination

The results of the classification analyzes using multivariate features are shown in Fig. 2.7. These features were extracted from stimulus-locked segments. The classification performance was measured in terms of AUC scores achieved by the outputs of RLDA classifiers on test data. The classifiers were trained to distinguish two of the three classes. Thus, there were three different class combinations to consider.

Fig. 2.7
figure 7

Classification performances based on ERP feature and novel feature combination. The distribution of reaction times for the sharp braking condition is depicted by boxplots. The areas of shaded color represent standard errors of the mean (SEM) of the AUC scores

Full size image

Figure 2.7 provides a time-resolved assessment of the classification performance, where the AUC score at each time point represents the accuracy achievable using the preceding 1500 ms long segment of data. The boxplots in Fig. 2.7 shows the distribution of reaction times (defined as the first above-threshold brake pedal deflection) for the sharp braking condition. Note that for the soft braking condition artificial stimulus onsets were sampled from the same distribution.

The classification results with respect to distinguishing sharp braking and no braking based on ERP features are similar to the results previously achieved by [12] (see Fig. 2.7a). The AUC scores of both single features and the proposed feature combination based on EEG exceed 0.6 after 260 ms post-stimulus. In addition, the score of both EMG and brake signal exceed 0.6 after 320 and 580 ms post-stimulus, respectively. The AUC of the EEG using the novel feature combination exceeds a value of 0.9 by 80 ms ahead of using ERP features alone.

The novel feature combination shows significantly better performance compared to the ERP features from 640 to 1200 ms post-stimulus (p < 0.05, the largest difference at 960 ms, zmin = 5.4, p ≈ 10−2).

The performance with respect to distinguishing the soft braking and no braking conditions is presented in Fig. 2.7b. For EEG, the scores using the novel feature combination are considerably higher than the scores obtained from using ERP features only. For ERP only features, the AUC exceed 0.6 at 440 ms post-stimulus, while for the novel feature combination it is 340 ms post-stimulus. In addition, the novel feature combination achieves significantly better performance than ERP-only features from 760 to 1200 ms post-stimulus (p < 0.05, the largest difference at 840 ms, zmin = 5.8, p ≈ 10−3).

Finally, the performance in classifying sharp braking and soft braking based on ERP features is dramatically lower compared to using EMG features in the entire time interval considered. On the other hand, the performance is improved by means of the proposed combination of EEG features, although the achieved scores are still lower than those obtained from EMG. The novel feature combination achieves an AUC score of 0.6 by 120 ms earlier than the corresponding ERP-only features. It is significantly better than the ERP features alone in the interval from 380 to 440 ms post-stimulus (significant with p < 0.05, the largest difference at 420 ms, zmin = 5.6, p ≈ 10−3). These results are presented in Fig. 2.7c. These results confirm that our novel feature combination is more informative for the detection of drivers’ braking intentions than ERP features alone.

4 Discussion and Conclusion

A positive signal similar to the typical P300 component is observed in a broad parieto-occipital region for all kinds of visual stimulus types in the analysis results of spatio-temporal ERP pattern. This neurophysiological properties of the ERPs are caused by our experimental paradigm similar to the classical oddball paradigm [28] in a respect that the emergency stimuli (oddball) are presented randomly during normal driving (normal ball).

The neurophysiological responses to the two kinds of braking (soft and sharp) show the different amplitude of positivity in the parieto-occipital region and this difference was helpful to distinguish these two kinds of situations.

There are three kinds of stimuli in case of sharp braking condition and the ERP patterns across the scalp evoked by these stimuli were different. However, we considered these patterns as the same class because these patterns evoked by pseudo-oddball paradigm and we could observe a positive potential in the parieto-occipital region [29] for all kinds of stimuli. In the end, we were able to detect the driver’s braking intention regardless of what kind of traffic situations occurred.

A negative signal in a central region (especially at the Cz electrode) evoked by the planning processes in the motor system related to the act for pushing the gas or brake pedal (i.e., the readiness potential) [30, 31] in all braking situations. Different negative deflection was evoked by reactive and spontaneous movements at the foot area of the motor cortex during and prior to muscle movement [6, 32, 33]. In addition, the reactive movement and spontaneous movement had a different start time of the negative deflection in the central region. The reactive movement was similar to cue-based motor execution and corresponding to sharp braking. On the other hand, the spontaneous movement was similar to self-paced motor execution and corresponding to soft braking. The start point of the pre-motion negative deflection in the central motor area related to a reactive movement was later than the start point of the spontaneous movement, in line with [13, 34, 35]. Thus, as aforementioned, we used the readiness potential as a feature because it provided important movement-related information.

There is about 150 ms of the time difference between ERD starts and EMG onset (i.e., ERD is faster than EMG) [36, 37]. The ERD was observed prior to the depression of the brake pedal (i.e., right foot movement from the gas to the brake pedal). Interestingly, the self-initiated movement (i.e., soft braking) and the movement triggered by an external stimulus (i.e., sharp braking) differed in aspects (i.e., magnitude and slope) of the band power. Thus, the ERD information related to foot movement for braking was also used as a feature.

The prediction performance of the sharp braking and soft braking conditions are assessed in time series based on AUC score. The AUC score of EEG increased faster than that of EMG. Although the peak AUC score of EMG is lower than that observed in a previous study [12], the trends of results are similar to a previous study.

The feasibility of smart vehicle technology for detection of driver’s braking intention was verified in this study by detecting the participants’ braking intention robustly in all of the simulated traffic situations. Especially, the prediction performance based on the novel feature combination is better than the prediction performance based on ERP features alone reported in a previous study [12] although the classification performance for sharp braking and soft braking based on EEG features was lower than the performance based on the EMG. Some important environmental stimuli (e.g., vehicle vibration, auditory stimuli) were omitted in our setting. However, it is complemented by the work of [3, 38], which shows that results identical or better to those of [12] can be obtained in a real-world setting.

Together, our study and [38] provide a converging evidence suggesting that smart vehicles that have an automatic braking assistance system integrating neurophysiological responses could be developed in practice. The methods and results sections have been taken from an own prior publication after summarization. For further details on the experiment and analysis, see [39].