Keywords

1 Introduction

Steady-State Visual Evoked Potentials (SSVEPs) are normal brain responses to visual stimuli, which are elicited in the visual cortex [5]. In SSVEP-based Brain-Computer Interfaces (BCIs), there are several stimuli flashing with different frequencies. By fixating on such a target, continuous brain responses are elicited in the brain, which correspond to the frequency of the stimulus. The frequency as well as its harmonics can be detected in the measured brain waves. The brain signals are usually recorded via Electroencephalography (EEG) and the BCI can interpret these brain responses online. This way, the attended target can be determined and the associated command is executed. Thus, the SSVEP paradigm allows hand-free communication, which can be useful in the development of various assistive technologies or as a method of control.

The speed and accuracy of the system are essential for developing a practical and user-friendly BCI. Great improvements have been made in terms of speed due to new signal classification algorithms. For example Nakanishi et al. [7] achieved an Information Transfer Rate (ITR) of 325.33 bpm (bit per minute) by utilizing spatial filtering, filter-banks, and ensemble-based classification.

The highest ITR values have been achieved when stimulus presentation was used with multiple targets on standard screens [7]. As the frame-based display of a standard monitor does not easily allow for many different frequency stimuli [3], some form of frame-based frequency approximation method is typically utilized (see e.g. Wang et al. [10]), enabling the implementation of visual stimuli at flexible frequencies. Visual flickers with a frequency resolution as low as 0.05 Hz have been realized with this technique [9], still eliciting SSVEP responses the BCI could differentiate between. Therefore, a similar system was used for multi-target BCIs, yielding an overall higher ITR [4].

Regarding frequency choice, the best SSVEP responses are usually obtained using stimulation frequencies between 5 and 20 Hz [3]. However, in terms of user-friendliness, these frequencies are considered annoying by many participants, which is why many researchers developed BCIs with higher flickering rates [1, 2, 8]. Higher stimulation frequencies, on the other hand, are harder to realize on a standard monitor due to the limitations of the vertical monitor refresh rate (typically 60 Hz, in this experiment: 30 Hz). They also evoke weaker neural responses, reducing the accuracy and speed of the whole BCI system.

In this paper, a Custom-Made Computer Display (CMCD) dedicated for the graphical presentation of VEP-evoking stimuli is presented and tested. The stimuli are presented by a change in illumination rather than in contrast. The monitor that was specially assembled for this purpose enables easy and exact high-frequency stimulation, without the need of approximation methods. The frequency of the flickering can be controlled freely, enabling easy setup and testing.

In order to investigate the effectiveness of this hardware, we tested three frequency sets utilizing both the typically used frequency approximation method, as well as the custom-made hardware to present the stimuli.

Fig. 1.
figure 1

A participant during the experiment (the recording for the offline analysis). The flickering is realized by the CMCD’s background illumination. In the picture, the segments 1, 4, and 5 are in an active state, while the other segments are not.

Full size image

2 Methods and Materials

This section provides a description of the equipment, the setup of the study (Fig. 1) and the details of the used BCI system. First of all, information about the participants is given, which is followed by the description of the CMCD. Then, the recording hardware, the signal processing, the experimental procedure, and finally the Graphical User Interface (GUI) are discussed.

2.1 Participants

Nine healthy volunteers (four female) participated in this study. The mean age of the participants was 24.00 years with an SD (standard deviation) of 3.16 years. Written informed consent was given by each subject before the experiment, in accordance with the Declaration of Helsinki. This study was approved by the ethical committee of the medical faculty of the University Duisburg-Essen. All information was stored anonymously during the experiment and results cannot be traced back to the participants. Subjects had the opportunity to opt-out of the study at any time. Spectacles were worn when appropriate. The subjects received a small financial reward for participation in this study.

2.2 Stimulus Presentation

Two different methods of stimulus presentation where utilized, LED illumination (background light) and frequency approximation (stimuli shown by rendering on the screen) [6].

2.3 CMCD

The monitor was assembled by putting controllable LED strips into a Fujitsu ScenicView P22W-5 TFT-Monitor (Figs. 1 and 2). The whole screen was divided into six segments, in which the LED strips were separately controllable. To minimize their crosstalk, the intensity at the intersection of the segments was reduced (Fig. 3). This means that two types of LED lights were used with different brightness. For the most part, HD LED strips (LED-Emotion GmbH, 6500 K pure white, 2330 lm/m, 19.2 W/m, 140 LEDs/m, 24 V) were utilized, but along the middle horizontal line, less bright strips (Abrams & Mantler GmbH & Co. KG, 7000 K cold white, 950 lm/m, 9.6 W/m, 24 V) were used. The lights were controlled by a Raspberry Pi 3 B (Raspberry Pi Foundation), which controlled the six flickering fields of the monitor via a Gravity: MOSFET Power Controller SKU: DFR0457 (DFRobot). By providing power the LED strips turned on, and the changing of these on/off phases generated the flickering.

Fig. 2.
figure 2

The concept of the used LED structure inside one of the segments of the CMCD. The Cold Cathode Fluorescent Light (CCFL) backlight provides the standard display on the screen, the LED-Emotion and LED-Konzept lights provide the flickering by illumination. Six fields are separated, which can thus flicker at different frequencies without affecting the neighboring fields.

Full size image
Fig. 3.
figure 3

On the left side the crosstalk between the segments of one row is shown, while on the right side the crosstalk between segments above each other is shown.

Full size image

2.4 LED Illumination with the CMCD

Figure 2 shows the arrangement of the LED strips used in the CMCD. The intensity of the LED illumination was set to a medium value, when the LEDs were in the active state, each segment was using 260 mA. The illumination produced with this setting made the produced SSVEP response easily detectable and prevented the user-discomfort felt at higher light intensities. Each segment contains 105 LED lights from LED-Emotion and 18 from LED-Konzept. The flicker frequencies were controlled by a Python program running on a Raspberry Pi. The program utilized the multiprocessing library, and when the contrast changing method was used for stimulation, the frequency of these LEDs was set to 60 Hz (the same as the monitor refresh rate).

2.5 Stimulus Approximation Method

In order to implement multiple targets on the LCD-screen, frame-based stimulus approximation was utilized for the contrast changing method [6, 10].

To realize the flashing pattern, a specific frequency f was assigned to each target; the transparencies of the targets where sinusoidally modulated in accordance with the frequency [6].

For a monitor refresh rate of 60 Hz, the stimulus sequence for the i-th target is given by

(1)

yielding values in the range of 0 to 1.

2.6 Signal Acquisition

The computer operated with Microsoft Windows 7 Enterprise running on an Intel processor. Standard Ag/AgCl electrodes were used to acquire the EEG signals. The reference electrode was located at CZ and the ground electrode at AFZ. Eight signal electrodes were placed according to the international 10/20 system of electrode placement: PZ, PO3, PO4, O1, OZ, O2, O9, and O10. Standard abrasive electrolytic electrode gel was applied between the electrodes and the scalp to bring impedances below 5 k\(\Omega \). An EEG amplifier, g.USBamp (Guger Technologies, Graz, Austria), was utilized.

The sampling frequency was set to 600 Hz. During the EEG signal acquisition, a digital band pass filter (between 2 and 100 Hz) and a notch filter (around 50 Hz) were applied. The data was sent from the amplifier in blocks of 30 samples.

2.7 Signal Processing

Canonical correlation analysis (CCA), was used for SSVEP signal classification [5]. In general, CCA is used to investigate the relationship between two sets of variables and . It computes two vectors and that maximize the correlation \(\rho \) between the linear combinations and by solving

(2)

To classify the attended target with the CCA, \(\cos \) and \(\sin \) templates are compared to the EEG signal matrix , which contains recorded EEG data; M is the number of collected samples and N is the number of signal channels. Let \(N_h\) be the number of harmonics that are considered for frequency detection and \(F_s\) the amplifier sampling rate. For each frequency \(f_i\), \(i=1,\ldots , K\) the \(\cos \) and \(\sin \) templates are given by

(3)

For each of these templates, CCA is used to compute the maximal canonical correlation to the signal matrix X. This yields correlations \(\rho _i\), \(i=1,\ldots , K\). The classified target T is then determined as

$$\begin{aligned} T=\max _i \rho _i\quad i=1,\ldots ,K. \end{aligned}$$
(4)

Here, CCA-classification was performed on the basis of thresholds, i.e. if the difference between the highest and second highest \(\rho _i\) did not surpass a specific threshold \(\beta \), more data was collected, so that M increased stepwise. More details about the threshold-based classification can be found in [5]. In this study, the minimal classification time window was set to 2 s, i.e. \(M\ge 2F_s\). Moreover, the number of signal channels was set to \(N=8\), the amplifier sampling rate was \(F_s=600\), and \(N_h=3\) harmonics were considered to identify which of the \(K=6\) targets was attended.

2.8 Experimental Protocol

Participants sat on a chair facing the screen at a distance of approximately 80 cm. After the preparation for the EEG recording, they went through an offline recording phase, an online copy spelling phase, and a brief questionnaire. These steps were done with three different frequency sets in the \(\alpha \) and \(\beta \) bands (for details see Table 1).

For all participants the following order of sessions was utilized:

  1. 1.

    Alpha-band stimulation frequency approximation method

  2. 2.

    Alpha-band stimulation frequency CMCD LED illumination

  3. 3.

    Lower Beta-band stimulation CMCD LED illumination

  4. 4.

    Lower Beta-band stimulation frequency approximation method

  5. 5.

    Higher Beta-band stimulation frequency approximation method

  6. 6.

    Higher Beta-band stimulation CMCD LED illumination.

The order of the experiments was fixed in order to prevent user-frustration by possibly not working higher frequency stimuli (the experiment was stopped if the participants could not control the BCI with both types of flickering for at least a few minutes). This was necessary as the BCI scenarios which were not working caused such levels of user-frustration, that further tests would have been heavily biased. In order to avoid fatigue, participants took small breaks between the sessions. After each session, users were asked to rate the perceived level of annoyance of the stimuli on the Likert scale (1-5, 1 - not annoying at all, 5 - very annoying). This was done to compare the stimulus methods from this point as well. A disturbing stimulation would render the whole system not user-friendly, which opposes the goals of BCI development.

Table 1. Stimulation sets: Three sets of six frequencies (\(f_1\)–\(f_6\)) were tested (both offline and online) with both the frequency approximation method for standard LCD screens as well as the CMCD LED illumination presented in this paper.
Full size table

Offline Recording Phase. In the offline recording phase, six boxes were presented to the user containing the numbers 1–6. Each of the boxes flickered with a specific frequency. Participants were asked to focus on each box three times, while EEG-data trials were recorded (Fig. 1 shows the offline recording phase). A green frame around the box indicated which target needed to be fixated on. After each trial, the next box was highlighted, the recording paused for one second, and the participant shifted his/her gaze to the next target.

Fig. 4.
figure 4

Graphical user interface of the six-target speller. The participants wrote “BCI” in the familiarization session. In order to write the letter “B”, two selections were required: First, the group containing the desired character (here: “ABCDEF”) needed to be selected, then, the letter itself needed to be chosen.

Full size image

Online Copy Phase. In the copy spelling phase, participants completed a spelling task by utilizing a six-target BCI spelling application. After each selection, the recording paused for two seconds, when the participant shifted his/her gaze to the next target (i.e. letter). The same classification threshold was used for each subject and task. After a brief familiarization run where participants spelled the word “BCI”, participants were asked to spell the word “KLEVE”. During the spelling task, accuracy and ITR were measured. As the interface provided a two-step selection mechanism for writing letters, a minimum of ten commands were necessary to write “KLEVE” (First the box “GHIJKL” had to be selected, followed by “K”, etc.). Errors had to be corrected by fixating on the last target for the “UNDO” function of the interface. The goal of this phase was to provide information about the performance of the BCI with different types of stimuli generators (CMCD and approximation method).

Table 2. Results of the online spelling tasks of the frequency approximation test. Provided are the averaged time, accuracy, and ITR for all the tasks “KLEVE”. For the higher Beta-band stimulation, participants who did not complete the task for both stimulation methods (marked with * or -) were excluded from the calculation of mean values.
Full size table

2.9 Graphical User Interface

For this experiment, a six-target BCI spelling application was designed. The GUI was arranged as a \(2\times 3\) stimulus matrix containing the letters of the alphabet as well as additional characters in five groups of six characters each (see Fig. 4). Each of these characters could be selected in two steps. The sixth box contained a correction option. Every command classification was followed by an audio feedback.

Table 3. Results of the online spelling tasks done with the CMCD LED illumination. Provided are the averaged time, accuracy, and ITR for all of the “KLEVE” tasks.
Full size table

3 Results and Discussion

Tables 2 and 3 show results from the online experiment. ITRs were calculated according to [11]. For the CMCD LED illumination, mean ITRs of 39.89, 37.68, and 15.76 bpm were achieved with the Alpha-band, lower Beta-band and higher Beta-band stimulation, respectively. For the frequency approximation method, ITRs of 36.58, 40.29, and 21.34 bpm were achieved, respectively.

A paired t-test was performed to investigate differences in online performance between frequency approximation method and CMCD illumination. However, no statistically significant difference was found for either stimulation set. The reason for this result could be the length of the time window (2 s). This prevented faster classification, which would have otherwise occurred. If we calculate the ITR with maximum accuracy and the fastest possible classification (2 s) and include the gaze shifting time of 2 s (except before the first classification), we get the maximum achievable ITR with this window length: 40.82 bpm. There are no significant differences between the results most likely because most subjects reached an ITR close to the maximum, which suggests that using shorter time windows, which allow a higher maximum ITR could result in more dispersed (and higher) ITR results which could be significantly different for the two types of stimulation. To further investigate the difference in performance, offline analysis was conducted.

Fig. 5.
figure 5

Offline comparison of the CMCD LED illumination and the frequency approximation method. Provided are the accuracies and ITRs for classification windows between 0.25 and 2 s from the offline recording. Each of the six targets was attended three times. For classification, CCA was utilized.

Full size image

The results of the offline analysis are provided in Fig. 5. A clear trend can be observed, showing that the CMCD LED illumination yields faster system speeds for all tested frequency bands. The outlier results of the high Beta-band stimulation with CMCD originate from one subject, who had exceptionally good control over the BCI, achieving close to maximum ITRs in the online phase, and really high ITR and accuracy in the offline analysis. The same subject achieved also high ITR with the approximation method, however, as can be seen on Fig. 5 this result is not considered an outlier as the variance of ITR across subjects was much larger as with the CMCD. This outlier, as well as some of the results from the online phase (e.g. Subject 5s’ ITR difference is largely impacting the t-test results) warrant a study with a higher number of subjects, longer spelling tasks, and appropriate settings for the classification time windows to find substantial differences and lessen the influence of outlier results.

Participants found the CMCD stimulation more annoying (see Fig. 6). This can be attributed to the fact that for the CMCD, the targets flickered continuously; there was no break for 2 s in the stimulation after a command was executed. Surprisingly, the lowest flickering speed was found the least annoying by most participants.

This can be a consequence of the study design; all participants started with the low-frequency set and could have become more annoyed by the flickering during the experiment. A higher number of subjects are required for further investigation.

Fig. 6.
figure 6

Subjective levels of user-friendliness for the frequency approximation method and the CMCD illumination. For each of the tested frequency sets, participants were asked to state the perceived level of annoyance on a 1-5 Likert scale.

Full size image

4 Conclusion

A custom-made monitor for VEP stimulation was presented. The system allows the setup of high stimulation frequencies without the typical limitations caused by the dependence on the monitor refresh rate. Though the system yielded superior performance in terms of offline ITR, some improvements and tests can still be made in the future, like the optimization of user-friendliness. In preliminary tests, we also tested flickering beyond the visible threshold (30 Hz). However, with the classification algorithms used here, only a few participants achieved reliable control over the system. We are planning further tests allowing for faster classification, and with more participants to fully assess the CMCD LED illumination. Higher stimulation frequencies, and the effect of the intensity of the LEDs are also planned to be examined in the future.