Keywords

1 Introduction

Brain Computer Interface (BCI) technology utilizes brain signals to output commands to devices for communication and environmental control [1]. This is significant, especially for providing disabled individuals an avenue of communication that requires no physical movement. BCIs commonly use electroencephalography (EEG) for acquiring brain signals at the scalp [1]. The system consists of the user attending to letters or icons that flash on a screen. Event-related brain potentials (ERPs) are detected by the computer that is able to recognize the user’s selection [2]. BCIs can use different types of ERP signals including the P300 and steady-state visual evoked potential (SSVEP) [3]. Specifically, our focus is on P300-based BCIs. The P300 is a positive ERP occurring about 300 ms after a visual stimulus [2]. The oddball paradigm is characterized by a P300 in response to a target stimulus [2, 4].

Using the oddball paradigm, P300-based BCIs allow individuals with severe disabilities to make visual selections without the need for movement or speech [2]. Individuals with Amyotrophic Lateral Sclerosis (ALS) suffer motor-neuron degeneration causing the loss of voluntary movements [5]. When individuals can no longer communicate due to the inability to speak or move their extremities, augmentative and alternative communication (AAC) methods are needed to convey their needs. Additionally, BCIs can also give access to environmental control [6,7,8]. This involves control of devices such as lights, TV’s, and smart outlets. Recent advancements in smart home technology such as Amazon Alexa and Google Assistant are making environmental control more accessible. Enabling individuals with ALS to control their environment would significantly increase their independence and quality of life [9].

BCIs have already been used as AAC for people with ALS. However, they require sustained and high levels of attention and cognitive load that can be fatiguing for users [10]. Additionally, long calibration times and complex set-up of traditional systems decrease the practicality of BCI for home use outside of a clinic setting [11]. Specifically, for individuals with ALS, the need to be upright and stare at a screen poses challenges. At any stage of the disease, neck weakness or dropped head syndrome can lead to the inability to hold the head up [12]. At later stages of the disease, individuals commonly find lying down or reclining most appropriate. Having to be propped up in an upright position can contribute to fatigue while using a BCI and deter individuals from using it. An ideal solution allows the user to utilize the system in any position.

Fig. 1.
figure 1

Our traditional BCI set-up with a computer monitor screen (left). Augmented Reality glasses worn with an EEG cap (center). Augmented Reality display (right)

Full size image

In this study, we aimed to combine newer portable display technologies with P300 based EEG-BCI to address some of the problems arising from home use. Specifically, we have adapted the screen for stimuli and output projection via an augmented reality (AR) glasses and utilized the BCI output for environmental control via Google Assistant. The use of AR glasses enables visualization in any position without the need for multiple recalibrations and adjustments of the user interface. Furthermore, AR glasses have advantages over virtual reality (VR) devices since the AR technology maps virtual objects onto the real world rather than a dark background or screen. Hence, the surrounding environment is still visible while wearing the glasses thereby reducing any sense of claustrophobia and allowing the user to see anything that is happening within the room. AR has become increasingly popular in medicine, as it allows for a unique way to display images [13]. A BCI in which targets are displayed on AR glasses could provide a solution to the problems of positional fatigue enabling use in any position (See Fig. 1). Finally, integrating BCI with smart home assistants like Google Assistant opens the door for future practical use scenarios.

2 Traditional System Set-up and Challenges

The current BCI system used at Temple Hospital’s Department of Neurology clinic under Dr. Terry Heiman-Patterson, in collaboration with ALS Hope Foundation, consists of a 16-channel wet EEG electrode array and follows the standard 10–20 system of electrode placement [14]. The system uses a P300 classifier to train and test a linear classifier for detection of the evoked related potentials collected with BCI2000 [15]. A MATLAB GUI was designed for the analysis of the BCI2000 data collected using a speller [16]. The P300 Speller module implements a BCI that uses evoked responses to select items from a rectangular matrix [2]. This module can deliver visual and auditory stimuli in user-defined or pseudo-random sequences. In simulation mode, the simulated EEG will simulate evoked responses on the respective items, allowing to test signal classification and spelling functionality. Once calibrated, the purpose of the P300 Speller is to choose arbitrary letters from the matrix or even spatial navigation icons instead of letters for control of virtual environments [17].

Significant challenges have limited the home deployment of our traditional BCI system. First, the research-grade original systems developed are cumbersome to use, consisting of wet electrode caps, multiple wired connections to multiple components, the need for a large amplifier, and a display that consists of a computer screen that ran two different types of software. MATLAB software is utilized for weight calculation, C++  for the actual stimulus presentation and output display, and software that was written for decoding the signals. Also, having many sub-components render the system susceptible to failures and malfunctions. Furthermore, it would be hard to troubleshoot for the user in the home,

The second issue is the long set up time ranging from at least 20 min to 45 min or even more. This is a difficult time constraint for a busy caregiver and requires training in electrode placement and use of the programming reducing the quality of the user experience.

The system can also be uncomfortable for the user, requiring a fixed computer monitor screen to display the keyboard flashes that act as the visual stimulus for the ERPs. This leads to two major issues. First, the user needs to be in an upright position to keep the screen in their line of sight for best performance accuracy. This can be challenging to the user with ALS or other neuromuscular disorders due to fatigue and discomfort. Additionally, the calibration in these systems can add to the challenges. The screen is fixed w.r.t the user’s head position. Given that individuals with ALS commonly experience cervical weakness, users will have difficulty maintaining head position for prolonged periods resulting in the likelihood that there will be a loss in screen calibration due to a shift in the neck position. Every recalibration adds 4–5 min to the set-up time before the system is up for use again [18].

Even when set up, with this system the process of communicating is time-consuming. It has been reported that on average a person speaks approximately 150 words per minute depending on individual, demographic, cultural, linguistic, psychological, and physiological factors [19]. This can roughly be equated to 500 characters per minute. A traditional BCI system can barely process 4–5 characters per minute in a practical test case [20]. This gap, between the characters that can be spoken versus processed by this system can end up being frustrating and time consuming.

Finally, the systems, as designed, has limited functionality especially based on the technologies now available. The traditional BCI system displays letters, numbers, and punctuations on the screen which may be used to spell out words and sentences. It is not interfaced to home control and there is a large dependence on a caregiver for set-up and to carry out a task that can easily be accomplished utilizing home control devices.

3 AR BCI Based Smart Home Set-up

3.1 Ultra-mobile EEG Headset

A significant drawback of the traditional system is the inconvenience of the EEG cap. It had 16 wet electrodes wired to a separate amplifier unit. This not only made putting on the EEG cap on the user an inexpedient task but also made it harder for the user to use the system lying down. Along with the hardware issues, the software used was very hard to run and set-up. These factors contributed to the interface not being home-use friendly.

We considered some basic points while choosing a new headset for our new BCI. Specifically, we opted for an ultra-mobile system that is battery-operated, wireless, and head-mounted. Before selection, we checked system specifications like sampling rate, battery life and type (e.g., rechargeable, or replaceable), how many EEG channels, electrode type (wet, dry, or both), usability (weight, wires, placement, configuration, and setup steps) After considering these hardware specific questions, we also took into consideration their respective software requirements. Does the headset come with its own software customization and extensions like software development kit (SDK) or application programming interface (API)? What is the developer community and available documentation for them? What are the different programming languages that the SDK/API is available in? How customizable is the SDK and can it be adapted to work with XR applications?

After considering these prerequisites, we decided to utilize Unicorn’s Hybrid Black EEG Headset kit that has 8 EEG channels with electrodes that can be used either dry or with gel, a 24-bit sampling rate of 250 Hz and 6–8 h battery life, which can be recharged using any micro-USB plug. The electrodes are connected to a Bluetooth hub mounted on the back of the head that communicates with the computer using a Bluetooth dongle. The SDK is available in C, C#, and Python. It also has Lab Streaming Later (LSL) and User Datagram Protocol (UDP) data transmission capabilities which allow the SDK to be used with 3rd custom party applications as well.

3.2 AR Headset

One of the things that made the traditional system cumbersome to use was the fact that the visual input and output was using a flat LCD computer monitor panel. The user had to be in an upright position. Even change in placement of neck could cause loss of calibration, adding to the inefficiency.

We opted to eliminate this issue by replacing the LCD computer monitor with a head-mounted AR display goggle. Because the goggles are mounted on the user’s head, they can be in any position they feel comfortable in; the screen displaying the inputs and outputs will always be at a constant location from their eyes. This eliminates both the issues of user fatigue and frequent loss of calibration.

We asked a lot of similar design questions while choosing an AR headset as well. Is the system built for commercial use and readily available? Was it wireless and had an inbuilt battery or was it wired to a computer and used that for power? How heavy is the whole headset? What is the field-of-view (FOV)? A small FOV would make the displayed image very small and centered on the goggles, making it harder to see. What is the refresh rate? We wanted a refresh rate of at least 30 Hz for our P300 speller display. If the refresh rate of the headset would have been lesser than 30 Hz, the user would have been able to notice each time the screen was redrawn, causing visual strain and fatigue. We also considered some software specific questions. Does the headset come with its own SDK/API? What are the different programming languages that the SDK/API is available in? Availability of the SDK documentation?

Finally, we selected commercially available DreamWorld’s DreamGlass AR headset. It has a Unity based SDK that is able to run on Windows. It has a refresh rate of 60 Hz and 90° diagonal FOV. The AR headset is wired to a computer via HDMI cable and the AR display is recognized as an additional (second) screen on the computer (See Fig. 1, right). We utilized DreamWorld’s AR Desktop app that mirrored the computer desktop onto an AR environment displayed on the goggles.

3.3 Raspberry Pi SoC

As we want the interface to be home-use friendly, portability was one of the biggest shortcomings of the traditional BCI system. The system was built for in-clinic research purposes. One of the main reasons for the low portability of the system was the high computing performance requirements of the traditional system. But, with the development of technology and the increase in the complexity and wide-spread availability of system-on-chip (SoC) computer systems, it has become possible to make portable architectures without a significant drop in computing performance.

The current AR BCI set-up works with 64-bit Windows 10 operating system and higher running on a quad-core chip with a clock speed in the range of 2.4 GHz to 3.8 GHz that can be overclocked to 4.1 GHz. The minimum memory requirement is 8 GB. The total graphics power of the hardware needs to have a base clock of 1150 MHz with a boost clock of 1340 MHz, a power consumption rating of 350 W, 2 GB of 128-bit DDR5 6.6 GBps memory with a core clock speed of 1024 MHz and 770 CUDA cores.

Although no single SoC currently available in the market met all those specifications, we decided to test a sub-system of the whole interface on a relatively low cost SoC to understand the feasibility of the task. Hence, we decided to go ahead with putting the smart assistant part of the system on the Raspberry Pi 3B+. It uses a Broadcom BCM2837B0 chip, with 64-bit ARM cores capable of running at 1.4 GHz and 32-bit VideoCore IV running at 400 MHz. These specifications were more than enough to support any smart assistant and its associated functionalities.

3.4 Smart Home Assistant

One of the limitations of the traditional BCI system, as mentioned above, was that the interface was built to be used only as a means of communication with the participant’s health aide. Addressing this in our new system would reduce the dependance of the user on the caregiver for certain tasks, increasing independence. We resolved this issue by integrating smart home assistant in our interface giving user the capacity to control their environment. The interface was only tested with basic entertainment and device on/off actions, but with the vast possible capabilities of smart home assistants, the applications are endless.

Both Amazon Alexa and Google’s Google Assistant can be set up in a relatively low-cost SoC like Raspberry Pi (we used 3B+). However, only Google provided the developers with documentation to modify the code to work with other triggers as like a physical button press, a software call etc. This allowed us to finally integrate the Assistant in our BCI interface which in-turn gave the user access to all of the Assistant’s functions.

3.5 Overall System Design

Fig. 2.
figure 2

Our new AR integrated P300 based home BCI system architecture

Full size image

In the preceding section, we have explained the changes that have been implemented in the hardware and software in order to develop a more effective and robust BCI system. The overall proof-of-concept BCI block diagram is depicted in Fig. 2.

The EEG cap is placed on the user’s head with the 8 electrodes placed at FZ, C3, CZ, C4, PZ, PO7, OZ, and PO8 following the Unicorn’s default 10–20 system of arrangement. The electrodes are wired to a Bluetooth module placed on the back of the head that communicates with the computer via bluetooth and sends the signals for further processing. The AR headset is placed on the user on top of the EEG cap to act as the stimulus display (the modified speller interface) and the output selection. The signals sent to the computer are processed in the Speller software. Once the required selection is made, the associated function call is made to the Google Assistant listening for it on the Raspberry Pi SoC. Once the assistant receives the input, it processes the input, sends the instructions to Google Cloud Services which then executes the corresponding action.

The new BCI system developed by our team is designed to provide improved access with a broader range of capabilities as compared to the original design. There are certainly additional improvements to be implemented as we will be performing a human test of our system with healthy volunteers. The new system is expected to provide ease of use with speller-based selection. However, increasing the speed of selection, and hence reducing the time of each icon selection remains one of the factors that can be improved upon in the future versions.

AR technology is also constantly evolving and expanding with rapidly improving hardware and software, as it is still in early phases of its development cycle and consumer adaption. Going forward, we look to improve the AR experience with better headsets and software. We are also hoping to reduce the size and portability of the overall BCI system.

As embedded computation units improve, we further expect that computational performance will increase, and hardware’s power consumption will decrease, enabling integration of all the processing into a single powerful SBC to make the overall BCI system more portable, streamlined, and easier to manage.

Future studies could also explore other BCI paradigms and other fields of application for interaction in augmented reality environments.

4 Conclusion

In this paper we explained a new proof-of-concept AR based EEG-BCI system targeting home use for people with ALS. This paper outlined the design of this initial protype, and the challenges faced, lessons learned during its development along with future directions. As BCI research and consumer electronics industry rapidly evolve, new opportunities emerge for developing enhanced minimally intrusive and non-invasive BCI systems.

Overall, we have proposed a scalable approach to guide the design of BCIs using AR. We believe that AR is a promising new medium that could be adapted and reused in other BCI types and could orient future research on BCI-based interactions in mixed reality.