Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

What Are Brain-Computer Interfaces?

A brain computer interface (BCI), also known as brain-machine interface (BMI) is a system that allows for direct communication between a human and a machine without using traditional channels of interaction, e.g. the muscles of the arm and hand and a keyboard, and instead relies on brain signals directly [30]. This makes BCI technology especially attractive for people with severe motor disabilities such as multiple sclerosis (MLS) or locked in syndrome [3]. In extreme cases such interface is the only way by which a person can communicate with the external world, which can greatly improve their quality of life. The idea of BCI was initially unattractive to science, the idea of deciphering human thoughts seemed weird and remote. BCI systems were limited to laboratory and clinical use; however the recent developments in machine learning technology and increase in computational power of personal computers made BCI accessible not only to researchers and clinicians, but also for everyday users. BCI systems can be categorised into invasive and non-invasive. In case of invasive approach the signal is acquired using electrodes placed inside the scalp. In non-invasive BCI the signal is acquired outside of the scalp. In this chapter, the results of investigation into the use of two commercially available BCI devices for gaming and virtual environments are presented. In section “Neuroimaging Techniques for BCI Systems”, BCI systems based on different neuroimaging techniques are described. Section “Electroencephalography-Based BCI (EEG-Based BCI)”, presents the theoretical bases for different EEG-based BCIs. Stages of BCI systems are illustrated in section “Stages of a BCI System”. Section “BCI Paradigms” demonstrates presents different BCI paradigms used in BCI games, followed by different areas of BCI applications in section “BCI and Serious Games”. Finally, the description of performed experiments is presented in sections “Investigating Commercial BCI Systems for Serious Games and Virtual Environments” and “Conclusions and Limitations” presents conclusions.

Neuroimaging Techniques for BCI Systems

Different indicators of brain activity as well as different neuroimaging techniques can be used for signal acquisition and these are described below.

There are two types of brain activity that can be used for this purpose: metabolic and electromagnetic. Only the latter type of systems is currently available for day-to-day BCI devices. For each of the brain activity modalities there are two different neuroimaging techniques that can be used. For BCIs based on metabolic brain activity we can distinguish two types of systems functional Magnetic Resonance Imaging (fMRI) based BCIs and functional Near Infrared Spectroscopy (fNIRS) based BCIs. For BCIs based on electromagnetic brain activity we can distinguish two types of systems magnetoencephalography (MEG) based BCIs and electroencephalography (EEG) based BCIs. These systems are discussed below.

For the first category, both rely on very different method of signal acquisition, and are consequently sensitive to very different variations in the signal. fMRI is primarily used in research, because it is not practical for day-to-day activities, whereas fNIRS, which may be more accessible to a wider audience, is still under development.

MEG is a non-invasive imaging technique that registers changes in magnetic field associated with the electrical activity of the brain. MEG and EEG record signal associated with the same neurophysiological processes. However, the magnetic field is less prone to distortions introduced by the skull and the scalp than the electric field, therefore the quality of signal provided by the MEG is better than in case of EEG. The main limitations of MEG-based BCI are similar to the ones of fMRI-based systems. These include high cost and large equipment that cannot be used outside of a laboratory.

Electroencephalography-Based BCI (EEG-Based BCI)

EEG is the oldest and most widely used neuroimaging technique. Since its discovery in 1929 [2], EEG has been used by scientists to answer question about the functioning of the human brain as well as by clinicians as a diagnostic tool. BCI systems also allow using EEG as neurofeedback in neurorehabilitation. One of the reasons for the popularity of EEG based systems is the relative low cost, portability and low complexity of these devices.

EEG signal collected on the surface of the skull is a result of the activity of neurons. Neuron is a call in the nervous system that processes and transmits information using electrical and chemical signals. A typical neuron possesses a cell body (soma), dendrites, and an axon. The cell body can give rise to many dendrites. Dendrites are thin structures that span for hundreds of microns and form a dendritic tree. The cell body can give rise to multiple dendrites, but can produce only one axon, however axons can branch out hundreds of times before they terminate. A human axon can extend for up to one meter. Dendrites and axons connect neurons forming neural networks. Information from one neuron to another is passed through synapses. In most cases a synapse connect an axon to a dendrite, however there are exceptions (e.g. axon can connect directly to the body of the neuron).

Neurons communicate through action potentials, i.e. electrical discharge produced by the soma of the cell. Action potential travels along the axon and when the action potential arrives at the synapses neurotransmitter is released. Neurotransmitter triggers change in potential of the membrane of the receiving cell (flow of ions through the cell membrane) and if this potential reaches a threshold, new action potential is triggered and the information is transmitted to another neuron.

The signal measured using EEG equipment is thought to be generated mostly by the pyramidal neurons located in the cerebral cortex [14]. Pyramidal neurons have large soma of a shape that resembles a pyramid and a large dendrite extending from the apex of the soma and is directed perpendicular to the surface of the cortex. Activation of an excitatory synapse creates excitatory post-synaptic potential, i.e. inflow of positively charged ions form the extracellular space to body of the neuron. As a result, the extracellular region of the synapse becomes negatively charged and in turn regions distant from the synapse become positively charged and cause a change of potential (extracellular current) to flow towards the region of the synapse. The spatio-temporal summation of these extracellular currents at hundreds of thousands of neurons with parallelly oriented dendrites creates the change of potential that is detectable on the surface of the scalp. If a large number of excitatory synapses are activated close to the surface of the cortex, the resulting potential, detectable on the surface of the scalp, is negative. If synapses of the same type are activated closer to the body of the pyramidal neurons, deeper in the cortex the resulting potential is positive. Reverse relation is observed for inhibitory synapses. Activation of a large number of inhibitory synapses close to the surface of the brain produces positive potential and activation of inhibitory synapses in the deeper layers of the cortex results in negative potentials recordable on the surface of the scalp. It is therefore possible to infer the type of synapses activated form the polarity of the signal acquired on the surface of the scalp.

EEG is the recording of electrical potentials along the scalp. EEG recordings are usually performed using small metal electrodes placed on the scalp in standardised positions. The number of electrodes can vary depending on the type of device used. To improve the conductivity between scalp and electrodes conductive gel or saltwater are used. The electric potentials recorded by EEG result for the neural activation within the brain. EEG is the most widespread neuroimaging technique and the most widely used modality in BCI. The popularity of EEG stands form the fact that electrical signal can be easily and cheaply recorded through electrodes placed on the scalp [1]. However, electric current has to cross the scalp, skull and other tissues surrounding the brain which significantly distorts the acquired signal. EEG signal is also distorted by the electrical noise in the environment and electric current produced by muscles.

Researches using EEG identified distinctive patterns in the EEG signal. These patterns are related to specific cognitive activities performed. Although, the exact meaning of most of these patterns is still unknown, some of them have been thoroughly studied and are used in BCI systems. The three EEG signal features that are most often used in BCI research are P300, sensorimotor rhythms and steady state visually evoked potentials [25].

Stages of a BCI System

BCI can be considered as artificial intelligence system that employs machine learning. Such a system consists of hardware and software components with the aim of recognising patterns in the signals emitted by the brain, and to translate them into practical commands. In a typical BCI system, five consecutive stages can be identified and these are presented below [25].

  1. 1.

    Signal acquisition – various types of signal are captured by a neuroimaging device such as electroencephalography (EEG); a BCI system may be acquiring several kinds of signals at the same time, provided they are synchronised and time-locked to the interaction with the device.

  2. 2.

    Signal pre-processing or signal enhancement – signal is prepared to further processing, including artefact removal (e.g. muscle movement, and noise reduction) are typically performed at this stage.

  3. 3.

    Feature extraction – discriminative features are identified and mapped onto a vector; these may include first order parameters, like amplitude of signal or latency, and second-order parameters that require more processing, like time-frequency parameters extracted from a Fourier transform.

  4. 4.

    Classification – involves the classification of the parameters previously extracted, with the aim of ascribing meaning to them; various techniques from machine learning can be applied, but this imposes an overhead in time and processing power that is not suitable to all BCI applications, which demands real-time interaction.

  5. 5.

    Control interface – results of classification are translated into commands and send to a connected machine such as a wheelchair or a computer, which provide the user with feedback and close the interactive loop between the user and the device.

These stages create a communication loop between the user and the machine which is the essence of BCI. Different ways of implementing this loop can be proposed and this can result in different control paradigms: active, passive and reactive. Below, we will discuss these paradigms in the context of computer games.

BCI Paradigms

Active BCI

In case of active BCI user modulates brain signal actively. In order to effectively control the computer this signal should be discriminative. Active BCI can be used to directly control an BCI application. Active BCI applications often make use of motor imagery as the control paradigm. Imagination of motor movements result in contra-lateral event related de-synchronization (ERD), and ipsi-latelar event related synchronisation (ERS). When the user imagines a right hand movement, the amplitude of the mu-rhythm in the left sensory-motor area increases while the amplitude of mu-rhythm in the right sensory-motor area decreases. These changes can be observed in mu-rhythm (7.5–12.5 Hz). An active BCI pinball game was presented by [26]. Pinball is a fast-paced game with rich environment that requires very fast and precisely timed reactions. The users were able to achieve good control of the game and rated their experience very highly.

Reactive BCI

In reactive BCI, the brain signal containing information is generated as a response to external stimulation. The user introduces changes in the signal by attending to the stimuli. Reactive BCI paradigms include steady state visual evoked potential (SSVEP) and P300. In SSVEP, the user concentrate on one of flickering stimuli. This results in an increase in activity of the same frequency (or a harmonic of that frequency) observed in visual areas and allows to identify the stimuli to which the user is attending to. Jackson et al. [8] proposed a first person shooter type game in which the user could move around fire the gun by concentrating their eyes on one of four flashing stimuli displayed on the screen.

In P300-based BCI systems, the user concentrates on one of many randomly activated stimuli. The P300 (also called P3) wave is a component of an event related potential (ERP) which is elicited by infrequent auditory, visual or somatosensory stimuli. P300 is a positive peak in EEG signal occurring around 300 msec after onset of the event. This change in EEG signal is normally elicited using oddball paradigm, as a response to a low probability stimulus that appears amongst high probability stimuli. P300 is also augmented when one perceives a stimulus that is regarded to be important; a stimulus one pays attention to. A P300-based BCI game, so called Mind Game was developed by [5]. It is a 3D checkerboard-styled board game with aim of visiting all trees placed on the board. The trees are also used as a target like in typical P300 seller paradigm, the squares underneath the trees are illuminated randomly. The user moves an avatar on the board by gazing their eyes on the target tree.

Passive BCI

The primary objective of passive BCI system is not to provide the user with the ability to intentionally control the device, but to monitor the user’s mental state. Using these systems does not require much effort the monitoring of the users mental state happens automatically. In passive BCI, the primary role of the system is not to give control to the user, and it does not require any effort on the user’s part; instead, it monitors the user’s mental states automatically. The level of desired mental state is quantified and used to facilitate the communication between the user and the system.

One of the most popular computer games has been adapted for BCI input. In Alpha-World of Warcraft [27] the level of alpha activity, recorded over the parietal lobe, was used to control one aspect of the game – the type of the avatar, other aspects of this game were controlled using traditional methods (keyboard and computer mouse). High level of alpha activity over the parietal lobe is believed to be indicative of relaxed alertness which is probably the bast state of mind for playing computer games. On the contrary, it was assumed that a low level of alpha indicated the state of distress. When the level of alpha was high the avatar would assume the form of a druid and a low level of alpha resulted the avatar changing its form to a bear, with sharp teeth and claws. In time of distress, the bear form of avatar was better suited for a fight, while in relaxed times the druid form was more fragile, but able to cast effective spells as well as heal herself.

BCI and Serious Games

This section describes different areas of application domains in which BCI games have been used. These include: research, medicine and commercial BCI games.

BCI Games for Research

EEG technology has been use in clinical and research applications for decades. Adding a BCI and gaming components greatly enhance the research possibilities. The first BCI games designed 1977 was used exactly for these purposes [28]. In the experiment the participants navigated a 2D maze using ERP differences in EEG signal, much like P300 paradigm. However, in this care the researchers analysed the negative change which occurred 100 msec. after the stimulus presentation. The users concentrated their eyes on one of four stimuli (top, down, left, right) and this allowed them to move within the virtual maze. A number of well known games like Packman, Pong and Tetris have been adapted to be used with Berlin brain-computer interface (BBCI) [10]. Other BCI games developed for research purposes include flight simulator [15] and a tight rope walker controlled to the changes in left and right hemisphere activity [21]. Reuderink et al. [19] developed a game based on packman. The game was used to induce frustration and investigate the influence of frustration on the quality if EEG signal. The preliminary results indicate that frustration may have an effect on the quality of EEG signal and may affect the signal features used for classification. Frustration and tiredness often leads to decrease in BCI performance. Research into the effects of embedding BCI in games found the opposite effects. User performance increased when playing BCI game, this is most likely due to the fact that BCI games offer reach environment which increases the entertainment that stimulate and motivates the users.

BCI Games for Medical Applications

Neuro-feedback training is an alternative to medication for treatment of some neurological disorders. The most popular of them is attention deficit hyperactive disorder (ADHD). A number of applications using BCI for this purpose have been developed. Most of them rely on modulation of slow cortical potentials. This methods have been used inn the clinical settings for decades [20]. The availability of inexpensive BCI devices made it possible to use these techniques at home. The main obstacle in popularisation of this type of neuro-feedback is the lack of interest form the potential patients – often children who due to their condition find if difficult to concentrate on performing boring and mundane tasks. The application of computer game technology can makes the training more entertaining and encourage children to engage in it. Strehl et al. [24] developed a game that allowed users to steer a ball to a target using modulation of slow cortical potentials. Another example of a passive BCI device used for medical purposes was developed by [18]. The researchers investigated the use of neuro-feedback for the treatment of ADHD. The participants were divided into two groups. One group used standard neuro-feedback, while the other used neuro-feedback embedded in a computer game. Correct brain activity pattern was rewarded by more responsive controls.

Commercial BCI Games

Large game companies like Microsoft, Sony and Nintendo are yet to move into the world of BCIs. The market is no dominated by, however very active players. Two probably most popular producers of commercially available BCI systems are Emotiv [4] and NeuroSky [17].

The game available commercially can be divided into two categories, those offering the participants the ability to control a an avatar in virtual environment and those using neuro-feedback for the purpose of altering the users mental state and thereby improving user’s well-being. The ‘control’ applications employ active paradigm and usually require training. An example of one of them is StoneHenge game developed by Emotiv [23]. In this game users are requested to rebuilt the Stonehange, by using motor imagery to lift, push, pull and rotate giant stones until they are placed in the desired position. Another game developed by Emotiv is ‘Mindala’ [16] in which users can train meditation skills by controlling a mandala displayed on the computer screen. This is an example of neuro-feedback, passive application. In order for BCI to become a popular amongst computer game enthusiasts relatively inexpensive BCI devices have to provide good gamer experience. In the next section we evaluate two inexpensive and very popular currently available device.

Investigating Commercial BCI Systems for Serious Games and Virtual Environments

This section presents the evaluation of BCI systems using two commercially available devices: NeuroSky and Emotiv. Firstly, we present the evaluation of two BCI systems utilising Emotiv devive section “Comparison of Serious Games Using Emotiv”. Secondly, the evaluation of two BCI systems based on NeuroSky is presented section “Comparison of Serious Games Using NeuroSky”. Finaly, a comparison between the devices is presented (Fig. 6.1).

Fig. 6.1
figure 1

BCI devices. On the left – Emotiv Epoch; image available at http://emotiv.com, On the right – Neurosky Mindset; image available at http://neurosky.com. Epoch is a trademark of Emotiv, Mindset is a trademark of Neurosky (Images used with permission)

Full size image

The Emotiv headset [4] is a neuro-signal acquisition and processing wireless neuro-headset with 14 wet sensors (and 2 reference sensors) which is capable of detecting EEG signal as well user’s facial expressions. The neural signal is analysed and a number of affective measures are provided, these include ‘engagement’, ‘frustration’, ‘meditation’ and ‘excitement’. This type of measures can be utilized in a passive BCI design. Moreover, a built-in classifier can be trained using different mental activity (e.g. motor imagination) and these can be assigned to operations in the virtual environment, such as ‘push’, ‘pull’, ‘rotate’, and ‘lift’. These parameters can be easily used for an active BCI application.

The NeuroSky MindSet [17] device is a wireless headset with speakers and a microphone and one eeg sensor place on the forehead. Most of the signal collected from by this sensor corresponds to the frontal lobe, which limits the types of mental activity that can be used for controlling BCI. NeuroSky provide two measures precomputed by a ‘black-boxed’ NeuroSky algorithm. These are ‘attention’ and ‘meditation’. The measure of attention is a modulation of the frequency band which is triggered by the intensity of the user’s level of mental ‘focus’ when user focuses. It increases when the user concentrates on a single thought or an external object and decreases when the user is distracted. Both of the measures are well suited for passive BCI applications.

Comparison of Serious Games Using Emotiv

In this section we present the evaluation of two BCI systems which were develop with Emotiv device; the BrainMaze Game and Roma Nova (see Fig. 6.2).

Fig. 6.2
figure 2

BCI games used with Emotiv device: from the left – Roma Nova, LEGO NXT Robot, LEGO NXT Robot (view of the labyrinth from the top) (Images used with permission from the Serious Games Institute, Coventry University)

Full size image

The ‘BrainMaze’ Game

BrainMaze was designed for the user to navigate a 3D version of the LEGO NXT Robot [29] inside a maze with main goal to find the different waypoints that will lead to the finish line. If the robot hits a wall, the position resets and starts again from the beginning. Users have to be precise and cautious in order to find the way to the end. The walls are relatively narrow, requiring precise control and avoidance of sudden movements that could cause a position reset. The game session follows the first training session, during which the user trained in the Control Panel, so that the user is familiar with basic brain control.

Roma Nova

Roma Nova is built upon Rome Reborn [6] one of the most realistic 3D representations of Ancient Rome currently in existence. This 3D representation provides a high fidelity 3D digital model which can be explored in real-time. Rome Reborn includes hundreds of buildings, thirty two of which are highly detailed monuments reconstructed on the basis of reliable archaeological evidence. The interactive game is a serious game that aims at teaching history to children (11–14 years old). The game allows for exploratory learning by immersing the players inside a virtual heritage environment where they learn different aspects of history through their interactions with a crowd of virtual Roman avatars. The implementation of the Roma Nova game includes: (a) a crowd of Roman characters in the Forum and (b) a highly detailed set of buildings that belong to the Rome Reborn model. Intelligent agents are wandering around in the gaming environment between predefined points of interest, whereas the player is able to move freely and his movement is controlled via BCI device. To interact with the intelligent agents, the BCI-controlled player needs to approach them [11].

Participants and Experimental Procedure

Thirty-one participants used each of the prototypes. In case of both BCI systems the participants first trained using control panel (application provided by Emotiv SDK) and after that performed a task in virtual environment. After finishing the task, the participants were asked to evaluate their experience by filling in a questionnaire; a short unstructured interview also took place. These suggestions are a very helpful contribution towards the improvement of the system, giving feedback that an ordinary questionnaire cannot capture. The comparison of the results is presented in Table 6.1.

Results

Table 6.1 Comparison of average rating values for Roma Nova virtual environment and virtual robot for Emotiv headset
Full size table

Table 6.1 presents the comparison of user evaluation results of Roma Nova virtual environment and virtual robot with Emotiv headset. No significant differences for the ability to control, responsiveness, interaction and naturality of experience were found. The lack of significant differences can be explained by the similar difficulty of the BCI task. Both games required two-dimensional control, and while the quality of the virtual environment can influence the user experience it is unlikely to make a significant change.

Comparison of Serious Games Using NeuroSky

In this section we present the evaluation of two BCI systems which were developed with NeuroSky device; Roma Nova and Tetris (see Fig. 6.3).

Fig. 6.3
figure 3

BCI games used with NeuroSky device: left – Roma Nova, right – Tetris (Images used with permission from the Serious Games Institute, Coventry University)

Full size image

Roma Nova

Roma Nova was used again to evaluate user experience while using the NeuroSky device. The participants were instructed to move to the particular point within the virtual environment. However, the main difference with the previous interaction paradigm is that only one sensor was used to fully control the avatar. In this case, the participants attempt to control the avatar by changing cognitive states such as meditation and attention, which is translated to two integer values (in the range 0–100). To turn right the participant has to concentrate as hard as possible, while in order to move left they have to defocus their attention. Moving straight ahead was possible only by maintaining a balance between the two states. Meditation was used to control the velocity of the avatar, with high level of meditation resulting in high velocity.

Tetris

The second application use to evaluate NeuroSky was the well known Tetris game [13]. The Tetris game’s purpose was to teach the players how to self-regulate their state of mind in a stressful demanding situation to their own benefit as the more meditative they manage to become the slower each shape will fall in the context of the levels speed. The difference this feature will make becomes more prominent as the levels increase. This serious game is a multi-threaded application where the speed of the current falling brick is determined by the number of milliseconds required for the shape to traverse one line down (move on the Y axis from y0 to y1). The bigger this step time value, the slower the brick will fall. Participants were asked to play the game three times with the end goal of scoring at least five lines each time. The speed of the falling shapes increased with each level and that a level was marked by the collapse of a line. The speed of the falling blocks was also dependent on the meditation level provided by the BCI device. The participants were given an unlimited training time in which they could get accustomed with the setting and rules of the game.

Participants and Experimental Procedure

31 participants used each of the prototypes. After completing each task, the users were asked to evaluate their experience using NASA TLX questionnaire [7], followed by a short unstructured interview also took place. The comparison of the results is presented in Table 6.2.

Results

Table 6.2 Comparison of average rating values for Roma Nova virtual environment and Tetris game for NeuroSky headset
Full size table

Table 6.2 presents the comparison of user evaluation results of Roma Nova virtual environment and the Tetris game with NeuroSky headset. The users found controlling the avatar in Roma Nova virtual environment to be more mentally, physically and temporally demanding. They also reported that the Tetris game was less frustrating, required less effort, was easier to learn and the users scored the performance of the Tetris game higher than the performance of Roma Nova environment. There was no significant difference in terms of satisfaction gained by interaction with the two systems.

Comparison of Emotiv and NeuroSky

The comparison between Nurosky and Emotiv devices was performed by [12]. In this case both devices were used to navigate an avatar in virtual environment of Roma Nova. The results are presented in Table 6.3 and they indicated that it is easier to control an avatar, achieved higher satisfaction and rated the performance higher when using Emotiv. Moreover, using Emotiv headset also required less effort than using NeuroSky.

Table 6.3 Comparison of average rating values between NuroSky and Emotiv headsets for games and virtual environments
Full size table

Conclusions and Limitations

The results presented in this chapter show that BCI technology is a viable option for use in serious games and virtual environments. The qualitative feedback provided by the users shows that they enjoyed the interactive experience, they were in favour of using EEG technology for interacting with games, even though it is not as accurate as joystick, computer mouse or keyboard. At this stage of development, its usability is however still limited. The results show no difference in user experience when an avatar or robot movement was controlled in 3D environment when a 14-channel device was used. The applications, which allow for control of an avatar in 3D environment require two dimensional control (dimension one: forward – backward, dimension two: left – right). In case of one-channel device, the user’s satisfaction was higher when the game used only one dimensional input (meditation level). Moreover, using one channel BCI device for the control of avatar in 3D environment results in lower user experience as compared to using BCI device with 14 channels.

The comparison of Emotiv and NeuroSky devices while using reach 3D environment shows that the 14-channel Emotiv device is better suited for this purpose. These results clearly indicate that the user experience was determined by the combination of the type of device used and the requirements of the game played. It is therefore important to match the requirements of the BCI application with the BCI device.

The investigations presented in this chapter were based only on self-reports provided by the users and the sample group included mostly students of computer science who may be biased when assessing new technology. Moreover, performing parametric statistical analysis on rating measurements, such as Likert scales gives rise to methodological problems. For more information on limitations of using measurements based on ratings in human-computer interaction see [31].

The potential of replacing the keyboard, computer mouse or joystick with a BCI device as the main channel of interaction for computer games seems far away. Substantial development in signal acquisition and signal precessing will have to be made to make this goal possible. In the mean time, many interesting and highly beneficial BCI applications can be proposed. In section “BCI Games for Medical Applications”, we described a BCI game used for combating the symptoms of ADHD. BCI games for symptoms of other diseases can be imagined. One of them can be migraine and possibly epilepsy. Neuro-feedback based on slow cortical potentials regulation has been successfully user to stabilize brain activity and reduce the number of epileptic seizures [9] as well as to help control migraines [22]. It has, however, not yet been adapted for popular games to be used with commercially available BCI devices.