EEG-EOG based Virtual Keyboard: Toward Hybrid Brain Computer Interface

Abstract

The past twenty years have ignited a new spark in the research of Electroencephalogram (EEG), which was pursued to develop innovative Brain Computer Interfaces (BCIs) in order to help severely disabled people live a better life with a high degree of independence. Current BCIs are more theoretical than practical and are suffering from numerous challenges. New trends of research propose combining EEG to other simple and efficient bioelectric inputs such as Electro-oculography (EOG) resulting from eye movements, to produce more practical and robust Hybrid Brain Computer Interface systems (hBCI) or Brain/Neuronal Computer Interface (BNCI). Working towards this purpose, existing research in EOG based Human Computer Interaction (HCI) applications, must be organized and surveyed in order to develop a vision on the potential benefits of combining both input modalities and give rise to new designs that maximize these benefits. Our aim is to support and inspire the design of new hBCI systems based on both EEG and EOG signals, in doing so; first the current EOG based HCI systems were surveyed with a particular focus on EOG based systems for communication using virtual keyboard. Then, a survey of the current EEG-EOG virtual keyboard was performed highlighting the design protocols employed. We concluded with a discussion of the potential advantages of combining both systems with recommendations to give deep insight for future design issues for all EEG-EOG hBCI systems. Finally, a general architecture was proposed for a new EEG-EOG hBCI system. The proposed hybrid system completely alters the traditional view of the eye movement features present in EEG signal as artifacts that should be removed; instead EOG traces are extracted from EEG in our proposed hybrid architecture and are considered as an additional input modality sharing control according to the chosen design protocol.

Introduction

Human-computer interaction (HCI) is a revolutionary and challenging research field attracting multi-disciplinary research efforts. Currently, an explosion of interest in investigating every possible input from the human body as a source of meaningful information in order to develop usable, practical innovative systems that can reduce or even eliminate the communication barriers between man and machine and integrate modern technology in every aspect of human life.

Brain-computer interfaces (BCIs) represent the most novel HCI research direction in recent years. Basically, a BCI allows users to interact with computer applications or control devices through their measured brain activity signals. The idea was initially sparked by the motive of helping severely disabled people to combat their isolation and dependence resulting from neurological diseases such as strokes or Amyotrophic lateral sclerosis.

In particular, EEG is the most widely researched input modality for BCIs that are continuously motivating and inspiring researchers, resulting in a steadily increasing number of scientific publications starting from a few papers in the early 90's up to a large body of research published in recent years especially from 2007 to 2011 (Hwang et al. 2013). However, current BCIs are more theoretical than practical suffering from numerous challenges such as; frustrating performance and low Information Transfer Rate (ITR).

Recent trends in BCI research involve integrating multiple brain activity patterns in one BCI system or extending it by including other human bioelectrical signals such as Electro-oculography (EOG) and Electromyogram (EMG) (Hwang et al. 2013; Punsawad et al. 2010) to develop what is called a hybrid BCI system (hBCI) or (brain/neuronal computer interaction) BNCI (Allison et al. 2012).

Adding new bio-potential control signals to the BCI is in fact opposing the very first concepts based on which these systems were initially designed, as a BCI's traditional definition emphasis is on the sole use of EEG signals as neuronal commands and assuming the lack of any other potential source of input (Wolpaw et al. 2002). However, BNCIs are strong candidates for the new trend of BCI technology as it allows the system to benefit from any residual nervous or muscle activity of the user in order to increase the performance of the system and decrease common frustration rates.

To support this new trend, research involving human bio-potential signals other than EEG must be surveyed and reviewed to provide researchers with well-organized information that may be useful in designing successful and innovative hBCIs. This paper focuses on the use of EOG signals as a very attractive means of communication to external devices or applications, mainly due to its simplicity for acquisition and analysis, low cost and efficiency.

Although considerable research work related to EOG based HCI systems was published in various international journals and conferences, there are no literature survey papers that summarize and investigate thoroughly the general trends and characteristics of research in this area.

The paper introduces a survey and organization of EOG based HCI research articles, with a special focus on the "Eye Typing" using a virtual keyboard as our primary area of interest and the most widely researched application in both EEG and EOG based assistive applications.

First, we introduced EOG based eye tracking and recording principles and then the systems were categorized according to their design approach, in addition to their application areas. Our survey investigated the following aspects of research: The number of published research articles and their chronological timeline, target HCI applications and the signal processing and machine learning techniques employed to develop such applications and concludes with the challenges facing this technology. Then, we present a review of the currently developed hybrid EEG-EOG BNCI as a possible solution that uses EOG signals coupled with EEG as input modalities highlighting the shared control protocols employed. Finally, we conclude our survey with a discussion of the potential advantages of this hybridization with recommendations for future designs.

This paper presents a proposed system architecture for hybrid EEG-EOG Brain Neuronal Computer Interface (BNCI) system. The proposed hybrid system is based on extracting the Eye movement features from the EEG signals, completely altering the traditional view of these features as artifacts that should be removed from a conventional EEG based BCI system; instead EOG in our proposed system is considered as an additional input modality sharing control according to the chosen design protocol.

Electro-Oculography (EOG)

“Electro-oculography is a method of tracking the ocular movement, based on the voltage changes that occur due to the modifications on the special orientation of the eye dipole” (Desai 2013). The resulting signal is called the Electro-oculogram (EOG) and can be measured with electrodes placed around the eyes.

Using EOG signal to control computers can be compared to other biological inputs such as; EEG and EMG in the context of Human Computer Interaction. On the other hand, it can also be compared to various eye tracking techniques.

EOG Advantages and Disadvantages

There is no universal agreement on which bioelectrical signals would be best to use as a HCI input; it may depend on the intended application. However, EOG based HCI became recently one of the most prominent systems compared to other bio-potential based systems and this may be due to several reasons:

• 1) Convenience: Eye movement can be considered a convenient and natural source of communication which can be used by both normal and handicapped people.

• 2) Simplicity: The signal is easy to be acquired non-invasively and has simple characteristic patterns that can be easily extracted due to the signal amplitude range that is relatively higher than EEG and EMG. Simplicity also is reflected in the data acquisition system that can be designed at a considerable low cost.

• 3) Reliability: Even in the case of severe paralysis due to various neurological diseases, eye movements can still be a reliable source of control, enduring even in the worst conditions (Pinheiro et al. 2011).

On the other hand, EOG can be compared to other eye tracking techniques namely Infrared Occulography (IROG) and Video Occulography (VOG) (Singh and Singh 2012). Favoring one method over the other is basically dependent on the application but EOG based eye tracking systems may be favored principally for using body worn sensors for measurement as opposed to the other methods that require expensive sophisticated cameras or infrared transmitters and detectors. Therefore, EOG can control a low cost and low power embedded system which is particularly suitable for unobtrusive prolonged recordings and real time analysis, which is mandatory for today's ubiquitous computing and mobile eye based HCI scenarios (Bulling et al. 2009a).

The most important disadvantages relate to the fact that the corneo - retinal potential is non-stationary, even for the same person in different environments, which is shared by all bio-potential signals especially EEG. EOG is affected by numerous factors other than the actual movement of the eye, such as; lighting stimulations and user fatigue. Consequently, the repeated calibration process becomes mandatory to compensate for the DC drift. Additional difficulties arise owing to muscle artifacts (EMG), variation in skin resistance and electrode slippage or polarization (Carpenter 1988).

Characteristic Eye Movements

Human ocular movement, whether voluntary or involuntary, has been researched thoroughly in neuroscience and psychology, revealing very important discoveries in cognitive and visual processing. These studies indicate that the extraocular muscles work together to achieve four basic types of eye movements, described in detail in (Carpenter 1988): Saccades, smooth pursuit movements, vergence movements and vestibuloocular movements.

Particularly, an Eye Saccade is: “The movement of the gaze from one point to another point in the perceived space” (Postelnicu et al. 2012) so basically, it is the intuitive mechanism of communication in eye movement based systems and there identification plays the main role in EOG based HCI systems so far. In addition, blinks and winks were extensively investigated as voluntary control signals.

By classification of the main characteristics associated with various eye movements, control commands have been performed depending on the intended HCI application.

EOG Recording Principles

The human eye is polarized. Several experiments show that the cornea, the front part of the eye, is a positive pole while the retina, the back part, is a negative one. This potential is due to the increased rate of metabolism in the retina compared to the front of eye cornea (Malik and Ahmad 2007).

The EOG signal is the measurement of this corneo - retinal electrical potential difference, in the range of 2 to 20 mV but with recorded signal magnitude ranges from 15-200 μV. Modeling the eyeball as a single dipole directed from the retina to the cornea, EOG becomes a function of the position of the eye relative to the head and hence can be used to measure eye movement and then as a gaze input.

EOG recording is performed traditionally by placing five recording electrodes: One pair placed vertically above and below the eye in order to record vertical movements. Another pair next to the lateral canthus to record horizontal movements and one over the forehead or behind the ear, acting as a reference (Pinheiro et al. 2011). Figure 1 shows traditional electrode montage to measure EOG signals.

Fig. 1

The positioning of the electrodes on the face for recording the horizontal (H-CH) and vertical (V-CH) eye movements, the ground is used as the reference.

The polarity manifested at each electrode is attributed to the potential difference between the cornea and the retina, rotating the eye from the center position towards any of the electrodes results in a positive charge in this electrode as it is facing the cornea, as opposed to the other electrode which becomes the negative pole. This recorded potential difference becomes a measure for the eye position and with adequate calibration, can be mapped to the angular position of the eyeball in the range of 2 degrees in the vertical direction and 1.5 degrees in the horizontal direction (Young and Sheena 1975; Shaviv 2002; Brown et al. 2006; Chen 2003). Figure 2 illustrates the typical EOG signals generated by the corresponding eye movements.

Fig. 2

a The eye as a moving dipole. b EOG recording from horizontal eye movement (Barea et al. 2012)

As the conversion factor varies, calibration is required at the beginning of each recording to be able to translate A/D values into angular degrees. Conventionally, calibrated angular ocular movements are mapped onto screen co-ordinates for control of a mouse pointer (Gips and Olivieri 1996).

New Trends in EOG Data Acquisition Hardware

Going “wearable” and “mobile” inspired the investigation of novel electrode configurations to implement EOG based Eye Tracker Systems for mobile use as shown in Fig. 3. EOG for measuring eye movements is certainly a perfect candidate due to its simplicity, low cost and low computational complexity. One of the most innovative and appealing designs was implemented by Bulling et al. (Bulling et al. 2009a). They introduced a novel embedded eye tracker, wearable goggles based system equipped with dry electrodes, light sensor and analog amplification board. An accelerometer was integrated into the frame and EOG signal processing was performed in real time with a DSP component designed to fit in a pocket. Both storage capacities, for data recording and streaming via Bluetooth functionality were taken into consideration.

Fig. 3

Wearable EOG goggles (Bulling et al. 2009a)

The goggles based system designed by Bulling et al. was aimed at mobile HCI applications, activity recognition and context awareness, the design is particularly ergonomic, aiming to provide the user with comfort and to remove the manual procedure of placing the EOG electrodes around the user’s eye. Similar novel wearable designs were implemented in (Bulling et al. 2009a; Bulling and Roggen 2011; Iáñez et al. 2013; Kirbiš and Kramberger 2009). These studies contribute considerably towards efficient and ergonomic use of EOG for future HCI.

EOG Based HCI Survey

This section surveys existing research and approaches towards realization of important applications of EOG based HCI systems. Before doing so, the survey begins with a historical note on EOG based HCI research and how it is related to BCI and then HCI systems were categorized according to both employed design approaches and application areas. In addition, interesting statistics resulting from our survey that might give an insight on the research trends in the field were discussed. Finally, the survey highlights contributions of EOG based systems in various HCI applications dedicated for helping people with severe disabilities, with a particular focus on EOG based virtual keyboards as one of the most widely researched applications in HCI, based on both EOG and EEG which makes it an important candidate for hybrid BNCI systems development. Moreover, virtual keyboards are of particular interest to the authors.

EOG based HCI History

The revolutionary idea of controlling robots using EEG signals from a human head started in 1988 with control of a robot using EEG signals (Millán et al. 2004). The next year, 1989, the first control of a robot using EOG signals took place (Bozinovski 2014). It was Jack Vidal who started this challenge in (Vidal 1977). In 1973 Vidal stated the challenge of moving objects using bio-signals. In the challenge, besides EEG, he mentioned EOG, EMG, GSR and ECG. Vidal himself was the first one responding to his challenge and in 1977 he achieved the first EEG control of a graphical computer screen object (Vidal 1977). In 1989 the EOG part of Vidal’s challenge was achieved with the first EOG control of a physical object, a robot (Bozinovski 2014).

Scope

Our literature survey categorizes EOG based HCI systems according to the intended application, emphasizing on the system software such as the techniques used for EOG signal pre-processing and translation and with less focus on technical and digital hardware design details. Due to the simplicity of EOG acquisition, almost all the researchers designed and implemented their digital circuits for EOG acquisition systems. This made it difficult to efficiently evaluate and compare the different designs which enforced our decision. Moreover, our survey is intended to help researchers to integrate EOG as an additional input modality for hybrid BCI systems and therefore technical EOG acquisition details will not be useful as EOG will be acquired with additional electrodes in the EEG acquisition system corresponding to the BCI. It is hoped that this paper will provide an insight into EOG based HCI system design methodologies and the potential benefits that might be acquired from integrating EOG to EEG in HCI.

EOG based HCI Categorization

EOG based HCI can be mainly categorized according to 2 principles: Application Area and the HCI System Design. Figure 4 illustrates a schematic diagram for the proposed categorization.

Fig. 4

EOG based HCI systems categorization

EOG based HCI systems according to the design approach

In the design of EOG based HCI systems, there are basically two different approaches that have been used: Pattern recognition and non-pattern recognition approaches (Aungsakun et al. 2012).

In the first approach, the EOG based HCI system is considered a typical pattern recognition system that relies on tailored signal processing techniques to extract and classify characteristic eye movement patterns for control and communication purposes. Figure 5 illustrates the general architecture of this approach which is practically the same approach for almost all EEG based HCI systems or BCIs.

Fig. 5

Basic elements of pattern recognition EOG based HCI

The second approach avoids using complex pattern recognition techniques. It takes advantage of the simplicity of EOG signals and relies on the signal characteristics, namely polarity and voltage level, in classification. The classifier module of the pattern recognition algorithm is replaced by a simple threshold comparison module to avoid implementation complexities and to minimize computational time especially in real time implementations, i.e. on microcontrollers. Eye-gaze Systems based on EOG is considered a major part of the Non-pattern recognition approach. Eye-gaze systems are based on directly mapping gaze input and eye angular movements through EOG signals instantaneously to the corresponding application parameters, such as; screen co-ordinates for eye typing and cursor movement applications (Gips and Olivieri 1996) or a wheelchair steering function for wheelchair navigation applications (Barea et al. 2002a).

EOG based HCI systems according to the application area

Typically, 3 major EOG based HCI application areas can be identified to have a great impact in helping the disabled community thrive, namely, "Communication and Control", "Motor Substitution" and "Entertainment". Table 1 summaries EOG based HCI references categorized according to both application area and the corresponding design approach.

Table 1 EOG-based HCI references categorization

This paper focuses on Communication and Control through Eye typing in section 4, highlighting software design issues and performance evaluation. Thorough analysis of the challenges and shortcomings is presented in section 5 before discussing the proposed hybrid EEG-EOG BNCI design as a potential solution in section 6.

Literature Search and Interesting Statistics

To survey the literature regarding EOG based HCIs, we consulted Google Scholar and Thomson Reuters's Institute for Scientific Information (ISI) Web of Knowledge, a database providing the largest collection of research data (Jo 2013). Figure 6 show the number of published EOG based HCI research articles from 1998 to 2015 in International journals and conferences.

Fig. 6

The number of EOG-HCI published research articles for each year from 1998 to 2015.

The graph relatively shows that the number of publications is very small when compared to the EEG research ongoing in the BCI field. The maximum number of publications was 17 research articles reached in 2009 and in 2012 while the number of published articles in BCI reached more than 250 only in 2011 according to the same database (Pinheiro et al. 2011).

One interpretation may be that the idea of controlling computers using only thoughts has a greater appeal to a large number of dedicated BCI labs and research groups all around the world, in addition to the wide range of data available through BCI competitions to motivate researchers in order to be able to decode complex EEG signals into a useful and practical way of communication.

In the published EOG research, many practical applications were developed with the purpose of helping people with different degrees of disabilities. Figure 7 displays the percentage of each type of application based on our survey, in the published EOG based HCI work during the last 10 years.

Fig. 7

The percentage of each type of application in the EOG based HCI publications during the last 10 years.

According to our survey, research focused primarily on two applications; namely, communication and spelling using virtual keyboards and wheelchair navigation. Both applications contributed with 39% and 33% respectively of the total number of EOG based HCI applications developed in the last 10 years. The remaining percentage was divided between the 10% for Robot arm control applications and 18% Entertainment applications such as controlling TVs or visual navigation.

Interestingly, we found that almost 76% of the published papers in EOG based HCI during the last 10 years followed the non-pattern recognition paradigm and were based mainly on simple threshold comparison modules. This can be considered an important indication of EOG signal simplicity in contrast with EEG which requires complicated feature extraction and classification methods.

EOG Based HCI for Communication and Control: Eye Typing

Communication through typing is of crucial importance for severely handicapped people to express their needs to caregivers. In many disabilities, especially severe paralysis due to neurological diseases, eye movements tend to be the last controlled movement available, which makes Eye Typing an extremely intuitive, convenient and practical communication option.

Basically, there are two methods for EOG based Eye typing Systems to produce text: The first approach is performed by continuously reflecting and mapping the focus of the eye gaze through the EOG signals, This may also be called: The "Eye Gaze" system. The second approach is performed by translating discrete patterns of characteristic eye movements, usually specified eye movement directions, then using it as input commands to produce text on a specified typing interface.

Eye typing interface is typically an on-screen virtual keyboard, which is simply an alphanumeric matrix or conventional keyboard layout displayed on screen allowing the user to navigate through the letters for typing selections.

To navigate through letters in Eye gaze systems, EOG measurements are calibrated and directly used to map the eye position and eye movement on the screen as if the eyes are replacing a conventional mouse. In the classification of eye movement directions, basically, up-down, left-right and center, these are translated to corresponding one step cursor movements on the virtual keyboard instead of mapping the sight angle.

After being able to navigate through the virtual keyboard, the character selection process can be performed by a simple focus on the corresponding character. Focusing can be done by fixating the gaze on one of the virtual keys on screen, i.e. staring at the keyboard key for a relatively short period before selection and waiting for a feedback to confirm this selection. This feedback response can be achieved by showing a cursor or simply highlighting the fixated letter.

Then the typing system must provide a "selection" command which may be by intentional blinks or using "dwell time", which is the duration for continuous fixations on the selected button, as an activation command. The maximum dwell time is usually about one second (Majaranta and Räihä 2002).

Eye typing literature review: EOG based Virtual Keyboard

One of the most important Eye Gaze systems in the literature is Eagle Eyes (Gips and Olivieri 1996) and is almost the only practical implementation of an EOG based HCI system for Eye Typing. This system is being successfully used in the Campus School at Boston College, USA for helping severely disabled patients to communicate by operating almost every possible commercial application using only their eye movement. The system is based on direct mapping of EOG signals to magnitudes representing the co-ordinates of the cursor location on the computer screen. The software scans the amplifier every 1/60 of a second to translate the Eye movement values to the corresponding mouse co-ordinates.

The selection process is performed using the “dwelling time”: which means that the cursor remains on a pre-specified location on screen for an adequate time. Feedback is given by a “mouse click” to insure the user his selection was processed successfully.

Learning to use the Eagle Eyes is reported as an acquired skill that a normal user may take up to 15 minutes in order to master spelling his name using a virtual on-screen keyboard (Gips and Olivieri 1996).

Reported rates for professional people were about one character per second. Eagle Eyes doesn't completely simulate all mouse functionalities such as holding the mouse down and issuing a double click. Another major challenge that is still facing Eagle Eyes and practically all developed attempts for all Eye Gaze cursor movement systems in general, is the "Midas Touch Problem" where a major conflict is created by using the eye as an input method for commands, continuously preventing the user from simply "looking". In this case, feedback can be given falsely when the user fixates his gaze to obtain information and the system confuses it with a selection command which leads to failure in communicating the user’s will to the system and unwanted responses.

In (Tecce et al. 1998) EOG feasibility for controlling virtual keyboards using continuous "eye gaze" was demonstrated. 14 normal volunteers' horizontal and vertical EOG signals were recorded. The on-screen virtual keyboard was displayed as a matrix of characters shown in Fig. 8 and the cursor used for selection was represented by a black square. Users attempted to select letters by controlling the movement of the black square with their eyes and trying to position it in one of the cells containing the character. Eye movements were mapped to their corresponding position on screen via an “exponential smoothing function” and the feedback was given to the user after a “dwelling time” of at least 50% of the time required for the previous 100 samples. The system provided auditory selection feedback and included correction options. Only three letter words were tested in word and sentence spelling experiments. An average speed of one character per 2.6 s was reported to exceed almost 10 times BCI processing speed (Farwell and Donchin 1988).

Fig. 8

Virtual keyboard (Tecce et al. 1998): Matrix layout of characters. The spelling task: "Get the car" is presented to the subject at the bottom of the matrix.

Besides Eye gaze systems, Eye typing can be based on identifying characteristic eye movements in specified directions. The main advantage over Eye Gaze systems is not using eye movement continuously as an input method thus it reduces the "Midas touch" problem. It was LaCourse and Hludik who first proposed the discrete Electrooculographic control system (DECS) in (Lacourse and Hludik 1990) to help the disabled. The system did not implement a virtual keyboard but simply proposed the study of EOG signals during discrete eye movements to reach specified targets on screen. The plan of the rotational movement of the eye was divided into quadrants and targets were arranged in discrete positions. This resulted in the standard 8 directional movements shown in Fig. 9. The authors reported almost 100% accuracy for identifying targets 0,1,3,5,7 but much lower accuracies for targets 2,4,6,8 which they attributed to head movements.

Fig. 9

Target test patterns in (Lacourse and Hludik 1990).

In (Hori et al. 2006) Hori et al. designed the virtual keyboard based on the eye movements in the basic 4 directions: Up, down, left and right in addition to one selection command using voluntary eye blinks as a mouse click. Their system was aimed to be a simple and efficient measuring of EOG only from 2 electrodes plus a reference electrode.

The reduced number of channels in this study was proposed for usability and operability and was compensated by the logical combination of horizontal and vertical EOG channels according to pre-specified upper and lower threshold bounds to extract the features of the 4 directional movements and then output step by step cursor movements on the virtual keyboard. The system performance was extremely dependent on the threshold setting which was specified during preliminary experiments searching for optimum threshold values that maximize the system accuracy; the performance reported for a 12 letter experiment was almost 94% accuracy, and average typing speed of 8 letters per minute.

In (Kherlopian et al. 2006) Kherlopian et al designed “Telepathix”. The system introduced a new design; the "ternary" keyboard shown in Fig. 10, the idea was to divide the keyboard into 3 sections: Right, left and center, requiring only 3 basic eye movements to navigate between them. Once a section is selected, the area of selection is iteratively divided to 3 sections and the splitting process remains until only one letter is selected with one of the 3 eye movements. These iterations of selections can be stopped when the user types a “stop sequence” by selecting 2 blank spaces.

Fig. 10

Ternary EOG based virtual keyboard (Kherlopian et al. 2006) .

In Telepathix, the 3 eye movements were classified based on selecting representative and selective features of the EOG signal corresponding to the desired eye movements. Features included Polarity, Slope, Mean Value, Threshold, Command Timing, Peak duration and Correlation of Model Peak. The authors employed a Hierarchical Clustering Procedure and 5 letter words were tested for spelling. Results reported an average typing speed of 6 letters per minute after users were trained, which was significantly slower than Eye gaze systems depending on direct EOG mapping on screen. This low speed of typing disadvantage affects user comfort significantly and is due to the relatively high number of commands required to select a single letter. In this regard, the “ternary” design was not beneficial.

In (Yamagishi et al. 2006) Yamagishi et al. discussed how the limited number of eye movements is the main reason for the slow performance of EOG based Eye typing systems and proposed combining 4 directions to the basic 4 directions, namely: Up, down, right, and left combined with: Up right, up left, down right and down left. The selection was also using intentional eye blinks as a mouse click. The EOG was measured with 2 electrodes as in (Hori et al. 2006). The proposed 8 directional cursor movements were extracted from logical comparisons of both channel values versus threshold settings tailored to each individual according to corresponding trials.

In their results, they reported that using 8-directional cursor movements instead of 4, increased the typing speed from 10 to 12 letters per minute while the accuracy of writing remained the same, almost 90.4%.

In (Borghetti et al. 2007) Borghetti et al also designed a step-by-step eye typing system that interpreted eye movements based on the continuous evaluation of parameters of eye potential variations such as: “peak amplitude”,” polarity”, “slope” and “duration”. Thresholds set on these parameters enabled differentiation of eye movements in the basic 4 directions: Left, right, up and down in addition blinks. The interface of the system included an alphanumeric selection matrix displaying capital letters (A-Z) and numbers (0-9). The user begins the selection process by directing his gaze into one of the 4 directions and does not expect a corresponding cursor movement before a verification step that involves a fast return of his gaze in the center position of the screen. The interface allowed online setting and changing of the voltage thresholds and eye movement identification parameters. Letter selection was performed using double eye blinks to avoid possible errors during involuntary blinks, users were given visual feedback and no spelling correction was allowed.

After practice sessions that lasted around 5 minutes, subjects were able to achieve an average typing speed of one character per 8.5 sec. in typing experiments that involved writing words of 2-5 letters, for a total of 12 words per subject. The typing speed was found highly dependent on the keyboard layout and the distance between subsequent characters in the word was more than the number of letters per word, this may be due to the verification step requirement which reflects on the total number of eye movements required to reach a certain letter, especially when it is located relatively far from the previously typed letter. Although the typing speed in this system was more than 66% slower than (Tecce et al. 1998), it still considerably exceeds the speed reported for BCIs at this time (Farwell and Donchin 1988).

The authors argued that their low system speed was counter balanced by their great system reliability, as there was almost no need for correction of wrong inputs, even if the eye movement was misinterpreted by the system, the double blink system prevented this wrong input. The only source of error was due to the wrong selection of characters or misspelling and amounted to almost 0.36%.

In (Tsai et al. 2008) Tsai et al. proposed the use of the eyes in writing directly without relying on a virtual keyboard. The system was based on recognizing the symbols that the user 'writes' by moving his eyes to trace the paths of predefined stroke patterns. The symbol set was the 10 Arabic numerals and 4 mathematical operators. The recognition of the eye written symbol was based on the similarity between the measured EOG waveform and the standard EOG waveform corresponding to that symbol.

Classification was performed using clustering first, then identifying the symbols from within the clusters using Artificial Neural Networks based on Linear Vector Quantization (LVQ). The authors suggested that this system has the advantage of portability and low cost as there is no screen and no virtual keyboard required but it cannot be considered an actual typing system in the sense of composing a message for communication. Performance was reported in terms of dependability and feasibility for each symbol. The recognition rates were highly variable for different symbols ranging from 50% to 100% and no speed of typing was reported.

Important research effort was done by A.B. Usakli in (Usakli and Gurkan 2010) the design strategy was also not involved in detecting the sight angle, only right-left, up-down eye movements and blinking were classified online using the Nearest Neighborhood algorithm (NN) directly on EOG time series (with Euclidean distance) without any other preprocessing or feature vector calculation. NN was chosen for its simplicity and time efficiency which is mandatory for real time implementation.

A simple user interface was designed, displaying the real-time ongoing EOG for both horizontal and vertical channels and the system utilities (Fig. 11).

Fig. 11

User Interface main menu displaying various system utilities: Upper left: Virtual Keyboard; upper right: Movement navigation; Lower menu: Communication with care givers options (toilet, bath, drinks, meals, medical, sleep, family, stroll) (Usakli and Gurkan 2010).

The user can navigate through this menu using his eye movements and when the cursor is positioned on the selection, the user confirms with an eye blink. Involuntary eye blinks are not identified by the system as selection commands. After the virtual keyboard selection from the main menu, two options are available for display as a sub-menu: A standard virtual PC keyboard and the classical BCI P300 speller keyboard proposed in (Fazel-Rezai et al. 2012). By using these keyboards (Fig. 12), results reported a speed of 5 letters/25 seconds for random five letter words and the classification performance was 95% (Usakli and Gurkan 2010).

Fig. 12

Left: Standard virtual PC keyboard with special characters. The message shown on the bottom line was written in 148 s. Right: BCI P300 Speller Virtual Keyboard with an added row to increase efficiency (Usakli and Gurkan 2010).

The author reported that EOG signals can discriminate between eye blinks and the basic eye movements within the 4 directions: Up, down, left and right without the need of complex and lengthy classification algorithms as opposed to EEG signals. One of the advantages of the developed system was the presence of digit characters on the virtual keyboard. Moreover, the options presented for possible communication needs with care givers such as: Meal, bath, stroll, etc. An average response of 3 seconds was reported for a "cleanup" command.

In (Usakli et al. 2009) Usakli et al. highlighted how the progressive nature of motor impairment illness can lead to complete paralysis and even loss of eye movement ability. He proposed the use of possible residual ability of the patient especially EEG based control which doesn't require any sort of motion. He didn't develop a complete hybrid system but he proposed the idea of a system that switched between the two modes of control according to the user’s needs and performance at a specific moment. As a step, he only compared his developed EOG based keyboard with the classical P300 based BCI speller and reported the system based on EOG more efficient in terms of accuracy, speed, applicability and cost efficiency. The speed reported for the EOG based virtual keyboard was almost 25 seconds to spell the word "water" with an accuracy of 95% compared to 105 seconds using an 8 channel EEG based virtual keyboard with an average accuracy of 80%. In terms of user friendliness, subjects needed a 5 minute session to use the EOG based system efficiently; compared to a 30 minute session in case of the EEG based system.

In (Nathan et al. 2012) D. S. Nathan designed an EOG based virtual keyboard where users can type using eye movements in 8 directions: Up, down, left, right, up-right, up-left, down-right and down-left. They proposed a new matrix based keyboard layout, with indexed rows and columns. Using this layout, any character can be referenced by its row-column index and hence typed using only 2 eye movements compared to the number of eye movements conventionally required in a ‘QWERTY’ design, which can vary from 1 to 9 eye movements to actually move the cursor step by step in order to reach the desired letter.

Their proposed keyboard design did not rely on cursor navigation on the letter cells of the matrix but instead, on referencing each letter by its row-column index. These indices were located in the 4 corners of the screen in addition to the midpoint between each 2 corners; hence 8 indices were labeled on the screen in positions corresponding to the 8 directions of eye movements: Up, down, left and right (Fig. 13). Green highlighted buttons corresponding to each index were designed for feedback. Users simply choose letters by performing only 2 eye movements for looking at the green buttons corresponding to the row-column indices of the chosen letter. Identification of the specific EOG patterns corresponding to the eye movements were based on setting threshold values within a specified time frame for both positive and negative components. This time frame constraint increased the robustness of signal identification by ensuring the occurrence of the EOG components within a specified period of time, in order to solve the problem of subject specific thresholds that may vary over time, Threshold values were determined through calibration. The performance measures reported a typing speed of 15 letters per min and 95.2% accuracy.

Fig. 13

D.S. Nathan proposed system with the new keyboard layout (Nathan et al. 2012).

In (Soltani and Mahnam 2013) a miniaturized wearable system mounted on a glass was designed as in (Bulling et al. 2009b). The system recognized 8 directional eye movements and used them to navigate in a high performance new graphical interface for a virtual keyboard, which enables users to select letters by just 2 moves and 2 selections. The selection was performed using eye blinks.

Classification was performed using adaptive thresholds on the derivative of the horizontal and vertical signals. Figure 14 shows the newly proposed keyboard. The developed system was low cost, light weight, with a high level of mobility and comfort. The reported performance was 5.88 letters per minute.

Fig. 14

The proposed keyboard design in (Soltani and Mahnam 2013). (a) Main page of the keyboard consists of 9 groups of letters and numbers. (b) The second page of the keyboard divides the group of selected characters into 9 groups including special characters.

In (Lopez et al. 2014) Lopez et al. designed a virtual keyboard based on Eye gaze detection as shown in Fig. 15; the user was required to continuously perform eye movements in the required gaze direction until the cursor was positioned on the desired key. The selection was performed using eye blinks as a left mouse click by default.

Fig. 15

The proposed keyboard design in (Lopez et al. 2014) with the dedicated buttons for mouse operation on the left

In their design, the authors proposed the addition of dedicated buttons for the mouse operation, such as; right click and drag, to the standard keyboard. These buttons can be selected by eye gaze, as with the rest of the keys of the virtual screen keyboard. If the user wants to perform a right click, he must first select the right click button then the eye blink command will be translated into a right click command. As EOG varies from one user to another, the system included software for EOG repeated calibration and also allowed manual calibration and parameter settings such as; mouse speed and double click speed. The reported performance was quite slow; the average typing rate was 4 letters per minute. No explanation was given for this slow rate, especially when compared to the other Eye gaze systems (Gips and Olivieri 1996; Tecce et al. 1998).

In (Ang et al. 2015) Ang et al. designed a cheap, user friendly, requiring no training, Eye Typing system based on controlling cursor movements on an on-screen Windows 7 standard keyboard. A single channel commercial, ergonomic, wearable EEG device, the NeuroSky Mind Wave Mobile Headset was used to measure the data.

The authors chose double blinks to control and guide an automatically moving cursor on screen. Their testing experiments on 8 subjects both in indoor and outdoor conditions demonstrated high inter-subject consistency of EOG signals due to double blinks. Their system was trained on the data of a single subject and then tested on the remaining subjects with no training. Identified double blink segments were adapted to advanced signal processing methods such as; wavelet filtering and SVM classification to compensate for the low SNR due to the single channel commercial device. The average performance reached 84.42% accuracy, testing speed was slow, an average of 3.3 per min. which was argued due to the high computational complexity of the system.

In (Teja et al. 2015) a wearable dry electrode mask was used to record EOG data in an Arduino interfaced system. A simple multi-threshold algorithm was used to classify blinks, left and right movements. Their keyboard design was based on automatically highlighting character cells vertically in a rate of 1 character per second. User selection of the highlighted character was performed using an eye blink. To control the highlighter column, users moved their eyes to the left or right outside the monitor’s dimension. Average accuracy and speed was reported to be 100% and 5 characters per minute respectively.

EOG based Virtual Keyboard: Summary

Considering EOG as the input modality for communication using a virtual keyboard is very promising and efficient in terms of typing rates and accuracy. Using Eye gaze directly to type a message reached a 1 character per second typing rate in the Eagle Eyes system (Gips and Olivieri 1996) but as the Eye is being continuously used as both an input modality and the organ used for sight, problems such as "Midas Touch" affect the usability and practicality of the system. Other Eye typing systems based on specific eye movements used discretely to navigate on the keyboard, reached 15 characters per minute which is still considered a considerably high speed when compared to current BCI systems developed for the same purpose (Farwell and Donchin 1988; Fazel-Rezai et al. 2012; Cecotti 2011).

Table 2 compares the performance of EOG based spellers as discussed above and highlights the advantages and disadvantages of every system.

Table 2 EOG-based spelling: Summary and comparison.

EOG based HCI Challenges and Unresolved Problems

Based on our survey, several challenges face this promising technology:

• Almost all the developed non-pattern recognition systems based on threshold comparison modules and signal manipulation, directly confront the fact that EOG, as any bioelectrical signal, is continuously changing even for the same person. Calculated thresholds and system performance may face change with time. Proposed solutions rotate around the calibration and recalibration of the EOG at least once in the beginning of the recording which may not be practical in some cases. Artificial intelligence and pattern recognition techniques may solve this problem to some extent but the problem of complex real time implementation will remain another challenge.

• Most of the EOG calibration is performed manually. This may be acceptable practice in clinical tests that use the EOG; this restriction hinders the EOG from being used independently as a control and communication tool by people with disabilities.

• An important challenge is facing EOG based HCI systems, "The Midas Touch" problem. This complication arises as the human eye is also considered a source of information or sensor when this sensor is used also for control commands. Users cannot control whether they really want to issue a command or they are simply looking at the interface. Solving this challenge is a mandatory issue for reliable and practical EOG based HCI systems.

• Another challenge is due to prolonged use of the eye as an input controller may easily accumulate fatigue on muscles and cause performance degradation due to problems such as; "eye jitter", numerous eye gazing at single objects, possible movements due to uncontrolled reflexes and maintenance of corresponding equipment.

• Unless eye movement is directly mapped on the screen and used as an eye tracking method, the number of usable eye movement patterns is usually quite limited, with a maximum number of 8 movements in addition to blinks, winks, frowns and fixations. This makes the HCI system limited in the number of the supported control commands and may not satisfy a multi-task situation.

Towards EEG-EOG Hybrid BCI

Against the challenges facing EOG based HCI, a hybrid system that combines EOG with EEG as an input modalities is expected to make full use of the advantages of both systems and avoid the disadvantages. BCI systems based on EEG signals are very popular in literature; they received the biggest interest among all the bio-electrical signals based systems as they involve control with only thoughts, without any trace of physical movement which may be the only solution for the most severe disability cases where people may even lose their ability to control their eyes. However, current BCI systems suffer from low performance and slow Information Transfer Rates (ITR) and rather theoretical than practical. One reason is because EEG is a very complex signal that requires sophisticated signal processing techniques and complex pattern classification approaches. Moreover, a BCI usually can handle one type of task and handling different kinds of situations leads to significant performance degradation.

Comparison between EEG and EOG Based Virtual Keyboards

In order to get more insight on the potential advantages of a hybrid EEG-EOG BCI it is useful to compare the performance of both input modalities in our application of interest: The virtual keyboard.

Spelling using EEG has been extensively investigated and developed in the BCI literature (Fazel-Rezai et al. 2012). Three underlying EEG electrophysiological phenomena were mainly used to control an EEG virtual keyboard namely: P300 Event Related Potential (ERP), Steady State Visual Evoked Potentials (SSVEP) and Motor Imagery, a good comprehensive review can be found in (Cecotti 2011)However most recent research for utilizing these potential in visual stimuli driven BCI systems are reviewed here for their benefits.

(Zhang et al. 2012) Introduces a novel BCI based on oddball paradigm using stimuli of facial images with loss of configural face information (e.g., inversion of face). The proposed novel paradigm investigated the effects of ERP components N170, vertex positive potential (VPP) and P300 on target detection for BCI. The proposed method used the linear discriminant analysis (LDA) without complicated feature extraction processing to classify eight classes. The online classification accuracy reached 88.7% with information transfer rate (ITR) of 38.7 bits/min using stimuli of inverted faces with only single-trial which reflects the efficiency of the proposed paradigm for the visual stimuli driven BCI applications.

(Zhang et al. 2013) Introduced a spatial-temporal discriminant analysis (STDA) to ERP classification. As a multiway extension of the LDA, the proposed STDA method tries to maximize the discriminant information between target and nontarget classes through finding two projection matrices from spatial and temporal dimensions collaboratively, which reduces effectively the feature dimensionality in the discriminant analysis, and hence decreases the number of the required training samples. STDA effectively reduced the system calibration time and enhanced the classification accuracy, thereby validating the practicability of ERP-based BCI.

(Wang et al. 2016) Presented a new approach that utilizes Spatio-temporal feature extraction with multivariate linear regression (MLR) to learn discriminative SSVEP features to enhance the detection accuracy in real-time SSVEP-BCIs. MLR is applied on dimensionality-reduced EEG training data and a constructed label matrix to find optimally discriminative subspaces. The proposed MLR method significantly outperforms other competing methods for SSVEP detection, especially for time windows shorter than 1 second.

In (Jiao et al. 2018) Multiset canonical correlation analysis (MsetCCA) has been successfully applied to optimize the reference signals by extracting common features from multiple sets of electroencephalogram (EEG) for recognizing the steady-state visual evoked potential (SSVEP) in BCI applications. A sophisticated extension of MsetCCA, called multilayer correlation maximization (MCM) model is proposed in this research for further improving SSVEP recognition accuracy. MCM combines the advantages of both CCA and MsetCCA by carrying out three layers of correlation maximization processes.

Current EEG virtual keyboards are far from practical use, suffering from poor performance in terms of both accuracy and speed. Future trends highly suggest focusing on different research directions to fulfill the BCI promises to the disabled community. In addition to the traditional focus on signal processing and machine learning to elicit underlying EEG phenomena and enhance performance, next generation virtual keyboard BCIs are expected to explore hybrid control mechanisms and innovative graphical user interfaces to release the current bottleneck (Orhan 2014; Williamson et al. 2009).

Performing a thorough literature review on EEG based virtual keyboards is not in the scope of this paper, however, a general comparison is required to highlight the potential advantages of the new hybrid system. Table 3 provides a quick comparison between EEG and EOG as input modalities for virtual keyboards.

Table 3 Comparison between EOG and EEG input modalities for virtual keyboard

EEG-EOG Hybrid BCI Design Protocols

Hybrid EEG-EOG BCI (hBCI) or Brain/Neuronal Computer Interfaces (BNCI) using EOG are especially useful in disability cases where there are residual eye movements (Allison et al. 2012) in order to benefit from every possible remaining movement they've got to gain a more efficient and robust level of control and avoid frustration.

Hybrid BNCI is considered an entirely new approach to designing a BCI system which traditionally relies only on brain activity. Fusing EEG with other bioelectric inputs requires a thorough understanding of the characteristics of the used input modalities and new creative design approaches to maximize the advantages of combining these input mechanisms together in a shared control environment. In addition, it requires new strategies to deal with possible emerging challenges.

A hybrid BCI design protocol can be “simultaneous” and/or “sequential” (Pfurtscheller 2010). In a simultaneous protocol, the user can operate the BNCI by using the chosen input mechanisms simultaneously. In a sequential design protocol, the input mechanisms share control of the system in an alternating manner, each one is controlling only part of the time based on the user's choice to avoid a problem or to increase usability. In that case, the secondary input mechanism should compensate for some weaknesses of the primary one (Allison et al. 2012). Figure 16 depicts a schematic diagram for the BNCI design protocols.

Fig. 16

Hybrid BCI Design Protocols

Current Hybrid EEG-EOG BNCI System Challenges

One of the main goals of this paper is to guide and inspire the design of new BNCI systems based on both EEG and EOG signals focusing on our application of interest: The virtual keyboard.

According to our survey, only 3 EEG-EOG virtual keyboards were proposed (Postelnicu and Talaba 2013; Koo et al. 2014; Lee et al. 2018). Both systems basically shared the same design concept, which is using the EOG input initially to reduce the number of chosen letters by half before using the P300 EEG component for finally selecting the required letter. Both systems reported approximately no change in accuracy after the fusion of EEG and EOG input modalities but highlighted a significant improvement in speed.

In (Postelnicu and Talaba 2013) a hybrid P300 EEG-EOG virtual keyboard was designed based on one of the most successful improvements of the classical P300 EEG Speller: The checkerboard paradigm (CBP) (Townsend et al. 2010). A novel paradigm for letter display was proposed, the “HCBP” half checkerboard paradigm. Postelnicu et al. based the “HCBP” design on splitting the CBP letters matrix in half, horizontal EOG was used initially to identify the area were the user gazed at and then only the corresponding symbols were flashed. In this manner only half of the symbols were considered based on the gaze direction determined by EOG signals and then the character selection process was based on the classical approach of classification of EEG p300 potentials. This splitting was expected to reduce the time required for a single symbol selection by almost the half. For evaluation the proposed “HCBP” paradigm was compared to the CBP paradigm in an online experiment. Results indicated similar mean accuracies for both interfaces, almost 90% and favored “HCBP” in terms of speed reporting 4.17 characters/min comparing to 3.24 characters/min for the “CBP”.

In (Koo et al. 2014) B. Koo et al. extended the idea of using EOG initially to reduce the number of stimulus or selections by half by designing a novel hybrid EOG-P300 BNCI system with dual monitors. The system first analyzes EOG to identify which monitor was focused on by the subjects and then, in the monitor, the P300 system identified the item focused by the subjects. The system was compared to conventional P300 for evaluation. Classification accuracies were almost the same but the hybrid BNCI reported almost 70% higher in practical bit rate (PBR), a performance measure defined in (Townsend et al. 2010).

Recently in (Lee et al. 2018) proposed the row-column visual feedback paradigm (RC-VF), to solve the problem of single trial P300 EEG spellers, by presenting visual feedback of the classification results for the user. The system allows users to actively select the target letter using a simple eye movement such as a wink in order to minimize the amount of EEG required in order to elicit the ERP for target letters. The accuracy of the proposed system reached 97.6% with 8.2s required for the spelling of each character.

Other hybrid EEG-EOG BNCI systems were developed for other applications such as machine control (Punsawad et al. 2010; Ma et al. 2015), GUI control (Iáñez et al. 2011; Jiang et al. 2014), robot arm (Postelnicu et al. 2011; Witkowski et al. 2014) and wheelchair control (Wang et al. 2014; Taylor et al. n.d.). In (Punsawad et al. 2010; Iáñez et al. 2011), EEG and EOG were combined to perform more complex tasks such as increasing the number of commands or adding more control options to the system. In (Ma et al. 2015; Postelnicu et al. 2011; Wang et al. 2014) both signals shared control in order to add a new dimension to the system functionality and relieve the user from continuously using one type of input modality which eventually causes stress, fatigue, frustration and hence performance degradation. In (Jiang et al. 2014; Witkowski et al. 2014; Taylor et al. n.d.) EEG was used as a secondary input to validate EOG control commands, giving the user more flexibility in moving his eyes and increasing system safety and reliability. Finally, in (Crea et al. 2018), a hybrid EEG-EOG brain/neural hand exoskeleton was tested to restore hand function to quadriplegics, for fluent, reliable and safe operation of a semi-autonomous whole-arm exoskeleton.

In general, current hybrid EEG-EOG BNCI systems for spelling reported challenges were the need to consider recalibration pauses of the EOG part of the system after every few spelled characters in order to accurately determine the user gaze direction. Unintended eye movements can present also a challenge for the system, which might be solved by increasing the number of channels used to record EOG separately to acquire more reprehensive information about the current movement. Another issue of great importance is that these systems are highly dependent on the user’s eye gaze control capabilities, which probably makes these systems less useful for patients. People suffering from neurodegenerative diseases, such as Amyotrophic Lateral Sclerosis (ALS) especially at later stages tend to lose their eye gaze control function. One of the possible solutions for that issue is the development of multi-modal systems that could include more than one recording modality for these challenging cases (Hong and Khan 2017). Finally, it should be noted that identifying EOG traces in EEG implies some design issues such as considering that the underlying EEG signatures used for operating the hybrid system will not be masked by the frontal EOG signatures. Generally, visual P300 ERPs are detected are prominent in the Parieto-occipital areas of the brain which makes them a good candidate for a hybrid system

Hybrid EEG-EOG BNCI System Potential Advantages

In the following, we attempt to highlight the advantages of combining both input modalities in order to gain an insight of the potential benefits of integrating them in a single BNCI system.

• 1) Combining EEG and EOG provides a solution to the "Midas Touch" problem as the eye won't be the sole method of control. EEG may be used as a switch to trigger EOG control mode. Another way might be to use EEG signals for EOG command validation, which may give the eye the freedom to look around as long as there is no command that could be validated using EEG.

• 2) Sharing control between EEG and EOG helps reduce the stress and fatigue exerted on eye muscles from repeatedly using eye movements as an input and also any possible mind stress caused by continuous concentration. The BNCI will then increase the system usability, practicality and performance.

• 3) The diversity of input modalities adds new dimensions to the kinds of tasks handled in a single application. The limited number of eye movements definitely benefits from other EEG characteristics such as; motor imagery patterns or event related potentials to support multi-task situations, through new designs that take full advantage of both input modalities.

• 4) Accurate and fast response of EOG as a control input contributes significantly to BCI performance. Users who still have eye movement control will have greater chances to experience the new input modality and have a more satisfactory control experience.

• 5) Eye movement commands may be more intuitive and easy to associate with various control commands. For example, right/left eye movements can be easily associated with turn left/right in a wheelchair control experiment, rather than imagination of right/left arm movement. This will probably reflect on the user experience, the system usability, training time and hence the performance.

EEG-EOG Hybrid BCI System Recommendations

Generally, we envision a practical hybrid system with based on extracting significant EOG features from EEG data, to avoid potential burden on the user from using an extra input modality. For EEG data acquisition, an ergonomic low cost measurement device that doesn’t require any setup burden could be used (Emotiv Systems 2014). Then, methods that were traditionally used for EOG artifacts rejection/correction could be explored for identifying EOG traces in EEG, such as ICA which particularly stands out among the others to separate efficiently EEG related components from the EOG related ones (Choi et al. 2005; Lee et al. 1999).

The hybrid system should be able to extract eye movement features from the EEG signals, completely altering the traditional view of these features as artifacts; instead EOG is proposed as an additional input modality sharing control according to the chosen design protocol, whether simultaneous or sequential. Finally, a Fusion Control Module should be responsible for implementing a properly designed control protocol dedicated to organize the alternation between the two input signals in order to achieve the Hybrid system performance goals and advantages.

Conclusions

EOG signals are considered one of the most successfully used bio-electrical signals in the field of HCI with a considerable number of publications investigating diverse applications, mainly targeted to help severely disabled patients with a very promising performance. After reviewing EOG based HCI literature, we conclude that almost all the developed systems are very promising in terms of performance and didn't involve complex signal processing and pattern recognition algorithms as opposed to the EEG based BCIs.

In general, EEG-BCI is widely more researched and investigated; however, a greater percentage of this research is more theoretical than practical due to the signal complexity, and low information transfer rates (ITR) that lead to unsatisfactory response times in practice. We recommend combining EOG to EEG in a Hybrid BCI system to solve challenges in both systems. This paper started with summarizing and organizing the EOG based HCI literature as a mandatory step in order to make this hybridization possible and gain insight on its potential advantages. Then a survey on current EEG-EOG hybrid BCI systems was presented. Finally, a general architecture was proposed for a new EEG-EOG Hybrid BCI system. The proposed hybrid system presented a novel design approach for signal extraction based on extracting the eye movement features from the EEG signals. Our proposed hybrid architecture considered EOG as an additional input modality sharing control with EEG rather than artifacts that should be removed.

EEG-EOG Hybrid BCI is still a relatively new research field. Extensive research should be performed in order to investigate possible shared control protocols. An ideal system should continuously target improvements in: 1. Speed and accuracy to avoid user frustration and maintain consistent performance levels, 2. User experience to minimize the mind and body effort and stress levels, 3. Multi-tasking and task variability to offer diversity and user friendliness and finally, 4. Creative Human-Computer interface corresponding to the new perspective of the bio-potential nature of the system input.

Hybrid BCI based on EOG and EEG signals might add a new dimension to HCI in serving healthy people, current BCI performance and ITR are not adequate for healthy users and EOG based interfaces are too stressful for the eye muscles causing user fatigue, by combining both input modalities the new system is expected to increase the accuracy and speed, to be less stressful on the mind and body and to be more versatile in terms of the options and functionalities. Therefore, EEG-EOG Hybrid BCI might provide an interesting control experience for healthy users especially in game control and entertainment purposes.

Future work will involve designing a hybrid EEG-EOG spelling system fusing both input modalities to achieve satisfactory performance levels and better user experience. An ultimate goal might be a fully hybrid system, where many types of biological signals: EEG, EOG, EMG, fNIRS (functional near-infrared spectroscopy), etc. could be integrated according to the user-specific needs and abilities.

Information Sharing Statement

All resources utilized in the described work can be accessed by the general public.