Keywords

1 Introduction

Now-a-days, computers are helping humans in every domain of life. Their application field spreads from solving complex problem to organizing day-to-day human activities. Individuals with physical disabilities often find it difficult to get full benefits of this modern tool. To reduce this barrier for physically disabled persons, Felzer et al. [1] have developed a software tool named as OnScreenDualScribe (OSDS). In addition to text typing, this tool also allows the user to select and click on-screen objects. OSDS is usually controlled by 18 physical keys which are located on a small number-pad. In this work, another software VirtualKeypad (VK) has been developed. VK responds to the user’s eye signals, i.e., blinks and closing both eyes. In response to these signals, VK generates virtual keystrokes. These keystrokes are picked by OSDS, which does not know if these keystrokes are generated by an actual keypad or by VK. This connectivity of VK and OSDS, allows the user to perform the functionalities of the standard keyboard and mouse by eye signals.

To capture the eye signals, two piezoelectric sensors have been used (Fig. 3). These sensors are attached to a readout circuit, which constantly monitors them and sends their raw data to the computer. Inside computer this data is received and processed by VK.

The remainder of the paper is organized as follows. In the next section, the state-of-the-art (literature review) is described. Section 3 presents the proposed system. Results and comparison with other publications is discussed in Sect. 4 and  Sect. 5 concludes the paper.

2 Related Work

In the past years, a number of attempts have been made to enter text by using the eyes. All of these techniques have two things in common. First, they have a sensor monitoring the eyes, and second, they have a software receiving raw data from the sensors and processing it. The eye sensors can be categorized into three main categories. The first category sensors work on the principle of electrooculography (EOG). In the second category, cameras are used to observe eye blinks and sometimes eye gaze, whereas the third category consists of piezoelectric sensors.

Electrooculography-based systems are presented in [2–4]. Borghetti et al. [2] have used 7 electrodes and have achieved a typing speed of 7.1 CPM. Soltani et al. [3] have developed a small PCB which monitors five electrodes and sends their data to the computer. Their system has shown a typing speed of 5.9 CPM. Tangusksant et al. [4] have used six electrodes and have achieved a typing speed of 2.4 CPM. Electrooculography needs many electrodes, which should be precisely placed for accurate detection. For a physically disabled person, such sensor placement is cumbersome or even impossible without help.

Królak and Strumiłło [5] have used a camera to detect eye blinks and have reported a typing speed of 5.0 CPM. Benefit of using a camera, is the elimination of facial sensors, whereas its drawback is the significant increases in required computational power. Several images per second has to be acquired and processed by the computer for successful operation of this technique. The complexity of this approach further increases if the user is wearing spectacles or suffering from gaze jitter problem.

Related to the last category of sensors, Felzer et al. [6] have used a single piezoelectric sensor, reporting a typing speed of 4.3 CPM. With a single sensor it is not possible to find which eye has been blinked. This problem makes the system less efficient and also reduces the typing speed.

3 Proposed System

A novel system is proposed in this work which requires less sensors as compared to EOG based systems and requires less computational power as compared to camera based systems. In this work two piezoelectric sensors have been used so that “right eye blink”, “left eye blink”, and the “both eyes closed” can be effectively distinguished. Proposed system is pictorially illustrated in Fig. 1. Sensor placement, sensor readout circuitry, and the VirtualKeypad (VK) software are described in the following subsections.

Fig. 1.
figure 1

System Setup (©Fotosearch.com)

Full size image

3.1 Sensors Placement

To find appropriate spots for the sensor placement, anatomy and physiology of the fascial muscles have been studied.

Eye blinks are caused by Orbicularis Oculi muscular rings. Placing sensors directly on these muscular rings would restrict the eyelashes movement and would cause discomfort.

Both sensors can also be placed on forehead because the movement of the Orbicularis Oculi muscle also sends vibrations in the forehead skin and the Frontalis muscle. Placing sensors on the forehead make it easier to detect the eye blinks however the determination of signal type (i.e. “right eye blink”, “left eye blink” or “both eyes are closed”) becomes very difficult. Main reason for this problem is the close proximity of sensors, resulting in almost same readings on both sensors when any eye is blinked.

To solve these problems the sensors are placed on the bone between Temporal and Orbicularis muscles on each side of the skull. These spots are shown in Fig. 2 (see also [7]). Each spot shows significant movement, when the eye on its side is blinked. Being on separate sides of the skull, each sensor receives minimum interference from the blink of other eye. The headband with piezoelectric sensor is shown in Fig. 3.

Fig. 2.
figure 2

Facial Muscles (©Fotosearch.com)

Full size image
Fig. 3.
figure 3

Headband with readout circuitry

Full size image

3.2 Sensor Readout Circuitry

Sensor readout circuit is also shown in Fig. 3. It has cable connections with the headband and the computer. Readout board contains biasing circuit of piezoelectric sensors, analog amplifiers, a microcontroller (Atmel Atmega 329P), a 600 mAH LIPO battery and a battery management circuit. Battery can be charged either by computer USB port or by micro USB charger.

3.3 VirtualKeypad (VK)

This software is developed in Matlab® and its user interface is shown in Fig. 4. It receives the raw data from the sensor readout circuitry and processes it to determine the type of eye signal which can be “Right eye blink”, “Left eye blink”, and “Both eyes closed”.

Fig. 4.
figure 4

VirtualKepad (VK)

Full size image

At the start, the user needs to calibrate the VK. The calibration process has three phases. In the first phase, the user is required to wear the headband and relax. Once the user is calm and stationary, the second phase can be started. In this phase, VK monitors the background noise. In the third phase, the user is required to provide at least three right and three left eye blinks. By evaluating background noise and intensity of deliberate signals, VK computes the detection envelopes for both sensors which are shown by dotted lines in Fig. 3. The calculation of the detection envelopes is explained in more detail in [8]. To determine the type of eye signal VK follows the algorithm shown in Fig. 5.

Fig. 5.
figure 5

Detection algorithm

Full size image

To scan the virtual number-pad keys, VK has two scanning schemes. In passive scanning, the system automatically scans rows/columns. By default, VK stops on each selected element for 800 ms before incrementing the row/column. This time can be adjusted from 500 ms to 4 s. A “Left eye blink” starts/stops row scanning, whereas a “Right eye blink” starts/stops column scanning. Whenever both eyes are closed, the selected key is “virtually” pressed, which means that its keycode is generated. This keycode is intercepted by OSDS.

In active scanning, the user has more control. The system waits for the user signal. “Left eye blink” results in row increment, “Right eye blink” increments the column, and “Both eyes closed” virtually presses the selected key.

4 Results and Comparison

To measure the accuracy and typing speed, an evaluation software called “Writing” is used, which is a part of the OSDS software. This software generates a random sentences on the screen and asks the user to type it. After completion of five sentences, the software writes required time and accuracy report to the computer’s hard disk.

The presented system has been tested on five able-bodied and one disable volunteer. To make the results comparable, all the volunteers have used ambiguous mode in OSDS and passive scanning in VK (in early pilot testing, this has been found to yield the higher entry rates, and active scanning is much less comfortable).

The test was divided into three parts. In the first part, volunteers were given some practice time. When they had become comfortable with the new system, the remaining two parts were performed. During the second part, volunteers were asked to enter five phrases using a physical number-pad, and in the last part, they have used the headband and the VK software to enter the test phrases. Results are summarized in Table 1. The average entry rate shown by six volunteers is 9.1 characters per minute (CPM). In Table. 2, this speed is compared with other publications.

Table 1. Showing the result of volunteers
Full size table
Table 2. Comparison with other publications.
Full size table

The disable volunteer in this work is a 45 years old male who was diagnosed with Friedreich’s Ataxia in 1985. Same volunteer had also participated in [6], where a single piezoelectric sensor was used. There he had achieved a typing speed of 4.3 CPM. It is interesting to note that with this novel system he was able to type at the speed of 8.9 CPM which is more than double of his previous speed.

From Table 2, it can be seen that the typing speed achieved in this work is better than all other approaches except the one reported in [9]. However, in [9], the speed test has been performed by entering a five character word. This test approach is inadequate, because the sample consisting of just one word of five letters is simply too small. Other than this, [9] does not have any mechanism to control the mouse.

5 Conclusion

In this work, a novel system is presented; which – with the help of the OnScreenDualScribe (OSDS) software – replaces computer keyboard and mouse. The presented system consists of a headband fitted with two piezoelectric sensors, a readout circuit, and software called “VirtualKeypad (VK)”. With this new system, the user can click and type by using only three eye signals: “Left eye blink”, “Right eye blink”, and “Both eyes closed”. During tests, it has allowed users to enter text at an average speed of 9.1 characters per minute (CPM).