Keywords

1 Introduction

Fatigue is a state of decreased awareness or alertness associated with a desire or tendency to fall asleep. It impairs judgment, causes longer reaction time, and impairs coordination. Detection of fatigue, leading to drowsiness, involves the observation of eye movements and blink patterns. Fatigue is definitely a natural phenomenon that must be dealt with well, of course by sleeping or taking rest. But fatigue in a situation that demands an attentive state could get disastrous. For example, in the transportation sector, the attentiveness of a driver while driving is of utmost importance.

In the USA, a series of studies by the National Transportation Safety Board (NTSB) have proved that sleepiness is one of the major factors in causing accidents. The NTSB came to the conclusion that 52% of 107 single-vehicle accidents involving heavy trucks were fatigue-related. Generally, driver drowsiness is a main factor in 25% of the accidents and 60% of these redound to death [1]. Detection of fatigue can be useful in monitoring the staff and workers at places requiring high level of vigilance. Further, it can be used to detect the fatigue state of security guards. It can also be used in educational institutions to monitor the students in a classroom to detect the drowsiness of students.

2 Methodology

The first category is related to physiological studies. It includes eyelid closure, electrooculogram (EOG) [2], and electroencephalogram (EEG) [3,4,5]. Although the EOG provides good results, it is not suitable for a real application environment because of the attached electrodes on the users face. This is an intrusive approach and is likely to hinder the natural movement of the user. Among the other methods, eyeblink duration is the most reliable parameter for the detection of the drowsiness level [6]. As eyes are believed to contain vital information of the persons fatigue state, which can be tracked by using digital image processing techniques, and also this system is nonintrusive involving only a camera.

In this paper, we aim to develop a fatigue detection system, by using a method that can constantly monitor the eyes of the person and extract signs of fatigue or drowsiness by tracking eye movements and blink patterns and generating an alert. The system will have a nonintrusive approach and will extract fatigue information about the person by studying the eye movements and blink patterns. Also, the analyzing of the eye is done in real time, and recorded videos are not processed for the same. Our approach to this would be to use a nonintrusive system that would involve the image processing of eyes by monitoring and tracking eye movements and analyzing the blink patterns. Analyzing eyeblink patterns would help to extract the fatigue information of a person and also to develop a system that would be implementable in real-world scenarios.

Levels of fatigue states that we have considered in determining the alert level are [7]:

  1. 1.

    Awake State—Conscious and aware of the surroundings

  2. 2.

    Low Vigilance State—Low level of alertness

  3. 3.

    Drowsy State—A state of being lethargic; half asleep

  4. 4.

    Sleepy State—In a ready state to fall completely asleep

We have considered two modules:

  1. 1.

    Alert Module: In this module, based on live video feed, we continuously monitor the eyes of the person and the iris is detected and tracked. Based on the eye tracking, alert is generated on tracing a sleep condition. Also alert is generated on tracking a fixated gaze, left or right iris deflection and also on the facial movements [8].

  2. 2.

    Fatigue level Module: In this module, based on live video feed and input from the alert module, we calculate the eyeblink rate, blink amplitude, blink duration, and blink frequency. From the calculated values, fatigue level is detected and also a vacant stare (fixated gaze) condition may also be detected if traced.

Initially, the image is acquired from a live video feed, which is basically a frame. The frame is now used to extract a face image by using the Voila Jones Algorithm. If the face is detected then the face image is considered as the ROI (region of interest), further from the frame, the eye region is extracted and this is set as the new ROI. The eye image thus acquired is now flown into the function for iris detection. Based on this iris detection, we try to conclude whether the eye is opened or closed. In other words, we try to detect a blink. Studying these blink patterns, and tracking the eye gaze and movements thus, we try to conclude whether the person is in awake state, or is fatigued, or is in a sleep state. On the basis of these conclusions, we sound the alarm so that the person becomes vigilant again (Table 1).

Table 1 Drowsiness stages based on blink behavior [9, 10]
Full size table

The eye open or closed state is determined by calculating the eccentricity and major axis length of the connected component which is supposedly high in the eye closure state. Parameters like the eyeblink rate, eyeblink amplitude, eyeblink duration, and blink frequency are calculated. If the blink duration (the time period taken for a blink) is short and blink interval (time period between two blinks) is long, it is an awake state. Similarly, if the blink duration is short and the blink interval is short, then it is considered as a low vigilance state and if the blink duration is long and blink interval is very long then it is a drowsy state. Sleepy state can be determined with very long blink durations (Fig. 1).

Fig. 1
figure 1

Basic block diagram

Full size image

Input image is converted into grayscale image and the grayscale image is complemented in order to highlight the high intensity eye parts. Then median filter is applied in order to filter the noise and connected components are labeled in the image and eccentricity and major axis length is found out of the connected components. If the eccentricity and the major axis length is less than the given threshold value, then proper alert is generated and this process continues. After iris detection process, alert module is processed. In alert module, we have considered eye opening, eye closure, and face turn detection, iris deflection toward extreme left or right and vacant stare which can be a case of day dreaming. In the Alert Module, alarms and short alerts are issued on detecting a sleep condition and on detecting extreme iris deflections, face turns, and vacant stares, respectively (Figs. 2 and 3).

Fig. 2
figure 2

Flowchart for iris detection

Full size image
Fig. 3
figure 3

Flowchart for alert module

Full size image

In the fatigue module, we have considered four parameters:

  1. 1.

    Blink Amplitude—It is the time duration between two consecutive blinks.

  2. 2.

    Blink Rate—It is the time taken for one blink.

  3. 3.

    Blink Frequency—It is the number of blinks in a given time.

  4. 4.

    Constant Gaze—When no blink happens in a certain amount of time, then that can be the case of day dreaming.

Blink frequency and final fatigue level are calculated using the formulae:

$$\text{Blink}\,\text{frequency} = \left( {\text{No}{.}\,\text{of}\,\text{Closed}\,\text{Frames} /\text{Total}\,\text{No}{.}\,\text{of}\,\text{Frames}} \right)\,*\,\,100$$
(1)
$$\begin{aligned} & \text{Fatigue}\,\text{level} = \left( {0.45{\kern 1pt} \,*{\kern 1pt} \,\text{Blink}\,\text{frequency}} \right) \, \\ & \quad \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, + \,\left( {0.45\,{\kern 1pt} *{\kern 1pt} \,\text{Blink}\,\text{amplitude}\,\text{duration}} \right)\, + \,\left( {0.1{\kern 1pt} {\kern 1pt} *{\kern 1pt} \,\text{Stare}\,\text{level}} \right) \\ \end{aligned}$$
(2)

After the fatigue level module runs, it gets a final alert level, and then an alarm is issued based on the level of the alertness (Fig. 4).

Fig. 4
figure 4

Flowchart for fatigue module

Full size image
  1. 1.

    Level 1—(If Final level  < 1.6) Alert State—no alarm

  2. 2.

    Level 2—(If Final level  > 1.6 & Final level  < 2.3) Low Vigilance State—no alarm

  3. 3.

    Level 3—(If Final level  > 2.3 & Final level  < 3) Drowsy State—mild alarm

  4. 4.

    Level 4—(If Final level  > 3) Sleepy State—heavy alarm (Figs. 5, 6, 7, 8 and  9).

     Fig. 5
    figure 5

    Left iris deflection detection

    Full size image
    Fig. 6
    figure 6

    Right iris deflection detection

    Full size image
    Fig. 7
    figure 7

    Face turn toward left detection

    Full size image
    Fig. 8
    figure 8

    Face turn toward right detection

    Full size image
    Fig. 9
    figure 9

    Vacant stare detected

    Full size image

3 Results and Discussions

The system has been tested for its accuracy as well as its usability. In order to measure the accuracy of eyeblink detection, video sequences of each user sitting very close to the camera are taken. It was important to test the system’s ability to determine the accuracy and performance speed of the system. So part of objective of these tests is to determine the threshold that can work for multiple users. We tested the detection modules under alert module as well as fatigue detection module. In the alert module, we tested for open eyes, closed eye, face turn, iris deflection as well as a sleep condition. In the fatigue module, we tested for the fatigue level using the eyeblink rate as well as the eyeblink interval. We also tested for the vacant stare condition of the user (Tables 2, 3 and 4).

Table 2 Testing for alert module
Full size table
Table 3 Percentage calculation for accuracy
Full size table
 Table 4 Accuracy for fatigue module
Full size table

The average accuracy of the alert module is 88% and the average accuracy of the Fatigue module is 85%. The detected iris is used to determine the condition of the eye. The centroid of the iris is determined which is compared with iris’s centroid of previous frames and change in centroid results in the deflection of the iris. Also, we keep tracking both the eyes and in case only one eye is detected, face turn alert is given.

4 Summary and Conclusions

Lack of sleep, improper diet lacking nutrition, tiredness, and lack of exercise often lead to a fatigue state, which can prove to be hazardous and fatal in case a person is required to be vigilant in certain situations like while driving, air traffic controlling, guard personnel at nuclear reactors, security guard cabins, and so on. Although a proper sleep is the only solution to cure a fatigued person, yet to help the person remain vigilant, necessary alarms and warning systems at the right time can help the person to remain awake. Also, if the iris deflects a lot toward the extreme left or right, then a safety warning alert is issued to prevent distraction. The module was tested and it could detect fatigue and alertness with 88% and 85% accuracy respectively. It can be further implemented for the designing of a proper product after necessary removal of the limitations. The system cannot be used for a person wearing spectacles or sunglasses due to the reflection of light and can be further worked upon to bypass the glasses of the spectacles and thus, read the eye in a proper manner [11].