Keywords

1 Introduction

The present transportation systems are an integral part of human activities. All of us can experience drowsiness while driving a vehicle, just after a short sleep, during long journeys, or a changed physical condition. Sleeping sensation reduces the driver’s alertness, creating dangerous situations and increases the likelihood of accidents occurring. Tiredness and drowsiness of the drivers are among the main factors behind traffic accidents on roads. Every year, they increase the number of deaths and accidental injuries worldwide. Humans have a simple need called rest because, with sufficient sleep, they permit the efficient performance of the faculties that realize a person. However, when this activity is interrupted by lots of factors, especially in drivers, the psychomotor and cognitive functions such as response time, monitoring capability, judgment, and attention are impaired. When the driver does not have sufficient rest, he will try to sleep while driving a vehicle. That is the primary indication of drowsiness of the driver. Generally, the driver will not give an attention and then comes the yawning, the try to close his eyes every point of time, and the movements of the head laterally. This leads to a vehicle accident that can be an unfortunate event for both the pedestrian and driver. There are many studies to determine a drowsiness recognition system which extracts the fundamental features of drowsiness of the drivers to find out drowsiness.

Therefore, it is important that another upcoming vehicles have an additional security function to alert the tired drivers and do auxiliary tasks with the hand movements [1]. Such device can be built using various techniques which may be intrusive. The intrusion considers biological variables such as electroencephalogram (EEG) and electrocardiogram (ECG). In this technique, however, electrodes must be attached to body of the drivers. Because it disturbs the drivers and usually the electrodes are expensive, and it is not especially preferred by the driver. A non-intrusive procedure may be vehicle-based mostly, e.g., however, because the vehicle-based ways depend upon the driving capability of the drivers, and the vehicle-side driving style of vehicles is driver dependent, and the sort of the road which the vehicle is traveling helps it to be difficult to create standardized guidelines. This paper uses a research of expressions (yawning, head tilt, and eye closure) that is considered to be the most likely method. For this system, a camera must be located in the car to capture the video recording of the drivers. Further, the video recording is processed into images, and control of the captured image is achieved by a histogram oriented gradient (HOG) where we extract the descriptor algorithm to identify the driver’s face in each image. To keep an eye on the yawning and drowsiness of the drivers, we first get 68 landmarks on the driver’s face and then compute the EAR and MAR from each image. Consequently, a threshold was used to check on the position of the drowsiness. When tiredness is recognized, the system delivers a warning in the form of alarm. The purpose of these systems is to increase the safety of people by using detection and security alarm to prevent traffic incidents triggered by the drowsiness of the drivers.

2 Literature Survey

Several methodologies were proposed to detect drowsiness. Some of the relevant works related to the fatigue detection are explained here. Chellappa et al. [2] use the physical and physiological indications to find the drowsiness of drivers. This technique will get information from both factors and uses a combination of these as parameters to detect the drowsiness of the driver. Physical inputs include yawning and blinking, and physiological inputs include pulse rate and body’s temperature. Eventually, they cause discomfort for the drivers and are therefore not reliable in comparison with non-intrusive techniques. Yang et al. [3] mentioned a real-time prototype to monitor the vigilance of drivers. On the dashboard of the vehicle were two CCD video cameras. The first camera was a wide-angle camera targeted at the driver’s monitor head movement and driver’s head to monitor as the second camera was a small-angle camera focused on the driver’s eye to control the motion of the eyelids. The system used visible bio-behaviors such as eyelid actions, facial orientation, and eye movements (pupil movements) that typically characterize the alertness of the drivers, but we use an individual camera focus on the eyelid, head movement, and yawning.

Bergasa et al. [4] suggested a non-intrusive computer perspective system to detect the driver’s monitoring instantly. They decided to go with six guidelines: ratio of ocular closure (PERCLOS), eye closure duration time, occurrence of blinking and assent, position of face and stare. These guidelines were combined by using a fuzzy classifier to infer the driver’s insufficient attention level; however, the entire process considers only the state of eye. Sarada Devi and Bajaj [5] suggested a strategy to locate and monitor a driver’s mouth using the cascade of classics suggested by Viola–Jones for encounters. Support vector machines (SVM) can be used to teach the mouth and yawn images. During drowsiness detection, the mouth is identified by cascades of classifiers of facial images. SVM is then used to classify the condition of the mouth and also to find yawning for drowsiness monitoring.

3 Proposed System

In recent years, the reduction of traffic accidents has not increased significantly. Among the many reasons of road accidents, driver fatigue, tiredness, and drowsiness will be the most important. Our system aim is to increase the safety of pedestrian, passenger, and driver by detecting the drowsiness of the drivers and alerting the drivers to avoid accidents in real-time situations. The sources of the driver tiredness may be because of insufficient sleeping or no appropriate sleeping environment or the hectic schedule of driver, especially in transportation vehicle. The drivers were not in a position to sleep for the very least amount of eight hours which indirectly results the health and triggers fatigueless. For instance, the individual becomes sick at any time; variants in the feeling react quickly to avoid improper events. Sometimes having less sleeping environment could also cause drowsiness for drivers. However, the sleeping conditions do not have the necessary supports or the conditions for space and recovery for sleeping. The major ratio of sleeping environment is within the trunk of vehicle. This causes sleep issues to the driver, and it is not appropriate to take rest place because of the sound of other vehicles.

This paper implements a technique where drowsiness recognition has been done by consistently monitoring driver’s mouth, head location, and eye at the time in order to implement a driver safety system to avoid accidents. From the block diagram, the suggested strategy is shown in Fig. 1. The camera which focuses the face of the drivers can be used to record consistently the video recording of the face of the drivers. This video is processed into images, and then, each and every image is processed with the help of Dlib library. Then, for every single image, face coordinates are identified, which are being used to determine the driver’s behavior. From these facial coordinates, upcoming variables are determined that are the size of the eyelid and mouth openings, the inclination and rotation of the head, and the lowering of head forward. The determined values from each image are compared with the threshold values in each case. In starting phase, the recognition thresholds were placed manually. Several experiments were conducted to choose the threshold values for each and every image to ensure the highest recognition efficiency of drowsiness. Next, a computerized calculation of these thresholds originated. In order to recognize, 68 facial points can be considered.

Fig. 1
figure 1

Block diagram

Full size image

Different features of the face can be detected through the accessing points as shown in Fig. 2. The mouth can be accessed through the points [48, 67], the right eyebrow through [17, 21], the remaining eyebrow through [22, 26], the right eye can be accessed using [36, 41], the left eye uses [42, 47], the chin via [0, 16], and the nose uses [27, 35]. The first step in this algorithm is the recognition of face in frame. The coordinates of face are determined by using an HOG. Then, your characteristic details of face are determined. The characteristic Dlib detector was trained on face marking dataset of the iBUG 300-W. Because of this, 68 landmarks were obtained on the face, which permit the recognition of elements such as chin, mouth, eyebrows, nose, and eyes. By studying the coordinate changes of the viewpoints, the next facial expressions were watched such as eye closure, yawning, and head movement. Three variants in the head position can be viewed for head movements such as tilt of head aside, head falling, and rotating. In this process, head forward option is best among all the three head movements to determine sleeping state of a driver.

Fig. 2
figure 2

Shape predictor 68 face landmarks

Full size image

3.1 Eye Blink Detection

The blinking is defined thought an unconditioned reflex of shutting the eyelids to damp the eyes. The recognition of the drowsiness is a symptom which depends upon the blinking occurrence. The eye position is identified using six landmark points that are shown in Figs. 3 and 4. Then, the EAR (Eye Aspect Ratio) of the six points is calculated using Eq. (1). In this, A, B, and C are the euclidean distance between the above six points, given in Eqs. (2), (3), (4).

Fig. 3
figure 3

Landmarks for open eye

Full size image
Fig. 4
figure 4

Landmarks for closed eye

Full size image
$${\text{EAR}} = \frac{A + B}{2*C}$$
(1)
$$A = \left| {\left| {43 - 47} \right|} \right|$$
(2)
$$B = \left| {\left| {44 - 46} \right|} \right|$$
(3)
$$C = \left| {\left| {42 - 45} \right|} \right|$$
(4)

The size of the eyelid closure (Eq. 1) is evaluated with the threshold value which is significantly less than 0.3, and the occurrence of frames is higher than or equal to 7; then, the sleep condition is recognized. Equations (2), (3), and (4) represent the n-dimensional euclidean distance between the points. In this work, we considered n = 1. So, after detecting the eye points using HOG descriptors, the EAR ratio is defined in Eq. (1). In Eq. (2) (43, 47) points and in Eq. (3) (44, 46) points give the straight distance between eye open points in vertical direction, and in Eq. (4) (42, 45) points give closed points in horizontal direction.

3.2 Yawn Detection

Yawning is defined as a reflex during which the air is inhaled. It is associated with fatigue and drowsiness. The detection of yawing is based on a continuous review of the width of the mouth opening. The oral position is described using 20 landmark points shown in Figs. 5 and 6.

Fig. 5
figure 5

Landmarks of closed mouth

Full size image
Fig. 6
figure 6

Landmarks of opened mouth

Full size image

From these 20 points, six points were selected to compute MAR (mouth aspect ratio) using Eq. (5) where D, E, and F represent the euclidean distance between these six points given in Eqs. (6), (7), and (8).

$${\text{MAR}} = \frac{D + E}{2*F}$$
(5)
$$D = \left| {\left| {61 - 67} \right|} \right|$$
(6)
$$E = \left| {\left| {63 - 65} \right|} \right|$$
(7)
$$F = \left| {\left| {60 - 64} \right|} \right|$$
(8)

The determined value of MAR is evaluated with the recognition threshold; if the MAR value is higher than 25, then the yawing condition is recognized.

3.3 Head Location Analysis

Tiredness makes it hard to keep your head up. Ability to find the inclination of head, it uses Y coordinates of both end points of nose and chin and then compares them with the threshold condition. The down falling of a head is an indication of sleeping. To detect this event, the distance between the end of the nose and the chin is considered. When the head is falling down, the length between these points will decrease.

4 Result

The proposed methodology is examined in real-time situations, and the recognition results for all the three cases are shown in Fig. 7a–c. In Fig. 7a, as the EAR is less than the specified threshold value, the sleeping condition is detected. Figure 7b represents the yawing condition as the MAR value is greater than the threshold. Figure 7c represents the sleeping condition as the tilting of the head condition is detected. Figure 7d, e represents the condition of sleeping detection even the person is wearing the spectacles. With this, we conclude that the proposed approach is the most efficient approach compared to the existing approaches because the fatigue detection analysis is considered in three cases. Software Tool: To detect drowsiness, determined functions from the Dlib libraries and OpenCV were used. OpenCV is a library of functions made for real-time image analysis and processing. Dlib is a cross-platform universal library for network support, data structure management, thread handling, and machine learning. The library has several functions, including a face detector and a detector with quality features.

Fig. 7
figure 7

a Sleeping detected, b yawning detected, c head dropping down detected, d face detection with spectacles, e sleep detection with spectacles

Full size image

5 Conclusion

In this paper, a smart system based on fusion is introduced to detect the drowsiness of the driver in real-time situations. Drowsiness can cause the drivers to react slowly and raise the risk of road accidents. By detecting the driver’s behavioral characteristics and deciding the driver’s condition of awareness, we can avoid possible drowsiness risks. In this work, three cases such as yawning, eye closure, and head tilting were used to detect the sleeping behavior of driver. The advantage of this approach is that it can be used for commercial applications, which makes the system robust. Another good thing about this process is that it is not depending on subjects, and the system does not need to learn. Experimental results show the higher efficiency of drowsiness detection of driver.