1 Introduction

Ambient intelligence and humanized computing making it possible to collect data from heterogeneous sources and to analyze it in real-time. Data induction is a major factor in the development of multimedia-based real-time behavioral monitoring systems. Data collection, processing, analytics and development of machine intelligent systems are ubiquitous and helpful for the human. It is helpful in different activities like the facilitation of the drivers on roads. Machine intelligence with smaller datasets is being replaced with deep learning and big data because it is easier to incorporate more features from big data as compared to smaller datasets. The applications of deep learning making complex decision making easier. It is more feasible now to monitor behavioral and risk activities in real-time using soft computing techniques.

Vehicle active security and safety system on roads is becoming more critical and necessary to diminish traffic accidents on a daily basis. Usage of active sensors for vehicle detection offered in commercial type vehicles, in limitation of active sensor usage, the increasing trend of vision-based vehicle sensing technology to detect an obstacle in front of the vehicle on roads in real-time scenarios (Sivaraman and Trivedi 2013). On roads, vehicle obstacle and collision warning systems play an essential role in a safety system for a vehicle on roads in the real-time scene. Vehicle detecting on roads in real-time scenarios using vision-based cameras is focused as a challenge in the research era because there are lots of things to be considered on roads like building, trees, people, shadows, railing and so on (Riaz et al. 2018). In-vehicle safety system, the research focus is mostly on vehicle detection, obstacle recognition, collision warning, driver drowsiness, alert the driver for distance to avoid crashes with obstacles, lane detection, number of vehicles on the road (vehicle count), monitoring driver activities monitor, and vehicle tracking. Appearance-based methods employ features i.e. Speeded-Up Robust Features (SURF) (Bay et al. 2006), a histogram of oriented gradients (HOG), Haar-like features, and so on. In addition, classifiers features are containing such as Adaboost, support vector machine (SVM), and so on (Niknejad et al. 2012). These methods and classifiers are used to consider accuracy as well as robustness for improving the vehicle tracking systems. Rear-end collision and forward-collision constitute a significant concern for road accidents. The collision warning system assists the driver to avoid significant injuries and accidents on roads. State of the art vehicle system adopts a smart methodology for safety and security that includes driver drowsiness, obstacle detecting, collision warning, an alert driver using an alarm, and take actions according to the scenarios (Raza et al. 2018).

Driver drowsiness determines driver behaviors in various ways of face deformations (in front, sleepy, aside, down, upper), head movements (right, left, up, down) and eye movements (closing or opening duration, the percentage of eye closure (PERCLOS), blink rate, and eyelid movements). The graph of social and economic losses due to road accidents is alarmingly high. Among its many other reasons, lethargic, inactive and fatigue condition of drivers due to less sleep or drowsiness are the most important. Drowsiness is involved in one in five crashes. Even though the driver is very expert and experienced, drowsiness makes it all null. Saudi Arabia has one of the highest road traffic accidents (RTA) rates worldwide. According to the World Health Organization (WHO), RTFs occurring due to RTAs in the KSA reach 27.4 for every 100,000 people in 2013. RTFs due to RTAs in other countries such as the USA, Canada, UK, and UAE were 10.6, 6.0, 2.9, and 10.9 for every 100,000 people, respectively (World Health Organization 2015). A significant cause of roadside accidents includes drowsiness, which takes away the driver’s capability of controlling the vehicle (Bioulac et al. 2018). Drowsiness status causes a loss of awareness of the surroundings. Prevention of accidents caused by drowsiness includes alarming the person or stopping the vehicle using some intelligent vehicle control system.

Similarly, vehicle automation and Vehicle Adhoc Network (VANET) based systems can further support the cause of improving driving safety measures. This study focus is majorly based on the development of the drowsiness detection systems and taking pre-measures of the surroundings of the vehicle. Experiments involve detection of eye blinking ratio and eye closing period as the measurements to calculate the driver’s drowsiness status at real-time observation (Pasaribu et al. 2019). Continuous monitoring brings forth the possibility of any chances of accidental factors like drowsiness, loss of awareness, or collision chances. This study can provide a basic building block for future integration of vehicle automation. Vehicle automation can help in stopping the vehicle while the driver loses consciousness. There are drowsiness detection systems in the vehicles to alert the driver about an abnormality in driving. However, they are needed to be updated with alerting action including preventive and protective measures. Without this enhanced model, the driver can easily ignore the altering actions that usually result in accidents. Also, the vehicles without these pre-installed systems, the danger of accidents due to the under-discussed reason become high.

This paper is further organized as follows. Section 2 covers the literature review for showing the impact of drowsiness in Saudi Arabia traffic accidents recorded over time. Furthermore, it covers different studies following different models and methodologies in providing the solution to similar problems of driver’s activeness detection. Section 3 is dedicated to representing the methodology devised to solve the arising problem of accidents due to drowsiness and collision. After this section, the results are discussed to show the effectiveness of the proposed system. Finally, in Sect. 4 the conclusion is presented.

2 Literature review

Due to long driving hours and working in shifts, the drowsiness of the driver has become one of the severe problems. Drowsiness is the state of human beings between awakening and sleep. Drowsiness leads to affect driver’s concentration. Diabetes, working in a shift, excessive calcium in the human body, sleep disorder, medicines are the main reasons for drowsiness. A significant cause of roadside accidents includes drowsiness, which takes away the driver’s capability of controlling the vehicle. Drowsiness status causes a loss of awareness of the surroundings. Prevention of accidents caused by drowsiness includes alarming the person or stopping the vehicle using some intelligent vehicle control system. Similarly, vehicle automation and VANET based systems can further support the cause of improving driving safety measures. This study focus is majorly based on the development of the drowsiness detection system and taking precautionary measures of the surroundings of the vehicle. Experiments involve detection of eye blinking ratio and eye closing period as the measurement to calculate the driver’s drowsiness status at real-time observation. This study will provide a basic building block for future integration of vehicle automation. Vehicle automation can help in stopping the vehicle while the driver loses consciousness.

Drowsiness has many impacts on the human body such as impaired thinking, memory, and vital signs. An estimated 328,000 crashes each year involves a drowsy driver (USA statistics) (National Safety Council 2019). According to a report from the USA, The National Highway Traffic Safety Administration conservatively estimates that 100,000 police-reported crashes are the direct result of driver fatigue each year. It results in an estimated 1550 deaths, 71,000 injuries, and $12.5 billion in monetary losses (Rendeiro 2017). In the case of the Kingdom of Saudi Arabia (KSA), It has a high ratio of roadside accidents annually reported and is continually increasing (Mansuri et al. 2015). According to a report in Saudi Gazette, the Head of Red Crescent Committee, Ahmad Al-Shaikha, 17 people died on a daily basis among 526,000 accidents reported annually (Saudi Gazette 2016). According to him, “a total of SR21 billion is spent annually on different cases of road accidents. Saudi Arabia is ranked 23rd on the list of countries witnessing the highest death rates in road accidents in the world. It is second among Arab countries regarding road deaths” (Saudi Gazette 2016). A review paper on road safety and road traffic accident in Saudi Arabia shows a result that according to the hospital records, there exist 8% increase in road accidents compared to police record of decrease of 27% from 2005 to 2010 years’ era (Gopalakrishnan 2012). Recent statistics released by Riyadh’s General Directorate of Traffic show that 4000 people died because of drowsy driving in Saudi Arabia in 2014 (News 2015). Researchers at King Saud University’s Center for Sleep Disorder stated that 33% of drivers who were part of the 6-month-long study claimed to have almost been involved in an accident at least once during the research period. Over 12% of the study participants said they were involved in a car accident because of sleep deprivation (News 2015).

A long list of precautionary and preventive measures become null if the driver is feeling sleepy. So, there should be some useful means to inform the driver that he should stop driving because vital signs are showing that he is too tired to drive safely. Drivers’ drowsiness can be detected through various technologies including steering pattern recognition, Vehicle position in lane monitoring, driver’s eye and face monitoring and physiological measurement. The first solution also called Steering Wheel Angles (Shi et al. 2017). It uses steering input from an electrical power steering system for detecting the drowsiness of the driver. The sensor mounted on the steering level collects the data usually based on the frequency of minor steering corrections. This method can only work in certain situations as it is more reliant on the geometrical features of the road. Lane monitoring camera is used for the second solution to alert the driver due to frequent drifting out of its lanes. Thus, to initiate the alerting action to handle the drowsiness and to make the driver awake so that the accident can be avoided. For the driver’s eye or face monitoring solution to driver’s drowsiness detection, In-Vehicle Camera is used for watching the driver’s face (Kumari and Kumar 2017; Lee and Chung 2012).

From the Automotive industry, many big names are giving solutions to driver’s drowsiness. Among those, some are as follows:

  • Rest Recommendation System by Audi (a German Automobile manufacturer) (AUDI 2017).

  • Active Driver Assistant with Attention Assistant by BMW (a German Luxury Vehicle, Motorcycle, and Engine manufacturer) (BMW 2017).

  • Lane Departure Warning System by Mazda (a Japanese Multinational automaker) (MAZDA 2017).

  • Driver Drowsiness detection by Bosch (German multinational engineering and electronics company) (GmbH 2017).

  • Driver Alert by Ford Motor Company (American Multinational automaker) (Ford Motor Company 2017).

  • Driver Condition Monitor and Driver Fatigue Alert by Jaguar Land Rover (A British multinational automotive company) (Jaguar Land Rover 2017).

  • Attention Assist by Mercedes-Benz (A German global automobile manufacturer) (Mercedes-Benz 2017).

A probabilistic model based on the visual cues model by Yao et al. (2010) is proposed for the detection of the driver’s fatigue level. It includes eyelid movement, gaze movement, and head motion. Another methodology by Kumari and Kumar (2017) is proposed for detecting the driver’s drowsiness for the pupil’s motion. If for several consecutive frames, the driver’s eyes are closed within a specific period, then he is determined to be fatigued. Otherwise, there is just the blinking of eyes, and there is not a diagnosis of fatigue. Yao et al. (2010) proposed a vision system to the vigilance level of the driver. They integrated some facial parameter including eyes, gaze, and mouth. Another module for the Advanced Driver Assistance System (ADAS) is presented by Alshaqaqi et al. (2013) for increasing transportation safety and hence to reduce the number of accidents due to driver’s fatigue. The algorithm to measure the PERCLOS through locating, tracking and analyzing the driver’s face and the eye is present in the literature. PERCLOS is a scientifically supported measure of drowsiness that is associated with the eye closure in slow motion.

As driver’s drowsiness can lead to accidents causing major injuries or death through roadside crashes in many cases. Research shows how crucial it is to represent an effective and real-time drowsiness detection system. There have been many methods introduced to measure drowsiness of driver’s which include machine learning-based classification using fuzzy wavelet packet extraction (Khushaba et al. 2011), heart rate analysis for fatigue detection, electroencephalogram (EEG) based detection (Qidwai et al. 2018), and using EEG for automatic detection through Artificial Neural Network (Kaur and kaur 2013). The majority of these solutions are based on sensors and physiological measures studied by many other researchers. With the current evolution of systems by involving image processing for training machine learning-based models to identify fatigue and drowsiness of the driver. Machine learning-based solution is highly adopted these days due to their deployment capacity and accuracy ratio while analyzing the environmental constraints. Automated Driving (AD) systems nowadays are being introduced by researchers to reduce the impact of drowsiness. AD-based systems can be used to take control of the vehicle while the driver drops sight of the environment due to drowsiness. Drowsiness can be measured using heart rate or brain signal or image-based eye status detection. The partial AD system can be used to accomplish safety measures of the vehicle (Habib et al. 2019).

3 Methodology

The research methodology has three modules. The first one is driver drowsiness and second is vehicle obstacle and collision warning. The third module is related to the reporting of the driver’s drowsiness to the authorities. There are two major phases for the development of the driver’s fatigue management system. The first phase is offline and lab work, in which the development of machine classifiers for the detection of human face and eyes are involved. In the later phase, the testing of the classifiers and checking the prototype takes part in the real-time. In traditional systems, the driver’s fatigue detection is monitored manually by other humans if present in the vehicle at that time. In the case of the absence of another human with the driver at a particular time, it is hard to detect the fatigue manually. So, there is a need for an automated and artificial intelligence system for the detection and management of fatigue as well as it is necessary to take appropriate action.

3.1 Driver drowsiness detection

The proposed mechanism “Smart Methodology for Safe Life on Roads with Active Drivers” is comprising of three major modules:

  1. 1.

    Driver’s drowsiness detection system

  2. 2.

    Alerting action generation (alert the driver in the form of alarming sound with the vehicle)

  3. 3.

    Report to authority and the driver.

The architectural diagram of the proposed system is shown in Fig. 1.

Fig. 1
figure 1

The architectural diagram of the smart methodology for a safe life on roads with active drivers

Full size image

The fatigue detection system can be implemented using four different methods or a combination of these methods. These methods can be steering pattern recognition, vehicle position in lane monitoring, driver’s eye and face monitoring and physiological measurement. In our system, currently, we have adopted the method of eye and face monitoring.

The first module is further composed of two steps; the first one is the training of classifiers for face detection, eye detection and checking the openness and closeness of eyes. The target images are called positive samples or images, which contain the target, and the images which are not containing target are called negative images. In this research, the targets are the human face and eyes. The training of the machine classifier is shown in Fig. 2 to detect the drivers’ fatigue in real-time by applying different classifiers on the test image. The second module takes the input from the first module and generates the alerting action at three stepwise levels with some defined conditions. On exceeding the counter limit, the output of the uncontrolled situation of the driver’s response to the alerting actions is passed on to the third module.

Fig. 2
figure 2

The training process of a machine classifier using positive and negative sample images

Full size image

The driver’s drowsiness detection system is shown in Fig. 3. Algorithm 1 shows steps for the driver’s drowsiness detection system. The steps are followed to detect accurate driver drowsiness with monitoring eye aspect ratio.

figure a
Fig. 3
figure 3

Driver’s fatigue detection system

Full size image

The connectivity sub-module in the ‘Report to Authority’ system is triggered once the driver’s response to the alerting part is not up to the required extent. Next is to send the driver’s location and vehicle information with sufficient information of driver’s fatigue to the police’ Base station. The brief working of both the module is shown in Fig. 4.

Fig. 4
figure 4

A module for alerting and report to authority and driver to take action

Full size image

Algorithm 2 is about alerting and reporting to the authority for action generation, which shows the steps that follow to produce alert for drowsiness through the detection system. The threshold value is calculated to monitor activeness or drowsiness of driver, which is further used for alert managing.

figure b

3.2 Vehicle obstacle and collision warning

The driver’s drowsiness and vehicle obstacle detection modules are integrated to avoid accidents. In obstacle detection, the camera takes a frontal view of the driver and detect obstacles in front of the vehicle. Detected obstacles are containing type, size, and distance for the collision. Then generate alert to the driver when there is something wrong like any obstacle occur on the road. Collision detected when distance measure is less than prescribed. Distance calculate using \({\text{Dist}} = {\text{d}}x - {\text{d}}y\). Where dx is calculated x distance among detected vehicles and do is y distance calculate. Obstacle detection and collision warning system framework are shown in Fig. 5. Vehicle obstacle and collision warning system in computer vision to detect and recognize vehicles on roads to determine obstacle type, size, and distance. Frontal view camera that is placed on the car to view the front scene of the road and detect vision-based measurements to find obstacles in front of the car. Detection of obstacle type, size, and distance for avoiding a collision is measured as given in Algorithm 3.

figure c
Fig. 5
figure 5

A framework of obstacle warning system

Full size image

4 Experiments and results

This section describes results on the detection of driver drowsiness and vehicle collision warning using the calculation of distance among obstacles.

4.1 Driver’s drowsiness detection on driver’s images

The driver drowsiness and normal states are detected as shown in Fig. 6 by using the developed system. Driver drowsiness detects the behavior of the driver in various states. Using real-time camera scene captured to detect and alert driver behavior during driving cars on roads. Driver states like face deformations, eye movements, and body positions are measured as driver drowsiness and fatigue.

Fig. 6
figure 6

Driver drowsiness detection results

Full size image

Figure 6 shows different detection done on the driver’s face taken by the camera in real-time scenarios to analyze the situation of the driver while driving on the road.

Table 1 describes the statistics of drowsiness detection concerning the system’s normal operations that are the obtained or retrieved responses of driver’s states. Test analysis is performed on the video for a duration of 46 s on different drowsiness parameters (front nodding, blink detection, yawn detection, the assent of the head to left and right, and distraction to left and right) based on the percentage of hits (successful detection) are presented therein this table.

Table 1 Drowsiness parameters for detection in normal conditions for a recorded video of 46 s
Full size table

Open and closed EAR of single eye blink is presented in Fig. 7. The eye movements describe eyelid positions like open and close to detect driver drowsiness for alert security and safety of the driver. When the driver’s eyes blink, an alert is generated for security purposes to ensure that drivers are properly protected during driving using threshold calculations. When the threshold is less than 0.2 means that EAR blinks and activates alerts to the system for driver drowsiness (Fig. 8).

Fig. 7
figure 7

Open and closed eye aspect ratio of single eye blink

Full size image
Fig. 8
figure 8

Eye to aspect ration (EAR) calculation using a threshold value

Full size image

Driver drowsiness detection using eye blink in real-time scenarios to measure fatigue of driver during driving is shown in Fig. 8.

4.2 Vehicle obstacle detection and collision warning

Obstacle detection on-road and collision detection using distance measuring among various vehicles from camera results is shown in Fig. 9.

Fig. 9
figure 9

Vehicle detection and distance calculation

Full size image

The proposed mechanism detects various objects on the road scene and recognizes them with their labels. Then detected objects are used to measure the distance with the frontal camera-based vehicle to avoid crashes on roads and alert drivers with alarm to take the best options. Distance is calculated among other vehicles to ensure safe driving on the road. On approaching threshold distance value, a warning is given to the driver.

4.3 Convolutional neural network (CNN) for face and drowsiness detection

Neural networks like CNN are composed of different layers. These layers have different types like input, output, and hidden layers. Each layer of neurons has its own functionality, and the functionality depends upon the features in the data. CNN has three main functional layers starting from the output, i.e., pooling layer, intermediate layer functionality, i.e., activation or ReLU and convolution. The pooling layer simplifies the output by reducing the number of samples from the input and by reducing the number of features. The function of the activation layer is to map the positive and negative values. This process is done by maintaining the positive values and assigning negative values to zeros. The purpose of the convolutional layer is to apply convolutional filters, which triggers certain features in the images for the training purposes. Three basic layers can be arranged with different combinations by adding multiple times a type of layer, e.g., a typical CNN layer arrangement can be as a first input layer, convolutional and ReLU layers, pooling layer, convolutional and activation layers, pooling, and an output layer.

Every square at the montage of both activations is a channel at the convolutional layer’s outcome. The channel activation means the area of interest in a particular image and the number of channels means the bit number of each pixel. White pixels of the original image reflect dark pixels in the activated image, and optimistic activations reflect strong activations. A channel that’s mostly grey will not trigger the input image. The job of the pixel at the channel’s activation corresponds to exactly precisely the location from the image. A pixel in a particular stage in a channel signals the channel is activated at the location. Resize that the activations in channel 32 to possess precisely the identical size because the image as shown in Fig. 10, and also display the exact activations.

Fig. 10
figure 10

Activations in channel 32 for the input image using CNN

Full size image

By exploring channels with activations, it is easy to locate interesting channels. Locating the channel using the activation resize is as shown in Fig. 11, and it also reveals that precise activations.

Fig. 11
figure 11

Finding the strongest activation channel for the input image using CNN

Full size image

The neural networks figure out how to find features like advantages and color. In the next layers that are convolutional, the system achieves to find capabilities in the form features. After their features are built up by layers by blending functions of levels. Inquire into the fifth layer that is convolution. The working of the different layers of CNN like determine and reshape is shown in Fig. 12, and activations of the layers are also revealed.

Fig. 12
figure 12

Investigate a deeper layer of the input image using CNN

Full size image

It can be found in detail about the attention to a few of the types. Exhibit the activation at the convolution five layers. Inside this instance, the entire activation channel is not quite as fascinating for step by step capabilities as several the others and reveals solid bad (dim) in addition to favorable (gentle) activation. This channel is concentrating on confronts. Channel 5 and three further layers can be seen. A number of the channels comprise regions of stimulation which are equally dim and gentle. All these are activations that are not required and favorable. However, the activations are utilized as of this linear component which follows the exact five layers that are convolution. To explore activations, replicate the diagnosis to visualize this layer’s activations.

The process of loading, training, predicting and deploying the deep learning-based neural network, i.e. CNN as shown in Fig. 13 is elaborated step by step. The mechanism of transfer learning can be utilized using a pre-trained machine learning model. The network is pre-training with low-level features with task-specific output layers. In the next step, the output layer is replaced with our required objects to be detected, which is face, eyes and the drowsiness of the face. The training of the network is performed based on the target. The training of the network is followed by testing of the prediction and the accuracy of the network. The deployment is done once the model is trained and tested.

Fig. 13
figure 13

A deep learning-based framework for driver’s drowsiness detection system

Full size image

In contrast to the convolution fifth layer’s activations the activations of these layers’ features of the image, which have powerful capabilities. Assess whether channels 5 and 3 of those layers that are a ReLU5 layer which is used for eyes’ trigger. Closing the eye and its comparison with the image’s activations is performed. It might be understood from the activations that equally channels 5 and also three triggers about the region across the mouth, and also to a level to eyes’ area. The opening of the eyes is very necessary for the driver although the system has been trained to know the states regarding eyes. These convolutional networks can find features, although complex machine-learning procedures frequently equipped features unique for the issue. It is necessary to understand how to differentiate eyes, which can support the system to differentiate among different objects.

5 Conclusion and future works

The research would be a brilliant way towards opening new horizons of applied smart fabric and interactive textiles to replace the scenario of the “impact of drowsiness on body condition” with other factors and parameters. Support to continuous better livelihood and better family future, Since several drowsy driving incidents, have resulted in jail sentences for the driver that mostly pushes their family life in a very miserable condition. Extending this research towards the product and its development on a large scale can make an excellent contribution to the KSA economy. Multi-million Saudi riyals can be saved. Firstly, paid by individuals and businesses in issue settlements with the families of crash victims due to road accidents. Secondly, wasted in the form of victimized vehicles, buildings, and other infrastructures. Least not the last, spent in the treatment of victims.

Many precious lives can be saved from minor injuries to a total loss. Monitoring of driver’s activeness in driving by setting and measuring fatigue threshold. It is necessary to keep the drivers active from drowsiness feelings during driving through various alerting actions. Safety of human life and property loss by taking precautionary and preventive measures towards road accidents resulting from fatigue to drowsiness through smart technology.