Keywords

1 Introduction

Microsleep is a transitional state between waking and sleeps in which vigilance, or the capacity to maintain one’s attention on a task, is generally decreased. This can be a significant issue when performing tasks that require sustained attention, such as driving [1]. Detection microsleep can be divided into two categories which are visual features and non-visual features. The visual features were focusing the techniques by using computer visions. Based on previous studies, visual features can be divided into eye state analysis, eye blinking analysis, mouth, and yawning analysis, and facial expression analysis [2, 3]. Besides, a drowsy individual can be identified by their gaze pupil movements [4]. It was evident that a sleepy person has a more restricted gaze region than an alert person. Additionally, a drowsy person has fewer saccadic movements than when they are awake. Subsequently, a fatigued individual can be identified by the eyelids’ movement [4]. Drowsy individuals blink significantly slower than awake individuals. A fatigued individual will close their eyes for a more extended period than an awake individual. To put it simply, a drowsy person has a longer duration of eye closure than an alert person [2].

The research concentrated on facial landmark detectors for determining the contours of the eyes and eyelids [5]. The eye aspect ratio (EAR) was derived from the detected images to estimate the eye-opening state [6]. If an eye is open, the EAR has been mostly constant. When an eye is closed, the EAR approaches zero. Because both eyes blink in perfect sync, the EAR of both eyes is averaged [5]. The face must be detected using a Viola-Jones face detector available in the OpenCV library to detect the eye landmarks [7].

Additionally, subjective measures can be used to analyze driver's sleepiness. There were five standardized subjective fatigue or sleepiness methods to analyze sleepiness, such as Karolinska Sleepiness Scale (KSS), Visual Analogues Scale (VAS), and Ep- worth Sleepiness Scale (ESS) [8]. Subsequently, KSS has become the most widely used tool for subjective self-assessment of microsleep in recent years. KSS consists of a 9-point scale at first (1 = extremely alert, 2 = very alert, 3 = alert, 4 = rather alert, 5 = neither alert nor sleepy, 6 = some signs of sleepiness, 7 = sleepy, but no difficulty remaining awake, 8 = sleepy but some difficulty to keep awake, and 9 = extremely sleepy, great difficulty to keep awake, fighting sleep). However, a modified version of KSS contains one other item: 10 = extremely sleepy. Thus, the modified KSS will be mating with the fatigued driver.

Existing microsleep detection methods are unsuitable for use while driving, as the device or components will be attached to the driver’s body. Thus, the purpose of this research is to develop a MDD capable of detecting the driver’s eye aspect ratio (EAR) while driving. The proposed MDD employs a camera mounted directly in front of the driver, which is more conducive to use while driving. Additionally, this study concentrated on the warning system. Therefore, once the driver exhibits signs of sleepiness, MDD will activate the alarm. Following that, the driver’s behavior will be classified using the Karolinska Sleepiness Scale (KSS).

2 Project Methodology

2.1 Microsleep Detector Device (MDD) Development

The working mechanism of the MDD for the entire process, starting from initialization to the warning system. The experiment starts with detecting the face using a Vio-la-Jones face detector available in the OpenCV Library. Next, the device received input from the camera attached to the driver and processes the frame captured for microsleep detection (Fig. 1).

Fig. 1
Flowchart of Micro sleep detector Device (MDD) begins with Start, Camera Started, Face Detection, Capture Frame, Eye Detection, Ear<= Threshold, Warning System Triggered, Close Application, End. Two conditions in Face Detection labeled No are connected with Ear<= Threshold, and another is connected with Close Application.

Flowchart of microsleep detector device (MDD)

Full size image

Previous studies have based their criteria on facial landmark detectors to localize the eyes and eyelid contours. First, as landmarks detected in the image with face, the EAR was derived from an estimate of the eye openness state. Then, each video frame is computed; the eye landmarks are detected. Finally, the EAR between the height and width of the eye is calculated based on the Eq. 1, where p1 till p6 are the 2D landmark locations, depicted in Fig. 2.

$${\text{EAR}} = {{\left( {\left| {\left( {\left| {{\text{p}}2 - {\text{p}}6} \right|} \right)} \right| + \left\| {{\text{p}}3 - {\text{p}}5} \right\|} \right)} / {\left( {2\left\| {{\text{p}}1 - {\text{p}}4} \right\|} \right)}}$$
(1)
Fig. 2
This picture represents landmark of the eye. The line joining the points p1 and p4 represents the width of the eye, whereas the points p2, p3, p5, p6 represents the corners of the iris.

Eye landmark [5]

Full size image

The value of EAR is constant when the eyes are opened and getting close to value zero when the driver is closing their eyes. Thus, the device collects EAR data for both eyes synchronously and averaged the value to find the threshold for every respondent. Subsequently, the EAR threshold for every respondent had been stored in the device to proceed with the warning system. As the value EAR of each respondent equaled or exceeded the threshold for five milliseconds, the alarm will be triggered to alert the driver.

2.2 Participants

The experiment was conducted with five male and five female drivers between the ages of 25 and 24 years old (Mean = 24.2, SD = 0.422). The experienced driving of respondents showed between one to eight years, which proved they had a valid driver's license. All the respondents have no record of smoking behavior, and their blood pressure is normal, which will not affect their performance during the experiment.

2.3 Experiment Setup

The experiment was carried at the Ergonomic Laboratory of the Faculty of Mechanical Engineering, UiTM, Shah Alam. A driving simulator was set up before the experiments start, as illustrated in Fig. 3. The MDD’s camera module has been mounted on the top of the computer to capture video of the driver, as shown in Fig. 3.

Fig. 3
A Structure layout Office with table Speaker, LCD Screen, Power supply, Camera Module,

Layout of device

Full size image

Next, the respondent’s blood pressure and temperature were recorded to ensure that they were in excellent health during the experiment. Subsequently, the respondent has been explained the procedure of the experiment. Thus, the respondent is given a consent form to ensure that the respondent agrees to participate in the experiment. Following that, before the experiments start, the respondent’s information, such as age, height, weight, and driving experience, was collected via General Questionnaire.

2.4 Experiment Procedure

After developing a device capable of detecting the EAR, an experiment on micro-sleep was conducted. The experiment procedure comprises four sessions within two and half hours. The duration for every session is 30 minutes following 10 minutes breaks. In all sessions, the participants were required to drive in driving simulator environment. The driving condition and environment is the same throughout the sessions. Following that, the first session starts with taking value EAR, and observed the driver’s behaviour simultaneously. Value EAR is taken by MDD for every 150 frames. Thus, every five minutes for 30 minutes, the researcher observed the driver’s behaviour using the modified KSS and recorded the frequency of their behaviour. The first session’s procedure is repeated until the third session.

Following that, the experimental procedure for the fourth session is continued with the warning alarm system. The third session's mean value of EAR was set in as threshold. The warning system is triggered during the fourth session when the threshold value exceeds the EAR. Following the completion of the fourth session, the respondent was required to complete a Google Form regarding feedback on the device’s performance.

2.5 Driver Behaviour Sleepiness Scale

The driver behaviour sleepiness scale has been mating with the criteria of KSS accordingly, as shown in Table 1. The KSS level of this experiment ranges from 4 to 8, with drivers exhibiting specific behaviours such as yawning, conversing, and rubbing their eyes, as illustrated in Table 2.

Table 1 Modified karolinska sleepiness scale (KSS)
Full size table
Table 2 Driver behavior
Full size table

2.6 Statistical Analysis

IBM SPSS Statistic was used to analyze all the respondents’ data after being sorted out in Microsoft Excel. IBM SPSS Statistic analyzed the data to reveal the significant result in each measurement. Repeated measure ANOVA and univariate measure ANOVA were used to analyze the Eye Aspect Ratio (EAR) and observed KSS level and Questionnaire as the software will determine the significance for the data that have been entered is less than (p < 0.05). The significance level or alpha level of 0.5 (5%) has been selected to run the test of all results in this experiment. Repeated measure ANOVA was used to determine the statistical significance in all gender results and univariate measure ANOVA was used to determine the statistical significance in the overall result of all participants. Wilk’s lambda is a test statistic that’s reported in how well each level of independent variable contributes to the model.

3 Results and Discussion

3.1 Physiological Measurement (EAR)

A repeated-measures ANOVA was conducted with EAR of session one till three, as the within-subjects factor. The results of the repeated measure ANOVA indicated a significant value of EAR on every session, Wilks’ Lambda = 0.718, F(18, 978) = 9.795, p < 0.05, η2 = 0.153.

Figure 4 illustrates the average value of EAR from session one to session three, separated by gender. The results of the repeated measure ANOVA indicated a significant value of EAR on every session, Wilks’ Lambda = 7.305, F(2.000, 497) = 9.795, p < 0.05, η2 = 0.029. Thus, from session one to session three, a male’s EAR value trend is increasing. However, the female EAR increased from session one to session two but decreased slightly from session two to session three. An evaluation of this result suggests that males are not sleepy during driving for the entire experiment as they are accustomed to driving for a prolonged time. This can be demonstrated that female drivers are more sleepy than male drivers. For the whole experiment, the difference in EAR values between genders was statistically significant (p = 0.003). Gender has been shown to profoundly affect lifestyles, stress levels, and, more broadly, all aspects of human behavior [9]. To illustrate this issue, homeostatic pressure to sleep increases more quickly in females, leading to a more decisive necessity for sleep in females than in males [10].

Fig. 4
A Graphical Representation. The X-Axis denotes Session ranges from (1 to 3) and Y-Axis Denotes Ear, ranges from (220 to 245), Blue denotes Male, Red denotes Female. Male gradually increases and Female gradually decreases after session 2.

Line graph of mean EAR by gender

Full size image

Figure 5 present the analysis of the overall mean ten respondents for each session. The results of the repeated measure ANOVA indicated a significant value of EAR on every session, Wilks’ Lambda = 0.974, F(2.000, 498.000) = 6.752, p < 0.05, η2 = 0.026. The value of EAR is slightly level up from session one to session two. This is plausible given the respondent’s excitement while driving the simulator for the first time.

Fig. 5
A Graphical Representation of the Eye Aspect Ratio. X-Axis denotes Session ranges from (1 to 3) and Y-Axis denotes Ear ranges from (224 to 234)

Line graph of mean EAR for the entire session

Full size image

The pairwise comparison revealed a significant value between session one and session two (p = 0.000). The graph indicates that the growth rate from session two to session three is noticeably slower. Therefore, this provides evidence that the respondent is getting sleepy after an hour of driving. The result of EAR is higher at session three than session one, which session three is supposedly lower. This is because the respondent begins the experiment with drowsy state as they sleep at 5 a.m. According to a previous study, drivers become sleepy when they get little sleep prior to driving [11].

3.2 Subjective Measurement (KSS)

A repeated-measures ANOVA was conducted with KSS of sessions one, two, three, and four, respectively, as the within-subjects factor. Also, ten respondents as between-subjects factors. The results of the repeated measure ANOVA indicated a significant value of KSS on every session, Wilks’ Lambda = 0.381, F(27, 140.827) = 2.042, p < 0.05, η2 = 0.275.

Figure 6 illustrates the average value of KSS from session one to session three, separated by gender. The results of the repeated measure ANOVA indicated a significant value of KSS on every session, Wilks’ Lambda = 0.964, F(3.000, 56.000) = 0.696, p < 0.05, η2 = 0.036. The trend line of male’s KSS increased dramatically from session one to session three. An evaluation of this analysis proves that the male’s driver is alert at the first session but changed to neither alert nor sleepy at session three. Throughout the experiment, the male driver demonstrated microsleep behavior. Additionally, the trendline of female’s KSS also increased up noticeably from session one to session three. Thus, the female driver’s KSS at session one is significantly different than the male driver’s KSS, indicating that the female driver is trying to stay alert while driving. Thus, overall, the genders had affected the KSS level during driving. As was the scenario in session four, a warning system was installed to alert the drowsy driver. Therefore, the KSS level is expected to decrease between sessions three and four for both female and male drivers. What can be seen is that the male KSS level decreases, indicating a transition from sleepiness to alertness. Following that, the female KSS level gradually decreased which neither awake nor sleepy while driving. To summarize, the graph indicates that they return to the same state as in session one for female drivers but are more alert than in session one for male drivers after the warning system is triggered.

Fig. 6
A Graphical Representation of the KSS by gender. The X-Axis denotes Session ranges from (1 to 4) and Y-Axis denotes KSS ranges from (4.50 to 6.50), Blue denotes Male, Red denotes Female. According to the graph, Male gradually decreases, Female gradually increases, and Error bars are 95% CI.

Line graph of mean KSS by gender

Full size image

Figure 7 depicts the average value of KSS for each session from one to four, based on the mean of ten respondents. The results of the repeated measure ANOVA indicated a significant value of EAR on every session, Wilks’ Lambda = 0.849, F(3.000, 57) = 3.366, p < 0.05, η2 = 0.151. The KSS level shows increasing value from session one to session three. According to the driver behavior sleepiness scale, drivers begin to exhibit signs of sleepiness such as yawning while driving (Table 3). What this chart clearly shows is the steadily maintained KSS level from session two to session three. This would be to say; the respondent exhibited the same level of KSS. Following that, the graph demonstrates a slight decrease between sessions three and four. The KSS level decreases due to the driver’s alertness while driving when the warning alarm sounds. Following that, the significance value between sessions one and two is p = 0.013 which is significant. The same holds for session one, and session three demonstrated a statistically significant value (p = 0.008). This indicated that the more hours spent driving, the higher the KSS level. This is due to the presence of a warning system during session four. This hypothesis is remarkably similar between sessions three and four, as indicated by the p-value (p < 0.016). Based on previous research, the participants who were in sleepy conditions, the sleepiness scale reported increasing, which leads to falling asleep [11].

Fig. 7
A Graphical representation of KSS for entire experiment, X-Axis denotes Session, ranges from (1 to 4), Y-Axis Denotes KSS, ranges from (4.80 to 6.00) and Error bars 95% CI.

Line graph of mean KSS for entire experiment

Full size image

3.3 Performance of Microsleep Detector Device (MDD)

Figure 8 depicted an average response to a questionnaire during an experiment. The results of the repeated measure ANOVA indicated a significant value of the questionnaire’s answer, Wilks’ Lambda = 0.106, F(4.000, 6.000) = 12.702, p < 0.05, η2 = 0.894. The average value of ten respondents’ responses regarding the camera’s position is (M:2), which disagrees with the statement. According to the respondent, the camera’s position did not obstruct them while driving because the camera was not attached to their body. Therefore, the respondent can drive the simulator without any obstruction.

Fig. 8
A bar graph with X-axis as Questionnaire and Y-axis as Scale, illustrates about the different scales for different questionnaires like the position of camera, the performance of device, the performance of alarm, time response and overcome the sleepiness. The higher scale is the time of response and the lower one is the position of camera.

Bar chart of questionnaire’s answer

Full size image

Following that, the device’s performance affirms the answer of (M:5), strongly agreed. Once the warning alarm is triggered after the threshold exceeds the EAR, the device’s performance can be justified. Additionally, the device can detect the respondent’s face and ear while driving. Furthermore, the MDD can detect the EAR even if the respondent is wearing spectacles. Regarding that, the question is about the alarm’s performance. The average response of ten respondents was (M:5), which is strongly agreed on such. The warning is triggered during the fourth session following the threshold setting in MDD. Thus, when the threshold was maintained for five milliseconds, the warning system sounded. Furthermore, the respondent is intent about the time of response. When the threshold exceeds the EAR, the warning system does not delay. Finally, the respondent agreed unequivocally that MDD could overcome microsleep. This is demonstrated when the respondent’s KSS level drops slightly from session three following the triggering of the warning.

4 Conclusion and Recommendation

In a nutshell, the physiological and subjective measurement analysis of driver behaviour (sleepiness) was indicated. The physiological measurements were focused on the driver's eye aspect ratio EAR by using MDD, while the subjective measurements using the KSS and a questionnaire. The EAR demonstrates statistical significance for each subject as p < 0.05. The analysis of EAR by gender statistically significant as the research concluded female drivers are more drowsy than male drivers. Furthermore, the EAR of ten respondents indicating statistical significance as the drivers become drowsy, their EAR approaching to zero. Next, the results of KSS were statistically significant for each subject, which means they tend to sleepy during driving. The statistical result of gender for KSS also showed significant value as male drivers showed more fatigue behaviour than female drivers. Additionally, the mean of KSS for ten respondents also showed the statistical result as p < 0.05. This demonstrates that the longer the driving hour, the higher the KSS level. The KSS level of the drivers decreasing as the warning system was triggered.

However, there is still room for improvement to present a great result to MDD and analyse driver behaviour (sleepiness). The MDD can be improved by utilizing advanced technology and a high-quality camera module to capture high-resolution video frames. Additionally, to capture genuine microsleep, the number of video frames should be reduced to 35. Subsequently, a suitable environment should be created for drivers to experience the actual driving situation, leading to microsleep. Finally, the number of respondents should be increased to 50, as the number of respondents is critical for accurately representing the population. Overall, this research confirms previous findings and presents novel findings that aid in evaluating existing technologies and inform future development of microsleep detection systems. In this sense, this research contributes to the evolution of tailored technology for microsleep detection and warning based on the characteristics of individual drivers.