Towards to a System for Predicting an Insufficient Wake State in Professional Drivers

Abstract

Sleep is one of the major responsible for road accidents. Their early detection can prevent many of these types of accidents. The current sleep detection systems are based mainly on the evaluation of the behaviour of the driver, which is expressed by the steering wheel movement, facial expression or eyes movement monitored by a camera. However, these systems only detect a sleep situation when it already exists, what is proved to be insufficient. It is fundamental to anticipate this state more than to realize that the driver is in the state of drowsiness, alerting the driver while he is still awake for the necessity to stop and rest. Nowadays, the technology focused on the health and well-being has been making considerable progress. Mobile and wearable devices, among other equipment with multiple sensors, are being used more frequently in areas which are directly or indirectly related to health. The increased quality and accuracy of these devices improves their reliability and credibility, allowing them to be used in more sensitive contexts, particularly in the health area. The current wearable devices have a set of sensors that allow the evaluation of biometric parameters, as well as information about body movement that can help to predict sleep state. This technology can be used to prevent accidents.

1 Introduction

1.1 Sleep Detection

The current sleep detection systems are based mainly on the evaluation of the behavior of the driver, expressed by the steering wheel movement, facial expression or eyes movement monitored by a camera. However, what the system does is to detect sleep when it already exists and this has proved to be insufficient.

It is fundamental to anticipate this state more than to realize that the driver is already in a situation of drowsiness, alerting him, while he is still awake, for the necessity to stop and rest.

The current wearable devices have a set of sensors that allows the evaluation of biometric parameters as well as information about body movement that can help to predict a sleep state [1].

The current systems are centred on equipment associated with the car [2], detecting a sleep state when it already exists.

In fact, these systems are not very used in a real context because they are expensive, are not practical or even impossible to use in the work context.

1.2 Antisleep Seat

The Antisleep Seat is intended to reduce the fatigue of the professional and prevent him from sleeping while driving. It was created by the companies Marcopolo, TWE and Multidisciplinary Center for Sleepiness and Accidents (Cemsa).

The armchair is equipped with four devices: vibrating seat; heating blanket; fans; and loudspeakers for messages, which act in the moments of fatigue and drowsiness of the driver in order to keep him alert. The controls of the chair are activated by an application of the mobile phone, developed by Cemsa, which communicates with the armchair by bluetooth.

This system acts in function of the drive time and not in function of the driver tiredness.

1.3 Brain Computer Interface (BCI)

The idea of this project is to anticipate the action of the individual by sending the vehicle an order to execute an action (such as locking) before his own initiative to carry out the action [3].

It is able to capture electrical waves from the brain and, based on these waves, predict the actions that the individual wishes to perform.

This system is interesting but it is used to anticipate the action of the individual, if his or her waking level is low, the system will also have a slow reaction, moreover, cannot be used in real context.

1.4 HealthyDrive and HealthyRoad

HealthyDrive analyses the driver’s face and sends out audible alerts when signs of tiredness and drowsiness are observed. The processing unit takes a series of metrics such as fatigue, drowsiness and stress, sounding an alert when determinate pattern occurs [4]. The HealthyRoad is a similar system that uses the mobile camera and an app that processes data. The problem with these systems is that the alert will be given when fatigue or sleep already exists. It is generic and not adjustable to each individual.

1.5 Cocomi

This system collects data through a t-shirt that is able to sense electrical signals on the driver’s body. If signs of great tiredness are detected on the driver, the t-shirt issues an alert [5]. The need to wear the t-shirt makes the system impractical for daily use, unless this t-shirt is part of the uniform.

1.6 SURF (Speeded Up Robust Feature)

This system extracts the facial features of the driver, namely the eyes condition. The proposed algorithm uses mathematical morphology in the segmentation of the eyes of the drivers, detecting if the eyes are open or closed.

The open and closed eye models are acquired at a previous stage of system calibration and should be specific to each driver [6].

1.7 Optalert

OptalertTM (Optalert 2009) are glasses in which is mounted an infrared proximity sensor. The sensor measures the amplitude of the blink of the eyes and the speed of the eyes. The glasses connect via USB to a computer (onboard) that rates intermittent eye features on a drowsiness scale. The system alerts the driver with auditory and visual alarms if drowsiness increases above a certain threshold. So far, there is a lack of in-depth evaluation and scientific analysis of the system [7].

1.8 Resume

The four most referenced characteristics of those systems are: to be suitable in the work context; to be personalized; to allow continuous monitoring; and to have learning capability. However, none of them have the four simultaneously. Table 1 shows, for each system, the presence or absence of these characteristics.

Table 1. Presence or absence of system characteristics.

In this table we can observe that systems that have learning capability, perform continuous monitoring and are customizable, are not suitable for routine use in the work context. On the other hand, the ones that are more adequate for use in a real context, don’t have the other characteristics.

So, those responsible for the development of the different existing systems emphasize on one or two characteristics, not them all.

2 Objectives

The system that we intend to create – Round Trip without Sleep – has the equipment associated with the driver, is cheap and predict the sleep before it occur. The system has as main objective to collect biometric data (heart rate and heart rate variability) and movement data (with an accelerometer and a gyroscope). Based on the individual biometric pattern it issues an alert in case of low vigilance level.

The Autonomic Nervous System (ANS), whose regulation modulates cardiovascular functions during sleep onset and the transition to different sleep stages, has a key role in the physiology of sleep [8]. It is also known that the Heart Rate Variability (HRV) analysis is a valid method for evaluating the activity of the ANS [9]. It is in this context that the HRV will be used as a marker to assess the waking state of the individual when he is driving. A reduced wake state will produce, relative to a normal wake state, a lower HR value and a higher HRV value.

There are in the market several mobile and wearable devices that allow to evaluate HR and HRV. They are devices with good reliability, precision and intuitive use, not requiring special abilities for their use [10]. The most of them use a sensor based on reflexive photoplethysmography [11, 12] (the emitter and receptor are at the same side). Some clinical devices use transmissive photoplethysmography (the emitter and receptor are at the opposite side), like the finger oximeter, but they are not practical for continuous use. So, in this project we used a wrist device with a sensor based on reflexive photoplethysmography.

To achieve that goal this project was divided in two phases. The first phase consists in create individual patterns (baseline and driving pattern phase) that will be the basis for the low vigilance level alert system. The seconds phase will consist in the improvement of the wrist device, in order to get the accurate data in an efficient way.

3 Method

We use a sample with 30 drivers that work in a passenger transport company in Aveiro, Portugal, whose data will be collected for 15 consecutive days. In first place, in order to predict an insufficient vigilance level, it is necessary to establish two patterns: baseline pattern and driving pattern.

The baseline pattern allows to determine the biometric characteristics of each individual during an entire day. The driving pattern defines those characteristics when the individual is driving. To determine any of these patterns, we need to collect data about HR and HRV. Beyond these data, for baseline pattern we will analyse the answers to a stress questionnaire (pathologic stress is a cause for low vigilance) that will be administrated in the beginning and at the end of the data gathering period. For driving pattern, we will collect data about movement (through the accelerometer and gyroscope present in a wearable device).

These data will be used to determine individual and driving patterns through classification algorithms. Figure 1 illustrates the method.

Fig. 1.

Method for determine de individual pattern and good vigilance pattern in drive.

At the end of the data collection at the moment each driver returned the device, we requested a response to a questionnaire of acceptability and usability (extracted from TAM3), in order to understand the level of usability of the system and its acceptability by users and to promote improvements based on the analysis of the given answers. This questionnaire was also administered to another set of individuals (not only drivers) who used the device for 3 days.

Later, these patterns will be used for comparison with the pattern produced during the continuous monitoring. If the difference between the patterns is significantly different, an alert is issued. In the case of the basic pattern, the driver must not start driving if the alert is issued. Figure 2 show the comparison for baseline pattern.

Fig. 2.

Pattern comparison before the begin of driving.

In the case of the driving pattern, signals will be issued to raise the level of vigilance and the driver must stop the vehicle as soon as possible. Figure 3 present the comparison process to driving pattern.

Fig. 3.

Pattern comparison during driving.

When a significant difference is detected between the observed pattern and the pattern defined as the driving pattern, an alert can be sent to various receivers, such as the bus, someone in a telephone exchange, a mobile phone, the own wearable device, or any other device that can be important. The driver will always be informed about his low alertness and must stop the vehicle as soon as possible, naturally, complying with the required safety regulations.

4 Results

For the collection of data, an application was developed for mobile devices. The application gathers information continuously from the wearable device and makes it available to the individual. The app was developed with Xamarin VisualStudio Platform for Android and IOS and the chosen database server was the PostGreSQL.

The wearable device used in this project has a 3-axis accelerometer, a gyroscope and a photoplethysmography sensor to get the HR and the HRV data. In the HR case, the device processes the data and presents the HR value with 1 Hz frequency. For HRV the device provides the time between heart beats. The accelerometer and gyroscope produce, in each second, several results.

At this stage of the project we are defining, together with the drivers and enterprise managers of the passenger transport company, the information to be made available to each one as well as the way of visualization, taking into account the ethical and legal aspects involved. Figure 4 shows an example of biometric data presentation.

Fig. 4.

Heart rate and heart rate variability in real time.

Also, with the enterprise managers collaboration, we created a set of dashboards to present other kinds of information produced by the system. Figure 5 show how the information is presented.

Fig. 5.

Dashboard example for several information type.

For the analysis of stress of each individual, whose values will be considered for the definition of the standards, a module was constructed to receive the responses of the users and the level of stress will be determined by the analysis to these answers. Figure 6 shows the questionnaire responses module contained in the system.

Fig. 6.

Questionnaire response example.

Individuals respond by touching a circle and can anulate the answer by tapping the already selected circle. To submit the answer, they must touch de red button (at the bottom right).

5 Conclusions, Limitations and Future Work

The work is in progress and for now we have a system that collects biometric data, gives feedback in real time for drivers, allows responses to a questionnaire and assess stress level. The system is easy to use and the dashboards make clear the information about the several parameters produced. The collaboration between the develop team and the final users was fundamental for the success of the system concerning with visualization.

However, the data that we have don’t allow, yet, to establish the two types of patterns that we want because some data was lost during the gathering process. The wearable device used has an android wear 1.0 operating system that needs a mobile phone to collect data. This revealed to be a problem. The communication between the wearable device and the mobile phone is made through Bluetooth what implies that the two devices must be close to each other for the system to work. This is a limitation and that’s why some data was lost. The android wear 2.0 solve this problem, since it doesn’t need the mobile phone, but for now there’s not a device that provides HRV information (provide only HR information). The good news is that some enterprises annunciate that they will launch these types of devices briefly.

The questionnaire on acceptability and usability, consisting of 9 questions (TAM1 to TAM9) closed on a 7-point Lickert scale (where 1 represented “Strongly Disagree” and 7 represented “Strongly Agree”), was answered by the 29 individuals who agreed to participate in the study.

The analytical process started with the evaluation of the Alpha-Cronbach value whose result (0.880) suggests that the items of the questionnaire have a good internal consistency contributing to the measurement of the latent variable (in this case the level of acceptance of the proposed technology).

The Mann-Whitney test was performed to find out if there is a significant difference in the responses regarding the gender and the respondent indicates that it should not consider that such a difference exists.

There are few correlated variables (questions) and when the correlation (Spearman’s correlation was used) exists it is weak. However, based on these correlations and in the application of a Wilcoxon test (matching all questions two to two), we could establish 3 groups: “The interaction with technology is clear and understandable”, “Interaction with technology does not require much mental effort”, “It’s easy to use technology”; “It’s nice to use technology”, “The procedure (process/steps) in the use of technology is enjoyable”, “It was fun to use technology”; and “If it is available in the future, I would use the technology”, “From the experience I had with technology, I would use it in the future”. The question “Technology does what I intend to do easily” it’s not correlated with any other question.

The question “Technology does what I intend to do easily” is clearly the one with the lowest value in the assessment, while the questions of the first group stands out positively. The questions of groups 2 and 3 have similar evaluations. Figure 7 shows the results of the questionnaire.

Fig. 7.

Dashboard example for several information type.

In summary, users appreciated the system favourably, indicating that it is intuitive and easy to use, and that they would use it in the future. Yet, some says that it does not do what they want so easily, seeming contradictory to the idea that it is easy to use. This difference is due to the fact that the system allows other actions (which are not related to the system being evaluated) and in these the operational complexity is greater.

The work continues and the next step is to analyse the collected data in order to establish the patterns needed to create the alert system.