Keywords

1 Introduction

Driving is a hard task that drivers always have to watch road and make right operation decisions on time. They are subsequently prone to fatigue after driving for a long time and the ability to control the vehicle will be weakening. Studies show more than 35% of serious traffic accidents are caused by fatigue driving [1]. Therefore, it is necessary to recognize driver’s fatigue state effectively by some advanced methods for reducing traffic accidents caused by fatigue driving.

In the existing literature, researchers have made full use of many factors to recognize fatigue driving, which include drivers’ facial expression [2, 3], physiological characteristics [4, 5] and operation behavior [6,7,8]. Facial expression-based methods are recognized by analyzing changes of facial expression, such as yawning and blinking. Nevertheless, many external factors, such as illumination and head position change, have an important impact on recognition accuracy. Physiological characteristics-based methods are high in accuracy, but majority of them need to place many sensors on drivers’ body for data acquisition, which often make the drivers uncomfortable. Operation behavior-based methods recognize drivers’ fatigue state by analyzing operation changes of steering wheel and vehicle status, such as the standard deviation of the steering wheel angle and lane departure, which can avoid the shortcomings of the previous methods and have become an important part of fatigue driving recognition research. Wei et al. [9] chose the lateral position and steering wheel angle as the input data of neural network and trained it by BP algorithm. His experiment results showed that this method is effective to recognize drivers’ fatigue level. Mcdonald et al. [10] analyzed the characteristics of drivers’ steering data and established a fatigue driving recognition model based on random forest algorithm, which had a higher classification accuracy.

However, the existing majority of operation behavior based methods are to collect the related data by installing various sensors on steering wheel and vehicle, which are too complicated to be practical. In recent years, wearable smart watch with a variety of sensors, have been widely popularized and used to detect human body motion in some research works. Mannini et al. [11] utilized wrist-worn accelerometer to collect wrist and ankle motion data and classified behavior into four classes: sedentariness, cycling, ambulation and other, and the results showed high classification accuracy for ankle data (95.0%) and wrist data (84.7%). Shoaib et al. [12] used wrist-worn motion sensors to recognize hand gestures and it can accurately detect 13 activities. These researches have showed that motion sensors in wearable devices can accurately reflect human’s behavior.

According to the above analysis, this paper studies a fatigue driving recognition method based on steering operation by using a smart watch. Thirteen characteristic parameters are extracted from smart watch’s sensors based on driving simulation experiment and the fatigue driving recognition model is established based on support vector machine. The test results prove that we can use smart watch to recognize the drivers’ fatigue or normal state. This work uses no other external sensors, bringing about a low cost, user-friendly and simple recognition system.

2 Experiments of Data Acquisition

2.1 Experimental Design

Driving experiments are carried out on the Multi-Vehicle Cooperative Simulation Platform, as shown in Fig. 1, in Key Laboratory of Dependable Service Computing in Cyber Physical Society Ministry of Education (CPS-DSC), Chongqing University. The simulation road is a 100 km long highway of part of the Yuxiang highway located between Chongqing and Hunan province, with two lane in one direction. The smart watch used in the experiment is TicWatch, and it has acceleration sensor, gyroscope sensor, gravity sensor and geomagnetic sensor. Ten participants (six men and four women have an average of 3.5 years driving experience) aged between 24 and 30 years old took part in the experiment from 13:00 pm to 14:30 pm, who are most likely to be fatigued in this period. They have had a good command of the driving scene before the experiment and are asked to wear smart watch for driving during experiments, as shown in Fig. 2. Besides, all participants are required to reduce the lane change and keep the speed of 80 ± 10 km/h when the vehicle is moving.

Fig. 1
figure 1

Driving simulation platform

Full size image
Fig. 2
figure 2

Simulation scene

Full size image

2.2 Data Acquisition

In the process of the whole experiments, three types of data are recorded synchronously, including:

  1. (1)

    The data values of smart watch’s acceleration sensor, gyroscope sensor, gravity sensor, and geomagnetic sensor are recorded at the frequency of 10 Hz;

  2. (2)

    Video of the drivers’ facial expression;

  3. (3)

    The drivers’ self-evaluation of fatigue or normal state every ten minutes according to the Karolinska Sleepiness Scale (KSS) [13].

The video of drivers’ facial expression and the drivers’ self-evaluation are used to judge the drivers’ states. There may be a deviation between drivers’ self-evaluation and actual fatigue degree. In order to improve the accuracy of the fatigue degree, three trained experts evaluate the drivers’ states according to the video in different periods and modify the KSS score of the drivers’ self-evaluation. In this study, driving state of the driver is classified into two classes. If KSS score ≤3, the driver is under normal state and is under fatigue state while KSS score ≥7 [14].

3 Methods

3.1 Method Design

In order to establish the fatigue recognition model and recognize the drivers’ states effectively, there are several key steps to do, including driving simulation experiment, coordinate system transformation, characteristic extraction (principal component analysis), training and testing recognition model of fatigue driving based on SVM, as shown in Fig. 3.

Fig. 3
figure 3

Flow diagram of fatigue recognition model

Full size image

3.2 Data Processing

The motion sensors of smart watch use a standard 3-axis device coordinate system to express data values. The X-axis is horizontal and points to the right, the Y-axis is vertical and points up, and the Z-axis points toward the outside of the screen, as shown in Fig. 4. The attitude of smart watch will be constantly changing in the course of data acquisition and the data values of gravitational acceleration on the three axes will be changing too. Therefore, it is critical to eliminate the effect of gravitational acceleration on the data values of acceleration sensor.

Fig. 4
figure 4

Device coordinate system

Full size image

We transform the data values of acceleration sensor from device coordinate system to the world’s coordinate system whose X-axis is tangential to the ground and roughly points East, Y-axis is tangential to the ground and points towards the magnetic North Pole, and Z-axis points towards the sky vertically (as shown in Fig. 5):

Fig. 5
figure 5

World’s coordinate system

Full size image
$$ \left[ {\begin{array}{*{20}c} {x^{\prime}} \\ {y^{\prime}} \\ {z^{\prime}} \\ \end{array} } \right] = R \cdot \left[ {\begin{array}{*{20}c} x \\ y \\ z \\ \end{array} } \right] = \left[ {\begin{array}{*{20}c} {R_{1} } & {R_{2} } & {R_{3} } \\ {R_{4} } & {R_{5} } & {R_{6} } \\ {R_{7} } & {R_{8} } & {R_{9} } \\ \end{array} } \right] \cdot \left[ {\begin{array}{*{20}c} x \\ y \\ z \\ \end{array} } \right] $$
(1)

where \( x,y,z \) express the values of acceleration sensor in the device coordinate system and \( x^{\prime},y^{\prime},z^{\prime} \) are in the world’s coordinate system, R expresses a 3 × 3 row-major matrix that can be computed with the values of gravity sensor and geomagnetic sensor. When the values of acceleration sensor in the world’s coordinate system is got, only the value of Z-axis contains the constant of gravitational acceleration and we can eliminate it directly.

3.3 Characteristics Extraction

3.3.1 Analysis and Extraction of Characteristics

The data of drivers’ steering operation includes rotation acceleration and angular velocity of steering wheel that we can get from smart watch’s motion sensors. Take rotation acceleration for an example, Figs. 6 and 7 are the data change graphs of rotation acceleration (relative to the world’s coordinate system) under normal driving and fatigue driving respectively. It can be seen from the graphs that the steering wheel is corrected frequently and the acceleration amplitude of correction is small under normal driving. However, the correction frequency decreases and the correction amplitude of acceleration becomes larger under fatigue driving. To summarize, the correction frequency and amplitude are different under two different driving states.

Fig. 6
figure 6

Acceleration under normal driving

Full size image
Fig. 7
figure 7

Acceleration under fatigue driving

Full size image

In order to describe different driving states more accurately, a series of statistical characteristic parameters are defined as listed in Table 1. Taking the duration of fatigue operation and real-time of the recognition model into account, the time window in data series is ten seconds when extracting each characteristic parameter. PLAA and PSAA mean the percentage of the acceleration amplitude that is greater or less than a threshold of the time window:

Table 1 Characteristic parameters and meaning
Full size table
$$ PLAA = \frac{{\sum\nolimits_{i = 1}^{n} {LA_{i} } }}{n} \times 100\% ,LA_{i} = \left\{ {\begin{array}{*{20}l} {1,} \hfill & {if\,A_{i} \ge mean\_Q2\left( {AF} \right)} \hfill \\ {0,} \hfill & {if\,A_{i} < mean\_Q2\left( {AF} \right)} \hfill \\ \end{array} } \right. $$
(2)
$$ PSAA = \frac{{\sum\nolimits_{i = 1}^{n} {SA_{i} } }}{n} \times 100\% ,SA_{i} = \left\{ {\begin{array}{*{20}l} {1,} \hfill & {if\,A_{i} < mean\_Q2\left( {AN} \right)} \hfill \\ {0,} \hfill & {if\,A_{i} \ge mean\_Q2\left( {AN} \right)} \hfill \\ \end{array} } \right. $$
(3)

where n is the data number of the time window, \( A_{i} \) expresses the value of synthetic acceleration in data series, \( mean\_Q2\left( {AF} \right){\kern 1pt} \) is the mean value of synthetic acceleration that is greater than median in data series under fatigue state, and \( mean\_Q2\left( {AN} \right){\kern 1pt} \) is the mean value of synthetic acceleration that is greater than median in data series under normal state. PLAVA and PSAVA mean the percentage of the angular velocity amplitude that is greater or less than a threshold of the time window:

$$ PLAVA = \frac{{\sum\nolimits_{i = 1}^{n} {LV_{i} } }}{n} \times 100\% ,LV_{i} = \left\{ {\begin{array}{*{20}l} {1,} \hfill & {if\,AV_{i} \ge mean\_Q2\left( {AVF} \right)} \hfill \\ {0,} \hfill & {if\,AV_{i} < mean\_Q2\left( {AVF} \right)} \hfill \\ \end{array} } \right. $$
(4)
$$ PSAVA = \frac{{\sum\nolimits_{i = 1}^{n} {SV_{i} } }}{n} \times 100\% ,SV_{i} = \left\{ {\begin{array}{*{20}l} {1,} \hfill & {if\,AV_{i} < mean\_Q2\left( {AVN} \right)} \hfill \\ {0,} \hfill & {if\,AV_{i} \ge mean\_Q2\left( {AVN} \right)} \hfill \\ \end{array} } \right. $$
(5)

where n is the data number of the time window, \( AV_{i} \) expresses the value of synthetic angular velocity in data series, \( mean\_Q2\left( {AVF} \right) \) is the mean value of synthetic angular velocity that is greater than median in data series under fatigue state, and \( mean\_Q2\left( {AVN} \right) \) is the mean value of synthetic angular velocity that is greater than median in data series under normal state. We got a total of 953 samples and randomly selected 600 samples are used to train the recognition model and the other samples are used to test the model.

3.3.2 Principal Component Analysis

We know the number of characteristic parameters is far more than ten because the coordinate system has three axes in Table 1. There may be a strong correlation among these parameters and they have a significant impact on the performance of the recognition algorithm. Thus, it is import to reduce dimensionality of these parameters consisting of some interrelated parameters, while retaining as much as possible of the variation present in them. In this paper, the normalized characteristic data is analyzed by the method of Principal Component Analysis (PCA) and the number of characteristic parameters is reduced from 18 to 13. As shown in Fig. 8, the final 13 principal component parameters are inputs of the recognition model.

Fig. 8
figure 8

Contribution rate of the principal components

Full size image

3.4 Recognition Model

3.4.1 Support Vector Machine (SVM)

Since the SVM was firstly proposed in 1995 by Cortes and Vapnik [15], it had been widely used for its good classification performance. The principle of support vector machines is to find an optimal hyperplane that separates two classes and is at equal distance from the two. The margin between the optimal hyperplane and the two classes is maximal. As shown in Fig. 9, H is an optimal hyperplane and it is at equal distance from H1 and H2.

Fig. 9
figure 9

Optimal hyperplane

Full size image

Suppose a sample set to be \( \left\{ {\left( {x_{i} ,y_{i} } \right)} \right\},i = 1,2, \ldots ,m,x \in R^{n} ,y_{i} \in \left\{ { - 1,1} \right\} \), where there are m samples for training the SVM, \( R^{n} {\kern 1pt} \) is the n-dimensional feature vector and \( {\kern 1pt} y_{i} {\kern 1pt} \) is the category label. When the feature vectors of samples in different classes are linearly separable, the hyperplane \( \omega \cdot x + b = 0 \) must satisfy the following inequalities that can separate all samples accurately:

$$ y_{i} \left( {\omega \cdot x_{i} + b} \right) \ge 1,i = 1,2, \ldots ,m $$
(6)

The margin can be calculated as:

$$ \mathop {\hbox{min} }\limits_{{\left\{ {x_{i} \left| {y_{i} = 1} \right.} \right\}}} \frac{{\omega \cdot x_{i} + b}}{\left\| \omega \right\|} - \mathop {\hbox{max} }\limits_{{\left\{ {x_{i} \left| {y_{i} = - 1} \right.} \right\}}} \frac{{\omega \cdot x_{i} + b}}{\left\| \omega \right\|} = \frac{2}{\left\| \omega \right\|} $$
(7)

The optimal hyperplane requires the maximization of the margin, which is to mean:

$$ \hbox{max} \frac{1}{2}\left\| \omega \right\|^{2} = \frac{1}{2}\left( {\omega \cdot \omega } \right) $$
(8)

After using the Lagrange dual transformation, the optimal solution of the following objective function is the optimal hyperplane:

$$ \left\{ {\begin{array}{*{20}l} {\hbox{max} \sum\limits_{i = 1}^{m} {\alpha_{i} } - \frac{1}{2}\sum\limits_{i,j = 1}^{m} {\alpha_{i} } \alpha_{j} y_{i} y_{j} \left( {x_{i} \cdot x_{j} } \right)} \hfill \\ {s.t.\,y_{i} \left[ {\left( {\omega \cdot x_{i} } \right) + b} \right] - 1 \ge 0,\sum\limits_{i = 1}^{m} {y_{i} \alpha_{i} = 0} {\kern 1pt} ,\alpha_{i} \ge 0,i = 1,2, \ldots ,m} \hfill \\ \end{array} } \right. $$
(9)

where \( {\kern 1pt} \alpha_{i} {\kern 1pt} \) is Lagrange multiplier of sample data i.

When the classification problem is nonlinearly separable, the n-dimensional feature vector can be mapped into a higher dimensional space by using the mapping relationship of \( z =\Phi \left( x \right) \) and the nonlinear classification problem will be linearly separable by using appropriate kernel function \( k\left( {x_{i} ,x_{j} } \right) \) in high-dimensional space. Then, the optimal solution of the following objective function is the optimal hyperplane:

$$ \left\{ {\begin{array}{*{20}l} {\hbox{max} \sum\limits_{i = 1}^{m} {\alpha_{i} } - \frac{1}{2}\sum\limits_{i,j = 1}^{m} {\alpha_{i} } \alpha_{j} y_{i} y_{j} k\left( {x_{i} \cdot x_{j} } \right)} \hfill \\ {s.t.\,y_{i} \left[ {\left( {\omega \cdot z_{i} } \right) + b} \right] - 1 \ge 0,\sum\limits_{i = 1}^{m} {y_{i} \alpha_{i} = 0} ,\alpha_{i} \ge 0,i = 1,2, \ldots ,m} \hfill \\ \end{array} } \right. $$
(10)

3.4.2 Model Training

To recognize the driver’s fatigue state accurately, this paper uses SVM algorithm for fatigue recognition. Polynomial kernel and radial basis function (RBF) kernel are two popular kernels commonly used in SVM classification at present. Considering that the RBF kernel can realize nonlinear mapping and less hyper parameters can reduce the complexity of the SVM model compared to polynomial kernel, we choose RBF as the kernel for recognition model of fatigue driving. There are two parameters C and g are not known before model training, where penalty factor C expresses the tolerance of error and g in the RBF kernel has an important impact on model prediction. In order to improve the prediction performance of the model, the grid search algorithm is used to optimize the parameters C and g in this paper and the results of optimization are shown in Fig. 10. Finally, the recognition model is trained by using the two optimal parameters and randomly selected 600 samples collected from previous experiments.

Fig. 10
figure 10

Results of parameter optimization

Full size image

4 Experimental Results and Discussion

After trained, the recognition model is tested with the other 353 samples, including 168 samples of normal state and 185 samples of fatigue state, and the test results are shown in Table 2. Accuracy is calculated as:

Table 2 The test results of the recognition model
Full size table
$$ accuracy = {{\left( {TP + TN} \right)} \mathord{\left/ {\vphantom {{\left( {TP + TN} \right)} T}} \right. \kern-0pt} T} \times 100\% $$
(11)

where TP expresses the number of fatigue state correctly recognized as fatigue; TN expresses the number of normal state correctly recognized as normal, and T means the total test sample number of fatigue and normal state.

Table 2 shows the recognition model, which can classify the drivers’ states into two classes, correctly identifies 131 samples of normal state and 163 samples of fatigue state and the accuracy of classification is 83.29%. Figure 11 shows the performance of the used model in this paper and the other related models for fatigue driving recognition. All of these models are trained and tested using the same samples with previous SVM. From Fig. 11, we can conclude that the classification accuracy of SVM used in this paper is higher than other models.

Fig. 11
figure 11

Performance comparison for different methods

Full size image

Besides, it is noticed that 37 of 168 normal state samples are incorrectly classified and misjudgment is up to 22.02% in Table 2. The reason for the high misjudgment may be that the current characteristic parameters are not enough and the personal driving style may have an impact on it. In order to improve the recognition accuracy, we will further analyze the data characteristics under different driving states and combine with other recognition methods. For the impact of driving style, we will also consider building personalized recognition model of fatigue driving by using smart watch in the following study.

5 Conclusion

The steering operation of a driver under fatigue and normal states will be different. This paper designs and implements a recognition method of fatigue driving based on steering operation by using wearable smart watch. In this proposed method, thirteen principal characteristic parameters are selected from smart watch’s sensors by using PCA, and fatigue driving recognition model is established based on SVM. The test results prove that we can use smart watch to recognize the drivers’ fatigue or normal state effectively. We also compare the used model in this paper to the other related models. Consequently, the classification accuracy of SVM is better than the other models. This method uses no other external sensors, is low cost, user-friendly and simple. We will further consider more characteristics and build personalized recognition model by using smart watch to improve the accuracy.