Keywords

1 Introduction

The construction industry is facing two major challenges, which are a high probability of fatal safety accidents and decreasing productivity. For the safety challenges, the construction industry had 1,102 fatal injuries in 2019, which was recognized as one of the most dangerous industries in the U.S. [1]. The number of fatal injuries in the construction industry accounted for 22.4% of the whole occupational fatalities in 2019 [2]. Meanwhile, the productivity of the construction industry is at a low level and the improvement is lagging behind comparing with other industries [3]. In addition, some studies even found a continuous productivity decrease in the construction industry for the past 40 years [4].

To tackle the safety concerns and productivity decrease issues in the construction industry, monitoring construction workers’ activities is critical. For example, monitoring and detecting hazards around workers can reduce the probability of safety accidents [5]. Also, the monitoring and classifying of workers’ activities can be used to measure construction workers’ productivity [6]. With the development of wearable sensors, it is possible to classify workers’ activities based on the collected data such as accelerations [7]. Machine learning methods, such as decision trees, K-nearest neighbors KNN, and neural networks can classify workers’ activities based on the acceleration data collected from the wearable sensors [8, 9]. Furthermore, the findings of the recent studies showed that the data fusion of multimodal data can improve the accuracy of the workers’ activities classification. For example, the data fusion of acceleration data and electromyography improved the classification of scaffolding activities [9]. The workers’ physiological data such as heart rate, EDA and skin temperature change [10], however, were not considered for workers’ activities classification in existing studies.

Therefore, the objective of this study is to investigate if the data fusion between workers’ acceleration data and physiological data can improve workers’ activity classification. To achieve the objective, onsite data was collected at an Indiana Department of Transportation (INDOT) facility. Then, multiple classification models were trained for the two datasets (1) with acceleration data only and (2) with the fusion of physiological data. Then, the classification results on the two datasets were compared and discussed.

2 Literature Review

Classification of workers’ activities using wearable sensors has attracted much research attention in the past years. The benefits of applying wearable sensors include relatively low cost and no occlusion issues by the complex objects in construction sites compared with computer vision methods [11]. Previous studies showed the acceleration data collected by the wearable sensors can classify various construction workers’ activities with high accuracy. For example, acceleration data collected by the wearable sensors was used for masonry workers’ activity classification and the accuracy was 88.1% [12]. Also, acceleration, angular velocity and magnetic field were used for highway workers’ activities classification and the results showed a higher than 95% accuracy on three different datasets [13]. The acceleration data was used to classify construction workers’ ten activities and the results showed 93.69% accuracy [14].

However, two major limitations exist in previous studies about the workers’ activities classification. First, previous studies collected workers’ activity data in a controlled environment [14, 15]. The evidence of the classification performance of workers’ onsite data should be provided to further push this technique to field application. Second, previous studies mainly used the sole type of data for workers’ activity classification. For example, the kinematic data (i.e., acceleration, angular velocity, etc.) was mainly used [8, 12, 16]. Another study showed data fusion between workers’ electromyography (EMG) data, which was one type of workers’ physiological data, and acceleration data improved the classification results [9]. Other types of physiological data such as heart rate, EDA and skin temperature also could change with different activities [10]; however, the data fusion with these additional physiological data were not considered before. Therefore, more types of physiological data should be fused with kinematic data to investigate if the classification accuracy can be improved.

To fill the gaps mentioned above, this study collected workers’ acceleration, heart rate, EDA, and skin temperature data in a fieldwork. Two classifiers were trained on the dataset, including the one with acceleration data only and the one fusing all types of collected data. The classification results on two datasets were also compared and discussed below.

3 Methodology

3.1 Data Collection

The data collection was performed in an INDOT facility in June 2022. The workers were asked to perform crack sealing and pothole patching activities. The E4 wristband included several sensors, such as 3-axis accelerometer, photoplethysmography (PPG) sensor, EDA sensors and infrared thermopile, which were used to collect workers’ 3-axis accelerations, EDA, skin temperature and heart rate, respectively [10]. The 3-axis accelerations data was recorded in 32 Hz, the heart rate data was calculated based on the photoplethysmography(PPG) signal and stored in 1 Hz, and the EDA signal and the skin temperature were both recorded in 4 Hz.

Workers were required to wear the E4 wristband on their dominant hand during the data collection as shown in Fig. 1, which was the commonly used method in previous study [10]. Figure 2 shows the three activities used for classification, including crack sealing, transferring materials, and walking.

Fig. 1
A photo of a worker standing with an E 4 wristband.

The worker wearing E4 wristband for the data collection

Full size image
Fig. 2
3 photos present 2 workers during 3 different activities involved in road maintenance.

Three activity classes identified for this study, crack sealing (left), transferring materials (middle), and walking (right)

Full size image

3.2 Data Analysis

The data analysis consisted of data labeling, data preprocessing, model training and testing for activity classification. The data collected by the E4 wristband was labeled manually by comparing the timestamps of the recorded videos, including 5,792 data points for crack sealing, 2,304 data points for transferring materials, and 3,008 data points for walking.

The data preprocessing included data resampling, data balancing, data standardization, and dataset development for training. First, linear interpolation was used to resample the heart rate, EDA, and skin temperature data into 32 Hz. Linear interpolation was commonly used to resample the data for machine learning [17, 18]. Second, 2,304 data points of each activity were extracted to form a balanced dataset with 6,912 data points in total. Each data point has six features (3-axis accelerations, heart rate, EDA, and skin temperature) and one label. The balanced dataset prevents the classification results from being biased toward the majority class and increases the classification accuracy for minority classes [19]. Third, six features were standardized to keep the impacts of each feature to be consistent [20] and improve the accuracy of classification [21]. Each feature was standardized by subtracting the mean of the feature and divided by the standard deviation of the feature [22]. Fourth, two datasets were developed with a window size of 2 s and a 1 s sliding window from the labeled data. The first dataset only had three features, which were the three-axis accelerations. The second dataset included six features, which were the three axis accelerations, heart rate, EDA, and skin temperature. Also, both datasets were split into training and testing datasets by the ratio of 80% and 20%. The both training datasets had 171 data points and both test datasets had 43 data points.

The KNN models were trained for the two datasets in this study, which were also applied in previous studies and showed high accuracy in classifying workers’ activities [8, 23]. A new instance was labeled by the most common class of the K nearest instances in the feature space. Euclidean distance was used in this study as shown in Eq. (1). In addition, the number of neighbors to determine the class of a new instance was tuned in the range of 1 to 20 for the two datasets. The best number of neighbors for both datasets is one based on the tuning results. Therefore, the K = 1 was selected and the performance of the models on two test datasets will be discussed in the next section.

$$Distance \left({X}_{i}, {X}_{new}\right)=\sqrt[2]{\sum_{r=1}^{d}{({X}_{i}-{X}_{new})}^{2}}$$
(1)

where d represents the number of features.

4 Results and Discussion

The performance of the KNN models on the testing dataset were shown in Fig. 3 and Fig. 4 respectively. Figure 3 shows the confusion matrix of the KNN model on the dataset without data fusion (i.e., with the 3-axis accelerations data only). The accuracies for classifications of crack sealing, transferring materials and walking were 1.00, 1.00 and 0.71, with the acceleration data only. Figure 4 shows the confusion matrix of the KNN model on the dataset with the data fusion of workers’ heart rate, EDA, and skin temperature data. The results showed the classification accuracy for crack sealing was still 1.00 and the classification accuracy for transferring material decreased from 1.00 to 0.93 and the classification accuracy for walking increased from 0.71 to 0.93. The overall classification accuracies for all three activities increased from 0.9069 to 0.9535 with the data fusion. Previous studies showed that different activities had different impacted on workers’ heart rate, EDA and skin temperature data [10]. Therefore, adding these data can improve the accuracy for workers’ activity classification. The conclusion is consistent with previous studies that data fusion between acceleration and physiological data can improve the classification accuracy [9].

Fig. 3
A confusion matrix of 3 cross 3. It presents true versus predicted labels for crack sealing, transferring material, and walking. The diagonal cells with values are shaded.

Confusion matrix of the KNN model on the dataset without data fusion

Full size image
Fig. 4
A confusion matrix of 3 cross 3. It presents the true versus predicted labels for crack sealing, transferring material, and walking. The diagonal cells with values are shaded.

Confusion matrix of the KNN model on the dataset with data fusion

Full size image

5 Conclusions

This study compared the performance of a machine learning method (i.e., KNN) on the classification of workers’ activities with acceleration data only and data fusion between acceleration and physiological data. Onsite data was collected at an INDOT facility, and three activities were identified for the classification. The results showed that with the data fusion, the overall classification performance increased. However, the case varied for different classes of activity. For example, the accuracy for classifying material transfer decreased with the data fusion whereas the classification accuracy for walking increased.

The authors acknowledge the following limitations of the study. First, the size of the dataset was small. More data should be collected in the future to train more complex classification models, such as deep learning models. Second, the study only considers three activities. More types of activities should be considered in the future.