1 Introduction

Improving the efficiency and sustainability of transportation infrastructure systems is a major undertaking in the future development of smart cities [1,2,3]. In this context, advanced sensing and data analytics offer unique capabilities for improving various components of transportation infrastructure systems. Currently, transportation infrastructure systems in developed countries, as key components of smart cities, are mostly outdated and are vulnerable to various risks [4, 5]. As these transportation infrastructure systems age, there is an increasing demand for cost effective and efficient tools to monitor and manage the systems due to the limited budget of municipal jurisdictions.

Currently, there are two common methods used to monitor existing transportation infrastructure systems. The first method involves pre-installing sensors on the infrastructure to continuously collect and analyze data. The second method involves occasionally dispatching engineers or technicians to the site to record the measurements and bring them back for analysis. In spite of the rapid development of these monitoring techniques, there are still challenges in terms of scalability, i.e., applying these techniques to all of the existing transportation infrastructure systems. For the first method described above, a large number of sensors must be pre-installed on the infrastructure components. The cost for sensors and the cost of the labor to install them are high, and providing stable power supply systems for this type of health monitoring system remains an unresolved problem. For the second method, the inspection intervals are usually long due to the high costs of labor and inspection equipment.

To overcome the abovementioned issues, researchers have proposed using vehicles that are equipped with various kinds of sensors to monitor the transportation infrastructure systems with the involvement of citizens to increase efficiency and reduce the cost to municipal departments [6, 7]. Owing to the high mobility of vehicles, they can efficiently monitor the health condition of a population of the existing infrastructure. Figure 1 includes a list of widely used sensors in vehicles. The monitoring strategy that uses crowdsourced sensors in vehicles is presented in Fig. 2. The sensors in vehicles can collect data while the vehicle is traveling along a road or over a bridge. Then, the data are transmitted to remote servers for further processing and decision-making. In employing this strategy, there is no need to install sensors locally, and monitoring the condition of the infrastructure can be accomplished efficiently with the involvement of a large number of vehicles.

Fig. 1
figure 1

Sensors that can be installed in vehicles (modified from [8])

Full size image
Fig. 2
figure 2

Overview of the transportation infrastructure monitoring strategy employing crowdsensing-based techniques

Full size image

It should be acknowledged that previous studies propose to use instrumented vehicles for transportation infrastructure [9, 10]. However, these instrumented vehicles are usually specially designed and are equipped with expensive sensors, which means they cannot be used in a scalable manner due to the high cost [11]. This paper focuses on the utilization of commercial grade or naturally installed sensors in normal vehicles to gather data from a large number of vehicles for analysis. This type of monitoring technology offers several advantages: first, it can significantly reduce the cost for monitoring due to the voluntarily involvement of citizens; second, it has the potential to monitor a population of transportation infrastructure in real time; third, owing to big data, the technology is more robust to operational effects; fourth, the technology can be fully automated after the system is established.

This paper presents three applications under a common framework for monitoring transportation infrastructure using crowdsourcing. The proposed framework aims to provide an efficient and cost-effective tool for high-level monitoring of a large number of transportation infrastructure components. The applications include three easy-to-access sensors in smart devices, i.e., accelerometers and gyroscopes in smartphones and in-vehicle cameras.

2 Application 1: bridge health monitoring using smartphones in vehicles

Moving sensors for bridge health monitoring have been investigated extensively in last decades [12,13,14,15,16,17,18]. Yang et al. [14] were the first to conduct a study to extract the frequencies of a bridge from a moving vehicle. In their paper, the authors found an analytical solution showing that the data collected from a moving vehicle includes the dynamic properties of the bridge. Afterwards, numerical analysis and experiments were conducted by various researchers to show the feasibility of extracting dynamic properties such as frequency, mode shape, or damping [13, 18,19,20,21]. The major challenge in this research field is to separate the bridge properties from signals that mix bridge and vehicle vibrations. The mixed signals could also be affected by a number of factors such as road roughness and environmental effects.

Most of the previous studies focused on extracting the dynamic characteristics of the bridge using a single vehicle, which is sensitive to environmental and operational effects. In this paper, a new data-driven method for bridge damage detection based on a large number of vehicles is introduced. This method has the potential to be implemented on cars belonging to commuters, on police cars, on emergency vehicles, or on maintenance vehicles and buses to reduce costs associated with infrastructure monitoring.

The overall method, shown in Fig. 3, includes two phases; the training phase is the baseline case and the testing phase corresponds to the unknown state of the bridge. For each phase, acceleration data can be collected from a number of vehicles (m1 vehicles for training phase and m2 vehicles for testing phase). Features can be extracted from training and testing phases. The distribution of the features will then be compared to determine the existence of damage. The logic behind this method is that a large number of vehicles could mitigate the operational effects such as weight, suspension system, the speed of the vehicles, and the influence of other vehicles. Any large shift in terms of features should be observed only if the status of the bridge changes. In the research described in this paper, Mel-frequency cepstral analysis (MFCA) is conducted for feature extraction, principal component analysis (PCA) is applied to decorrelate the features, and then the root mean square deviation (RMSD) of the mean values of feature distributions is calculated for damage detection. Details of the method can be found in a previous study by the authors of the present work [6, 22].

Fig. 3
figure 3

A crowdsensing-based bridge damage detection method

Full size image

To verify the method described above, a lab experiment using smartphones in a robot car is conducted as presented in Fig. 4. In the experiments, a robot car passes through the bridge deck multiple times with different characteristics (different weight, spring, and speed) to simulate different vehicles crossing the bridge. Currently, only a vehicle is considered in each run and in future work, the influence of multiple vehicles will be considered by adding ambient vibration or using more robot cars simultaneously in the experiments. The robot car, as shown in Fig. 5, is designed with the ability to change weight, spring constant, and speed to mimic the behavior of vehicles of different configurations. The weight of the top plate could vary between 0.898, 0.988, 1.084, 1.170, and 1.270 kg; the spring constant could vary between 155, 288, 425, 615, and 726 N/m; and the speed of the robot car could be either 0.25, 0.33, or 0.40 m/s. Furthermore, each test is repeated three times to consider other effects such as road roughness. Two G-Link®-200 wireless accelerometers from Microstrain Inc. and a Galaxy S5 smartphone from Samsung Group are mounted on the top plate of the robot car. The sampling frequencies of the accelerometers and the smartphone are 128 Hz and 100 Hz, respectively. An Android app was developed specifically for this purpose and installed on the smartphone to collect vibration data. Details of the Android app can be found in a previous study by the authors of the present work [6].

Fig. 4
figure 4

Lab experiment setup

Full size image
Fig. 5
figure 5

Robot car configuration

Full size image

In total, three damage cases (DCs) are applied to the bridge: (1) 15% section area reduction at the mid-span (see Fig. 6); (2) 15% section area reduction at the ¼ span (see Fig. 6); and (3) boundary condition changes at both ends, as shown in Fig. 7a, b. For each damage case, a total of 5 × 5 × 3 × 3 = 225 tests are completed considering the combination of all the possible robot car configurations and the repetition of three times. In the analysis, 30 trials are conducted. In each trial, 50% of the 225 tests for each case are randomly selected to simulate the randomness of the vehicle configurations.

Fig. 6
figure 6

Local damage created by section area reduction

Full size image
Fig. 7
figure 7

Global damage created by boundary condition changes

Full size image

As shown in Eq. (1), damage feature is defined as the RMSD of the averages of two feature distributions.

$$DF(F_{{{\text{baseline}}}} ,F_{{{\text{unknown}}}} ) = \sqrt {\frac{{\sum\nolimits_{i = 1}^{n} {(\mu_{{{\text{unknown}},i}} - \mu_{{{\text{baseline}},i}} )^{2} } }}{n}} ,$$
(1)

where Fbaseline and Funkonwn are the n features (30 in this paper) extracted for baseline and unknown cases using MFCA and PCA, and μbaseline,i and μunknown,i are the mean values of the ith features for baseline and unknown cases. The damage feature is designed to measure the shift of the feature distribution when the status of the bridge changes.

The damage features calculated using the data collected from the accelerometers and the smartphone are presented in Figs. 8 and 9. Each trial is related to a random sampling procedure of the vehicle configurations. As shown in the figures, the damage features are stable while different vehicles are used but the state of the bridge is unchanged. When the damage of the bridge is introduced (i.e., damage cases DC1, DC2, and DC3), the damage features become higher against the baseline case. This shows that the existence of damage is successfully identified. Comparing Figs. 8 and 9, we can see that the patterns of damage features from the accelerometers and the smartphone are very similar, which proves that the smartphone is suitable for this application even though it has lower resolution and sampling frequency.

Fig. 8
figure 8

Damage features for baseline and three damage cases from the sensors

Full size image
Fig. 9
figure 9

Damage features for baseline and three damage cases from the smartphone

Full size image

3 Application 2: road inclination measurement using smartphones in vehicles

The gyroscope sensor can provide information regarding the orientation of the smartphone. In this section, a lab experiment investigating the feasibility of this sensor for road inclination measurement is introduced. As can be seen in Fig. 10a, four wooden decks are placed on the ground, and some small steel blocks are placed underneath these decks to create inclinations. The dimensions of the setup can be found in Fig. 10b. There are, in total, four zones in the setup, where zone 1 is flat, zone 2 is a decline with 1.37° inclination, zone 3 is an incline with 0.96° inclination, and zone 4 is a decline with 1.37° inclination.

Fig. 10
figure 10

The deformed road setup and the dimensions (in mm)

Full size image

As shown in Fig. 11a, b, another version of the robot car similar to that used in the last section is used for this test. In this robot car, four rods instead of two rods are used for the stability of the top plate. The total weight of the top plate is 1.2 kg. One wireless accelerometer and one smartphone are placed on the top plate. It should be noted that data from the wireless accelerometer are not collected in this experiment. The Matlab app on the Android smartphone is used for this experiment.

Fig. 11
figure 11

Road deformation measurement experiment

Full size image

The experiment includes three trials. As shown in Fig. 12, the measured orientations are compared with the theoretical values (red dashed lines) calculated from the dimensions. As shown in the figure, the gyroscope sensor in the smartphone provides useful information about the deformation of the road. Some measurement errors exist, which could occur because the gyroscope sensor in the smartphone did not go through the calibration process before the tests. Also, the measurement is not accurate when the inclination of the decks changes because the robot car could cross over two decks at such locations. In a real-life scenario, the orientation information of the smartphones in multiple moving vehicles can be synchronized with the GPS data to show the deformation of the roads at city level. Challenges such as human factors should be resolved in real-life applications.

Fig. 12
figure 12

Comparison of test results and theoretical values

Full size image

4 Application 3: road crack detection using sport cameras attached to vehicles

In addition to smartphones, backup cameras are another type of widely installed sensors in vehicles. In this section, the feasibility of applying such cameras for transportation infrastructure monitoring is investigated. Since current vehicles typically do not allow easy access to the backup camera system due to safety reasons, a commercial grade sport camera, GoPro, is mounted at the rear of the vehicle to mimic the behavior of a backup camera (see Fig. 13). A sport camera can continuously capture videos at a high shutter speed.

Fig. 13
figure 13

Mounting of GoPro sport camera

Full size image

In recent years, due to its ability to process massive data accurately and effectively, deep learning methods have attracted great attention from researchers in civil infrastructure monitoring [23]. They have been used to process vibration data [24–27] and image data [28–31] in different structures including bridges, buildings, railways, and roads. For example, Rafiei and Adeli [24] designed a structural health index (SHI) based on synchrosqueezed wavelet transform, fast Fourier transform, and unsupervised deep Boltzmann machine. They used this index to assess the local and global condition of the structure. Dong et al. [32] proposed deep learning-based full field optical flow methods for structural displacement monitoring.

Particularly, deep learning-based methods have shown superior performance in the context of crack detection problems [28, 29, 33–35]. In this section, a novel deep learning algorithm is developed for road crack detection that considers the connectivity of pixels. The architecture of the deep neural network is presented in Fig. 14. A batch size of 16 is used. Taking 256 × 256 color image patches extracted from GoPro as input, an encoder–decoder procedure is applied with multiple level feature fusion. The output of the deep neural network is a connectivity map, proposed in Mei et al. [36], representing the neighboring relationship of crack pixels. A depth-first search algorithm, as proposed in a previous study by the present authors [37], is applied to the output of the deep neural network to generate the binary mask for cracks. More details of the deep neural network can be found in the study authored by Mei et al. [36].

Fig. 14
figure 14

Deep neural network architecture

Full size image

The deep neural network is first trained on a general image datasets ImageNet [38] as pre-training. Then, the proposed method is trained and tested on a dataset called EdmCrack600 released by our research team [39]. The EdmCrack600 dataset includes 600 images extracted from videos taken during approximately 10 h of driving in Edmonton, Canada. All the images were annotated manually at pixel level by the authors. The dataset was collected during road tests and consists of the various objects one can encounter during driving, such as snow, shadows from trees, other vehicles, etc.

In this study, the EdmCrack600 dataset is split into 420/60/120 images for training, validation, and testing purposes, respectively. After training for 20 epochs on EdmCrack600 dataset, the precision, recall, and F1 score, as defined in the work by Shi et al. [40], are calculated to measure the performance on the test set. The obtained precision, recall, and F1 score are 0.8469, 0.6994, and 0.7472, respectively. It took 251.09 s to process all 120 test images on a PC with Intel 8700 k CPU, 32 GB memory, and Nvidia Titan V GPU. Considering the memory limit of 11 GB on GPU, the batch size of 16 is used, and overall implementation time includes the inference time from the deep neural network and the time for outputting the results The performance of this method on the EdmCrack600 dataset is comparable with the ones reported in a previous study by the present authors [41].

Two sample images from the EdmCrack600 dataset and their corresponding identification results are shown in Fig. 15. The images include many extraneous objects such as lane markers, shadows, and other vehicles which may affect the accuracy of crack detection. Comparing the ground truth and the identification results from our method, it can be seen that the cracks are correctly identified, and the influence of other objects is successfully excluded.

Fig. 15
figure 15

Sample identification results from the deep learning-based method

Full size image

With the high success rate of road crack detection using our method, further analysis can be conducted by synthesizing the camera data with GPS signals collected by the GoPro. To quantitatively reflect the road condition, a simple crack index (CI) is designed as shown in Eq. (2).

$${\text{CI}} = \frac{{N_{{{\text{crack}}}} }}{{N_{{{\text{total}}}} }} \times 1000,$$
(2)

where \(N_{{{\text{crack}}}}\) is the number of pixels that are identified as crack in an image and \(N_{{{\text{total}}}}\) is the total number of pixels in the image, i.e., resolution. A higher CI represents a worse road condition in terms of cracks.

Figure 16 is generated by calculating the CIs for a series of images taken at 0.5 s intervals over a time period of 410 s. The vehicle is driven such that it maintains the same speed as the surrounding traffic, and no extra effort is made to control the vehicle’s speed. It should be noted that the images used herein were not annotated and not included in EdmCrack600 dataset. In the figure, each dot represents an image. The images with a CI smaller than 2 are colored green, the images with a crack index between 2 and 5 are yellow, and the others with a crack index higher than 5 are indicated in red. As shown in Fig. 16, the road condition is worse in some locations than others.

Fig. 16
figure 16

Crack index map calculated by synchronizing image and GPS data

Full size image

Images at two locations, as labelled in Fig. 16, and their corresponding results are presented in Fig. 17. The CIs for these two locations are 6.8 and 2.6, respectively. As illustrated in Fig. 17, the cracks in the images are correctly identified, and the location 2 indeed has better road condition than location 1.

Fig. 17
figure 17

Images and results from the two locations labelled in Fig. 16

Full size image

5 Discussion

Although some experiments in the present study show the potential for the framework of crowdsensing-based transportation infrastructure monitoring, there are still many issues that need to be resolved before the framework can be applied to real-life monitoring. An important one is that the data collected from different vehicles using smartphones are influenced by vehicle and device properties. To investigate these effects, preliminary real-life data are collected from four bridges in Edmonton, i.e., Walterdale Bridge, MacDonald Bridge, Low Level Bridge and High Level Bridge. Two vehicles, one Honda Civic and one Honda Pilot, were used representing two vehicle types, sedan and SUV, respectively. In addition, in each vehicle, two smartphones with sampling frequencies of 200 and 400 Hz are used, and two wireless accelerometers are used in the sedan as a benchmark to compare the performance of the smartphones.

To present the results of the aforementioned investigation of the real-life issues due to the effects of vehicle and device properties, the frequency content of the data recorded on all devices while passing over the High Level bridge (Fig. 18) is illustrated in Fig. 19. The top and bottom plots show the sensor and smartphone data, respectively. The two sensors are installed on sedan, while the first two smartphones, denoted by SP1 and SP2, are installed in the SUV, and the other two smartphones, denoted by SP3 and SP4, are installed in the sedan. In addition, the sampling frequency of SP1 and SP3 is 200 Hz, while that of SP2 and SP4 is 400 Hz. As seen in Fig. 19, the frequency content of the sensors and the smartphones located in the sedan, i.e., SP3 and SP4, follow similar patterns in the lower frequencies which prove the efficient performance of the smartphone in capturing major contents in lower frequencies, which are the focus for indirect bridge monitoring. Furthermore, comparing the smartphones with the 200 Hz sampling rate, i.e., SP1 and SP3, to the smartphones with the 400 Hz sampling rate, i.e. SP2 and SP4, shows similar agreement in lower frequencies, which eliminates the need to use higher sampling rates. On the other hand, comparing the data from the SUV, i.e., SP1 and SP2, with the data from the sedan, i.e. SP3 and SP4, shows major differences, which indicates that the vehicle’s features are the most significant factor affecting the frequency content of the acceleration signal recorded for the vehicle, and without considering this effect, it is difficult to employ indirect methods to capture frequency content of the bridge.

Fig. 18
figure 18

Image of the High Level Bridge used for real-life study [42]

Full size image
Fig. 19
figure 19

Frequency content of recorded acceleration signals while passing over the High Level Bridge

Full size image

To specifically study the effect of the vehicle, heat map plots of short time Fourier transform of each car while moving off-bridge and also while passing over the High Level bridge are illustrated in Fig. 20. The off-bridge plots represent the condition in which the car is stopped, starts moving, and then stops, while the on-bridge plots are showing passing over the bridge with a constant speed. Comparing off-bridge and on-bridge plots corroborates the fact that the data collected from a vehicle passing over the bridge are significantly affected by the car type and features. These real-life data analyses prove that there is no general indirect monitoring method expected for extracting dynamic features of the bridge without considering the effect of the vehicle. One of the solutions to this issue is to create a filter to suppress car-related frequency content and hence amplify bridge-related content. To have a unique filter for each vehicle, the authors proposed an inverse filtering approach where off-bridge data before entering a bridge can be used to design such unique filters automatically for each vehicle [43]. Another solution, as presented in Sect. 2, could be to increase the number of the test vehicles in order to average out the effect of vehicles.

Fig. 20
figure 20

Heat map of short time Fourier transform of recorded acceleration signals while moving off-bridge and while passing over the high level bridge

Full size image

6 Challenges

Except the issues described in previous sections, there are many other challenges that have to be resolved while building the crowdsensing-based infrastructure monitoring system. One important challenge is the influence of the vehicles themselves and of human beings on the collected data. The moving and vibration of vehicles could corrupt the data like vibration data or images collected by the sensors. Also, devices in vehicles could be moved and disturbed by the drivers or passengers during the data collection. Studies regarding these effects should be taken. Advanced signal or image processing techniques are need to be applied to eliminate the effects of the vehicles on the data.

Also, a trade-off must be made between the limited bandwidth available to transmit the raw data and the large amount of computational power required to process the data. The onboard implementation of computationally heavy algorithms, such as deep learning algorithms, to process the data is difficult, and the computational capacity of vehicles and mobile devices is not as strong as that of remote servers. However, the transmission of the raw data to remote servers for further analysis is limited by the bandwidth available.

In addition, since the moving vehicles are continuously collecting data while driving, it is challenging to attract a large number of users and keep them engaged in contributing to the monitoring purpose. Also, privacy issues should be addressed while collecting the data from the users. Relevant laws and regulations are need to be made regarding the disclosure of this type of data.

7 Conclusions

This paper proposes a framework for transportation infrastructure monitoring using moving vehicles. The feasibility of this framework and some relevant applications under this framework is investigated. It is demonstrated in this paper that the crowdsensing-based monitoring framework may offer a promising high-level prescreening tool which has the potential to monitor various components of the transportation infrastructure system with reduced costs and increased efficiency.

Under this framework, three applications of smartphones and in-vehicle cameras for bridge and road health monitoring are presented. It is shown that it is feasible to use the existing sensors in personal mobile devices in moving vehicles for a high-level monitoring of the transportation infrastructure. Based on the information extracted from this high-level monitoring, more specific monitoring and inspection can be conducted on the structures when needed. Specifically, the following conclusions can be drawn from this paper:

  1. 1.

    A methodology combining feature extraction and statistical comparison of these features are proposed to take advantages of the vibration data collected from different vehicles at different times, which enables indirect bridge health monitoring using a large amount of data. Laboratory experiments are conducted to verify this method.

  2. 2.

    The gyroscope that can report the orientation of the smartphone is studied in this paper for monitoring of the highway infrastructure. The values reported by the gyroscope are compared with the actual inclinations in the road model tested in the laboratory. The results show that the gyroscope in the smartphone has the potential to identify the large deformations in roads, which can happen due to seasonal effects and climate change.

  3. 3.

    A deep learning algorithm combining connectivity maps and depth-first search is proposed to identify cracks from moving vehicles. Testing on the EdmCrack600 dataset collected by our group, the proposed method can achieve the state-of-the-art performance. This algorithm enables efficient and cost-effective inspection of the road pavements. The image data are fused with GPS data in this study to provide the detailed information such as crack index of the pavement health condition in the neighborhood.

In the future, in the context of smart city, other sensors in smart devices and fusion of sensors will be investigated to provide more valuable information about current civil infrastructure systems.