Abstract
Road damage is an inconvenience and a safety hazard, severely affecting vehicle condition, driving comfort, and traffic safety. The traditional manual visual road inspection process is pricey, dangerous, exhausting, and cumbersome. Also, manual road inspection results are qualitative and subjective, as they depend entirely on the inspector’s personal experience. Therefore, there is an ever-increasing need for automated road inspection systems. This chapter first compares the five most common road damage types. Then, 2-D/3-D road imaging systems are discussed. Finally, state-of-the-art machine vision and intelligence-based road damage detection algorithms are introduced.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
Source code is publicly available at https://github.com/ruirangerfan/unsupervised_disparity_map_segmentation.
- 2.
A demo video can be found at https://vimeo.com/337886918.
- 3.
Source code is publicly available at: https://github.com/hlwang1124/AAFramework.
- 4.
- 5.
3-D road point clouds are publicly available at https://github.com/ruirangerfan/stereo_pothole_datasets.
References
R. Fan et al., Road surface 3d reconstruction based on dense subpixel disparity map estimation. IEEE Trans. Image Process. 27(6), 3025–3035 (2018)
C. Koch et al., A review on computer vision based defect detection and condition assessment of concrete and asphalt civil infrastructure. Adv. Eng. Inform. 29(2), 196–210 (2015)
R. Fan et al., Pothole detection based on disparity transformation and road surface modeling. IEEE Trans. Image Process. 29, 897–908 (2019)
T. Kim, S.-K. Ryu, Review and analysis of pothole detection methods. J. Emerg. Trends Comput. Inf. Sci. 5(8), 603–608 (2014)
R. Fan et al., We learn better road pothole detection: from attention aggregation to adversarial domain adaptation, in European Conference on Computer Vision Workshops (Springer, Berlin, 2020), pp. 285–300
S. Mathavan et al., A review of three-dimensional imaging technologies for pavement distress detection and measurements. IEEE Trans. Intell. Transp. Syst. 16(5), 2353–2362 (2015)
R. Fan et al., Rethinking road surface 3-d reconstruction and pothole detection: from perspective transformation to disparity map segmentation. IEEE Trans. Cybern. 52(7), 5799–5808 (2022)
R. Fan, Long-awaited next-generation road damage detection and localization system is finally here, in 29th European Signal Processing Conference (EUSIPCO) (IEEE, 2021), pp. 641–645
Y.-C. Tsai, A. Chatterjee, Pothole detection and classification using 3d technology and watershed method. J. Comput. Civ. Eng. 32(2), 04017078 (2018)
M.R. Jahanshahi et al., Unsupervised approach for autonomous pavement-defect detection and quantification using an inexpensive depth sensor. J. Comput. Civ. Eng. 27(6), 743–754 (2013)
R. Fan et al., Real-time dense stereo embedded in a UAV for road inspection, in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (IEEE Computer Society, 2019), pp. 535–543
L. Zhang, Road crack detection using deep convolutional neural network, in IEEE International Conference on Image Processing (ICIP) (IEEE, 2016), pp. 3708–3712
R. Fan et al., Graph attention layer evolves semantic segmentation for road pothole detection: a benchmark and algorithms. IEEE Trans. Image Process. 30, 8144–8154 (2021)
A. Dhiman, R. Klette, Pothole detection using computer vision and learning. IEEE Trans. Intell. Transp. Syst. 21(8), 3536–3550 (2019)
D.S. Mahler et al., Pavement distress analysis using image processing techniques. Comput.-Aided Civ. Infrastruct. Eng. 6(1), 1–14 (1991)
H.N. Koutsopoulos, A. Downey, Primitive-based classification of pavement cracking images. J. Transp. Eng. 119(3), 402–418 (1993)
J. Laurent et al., Road surface inspection using laser scanners adapted for the high precision 3d measurements of large flat surfaces, in Proceedings. International Conference on Recent Advances in 3-D Digital Imaging and Modeling (Cat. No. 97TB100134) (IEEE, 1997), pp. 303–310
D. Joubert, A. Tyatyantsi, J. Mphahlehle, V. Manchidi, Pothole tagging system, in Robotics and Mechatronics Conference of South Africa, CSIR International Conference Centre, Pretoria (2011), pp. 23–25
I. Moazzam, K. Kamal, S. Mathavan, S. Usman, M. Rahman, Metrology and visualization of potholes using the microsoft kinect sensor, in 16th International IEEE Conference on Intelligent Transportation Systems (ITSC 2013) (IEEE, 2013), pp. 1284–1291
M. Barbato, G. Orlandi, M. Panella, Real-time identification and tracking using kinect
A.M. Andrew, Multiple view geometry in computer vision, in Kybernetes (2001)
G. Jog, C. Koch, M. Golparvar-Fard, I. Brilakis, Pothole properties measurement through visual 2d recognition and 3d reconstruction. Comput. Civ. Eng. 2012, 553–560 (2012)
S. Ullman, The interpretation of structure from motion. Proc. R. Soc. Lond. Ser. B. Biol. Sci. 203(1153), 405–426 (1979)
H. Wang et al., CoT-AMFlow: adaptive modulation network with co-teaching strategy for unsupervised optical flow estimation, in Conference on Robot Learning (CoRL) (2020)
B. Triggs, P.F. McLauchlan, R.I. Hartley, A.W. Fitzgibbon, Bundle adjustment-a modern synthesis, in International workshop on vision algorithms (Springer, Berlin, 1999), pp. 298–372
M.U.M. Bhutta et al., Loop-box: multiagent direct slam triggered by single loop closure for large-scale mapping. IEEE Trans. Cybern. 52(6), 5088–5097 (2022)
N. Ma et al., Computer vision for road imaging and pothole detection: a state-of-the-art review of systems and algorithms. Transp. Saf. Environ. (2022) (in press)
H. Hirschmuller, Stereo processing by semiglobal matching and mutual information. IEEE Trans. Pattern Anal. Mach. Intell. 30(2), 328–341 (2007)
J. Sun, N.-N. Zheng, H.-Y. Shum, Stereo matching using belief propagation. IEEE Trans. Pattern Anal. Mach. Intell. 25(7), 787–800 (2003)
H. Wang et al., Pvstereo: pyramid voting module for end-to-end self-supervised stereo matching. IEEE Robot. Autom. Lett. 6(3), 4353–4360 (2021)
R. Danzl, F. Helmli, S. Scherer, Focus variation–a new technology for high resolution optical 3d surface metrology, in The 10th International Conference of the Slovenian Society for Non-Destructive Testing (Citeseer, 2009), pp. 484–491
Y. Sun, S. Duthaler, B.J. Nelson, Autofocusing algorithm selection in computer microscopy, in 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems (IEEE, 2005), pp. 70–76
S.K. Nayar, Y. Nakagawa, Shape from focus. IEEE Trans. Pattern Anal. Mach. Intell. 16(8), 824–831 (1994)
C. Wöhler, Triangulation-based approaches to three-dimensional scene reconstruction, in 3D Computer Vision (Springer, Berlin, 2013), pp. 3–87
A. Kuhl, C. Wöhler, L. Krüger, P. d’Angelo, H.-M. Groß, Monocular 3d scene reconstruction at absolute scales by combination of geometric and real-aperture methods, in Joint Pattern Recognition Symposium (Springer, Berlin, 2006), pp. 607–616
M. Subbarao, G. Surya, Depth from defocus: a spatial domain approach. Int. J. Comput. Vision 13(3), 271–294 (1994)
S. Nayar, K. Ikeuchi, T. Kanade, Surface reflection: physical and geometrical perspectives, in Proceedings: Image Understanding Workshop (1990), pp. 185–212
R.J. Woodham, Photometric method for determining surface orientation from multiple images. Opt. Eng. 19(1), 191139 (1980)
S. Barsky, M. Petrou, The 4-source photometric stereo technique for three-dimensional surfaces in the presence of highlights and shadows. IEEE Trans. Pattern Anal. Mach. Intell. 25(10), 1239–1252 (2003)
J.G. Fujimoto, C. Pitris, S.A. Boppart, M.E. Brezinski, Optical coherence tomography: an emerging technology for biomedical imaging and optical biopsy. Neoplasia 2(1–2), 9–25 (2000)
W.J. Walecki, P. Van, Determining thickness of slabs of materials by inventors, 3 Oct 2006, uS Patent 7,116,429
D. Scharstein, R. Szeliski, High-accuracy stereo depth maps using structured light, in Proceedings 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003, vol. 1 (IEEE, 2003), p. I
V. Muzet, C. Heinkele, Y. Guillard, J.M. Simonin, Surface deflection measurement using structured light, in Testing in Civil Engineering, Nantes, France (2009)
T. Oggier, M. Lehmann, R. Kaufmann, M. Schweizer, M. Richter, P. Metzler, G. Lang, F. Lustenberger, N. Blanc, An all-solid-state optical range camera for 3d real-time imaging with sub-centimeter depth resolution (swissranger), in Optical Design and Engineering, vol. 5249 (International Society for Optics and Photonics, 2004), pp. 534–545
D. Anderson, H. Herman, A. Kelly, Experimental characterization of commercial flash ladar devices, in International Conference of Sensing and Technology, vol. 2 (Citeseer, 2005), pp. 17–23
C.F. Özgenel, Concrete crack images for classification, in Mendeley Data, V1 (2018)
M. Maguire, S. Dorafshan, R.J. Thomas, Sdnet2018: a concrete crack image dataset for machine learning applications (2018)
Z. Zhang, Advanced stereo vision disparity calculation and obstacle analysis for intelligent vehicles, Ph.D. dissertation, University of Bristol (2013)
A. Alzoubi, PotDataset 4 (2018). https://cord.cranfield.ac.uk/articles/dataset/PotDataset/5999699
R. Fan, M. Liu, Road damage detection based on unsupervised disparity map segmentation. IEEE Trans. Intell. Transp. Syst. 21(11), 4906–4911 (2019)
U. Ozgunalp, Vision based lane detection for intelligent vehicles, Ph.D. dissertation, University of Bristol (2016)
E. Buza, S. Omanovic, A. Huseinovic, Pothole detection with image processing and spectral clustering, in Proceedings of the 2nd International Conference on Information Technology and Computer Networks, vol. 810 (2013), p. 4853
S.-K. Ryu, T. Kim, Y.-R. Kim, Image-based pothole detection system for its service and road management system. Math. Probl. Eng. 2015 (2015)
C. Koch, I. Brilakis, Pothole detection in asphalt pavement images. Adv. Eng. Inform. 25(3), 507–515 (2011)
G.W. Zack, W.E. Rogers, S.A. Latt, Automatic measurement of sister chromatid exchange frequency. J. Histochem. Cytochem. 25(7), 741–753 (1977)
G. Beylkin, R. Coifman, V. Rokhlin, Fast wavelet transforms and numerical algorithms, in Fundamental Papers in Wavelet Theory (Princeton University Press, 2009), pp. 741–783
L. Najman, M. Schmitt, Watershed of a continuous function. Signal Process. 38(1), 99–112 (1994)
R. Labayrade, D. Aubert, J.-P. Tarel, Real time obstacle detection in stereovision on non flat road geometry through “v-disparity” representation, in Intelligent Vehicle Symposium, vol. 2 (IEEE, 2002), pp. 646–651
P. Pedregal, Introduction to Optimization (Springer Science & Business Media, 2006), vol. 46
R. Fan, N. Dahnoun, Real-time stereo vision-based lane detection system. Meas. Sci. Technol. 29(7), 074005 (2018)
U. Ozgunalp et al., Multiple lane detection algorithm based on novel dense vanishing point estimation. IEEE Trans. Intell. Transp. Syst. 18(3), 621–632 (2016)
R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, S. Süsstrunk, Slic superpixels compared to state-of-the-art superpixel methods. IEEE Trans. Pattern Anal. Mach. Intell. 34(11), 2274–2282 (2012)
Y. Hu, T. Furukawa, A self-supervised learning technique for road defects detection based on monocular three-dimensional reconstruction, in International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, vol. 59216 (American Society of Mechanical Engineers, 2019), p. V003T01A021
A. Krizhevsky, One weird trick for parallelizing convolutional neural networks (2014). arXiv:1404.5997
K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition, in International Conference on Learning Representations (ICLR) (2015)
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A. Rabinovich, Going deeper with convolutions, in Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR) (2015), pp. 1–9
C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, Z. Wojna, Rethinking the inception architecture for computer vision, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016), pp. 2818–2826
K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016), pp. 770–778
M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, L.-C. Chen, Mobilenetv2: inverted residuals and linear bottlenecks, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018), pp. 4510–4520
N. Ma, X. Zhang, H.-T. Zheng, J. Sun, Shufflenet v2: practical guidelines for efficient CNN architecture design, in Proceedings of the European Conference on Computer Vision (ECCV) (2018), pp. 116–131
M. Tan, B. Chen, R. Pang, V. Vasudevan, M. Sandler, A. Howard, Q.V. Le, Mnasnet: platform-aware neural architecture search for mobile, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2019), pp. 2820–2828
Y. LeCun, Y. Bengio, G. Hinton, Deep learning. Nature 521(7553), 436–444 (2015)
R. Fan, Road crack detection using deep convolutional neural network and adaptive thresholding, in IEEE Intelligent Vehicles Symposium (IV) (IEEE, 2019), pp. 474–479
J. Fan et al., Deep convolutional neural networks for road crack detection: qualitative and quantitative comparisons, in 2021 IEEE International Conference on Imaging Systems and Techniques (IST) (IEEE, 2021)
J. Hoffman, E. Tzeng, T. Park, J.-Y. Zhu, P. Isola, K. Saenko, A. Efros, T. Darrell, Cycada: cycle-consistent adversarial domain adaptation, in International Conference on Machine Learning (PMLR, 2018), pp. 1989–1998
A. Chattopadhay, A. Sarkar, P. Howlader, V.N. Balasubramanian, Grad-cam++: generalized gradient-based visual explanations for deep convolutional networks, in IEEE Winter Conference on Applications of Computer Vision (WACV) (IEEE, 2018), pp. 839–847
J. Hu, L. Shen, G. Sun, Squeeze-and-excitation networks, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018), pp. 7132–7141
R. Girshick, J. Donahue, T. Darrell, J. Malik, Rich feature hierarchies for accurate object detection and semantic segmentation, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2014), pp. 580–587
R. Girshick, Fast R-CNN, in Proceedings of the IEEE International Conference on Computer Vision (2015), pp. 1440–1448
S. Ren, K. He, R. Girshick, J. Sun, Faster R-CNN: towards real-time object detection with region proposal networks. Adv. Neural. Inf. Process. Syst. 28, 91–99 (2015)
J. Redmon, S. Divvala, R. Girshick, A. Farhadi, You only look once: Unified, real-time object detection, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016), pp. 779–788
J. Redmon, A. Farhadi, Yolo9000: better, faster, stronger, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017), pp. 7263–7271
J. Redmon, A. Farhadi, Yolov3: an incremental improvement. CoRR (2018)
A. Bochkovskiy, C.-Y. Wang, H.-Y. M. Liao, Yolov4: optimal speed and accuracy of object detection. CoRR (2020)
J.R. Uijlings, K.E. Van De Sande, T. Gevers, A.W. Smeulders, Selective search for object recognition. Int. J. Comput. Vision 104(2), 154–171 (2013)
L.K. Suong, J. Kwon, Detection of potholes using a deep convolutional neural network. J. Univers. Comput. Sci. 24(9), 1244–1257 (2018)
W. Wang, B. Wu, S. Yang, Z. Wang, Road damage detection and classification with faster R-CNN, in IEEE International Conference on Big Data (Big Data) (IEEE, 2018), pp. 5220–5223
E.N. Ukhwah, E.M. Yuniarno, Y.K. Suprapto, Asphalt pavement pothole detection using deep learning method based on yolo neural network, in International Seminar on Intelligent Technology and Its Applications (ISITIA) (IEEE, 2019), pp. 35–40
N. Camilleri, T. Gatt, Detecting road potholes using computer vision techniques, in 2020 IEEE 16th International Conference on Intelligent Computer Communication and Processing (ICCP) (IEEE, 2020), pp. 343–350
K. He, G. Gkioxari, P. Dollár, R. Girshick, Mask R-CNN, in Proceedings of the IEEE International Conference on Computer Vision (2017), pp. 2961–2969
R. Fan, H. Wang, P. Cai, J. Wu, M.J. Bocus, L. Qiao, M. Liu, Learning collision-free space detection from stereo images: homography matrix brings better data augmentation. IEEE/ASME Trans. Mechatron. 27(1), 225–233 (2021)
R. Fan, H. Wang, P. Cai, M. Liu, Sne-roadseg: incorporating surface normal information into semantic segmentation for accurate freespace detection, in European Conference on Computer Vision (Springer, Berlin, 2020), pp. 340–356
H. Wang, R. Fan, P. Cai, M. Liu, Sne-roadseg+: rethinking depth-normal translation and deep supervision for freespace detection, in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE, 2021), p. (to be published)
H. Wang, R. Fan, Y. Sun, M. Liu, Dynamic fusion module evolves drivable area and road anomaly detection: a benchmark and algorithms. IEEE Trans. Cybern. (2021). https://doi.org/10.1109/TCYB.2021.3064089
J. Long, E. Shelhamer, T. Darrell, Fully convolutional networks for semantic segmentation, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2015), pp. 3431–3440
V. Badrinarayanan, A. Kendall, R. Cipolla, Segnet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39(12), 2481–2495 (2017)
O. Ronneberger, P. Fischer, T. Brox, U-net: convolutional networks for biomedical image segmentation, in International Conference on Medical Image Computing and Computer-Assisted Intervention (Springer, Berlin, 2015), pp. 234–241
H. Zhao, J. Shi, X. Qi, X. Wang, J. Jia, Pyramid scene parsing network, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017), pp. 2881–2890
L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, H. Adam, Encoder-decoder with atrous separable convolution for semantic image segmentation, in Proceedings of the European Conference on Computer Vision (ECCV) (2018), pp. 801–818
M. Yang, K. Yu, C. Zhang, Z. Li, K. Yang, Denseaspp for semantic segmentation in street scenes, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018), pp. 3684–3692
T. Takikawa, D. Acuna, V. Jampani, S. Fidler, Gated-SCNN: gated shape CNNs for semantic segmentation, in Proceedings of the IEEE/CVF International Conference on Computer Vision (2019), pp. 5229–5238
H. Wang, R. Fan, Y. Sun, M. Liu, Applying surface normal information in drivable area and road anomaly detection for ground mobile robots, in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE, 2020), pp. 2706–2711
C. Hazirbas, L. Ma, C. Domokos, D. Cremers, Fusenet: incorporating depth into semantic segmentation via fusion-based CNN architecture, in Asian Conference on Computer Vision (Springer, 2016), pp. 213–228
W. Wang, U. Neumann, Depth-aware CNN for RGB-D segmentation, in Proceedings of the European Conference on Computer Vision (ECCV) (2018), pp. 135–150
J.-Y. Zhu, T. Park, P. Isola, A.A. Efros, Unpaired image-to-image translation using cycle-consistent adversarial networks, in Proceedings of the IEEE International Conference on Computer Vision (2017), pp. 2223–2232
J. Fan, M.J. Bocus, B. Hosking, R. Wu, Y. Liu, S. Vityazev, R. Fan, Multi-scale feature fusion: learning better semantic segmentation for road pothole detection, in 2021 IEEE International Conference on Autonomous Systems (ICAS) (IEEE, 2021), pp. 1–5
L.-C. Chen, G. Papandreou, F. Schroff, H. Adam, Rethinking atrous convolution for semantic image segmentation. CoRR (2017)
R. Fan, H. Wang, B. Xue, H. Huang, Y. Wang, M. Liu, I. Pitas, Three-filters-to-normal: an accurate and ultrafast surface normal estimator. IEEE Robot. Autom. Lett. 6(3), 5405–5412 (2021)
R. Ravi, D. Bullock, A. Habib, Highway and airport runway pavement inspection using mobile lidar. Int. Arch. Photogramm., Remote. Sens. Spat. Inf. Sci. 43, 349–354 (2020)
A. Hast, J. Nysjö, A. Marchetti, Optimal ransac-towards a repeatable algorithm for finding the optimal set (2013)
C. Yuan, H. Cai, Automatic detection of pavement surface defects using consumer depth camera, in Construction Research Congress 2014: Construction in a Global Network (2014), pp. 974–983
B.-h. Kang, S.-i. Choi, Pothole detection system using 2d lidar and camera, in 2017 Ninth International Conference on Ubiquitous and Future Networks (ICUFN) (IEEE, 2017), pp. 744–746
N. Otsu, A threshold selection method from gray-level histograms. IEEE Trans. Syst. Man Cybern. 9(1), 62–66 (1979)
M.J. Flynn, Very high-speed computing systems. Proc. IEEE 54(12), 1901–1909 (1966)
S. Lee, W.W. Ro, Parallel GPU architecture simulation framework exploiting work allocation unit parallelism, in IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) (IEEE, 2013), pp. 107–117
Acknowledgements
This work was supported by the National Key R &D Program of China, under grant No. 2020AAA0108100, awarded to Prof. Qijun Chen. This work was also supported by the Fundamental Research Funds for the Central Universities, under projects No. 22120220184, No. 22120220214, and No. 2022-5-YB-08, awarded to Prof. Rui Fan.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this chapter
Cite this chapter
Fan, R., Guo, S., Wang, L., Junaid Bocus, M. (2023). Computer-Aided Road Inspection: Systems and Algorithms. In: Hosny, K.M., Salah, A. (eds) Recent Advances in Computer Vision Applications Using Parallel Processing . Studies in Computational Intelligence, vol 1073. Springer, Cham. https://doi.org/10.1007/978-3-031-18735-3_2
Download citation
DOI: https://doi.org/10.1007/978-3-031-18735-3_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-18734-6
Online ISBN: 978-3-031-18735-3
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)