Abstract
This paper proposes a method that synthesizes realistic sonar images using a Generative Adversarial Network (GAN). A ray-tracing-based sonar simulator first calculates semantic information of a viewed scene, and the GAN-based style transfer algorithm then generates realistic sonar images from the simulated images. We evaluated the method by measuring the similarity between the generated realistic images and real sonar images for several objects. We applied the proposed method to deep learning-based object detection, which is necessary to automate underwater tasks such as shipwreck investigation, mine removal, and landmark-based navigation. The detection results showed that the proposed method could generate images realistic enough to be used as training images of target objects. The proposed method can synthesize realistic training images of various angles and circumstances without sea trials, making the object detection straightforward and robust. The proposed method of generating realistic sonar images can be applied to other sonar-image-based algorithms as well as to object detection.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
H.-S. Kim and D. Lee, “Intelligent psr estimator for feature extraction of a passive sonar target,” International Journal of Control, Automation and Systems, vol. 8, no. 2, pp. 677–682, 2010.
T. Kim, J. Kim, and S.-W. Byun, “A comparison of nonlinear filter algorithms for terrain-referenced underwater navigation,” International Journal of Control, Automation and Systems, vol. 16, no. 2, pp. 2977–2989, 2018.
M. B. Loc, H.-S. Choi, J.-M. Seo, S.-H. Baek, and J.-Y. Kim, “Development and control of a new auv platform,” International Journal of Control, Automation and Systems, vol. 12, no. 2, pp. 886–894, 2014.
M. G. Joo and Z. Qu, “An autonomous underwater vehicle as an underwater glider and its depth control,” International Journal of Control, Automation and Systems, vol. 13, no. 2, pp. 1212–1220, 2015.
M. H. Lee, J.-H. Moon, I.-S. Kim, C. S. Kim, and J. W. Choi, “Pre-processing faded measurements for bearing-and-frequency target motion analysis,” International Journal of Control, Automation, and Systems, vol. 6, no. 2, pp. 424–433, 2008.
J. Kim and S.-C. Yu, “Convolutional neural network-based real-time rov detection using forward-looking sonar image,” Proc. of IEEE/OES Autonomous Underwater Vehicles (AUV), IEEE, pp. 396–400, 2016.
D. P. Williams, “Underwater target classification in synthetic aperture sonar imagery using deep convolutional neural networks,” Proc. of 23rd international conference on pattern recognition (ICPR), IEEE, pp. 2497–2502, 2016.
H. Horimoto, M. Toshihiro, K. Kofuji, and T. Ishi-hara, “Autonomous sea turtle detection using multi-beam imaging sonar: Toward autonomous tracking,” in 2018 IEEE/OES Autonomous Underwater Vehicle Workshop (AUV), IEEE, pp. 1–4, 2018.
J. Kim, M. Sung, and S.-C. Yu, “Development of simulator for autonomous underwater vehicles utilizing underwater acoustic and optical sensing emulators,” Proc. of 18th International Conference on Control, Automation and Systems (ICCAS), IEEE, pp. 416–419, 2018.
S. Kwak, Y. Ji, A. Yamashita, and H. Asama, “Development of acoustic camera-imaging simulator based on novel model,” Proc. of IEEE 15th International Conference on Environment and Electrical Engineering (EEEIC), IEEE, pp. 1719–1724, 2015.
J. Riordan, F. Flannery, D. Toal, M. Rossi, and G. Dooly, “Interdisciplinary methodology to extend technology readiness levels in sonar simulation from laboratory validation to hydrography demonstrator,” Journal of Marine Science and Engineering, vol. 7, no. 5, p. 159, 2019.
R. Cerqueira, T. Trocoli, G. Neves, S. Joyeux, J. Albiez, and L. Oliveira, “A novel gpu-based sonar simulator for real-time applications,” Computers & Graphics, vol. 68, pp. 66–76, 2017.
H. Joe, J. Kim, and S.-C. Yu, “Sensor fusion-based 3d reconstruction by two sonar devices for seabed mapping,” IFAC-PapersOnLine, vol. 52, no. 2, pp. 169–174, 2019.
B. Kim, J. Kim, M. Lee, M. Sung, and S.-C. Yu, “Active planning of auvs for 3d reconstruction of underwater object using imaging sonar,” Proc. of IEEE/OES Autonomous Underwater Vehicle Workshop (AUV), IEEE, pp. 1–6, 2018.
P. C. Etter, “Underwater acoustic modeling: principles, techniques and applications,” 1995.
M. Palmese and A. Trucco, “Acoustic imaging of underwater embedded objects: Signal simulation for three-dimensional sonar instrumentation,” IEEE transactions on instrumentation and measurement, vol. 55, no. 2, pp. 1339–1347, 2006.
B. K. Novikov, O. V. Rudenko, and V. I. Timoshenko, Nonlinear Underwater Acoustics, American Institute of Physics, New York, 1987.
B. Kim and S.-C. Yu, “Imaging sonar based real-time underwater object detection utilizing adaboost method,” in 2017 IEEE Underwater Technology (UT), IEEE, pp. 1–5, 2017.
C. H. Yu and J. W. Choi, “Interacting multiple model filter-based distributed target tracking algorithm in underwater wireless sensor networks,” International Journal of Control, Automation and Systems, vol. 12, no. 2, pp. 618–627, 2014.
D. Kim, J.-U. Shin, H. Kim, H. Kim, D. Lee, S.-M. Lee, and H. Myung, “Development and experimental testing of an autonomous jellyfish detection and removal robot system,” International Journal of Control, Automation and Systems, vol. 14, no. 2, pp. 312–322, 2016.
M. Sualeh and G.-W. Kim, “Simultaneous localization and mapping in the epoch of semantics: A survey,” International Journal of Control, Automation and Systems, vol. 17, no. 2, pp. 729–742, 2019.
S. Lee, B. Park, and A. Kim, “Deep learning from shallow dives: Sonar image generation and training for underwater object detection,” arXivpreprint arXiv:1810.07990, 2018.
J. L. Chen and J. E. Summers, “Deep neural networks for learning classification features and generative models from synthetic aperture sonar big data,” Proceedings of Meetings on Acoustics 172ASA, vol. 29, no. 1, ASA, p. 032001, 2016.
A. S. Glassner, An Introduction to Ray Tracing, Elsevier, 1989.
E. Catmull, “A subdivision algorithm for computer display of curved surfaces,” Utah Univ. Salt Lake City School of Computing, Tech. Rep., 1974.
P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1125–1134, 2017.
O. Ronneberger, P. Fischer, and T. Brox, “U-net: convolutional networks for biomedical image segmentation,” Proc. of International Conference on Medical Image Computing and Computer-assisted Intervention, Springer, pp. 234–241, 2015.
J. Moon, H. Kim, and B. Lee, “View-point invariant 3d classification for mobile robots using a convolutional neural network,” International Journal of Control, Automation and Systems, vol. 16, no. 2, pp. 2888–2895, 2018.
S.-H. Kim and H.-L. Choi, “Convolutional neural network for monocular vision-based multi-target tracking,” International Journal of Control, Automation and Systems, vol. 17, no. 2, pp. 2284–2296, 2019.
J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 779–788, 2016.
E. Belcher, W. Hanot, and J. Burch, “Dual-frequency identification sonar (didson),” in Proceedings of the 2002 In-terntional Symposium on Underwater Technology (Cat. No. 02EX556), IEEE, pp. 187–192, 2002.
Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 2, pp. 600–612, 2004.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Recommended by Associate Editor Jinwhan Kim under the direction of Editor Keum-Shik Hong. This work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT) (No. 2017R1A5A1014883).
Minsung Sung received his B.E. degree from the Pohang University of Science and Technology (POSTECH), Pohang, Korea, in 2016, where he is currently pursu- ing a Ph.D. degree with the Department of IT Engineering. He is a member of the Hazardous and Extreme Environment Robotics (HERO) Laboratory, POSTECH. His research interests include deep learning, sonar image processing, underwater perception with sensor fusion, and SLAM.
Jason Kim received his B.E. degree in computer engineering in 2018 from the Ul-san National Institute of Science and Technology (UNIST), Ulsan, Korea. Currently, he is an M.S. candidate in the Department of IT Engineering at Pohang University of Science and Technology (POSTECH), Pohang, Korea. His research interests include computer simulation, underwater optical/sonar image processing, and SLAM. Meungsuk Lee received his B.E. degree in hanical engineering in 2018 from Pohang University of Science and Technology (POSTECH), Pohang, Korea, where he is currently pursuing an M.S. degree with the Department of Electrical Engineering. His research interests include underwater robotics and underwater vision.
Byeongjin Kim received his B.E. degree in electrical engineering from the Pohang University of Science and Technology (POSTECH), Pohang, Korea, in 2015, I where he is currently pursuing a Ph.D. I degree with the Department of IT Engi-I neering. His research interests include un- derwater robotics, sonar image processing, and SLAM.
Taesik Kim received his B.E. degree in IL mechanical engineering from the Ulsan National Institute of Science and Technology (UNIST), Ulsan, Korea. He is currently pursuing a Ph.D. degree with the Department of IT Engineering, Pohang University of Science and Technology (POSTECH), Pohang, Korea. His research interests include underwater robotics, hydrodynamics, and manipulation.
Juhwan Kim received his B.E. degree in electrical engineering in 2015 from the Pohang University of Science and Technology (POSTECH), Pohang, Korea, where he is currently pursuing a Ph.D. degree with the Department of IT Engineering. His research interests include underwater robotics, machine learning, and multi-agent underwater manipulation.
Son-Cheol Yu received his M.E. and Ph.D. degrees from the Department of Ocean and Environmental Engineering, University of Tokyo, in 2000 and 2003, respectively. He is an Associate Professor of the Department of IT Engineering, Electrical Engineering, and Advanced Nuclear Engineering with the Pohang University of Science and Technology (POSTECH), Korea. He is also the Director of Hazardous and Extreme Environment Robotics (HERO) Lab, IEEE Ocean Engineering Society Korea Chapter, Gyeongbuk Sea Grant Center. He has been a Researcher of mechanical engineering with the University of Hawaii from 2004 to 2007 and an Assistant Professor of mechanical engineering with the Pusan National University from 2008 to 2009. His research interest is autonomous underwater Vehicles, underwater sensing, and multi-agent-based robotics.
Rights and permissions
About this article
Cite this article
Sung, M., Kim, J., Lee, M. et al. Realistic Sonar Image Simulation Using Deep Learning for Underwater Object Detection. Int. J. Control Autom. Syst. 18, 523–534 (2020). https://doi.org/10.1007/s12555-019-0691-3
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12555-019-0691-3