Abstract
Underwater cameras are widely used to observe the sea floor. They are usually included in autonomous underwater vehicles (AUVs), unmanned underwater vehicles (UUVs), and in situ ocean sensor networks. Despite being an important sensor for monitoring underwater scenes, there exist many issues with recent underwater camera sensors. Because of light’s transportation characteristics in water and the biological activity at the sea floor, the acquired underwater images often suffer from scatters and large amounts of noise. Over the last five years, many methods have been proposed to overcome traditional underwater imaging problems. This paper aims to review the state-of-the-art techniques in underwater image processing by highlighting the contributions and challenges presented in over 40 papers. We present an overview of various underwater image-processing approaches, such as underwater image de-scattering, underwater image color restoration, and underwater image quality assessments. Finally, we summarize the future trends and challenges in designing and processing underwater imaging sensors.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
1 Introduction
Underwater optical imaging (OPI) is a challenging field in computer vision research [1]. As opposed to land photography, there exist many constraints in underwater imaging [2, 3]. First, due to the medium, scattering always causes a blurring effect in underwater photography; this rarely occurs in land photography. Second, wavelength absorption usually causes a color reduction in the captured images, which rarely occurs in air. Third, except for electronic noise, the sediments in the water also affect high dimensional imaging. Another problem occurs because artificial lighting is widely used for underwater photography, and this non-uniform lighting causes vignetting in captured images [4]. Furthermore, the flickering affects always exist in sunshine day. This will cause the captured images with strong highlights in the shallow ocean. Consequently, underwater images have specific characteristics that need to be taken into account during gathering and processing. Figure 1 shows the concept map of ocean observing network, which is proposed by Lu et al. [2]. Cameras are usually used in many devices. Common issues with underwater images, such as light attenuation, scattering, non-uniform lighting, shadows, color shading, suspended particles, or the abundance of marine life, can be overcome via underwater optical image processing.
The volume scattering function describes the angular distribution of light scattered by the suspension of particles in a direction at a given wavelength. Scatters redirect the angle of the photon path, absorption removes the photons from the light path. Absorption in pure water indicates that blue wavelengths are more sensitive to absorption than red wavelengths. However, in phytoplankton water, red wavelengths are not terminated more than blue wavelengths. Therefore, it is difficult to measure absorption rates in practice. On the other hand, the wavelength absorption is relayed on the geographic location of the seawater. Different salinity of seawater has a different wavelength absorption coefficients.
In this paper, we first review two categories of underwater image de-scattering methods: hardware-based approaches and software-based approaches. Then, we summarize four typical underwater image color restoration methods. Next, we cover two underwater image quality assessment methods: reference-based indexes and non-reference indexes. Finally, we summarize this paper and elaborate upon future trends in this research field.
2 Types of underwater imaging
There are several existing methods for underwater image processing. In general, these methods can be categorized into two approaches: hardware-based methods [5–10] and software-based methods [11–14]. There are four traditional hardware approaches to underwater imaging: polarization, range-gated imaging, fluorescence imaging [15, 16], and stereo imaging [17]. Light has the properties of intensity, wavelength, and polarization. Natural light is without polarization, while light reaching an imaging sensor often has biased polarization. Preliminary studies have verified that backscatter can be reduced using polarization. There are two classic methods for underwater polarization imaging, one is to use a polarization filter attached in front of the camera to receive the biased images [5, 6]. The other method is to use a polarized light source to capture different illuminated images of the same scene. The polarization method is designed to capture images quickly as well as to reduce the noise significantly. Schecher et al. [6] proposed the state-of-the-art polarization imaging method for underwater. The result is shown in Fig. 2.
Polarization imaging is a passive imaging method, while range-gated imaging is an active imaging method and widely used for laser-imaging systems in turbid water. The most of recent underwater laser-imaging methods were summarized in Ref. [8]. In a laser-imaging system, the camera is adjacent to the light source, and the target is behind the turbid medium. This system operates by selecting the reflected light from the object and blocking the backscatter by closing a flash gate. However, the laser-imaging methods have the disadvantage of being susceptible to environment and device setting are complex. Consequently, laser imaging instruments are seldom used in industrial applications.
Treibitz et al. [15] (see Fig. 3) proposed a fluorescence method to recover the shape of an underwater scene. Fluorescence imaging can be used to detect microorganisms in coral reefs. They also proposed a different direction lighting method to fuse the turbidity of a hazy image [16]. The fusion method removed the highlight of artificial light well. Roser et al. [17] proposed a stereo-imaging method to recover underwater images by estimating the visibility coefficients. This stereo-imaging method was designed by real-time algorithms and was applied into AUVs.
3 Underwater image processing
3.1 De-scattering
Recently, there have been many software-based approaches to underwater imaging. Depending on the outcome of the results, we can divide these approaches into two methods: wavelength compensation (sediment scattering) and color reconstruction (light absorption). To solve the scattering problem, many researchers have proposed both physical model-based methods and non-physical model-based methods. Traditional physical model-based methods are as follows. Fattal [18, 19] designed a color-lines method to estimate the turbidity of haze and then used a Markov Random Field model to recover clean images. He et al. [20] proposed using a dark channel prior to estimating the depth map. Then, they employed guided filtering to refine the depth map and obtain clear images. This method can achieve real-time processing. Chiang et al. [21] proposed a wavelength compensation and dehazing method for underwater image restoration. It is the first time to consider the wavelength absorption in the imaging model. Lu et al. [22] found that some flickers exist in captured underwater images and proposed a corresponding robust ambient light estimation method and underwater median dark channel prior for de-scattering. Results of some conventional physical model-based de-scattering are shown in Fig. 4.
Similarly, there have been multiple proposals for non-physical model-based methods. Garcia et al. [23] proposed local histogram equalization to address non-uniform lighting and haze. In many cases, local histogram equalization does not performs well in very dark environment. Zuiderveld et al. [24] proposed contrast limited adaptive histogram equalization (CLAHE) to adjust the target region according to an interpolation between the histograms of neighboring regions. However, non-uniform light remains on the processed image, because it operate on local regions instead of entire image. Inspired by HDR imaging, Ancuti et al. [25] proposed an espouse fusion method to combine the different exposed images via filtering. In high turbid water, the espouse fusion method cannot remove the scatter well. Galdran et al. [26] proposed a red channel-based underwater image restoration method. As mentioned in Section 1, red color channel is not always the minimum channel of the RGB color space. Gibsion et al. [27] tried to solve the scatter and noise problems simultaneously and proposed a variable Kernel size de-scattering method. After de-scattering, some halos and artifacts remain in the image. Lu et al. [28] proposed a single image dehazing method using depth map refinement. The improved bilateral filtering can smooth the depth map, while there are some residual noises exist on the image. Bazeiile et al. [29] proposed a frequency domain filtering method in YUV color space to enhance the images. The result images severe serious color distortion. Iqbal et al. [30] proposed an underwater image enhancement algorithm using an integrated color model. This method also has the color distortion issue. Experimental results of some conventional non-physical model-based methods de-scattering are shown in Fig. 5.
3.2 Underwater image color restoration
As a light absorption recovering method, Torres-Mendez et al. [31] proposed a Markov Random Field (MRF) learning method to estimate the related color value of each pixel. Alhen et al. [32] developed a hyperspectral imaging and mathematical stability model to compute the attenuation coefficients using the depth map. Petit et al. [33] designed a light attenuation inversion after processing the RGB color space contraction using quaternions. Lu et al. [34] modeled the spectral response function of a camera as a function of the wavelength of the light to recover the contrast of the colors, the experimental result is shown in Fig. 6. In this method, the artificial vignetting has also been solved.
4 Underwater image quality assessment
An underwater image quality assessment is also important to measure the performance of different underwater image processing methods. Wang et al. [35] proposed a structural similarity index, which treats the image degradation as a structural distortion. Panetta et al. [36] proposed a non-reference underwater image quality measure (UIQM), which combines the colorfulness measure, the sharpness measure, and the contrast measure of underwater images. Lu et al. [37] introduced a human perception method, the High-Dynamic Range Visual Difference Predictor 2, to predict both the visibility of artifacts and the overall quality in images. Yang et al. [38] proposed an underwater image quality evaluation (UCIQE), which uses a linear combination of the chroma, saturation, and contrast of underwater images in CIElab color space. Hou et al. [39] used the images’ sharpness to evaluate the image quality. Arredondo et al. [40] proposed a mean angular error to assess the robustness and behavior of methods with respect to underwater noises. Lu et al. [41] proposed a new index, Qu, which can evaluate the similarity of both the structures and colors of underwater images.
There are also existed some other issues for underwater optical image processing. The non-uniform artificial lighting, inhomogeneous de-scattering, high turbidity image reconstruction, image reflection, and computational underwater imaging et al. will be focused in near future. Other related research fields will be studied for underwater image processing, such as deep learning [42], cloud computing [43, 44], and internet of things [45].
In Table 1, we summarize the different underwater image enhancement methods along with some representative studies.
5 Conclusions and future trends
In this paper, we presented a comprehensive review of underwater image processing. We divided the underwater image processing methods into two categories according to their imaging types. The state-of-the-art approaches of the two classes were discussed and analyzed in detail. For software-based underwater image processing, wavelength compensation approach, e.g. physical model, non-physical model and color reconstruction approach are discussed. Finally, the quality assessment methods and future trends are summarized.
References
McGlamery B (1979) A computer model for underwater camera system. Proc SPIE 208:221–231
Lu H, Li Y, Serikawa S (2016) Computer vision for ocean observing. Artificial Intelligence and Computer Vision:1–16
Bonin F, Burgeara A, Oliver G (2011) Imaging system for advanced underwater vehicles. Journal of Maritime Research 8(1):65–86
Lu H, Li Y, Xu X, Li J, Liu Z, Li X, Yang J, Serikawa S (2016) Underwater image enhancement method using weighted guided trigonometric filtering and artificial light correction. Journal of Vision Communication and Image Representation 38:504–516
Yemelyanov K, Lin S, Pugh E, Engheta N (2006) Adaptive algorithms for two-channel polarization sensing under various polarization statistics with nonuniform distributions. Appl Opt 45(22):5504–5520
Schechner Y, Averbuch Y (2007) Regularized image recovery in scattering media. IEEE Trans Pattern Anal Mach Intell 29(9):1655–1660
Tan C, Sluzek A, Jiang T (2007) Range gated imaging system for underwater robotic vehicle. In: Proceedings of IEEE international symposium on the applications of ferroelectrics, pp 1–7
Kocak D, Dalgleish F, Caimi F, Schechner Y (2008) A focus on recent developments and trends in underwater imaging. Mar Technol Soc J 42(1):52–67
Tan C, Sluzek G, He D (2005) A novel application of range-gated underwater laser imaging system in near target turbid medium. Opt Lasers Eng 43(9):995–1009
Li H, Wang X, Bai T, Jin W, Huang Y, Ding K (2009) Speckle noise suppression of range gated underwater imaging system. Appl Opt 38(18):3937–3944
Arnold-Bos A, Malkasse J, and Kerven G (2005) A pre-processing framework for automatic underwater images denoising. In: Proceedings of the European conference on propagation and systems, breast, France, pp 1–8
Rizzi A, Gatta C, Marini D (2003) A new algorithm for unsupervised global and local color correction. Pattern Recogn Lett 24(11):1663–1677
Arnold-Bos A, Malkasset J, and Kervern G (2005) Towards a model-free denoising of underwater optical images. In: Proceedings of the IEEE Europe Oceans Conference, pp 527–532
Chambah M, Demani D, Renouf A, Courtellemont P, and Rizzi A (2004) Underwater color constancy: enhancement of automatic live fish recognition. In Proceedings of SPIE, pp 157–168
Murez Z, Treibitz T, Ramamoorthi R, and Kriegman D (2015) Photometric stereo in a scattering medium. In: Proceedings of IEEE international conference on computer vision, pp 3415–3423
Treibitz T, Schechner Y (2012) Turbid scene enhancement using multi-directional illumination fusion. IEEE Trans Image Process 21(11):4662–4667
Roser M, Dunbabin M, and Geiger A (2014) Simultaneous underwater visibility assessment, enhancement and improved stereo. In: Proceedings of IEEE international conference on robotics and automation, pp 1–8
Fattal R (2008) Single image dehazing. ACM Trans Graph 27(3):1–8
Fattal R (2014) Dehazing using color-lines. ACM Trans Graph 34(1):1–10
He K, Sun J, Tang X (2011) Single image haze removal using dark channel prior. IEEE Trans Pattern Anal Mach Intell 33(12):2341–2353
Chiang J, Chen Y (2012) Underwater image enhancement by wavelength compensation and dehazing. IEEE Trans Image Process 21(4):1756–1769
Lu H, Li Y, Zhang L, Serikawa S (2015) Contrast enhancement for images in turbid water. Journal of Optical Society of America A 32(5):886–893
Garcia R, Nicosevici T, and Cufi C (2002) On the way to solve lighting problems in underwater imaging. In Proc. of MTS/IEEE Oceans ‘02, pp 1018–1024
Zuiderveld K (1994) Contrast limited adaptive histogram equalization. Academic Press, Graphics gems IV, pp 474–485
Ancuti C, Ancuti CO, Haber T, and Bekaert P (2012) Enhancing underwater images and videos by fusion. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Providence, USA, pp 81–88
Galdran A, Pardo D, Picon A, Gila AA (2015) Automatic red-channel underwater image restoration. J Vis Commun Image Represent 26:132–145
Gibson K (2015) Preliminary results in using a joint contrast enhancement and turbulence mitigation method for underwater optical imaging. In Proc of IEEE Oceans 2015:1–5. https://searchworks.stanford.edu/view/10257123
Serikawa S, Lu H (2014) Underwater image dehazing using joint trilateral filter. Comput Electr Eng 40(1):41–50
Bazeille S, Quidu I, Jaulin L, and Malkasse JP (2006) Automatic underwater image pre-processing. In: Proceedings of Caracterisation Du Milieu Marin, pp 1–8
Iqbal K, Salam R, Osman A, Talib A (2007) Underwater image enhancement using an integrated color model. Int J Comput Sci 34(2):1–8
Torres-Mendez L, and Dufdek G (2005) Color correction of underwater images for aquatic robot inspection. In: Proc. of 5th International workshop on energy minimization method in computer vision and pattern recognition, pp 60–73
Ahlen J, Sundgren D, Bengtsson E (2007) Application of underwater hyperspectral data for color correction purposes. Pattern Recognition and Image Analysis 17(1):170–173
Petie F, Capelle-Laize A, and Carre P (2009) Underwater image enhancement by attenuation inversion with quaternions. In: Proceedings of IEEE international conference on acoustics, speech, and signal processing, pp 1177–1180
Lu H, Li Y, and Serikawa S (2015) Single underwater image descattering and color correction. In: Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing, pp 1623–1627
Wang Z, Bovik A, Sheikh H, Simoncelli E (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612
Panetta K, Gao C, Agaian S (2015) Human visual system inspired underwater image quality measures. IEEE J Ocean Eng:1–11
Lu H, and Serikawa S (2014) Underwater scene enhancement using weighted guided median filter. In: Proceedings of IEEE international conference on multimedia and expo, pp. 1–6
Yang M, Sowmya A (2015) An underwater color image quality evaluation metric. IEEE Trans Image Process 24(12):6062–6071
Hou W, Weidemann A (2007) Objectively assessing underwater image quality for the purpose of automated restoration. Proc SPIE 6575:1–10
Arredondo M, and Lebart K (2005) A methodology for the systematic assessment of underwater video processing algorithms. In: Proceedings of IEEE Europe oceans conference, pp 362–367
Lu H, Li Y, Xu X, He L, Dansereau D, and Serikawa S (2016) Underwater image descattering and quality assessment. In: Proceedings of IEEE international conference on image processing, pp. 1998–2002
Li Y, Lu H, Li J, Li X, Li Y, Serikawa S (2016) Underwater image de-scattering and classification by deep neural network. Computers & Electrical Engineering 54:68–77
Zhang Y (2016) GroRec: a group-centric intelligent recommender system integrating social, mobile and big data technologies. IEEE Trans Serv Comput 9(5):786–795
Zhang Y, Chen M, Huang D, Wu D, Li Y (2016) iDoctor: personalized and professionalized medical recommendations based on hybrid matrix factorization. Futur Gener Comput Syst:1–15
Zhang Y, Chen M (2016) Definable networking. Cloud based 5G wireless networks, pp 33–58
Schettini R, Corchs S (2010) Underwater image processing: state of the art of restoration and image enhancement methods. EURASIP Journal on Advances in Signal Processing 2010:1–14
Hou W (2013) Ocean sensing and monitoring. SPIE Press
Acknowledgements
This work was supported by JSPS KAKENHI (17K14694), Leading Initiative for Excellent Young Researcher (LEADER) of Ministry of Education, Culture, Sports, Science and Technology-Japan (16809746), Research Fund of Chinese Academy of Sciences (MGE2015KG02), Research Fund of State Key Laboratory of Marine Geology in Tongji University (MGK1608), Research Fund of State Key Laboratory of Ocean Engineering in Shanghai Jiaotong University (1315; 1510), and Research Fund of The Telecommunications Advancement Foundation and Fundamental Research Developing Association for Shipbuilding and Offshore.
Author information
Authors and Affiliations
Corresponding authors
Rights and permissions
About this article
Cite this article
Lu, H., Li, Y., Zhang, Y. et al. Underwater Optical Image Processing: a Comprehensive Review. Mobile Netw Appl 22, 1204–1211 (2017). https://doi.org/10.1007/s11036-017-0863-4
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11036-017-0863-4