Keywords

1 Introduction

Outdoor images and some works about computer vision are often influenced by bad weather. In any outside scenes, the light reflected from object is influenced from air scattering more or less before they reach camera. Under different weather conditions, the particles in the air and the degrees that the images are depleted are different. In the aerology, with the thickness of haze, the fog is often divided into three classes by the size of visibility, namely, dense fog, fog and mist. At present, the technology of image haze removal and restoring are separated into two classes: one is based on physical model of air scattering, which is oriented from physical reasons, and builds a image degradation physical model by the prior knowledge which are known or partially known, then restores haze removal image by simulating its reverse process; the other one is based on non-physical model, in other words, image enhancement method, which ignores the image degradation, and improves images contrast to realize image haze removal.

To remove haze from a single image, Tan [1] observed that the haze-free image must have higher contrast compared to the haze image and he removes the haze by maximizing the local contrast of the restored image. Fattal [2] estimates the albedo of the scene and then infers the medium transmission, under the assumption that the transmission and surface are locally uncorrelated. He [3] estimates the albedo by utilizing the dark channel prior, then corrects it by soft matting, and finally restores the image. Later He proposed the guided filter [4], and utilized it to replace soft matting, which can not only get a smooth edges, but also reduce the running time. Tarel [5] proposed a algorithm whose main advantage is its speed and its ability to handle both color images or gray level images. Histogram equalization is widely used for contrast enhancement in a variety of application due to its simple function and effectiveness. For images which contain local regions of low contrast, namely, bright or dark regions, adapted histogram equalization [6] can be used on such images for better results, but it expands the noise from small regions to global image. Contrast limited histogram equalization [7] can overcome the shortage in the AHE. The idea of the Retinex was conceived by Land [8] as a model of the lightness and color perception of human vision. Land introduced random walk [9] from one pixel to next pixel whose calculation is very complex. Later Frankle introduced the McCann [10] algorithm which is a Retinex based on multiple iteration, then McCann [11] proposed the McCann99 algorithm which utilizes the Gaussian pyramid to iteration. Later, Jobson, Rahman and Woodell proposed single-scale Retinex [12] and multiple-scale Retinex [13] which adopt center/surround Retinex to estimate brightness. Later Rahman [14] proposed the MSRCR algorithm which restores the color by utilizing color proportion factor of each waves in the original image. In this paper, we will review a variety of one-image haze removal methods based either on atmospheric scattering model or not, and for the different types of fog: dense fog, fog and mist, realize the single image haze removal with the various of algorithms. With the contrast analysis for the results, we estimate the appropriate algorithm to haze image for each kind of fog.

2 Haze Removal Algorithm

2.1 The Haze Removal Algorithm Based on Physical Model

During the light go through some mediums with scattering, there will be some light attenuation in the original direction, which will be dedicated to other directions. In computer vision and computer graphics, the model widely used to describe the formation of a haze image is as follows [15]:

$$\begin{aligned} I\left( x\right) =t\left( x\right) J\left( x\right) +A\left( 1-t\left( x\right) \right) \end{aligned}$$
(1)

where I is the observed intensity, J is the scene radiance, A is the global atmospheric light, and t is the medium transmission describing the portion of the light that is not scattered and reaches the camera. The goal of haze removal is to recover J, A, and t from I. The first term \(t\left( x\right) J\left( x\right) \) on the right hand side of Equation 1 is called direct attenuation [1], and the second term \(A\left( 1-t\left( x\right) \right) \) is called airlight [16].

Haze removal based on one image, as it is just depending on the input I, estimating atmospheric light A and medium radio t by exploiting some assumption or knowledge prior, achieving finally the restored image J.

2.1.1 Haze Removal Using Dark Channel Prior

The dark channel prior is based on the following observation on haze-free outdoor images: in most of the non-sky patches, at least one color channel has very low intensity at some pixels. In other words, the minimum intensity in such a patch should have a very low value. Formally, for an image J, it can be defined

$$\begin{aligned} J^{dark}\left( x\right) =\underset{y\in \varOmega \left( x\right) }{min}\left( \underset{c\in \left\{ r,g,b\right\} }{min}\left( J^{c}\left( y\right) \right) \right) \end{aligned}$$
(2)

where \(J^{c}\) is a color channel of J and \(\varOmega \left( x\right) \) is a local patch centered at x. Our observation says that except for the sky region, the intensity of \(J^{dark}\) is low and tends to be zero, if J is a haze-free outdoor image. We call \(J^{dark}\) the dark channel of J, and we call the above statistical observation or knowledge the dark channel prior.

Before we deduce the transmission \(t\left( x\right) \), we need to get atmospheric light A by the dark channel prior firstly. Then we can get the haze-free image during the reverse process of atmospheric scatting model.

2.1.2 Dark Channel Prior Based on Guided Filter

He [4] simply filter the raw transmission map under the guidance of the hazy image: first apply a max filter to counteract the morphological effects of the min filter, and consider this as the filtering input of the guided filter. Using guided filter, on the one hand we can get a more smooth edges, on the other hand it has a faster running time.

We first define a general linear translation-variant filtering process, which involves a guidance image I, an filtering input image p, and an output image q. Both I and p are given beforehand according to the application, and they can be identical. The filtering output at a pixel i is expressed as a weighted average:

$$\begin{aligned} q_{i}=\underset{j}{\sum }W_{ij}\left( I\right) p_{j} \end{aligned}$$
(3)

where i and j are pixel indexes. The filter kernel \(W_{ij}\) is a function of the guidance image I and independent of p. This filter is linear with respect to p.

The key assumption of the guided filter is a local linear model between the guidance I and the filtering output q. We assume that q is a linear transform of I in a window \(\omega _{k}\) centered at the pixel k:

$$\begin{aligned} q_{i}=a_{k}I_{i}+b_{k},\forall i\in \omega _{k} \end{aligned}$$
(4)

This local linear model ensures that q has an edge only if I has an edge. To determine the linear coefficients\(\left( a_{k},b_{k}\right) \). By the linear ridge regression model [17], the output image calculated as:

$$\begin{aligned} q_{i}=\frac{1}{\mid \omega \mid }\underset{k\mid i\in \omega _{k}}{\sum }\left( a_{k}I_{i}+b_{k}\right) \end{aligned}$$
(5)

If the guidance image I and input image are identical, that is \(I=p\), we can have \(a_{k}=\sigma _{k}^{2}/\left( \sigma _{k}^{2}+\varepsilon \right) \),\(b_{k}=\left( 1-a_{k}\right) \mu _{k}\). According to the different \(\varepsilon \), with \(q_{i}=\bar{a}_{i}I_{i}+\bar{b}_{i}\), we can get output image.

2.1.3 No-Black-Pixel Constraint (NBPC)

The method is proposed by Tarel [5] in 2009, its main advantage compared with other is speed, another advantage is the possibility to handle both color images or gray level images. On a gray level image, the model of the effect of the fog is established by Koschmieder:

$$\begin{aligned} I\left( x,y\right) =R\left( x,y\right) \left( 1-\frac{v\left( x,y\right) }{I_{s}}\right) +V\left( x,y\right) \end{aligned}$$
(6)

This model is directly extended to a color image by applying the same model on each RGB component. This method can be decomposed into several steps: estimation of \(I_{s}\), inference of \(V\left( x,y\right) \) from \(I\left( x,y\right) \), estimation of \(R\left( x,y\right) \), smoothing to handle noise amplification and final tone mapping.

2.1.4 Fast Haze Removal Based Average Filter

The algorithm [18] is based on the analysis of physical-based model, and estimate airlight and atmospheric light utilizing the simple average filter.

To estimate the airlight \(L\left( x\right) \), we utilize average filter to estimate \(t\left( x\right) \) firstly, and set a offset for the result which is computed by average filter. for estimating the atmospheric veil, compare to dark channel prior, we apply a simply way, that is the maximum of the whole pixel RGB components in original image and average value among the maximum of dark channel.

$$\begin{aligned} A=\frac{1}{2}\left( max\left( \underset{c\in \left\{ r,g,b\right\} }{max}\left( H^{c}\left( x\right) \right) \right) +max\left( M_{ave}\left( x\right) \right) \right) \end{aligned}$$
(7)

which not only avoids mistakes which regard the brightness gray value as global airlight directly, but also improves the speed of running time.

2.2 The Haze Removal Algorithm Based on Image Enhancement

2.2.1 The image Enhancement Based on Frankle-McCann Retinex

Retinex is the term which combines the words both Retina and Cortex. Retinex theory [19] is that as a precondition of constancy theory accorded with the human eye color. It is assumed that the foundation of establishment of the human eyes perceived color and brightness of object determined by atmospheric brightness and reflection of object surface. The main idea of preserving color constancy is that estimating brightness, extracting the influences of atmospheric brightness, finally we achieve the correct color and brightness of object.

Frankel-McCann Retinex [10] adapt one-dimensional path which is specific spiral structure to estimate brightness. In each wave of RGB space, we estimate the destination pixel brightness by make a comparison with the other pixels. If there are n pixels along the path which is between the pixel \(S\left( x_{1},y_{1}\right) \) and the pixel \(D\left( x_{2},y_{2}\right) \), whose brightness are \(\left( d_{1},d_{2},\ldots d_{n}\right) \), so the bright relation between S and D can be expressed with a series of multiplication, as follows:

$$\begin{aligned} \frac{D}{S}=T\left( \frac{d_{2}}{d_{1}}\right) \times T\left( \frac{d_{3}}{d_{2}}\right) \times T\left( \frac{d_{4}}{d_{3}}\right) \times \cdots \times T\left( \frac{d_{n}}{d_{n-1}}\right) \end{aligned}$$
(8)

where \(T\left( \cdot \right) \) is threshold function \(T\left( x\right) ={\left\{ \begin{array}{ll} \begin{array}{c} x\\ 1 \end{array} &{} \begin{array}{c} x\le 1\\ x>1 \end{array}\end{array}\right. }\).

In practice, FMR can be decomposed into for steps which are comparing, multiply, reset and averaging. There are more pixels near the predicted center, that is the reason why there is a better correlation near the center.

2.2.2 The Image Enhancement Based on McCann99

McCann99 Retinex algorithm [11] is the same as FMR algorithm in essence, and it need comparing, multiplying, reset and averaging to iterate. But McCann99 Retinex adapts image pyramid construction instead of spiral path, McCann99 Retinex restrains the length and width of input image, it need them to be formulated into \(nrows\cdot 2^{n}\times ncols\cdot 2^{n}\) and \(cols\ge rows\), \(1\le cols,rows\le 5\). This constraint arises from the fact that each level of the image pyramid differs from previous levels by a factor 2 in each dimension, the size of top is \(rows\times cols\), the bottom is the original image. McCann99 algorithm estimates the reflection component by comparing each pixel to each of its 8 immediately neighboring pixels in clockwise order; then by interpolating the result of last level, make the size both the width and height same as next level; and iterating the above steps; finally, with comparing the original image to its 8 neighborhood, we can achieve the enhancement image.

2.2.3 The Image Enhancement Based on Multiple Scale Retinex with Color Restoration

Among the many improved center/surround algorithm, both single-scale Retinex(SSR) [12] and multi-scale Retinex (MSR) [13] adapt the Gaussian fuzzy about processing the brightness component. Compare to the SSR, the difference of MSR algorithm is that MSR need to provide Gaussian Fuzzy to each scale in the original image, then accumulate every scale about calculating the

\(Log\left[ R\left( x,y\right) \right] =Log\left[ R\left( x,y\right) \right] +Weight\left( i\right) *\left( Log\left[ I_{i}\left( x,y\right) \right] -Log\left[ L_{i}\left( x,y\right) \right] \right) \) .

The multi-scale Retinex with color restoration combines the dynamic range compression of the small-scale Retinex and the tonal rendition of the large scale Retinex with a universally applied color restoration. MSRCR overcomes the problem that the haze-free image during the both SSR and MSR exist biasing color. In previous, more rarely, the gray-world violations can simply produce an unexpected color distortion. Jobson [14] provides good color rendition for images that contain gray-world violations.

2.2.4 Contrast Limited Adaptive Histogram Equalization (CLAHE)

Histogram equalization primary idea is that to concentrate gray histogram in the local region of original image into a whole coverage gray region. A modification of histogram equalization called the Adaptive Histogram Equalization [6] can be used on images which contain local regions of low contrast bright or dark regions for better results. It works by considering only small regions and based on their local cdf, performs contrast enhancement of those regions.

Contrast limited adaptive histogram equalization [7] is different with adaptive histogram equalization, the main difference is limited contrast. In CLAHE, the contrast of each small regions must be limited, which can overcome oversize noise about AHE. To restrict the contrast, we can clip the histogram by setting a threshold earlier before computing cdf. The value clipped from histogram, that is limited contrast, is determined from distribution of histogram, which is the size of neighbor.

2.3 Contrast Results

2.3.1 For the Mist

Figure 1 show the haze-free images of mist based on physical model method.

Fig. 1
figure 1

Haze removal results of mist based on physical model, left the original images, right the haze-free images. From top to bottom dark channel prior, guided filter, NBPC, average filter

Full size image

From Fig. 2, we can see the haze-free images of mist based on image enhancement method.

Fig. 2
figure 2

Haze removed images of mist based on image enhancement, left the original images, right the haze-free images. From top to bottom FMR, RM99, MSRCR, CLAHE

Full size image

According Figs. 1 and 2, we can find that for the mist images whose contrast are not low, and the distribution of fog is homogeneous, the algorithms based on dark channel achieve better effects while consuming more time. Other algorithms based on CLAHE or Retinex require less running time, but lead to worse haze removal ability while creating bias. It appears the algorithm based on average filter preserves the advantage of algorithms based on dark channel and decreases time cost.

2.3.2 For the Fog

Figure 3 show the haze-free images of fog based on physical model method.

Fig. 3
figure 3

Haze removal results of fog based on physical model, left the original images, right the haze-free images. From top to bottom dark channel prior, guided filter, NBPC, average filter

Full size image
Fig. 4
figure 4

Haze removal results of fog based on image enhancement, left the original images, right the haze-free images. From top to bottom FMR, RM99, MSRCR, CLAHE

Full size image

From Fig. 4, we can see the haze-free images of fog based on image enhancement method.

Fig. 5
figure 5

Haze removal results of dense fog based on physical model, left the original images, right the haze-free images. From top to bottom dark channel prior, guided filter, NBPC, average filter

Full size image
Fig. 6
figure 6

Haze removal results of dense fog based on image enhancement, left the original images, right the haze-free images. From top to bottom FMR, RM99, MSRCR, CLAHE

Full size image

Take a look at figures from Figs. 3 and 4, method based on CLAHE bring about a serious problem of blocks in images, and the haze removal techniques based on Retinex lead to image-free so bright that there is something invisible in images. Although the contrast and brightness of the fog are great obstacles, the haze removal techniques based on dark channel managed to solve the problem.

2.3.3 For the Dense Fog

Figure 5 show the haze-free images of dense fog based on physical model method.

From Fig. 6, we can see the haze-free images of dense fog based on image enhancement method.

For the dense fog situation with which the contrast, brightness and quality of image are extremely terrible, so the haze removal techniques based on image enhancement reveal its excellent power to solve the problem compare to the ones based on physical model. From Figs. 5 and  6, we can find that the algorithms based on CLAHE and Retinex have better removal effects.

3 Conclusion

In this paper, we summed up a series of haze removal techniques and applied them with a few hazy images under several weather conditions. Firstly, The haze image under dense fog gain properties including low contrast ratio and low luminance, but the haze removal algorithms based on Retinex and CLAHE perform well on these images. Secondly, The haze removal algorithms based on dark channel present good results on the hazy images under mist. Eventually, we could use average filtering algorithm based on dark channel to deal with hazy images which have homogeneous mist and not bad contrast ratio, since it preserves the advantage of algorithms based on dark channel and decreases time cost. According to the experiment results, we could find that methods based on physical model increase the computational complexity and time cost, since the methods always use the model parameters estimated by optimal method. However, among the methods based on non-physical model, some could reach the real-time requirement, but the performance does not reach the restriction. In the practical applications of computer vision, the image haze removal algorithms retain the following problems. (1) Automatic. The methods based on multiple images or user interaction are restricted by scene, weather, imaging equipment and so on. However, the applications of intelligent transportation and video surveillance demand the algorithms focusing on single image and executing automatically. (2) Real-time. The algorithm should have the property of real-time, in order to adapt to the needs of practical application.

The image haze removal technique is a hotspot in the field of image processing and computer vision and it is widely used in the outdoor visual system. Since the robustness, aromaticity and real time of the algorithm are required in practical applications. The system should select the algorithms which perform better, and further achieve real-time processing so as to reach the requirement of practical applications.