Texture filtering based physically plausible image dehazing

Abstract

To address the issues of false candidate atmospheric light, halo effects and color distortion in sky regions, a physically plausible single image dehazing algorithm is proposed based on texture filtering. First, gamma correction based preprocessing is applied to the luminance channel of the haze image, which improves the luminance and contrast of the haze image simultaneously. Second, a support vector machine based classifier is trained and utilized to reject the false candidate atmospheric lights. Third, the haze image is decomposed into sky and non-sky regions with a histogram analysis based sky detection and segmentation method. And, color correction of the sky regions is carried out with a pixel distribution shifting based white balance method. The non-sky regions are smoothed with a patch shift based bilateral texture filtering process, which can preserve edges and eliminate redundant details. Fourth, a transmission estimation method based on hybrid filtering is proposed to eliminate the halo effects. Finally, the haze-free non-sky regions are recovered by solving the haze imaging model, which are then merged with the color-corrected sky regions to form the final haze-free image. Experimental results demonstrate that our algorithm can locate the valid atmospheric light, diminish the halo effects and improve the visibility remarkably, which outperforms the state-of-the-art image dehazing methods.

1 Introduction

The light received by the observer includes the direct radiance reflected by the object surface and the scattering light from the surroundings. Aerosols such as small dust particles or liquid droplets scatter the sunlight in the atmosphere, which can reduce the visibility of outdoor images. Common weather conditions like haze and fog in the hazy or dusty days can result in photographs with bad quality, especially for the long-distance photographing with relatively low contrast and weak visibility. However, haze or fog-degraded image can be unbearable for such application fields of computer vision as video surveillance, visual navigation and object detection, which demand clear and sharp input images. How to recover the haze-free images efficiently and promptly has become a hot research topic in the cross fields of computer graphics, computer vision and pattern recognition.

As far, many image dehazing methods have been put forward. The multiple images based image dehazing methods require additional prerequisites, such as several views of the same scene with different shooting conditions or the depth information of the scene, which present high requirements for the users and thus constrain its application fields. The single image based methods can be roughly divided into two categories. Image enhancement based methods increase the contrast of haze image to produce visually acceptable results, but may incur color distortion and noising effects. Prior based image dehazing methods, for example, the dark channel prior or color line prior based approaches, can obtain the haze-free image with high visibility, but may lose edge and detail information and result in halo artifacts.

To address the unsolved problems in the single image dehazing methods, we propose a texture filtering based physically plausible image dehazing algorithm, which shows superior performance than the state-of-the-art image dehazing methods. Our main contributions can be summarized as the following three aspects: (1) a support vector machine based classifier is trained to verify the candidate atmospheric lights and reject false ones. (2) The haze image is segmented into sky and non-sky regions, and color correction is applied to the sky regions to avoid color distortion. (3) A texture filtering based image dehazing approach is proposed to the non-sky regions to eliminate halo effects and improve the visibility of haze free results.

Fig. 1

The flow chart of our texture filtering based physically plausible image dehazing algorithm

2 Previous work

The state-of-the-art image dehazing methods fall into two categories, i.e. multiple images and single image based ones. The formers require additional information based on multiple images. Nayar et al. [1–3] restore the haze-free images by introducing additional atmospheric light, which is extracted from multiple strictly aligned images of the same scene taken under different weather conditions. Thus, it is hardly to be used in real-life image dehazing applications. Schechner et al. [4, 5] require multiple images under different polarization angles, and their haze-free results might be affected by noise easily [6]. Kopf et al. [7] demand a 3D model to provide the depth information of the scene so that the accurate transmission map could be obtained. However, a 3D model of the scene is unavailable in most cases, which is a big obstacle for practical use.

Due to the application limitations of the multiple images based dehazing methods, single image based dehazing methods draw more attention from the academic community in recent years. Tan [8] removes the haze by maximizing the contrast of the expected haze-free image as well as the smoothness of the transmission map, which presents visibility enhancement advantage with over-saturated color defects. Kim et al. [9] maximize the contrast while minimize the information loss; however, the loss of details and edges in the recovered haze-free image is inevitable. The above enhancement-based image dehazing methods [8, 9] also suffer from such physically invalid problems as unnatural appearance. To handle these problems, prior based image dehazing methods are designed on the basis of prior knowledge. Assuming the statistically uncorrelated relationship between the transmission and reflectance components of a local image patch, Fattal’s method [10] infers the haze-free image with independent component analysis. Nevertheless, it fails when the fog is dense in the image. Considering the 1D color distribution characteristics in the RGB color space for pixels in a small image patch, the color line prior based method [11] solves and verifies the transmission for all pixels, and the haze-free image is achieved with the interpolation reconstructed transmission map, which preserves high contrast and visibility, but looks unnatural and lacks depth perceptions.

Through statistical observations of massive haze-free outdoor images, He et al. [12] found that there are some pixels in most image patches except the sky, whose intensity is very low and close to zero in at least one color channel, which is called the dark channel prior. The dark channel prior based method [12] can obtain compelling results in most cases, but fails for some exceptions that do not hold this law. What is more, the high-intensity light sources such as the sunlight and the lights of cars, usually lead to cause higher atmospheric light values and darker haze-free images. And, the original method [12] cannot preserve edges where depth varies. Consequently, the edge-destroyed coarse transmission map results in halo effects. Soft matting can be adopted to refine the transmission map, while it is too computationally time-consuming to be applied for real-time dehazing applications. Guided image filtering [13] possesses low computation complexity while it still cannot eliminate the halo effects completely. In order to solve the left problems in the dark channel prior based methods, we propose a texture filtering based physically plausible image dehazing algorithm.

3 Texture filtering based physically plausible image dehazing algorithm

The atmospheric imaging model for haze widely used in computer vision applications is

$$\begin{aligned} I(x)=J(x)*t(x)+A*(1-t(x)), \end{aligned}$$

(1)

where I is the input image with haze, t is the transmission map, A is the atmospheric light and J is the scene radiance. x denotes one pixel in the image I. Assuming t and A have been obtained, we can transform Eq. 1 and get J as follows:

$$\begin{aligned} J(x)=\frac{I(x)-A}{\max (t(x),t_0)}+A. \end{aligned}$$

(2)

$t_0$ is the threshold introduced to prevent the denominator to be zero, which is set to 0.1 in most cases. Assuming

$$\begin{aligned} k=\frac{1}{\max (t(x),t_0)}, \end{aligned}$$

(3)

we can re-formulate Eq. 2 as

$$\begin{aligned} J(x)=k(x)*I(x)+(1-k(x))*A. \end{aligned}$$

(4)

Our algorithm first adopts gamma correction based preprocessing to improve the luminance and contrast of the haze image. Second, we put forward a support vector machine (SVM) based atmospheric light validation approach to reject false candidate atmospheric lights and get the valid one. Third, we partition the haze image into sky and non-sky regions and apply color correction to the sky regions. Fourth, the non-sky regions are smoothed with patch shift based bilateral texture filtering, which preserves strong edges and removes redundant details. And, we propose a hybrid filtering based transmission estimation method to calculate a physically plausible transmission map for the non-sky regions. Finally, the recovered non-sky regions are merged with the corrected sky regions to generate the haze-free image. The flow chart of our algorithm is shown in Fig. 1.

Fig. 2

Image dehazing example for the “train” image. a The input image. b Gamma correction with atmospheric light. c Sky detection result. d Bilateral texture filtering result. e Coarse transmission map [12]. f Our refined transmission map. g Dark channel prior based result [12]. h Color line prior based result [11]. i Our image dehazing result

3.1 Gamma correction based preprocessing

We can observe a fact that the luminance of most haze-free results obtained by dark channel prior based image dehazing methods is decreased. Equation 4 shows that any pixel whose value below atmospheric light is decreased after the dehazing process. In addition, the luminance histograms of most haze images do not take up the whole gray-scale color space. Therefore, if we increase the luminance and contrast of the input haze images, the corresponding values of the haze-free results will be promoted as well, which also avoid the underflow problem for the haze-free pixel values. Thus, gamma correction based preprocessing is applied to the luminance channel of the input haze image.

Gamma correction can be formulated as the following power-law expression:

$$\begin{aligned} I_{c_\mathrm{gamma}}=A*I_{c}^{\gamma }, \end{aligned}$$

(5)

where $I_c$ is the luminance channel of the original input haze image I, $I_{c_\mathrm{gamma}}$ is the result of gamma correction for $I_c$, A is set to 1, $\gamma \le 1$ is the gamma value and $\gamma$ is set to 0.5 in our proposed method. The preprocessing with gamma correction is applied to the luminance channel of haze image in the LAB color space.

Figures 2 and 3 illustrate two examples for the gamma correction based preprocessing. Figures 2a and 3a are the original input haze images, and their gamma correction results are shown in Figs. 2b and 3b, respectively. We can see that the luminance and contrast of haze images are improved simultaneously, which has a positive impact on the transmission estimation.

Fig. 3

Image dehazing example for the “Lviv” image. a The input image. b Gamma correction with atmospheric light. c Sky detection result. d Bilateral texture filtering result. e Coarse transmission map [12]. f Our refined transmission map. g Dark channel prior based result [12]. h Color line prior based result [11]. i Our image dehazing result

3.2 SVM-based atmospheric light validation

The estimation of atmospheric light can be misled by the brighter objects in the dark channel prior based image dehazing method [12]. A support vector machine based classifier is trained and utilized to verify the validity of candidate atmospheric lights and reject those false ones here. Considering that the atmospheric lights chosen by dark channel prior based method are reasonable in most cases, only minor false candidates need to be distinguished from the correct ones with discriminative features. As observed, the genuine atmospheric lights are usually located at the most haze-opaque regions, which should be smooth as the sky in Fig. 4a, b. However, the local regions around the brighter objects such as the light of train in Fig. 4c and the white wall in Fig. 4d, often contain strong edges. On the other hand, the positions of the most haze-degraded regions generally fall in the upper area of a haze image. As a result, for an image patch with radius r centering at the candidate atmospheric light with pixel p, the variance $\sigma$ of pixel values and the relative value of the vertical coordinate of p to the height of the image, i.e. $\Delta {y}$, are extracted as discriminative features to distinguish the valid atmospheric light from the misled brighter objects.

To collect samples for the SVM training process, we take the haze image as input, calculate the candidate atmospheric light and manually label its validity with 1 (for positive samples) and $-1$ (negative samples). Given the l training samples [14] with feature vectors $x_{i}=(\sigma _{i},\Delta {y_i})$ and their class labels $y_{i}\in \{1,-1\}$ $(i=1,\ldots ,l)$, we adopt a basic C-SVC model in the LIBSVM package [15] for the SVM model training process, i.e.

$$\begin{aligned} \begin{array}{ll} \min _{\omega ,b,\xi } &{} \frac{1}{2}\omega ^{T}\omega +C \sum \nolimits _{i=1}^{l}\xi _{i},\\ \mathrm{s.t.} &{} y_{i}(\omega ^{T}\phi (x_i)+b)\ge 1-\xi _{i},\\ &{} \xi _{i}\ge 0,i=1,\ldots ,l. \end{array} \end{aligned}$$

(6)

where $\phi (x_i)$ is a mapping function to transform $x_{i}$ to a higher dimensional space. $C >0$ is a penalty parameter of the error term $\xi _i$. Furthermore, we denote $K(x_i,x_j)=\phi (x_i)^{T}\phi (x_j)$ as the kernel function. A radial basis function (RBF) kernel is chosen with the following form:

$$\begin{aligned} K(x_i,x_j)=\exp (-\gamma ||x_i-x_j||^{2}) \end{aligned}$$

(7)

$\gamma >0$ is specified as 0.00781. Cross validation strategy is adopted to ensure the reliability of our model, and our cross validation rate is more than 95 %, which is pretty remarkable.

During the atmospheric light validation process, we choose the pixel with brightest value in the dark channel image as the first candidate atmospheric light. The related variance $\sigma$ and the relative position value $\Delta {y}$ are extracted as the input features for the SVM classifier, which returns the validity of current candidate atmospheric light. If it is rejected, we discard the whole image patch centering it. Then, the second candidate is chosen from the rest pixels until our classifier accepts a valid one. Figures 2b and 3b show the accepted atmospheric lights marked with a green circle, which demonstrate that our classifier can reject brighter light sources or white objects. Therefore, our dehazing results are brighter and more physically valid than the dark channel prior based results [12].

Fig. 4

Typical positive and negative samples of atmospheric lights for the SVM training process. a Positive samples 1. b Positive samples 2. c Negative samples 1. d Negative samples 2

3.3 Sky detection and color correction

The dark channel prior is statistically valid, which relies on the observation of pixels in the non-sky regions of the haze-free outdoor images. However, in real dehazing applications, most haze outdoor images contain sky regions, whose pixel values do not obey the dark channel prior obviously. As a result, the transmission map in sky regions typically have relatively low values. On the other hand, the color distribution in sky regions ought to be smooth. However, minor gradient differences still exist in sky regions in natural haze images. The differences come from two aspects. It may result from the limitations of camera’s sampling precision during the capture process. The image compression which causes data loss may also result in this gradient difference. As a result, the minor differences would be amplified by low transmission values which leads to color distortions in sky regions. As sky regions do not satisfy the dark channel prior, color correction methods should be applied to obtain reliable results.

We employ a histogram analysis based sky detection and segmentation method [16] to detect the sky regions and divide the haze image into sky and non-sky regions. The sky regions are straightly corrected using a pixel distribution shifting based white balance method [17]. And, the non-sky regions are restored with the following proposed texture filtering based image dehazing method. The recovered two parts are then combined to obtain the final haze free image.

Figures 2c and 3c demonstrate two sky detection and segmentation results with white sky pixels and black non-sky pixels. As illustrated in Fig. 2c, no sky regions exist in the haze image; therefore, the color correction process for the sky regions is canceled. In Fig. 3c, the recognized sky regions contain the true sky regions and few false ones. However, we find that the latter are usually the brighter white objects, which do not obey the dark channel prior as the sky regions do. Thus, they should be processed with a color correction procedure in the same way as the sky regions, which is proved to be helpful for the final dehazing result as shown in Fig. 3i.

3.4 Patch shift based bilateral texture filtering

Theoretically, we can recover the expected haze-free image with the accurate depth and transmission map of a haze image. However, accurate depth map cannot be obtained from a single haze image; therefore, we make use of prior knowledge like dark channel prior to estimate the transmission map. According to the haze imaging model, the transmission and depth information of a pixel have an inverse relationship, and transmission varies with the scene depth in the haze image. Yet, it is not strictly observed in the obtained transmission map with the dark channel prior, because the dark channel prior based image dehazing method operates on image patches, which produces edge shifting resulted haloing and blocking effects inevitably. What is more, color change of an object such as texture can also incur transmission variation, which has a negative effect on the contrast of haze-free images. However, the authentic depths may actually keep same within the object; therefore, the transmissions insides should be a constant. We take advantage of texture filtering technique to overcome the bad influence of textures and details on the haze image.

The patch shift based bilateral texture filter [18] is resorted to smooth textures and eliminate redundant details in the haze image. Its key idea lies in substituting a texture description image G in the range kernel g of the bilateral filter, i.e.

$$\begin{aligned} J_p=\frac{1}{k_p}\sum _{q\in \varOmega _p}f(||q-p||)*g(G_q-G_p)*I_q, \end{aligned}$$

(8)

where $k_p$ is a normalizing term. The output $J_p$ at pixel p is a weighted average of $I_q$ in an image patch $\varOmega _p$, which is centered at p with radius r. The spatial kernel f and the range kernel g are typically Gaussian functions. Each pixel p can be contained in a total of $(2r+1)^2$ patches $\varOmega _{q},q=1,2,\ldots ,(2r+1)^2$ in the image I. We denote $\Delta {I}(\varOmega _q)=I_\mathrm{max}(\varOmega _q)-I_\mathrm{min}(\varOmega _q)$, where $I_\mathrm{max}(\varOmega _q)$ and $I_\mathrm{min}(\varOmega _q)$ are, respectively, the maximum and minimum pixel values for all pixels within the patch $\varOmega _q$. The average intensity of the optimal patch with the minimum $\Delta {I}$ among the $(2r+1)^2$ patches is specified as the representative texture value $G_p$ at p.

The texture filtering process aims to smooth the area within an object and preserve the edges of input haze image, which are desirable for the transmission map. As shown in Figs. 2d and 3d, the texture filtered haze images are utilized to estimate the physically plausible transmission maps based on the following hybrid filtering approach, which are favorable for the improvement of contrast and saturation in the recovered haze-free images.

Fig. 5

Comparisons of different image dehazing algorithms. a Input images. b Dark channel prior [12]. c Color line prior [11]. d Our image dehazing results

3.5 Hybrid filtering based transmission estimation

The dark channel prior based image dehazing method gains a coarse transmission map $\widetilde{t}$ as follows:

$$\begin{aligned} \widetilde{t}(x)=1-\min _{y\in \varOmega (x)}\left( \min _{c}\frac{I^{c}(y)}{A^c}\right) \end{aligned}$$

(9)

Comparing the input image I and $\widetilde{t}$ in Figs. 2 and 3, we can find that edges in I are shifted in $\widetilde{t}$, which causes halo effects in haze-free images. Here, we propose a patch shift based edge preserving transmission estimation approach to gain the fine transmission map with high accuracy, and the transmission values of few pixels near the sharp edges are corrected with the guided filtering to avoid halo effects.

Each pixel x in $\widetilde{t}$ is included in altogether $d^2$ image patches $\varOmega _y, y=1,2,\ldots ,d^2$ of $d \times d$ size. Let $\Delta {\widetilde{t}}(\varOmega _y)=\widetilde{t}_\mathrm{max}(\varOmega _y)-\widetilde{t}_\mathrm{min}(\varOmega _y)$, where $\widetilde{t}_\mathrm{max}(\varOmega _y)$ and $\widetilde{t}_\mathrm{min}(\varOmega _y)$ are the maximum and minimum coarse transmission values within $\varOmega _y$. During the patch shift based transmission estimation process, we find the optimal patch $\varOmega _z$ for x with the minimum $\Delta {\widetilde{t}}$ among $\varOmega _y, y=1,2,\ldots ,d^2$ and then update the transmission of x with the coarse transmission value $\widetilde{t}(z)$ of z. Although patch shift based transmission estimation strategy can avoid most edge shifting incurred halo effects, it may not handle few pixels near sharp edges properly. Thus, guided filtering is followed to refine the transmission map and avoid the halo effects, which takes the texture filtered haze image as the guidance image.

Compared with the coarse transmission maps in Figs. 2e and 3e, our refined transmission maps in Figs. 2f and 3f show distinct advantage, which can preserve strong edges and remove redundant details in the haze image. Finally, the non-sky regions in the gamma-corrected haze image are recovered with our refined transmission map, which are then combined with the color-corrected sky regions to generate our haze-free result in Fig. 2i and 3i. Our dehazing results possess higher visibility and look more vivid, which are better than the dark channel prior based dehazing results [12] in Figs. 2g and 3g and the color line prior based dehazing results [11] in Figs. 2h and 3h.

Fig. 6

More image dehazing results with our algorithm. (a) Input images (b) our image dehazing results

4 Experimental results and discussions

Our algorithm is implemented by Visual C++ with OpenCV 3.0 on Windows 8.1. The experimental platform holds Intel$^\circledR$ Core$^\mathrm{TM}$ i5-3210M CPU and 6 GB memory.

Figure 5 demonstrates the comparison of different image dehazing methods. Figure 5a comprises the input haze images. The dark channel prior based image dehazing results [12] are shown in Fig. 5b, which present the lowest local contrast and global brightness with weak halo effects. The bad effects are improved to some extent in the color line prior based image dehazing results [11] in Fig. 5c, but they look unnatural and lifeless with severe color distortion and halo effects. In addition, the haze-free results from the two methods [11, 12] may remain little residual fog as illustrated at the bottom of Fig. 5b, c. In contrast, our dehazing results in Fig. 5d can remove the fog thoroughly, show highest visibility and look more vivid, although slight color shifts happened in the yellowish buildings of the top instance and the bluish windows of the middle instance.

Figure 6 shows more image dehazing results with our algorithm. We can conclude that our algorithm can obtain visually haze-free results with better contrast and richer details, which is superior than the existing image dehazing methods.

5 Conclusions

A texture filtering based physically plausible image dehazing algorithm is proposed to improve the dark channel prior based method in this paper. We first adopt gamma correction based preprocessing to promote the luminance and contrast of haze image. Then, a support vector machine based classifier is trained and utilized to reject false candidate atmospheric lights. Next, we apply a histogram analysis based sky detection and segmentation method to separate sky from non-sky regions in the haze image. And, the color of sky regions is corrected by a pixel distribution shifting based white balance method. The non-sky regions are smoothed by a patch shift based bilateral texture filter, which are then restored with a fine transmission map obtained by a hybrid filtering based transmission estimation method. Finally, the haze-free image is formed by merging the color-corrected sky regions and the haze-removed non-sky regions. Experimental results prove that halo effects are eliminated while high contrast and visibility are achieved with our algorithm.

Nevertheless, some odd color might be introduced when the selected atmospheric light implies the intrinsic color of the object, which is a technical challenge to be solved on our research plan.