An Enhanced Depth Approximation Model for Haze Removal Using Single Image

Abstract

Contrast of images which are captured in poor weather (i.e., hazy, foggy, rainy and cloudy) degrades due to optical and physical properties of light and atmospheric particles. This degradation lessens the performance of computer vision assisted systems. The weight of degradation at a pixel depends upon depth of the pixel from camera. Therefore, accurate depth estimation of a pixel is essential to improve quality of image. This paper presents color attenuation prior-based depth approximation model to approximate depth of a pixel from the camera using a single degraded image. The proposed method observed that the depth of a pixel from the camera is directly proportional to the difference of the saturation from the sum of brightness and hue. Visual quality and quantitative metrics are used to compare results of the proposed method with prominent existing methods in literature.

1 Introduction

Performance of surveillance systems, object recognition in outdoor systems, intelligent transportation systems, traffic monitoring using outdoor vision systems is highly interrupted by poor weather. Computer vision assisted systems work effectively only if input is noiseless. Image captured in mediocre weather degrades due to absorption and scattering of light by the particles present in atmosphere (such as fog, haze, and smoke), which fade color and contrast of the captured image. Camera receives sum of reflected light and scattered light as radiance of a pixel [2]. It is observed that amount of scattered light is more than absorbed light [28]. Thus, removal of fog or haze (dehazing) is highly essential. Main task of dehazing is to improve quality of degraded image using atmospheric scattering model [2, 21]. The quantity of degradation at a scene point depends upon depth of the scene point from camera. [2].

Histogram equalization and contrast stretching are generally used to enhance degraded images earlier. However, these methods focus on improvement of brightness and contrast, which make these methods unable to produce ideal dehazed images [4, 5, 19].

Methods in [2, 22, 23, 29] are based on atmospheric scattering model and generate quality results. However, the performance of these methods depends upon filtering approach. Due to inappropriate filtering or inexact depth estimation, these methods do not restore degraded edges.

It is observed that the depth of a pixel from the camera is directly proportional to the difference of the saturation from the sum of brightness and hue. Therefore, an enhanced model to approximate depth of each pixel is proposed. Original edges are preserved, and degraded edges are restored by the proposed method. Visual quality of results which are obtained by the proposed method is validated qualitatively and quantitatively.

Work in literature is presented in Sect. 2. Introduction to atmospheric scattering model and problem formulation is presented in Sect. 3. Mathematical foundation of the proposed work is discussed in Sect. 4. In Sect. 5, the proposed work is modeled mathematically. The process to restore original clear day image from hazy image is given in Sect. 6. Result analysis based on visual quality and quantitative metrics is discussed in Sect. 7. In Sect. 8, conclusion is presented.

2 Related Work

Concentration of haze changes with unknown depth of each scene point, which makes dehazing a tiring task. Objective of dehazing is to recover unknown scene depth using degraded image. Dehazing is mainly classified into three categories based on type of input; (1) extra information based [14, 20], (2) multiple images based [11,12,13, 17, 18], and (3) single image based [1, 2, 21, 22, 29].

Extra information-based methods need some additional input like cues of depth, which are obtained through different camera positioning [14, 20]. Due to extra need, these methods are not good for real-time applications. Multiple image-based methods [17, 18] require more than one image of a scene, which are taken at varying degree of polarization. However, these methods need extra hardware, which increases hardware cost and other expenses.

Thus, single image-based methods [1, 2, 21, 22, 29] have been proposed, which solves problem of dehazing by putting many constraints. Performance of these methods depends upon strong assumptions and priors. In [21], a method to maximize local contrast is proposed, which is based on an observation that haze free images have more contrast than hazy images. However, this method produces blocky artifacts due to window-based filtering. In [1], it is assumed that transmission and local surface shading are not correlated. However, method in [1] performs poorly in dense haze.

A most prominent work is proposed in [2]. In [2], it has been observed that one of the color channel of outdoor haze free images has very low intensity in non-sky region, which is used to estimate transmission. However, this method performs incorrectly in the presence of sky region, or an object brighter than atmospheric light.

More fast method is presented in [29], which estimates scene depth to recover scene transmission. This method is fast and handles sky regions to certain extent. However, it loses a few edges and does not recover degraded edges in certain hazy conditions due to inaccurate estimated depth.

3 Problem Formulation

Color and contrast of images change due to scattering of light reflected from scene point by atmospheric particles. Type, size, orientation, and distribution of particles decide severity of scattering [10]. Figure 1 shows the process of image formation in outdoor environment. A light beam reflected from surface of an object is attenuated due to atmospheric scattering. Camera receives fraction of non-attenuated light (direct attenuation) and attenuated light (airlight), which is described by atmospheric scattering model [10, 13]. Mathematical expression of atmospheric scattering model is given in Eq. 1.

Fig. 1

Image formation based on optical model

$$\begin{aligned} I_2(y)=\underbrace{I_1(y)*\text {Tr}(y)}_{\text {Direct Attenuation}}+\underbrace{\text {Ar}*(1-\text {Tr}(y))}_{\text {Airlight}} \end{aligned}$$

(1)

where y is position (usually coordinate) of a scene point, intensity of degraded image at position y is $I_2(y)$, intensity of original clear day image at position y is $I_1(y)$, $\text {Ar}$ is atmospheric light, transmission at position y is represented by $\text {Tr}(y)$, and given by Eq. 2.

$$\begin{aligned} \text {Tr}(y)=e^{-\gamma \text {Dep}(y)} \end{aligned}$$

(2)

where scattering coefficient is $\gamma$, $\text {Dep}(y)$ is depth of a pixel at position y from camera. If atmosphere contains homogeneous and little particles in size, then $\gamma$ and Ar will be constants. Approximation of $I_1(y)$, $\text {Tr}(y)$, and A from single degraded image $I_2(y)$ is the main goal of dehazing. If $\text {Dep}(y)$ is known, then $\text {Tr}(y)$ can be obtained using Eq. 2. Obtained $\text {Tr}(y)$ can recover image $I_1(y)$ using Eq. 1.

4 Mathematical Foundation of the Proposed Model

Information about depth of a scene point is not contained in single image. Thus, dehazing is challenging due to unknown depth. To restore clear day image, depth recovery is essential. The proposed work is influenced by color attenuation prior [29] to approximate depth. According to [29], the depth of a pixel increases with increased difference of saturation and brightness at same pixel. The results produced by [29] are good; however, this method loses existing edges due to lack of accuracy in depth estimation. This inspired the proposed work to introduce an enhanced depth approximation model. Method in [29] works in HSV color space which can be best described by following transformation equations [7].

$$\begin{aligned} H(y)={\left\{ \begin{array}{ll} \tfrac{1}{6}* \left( \frac{g^{\prime }(y)-b^{\prime }(y)}{\delta (y)}\right) , &{} \text {if} \quad r^{\prime }(y)=\text {MAX}(y) \\ \tfrac{1}{6}* \left( 2+\frac{b^{\prime }(y)-r^{\prime }(y)}{\delta (y)}\right) , &{} \text {if} \quad g^{\prime }(y)=\text {MAX}(y)\\ \tfrac{1}{6}* \left( 4+\frac{r^{\prime }(y)-g^{\prime }(y)}{\delta (y)}\right) , &{} \text {if} \quad b^{\prime }(y)=\text {MAX}(y) \end{array}\right. } \end{aligned}$$

(3)

$$\begin{aligned} S(y)=\frac{\delta (y)}{\text {MAX}(y)}, \qquad V(y)=\text {MAX}(y) \end{aligned}$$

(4)

$$\begin{aligned} \text {MAX}(y)=\max (r^{\prime }(y),g^{\prime }(y),b^{\prime }(y)), \quad MIN(y)=\min (r^{\prime }(y),g^{\prime }(y),b^{\prime }(y)) \end{aligned}$$

where $[r^{\prime }(y),g^{\prime }(y),b^{\prime }(y)]$ are normalize triplets representing r, g, b color intensities of input image at location y, and $\delta (y)=\text {MAX}(y)-\text {MIN}(y)$. Saturation, brightness, and hue of input image at location y are represented by S(y), V(y), and H(y), respectively.

Equation 1 represents that original clear day image $I_1(y)$ is degraded due to, (1) multiplicative reduced transmission $\text {Tr}(y)$ and (2) additive airlight $Ar(1-\text {Tr}(y))$. It is proved that the transmission $\text {Tr}(y)$ depends upon depth Dep(y) of each scene point. Thus, the degradation increases with depth. Therefore, additive airlight increases with depth due to which MAX(y) and $\text {MIN}(y)$ start approaching to airlight. At long distance, transmission will be zero and airlight will be maximum. Therefore, $\text {Tr}(y)$ and $\text {MIN}(y)$ will be almost same. Thus, $\delta (y)$ will decrease with depth and can be expressed as a function of depth $\text {Dep}(y)$ as shown in Fig. 2

Fig. 2

$\delta (y)$ as function of depth $\text {Dep}(y)$

Figure 2 shows $\delta (y)$ as a function of depth $\text {Dep}(y)$. Two solid lines represent values of $\text {MIN}(y)$ and $\text {MAX}(y)$ at varying level of depth $\text {Dep}(y)$. Difference of $\text {MAX}(y)$ and $\text {MIN}(y)$ is the value of $\delta (y)$ and represented by dashed lines. It can be observed that increased airlight causes increase in $\text {MAX}(y)$ and $\text {MIN}(y)$ intensity. However, at very long distance $\text {MAX}(y)$ and $\text {MIN}(y)$ will be almost equal due to reduced transmission and increased airlight. Thus, $\delta (y)$ decreases with depth which causes saturation S(y) to decrease and brightness V(y) to increase. Therefore, [29] considered difference of brightness and saturation as a function of depth.

However, it can be observed from Eq. 3 that hue H(y) increases with depth due to reduction in $\delta (y)$. This implies that hue H(y) and brightness V(y) are positively correlated with depth while saturation S(y) is negatively correlated.

Consider objects which are not too close to camera and not too far from camera. For these objects, maximum difference of $V(y)- S(y)$ will be one, which is not true estimation of depth by [29]. Thus, difference of brightness and saturation is unable to estimate true depth. However, the value of $H(y)+V(y)-S(y)$ will be more for same objects. Therefore, the model in [29] and the proposed model are combined to estimate the scene depth more accurately. This combination can be best represented by a combination of hue, brightness, and saturation as:

$$\begin{aligned} \text {depth}(y) \alpha (\text {hue}(y)+\text {brightness}(y)-\text {saturation}(y)) \end{aligned}$$

(5)

5 Mathematical Modeling

Mathematical model of the proposed enhanced depth approximation model is described as:

$$\begin{aligned} D(y)=(c_1+c_2B(y)+c_3H(y)-c_4S(y))/\alpha +\epsilon (y) \end{aligned}$$

(6)

where D(y) is depth, B(y) is brightness, H(y) is hue, S(y) is saturation, $c_1,c_2,c_3$ and $c_4$ are linear coefficients, $\alpha$ is used to normalization constant and $\epsilon (y)$ is random image to represent random error of the model.

5.1 Computation of Linear Coefficients

Linear coefficients are computed with an objective to improve structural similarity index (ssim), which measures variance of mixed effect of contrast, structure and luminance of two images [16, 26].

$$\begin{aligned} ssim(j,k)=l(j,k)*c(j,k)*s(j,k) \end{aligned}$$

(7)

where l(j, k) is luminance, c(j, k) is contrast, s(j, k) defines structure of two images j and k. Formulas to compute the value of l(j, k), c(j, k), and s(j, k) are defined as:

$$\begin{aligned} l(j,k)=\frac{2\mu _j\mu _k+C_1}{{\mu _j}^2+{\mu _k}^2+C_1}, \quad c(j,k)=\frac{2\sigma _j\sigma _k+C_2}{{\sigma _j}^2+{\sigma _k}^2+C_2} \quad s(j,k)=\frac{\sigma _{jk}+C_3}{\sigma _j\sigma _k+C_3} \end{aligned}$$

(8)

where $\mu _j,\mu _k$ are local mean of images j and k, respectively, $\sigma _j, \sigma _k$ are standard deviations of images j and k, respectively, $\sigma _{jk}$ is cross covariance. The value of $ssim(j,k)=1$ is possible, if and only if $l(j,k)=1$, $c(j,k)=1$, and $s(j,k)=1$. Thus equating $l(j,k)=1$, $c(j,k)=1$, and $s(j,k)=1$.

$$\begin{aligned} (\mu _j-\mu _k)^2=0, \quad (\sigma _j-\sigma _k)^2=0, \quad \sigma _j\sigma _k=\sigma _{jk} \end{aligned}$$

(9)

It can be inferred from Eq. 9 that ssim between two images will be high if squared difference of those images is low. Thus, linear coefficients are computed such that squared difference(error) is minimized. Therefore, ordinary least square estimation (OLS) is used to compute linear coefficients. Generalized regression for Eq. 6 can be expressed as:

$$\begin{aligned} D_i(y)=\left( \frac{c_1}{\alpha }+\frac{c_2}{\alpha }B_i(y)+\frac{c_3}{\alpha }H_i(y)-\frac{c_4}{\alpha }S_i(y)\right) +\epsilon _i(y) \end{aligned}$$

(10)

where $D_i(y)$ dependent variable of regression and represents random depth of ith sample. Brightness, hue, and saturation of ith sample are represented by $B_i(y),H_i(y)$, and $S_i(y)$, respectively. Random error of ith sample is $\epsilon _i(y)$.

It is assumed that the random error $\epsilon _i(y)$ is based on normal distribution with $\sigma ^2$ variance. If $\frac{c_1}{\alpha }$,$\frac{c_2}{\alpha }$,$\frac{c_3}{\alpha }$ and $\frac{c_4}{\alpha }$ are replaced by $\beta _0,\beta _1,\beta _2$ and $\beta _3$, respectively in Eq. 10, then

$$\begin{aligned} D_i(y)=(\beta _0+\beta _1 B_i(y)+\beta _2 H_i(y)-\beta _3 S_i(y))+\epsilon _i(y) \end{aligned}$$

(11)

where $\beta _{i} \le 1$ for $(i=0,1,2,3)$. Using Eq. 11, the sum of square of errors s is given by following equation.

$$\begin{aligned} s=\sum _{k=1}^{\text {num}} (D_i(y)-(\beta _0+\beta _1 B_i(y)+\beta _2 H_i(y)-\beta _3 S_i(y)))^2 \end{aligned}$$

(12)

where $\text {num}$ are number of samples used in OLS. The following equations are derived by equating partial differentiation of Eq. 12 to zero (i.e., $\frac{\partial s}{\partial \beta _0}=0, \frac{\partial s}{\partial \beta _1}=0, \frac{\partial s}{\partial \beta _2}=0$ and $\frac{\partial s}{\partial \beta _3}=0$).

$$\begin{aligned} \sum _{k=1}^{\text {num}} D_i(y)=\text {num}\beta _0+\beta _1\sum _{k=1}^{\text {num}}{B_i(y)}+\beta _2\sum _{k=1}^{\text {num}}{H_i(y)}-\beta _3\sum _{k=1}^{\text {num}}{S_i(y)} \end{aligned}$$

(13)

$$\begin{aligned} \sum _{k=1}^{\text {num}} D_i(y)*B_i(y)=\beta _0\sum _{k=1}^{\text {num}}{B_i(y)}+\beta _1\sum _{k=1}^{\text {num}}{B_i(y)}^2 \nonumber \\ +\beta _2\sum _{k=1}^{\text {num}}{H_i(y)*B(y)}-\beta _3\sum _{k=1}^{\text {num}}{S_i(y)*B_i(y)} \end{aligned}$$

(14)

$$\begin{aligned} \sum _{k=1}^{\text {num}} D_i(y)*H_i(y)=\beta _0\sum _{k=1}^{\text {num}}{H_i(y)}+\beta _1\sum _{k=1}^{\text {num}}{B_i(y)*H_i(y)} \nonumber \\ +\beta _2\sum _{k=1}^{\text {num}}{H_i(y)}^2-\beta _3\sum _{k=1}^{\text {num}}{S_i(y)*H_i(y)} \end{aligned}$$

(15)

$$\begin{aligned} \sum _{k=1}^{\text {num}} D_i(y)*S_i(y)=\beta _0\sum _{k=1}^{\text {num}}{S_i(y)}+\beta _1\sum _{k=1}^{\text {num}}{B_i(y)*S_i(y)} \nonumber \\ +\beta _2\sum _{k=1}^{\text {num}}{H_i(y)*S_i(y)}-\beta _3\sum _{k=1}^{\text {num}}{S_i(y)}^2 \end{aligned}$$

(16)

Solution of Eqs. 13, 14, 15, and 16 gives values of $\beta _0,\beta _1,\beta _2$ and $\beta _3$. Linear coefficients $c_1,c_2,c_3$ and $c_4$ can be obtained if $\alpha$ is known. As discussed that $\alpha$ is used to normalize depth. Thus, it can be obtained as:

$$\begin{aligned} \alpha = {\left\{ \begin{array}{ll} 1 \qquad if \quad d(y)\le 1\\ d_{\text {max}} \qquad \text {otherwise} \end{array}\right. } \end{aligned}$$

(17)

where $d_{\text {max}}$ is maximum of scene depth D(y).

5.2 Data Preparation for Regression Analysis Using Ordinary Least Square Estimation Method

Ground truth of the depth is unavailable due to constraint of nature (Depth of an object in a scene may change over time). Thus, 200 images captured in fine weather of outdoor scenes (mountains, animals, trees, etc.) are used to prepare sample space. For each sample image, depth $D_i(y)$ is obtained randomly using Gaussian distribution with parameters $\mu =0,\sigma ^2=0.5$. Experiments have been conducted to select value of $\sigma ^2$ to bring proper diversity in the depth map. It is found that $\sigma ^2=0.5$ is sufficient to bring proper diversity. Atmospheric light Ar is randomly obtained using uniform standard distribution. Hazy images with respect to each clear day image are prepared using Eq. 2. Curve of Eq. 11 is fitted on this sample space using Eqs. 13, 14, 15, and 16. This curve fitting gives values of linear coefficients ($c1=0.0122$, $c2=0.9592$, $c3=0.9839$, and $c4=0.7743$).

6 Properties of the Enhanced Depth Approximation Model and Scene Restoration

6.1 Edge Preserving Property

Existing edges are preserved and degraded edges are recovered by enhanced depth approximation model. Gradient of Eq. 6 will be as in Eq. 18 [9].

$$\begin{aligned} \Delta D(y)= c_2 \Delta B(y)+c_3 \Delta H(y)-c_4 \Delta S(y)+\Delta \epsilon \end{aligned}$$

(18)

where $\Delta \epsilon = 0$ according to principle of OLS Eq. 18 proves that gradients of D(y) depend upon gradient of B(y), H(y), and S(y), which indicates presence of edge in D(y) if and only if there is an edge in B(y), or H(y), or S(y). Figure 3 shows effect of hue on color attenuation prior. It can be noticed from Fig. 3f, g that edges preserved by the proposed model are more accurate. Peak signal-to-noise ratio(psnr) obtained by the proposed model proves its accuracy.

Fig. 3

Hazy image (a), Ground truth of clear day image (b), Edges in Fig. 3a (c), Edges in Fig. 3b (d), Approximated depth using method in [29] (e), and Edges in Fig. 3e ($psnr=62.85$) (f), Approximated depth obtained by the proposed method (g), Edges in Fig. 3f ($psnr=63.71$) (h)

6.2 White Regions Handling

Due to additive airlight, amount of whiteness increases in degraded image. Thus, differentiation of real white objects from atmospheric light becomes difficult. The proposed method may approximate wrong transmission in the presence of white objects in the scene. Presence of white objects results in increased brightness, low saturation, and moderate hue.

In [2, 29], the problem of white region is solved by assuming that pixels are locally at same depth. Minimum filter is used by these methods to refine depth map locally. However, minimum filters results in loss of existing edges as shown in Fig. 3f. Therefore, median filter is used by the proposed method, which solves problem of white object upto a level and preserve existing edges. Thus, the refined transmission is expressed as:

$$\begin{aligned} T_r(y)=\text {med}_{x \epsilon \omega _r(y)}{\text {Tr}(x)} \end{aligned}$$

(19)

where $T_r(y)$ is refined transmission, $\omega _{r(y)}$ is window of size $y\times y$, and $\text {Tr}(x)$ is approximated transmission. To reduce blocking artifacts which are introduced due to window-based operation, refined depth map is further smoothed using guided filter [3].

6.3 Restoration of $I_1(y)$

Atmospheric light $A^c$ in each c color channel is estimated using method of [2], where $c\epsilon (R,G,B)$. The proposed method further takes minimum of atmospheric light of each color channel as global atmospheric light, which is given by Eq. 20.

$$\begin{aligned} Ar=\min _{c\epsilon (R,G,B)}(A^c) \end{aligned}$$

(20)

where min is a function to compute minimum of given values, Ar is global atmospheric light. Equation 6 is used to approximate scene depth D(y), transmission is obtained using Eq. 2 and refined using Eq. 19. Once Ar and Tr(y) are obtained then Eq. 1 can be used to restore image $I_1(y)$. The value of $\gamma$ is critical in restoration of $I_1(y)$. Low value of $\gamma$ generates residual haze and its high value increases dehazing level, which darkens the dehazed image. Thus, the proper value of $\gamma$ is vital for restoration. The proposed work considered $\gamma =1$.

7 Experimental Analysis

The MATLAB version R2014a is used to implement the proposed method. Waterloo IVC dehazed image data set(WID) [6] is used to verify effectiveness of the proposed method. This data set consists of 25 hazy images of outdoor scenes. Results obtained by the proposed method are compared with method in [2, 8, 22, 23, 25, 29].

7.1 Qualitative Evaluation

Fig. 4

Comparison of visual quality Hazy image (a), [2] (b), [23] (c), [8] (d), [22] (e), [29] (f), [25] (g), and proposed method (h)

Figure 4 shows comparison of the results based on visual quality. Figure 4b shows that method of [2] generates artifacts near depth discontinuities and color distortion in sky region. Figure 4c shows the results obtained in [23]. It can be observed from Fig. 4c that results are dark due to overestimation of transmission. Results obtained in [8] are promising as shown in Fig. 4d. However, method of [8] is time consuming due to regularization. Method of [22] obtains better results, which are shown in Fig. 4e. However, this method produces wrong results in dense haze. Method of [29] is fast. However, this method wrongly estimates depth in the presence of white object as shown in Fig. 4f. Method in [25] is fast in comparison with other methods. However, results obtained by method in [25] are not visually pleasant as shown in Fig. 4g. The proposed method obtains better visual results as shown in Fig. 4h. It restores natural colors in sky region as well as in non-sky region.

Furthermore, Fig. 5 shows the effect of varying $\gamma$ on restored images. Image shown in Fig. 5a is restored with varying values of $\gamma =[1,1.2,1.5]$ using the proposed method. Restored images are shown in Fig. 5b–d. It can be observed that darkness of restored images increases with increased value of $\gamma$. Thus, adaptive $\gamma$ is essential for accurate dehazing.

Fig. 5

Effect of varying $\gamma$ on restored images. Hazy image (a), are restored images (b–d)

7.2 Quantitative Evaluation

Fig. 6

Images used to compute parameters e and $\overline{r}$ as given in Tables 1 and 2, respectively

Table 1 Comparison on the basis of e of images shown in Fig. 6 [15]

Table 2 Comparison on the basis of $\overline{r}$ of images shown in Fig. 6 [15]

The proposed method is validated using quantitative metrics to measure strength on the basis of restored edges, structure, and texture. Metrics e and $\overline{r}$ are computed, which quantifies the strength of the proposed method to restore and preserve edges [24, 27] using WID data set. Increasing values of e and $\overline{r}$ indicate improved quality of results.

Obtained values of metrics e and $\overline{r}$ for images shown in Fig. 6 are given in Tables 1 and 2. The proposed method performs well in comparison with method in [2, 8, 22, 23, 29] on the basis of obtained values of e and $\overline{r}$.

8 Conclusions

An enhanced depth approximation model has been proposed. The proposed depth approximation model is based on an observation that depth of a pixel from the camera is directly proportional to the difference of the saturation from sum of brightness and hue. Transmission obtained by the proposed method is further refined using local median filtering, which helps in preserving existing edges. Accuracy of the proposed model is proved on the basis of visual quantity and qualitative metrics. However, the proposed method is based on homogeneous scattering of light. Thus, this issues will be part of future work.