Keywords

1 Introduction

In many cases, the quality of the captured images may be affected by environment, equipment and technology. Many useful information of these images may be hidden. On the one hand, images with high visibility are critical to computer vision technology including target recognition and tracking, medical research, can improve the accuracy of algorithms. On the other hand, the clear pictures have good visual effects, which are valuable images, such as space and archaeology. Therefore, the study in low-light image enhancement technology has important value and significance.

Many classic algorithms to deal with low-light images were proposed from different perspectives. Low illumination image intuitive sense is caused by the uneven distribution of pixel level. The most intuitive approach is to stretch the contrast of the image directly. However, this method of stretching contrast may lead to excessive brightness in the local area. Histogram equalization (HE) strategies [1, 2], contextual and variational contrast enhancement (CVC) [3], by seeking a layered difference representation of 2D histograms (LDR) [4], although it is possible to avoid pixel overflow and over-boost by forcing the pixel value in the range of [0,1]. However, the method only focuses on improving the contrast of the image, unable to solve the problem of color saturation. Retinex theory is a method based on human vision systems, which is widely used in image enhancement. Naturalness preserved enhancement algorithm (NPEA) was proposed in [5], the algorithm by non-uniform enlarge the contrast of brightness figure. Though solving the problem of low-light, the method may suffer from under-enhancement and color distortion. In [6], the bilateral filter can be used to generate different filter effects according to different pixel location and gray value. However, this method may appear artifacts in the edge regions of high contrast. The structure-aware smoothing model was proposed in [7], can construct the brightness map rapidly through different weighting schemes. But the algorithm will suffer color supersaturated, causing the enhanced images to be unnatural. Based on the dark channel algorithm in [8,9,10,11], the low-illumination image operates in a similar way to the fog image after inversion, and the enhancement of the images are realized by using the dark channel theory. The work proposed in [12] has applied deep learning used in image processing. Though the method has achieved good results, the global atmospheric light estimation is a challenge. The image fusion method based on camera model was proposed in [13, 14]. This algorithm can make the image enhancement more natural. However, the method is not ideal for images treatment with severe underexposure.

In this paper, we proposed different solutions to different degrees of low-light images. Experiments on a number of challenging images are conducted to reveal the advantages of our method, which can obtain results with less contrast and lightness distortion in comparison with other state-of-the-art methods.

2 Methodology

2.1 Framework Construction

In many cases, the quality of the image is not only affected by the illumination, but the exposure rate of the camera is also the cause of poor visibility. Under the same conditions, we get more information about the better exposure, as shown in Fig. 1. In [14], the mapping function of a series of different exposure rate images is obtained through the input graph, which is called the luminance transformation function (BTF).

$$ P_{i} \; = \;g(P_{0} ,k_{i} ), $$
(1)

where g is the brightness transformation function, and k is the exposure rate. By varying k, we can get a series of images P with different exposure rates.

Fig. 1.
figure 1

Images with different exposure rates from the HDR dataset. The exposure ratio is −4, −2, 0, 1.7 respectively.

Full size image

Many algorithms have a good effect on the low illumination image. However, they may cause over-enhancement, color distortion and unnatural to some extent. We found that images fusion technology can coordinate multiple sensor information in the same scene and help to obtain the information of target or scene more accurately and comprehensively. Therefore, in order to solve the above mentioned problems, we construct an image fusion frame.

$$ E\; = \;W\; \circ \;L\; + \;(1\; - \;W)\; \circ \;g(P,k), $$
(2)

where E is the output, W is the weight, the operator \( \circ \) represents element-wise multiplication. In order to achieve a better enhancement, in different situations, the choice of L is different. It is specifically introduced in Sect. 2.4.

2.2 Weight Matrix and Reflection Map Estimation

In the fusion framework, the selection of weight is the key to retain the advantages of each part and avoid weaknesses. To avoid over-enhancement, the building of the weight depends on the illumination. The weighting matrix is calculated as in [13]:

$$ W\; = \;T^{\mu } , $$
(3)

where T is the brightness map of the input image, \( \upmu \) is a parameter that controls the degree of enhancement, the value range of which is [0, 1]. In order to avoid over-enhancement, our algorithm selection μ = 0.5.

The Initialization of the Illumination Map

To preserve the brightness of the image, we initialize the brightness image T as:

$$ \hat{T}(x)\; = \;\mathop {\hbox{max} }\limits_{{c\, \in \,\{ R,G,B\} }} P^{c} (x), $$
(4)

for each individual pixel x. illumination map T is solved by optimization.

Optimized Illumination Map

In order to better preserve the edge information of the image, we chose the guide filter to optimize the brightness image T, which was proposed in [15].

$$ e(a,b) = \sum\limits_{{i \in w_{k} }} {\left( {(a_{k} P_{i} + b_{k} - \hat{T}_{i} )^{2} + \epsilon a_{k}^{2} } \right)} , $$
(5)

where the input image P denote the guide image, \( \hat{T} \) is the rough estimate of the illumination image, \( \varepsilon \) is the regularization parameter, and \( w_{k} \) is the local area of the center of the pixel point k in the guide image. Parameters of \( a_{k} \) and \( b_{k} \) can be calculated by the linear ridge regression method.

Through the processing of the above, we can get further optimized illumination map T:

$$ T(x)\; = \;\frac{1}{|w|}\sum\limits_{{k,i\, \in \,w_{k} }} {(a_{k} P_{i} \; + \;b_{k} )} , $$
(6)

where \( |w| \) is the number of pixels in the local area \( w_{k} \).

Reflection Estimation

Based on the Retinex theory, we can obtain the reflection map R of the scene through the illumination map T

$$ R\; = \;\frac{1}{T}\; \circ \;P, $$
(7)

where R and T are the reflectance map and the illumination map, respectively.

2.3 Exposure Rate Estimation

Adaptive Threshold Selection

As we mentioned earlier, globally enhanced methods may lead to over-enhancement in some regions. Therefore, we proposed the adaptive threshold to filter the pixels. The pixels below the threshold were extracted, and the pixels above the threshold were ignored.

$$ Q\; = \;\{ P(x)|T(x)\; < \;Th\} , $$
(8)

where Q contains all the lower pixels. In this article, the selection of threshold \( Th \), we use the \( Otsu \) threshold method, which can according to the image adaptive selection of the corresponding threshold.

The Determination of Exposure Rate

Intuitively, well-exposed images transfer to us more information than under-exposed images. Image entropy is a statistical form of feature, which reflects the average amount of information in the image. As in [13], the image entropy is calculated as follows:

$$ {\mathcal{H}}(B)\; = \; - \sum\limits_{i\, = \,1}^{N} {p_{i} } \cdot \log_{2} p_{i} $$
(9)

where \( B = \sqrt[3]{{Q_{r} \circ Q_{g} \circ Q_{b} }} \), \( Q_{r} ,Q_{g} \) and \( Q_{b} \) are the red, green, and blue channel of Q respectively. \( p_{i} \) is the \( i \)-th bin of the histogram of B and N is the number of bins. The optimal k can make the maximum amount of information. As proposed in [13], we obtain the optimal exposure rate by maximizing the image entropy:

$$ \hat{k} = \arg \max\nolimits_{k} {\mathcal{H}}(g(B,k)), $$
(10)

2.4 Fusion Strategy

For Slightly Under-Exposed Images

In many cases, a poorly quality image is only partially under-exposed, and the other part is still well. In order to better use this part of the good information, we select the original map to fuse with the image that enhances the exposure rate.

$$ E = W \circ P + (1 - W) \circ g(P,k), $$
(11)

where, P is the input image.

For Severely Under-Exposed Images

In the case of severely underexposed images, the above method is not satisfactory. However, images captured in low-light conditions are often of low visibility, which cause a lot of information to be hidden in the dark. To visualize the hidden information, we propose the following fusion method:

$$ E = W \circ R + (1 - W) \circ g(P,k), $$
(12)

where R is the reflection of the scene.

3 Experiments

In this section, a public dataset with a standard comparison graph (Robust Patch-Based HDR Reconstruction of Synamic Scenes) is used to visualize the vividly and intuitively of our proposed algorithm. We compare our method with other seven algorithms including CVC, HE, LDR, WAHE, Dong, LIME and CAIP in our experiments.

As can be seen from the comparison of Fig. 2, our algorithm is superior to algorithms of CVC, HE, LDR, WAHE and Dong, which can enhance the picture and keep the picture color undistorted. Compared with the LIME method, our algorithm is more natural and more consistent with human visual perception. Figure 3 shows the enhanced effect of our algorithm.

Fig. 2.
figure 2

Visual comparison among the competitors on the HDR dataset.

Full size image
Fig. 3.
figure 3

More results enhanced by our method.

Full size image

The relative order of image brightness can not only indicate the direction of light source but also the relative change of brightness. The natural degree of image enhancement is closely related to the relative order of brightness in different local regions [5, 7]. In other words, the relative order of the brightness of different regions of the picture should remain unchanged before and after the enhancement. The originally relatively bright region is still brighter after enhancement, and the originally darker region still maintains a darker brightness after enhancement. Therefore, we use the luminance order error (LOE) as a measure of the brightness fidelity performance of the algorithm. The definition of LOE is as follows:

$$ LOE = \frac{1}{m}\sum\limits_{x\, = \,1}^{m} {\sum\limits_{y\, = \,1}^{m} ( } U({\mathbf{Q}}\left( x \right),{\mathbf{Q}}\left( y \right)) \oplus U({\mathbf{Q}}_{r} \left( x \right),{\mathbf{Q}}_{r} \left( y \right))), $$
(13)

where m is the number of pixels and \( \oplus \) is the exclusive-or operator. \( {\mathbf{Q}} \) is an enhancement image and \( {\mathbf{Q}}_{r} \) is a reference image. If \( p \ge q \), then the function \( U(p,q) \) returns 1, otherwise it returns 0. In addition, \( {\mathbf{Q}}(x) \) and \( {\mathbf{Q}}_{r} (x) \) respectively are the maximum values of R, G and B at the x position of enhanced image and reference image. The smaller the \( LOE \), the better the natural degree of brightness. As a result, the effect of image enhancement is better (Table 1).

Table 1. The lightness order error.
Full size table

4 Conclusions

In this paper, we propose a fusion framework and adaptive enhancement algorithm. This model can better solve the problem of insufficient to enhance or increase, can better keep the naturalness of the image. The selection of weights is the key of fusion, and we use the guide filter to optimize the illumination diagram, thus obtaining more accurate weight. Moreover, different fusion schemes make our algorithm more applicable. The experimental results demonstrated that the proposed method is an effective and feasible low-light image enhancement algorithm.