Adaptive Low-Light Image Enhancement Optimization Framework with Algorithm Unrolling

Abstract

Images captured in a dark environment may cause low visibility and lose significant details leading to poor performance of vision-based recognition systems. Recently, deep learning-based methods have been proposed for low-light image enhancement (LIE) with different priors or training schemes. However, even those LIE methods may introduce visual artifacts into the enhanced images. This paper proposes an adaptive LIE optimization framework that allows to re-optimize different deep learning-based LIE methods based on an adaptive quality evaluation (QE). Specifically, we design an interpretable and learnable LIE-QE module for LIE. To find the optimal structure of the LIE-QE module, we propose an algorithm unrolling method to design the LIE-QE module, where the each layer of the decomposition component of the LIE-QE module can be interpreted as WLS edge-aware smoothing. Both qualitative and quantitative experiments were conducted, and the evaluation verified the effectiveness of the proposed learnable deep unrolled LIE-QE module for LIE. The results shows that the proposed LIE framework can effectively improve different deep learning-based LIE methods indicating the potential of the optimization framework with LIE-QE module to re-optimize existing DNN-based LIE methods.

1 Introduction

In our daily life, low-light images are fairly common due to insufficient illumination or limited exposure time. Low-light images not only have poor visibility, but also degrade the performance of vision-based recognition systems that designed for high-quality images.

Low-light image enhancement (LIE) has been an active research topic for decades, and many classic methods are based on intensity transform [18], filtering operations or Retinex theory that utilize intrinsic properties of scene or objects to obtain better LIE [4, 7, 10, 11, 14].

Recently, many learning-based methods have been proposed for LIE with deep neural network (DNN). One line of works modify existing DNN with different priors or training schemes for LIE, such as Zero-Reference [6], Zero-DCE++ [12], DALE [9], EnlightenGAN [8], LLFlow [20] and SNR-Aware transformer [24]. Another line of works takes intrinsic image decomposition or Retinex theory [10, 11] as a main illumination constraints to design DNN structure for LIE, and integrates additional schemes to further optimize the networks, such as RetinexNet [22], LightenNet [13], KinD [26], KinD++ [25], URetinex-Net [23] and LIE with semantic segmentation [2].

Fig. 1.

Visual artifacts introduced using some recent low-light enhancement methods, where (a) is the ground-truth image, (b) is the low-light image, and (c)–(f) are the results of RetinexNet [22], KinD [26], DALE [9] and Zero-DCE++ [12], respectively.

However, experiments indicate that even though the state-of-the-art LIE methods may inevitably introduce some visual artifacts into the enhanced output, like underexposure, color shift or loss of details, as shown in Fig. 1. It indicates that LIE is still an unsolved problem. It would be significant to develop a framework to further improve the performance of existing DNN-based methods without completely re-designing the trained DNN structure.

Therefore, this paper proposes an adaptive LIE optimization framework that enables to re-optimize different deep learning-based LIE methods based on an adaptive quality evaluation (QE) of the enhanced low-light images. Specifically, we design the LIE optimization framework with an interpretable and learnable LIE-QE module. We employ the algorithm unrolling [5, 15, 17] method to determine the optimal structure of the LIE-QE module, where the each layer of the decomposition component of the LIE-QE module can be interpreted as WLS edge-aware filtering [3].

Both qualitative and quantitative experiments were conducted to evaluate the effectiveness of the proposed learnable deep unrolling LIE-QE module for LIE. The experimental results demonstrate that the proposed LIE framework can significantly further improve the performance of different deep learning-based LIE methods, such as RetinexNet [22], KinD [26], DALE [9] and Zero-DCE++ [12]. Furthermore, the results also indicate the potential of the optimization framework with LIE-QE module to re-optimize a variety of existing DNN-based LIE methods.

The main contributions are summarized as follows:

• 1. An adaptive LIE optimization framework is constructed that can re-optimize different deep learning-based LIE methods.

• 2. An interpretable and learnable LIE-QE module is proposed to evaluate the quality of LIE methods, and the DNN structure of the LIE-QE module can be interpreted as WLS edge-aware filtering from an algorithm unrolling perspective.

• 3. Qualitative and quantitative experiments indicates that our LIE optimization framework with the LIE-QE module can effectively improve deep learning-based LIE methods, including RetinexNet [22], KinD [26], DALE [9] and Zero-DCE++ [12].

2 LIE Optimization Framework with Algorithm Unrolling

We design an adaptive LIE optimization framework to optimize the LIE methods and guide them generate higher quality results that are more visually pleasing to humans. The framework consists of the low-light image enhancement method and unrolling LIE-QE module, as shown in Fig. 2. The quality evaluation serves as the quality constraint to optimize and improve the LIE module, leading to higher quality and better performance which is more preferable to the human visual system. The LIE optimization framework can be applied to a variety of deep-learning LIE methods, such as KinD [26], ZeroDce [6] and the detailed evaluations were shown in Sec. 3.

Fig. 2.

The overall structure of our unrolling adaptive LIE optimization framework.

2.1 Unrolling LIE-QE Module

As shown in Fig. 2, our Unrolling LIE-QE model mainly contains the Unrolling Decomposition Module (UDM) that is the key component of Unrolling LIE-QE and the Feature Extraction Module.

Unrolling Decomposition Module. Our Unrolling Decomposition Module (UDM) is a novel decomposition module which unrolls the decomposition iterations into a neural network by algorithm unrolling. UDM contains n Unrolling blocks, with each block representing an iterative step of the solution, as shown in Fig. 3. UDM can be interpreted as WLS edge-aware smoothing [3], combining the prior knowledge of WLS [3] with the advantages of a data-driven neural network. This leads to better performance and interpretability of the module.

Fig. 3.

The overall structure of Unrolling Decomposition Module(UDM).

We designed an objective function as the WLS filtering [3] and unroll the structure of iterative WLS, whose the constraint term is a related term of the output U:

$$\begin{aligned} \left\{ \begin{array}{cc} min. \{ (U-I)^{2} + \lambda w(Z)\},\\ s.t.\,U=Z. \end{array} \right. \end{aligned}$$

(1)

where, I is the input image, U is the output, w(Z) is a constraint term of the output U, $\lambda$ is the weight of w(Z) and Z is an auxiliary variable to repalce U for making the problem easy to solve.

Alternating direction method of multipliers (ADMM) [1] is used to solve Eq. (1). Then, we obtain the following equations:

Here, $Z^{k}$, $\mu ^{k}$ and $\alpha ^{k}$ denote the auxiliary variable Z  and the constraint coefficients $\mu$ and $\alpha$ in the k th iteration, respectively. $U^{k+1}$, $Z^{k+1}$, $\mu ^{k+1}$ and $\alpha ^{k+1}$ denote U, Z, and $\alpha$ in the $k+1$ th iteration. h() is a nonlinear shrink function designed as fully convolutional neural network.

UDM unrolls the iterative update solution of Eq. (2a), (2b), (2c) and (2d) into a neural network using the algorithm unrolling, as shown in Fig. 3. The iteration is converted into a data-driven training process using neural network training techniques. $U^{k}$, $Z^{k}$, $\mu ^{k}$ and $\alpha ^{k}$ obtained in the kth iteration, are inputted for $u^{k+1}$, $Z^{k+1}$, $\mu ^{k+1}$ and $\alpha ^{k+1}$ respectively. That iterations are repeated until the final decomposition result u is got. The U module, Z module, $\mu$ module, and $\alpha$ module of the Unrolling block in Fig. 3 correspond to Eq. (2a), (2b), (2c) and (2d), respectively.

Algorithm unrolling allows each hidden layer in the network to have a certain meaning, which can be interpreted as a certain step in the iteration. Each Unrolling block in UDM acts as a WLS filter [3], showing a similar effect in decomposition as the WLS filter [3].

As shown in Fig. 4, UDM is capable of decomposing the illumination-independent structure map and the illumination detail map from the low-light image. The illumination detail map contains the illumination information of the image, showing the regions impaired by the low-light environment. This illumination detail map can be used to evaluate the quality of low-light enhanced images and optimize LIE methods more effectively.

Moreover, the processing time of UDM in decomposition is 0.00789 s per image, which overcomes the disadvantage of the high-time-complexity iterations.

Fig. 4.

Outputs of Unrolling Decomposition Module. (a) is the low-light image, (b) is the structure map and (c) is the illumination-detail map from UDM.

The input image I is converted from RGB color space to CIELAB color space to obtain the L-channel $I_L$ containing illumination information. UDM decomposes the L channels of the input image to obtain the illumination detail map $u_{id}$ and the structure map $u_s$:

$$\begin{aligned} U_s = UDM(I_L), \end{aligned}$$

(3)

where $I_L$ is the L-channel image of input image I, $U_s$ is the structure map of I.

After obtaining structure map $U_s$ using UDM, it can be subtracted from the original low-light image $I_L$ to obtain the illumination-detail map $u_{id}$ containing information about the illumination in the image:

$$\begin{aligned} U_{id}=I_L - U_s, \end{aligned}$$

(4)

where $U_{id}$ is the illumination-detail map of I.

Feature Extract. The input enhanced image $I_{en}$ and the reference $I_{ref}$, as well as their respective structure map $U_{s\_en}$, $U_{s\_ref}$ and illumination detail map $U_{id\_en}$, $U_{id\_ref}$ are fed into the Feature Extraction Module. The module extracts perceptual features, structure features and illumination features of $I_{en}$ and $I_{ref}$ using pretrained VGGNet [19]. It benefits the quality evaluation for low-light image enhancement. We choose the outputs of the five latent layers of the VGGNet [19] as our features.

Quality Evaluation. The final image quality evaluation is calculated by taking the weighted sum of the feature similarities, which are obtained by L2 distance between the perceptual features, structure features and illumination features of $I_{en}$ and $I_{ref}$. This evaluation can be used to optimize LIE methods more effectively.

2.2 Loss of the LIE Optimization Framework

The loss of our framework is given by:

$$\begin{aligned} \mathcal {L} =\mathcal {L}_{en}(I_{dark},I_{en}) +\beta *\mathcal {L}_{q}(I_{en},I_{ref}) , \end{aligned}$$

(5)

where $\mathcal {L}_{en}$ is the enhancement loss of the framework from LIE module, $\mathcal {L}_{q}$ is the quality constraint. $\beta$ weights the influence of $\mathcal {L}_{q}$ which is set as 0.2 in our experiments. $I_{dark}$ is low-light image inputted to the LIE method, $I_{en}$ is enhancement image of the LIE method and $I_{ref}$ is the reference image.

$\mathcal {L}_{en}$ depends on the LIE methods used in our framework, calculated by the low-light image $I_{dark}$ and enhanced image $I_{en}$. It is typically as same as the loss function of the selected LIE method in general. To ensure the versatility of our framework and preserve the original characteristics and advantages of the selected LIE method, it is recommended to keep $\mathcal {L}_{en}$ consistent with the original loss function of the chosen method.

$\mathcal {L}_{q}$ is the result of Unrolling LIE-QE calculated by enhanced image $I_{en}$ which is the output of the LIE method and reference image $I_{ref}$. $\mathcal {L}_{q}$ is the quality constraint, which is used to optimize the LIE method for better and higher quality enhanced results. It guides the LIE method to produce results that are more visually pleasing to humans.

3 Experiment

We evaluated our method using LOL dataset [22] which is a real-world dataset comprising low-light and normal-light image pairs, as well as Large Scale Low-light Synthetic(LSLS) dataset [16]. We selected low/normal-light image pairs in LOL dataset [22] and synthesized dark images and their high-contrast images from LSLS [16] for our experiments.

3.1 Evaluation of the Proposed Framework

To evaluate the effectiveness of our framework, we implemented LIE-QE of the framework using WLS [3] instead of UDM. The framework implemented with WLS [3] is tested to optimize KinD [26] and Zero-DCE++ [12] in comparison with the original KinD [26] and Zero-DCE++ [12].

Table 1. Experiment results of the framework implemented with WLS [3].

The experiments were conducted on LOL dataset [22] and LSLS dataset [16]. We used PSNR and SSIM [21] as the image quality metrics in the experiment. The results are presented in Table 1, where “WLS-Optimized” indicates that the method is optimized using the framework with the LIE-QE module implemented using WLS [3]. PSNR and SSIM [21] of the WLS-Optimized images are higher than the original ones, indicating that our framework can lead to better enhancement and higher quality. It demonstrates that our framework has the ability to optimize LIE methods.

3.2 Evaluation of Unrolling Decomposition Module

Decomposition Results. We tested the effectiveness in decomposition of UDM compaired with WLS [3]. As shown in Fig. 5, our UDM is effective in decomposing the illumination-independent structure map and the illumination-detail map from the low-light image. The illumination map from UDM contains the illumination information of the image, showing the regions of image affected by the low-light environment. It can guide the quality evaluation of low-light enhanced images and optimization for low-light enhancement algorithms more effectively. Moreover, the decomposition results indicate that UDM is similar to WLS [3]. This suggests that UDM can be interpreted as WLS [3], increasing the interpretability of the model.

Fig. 5.

Decomposition results of UDM. (a) is the low-light image, (b) is the reference image, (c) is the structure map from UDM, (d) is the illumination-detail map from UDM and (e) is the illumination-detail map from WLS [3].

Ablation Studies. We conducted ablation studies to evaluate the important role of UDM playing in our unrolling LIE optimization framework. We compared the performance of UDM and WLS in our framework by implementing them in the LIE-QE module to optimize the LIE methods.

Fig. 6.

Experiment results of ablation studies in LOL dataset [22]. (a) are the reference images, (b) are the low-light images, (c) are the results of WLS-Optimized RetinexNet, (d) are the results of UDM-Optimized RetinexNet, (e) are the results of WLS-Optimized SNR and (f) are the results of UDM-Optimized SNR.

Fig. 7.

Experiment results of ablation studies in LSLS dataset [16]. (a) are the reference images, (b) are the low-light images, (c) are the results of WLS-Optimized ZeroDCE++, (d) are the results of UDM-Optimized ZeroDCE++, (e) are the results of WLS-Optimized SNR and (f) are the results of UDM-Optimized SNR.

Table 2. Ablation studies results of UDM and WLS [3].

The results in Fig. 6, Fig. 7 and Table 2, demonstrate that our framework with UDM outperforms the framework with WLS [3] in terms of optimization for LIE methods. “UDM-optimized” means that the method is optimized by Unrolling LIE optimization framework whose LIE-QE implemented with UDM and “WLS-optimized” means the method is optimized by the framework with LIE-QE implemented with WLS [3]. The experiment results indicate that UDM can extract illumination information from image better and more effectively than WLS [3]. UDM was found to be more effective at extracting illumination information from the image, resulting in better image quality and more natural light and details in the enhanced images, which are more preferable for the human visual system. The ablation studies confirmed that UDM is beneficial to LIE and plays an important role in the effectiveness of our framework in improving the performance of LIE methods.

3.3 Comparison with Related Methods

Since UDM has better decomposition and optimization performance than WLS [3], we selected Unrolling LIE Optimization Framework with UDM as the LIE optimization framework. We applied our framework to optimize four deep-learning LIE methods, namely RetinexNet [22], KinD [26], Zero-DCE++ [12], and SNR [24], on both the LOL [22] and LSLS [16] datasets. We used PSNR and SSIM [21] as the image quality metrics. In the results, “UDM-optimized” means that the method is optimized by the framework whose LIE-QE is implemented with UDM.

Qualitative Evaluation. As demonstrated in Fig. 8 and Fig. 9, the UDM-Optimized enhancement results have more natural light and more light details compared with the original ones. They are more preferable for human visual system than before.

Fig. 8.

Optimization experiment for existing low-light image enhancement methods in LOL dataset [22]. (a) are the low-light images, (b) are the results of RetinexNet [22], (c) are the results of optimized RetinexNet, (d) are the results of KinD [26], (e) are the results of optimized KinD, (f) are the reference images, (g) are the results of Zero-DCE++ [12], (h) are the results of optimized Zere-DCE++, (i) are the results of SNR [24] and (j) are the results of optimized SNR.

Fig. 9.

Optimization experiment for existing low-light image enhancement methods in LSLS dataset [16]. (a) are the low-light images, (b) are the results of RetinexNet [22], (c) are the results of optimized RetinexNet, (d) are the results of KinD [26], (e) are the results of optimized KinD, (f) are the reference images, (g) are the results of Zero-DCE++ [12], (h) are the results of optimized Zere-DCE++, (i) are the results of SNR [24] and (j) are the results of optimized SNR.

Quantitative Evaluation. In Table 3, PSNR and SSIM [21] of the UDM-Optimized images optimized by Unrolling LIE Optimization Framework are higher than the originals, indicating better quality. The enhanced image quality of the four LIE methods is improved after optimization with our framework. The results show that LIE methods optimized by our framework have better enhancement for the low-light images and higher quality results.

Table 3. Optimization results of our Unrolling LIE Optimization Framework.

Moreover, both quantitative and qualitative experiments of the framework demonstrated that our framework is applicable to a variety of deep-learning LIE methods, which indicating the potential of our framework to improve the performance of various LIE methods.

4 Conclusion

We propose a novel Unrolling-based adaptive LIE optimization framework, to address some visual artifacts like underexposure and color shift of state-of-the-art LIE methods, which can result in higher quality enhanced images. Our framework incorporates the output of the Unrolling LIE-QE model into the loss function as a quality constraint during optimization. Our UDM implemented in LIE-QE takes advantage of algorithm unrolling, which unrolls the iteration into a neural network, to decompose structure map and the illumination-detail map that contains the illumination information result in better enhancement from the input image. The experimental results demonstrate that our method can effectively improve and re-optimize the LIE methods to produce higher quality and visually-pleasing results. The results indicate the potential of framework with LIE-QE module to re-optimize various existing DNN-based LIE methods.