Keywords

1 Introduction

As an emerging field, automatic drive has vast potential for future development, so one of the important component in autonomous driving, image identification, get people’s attention. Compared to other methods, the identification based on visible light camera is a cheaper and more deployable way.

Other the other hand, this method also comes with its issue. For example, the result will heavily affected by lighting conditions. If there is a hash condition, which can cause blurry object shape or inaccurate color, the detection accuracy will significantly drop. We test some common object recognition algorithms, faster-RCNN [11] and Yolo [8]. They are designed for general scenario, and not are optimized for the light condition. So, when the system is operated under undesired lighting conditions, In the actual usage scenario, at noon or in the late night, these influence may result accidents.

One of the most discerning autonomous driving systems in the industry, Tesla’s Autopilot system [9] is a vision-based Autopilot system, which mainly relies on camera sensors to obtain surrounding information. But this system still experiences accidents under poor lighting conditions. In 2016, a Tesla didn’t identify a front lorry, eventually cause a rear-end collision. Other Autopilot system, like Google’s Waymo [10], Baidu’s Apollo Go [3], and so on, they all have same problem.

To address the problem of accuracy faced by autopilot in complex lighting conditions, and does not affect the recognition ability under normal brightness, we develop a algorithm. In this paper, after introducing the related works, we will present our function into two parts, Firstly, we will introduce light-selection. We use the gray processing of the image, and the anomaly image is recognized and classified. Second we will present an image restoration technique based on MSRCP [7] and how it’s applied in the autonomous driving scenario. It can enhance the performance of target detection under abnormal lighting conditions by considering the lighting conditions and preserving the color information of the image. This method adjusts the brightness and color distribution of the image to enhance the edges and contours of the target, making it easier to detect under abnormal lighting conditions. Next, in the Experiment and Result section, we test the algorithm, 3000 images with abnormal brightness were selected from bdd-100k [13] as the data set of this project. During the test, we use the same model to identify image to find out the progress of our algorithm. The results showed that our method improves the total mAP@[.5, .95] from 15.22% to 19.57%, this algorithm can also used on yolov3 or other version. It can improve the recognition results of abnormal brightness images.

2 Related Work

2.1 Image Enhancement Algorithm Based on Retinex

The image enhancement algorithm is based on Retinex theory [5, 6]. The term “Retinex” stands for “Retina with Cortex” it simulates the biological processes of human’s retina and brain cortex, and it suggests that the image I observed by human eyes is the product of light image L and object reflection image R.

$$\begin{aligned} I(x, y)=L(x, y)^* R(x, y) \end{aligned}$$
(1)

Take log for both side:

$$\begin{aligned} \log [R(x, y)]=\log [I(x, y)]-\log [L(x, y)] \end{aligned}$$
(2)

We can get a reflection property image that preserves the essence of the object from the received image I and the light image L. This method is called “Single Scale Retinex (SSR)” [7, 14], and its workflow is shown as follows:

  1. 1

    Take log for I(x,y)

  2. 2

    Use Gaussian blur for l(x,y), the result is \(\log [L(x, y)]\)

  3. 3

    Calculate \(\log [R(x, y)]\) by \(\log [I(x, y)]\) and \(\log [L(x, y)]\)

  4. 4

    Transfer the \(\log [R(x, y)]\) to RGB image, and this is the result.

Further improved can be achieved by, for example, Muti-Scale Retinex(MSR). When calculating the value of \(\log [R(x, y)]\), the algorithm performed multiple SSR operations and weighted the calculation. So compare with SSR, this algorithm has better performance on color.

The aforementioned will still face some issues, such as, color cast, which can be mitigated by applying Muti-Scale Retinex with Color Restoration(MSRCR). It takes into account multiple scales or spatial frequencies of the image for more comprehensive enhancement. The algorithm adds multiple parameters to produce color recovery factor C, adjusts the ratio of the 3 channels of the image, reduces the color cast.

At the same time, the MSRCP [1] algorithm was proposed, which carries out an MSR processing on the light intensity image of the original color image to map to each channel in the same proportion as the original image RGB channel. This algorithm can preserve the color distribution of the original image well.

2.2 Object Detection Algorithm Based on YOLOv4 Model

In this paper, YOLO-V4 [2, 12] is used as the object detection network. The YOLO network is a single-stage algorithm. Be careful about the comparison between YOLO and two-stage algorithms, such as Fast RCNN [4]. YOLO has been improved a lot over the past a few years. But Fast RCNN and some other algorithms haven’t changed since they were published. Various methods are also used to improve accuracy.

The yolo-v4 used in this paper has been optimized in many aspects, making the accuracy significantly improved. The data processing aspect includes mainly Mosaic data enhancement, which can enrich the background of the image and increase the batch size in a different way. Backbone includes the new CSPDarknet53, which replaced ReLu by Mish as the activation function, and the CSP structure prevents different layers from learning the same gradient information. In addition, the loss function has been updated, and the prediction box filtering has been improved. Through this series of optimization, it finally makes the real-time target detection algorithm in the highest accuracy.

3 Method

In this paper, brightness selection algorithm and image restoration algorithm based on MSRCP are used, and then object detection algorithm based on Yolo-v4 is adopted to identify the target. The algorithm flow in this paper is shown in the Fig. 1.

Fig. 1.
figure 1

The Algorithm Worklow

Full size image

After the input image goes through the brightness selection algorithm, the color information of the image with abnormal brightness will be restored by the image recovery algorithm. Finally, the image recognition will be carried out in the model trained by yolo-v4 and the results will be obtained.

3.1 Brightness Selection Algorithm

Firstly, the brightness selection algorithm is introduced. Through this algorithm, we screen out the images with abnormal brightness. Since brightness equals gray level in gray level image, we can judge brightness by converting the image into gray level image. We first calculate the difference between the gray value pg and 128 of each pixel point:

$$\begin{aligned} g a=\sum (p g-128) / N \end{aligned}$$
(3)

Then the gray-scale histogram is used to obtain the number of pixels of each gray value between 0 and 255, and the weighted average deviation is calculated with the number of pixels as the weight.

$$\begin{aligned} M a=\sum |(p g-128)-g a|^* \textrm{G}[\textrm{i}] / \sum \textrm{G}[\textrm{i}] \end{aligned}$$
(4)

Finally, we calculated according to the average deviation to judge whether the image brightness is unusual, through access to online data, we set threshold is |ga|.

3.2 Image Enhancement Algorithm Based on MSRCP

After brightness selection, we need to process the abnormal brightness image. Here we compare several algorithms and finally choose MSRCP. Compared with MSRCR and MSR, MSRCP can increase the highest recognition rate (Fig. 2).

Fig. 2.
figure 2

Result of Different Enhancement Algorithm

Full size image

In the process of brightness selection, we evaluate the gray image, and then we can compute the intensity image I according to the image component of the RGB three-channel of the original image.

$$\begin{aligned} I=\left( I_R+I_G+I_B\right) \end{aligned}$$
(5)

In order to make the color of the enhanced image consistent with that of the original image, the amplification factor A is obtained through the intensity image of the single channel:

$$\begin{aligned} B={\text {Max}}\left( I_R+I_G+I_B\right) \end{aligned}$$
(6)
$$\begin{aligned} A={\text {Min}}\left( \frac{255}{B}, \frac{I_{n t}}{I_{n t i}}\right) \end{aligned}$$
(7)

In the equation, I is the pixel of the image. Color enhancement is carried out on RGB channels respectively, and the final result is obtained (Fig. 3):

$$\begin{aligned} I_E=A * I(R, G, B)(x, y) \end{aligned}$$
(8)
Fig. 3.
figure 3

Image Enhancement Algorithm for Abnormal Light Image

Full size image

4 Experiment and Result

4.1 Experimental Setting

Dataset. The BDD100K dataset is used in this work to evaluate the proposed algorithm. This dataset was created by Berkeley AI Laboratory, we select 3875 random images in it, including images of different Harsh light environment. The label distribution of the data set is shown in Fig. 4:

Fig. 4.
figure 4

Image Enhancement Algorithm for Abnormal Light Image

Full size image

Evaluation Step. Experiment Setup

  • * Operating system: Ubuntu 18.04

  • * Python version: Python3.8

  • * PyTorch version: Pytorch 1.10

  • * GPU model: NVIDIA GeForce RTX 2080Ti

4.2 Experiment Result

After producing the data set, we trained it for 1000 generations by yolo-v4 and yolo-v3. The learning rate was adjusted by cosine annealing with the interval of 0.01–0.0001, and SGD was used for optimization. In terms of data enhancement, mosaic enhancement is used in the first 70% generation of training, and the probability of using it in each training step is 50%. After Mosaic is used, mixup data enhancement will continue to be used, and the probability of using it is also 50%. The final training results are as follows (Fig. 5):

Fig. 5.
figure 5

Loss for Train Process

Full size image

After training, we test for the MSRCP image recovery algorithm. We selected 746 pictures containing different elements as the test set of the experiment. And we tested them with and without the algorithm. The results are shown in the below (Fig. 6):

Fig. 6.
figure 6

Image Enhancement Algorithm for Abnormal Light Image

Full size image

As the result, the recognition ability after the processed image has been significantly improved. The overall mAP increased about 10%, and traffic lights in particular have improved by 14% and 17%, respectively. Less improved object categories include people and car. We believe that the unidentified car targets are mainly distant targets and obscured targets. When these objects appear in under dark conditions, more color blocks will be generated after our algorithm is restored, which will cause greater interference, and the same is true for person. And traffic lights themselves emit light, so they’re friendly to our algorithm. Totally, the whole recognition rate of image has been improved obviously.

5 Conclusion

In this paper, to mitigate the problem of abnormal image brightness in automatic driving, brightness selection algorithm and image restoration algorithm based on MSRCP are used to classify images and carry out color restoration. Combined with yolo-v4 target recognition, common targets in automatic driving scenes are identified, which ultimately effectively improves the image recognition rate and provides guarantee for safe driving. On the other hand, there are still issues need further investigation. First, images recovered using MSRCP will blur the image and in some cases even render the object that could have been recognized unrecognizable. Second, in the test of this paper, there are still some kinds of targets that cannot be confirm to improve by the insufficient number of the test set. Later, we will increase the size of the test set and conduct further tests on trucks, bus. We will also increase the recognition of different types of objects to make it closer to real driving scenarios. In the future, we will optimize the algorithm and integrate more advanced image recognition methods, such as YOLOv7, to attempt real-time recognition in low-light conditions.