Median filters combined with denoising convolutional neural network for Gaussian and impulse noises

Abstract

Elimination of combined Gaussian and impulse noises in digital image processing with preservation of image details and suppression of noise are challenging problem. For this purpose, a new filter which is median filters combined with convolutional neural network for Gaussian and salt & pepper noises. The previous methods are application dependents; some used for impulse noise and other employed only for Gaussian noise. The elimination of Gaussian and impulse noise completed into two steps. First the detection of impulse noise with the rejection of noise by employed of 3 × 3 and 5 × 5 window size median filters. In the second step removal of Gaussian noise performed by residual learning denoising convolutional neural network. It is very favorable and the ability of learning and denoising performance in the field of digital image processing. Denoising convolutional neural network also has active Gaussian noise with an unknown level of noise. Experimental work showed that the proposed method can achieve low loss and root mean square error during training, high peak signal to noise ratio, low mean square error, image quality assessment with good quality and mean absolute error for close prediction between denoised and original color images.

1 Introduction

Digital Image de-noising is an essential active topic during the process of image low-level vision, acquisition and transmission in many practical applications. The aim of image de-noising in image processing is to recover a clean image from noisy. Digital images distorted by several types of noise like; Gaussian noise [21], Fixed value impulse noise, impulse noise, shot noise so on. During transmission in noisy channel, low contrast, deteriorate edges and details between target and background image [5, 11, 13, 15, 18, 19] and nonlinear median filters are also used for denoising in practical life [1, 16, 20], while smoothing impulsive noise [4] and image degradation by Gaussian noise [17]. Our research focus was fixed valued impulse noise called Salt & Pepper noise with Gaussian noise in RGB images. Salt & Pepper noise generates two values 0 and 255, where 0 and 255 belong to black and white noise in each channel of RGB image respectively. While Gaussian noise creates due to poor acquisition illumination and high temperature or transmission. In the past years different models have been developed for image de-noising, including “Beyond a Gaussian Denoising: Residual Learning of DnCNN for Image Denoising (CNN)”[21], “Median Filter (MF)”[17], “Adaptive Centre Weight Median Filter (ACWMF)”[14], “Decision-Based Algorithm (DBA)”[8], “Noise Adaptive Fuzzy Switching Median Filter (NAFSM)”[7], and “An Efficient Edge-Preserving Algorithm for Removal of Salt-and-Pepper Noise (NASEPF)”[2], and Fast and flexible discriminate CNN based Image Denoising [22]. These high denoising quality filters are handy for a single time of noise while suffering from a combination of Salt & Pepper and Gaussian Noises. To remove both noises, we present a model which denoise the image with high Peak Signal to Noise (PSNR) ratio for RGB images [12], low Mean Square Error (MSE) [23], Image quality assessment (SSIM) [3] and model performance measure (MAE) [6]. Remove noise by the median filter is also depends upon the window size of the filter, higher the window size filter has good suppression capability but details (corner, edges) is limited. As for preserve details of the image, the window size must be smaller, but it causes noise suppression problem. While Gaussian noise image has a degradation problem and without computational efficiency, most of the method cannot achieve high accuracy.

It is a standardized measure to remove a single type of noise and has been performed by the previous study. However, the combination of Gaussian with Impulse noises affects these algorithms. In this paper, the median filter with convolutional neural network has been used to analyze and remove combined noises. The approach can improve noise suppression with high quality and efficiently preserve the details of the image. The study presented in this paper has initially used median filter to suppress high-density noises with the noise-free pixels. After that, the convolutional neural network has employed to remove Gaussian noise and obtain clear image. For training the convolutional neural network L2 regularization, constant learning drop and validation used to reduce the loss and root mean square error.

2 Related work

Analyzing of noise removal has been studied for several years with a different approach for a single type of noises. Previous studies have focused on either Gaussian noise or Impulse noise. The IAPR TC-12 and Berkeley Segmentation datasets are used for information retrieval and edge preservation. These approaches have been well designed, and it is illustrated as follows.

2.1 Datasets

There are two datasets IAPR TC-12 and Berkeley Segmentation that are useful for information retrieval in image processing. IAPR TC-12 is used for studying multimedia information retrieval. It consists of 20,000 natural images which have been taken from different locations around the world. These images are including of different photographs of people, sports and actions, animals, cities, and other aspects of life. Furthermore, all images have high quality, multi-objects and color photographs with strict image selection rules. While Berkeley Segmentation dataset is mostly used for edge-preserving. The dataset consists of 1000 Corel images, but 300 images are available for public benchmark. The public dataset is divided into 200 images for training and test set of 100 images.

2.2 Median filter

In the Salt & Pepper noise removal techniques, the most common nonlinear filter is the standard median filter (MF) [13]. The MF replaces the pixel value by the median of the filtering window. If the nearest pixels value taken than P1, P2, P3…Pn is the sequence. So, the mathematical form as following:

$$ MF= median(Pi)=\kern0.5em \frac{\sum \limits_{i=1}^n Pi\left(n+1\right)}{2}, if\ n\ is\ odd\kern5.75em $$

(1)

$$ \mathrm{MF}= median(Pi)=+\sum \limits_{i=1}^n Pi\left(\frac{n}{2}\right)+1, if\ n\ is\ even\kern4.75em $$

(2)

where i=1,2,3... n

2.3 Adaptive center weighted median filter

It is used to obtain optimal weight, efficient scalar quantization and least mean square of the median-based filters as a noise removal [14]. Where x(p), s(p) and n(p) represents input, noise free and noisy pixels.

x(p) = s(p), with probibility of 1 − P if noise free image pixels. Similarly x(p) = n(p), with probibility of P,if noisy pixels image.

2.4 Decision-based algorithm

Decision-Based Algorithm is used to get high quality and quantity of image with preserve edges as a nonlinear filter for high noisy density image. If P (A, B) is not corrupted pixel than Pmin < P(A, B) < Pmax, Pmin > 0 and Pmax<255. Otherwise, P(A, B) is corrupt and replace by median value Pmi, n < Pmed < Pmax and 0 < Pmed < 255. Else replace median Pmen by a neighborhood pixel value, if Pmin < Pmed < Pmax or 255 < Pmed = 0 [8].

2.5 Noise adaptive fuzzy switching median filter

This algorithm has two stages one identifies noisy pixel by histogram and the second stage used for the filter process to remove noise from the image [7].

$$ {\displaystyle \begin{array}{c}P\left(i,j\right)=\sum N\left(i,j\right)\\ {}N\left(i,j\right)=\left\{\begin{array}{c}0,\\ {}1,\end{array}\kern0.66em \begin{array}{c}{L}_{salt}\kern0.33em and\kern0.33em {L}_{pepper}\\ {} otherwise\end{array}\right.\end{array}} $$

(3)

Where P(i, j) is pixel with the intensity of “P” at (i, j) location, N(i, j) = 1 is noise-free pixels and 0 for the noisy pixel.

2.6 Efficient edge-preserving algorithm for removal of salt-and-pepper noise

It is used for impulse noise and salt-and-pepper and preserve edges by employ efficient impulse noise detector and edge-preserving filter. The construction of the algorithm is straightforward and efficient [2].

$$ {\displaystyle \begin{array}{c}{N}_{i,j}=\frac{\sum P\left(i,j\right)}{Total\kern0.33em number\kern0.33em of\kern0.33em pixel s}\\ {}P\left(i,j\right)=\left\{\begin{array}{c} Noisy\kern0.33em pixel,\kern0.33em if\kern0.33em \left({N}_{i,j}=P\max \right)\kern0.33em or\kern0.33em \left({N}_{i,j}=P\min \right)\\ {} Noise\kern0.33em Free\kern0.33em Pixel,\kern0.33em otherwise\end{array}\right.\end{array}} $$

(4)

2.7 De-noising convolutional neural network

The second step of the proposed method is DnCNN which is used as a denoising technique in the field of image processing. It can tackle the different task in image denoising like; Gaussian denoising, JPEG image deblocking, and single image super-resolution. The technique used for image denoising is residual learning which separates noise from the image. Speed up training and quality information achieved by the combination of batch normalization [9] and residual learning integration [21].

$$ B=A+V\kern15.25em $$

(5)

Where A is predicted clean image, V is the noise. The average mean squared error between input A and residual image from noisy input as:

$$ {Error}_x=\frac{1}{2x}\sum \limits_{i=1}^x\left\Vert R\left({B}_i;x\right)-\right({B}_i-{A}_{i\Big)}\mid {\left.\kern0em \right|}^2\kern1.5em $$

(6)

3 Methodology

Today different de-noising techniques are available, but in these algorithms, some use for salt & pepper noises and others use for Gaussian noises. Removing both noises at the same time is tough. For this purpose, we will develop a method for RGB images de-noising, which is typically base on De-noising Convolutional Neural network with median filtering. In this technique, our main goal is the removal of salt & pepper with Gaussian noise. The performance evaluation PSNR, MSSIM, MAE and MSE. There are different algorithms for noise removal, while these algorithms are application dependent. Image processing has two main, and critical stages are detection stage and enhancement stage called image de-noising. Given the technique of our research provides optimize result with 3 × 3 and 5 × 5 window size. The losses of the image details are shallow and achieved better quality with informative image results. The proposed method, the whole procedure, is completing the following steps. First, take an RGB image and divided into separate red, green and blue channels. Then apply Gaussian noise with different variance and Salt & Pepper noise with percentage deviations and each color channel independently noisy. After make image noisy, now time to remove noises with by different algorithms and our proposed method. This de-noising process could divided into two stepwise. Our contribution to the following work as follows. Instead of only Salt & Pepper noise or Gaussian noise removal process separately, our proposed method filter both noises in one operation.

3.1 Improve median filter

Initially, selected an RGB image and extracted into red, green, and blue channels respectively. Find the noise in each channel and ride noise in the channel by replacing with a median. As we use 3 × 3 and 5 × 5 filters size of windows, so nine elements in 3 × 3 window filtering and twenty-five elements in 5 × 5 window filtering. By excluding of central pixel of 3 × 3 and 5 × 5 window, we can calculate the maximum and minima of other pixels. Now apply to switch median filters to detect noisy and noise-free pixels.

a) When the value of the central pixel is equal to 0 or 255 then it considered as a noisy pixel and got a ride by noise adaptive median filter used for suppression of high-density noises to replace with the noise-free pixel. Edge preserving applied for image edges and details, while non-local mean filtering⁴ used for the improvement of noise the suppression and edge details maintenance.

b) If the pixel value of the central pixel is not 0 and neither 255, then it is treated as a noise-free pixel and by switching filter select noise-free pixel. Because Salt & Pepper noises change pixel value to 0 or 255 in each channel of RGB. Where P central pixel with window size generate P points for a noisy pixel to take the average. The P central pixel weighted sum is

$$ {P}_{\left(i,j\right)=}^{\prime}\sum \limits_{i=1}^j{W}_{\left(i,j\right)}.{P}_{\left(i,j\right)}\kern15em $$

(7)

Where ${P}_{\left(i,j\right)}^{\prime }$ is a noise-free pixel and W_(i, j) is a weighted sum of P_i and P_j and weight measure by distance as,

$$ {W}_{\left(i,j\right)}=\frac{d\left({P}_{\left(i,j\right)},{P}_{\left(i,j\right)}^{\prime}\right)}{\sum d\left({P}_{\left(i,j\right)},{P}_{\left(i,j\right)}^{\prime}\right)}\kern13.5em $$

(8)

As we know,

$$ d\left({P}_{\left(i,j\right)},{P}_{\left(i,j\right)}^{\prime}\right)=\sqrt{{\left(P1-{P}^{\prime }1\right)}^2+\dots {\left( Pn-{P}^{\prime }n\right)}^2}\kern2.25em $$

(9)

3.2 Convolutional neural networks

In the previous filtration process by median filtering, we got an image with Gaussian noise and farther de-noising image by De-noising Convolutional Neural Network. DnCNN is very beneficial for Gaussian noise (mostly as a single noisy level image), and due to residual learning and batch normalization, one could get an efficient noise-free image. Therefore, we applied DnCNN again de-noised the image with the following conditions;

1) Latent clean image obtained by end-to-end deep CNN residual learning to remove Gaussian noise. Image with Gaussian noise is y = x + v, which is a noisy observation of DnCNN.

2) Speed and functional performance of noise-free image performed by batch normalization and residual learning during Convolutional Neural Network learning.

3) The network depth D is with (2D + 1) x (2D + 1) and size of the convolutional filters are 3 × 3 with no pooling layers.

$$ \mathrm{B}=\left|\sum \limits_{\mathrm{i}=1}^{\mathrm{q}}\left(\sum \limits_{\mathrm{i}=1}^{\mathrm{q}}{\mathrm{f}}_{\mathrm{i},\mathrm{j}}{\mathrm{d}}_{\mathrm{i},\mathrm{j}}\right)\right|\kern11.5em $$

(10)

Where f_{i, j} is a convolutional window coefficient with position i, j, d_{i, j} is a pixel value convolve with f_{i, j}, q is a window size and B is the output of convolutional.

$$ {f}_{i,j}=\frac{w-{f}_{i-1,j-1}+2p}{s}+1\times \frac{h-{f}_{i-1,j-1}+2p}{s}+1 $$

Where, w and h are width and height. While p and s are padding and stride of the filters. f_{i − 1, j − 1} is the previous window size.

Finally, ridge regression (L2 regularization used which adds square of the magnitude of coefficient as a penalty term to the loss function. L2 regularization avoids the training process from overffitng and improve the accuracy of the model.

$$ Loss= Error\left(A,B\right)+\lambda \sum \limits_{i=1}^N{w}_i^2 $$

A is true value and B is predicted value. $\lambda \sum \limits_{i=1}^N{w}_i$ is a L2 regularization co-efficient which find the right co-efficient through iterative updates with optimizer algorithms. λ should be greater than 0 to avoid facing with least square regression. The flowchart of the proposed method is given in Fig. 1. The input is a noisy image with Gaussian and Impulse noises and followed by windows selections. The channel selection is applied to detect the red, blue and green channels and process it. The conditional section detects the noisy and noise free pixels of the channels. Salt and Pepper noise get ride by replacement of improved median filter. After all, the convolutional neural network is used to remove Gaussian noise from input and give output a noise free image.

Fig. 1

Flow chart of given method

The experimental results show that our given method achieved high performance in denoising process with low computational power and enable image restore with low loss of information and high image quality.

4 Simulation and results

The experimental work of our given method performed by MATLAB R2018a (9.4) software and use Salt & Pepper noise and Gaussian noise for our proposed method as image denoising. While optional parameters for training are epoch = 20, Gaussian Noise Level = 0.01:0.3, mini-batch size = 64, patches per epoch is 64, Patch size = 50, Learn rate drop factor with 0.001 and constant, Channel format = Grey Scale, total iteration = 4520 with 100 and 126 per epoch iteration with L2 regularization and using Adam optimization [10]. We have used DnCNN as pre-trained for IAPR TC-12 Benchmark only 100 images out of which has 20 thousand images with all completed annotations in English, Germany, and Random and 100 images of Berkeley segmentation dataset which included on 300 images with Lenna and onion images. Our technique applied to the “Lenna” RGB digital image, which is very popular in image processing. This image is susceptible to noise and became very useful if we apply Gaussian and Salt & Pepper noises simultaneously. The size of the image is 135x198x3, where 135, 198 and 3 are height, width, and channels respectively. The picture was initially noise-free, and we added Gaussian and Salt & Pepper noises respectively using MATLAB R2018a. The proposed technique used for RGB images from different size and fields and got fewer losses of information and quality image results. We simulated this task at a different system with different configurations and achieved excellent results within a few seconds, but mainly Manufacturer of the system required; Pavilion dv5 Notebook with 4GB RAM and Core ™2 Duo CPU. The performance of the proposed method of simulation is on the base of the high gain of PSNR, Compression Quality Factor, measuring image quality by MSSIM, the absolute value of the error MAE and mean square normalizes error MSE.

$$ \mathrm{PSNRrgb}=10\ \mathrm{x}\ {\log}_{10}\left(\frac{{\left(\max \right)}^2}{MSE}\right)\kern6.75em $$

(11)

$$ \mathrm{MSE}\left(\mathrm{A},\mathrm{B}\right)=\frac{\sum \limits_{\mathrm{i}=1}^{\mathrm{m}}\sum \limits_{\mathrm{j}=1}^{\mathrm{n}}\sum \limits_{\mathrm{k}=1}^{\mathrm{p}}{\left({\mathrm{A}}_{\mathrm{i},\mathrm{j},\mathrm{k}}-{\mathrm{B}}_{\mathrm{i},\mathrm{j},\mathrm{k}}\right)}^2}{\mathrm{m}\times \mathrm{n}\times \mathrm{p}}\kern3em $$

(12)

Where PSNRc is a peak signal to noise ratio between the original image and restored the image of RGB images and MSE is a mean square error. K and L are lengths, width, and channels of the image respectively. The maximum (max) value for each channel is 255. A_{i, j} and B_{i, j} are components of original and filtered image respectively.

$$ SSIM\left(A,B\right)=\frac{\left(2{\mu}_A{\mu}_B+{K}_1\right)\left(2{\sigma}_{AB}+{C}_2\right)}{\left({\mu}_A^2+{\mu}_B^2+{C}_1\right)\Big({\sigma}_A^2+{\sigma}_B^2+{C}_{2\Big)}}\kern3.25em $$

(13)

Where C₁, C₂ are luminance, constant and structural terms for regularization and μ_A, μ_B, σ_A, σ_B, σ_AB are cross covariance, local means and standard, deviations for images A and B. Dynamic range for the image is 255 and constant in the structural similarity index is 0.01 to 0.03.

$$ MAE=\frac{1}{N}{\sum}_{i=1}^N\mid {A}_{uv}-{B}_{uv}\mid \kern11.75em $$

(14)

Where N = number of errors and u, v is rows and columns summations. In last we show a different variation of Gaussian and impulse noises of our proposed method between different median filters and DnCNN and median filters algorithms combined with DnCNN.

In Table 1 we can see that the performance of median filters for Salt & pepper noise are higher PSNR with 3 × 3 and 5 × 5 filters compare to DnCNN. We compare these median filters and DnCNN by PSNR, MSSIM, MAE and MSE for “Lenna” image.

Table 1 3 × 3 and 5 × 5 filtering for Salt & Paper Noise with different parameters

We can see that results of “Lenna” images clearly shows Salt & Pepper noise difference in median filters and DnCNN.

Table 2 shows that DnCNN has higher PSNR for Gaussian noise, while median filters effect by it.

Table 2 DnCNN, FFDNet and filtering for Gaussian Noise with different parameters and noise level

The compression quality factor uses for image loss during compression with an integer value from 1 to 100. The image quality at different compression level shows that image degraded by noises and using previous methods still has a low-quality image. Which clearly shows that these methods are effect by combined Gaussian and Impulse Noises. While we applied our proposed method and achieved a good quality image with compression and near to original image quality. In Fig. 2, graph “a” include with previous methods and the proposed method for same noise value with 30%, which clearly shows that our proposed method has very loss and approximately equal to the original image. While graph “b” used for different noises, which proved that our proposed method has better performance and low loss with high noise ratio.

Fig. 2

Image Quality at different compression levels for previous vs proposed method. Figure 2a include on proposed method and previous methods, while 2b has only proposed method with different noise ratio

The results of the proposed method in Table 3 shows that the performance of median filters with DnCNN is satisfied with higher PSNR and other parameters. These tables show the 3 × 3, and 5 × 5 median filters combination with DnCNN achieved excellent results for both Gaussian and Salt & Pepper noises.

Table 3 Proposed method (PM) for both Gaussian and Salt & Pepper Noises (3 × 3 and 5 × 5)

By comparative evaluation of images with separate denoising process by previous methods are the effect if both noises given in image. Our proposed method can achieve low losses clearly show that if both Gaussian and Salt & Pepper noises are given in image than we can get a good result.

The advantage of the proposed method is to use L2 regularization, constant learning drop and validation which reduced the loss and root mean square error during training. Without these parameters, the loss was high and effected the performance of the algorithm for practical use.

Table 4 magnify the training accuracy with different parameters for IARP_TC and Berkley datasets. A series of experiments have been conducted on IARP_TC and Berkley datasets using advanced techniques of convolutional neural networks for optimization and regularization of hyper-parameters. The training graph in Fig. 4 shows that if L2 regularization used with constant learning drop than the RMSE and Loss will be very low.

Table 4 RMSE and LOSS of Datasets

Figure 3 shows the difference between pre-processing of the image from 3(a) to 3(i) respectively. While the proposed method result is 3(j) clearly de-noised with low losses mean higher image quality and very less information losses. The results show that MF, ACWMF, DBA, NAFSM, NASEPF, DnCNN and FFDnet cannot filter properly if both Gaussian and Salt & Pepper noises except proposed method. The disadvantage of the proposed method is the low computational power during training, which taken days for training (Fig. 4).

Fig. 3

Denoising Results on Lenna with comparison of previous and proposed methods

Fig. 4

Training graph for the dataset with 20 epochs

5 Conclusion

The Proposed technique proves to get high quality factor, good PSNR, MSSIM, MAE, and MSE for both Gaussian and Salt & Pepper noises in the image instead of previous methods separately. Practically noises are in multiple forms not in one shape, and it is very efficient to remove both noises from the image. Our mean purpose was to develop a method for RGB images, which detect Gaussian and Salt & Pepper noises and remove it. It gives high PSNR, low MSE, SSIM with perceived quality and MAE for close prediction between original and fixed image. The proposed method has the following benefit: This method has a primary advantage on other filter is to remove various noises, while other filters were used for a specific noise. Second, it has tested on Berkeley and IARP TC Datasets, onion image and Lenna Image, which is very popular for noises and easily damage by various noises, while we tested it on another image to get quantity results.

Finally, our motivation about design method “Median Filters combined with Denoising Convolution Neural Network,” and Tables 1, 2 and 3 show that our algorithm is the capability of removing Gaussian and Salt & Pepper noises for susceptible noise affected images.