A modified non-local means using bilateral thresholding for image denoising

Abstract

In non-local means (NLM) method, weight of each pixel in the neighbor centered on the reference noisy pixel plays a different role for performance of NLM for image denoising. The reference noisy pixel is called center pixel. Usually, weight of each pixel including center pixel in the neighbor (neighbor pixel) is computed based on the distance between neighbor pixel and center pixel. This paper proposes a novel weight by studying the recent statistical nearest neighbors distance measurement (SNNDM) and gradient domain filter. The difference of each neighbor pixel including center pixel is considered sufficiently. The proposed weight is called the bilateral thresholding since it is similar to bilateral filtering in form. Test results show that the proposed method can deal with each neighbor pixel differently so that the desired performance is achieved.

1 Introduction

In the Multimedia era, denoising becomes a key challenge in the field of image processing [11]. A good denoised image can pave a way to many subsequent processing tasks such as image segmentation, object recognition and so on. For example, in [12], image denoising method was used for image recognition. The goal of image denoising is to reconstruct the latent clean image from its noisy counterpart while preserving the fine structures, textures and details as accurately as possible. Currently, a wide variety of strategies including total variation [10, 19] have been proposed to remove the noise from the degraded image. Especially, non-local means (NLM) proposed by Buades et al. in 2005 [1] has always been popular. The denoising mechanism of NLM is: denoising of a given pixel is obtained as a weighted average of the surrounding pixels, with weights proportional to the patch similarity measurement. In NLM, patch similarity is called non-local self-similarity (NLSS). The emergence of NLSS has triggered a revolutionary change in the field of image processing. A number of the state-of-the-art methods armed with NLSS have emerged. For example, BM3D was proposed in [2]. This method obtains good results by grouping of similar image and specially developed collaborative Wiener filtering. BM3D is a milestone work employing a NLSS for grayscale image denoising. Another celebrated evolved version of NLM is the weighted nuclear norm minimization (WNNM) method in [8]. The method represented NLSS by using the weighted nuclear norm of similar patch matrices. Compared to BM3D, WNNM achieves better results with large performance gain. To better restore the structure and the edge of image contaminated by noise, a novel multi-scale weighted group sparse coding model (MS-WGSC) was proposed for image denoising by simultaneously exploiting the NLSS prior from the original scale and a coarse scale in [17]. In the recent years, NLSS is also incorporated into the deep learning architecture so that more powerful denoising performance is shown. For example, in [13, 24], the NLSS denoising algorithms based on deep learning techniques were used to successfully deal with the different noisy image. Although these methods have achieved good performance, they are becoming increasingly complex. So, some scholars have been paying attention to NLM itself, optimizing and improving the original NLM. This is also the motivation for the proposed method.

1.1 Related work

Currently, there are also some important studies in which original NLM filter is discussed deeply. In these researches, the weighted average mechanism of NLM filter is further revealed and optimized. These studies have to some extent overcome the shortcomings of NLM, such as high computational complexity, difficulty in determining parameter values, and unreasonable center weight values. For example, the literature [7, 22] studied how to accelerate NLM to make it useful in practical application. The literature [16, 18, 21, 26,27,28,29, 31] and [3, 9] concentrated their attention on how to determine the smoothing parameter, how to compute the weight of the center pixel (center weight) and how to select the weight function, respectively. These studies further improve NLM from different perspectives. This paper focuses on the distance between the pixels so that the proposed weight computes the similarity between the pixels properly.

In [5], Alessandro Foi et al. installed the foveated self-similarity into NLM (NLM-Fov). Their studies show that the foveation operators producing the best denoising performance are in complete line with the directional preference of the human visual system. In [4], Vadim Fedorov et al. exploited the affine invariant self-similarities to design a new NLM filter. The similarity measure compares patches related by a local affinity and considers shape adaptive patches so that a better denoised result is provided. In [6], the statistical nearest neighbors distance measurement (SNNDM) was introduced into the NLM filter (NLM-SNN) for measuring the distance between two patches so that an improvement is obtained in the perceived image quality. In [14], the pixel-pixel unbiased distance, patch-patch unbiased distance and coupled unbiased distance were used to construct the unbiased distance based non-local adaptive means. The tests in [14] show that a better performance than the state-of-the art denoising methods can be achieved by this method. All these methods tap the potential of different similarity measurement ways from different views. But, the methods in [4, 5] have a little complex. The method in [14] combines a few unbiased distance so that there is more complex. In contrast, the method in [6] is simple and effective. In addition, in [5, 14], NLM is carried out with the pixel wise scheme. In [4, 6], the patch wise scheme of NLM is implemented. By comparison, the pixel wise implementation is simpler and more convenient. So, starting from the SNNDM proposed in [6], this paper will propose a novel distance similarity measurement method for the pixel wise NLM by a simple design.

It is noted that the proposed method is different from the recent closely related pixel wise NLM methods [27,28,29]. In [27], the center weight (CW) was researched carefully and then a wiener filter center weight (WFCW) was proposed. This NLM method is called NLM-WFCW. In [28], a multidirectional gradient domain nonlinear diffusion image denoising scheme (MGDID) was proposed by introducing the remedy reactor of the step size from the view of NLM. In essence, MGDID presented a novel center weight in iterative implementation of NLM. In [29], a two-step iterative NLM scheme (TSNLM) was proposed by considering the role of the different center weights. The disadvantage of these NLM methods is that they only consider the role of center weight. Furthermore, more NLM filters are used to design MGDID in [28] and TSNLM in [29]. This increases the complexity of algorithm design and computation. The proposed method is like the original NLM in [1] and WFCW in [27], using only one NLM. The novelty of the proposed method lies in designing a new similarity measurement method that distinguishes and calculates the weights of different pixels.

The main contribution of the proposed method is in: (1) the statistical nearest neighbors distance measurement (SNNDM) is analyzed carefully; (2) to sufficiently consider the role of different neighbor pixel including the center pixel, the geometric distance is introduced into the computations of the weights of all neighbor pixels including the center pixel around the reference noisy pixel; (3) the relation between the parameters is discussed deeply; and (4) the tests show that the proposed method is effective by providing desired peak signal-to-ratio (PSNR), structural similarity (SSIM) and good visual effect. Furthermore, the proposed method is very simple.

The paper is organized as follows. In Section 2, one provides a detailed description of the proposed method. Then, in Section 3, the experimental results and comparisons are presented to demonstrate the effectiveness of the proposed method. Finally, one concludes this paper in Section 4.

2 The proposed method

In this section, the several aspects of the proposed method are detailed in turn. NLM filter is firstly reviewed. Then, the statistical nearest neighbors distance measurement (SNNDM) is introduced and analyzed. After that, bilateral thresholding is proposed to compute the weight of different pixel. At last, the parameters are discussed carefully.

2.1 Non-local means filter

For any pixel at (x, y) in the image domain Ω, the observation model has the following formulation:

$$I\left(x,y\right)=s\left(x,y\right)+\eta \left(x,y\right)$$

(1)

where I(x, y) denotes noisy image, s(x, y)represents original noise free image, and η(x, y) is independent and the zero-mean white Gaussian noise of variance σ². For any pixel of interest (x, y), an estimate u(x, y) with the NLM filter is given by:

$$u\left(x,y\right)=\frac{\sum\limits_{\left({x}^{\prime },{y}^{\prime}\right)\in \textrm{R}}I\left({x}^{\prime },{y}^{\prime}\right)\omega \left(x,y,{x}^{\prime },{y}^{\prime}\right)}{\sum\limits_{\left({x}^{\prime },{y}^{\prime}\right)\in \textrm{R}}\omega \left(x,y,{x}^{\prime },{y}^{\prime}\right)}$$

(2)

where R is a search window around (x, y) and $\omega\left(x,y,{x}^{\prime },{y}^{\prime}\right)$ is the weights that compare the neighborhood patches P(x, y) and $P\left(x^\prime,y^\prime\right)$ around (x, y) and $\left(x^\prime,y^\prime\right)$, respectively. The original weights are defined as:

$$\omega \left(x,y,{x}^{\prime },{y}^{\prime}\right)=\exp \left(-\frac{{\left\Vert I\left(P\left(x,y\right)\right)-I\left(P\right({x}^{\prime },{y}^{\prime}\left)\right)\right\Vert}_{2,\rho}^2}{h^2}\right)$$

(3)

where $h>0$ is the smoothing parameter controlling the filtering degree. $\left\|I\left(P\left(x,y\right)\right)-I\left(P\left(x^\prime,y^\prime\right)\right)\right\|_{2,\rho}^2$ is a weighted Euclidean distance between two image patches $P\left(x,y\right)$ and $P\left({x{^\prime}},\;y{^\prime}\right)$ of size $\left(2L_p+1\right)\times\left(2L_p+1\right)\left(L_p\geq0\;is\;a\;integer\right)$ defined as

$${d}_G^2\left(x,y,{x}^{\prime },{y}^{\prime}\right)={\left\Vert I\left(P\right(x,y\left)\right)-I\left(P\right({x}^{\prime },{y}^{\prime}\left)\right)\right\Vert}_{2,\rho}^2=\sum_{i=-{L}_p}^{L_p}\sum_{j=-{L}_p}^{L_p}{G}_{\rho}\left(i,j\right){\left(I\left({x}^{\prime }+i,{y}^{\prime }+j\right)-I\left(x+i,y+j\right)\right)}^2$$

(4)

where ${G}_{\rho}\left(i,j\right)=\frac{1}{\sqrt{2\pi}\rho}\mathit{\exp}\left(\frac{i^2+{j}^2}{2{\rho}^2}\right)$ is the Gaussian kernel with the smoothing parameter ρ > 0, and the (i, j) varies in the similar window $\mathrm R^\prime$. Some alternative strategies have been proposed to measure the distance between two different pixels to obtain the high image processing quality [4,5,6, 14]. In term of (3), the computation of the distance between the same two pixels I(x, y) will produce the center weight (CW) ω(x, y, x, y), and CW is always 1. This is unreasonable in practice. So, the CW problem has been discussed in [16, 18, 26,27,28,29] and some more effective CWs have been presented. In this paper, the proposed weight includes the computation of CW and all weights of other neighboring pixels. It is noted that (2) is expression of pixel wise NLM filter. (2) is turned into patch wise NLM filter when $I\left(x^\prime,y^\prime\right)$ is replaced with $I\left(P\left(x^\prime,y^\prime\right)\right)$. At this time, apart from the aggregation step, the estimate u(P(x, y)) of the noise-free patch of noisy patch is obtained. In the patch wise implementations of NLM filter, the weight of the patch $I\left(P\left(x^\prime,y^\prime\right)\right)$ is computed as $\omega\left(x,y,{x}^{\prime },{y}^{\prime}\right)$ in term of the pixel wise implementation. That is that computation of weight of pixel $I\left(x^\prime,y^\prime\right)$ in the pixel wise NLM is used as computation of weight of patch $I\left(P\left(x^\prime,y^\prime\right)\right)$ in the patch wise NLM.

2.2 Statistical nearest neighbors distance measurement (SNNDM)

If the reference patch P(x, y) and its neighbor $P\left(x^\prime,y^\prime\right)$ are two noisy replicas of the same clean patch, one gets the expectation of the distance.

$$E\left[{d}^2\left(x,y,{x}^{\prime },{y}^{\prime}\right)\right]=2{\sigma}^2$$

(5)

where.

$${d}^2\left(x,y,{x}^{\prime },{y}^{\prime}\right)={\left\Vert I\left(P\right(x,y\left)\right)-I\left(P\right({x}^{\prime },{y}^{\prime}\left)\right)\right\Vert}_2^2=\frac{1}{{\left(2{L}_p+1\right)}^2}\sum_{i=-{L}_p}^{L_p}\sum_{j=-{L}_p}^{L_p}{\left(I\left({x}^{\prime }+i,{y}^{\prime }+j\right)-I\left(x+i,y+j\right)\right)}^2$$

(6)

The above (6) is called the nearest distance. From (5), one can draw a conclusion that the expected distance is not zero for two noisy replicas of the same patch. Therefore, for patch wise NLM filter, unlike traditional distance measure method between two patches, the statistical nearest neighbors distance measurement (SNNDM) is adopted in [6]. According to SNNDM, one computes the similarity between patches whose distance from the reference patch is instead closed to its expectation. The SNNDM between two patches $P\left(x^\prime,y^\prime\right)$ and P(x, y) is defined as:

$${d}_{SNNDM}^2\left(x,y,{x}^{\prime },{y}^{\prime}\right)=\max \left({d}^2\left(x,y,{x}^{\prime },{y}^{\prime}\right)-2{\sigma}^2,0\right)$$

(7)

In (7), the $P\left(x',\;y'\right)$ must satisfy:

$$d_{SNN}^2\left(x,y,x',y'\right)=\left|d^2\left(x,y,,x',y'\right)-2\widetilde o\sigma^2\right|<\tau$$

(8)

where $\overset{\sim }{o}$ is 1 for the SNNDM approach. Here, τ ≥ 0is a threshold on $P\left(x^\prime,y^\prime\right)$. τensures that the enough similar patches can be selected for every P(x, y). The traditional Euclidean nearest neighbors distance measurement is a special case of (7) with ${d}_{SNN}^2\left(x,y,{x}^{\prime },{y}^{\prime}\right)$ when $\overset{\sim }{o}=0$. In [6], the SNNDM is used to improve the patch wise NLM filter. Compared to the nearest neighbors distance measurement (NNDM) NLM filter, the better denoising effect is achieved under the case that the fewer neighbor patches in search window are used. The details can be found in [6]. Inspired by [6], this paper will develop the weight function based on the distance measurement in the pixel wise NLM filter (2).

2.3 Bilateral thresholding

The proposed bilateral thresholding weight function is produced from three perspectives. The rationality of SNNDM will be incorporated into the new design of the weight function for the pixel wise NLM.

As mentioned earlier, SNNDM is produced by discussing two noisy replicas of the same clean patch. On the contrary case, NNDM is proper for two noisy image patches containing the same noise patch. At this time, ${d}^2\left(x,y,{x}^{\prime },{y}^{\prime}\right)$ will embody the ideal distance between P(x, y) and , but SNNDM will bring bigger distance error because some corresponding patches may have been abandoned. For example, for the case that the ${d}_{SNN}^2\left(x,y,{x}^{\prime },{y}^{\prime}\right)>\tau$ but also ${d}^2\left(x,y,{x}^{\prime },{y}^{\prime}\right)<-\tau +\overset{\sim }{o}\cdot 2{\sigma}^2$, one should pay more attention since as mentioned above that the pixel with smaller ${d}^2\left(x,y,{x}^{\prime },{y}^{\prime}\right)$ maybe has more contributions for NLM filter. In fact, the weight of the unselected (abandoned) patch is zero for SNNDM and NNDM for the patch wise NLM filter. This irrationality should not appear in the proposed pixel wise NLM filter. All neighborhood pixels should contribute to denoising. The roles of different pixels should be reflected by different weights. This is the fist perspective producing the proposed weight.

In fact, after a series of manipulations, the non-local scheme (2) can be re expressed as:

$$u\left(x,y\right)=I\left(x,y\right)+\frac{1}{C\left(x,y\right)}\sum_{\left({x}^{\prime },{y}^{\prime}\right)\in N\left(x,y\right)}{e}^{-\frac{{\left\Vert I\left(P\left(x,y\right)\right)-I\left(P\right({x}^{\prime },{y}^{\prime}\left)\right)\right\Vert}_{2,\rho}^2}{h^2}}\left(I\left({x}^{\prime },{y}^{\prime}\right)-I\left(x,y\right)\right)$$

(9)

where $C\left(x,y\right)=\sum_{\left({x}^{\prime },{y}^{\prime}\right)\in \textrm{R}}\omega \left(x,y,{x}^{\prime },{y}^{\prime}\right)$. It can be seen that (9) can be considered as one step multidirection gradient domain filter scheme. In (9), the information of multiple directions is used. The multidirection window is shown in Fig. 1. The multidirectional gradient domain image denoising (MGDID) presented the good denoised results [28]. Let represents the differences of different distances in the different direction (Δx, Δy) centered at (x, y) in the “square” search window, respectively. ∇I^(Δx, Δy)(x, y) is called gradient coefficient at (x, y) with directions (Δx, Δy). In [30], one implemented the local adaptive Wiener filter in the gradient domain because the gradient coefficients of image can be modeled as zero-mean Gaussian random variables with high local correlation. The gradient coefficient variance is estimated by using an approximate maximum likelihood (ML). So, the “signal” variance of the gradient coefficient of the latent clean image is expressed as:

$$\nabla {I}^{\left(\varDelta x,\varDelta y\right)}\left(x,y\right)=I\left(x+\varDelta x,y+\varDelta y\right)-I\left(x,y\right)=I\left({x}^{\prime },{y}^{\prime}\right)-I\left(x,y\right)$$

(10)

$${d}_{SVDM}^2\left(x,y,{x}^{\prime },{y}^{\prime}\right)=\max \left(q\left(x,y\right)-2o{\sigma}^2,0\right)$$

(11)

where ο ≥ 0, and the q(x, y) in the size (2L_p + 1) × (2L_p + 1) local window is computed as $d^2\left(x,y,x^\prime,y^\prime\right)$ by (6). Here, (11) is called the “signal” variance distance measurement (SVDM). From NLM filtering (2) and gradient domain filtering (9), one can draw a conclusion ${d}_{\boldsymbol{SVDM}}^2\left(x,y,{x}^{\prime },{y}^{\prime}\right)$ is another statistical distance measurement. The ${d}_{\boldsymbol{SVDM}}^2\left(x,y,{x}^{\prime },{y}^{\prime}\right)$ is zero when q(x, y) ≤ 2οσ² for patch $P\left(x^\prime,y^\prime\right)$ and P(x, y). But, in SNNDM, ${d}_{\boldsymbol{SNNDM}}^2\left(x,y,{x}^{\prime },{y}^{\prime}\right)$ is considered infinity when ${d}_{SNN}^2\left(x,y,{x}^{\prime },{y}^{\prime}\right)=\left|{d}^2\left(x,y,{x}^{\prime },{y}^{\prime}\right)-2\overset{\sim }{o}{\sigma}^2\right|>\tau$ for patch $P\left(x^\prime,y^\prime\right)$ and P(x, y). This means that the patch (pixel) with ${d}_{SNN}^2\left(x,y,{x}^{\prime },{y}^{\prime}\right)>\tau$ will be abandoned in the process of denoising .

Fig. 1

The “square” search window

The difference with SNNDM (7) and SVDM (11) lies mainly in that the two measurements have different expressions for ${d}_{SNN}^2\left(x,y,{x}^{\prime },{y}^{\prime}\right)>\tau$. In fact, both two different expressions have some disadvantages. For SNNDM, the disadvantage is that the unselected patch (pixel) is not considered as mentioned earlier. For SVDM, the disadvantage is also obvious. That is that the distance between two clean patches should all not be always set to zero when q(x, y) ≤ 2οσ². In practice, two clean patches have difference on average. This is the second perspective producing the proposed weight.

The above analysis gave birth to a new weight based on the distance measurement. It is noted that if $\omega\left(x,y,x^\prime,y^\prime\right)$ in (2) is turned into:

$${\omega}_{bi}\left(x,y,{x}^{\prime },{y}^{\prime}\right)=\exp \left(-\frac{{\left(x-{x}^{\prime}\right)}^2+{\left(y-{y}^{\prime}\right)}^2}{h_s^2}\right)\exp \left(-\frac{{\left(I\left(x,y\right)-I\left({x}^{\prime },{y}^{\prime}\right)\right)}^2}{h^2}\right)$$

(12)

and then NLM filter is changed into bilateral filtering in [20]. In (12), ${h}_s^2$ and h² are two different smoothing parameters used in spatial and range weights. h² in bilateral filtering plays the same role as that in NLM filter. Compared with bilateral filtering (12), NLM only computes the range weight. In addition, the range distance between two pixels takes into account two image patches instead of two single pixels. In NLM filter, the spatial distance is discarded. In the proposed method, the role of the spatial distance will be brought into full play. This is the third perspective producing the proposed weight. The pixel close to the center pixel in the spatial distance will have greater weight, and which has achieve success in bilateral filtering.

The proposed weight function adopts the following form:

$${\omega}_{pr}\left(x,y,{x}^{\prime },{y}^{\prime}\right)=\underset{\phi_s\left(x,y,{x}^{\prime },{y}^{\prime}\right)}{\underbrace{\exp \left(-\frac{{\left(x-{x}^{\prime}\right)}^2+{\left(y-{y}^{\prime}\right)}^2}{h_s^2}\right)}}\underset{\phi_r\left(x,y,{x}^{\prime },{y}^{\prime}\right)}{\underbrace{\exp \left(-\frac{d_{pr}^2\left(x,y,{x}^{\prime },{y}^{\prime}\right)}{h^2}\right)}}$$

(13)

where.

$${d}_{pr}^2\left(x,y,{x}^{\prime },{y}^{\prime}\right)=\max \left({\left\Vert I\left(P\right(x,y\left)\right)-I\left(P\right({x}^{\prime },{y}^{\prime}\left)\right)\right\Vert}_{2,\rho}^2-{o}_{pr}{\sigma}^2,0\right)$$

(14)

From (13), the proposed distance measurement is as follow as:

$${d}_{PRDM}^2\left(x,y,{x}^{\prime },{y}^{\prime}\right)=-\left[\ln {\phi}_s\left(x,y,{x}^{\prime },{y}^{\prime}\right)+\ln {\phi}_r\left(x,y,{x}^{\prime },{y}^{\prime}\right)\right]$$

(15)

Here, for the sake of the convenience, the smooth parameter ${h}_s^2$ and h² are incorporated into (15). Compared to the traditional NLM filter, the proposed distance measurement includes both spatial distance measurement and range distance measurement based on the pixel intensities. Besides this, compared to the “signal” variance distance measurement ${d}_{\boldsymbol{SVDM}}^2\left(x,y,{x}^{\prime },{y}^{\prime}\right)$, a key difference lies in that the Gaussian weighted Euclidean distance between two image patches is used. Compared to the SNNDM ${d}_{\boldsymbol{SNNDM}}^2\left(x,y,{x}^{\prime },{y}^{\prime}\right)$ with (7), the proposed distance aims to employ the difference of the different pixels in the neighbor of the pixel at (x, y) for the pixel wise NLM filter. And all the neighbors pixels are used. The patch wise NLM filter with (7) is a two-step algorithm in essence. The first step is the weighted computation based on the similar patches. The second step is the aggregation of multiple estimates of the reference noisy pixel. The proposed pixel wise NLM filter is one step weighted method. The proposed method employing all neighbor pixels will demonstrate the satisfied performance.

Currently, the remained problem is the determination of the weight of the center pixel (CW) at (x, y). The proposed weight ${\omega_{pr}}\left(x,y,{x}^{\prime },{y}^{\prime}\right)$ is product of the spatial kernel ${\phi_{s}}\left(x,y,{x}^{\prime },{y}^{\prime}\right)$ . For the CW ω_pr(x, y, x, y), both ϕ_s(x, y, x, y) and ϕ_r(x, y, x, y) are 1 in term of (13). As a matter of fact, the CW plays an important role for the performance of NLM filter. So, some optimal weights are presented in the recent years. In [27], the Wiener filter center weight (WFCW) achieves the very good effect. For the proposed method, the proposed center pixel distance is.

$${\overline{d}}_{pr}^2\left(x,y,x,y\right)=\frac{1}{\mid {\textrm{R}}_0\mid}\sum_{\left({x}^{\prime },{y}^{\prime}\right)\in {\textrm{R}}_0}{\left\Vert I\left(P\right(x,y\left)\right)-I\left(P\right({x}^{\prime },{y}^{\prime}\left)\right)\right\Vert}_{2,\rho}^2$$

(16)

where R₀ = R − (x, y). The signal variance at (x, y) is computed as:

$${d}_{pr}^2\left(x,y,x,y\right)=\max \left({\overline{d}}_{pr}^2\left(x,y,x,y\right)-v{\sigma}^2,0\right)$$

(17)

At last, the range kernel of the proposed CW is expressed as:

$${\phi}_r\left(x,y,x,y\right)=\frac{v{\sigma}^2}{d_{pr}^2\left(x,y,x,y\right)+v{\sigma}^2}$$

(18)

The constant v is used to tune the denoising extent in local Wiener filter ϕ_r(x, y, x, y). As with [27], (18) shows ϕ_r(x, y, x, y) is larger in the flat domain, and smaller in the edge and other characteristic areas. This will be conducive to the preservation of image features in the process of denoising.

The proposed complete weight function is produced as:

$${\omega}_{cpr}\left(x,y,{x}^{\prime },{y}^{\prime}\right)=\left\{\begin{array}{c}\underset{\phi_s\left(x,y,{x}^{\prime },{y}^{\prime}\right)}{\underbrace{\exp \left(-\frac{{\left(x-{x}^{\prime}\right)}^2+{\left(y-{y}^{\prime}\right)}^2}{h_s^2}\right)}}\underset{\phi_r\left(x,y,{x}^{\prime },{y}^{\prime}\right)}{\underbrace{\exp \left(-\frac{d_{pr}^2\left(x,y,{x}^{\prime },{y}^{\prime}\right)}{h^2}\right)}}\kern0.5em \textrm{for}\ \left(x,y\right)\ne \left({x}^{\prime },{y}^{\prime}\right)\\ {}\frac{v{\sigma}^2}{d_{pr}^2\left(x,y,x,y\right)+v{\sigma}^2}\kern13.5em \textrm{for}\ \left(x,y\right)=\left({x}^{\prime },{y}^{\prime}\right)\end{array}\right.$$

(19)

And then, the restored image with the proposed method is computed as:

$$u\left(x,y\right)=\frac{\sum_{\left({x}^{\prime },{y}^{\prime}\right)\in \textrm{R}}I\left({x}^{\prime },{y}^{\prime}\right){\omega}_{cpr}\left(x,y,{x}^{\prime },{y}^{\prime}\right)}{\sum_{\left({x}^{\prime },{y}^{\prime}\right)\in \textrm{R}}{\omega}_{cpr}\left(x,y,{x}^{\prime },{y}^{\prime}\right)}$$

(20)

The flowchart of the proposed method is shown in Fig. 2. It clearly and concisely indicates that the key in the proposed method lies in the calculation of the weight values in term of (19).

Fig. 2

The flowchart of the proposed modified NLM method

2.4 Parameters

In the test, one takes one image as test image. The optimal parameters by hand can be obtained for the test image. And then, the optimal parameters of other images can be roughly considered to be around the optimal parameters of the test image. This is because each natural image is composed of similar features such as smooth domains, boundaries, textures, and details and etc. Thus, the parameters of other images can be roughly obtained. This strategy has been successfully used in the gradient domain methods in [28, 30].

Many parameters are involved in the proposed method. They include the size of search widow and similar window, the smoothing parameter ${h}_s^2$ of $\phi_{s}\left(x,y,{x}^{\prime },{y}^{\prime}\right)$ and h² of $\phi_{r}\left(x,y,{x}^{\prime },{y}^{\prime}\right)$, the offset parameter ο_pr in (14), the smoothing parameter ρ of Gaussian kernel, etc.

For the sake of fair comparison, the size of search window and similar window is set to 11 × 11 which is the same as selection of [27]. Then selection of Gaussian kernel is considered. If the smoothing parameter ρ of Gaussian kernel is too small, it cannot achieve the proper filtering effect, and if the smoothing parameter ρ of Gaussian kernel is too large, it will blur the image features. After a lot of attempts, the smoothing parameter ρ of Gaussian kernel is taken as 2 by hand. Based on which, Gaussian kernel G_ρ is produced by the MATLAB file “fspecial”. In the test, it is found that the difference between results obtained by using the proposed Gaussian kernel and results obtained by Gaussian kernel used by Manjon Herrera and Buades is subtle for the proposed modified NLM method, and due to fair comparison, the Gaussian kernel used by Manjon Herrera and Buades in [15] is adopted in the proposed modified NLM. Gaussian kernel used by Manjon Herrera and Buades avoids selection of the smoothing parameter ρ of Gaussian kernel. The details can be found in [15].

As for v in (19), it is taken to 2. This is because that ${\overline{d}}_{pr}^2\left(x,y,x,y\right)$ is average of noisy gradient coefficient of different direction and different distance gradient domain in neighbor around (x, y). From [30], the variance of Gaussian noise component contained in every noisy gradient coefficient is 2σ². For so, 2 is a reasonable selection.

How to determine the value of the smoothing parameter ${h}_s^2$, h² and the offset parameter ο_pr? Let ${h}_s^2={o}_s{h}_{s0}$ and h² = ο_rh_r0(h_s0 and h_r0 are the initial value). Here, the offset parameter ο_s and ο_r is used to tune the corresponding parameter value. And then, one presents a simple fitted approach to describe ο_pr, ο_s and ο_r. Assume that these offset parameters vary linearly with noise level. The function model is as follows:

$$f={f}_0+\theta \left(\sigma -{\sigma}_0\right)\kern0.5em \left(\ f\in \left\{{o}_{pr},{o}_s,{o}_r\right\}\right)$$

(21)

where f₀ is preset optimal value under some noise level σ₀ for some image. The (21) is used to get the corresponding functions of different offset parameter ο_pr, ο_s and ο_r.

Here, the peppers image is taken to obtain the initial values and optimal values. Let σ₀ = 10, under this noise level, the corresponding parameters are tuned. For example, h_s0 and h_r0 are 40 and σ², respectively. For ο_pr, f₀ = ο_pr0 = 1. For ο_s, f₀ = ο_s0 = 1. For ο_r, f₀ = ο_r0 = 1. To determine the θ, let σ = 25. The corresponding ο_pr, ο_s and ο_r can be presented. And then, the different θ is determined. For ο_pr, θ = 0.08. For ο_s, θ =  − 0.02. For ο_r, θ =  − 0.02. Three functions are shown in the following, respectively.

$${o}_{pr}=1+0.08\left(\sigma -10\right)$$

(22)

$${o}_s=1-0.02\left(\sigma -10\right)$$

(23)

$${o}_r=1-0.02\left(\sigma -10\right)$$

(24)

The function (22) demonstrates that ο_pr increases as σ increases. At this time, ${d}_{pr}^2\left(x,y,{x}^{\prime },{y}^{\prime}\right)$ decreases. This means more ${d}_{pr}^2\left(x,y,{x}^{\prime },{y}^{\prime}\right)$ is zero. More pixels around (x, y) will have the same value 1 of $\phi_{r}\left(x,y,{x}^{\prime },{y}^{\prime}\right)$ for higher noise level. At this time, the proper ο_s and ο_r will tune the denoising contribution of these pixels with the same ${d}_{pr}^2$. The differences of pixels will be fully utilized. Some important notations in the proposed method are summarized in Table 1.

Table 1 Some important notations for the proposed method

3 Experimental results

The aim of this section is to validate the performance of the proposed method. The comparison results are obtained by some NLM based methods using a NLM filter. They include the pixel wise methods using a NLM filter such as NLM-Stein [18], NLM-LJS [26], NLM-Max [15], NLM-WFCW [27] and NLM-Fov [5] since the proposed method is a kind of pixel wise design using a NLM filer. Among these methods, NLM-Fov is a state-of-the-art method for NLM filter. To further verify the performance of the proposed method and also for the proposed method is variant of NLM-SNN, the patch wise NLM filter NLM-SNN [6] is also used to compare. All the simulations are performed under the MATLAB R2017a environment. The code of the proposed NLM comes from NLM-Max [15] and can also originate from NLM-LJS [25]. The performance is similar. Therefore, the result is produced by modified code only from [15]. The main difference lies in that the running time is different. The 10 grayscale test images including Cameraman (256 × 256), Lena (512 × 512), Montage (256 × 256), House (256 × 256), Pepper (256 × 256), Barbara (512 × 512), Boat (512 × 512), Hill (512 × 512), Couple (512 × 512) and Fingerprint (512 × 512) are used to evaluate subjective and objective performance of the different NLM filter.

For the proposed method, the parameters have been determined in Section 2.4. For NLM-Stein, NLM-LJS and NLM-WFCW, as with [27], for the smoothing parameter h, the value is σ. The search window and similar window is the same as that of the proposed method. The used Gaussian kernel is that used by Manjon Herrera and Buades in [15]. For NLM-Fov and NLM-SNN, the parameters of each method have been set according to the values given by their respective authors in the corresponding referred codes. Only for NLM-SNN, his taken to σ so that the better result is get.

The denoising performance is measured in term of the peak signal-to-ratio (PSNR) and structural similarity (SSIM) index. PSNR measures the intensity similarity between the noise-free image s and the denoised image u. It is defined as.

$$\textrm{PSNR}=10{\log}_{10}\left(\frac{255^2}{\textrm{MSE}}\right)\textrm{dB}$$

(25)

where MSEis the mean squared error between the original noise-free image s and the reconstructed u. SSIM [23] can understand the structure between the noise-free image s and the denoised image u. It is defined as.

$$\textrm{SSIM}\left(s,u\right)=\frac{1}{N_1{N}_2}\sum_{i=1}^{N_1{N}_2}\frac{\left(2{\mu}_{s_i}{\mu}_{u_i}+{c}_1\right)\left(2{\sigma}_{s_i{u}_i}+{c}_1\right)}{\left({\mu}_{s_i}^2+{\mu}_{u_i}^2+{c}_1\right)\left({\sigma}_{s_i}^2+{\sigma}_{u_i}^2+{c}_2\right)}$$

(26)

where ${\mu}_{s_i}$ and ${\mu}_{u_i}$ are the average of local image patch s_i and u_i, respectively. ${\sigma}_{s_i}^2$, ${\sigma}_{u_i}^2$ are variances and ${\sigma}_{s_i{u}_i}$ is the covariance of s_i and u_i. c₁ and c₂ are constants. Author Wang et al. of [23] suggests using the down sampling version SSIM. The corresponding Matlab code comes from https://ece.uwaterloo.ca/~z70wang/research/ssim/.

3.1 Role of spatial kernel ϕ _s

The denoised image with ω_cpr = ϕ_r is yielded from an estimate using the proposed method. Although the noise is removed, excessive smoothness affects the visual effect in a sense (see Fig. 3b). Then the denoised image with ω_cpr = ϕ_sϕ_r avoids the disadvantage of excessive smoothness. The result in Fig. 3c is more natural than that in Fig. 3b (see the capsicum in the lower part of Figs. 3a, b and c). In addition, the PSNR and SSIM values of the denoised image in Fig. 3b are 28.45 dB and 0.8210, respectively; the PSNR and SSIM values of the denoised image in Fig. 3c are 28.67 dB and 0.8341, respectively. In a word, compared to the result without spatial kernel, the results using the proposed method with spatial kernel have different degrees of ascension in term of subjective and objective performance.

Fig. 3

Comparison of denoised result without ϕ_sand denoised result with ϕ_s using the proposed method for Peppers (256 × 256). a Original image; b denoised image without ϕ_s; c denoised image with ϕ_s

3.2 Comparison of the proposed method and the related NLM methods

Tables 2, 3 and 4 present the PSNR and SSIM results of NLM-Stein, NLM-LJS, NLM-Max, NLM-WFCW and the proposed method forσ = 10, 20, 30, respectively. One can find that the proposed method always outperforms other several methods in PSNR and SSIM in almost all cases. Except for Fingerprint and House at σ = 10, the proposed method is slightly inferior to the highest PSNR or SSIM among the obtained results. For Fingerprint, in PSNR performance, 31.09 dB with the proposed method is inferior to 31.19 dB with NLM-LJS. For House, in SSIM performance, 0.8963 with the proposed method is inferior to 0.8967 dB with NLM-LJS. However, in all other cases, the proposed method achieves the best results. Furthermore, the advantage of the proposed method becomes more obvious when the noise level increases.

Table 2 PSNR and SSIM values on Proposed and several other NLM methods for σ = 10

Table 3 PSNR and SSIM values on Proposed and several other NLM methods for σ = 20

Table 4 PSNR and SSIM values on Proposed and several other NLM methods for σ = 30

Table 5 presents PSNR and SSIM results averaged over all the test images for every noise level σ (σ ∈ {10, 20, 30}) and for all three noise levels for Proposed and several other NLM methods. It is noted that the center weight strategy of the proposed method comes from my previous work NLM-WFCW proposed in 2021. The comparison with NLM-WFCW is an ablation study. The comparison shows that the proposed method has weak improvement in noise level 10, while the improvement has become increasingly significant in noise levels 20 and 30. At the noise 30, compared to NLM-WFCW, the averaged gains are 0.40 dB in PSNR and 0.0128 in SSIM, respectively. For all three noise levels, compared to NLM-WFCW, the averaged gains are 0.24 dB in PSNR and 0.0085 in SSIM, respectively.

Table 5 PSNR and SSIM results averaged over all the test images for every noise level σ and for all three noise levels for Proposed and several other NLM methods

In a word, compared to other methods, the proposed method is more stable and excellent in PSNR and SSIM.

Figures 4 and 5 present the visual comparisons of results of NLM-Stein, NLM-LJS, NLM-Max, NLM-WFCW and the proposed method.

Fig. 4

Comparison of the restoration results from the different methods. Zoom into file for a better view. a Original Barbara (512 × 512) image; b noisy image (σ = 20); c-g shows restored Barbara images using NLM-Stein, NLM-LJS, NLM-Max, NLM-WFCW and the proposed method, respectively; h shows the bright dot block on table legs in the lower left corner with original Barbara, NLM-LJS, NLM-WFCW and the proposed method in turn from left to right

Fig. 5

Comparison of the restoration results from the different methods. Zoom into file for a better view. a Original Peppers (256 × 256) image; b noisy image (σ = 10); c-g shows restored Peppers images using NLM-Stein, NLM-LJS, NLM-Max, NLM-WFCW and the proposed method, respectively

In Fig. 4, noisy Barbara (512 × 512) image with σ = 20 is processed. Figures 4c, d, e, f and g are the denoised results with NLM-Stein, NLM-LJS, NLM-Max, NLM-WFCW and the proposed method, respectively. Figure 4h shows the bright dot block on table legs in the lower left corner with original Barbara, NLM-LJS, NLM-WFCW and the proposed method in turn from left to right. One notices that there is a rectangular area in the lower right corner of Fig. 4g for the proposed method. In Figs. 4c and e, the dot block in this area is invisible compared to that in Fig. 4g. This is because the smooth strength in the edges and details domain for the proposed method is reasonably reduced in the whole denoising process compared to NLM-Stein and NLM-Max. For NLM-LJS, NLM-WFCW and the proposed method, the area is well preserved. But, for the bright dot block on table legs in the lower left corner of Fig. 4a, the presentations with NLM-LJS, NLM-WFCW and the proposed method have some differences. The result with proposed method is more faithful to the original image. For example, for the original image, the mean and standard deviation of the bright dot block is 46.62 and 22.81, respectively. For the result Fig. 4d with NLM-LJS, the mean and standard deviation of the bright dot block is 46.08 and 10.83, respectively. For the result Fig. 4f with NLM-WFCW, the mean and standard deviation of the bright dot block is 44.78 and 10.97, respectively. For the result Fig. 4g with the proposed method, the mean and standard deviation of the bright dot block is 45.61 and 13.31, respectively. It can be see that the proposed method is closer to the original image features in term of standard deviation compared to NLM-LJS and NLM-WFCW. Although the mean with NLM-LJS is closer to that of original image, there are many obvious noise points in the result with NLM-LJS (see Fig. 4h). Visually, the bright dot block on table legs with the proposed method is brighter in comparison with that with NLM-WFCW and NLM-LJS. And noise is handled well with the proposed method (see Fig. 4h).

From Fig. 4, the proposed method is better in the preservation of image features. This merit is shown in Fig. 5. In which, noisy Peppers image with σ = 10 is processed. Figures. 5c, d, e, f and g are the denoised results with NLM-Stein, NLM-LJS, NLM-Max, NLM-WFCW and the proposed method, respectively. One can find that there is a rectangular area in the lower left corner of Fig. 5g for the proposed method. In Fig. 5e, the area emerges in the same position, but this area is over-smoothed compared to that in Fig. 5g. For NLM-Stein and NLM-LJS, the area is well preserved. But, there are the left noise in a lot of other areas in Figs. 5c and d because of the insufficient denoising. For NLM-WFCW and the proposed method, the good visual effect is achieved.

In a word, compared to other methods, the proposed method can better preserve textures, details and remove the noise. This is because the difference of pixels is used sufficiently in the proposed method.

NLM-Max is a classical method, and other NLM filters are produced by replacing the center weight (CW) of NLM-Max. Only for the proposed method, the spatial kernel need also be computed. Therefore, compared to the NLM-Max, for each pixel, the proposed method need only mainly a few addition and product operations in calculating the variance of pixel in the proposed CW and spatial kernel (please see Eq. (19)), so the computational time increment for the proposed method is negligible. Here, for NLM-Max and the proposed method, the average running time is measured by code from [15]. For example, for the size 256 × 256 and 512 × 512 image, the running times with NLM-Max are 20.12 s and 80.33 s, respectively. The running times with the proposed method are 20.13 s and 80.34 s, respectively. It can be said that the proposed method has achieved good performance with little increase in computing time.

3.3 Comparison of the proposed method and two state-of-the-art NLM methods

To further verify the effectiveness of the proposed method, two state-of-the-art NLM filters are used to compare. Here, the compared code presented by the corresponding authors is written in a similar form to that of [25]. For the sake of fair comparison, the running time is measured by modified code from [25] for the proposed method.

Figures 6, 7 and Tables 6 and 7 present the comparative results between NLM-Fov and Proposed. Visually, the result with NLM-Fov looks smoother in the case of shrinking and appears many minor false details in case of magnification (see Figs. 6 and 7). Although the proposed method show the block effects, it better preserves the contrast of structure, has less artifacts around edges and looks more natural (see the capsicum in the lower part of Figs. 7a, b and c). In a word, two methods have their own advantages. These have been embodied in PSNR and SSIM in Tables 6 and 7. The performance of two methods is comparative. Although the foveated NLM filter has shown the state-of-the-art performance, the current research demonstrates that the proposed window based NLM filter by specific design also can compete with the foveated NLM filter. Furthermore, the proposed method can hardly increase the calculation time compared with NLM-Max. But NLM-Fov increases the calculation time by over 50% compared with NLM-Max (For fair comparison, the implementation by NLM-Max comes from code of NLM-Fov). This is because that the proposed method only uses a Gaussian filter in the computation of weight but NLM-Fov employs more filters in essence.

Fig. 6

Comparison of the restoration results from NLM-Fov and Proposed for noise level σ = 20. Zoom into file for a better view. a Original House (256 × 256) image; b-c shows restored House images using NLM-Fov and the proposed method, respectively

Fig. 7

Comparison of the restoration results from NLM-Fov and Proposed for noise level σ = 30. Zoom into file for a better view. a Original Peppers (256 × 256) image; b-c shows restored Peppers images using NLM-Fov and the proposed method, respectively

Table 6 PSNR and SSIM values on Proposed and NLM-Fov for σ = 10, 20, 30

Table 7 PSNR and SSIM results averaged over all the test images for every noise level σ and for all three noise levels for Proposed and NLM-Fov

Figures 8, 9 and Tables 8 and 9 present the comparative results between NLM-SNN and Proposed. Figure 8 with the proposed method produces more obvious artifacts around the edges (see “Hello World” in Fig. 8c). But, Fig. 9c with proposed method better preserves the features and contrasts in camera compared to Fig. 9b with NLM-SNN. Likewise, from Tables 8 and 9, these have been embodied in PSNR and SSIM. In general, the performance of the two methods is comparable. They have their own characteristics and strength. Furthermore, the proposed method can produce the result with less time. For NLM-SNN, selection of patch and aggregation of multiple estimates costs more times. For example, for the size 256 × 256 and 512 × 512 image, the average running times with the proposed method are about 0.70 s and 2.68 s, respectively. But, for the size 256 × 256 and 512 × 512 image, the average running times with NLM-SNN are 9.02 s and 62.05 s, respectively. Obviously, the proposed method saves plenty of time.

Fig. 8

Comparison of the restoration results from NLM-SNN and Proposed for noise level σ = 10. Zoom into file for a better view. a Original Montage (256 × 256) image; b-c shows restored Montage images using NLM-SNN and the proposed method, respectively

Fig. 9

Comparison of the restoration results from NLM-SNN and Proposed for noise level σ = 30. Zoom into file for a better view. a Original Cameraman (256 × 256) image; b-c shows restored Cameraman images using NLM-SNN and the proposed method, respectively

Table 8 PSNR and SSIM values on Proposed and NLM-SNN for σ = 10, 20, 30

Table 9 PSNR and SSIM results averaged over all the test images for every noise level σ and for all three noise levels for Proposed and NLM-SNN

By comparison, the proposed method is superior to the several excellent NLM filters such as NLM-Stein, NLM-LJS, NLM-Max and NLM-WFCW in subjective and objective performance. At the same time, the proposed method is also a state-of-the-art NLM filter by the comparative results with NLM-Fov and NLM-SNN.

4 Conclusion

By exploring the spatial kernel to remedy the statistical nearest neighbor distance measurement (SNNDM) in the weight computation in NLM filter, in this paper one proposes a bilateral thresholding to calculate the weight of each pixel in neighbor at the reference noisy pixel. In order to make full and proper use of the differences of different pixels, the center weight and other weights are also treated differently. Some simulations are taken on several grayscale natural images to investigate the performance of the proposed method. The results prove the proper and simple weight function design can bring superiority with negligible differences in calculation time to other related NLM filter approaches such as NLM-Stein, NLM-LJS, NLM-Max and NLM-WFCW in PSNR, SSIM and visual performance. Furthermore, the proposed method is on par with the-state-of-the-art NLM methods such as NLM-Fov and NLM-SNN in PSNR, SSIM and visual comprehensive performance, but the proposed method can save plenty of running time. In addition, it is noted that the proposed pixel wise NLM method originates from the patch wise SNNDM and other analysis. In the future, one will seek more appropriate distance measurement ways to compute the weight of neighbor pixel to improve patch wise NLM filter.