1 Introduction

In the current years, the field of image deblurring has gained significant consideration from researchers due to massive usage of high-quality multimedia in an array of valuable applications such as medical images [1,2,3], security and surveillance [4, 5], remote sensing [6,7,8], underground object detection [9], etc. Unfortunately, the images captured with either personal hand-held or surveillance cameras are degraded due to different types of contaminations such as noise [10], blur [11,12,13,14,15,16,17,18,19,20], and fog [21]. The primary cause of this kind of degradation is blurring which corrupts the visual quality of latent images by averaging the rapid changes in pixel intensities. The blurring is determined by the relative translation or rotation between the objects and the camera. It removes the high spatial frequencies, thereby obscuring or distorting the high-level details from the image. In this regard, several methods have been projected to enhance the visual quality of the corrupted images in which, Image deblurring is the primary countermeasure for restoring the quality of degraded images through precise estimation of the convolution kernel that causes blurriness. There are various kinds of blurring functions such as Gaussian, Motion, Average, or Defocus, etc.

On the basis of prior knowledge regarding the convolved blur kernel and the original sharp image, image deblurring techniques are broadly classified into two major categories i.e., Blind image deblurring (BID) and Non-Blind image deblurring (NBID) [22,23,24]. BID is defined as a kind of deblurring where the blur kernel is not known. In contrast, NBID is defined as the kind of deblurring where the convolved kernel causing blurriness is known. According to the literature, it has been inferred that BID is a challenging problem due to unstable and non-unique solutions [25,26,27,28,29]. Therefore, in our proposed research work, we have more focused on the techniques of BID for image restoration.

In case of blind image deblurring it is very important to estimate the unknown blur kernel accurately. It has been observed that well-detected edges of objects in degraded images are fundamental in the correct estimation of the blur kernel. Likewise, the poorly detected edges can neither determine the blur kernel effectively nor are able to remove ringing effects near strong edges [30,31,32]. Most of the Image deblurring techniques have not considered the weak edges and their rotation symmetry while estimating the blur kernel. The negligence of such minor structures results in poor detection of blur kernel and do not provide minimum cost function for sharp image. A range of additional image priors are required to accurately estimate the blur kernel that can restore the high-quality sharp image.

In this research paper, we have focused on accurately estimating the blur convolution kernel that gives high quality sharp image with minimum cost function. The accurately estimated blur convolution kernel is vital as it is then used in combination with any known image deblurring technique for recovering the latent sharp image. The normalized form of regularization which was used as a measure of sparsity in [33] is utilized in our work in combination with gradient image that gives low cost for sharp images. The blur in the image always attenuates the high frequencies of sharp image and hence decreases value of the L1 norm as it is scale variant. The benefit of using normalized form of regularization is that blur decreases both L1 and L2 norm but L2 norm decreases more that eventually increases the ratio of this normalized regularization and results in minimum cost function for the sharp images. The whole procedure is divided into three main phases: (i) Edge Enhancement by using the Scharr Edge-based (SEB) operator (ii) Estimation of blur kernel from normalized regularization of gradient image and (iii) recovery of latent sharp image. The approach of SEBR is simple as it does not have to apply any complex schemes followed by other techniques to overcome the shortcoming of already developed deblurring models. The kernel estimated with SEBR is more accurate because it can retain the salient edges and minor texture areas of the objects from the degraded image which results in recovering a fine quality sharp image with minimum cost function. The key contributions of this research work are mentioned below.

  • An improved approach for kernel estimation based on Scharr Edge-based (SEB) operator and normalized regularization is proposed that highly optimizes the salient edge extraction from blur images and achieves perfect rotation symmetry that results in an accurate kernel with lowest cost function for sharp images.

  • The proposed SEBR can detect the outliers in the degraded image and minimize the ringing artifact near the strong edges while improving the contrast of the sharp image with a comparatively low cost of computation.

  • The blur convolution kernel estimated with SEBR is scale-invariant because of normalized regularization, flexible to various types of blurriness having different centers and rotations, and applicable to an extensive range of deblurring problems irrespective of images having different characteristics.

  • The proposed technique effectively processes the real blur images having low light, complex scenes, and uniform and non-uniform blur and give favorable results against other state-of-the-art deblurring approaches.

The rest of the research paper is divided into the following sections: Sect. 2 includes the critical review of the techniques proposed in the BID. Section 3 presents edge enhancement with Scharr Edge-based detection operator and Sect. 4 presents the proposed approach of SEBR for the blind image deblurring process in detail. Experimental deblurring results and discussions are presented in Sect. 5. Lastly, Sect. 6 concludes the proposed approach for the BID.

2 Related Work

In the previous years, an extensive variety of deblurring models have been proposed for blind deblurring of corrupted images, however, they were not able to give high-quality results because of restrictions imposed on kernel priors and latent image. The techniques used for estimation of blur kernel are largely classified into two main categories based on their various types of properties: (i) Variational Bayes-based techniques (VB-based) [16,17,18, 34], and (ii) Maximum a Posterior-based technique (MAP-based) [11, 13, 19, 32, 35,36,37,38]. VB-based techniques avoid trivial solutions to the problems, however, are not cost-effective compared to the MAP-based approaches used for image deblurring. Contrary, in MAP-based approaches, Blur kernel is estimated in two ways: (1) Salient Structures (based on /utilizing the salient structures of objects in images) [11, 13, 19, 32, 36, 37, 39,40,41] and (2) sparse regularization based methods [10, 13, 15, 28, 35, 38, 42, 43].

In this research paper, we have explored MAP-based approaches due to their cost-effectiveness. The techniques proposed in [11, 19] for single image deblurring were not able to yield effective results due to low-quality blur kernel estimation, and other variables. These techniques were based on edges of objects in gradient images by calculating their L1 norm. The estimated PSF recovered the latent image with high quality but only under some specific conditions. The L1 norm in combination with another technique based on ringing affect suppression were proposed in [37] to enhance the visual quality of the recovered sharp image. The approach gives comparatively better deblurring results, but some conditions and parameters are required to be set for good quality results. For further improvement of the recovered image, a new regularization method based on L1/L2- norm was anticipated that uses the ratios of L1- norm to L2—norm applied the high pixel values in degraded images [28]. This method performs better but lacks the capability for modeling both the image sparsity and blurry gradients. Consequently, Resultant deblurred images had ringing effects near the edges and were also over-smoothed in some cases.

Another technique based on random transform was introduced in [10] for estimating the convolved blur kernel. It deblurred and denoised the image by using singular values for decomposition and recovered the sharp image. The random transformation of each image was estimated by the application of several directional operators in various orientations. It performs better but the ringing effect was visible in the restored sharp image. A generalized sparse representation-based deblurring technique was introduced in [38] that worked on removing the blur caused by both uniform and non-uniform motion. This technique had mathematically proven the accuracy of the L0-norm for the sparse representation, and it also converged in a few iterations as no extra filters were required for the optimization. However, the proposed method was not able to give good results on images having text regions. An effective kernel estimation technique was introduced in [14] that was based on data fidelity terms for suppressing the negative effect of outliers. This method effectively restored the blurred image but at the expense of greater computational cost. A generalized iteration-wise shrinkage threshold method was proposed to preserve the smooth structure of objects in the restored image [32]. In this method, the generalized shrinkage thresholding algorithm (GST) was used to eliminate the small details while sharpening the salient edges of objects in blur images, but it was not able to accurately estimate the kernel. Dark channel-based image priors were employed in [15] with the assumption that the dark channel image priors are less sparse which achieves good results in different type of scenes. Another method proposed in [43] used Gaussian priors for the estimation of the blur kernel. However, the proposed methodology was incapable of retaining the salient structures of objects because of two main reasons. First, it lacked sparse representation which led to a noisy solution having dense structures, and secondly, the kernel recovered was not of fine quality.

A new MAP-based framework was introduced for kernel estimation in [42] that used the edge priors specifically related to the scene. It works better on some images but fails on a variety of scene images. Sparsity based on Row-Column was utilized for blind image deblurring in [44]. In this method, the problem of sparsity optimization was solved by the inclusion of rows and columns for the kernel estimation. Later, a new method [45]is proposed that combines the dark and black channel and utilizes the extreme channel prior for deblurring the corrupt images. The methods achieved better results, but they have certain drawbacks as the image has no evident bright and dark pixels which poorly estimates the kernel. Inspired from dark channel prior a new technique was presented in [46] that is based on sparse regularization by using the local minimum pixels in the images that speed up the deblurring process greatly. Another blind deblurring technique [47] based on the prior of local maximum gradient prior (LMG) is presented that gives good performance for many different scene images. The method presented in [41] simplifies the work of LMG and introduces the patch based maximum gradient that lowers the cost of calculations. The blind deblurring approach in [40] have designed a new image prior based on the ratio of extreme pixel values which are affected by blur. The image is divided into various patches for extraction of features and finally overlapped all patches for getting detailed information. It gives good results but its performance decreases when the image has severe types of noise. Its computation complexity and run time are also high. An effective regularization method based on probability-weighted moments was proposed in [48], the method effectively preserved the trivial structure of the object in an image and restores the sharp image with good quality but it has large computation time.

As the gradient and intensity based priors are employed mostly to the single or adjacent pixels, it does not account for the relationship among pixels on larger scales. For better reflection of the relationship of pixels in the image, patch based deblurring approaches have been developed. A method inspired by the statistical properties of natural images is presented in [49] that adopts two priors based on edges in patches. Another patch based method is proposed in [50] that combines the similar characteristic in different patches of blur images. In [51] a new deblurring approach combining the selection of salient edges and low ranks constraints is presented. A deblurring method based on minimum patch constraint imposing zero is presented in [52]. These patch-based methods require frequent searching, so it requires much more time to execute.

The above discussed image deblurring techniques [19, 42, 46, 47, 51, 53] mostly follows the MAP based or gradient prior based approaches for deblurring corrupted images. In the MAP based deblurring frameworks the blurring kernel estimation is highly dependent on the extraction of sharp edges. Presently the main edge extraction method relies on the selection of threshold value for retrieving strong edges. The deblurring methods based on explicitly selecting edges from corrupted images have many drawbacks. As some images don’t contain any noticeable edges that can be retrieved so the resultant deblurring images are mostly over sharp and amplify the noise in images. They are also very sensitive to the selection of parameters for a variety of images. The regularization based techniques [37, 43, 48, 54] needs the finer details of the images to be used for kernel estimation and they favors the blurry images rather than sharp images. Secondly, they lacked sparsity which led them to produce dense and noisy output images. Therefore, it is required to develop a prior that is relevant to gradient information of degraded image but favors low cost for sharp images as compared to blurry images. This work presented a Scharr edge-based regularization (SEBR) prior that utilizes both the gradient information of image and normalized form of regularization to accurately estimates the blur kernel that gives the minimum cost for recovering sharp image.

3 Edge Extraction with Scharr Edge-Based Gradient Operator

In deblurring tasks, edges being the high-frequency components plays a significant role in enhancing the quality of recovered image. Moreover, such information is also directly associated with visual cognition and human perception. It has also been observed that clean and clear images have sharp and fine quality edges whereas, in blurred images, abrupt changes in intensity values smoothens out due to averaging of neighboring pixel areas. The edge gradient reduced in a gaussian blur, therefore, edge information around the patches is weak. Likewise, edge information becomes dense and unstable due to the ghosting effect in the motion-blurred images. Owing to this deficient edge-based information, our proposed model will focus on these gradient regions and recover the sharp image.

Numerous state-of-the-art operators are available for extracting edges from images such as Sobel, Prewitt, Robert, Canny, Laplacian, Laplacian of Gaussian (LoG) [55]. These edge detectors smoothen the image while calculating the gradient during removal of noise from image. The convolution of image with Sobel edge detector is only the approximation process of vertical and horizontal pixel intensity differences and it does not contain any information of rotational symmetry [56, 57]. Scharr et al. [58] have improved the Sobel edge detection filter by minimizing the weighted angle difference in the frequency domain and proposed a new edge detection filter known as Scharr filter. The Scharr filter is also fast like Sobel edge filter, but its accuracy is high when it is calculating the derivatives of the image. Different sizes of Scharr edge based gradient operators can be used for detecting edges from the blurred image [59].

The horizontal and vertical masks of Scharr edge detector of size 3 × 3 are given below.

figure a

We have used the Scharr edge-based gradient operator to extract the salient edges from the captured degraded images The motivation for using Scharr edge-based gradient operator to extract the edge-based information from the degraded images is twofold: (1) rotation symmetry, and (2) these edge masks facilitate in suppressing noise. SEB gradient operator has the capability to achieve the perfect rotation symmetry which leads to the detection of edges along all its orientations and optimizes edge extraction from blurry and noisy images. SEB operator will perform the same amount of smoothing in all directions. Generally, it is known in advance that edges in an image are not oriented in some specific direction; therefore, there is no prior reason to perform more smoothing in one direction than others. The property of rotational symmetry infers that the Scharr edge masks are not biased to successive edge detections in any specific direction.

Scharr Edge masks will behave like a single lobe filter and remove the noise by replacement of every image pixel value with a weighted average of the neighboring pixel values. The weight assigned to the neighboring pixel in masks decreases monotonically as the distance increases from the center pixel because the edge is the local feature in the image and if more significance is given to the far away pixel it will distort the features during filtering [58]. The Fourier spectrum of the SEB operator also shows a single hemisphere in the frequency domain. Mostly the images captured are corrupted by unwanted frequency signals like gaussian noise and other fine textures. Edges being the most important features within images have components at both high and low frequencies. The single lobe of the SEB operator in the Fourier spectrum will give an output image that is not corrupted by high-frequency components and will retain most of the desirable details in an image. Scharr edge filter simply calculates the magnitude of gradients and efficiently detects both weak and strong edges because it is also equally sensitive to the narrow and wider boundary lines as shown below in Fig. 1.

Fig. 1
figure 1

Extraction of edges from blur image a real blur image, b Edge detection with Sobel, c edge detection with Scharr

Full size image

4 Proposed Methodology of SEBR

In this research article, an improved method named Scharr Edge-based regularization (SEBR) has been proposed for the blind image deconvolution of images. The proposed SEBR is executed in three main steps: (1) Edge Enhancement by using the Scharr Edge-based (SEB) operator, (2) Estimation of a convolved blur kernel, and (3) Recovery of a latent sharp image. Figure 2 below presents the proposed approach used for blind image deblurring. The observed blur image (f (x, y)) is down sampled to a certain size. The state-of-the-art Scharr edge detection operator is then applied to the down-sampled blurry images to extract the edge-based information of objects at different levels. The resultant edge map (Edge-Map) consists of both weak and strong edges. Subsequently, the trilateral filter is then applied to the resultant edge map (Edge-Map). The trilateral filter not only enhances the weak edges but also mitigates the effect of noise. Moreover, the trilateral filter is the fine quality version of the bilateral filter that suppresses the impulse noise effectively and sharpens the edge areas of objects in the blurred image. The kernel estimated with SEBR will effectively recover the sharp images that are either affected by gaussian or impulse noise. The kernel (k) is estimated by mathematical operation. Afterward, the initial image (f (x, y)) is updated by using the fast Iterative Shrinkage-Thresholding Algorithm (ISTA). Finally, a high-quality deblur image is obtained by using the non-blind deblurring method.

Fig. 2
figure 2

Proposed framework for blind image deblurring

Full size image

The following equation is used to model the captured blur image.

$$ {\mathbf{B}} = {\mathbf{I}} \otimes {\mathbf{k}} + {\mathbf{N}} $$
(1)

In the above equation, the symbol B is used to show the captured corrupted image having blurry effects, the symbol I is used for showing an original sharp image that becomes blur after convolving with kernel k, and ⊗ symbol is used to show the operator of the convolution. N shows the environmental noise that also degrades the image during capture. The main objective of our work is to develop an image deblurring technique that gives an efficient solution to the problem and restores the original image I from the input blur image B through estimation of high-quality blur convolution Kernel k.

Afterward, the process of blind image deblurring comprises of two main steps i.e., (1) estimating unknown blurring convolution kernel, and (2) recovery of original sharp image. The quality of the recovered image I can be highly improved through precise estimation of the unknown blur convolution kernel k. The standard process used for blind image deblurring is revealed below in the following algorithm.

figure b

4.1 Blind Estimation of Blur Convolution Kernel

In the proposed approach, blur convolution kernel k that convolved with the captured image and make it blur is estimated on the highest frequencies of the image as it was performed in [28]. We take blurred image B as input and apply the Scharr edge based (SEB) filters to get a high-frequency image y. The horizontal and vertical edge-based information is extracted by convolving xmask and ymask, respectively with the down-sampled image (shown in Eqs. (2), (3) and Eq. (4) gives the strength of both horizontal and vertical edges.

(2)
(3)
$$ G_{{{{mag}}}} = \sqrt {G_{x}^{2} + G_{y}^{2} } $$
(4)

Here, \(G_{x}\), \(G_{y}\) presents the vertical and horizontal edge-based information obtained by convolving the image (I) with the xmask and ymask, respectively. Worth noting that the stride of 1 is set to preserve the continuity in edges. The model equation for proposed SEBR based blind image deblurring is shown below in (Eq. 5)

(5)

Here k represents the unknown blur convolution kernel, and x represents the original sharp image which is also unknown. Whereas λ and \(\sigma\) are the parameters used for optimization and stability of both the kernel k and image regularizing, the symbol represent the 2D convolution operation. Considering the physical principles of the formation of blur in the sharp image, there are two types of restrictions that bounds the blur kernel k: (1) k ≥ 0, and (2) \(\mathop \sum \limits_{i} ki\) = 1.

The Eq. (5) comprises of 3 main parts. The 1st term shows the likelihood of the deblurring model shown earlier in Eq. (1). The 2nd term shows the Scharr edge-based regularization which is based on the detection of salient edges at different levels and applying the L1/l2 regularization to the image x which will be \(\left| {\left| x \right|} \right|_{1} /\left| {\left| x \right|} \right|_{2}\). The SEBR will further enhance the salient edge detection for kernel estimation K and encourages the scale invariant sparsity during deblurring process. The third term shows that L1—norm-based regularization is then applied to the convolved kernel k to minimize the noisy effects from blurry image and for its sparse representation. To get a highly optimized solution to the blurring problem, we must assume an initialization for the values of latent image x and blur convolution kernel k, and then both will be updated in an alternate manner. Inspired by the method proposed in [28], the sharper image x and blurring kernel k are estimated by using alternating steps.

For the updating of x and k, Eq. 5 is divided further into 2 sub equations as given below:

$$ \min_{x,k } \lambda \left| {\left| {x \otimes k - y} \right|} \right|_{2 }^{2} + {{ SEBR}} $$
(6)
$$ \min_{x,k } \lambda \left| {\left| {x \otimes k - y} \right|} \right|_{2 }^{2} + \sigma \left| {\left| k \right|} \right|_{1} $$
(7)

4.1.1 Updating of Intermediate Image x

The Eq. (6) is highly nonconvex because of the inclusion of normalized form of regularization term \(\left| {\left| x \right|} \right|_{1} /\left| {\left| x \right|} \right|_{2}\). But it can be made convex if the denominator is fixed from the last iteration which makes it L1 regularized. To solve this L1 regularized problem we than used the Fast iterative shrinkage thresholding algorithm (ISTA) [60] because of its enhanced rate of convergence and increased capability of restoration. The fast ISTA algorithm is used for solving the linear inverse problem shown below in Algorithm 2.

figure c

In the above algorithm, S stands for soft shrinkage operator that is applied to the input vector of image x. This operator is used for shrinking every component of the input blurry image vector towards zero as shown below in Eq. 8.

$$ S_{\alpha } = \max \, (x_{i} - \alpha , \, 0)\;{{sign }}(x_{i} ) $$
(8)

This algorithm is the fastest and simplest one and it involves only the multiplications of the matrices of kernel k and blur image x followed by the shrinking process componentry. The ISTA algorithm is applied to innermost iterations in the x-updating algorithm and the outer loop of ISTA is used for re-estimating the weightage of the likelihood term ||x||2 used in Eq. (6). Algorithm 3 is used for updating of initialized image x.

figure d

In the above algorithm, the symbol kT is used for the conjugate transpose matrix of estimated kernel k and the symbol λmax(kTk) stands for maximum eigenvalues for kTk. The input value of xM is used for the estimation of the convolved kernel that causes blur for solving the sub-problem of updating k. Our proposed approach is used for the estimation of blur kernel as elaborated in algorithm 3. The estimation based on the gradient values of the image has proved to give more precise results [16]. Because of the normalized form of regularization the problem is highly non convex so we use of the outer and inner iterations to highly reduced the cost function as shown in Eq. (6).

4.1.2 Updating of the Kernel k

After updating the image x with Eq. (6), the convolved kernel k should be updated by using Eq. (7). The Iterative re-weighted Least Squares based technique [61] was applied for the estimation of kernel k without setting any constraints. It has used the weights gained from the former updating of matrix k having only one outer iteration. At the finer level, the estimated kernel k has values that can be either neglected or can be normalized to zero for increasing its robustness towards noise that enhances its efficiency. The kernel k is estimated in the multiscale process and a coarser to the finer pyramid of image resolution is followed.

For converging to a good quality solution, many updates are required on x and k in case of very large kernels. To alleviate this trouble, we have estimated the blurring kernel in a multiscale manner by using a coarser to finer levels of images as used in [16]. We have used the images at different scale levels with a resolution size of ratio √ 2 in every dimension. The size of kernel k is kept 3 × 3 at the coarsest level. We have down sampled the input blur image and then used Scharr edge-based gradient (SEB) operator at each level to get the discrete values gradients of image y. At every scaling level, alternate updating of image x and kernel k is performed. Once we get the blur convolution kernel k and sharper image x then up sampling is performed to be used as the initialized kernel and sharp image for subsequent levels.

4.1.3 Recovery of Sharp Image

After estimating of the blurring convolution kernel K, an extensive number of non-blind image deblurring techniques are available for recovering original image I from the captured blur version image B [28, 61]. The simplest deblurring method is Richardson-Lucy (RL) technique, however, this is very sensitive to an incorrect kernel estimation and ultimately produces ringing artifacts in the restored image. We have used the Hyper-Laplacian priors technique [62] for restoring image after the estimating blurring kernel to avoid high sensitivity towards inaccurate estimation of kernel. This technique for image deblurring proves to be faster than others and it is also robust to the minor errors in kernel estimation. The hyper-Laplacians based method uses a continuous method for solving the cost function as mentioned below in Eq. (9):

$$ \min_{i } \lambda \left| {\left| {I \otimes k - B} \right|} \right|_{2 }^{2} + G_{X } Bc + G_{Y} Bc $$
(9)

Here Gx and Gy represent the horizontal and vertical Scharr operators used aforesaid in Part 4.1. It is important to mention that the value of c is kept at 0.81 and the value of λ is kept at 3000 as used in [28]. Also, we have used the Lp-based regularization instead of using L1/L2 norm for the restoration of the image because non blind deconvolution process is much less ill posed than blind image deblurring.

4.2 Fast and Robust Nature of SEBR

In our approach the blur kernel is estimated in a multiscale iterative manner by using a coarse to fine levels of images which requires only few convolution operations between the updated intermediate image x and blur kernel k which makes it very fast. The cost function of our deblurring model is very simple as shown earlier in Eq. (5). SEBR takes less time than the deblurring techniques of [25, 50]. From the experimental analysis, it is proved that proposed SRBR is robust to selection of parameters. We kept the same values of parameters for the deblurring outputs stated below in the results section. The robustness of SEBR is the major improvement over existing techniques that need to adjust the various parameters for different types of blur data.

5 Experimental Deblurring Results

For validating the effectiveness of the proposed SEBR technique, several experiments are made on synthetically blur images and real blur data. The obtained results of the algorithm are evaluated with the well-known performance matrices. The research paper is further sub-divided into three parts that include: (1) Evaluation Metrics, (2) Results on synthetically blurred images and real blur image data (dataset is available at [63]), and (3) discussion of the obtained results.

5.1 Evaluation Metrics

The renowned evaluation metrics used for performance evaluation of results are Peak Signal to Noise ratio (PSNR), a measure of structure similarity index (SSIM), error ratios, and time of computation taken in seconds.

The following Eq. (10) is utilized to calculate the PSNR value of sharp image recovered with the proposed algorithm.

$$ {{PSNR}} = 10 \cdot \log _{10} \left( {\frac{{\left( {{{Max}}} \right)}}{{{{MSE}}}}^{2} O} \right) $$
(10)

In the above equation ‘Max’ shows the maximum values of pixels in the original sharp image. 'MSE’ shows the means of square error between the latent sharp image ‘O’ and deblur image ‘R’.

‘MSE’ will be calculated by using Eq. (11) as shown below. The m and n symbols are used to show the height and breadth of the original sharp image.

$$ {{MSE}}\left( {O, \, R} \right) \, = \left( \frac{1}{mn} \right) \mathop \sum \limits_{i = 1}^{m} \mathop \sum \limits_{i = 1}^{n} \left( {O - R} \right)^{2} $$
(11)

The PSNR value is a very common performance measure for evaluating the quality of the restored images but the problem with the PSNR value is that it ignores the characteristics of the naked eye visual system as the eyes are extra sensitive to brightness levels as compared to chromatic aberration. Therefore, we also use the value of SSIM along with PSNR value to reflect the subjective feel of a human being’s eyes.

The SSIM of the restored image is computed as follows:

$$ {{SSIM}}\left( {O,R} \right) \, = \frac{{\left( { 2\Omega _{O}\Omega _{R} + C_{1} } \right) \left( {2 \beta_{OR} + C_{2} } \right) }}{{\left( {\Omega _{O}^{2} +\Omega _{R}^{2} + C1} \right)\left( {\beta_{O}^{2} + \beta_{R}^{2} + C_{2} } \right)}} $$
(12)

where the symbols \(\Omega _{O} \;{{and}}\; \Omega _{R}\) shows the mean values, the symbols \(\beta_{O}^{{}} \;{{and}}\;\beta_{R}^{{}}\) are used to represent the variance values and \( \beta_{OR} \) shows the covariances of original and restored sharp images while ‘C1' and 'C2' shows the constant values.

The Error ratio is used for the comparison of the performance of proposed algorithm by computing the standard deviation between the images recovered with estimated blur kernel and true blur kernel. The Error ratio will be computed by the following equation:

$$ R_{{{{Error}}}} = \frac{{{{STD}}\left( {I_{EK} , I_{R } } \right)}}{{{{STD}}\left( {I_{RK} , I_{R } } \right)}} $$
(13)

where STD is used to present the standard deviation while IR represents the Real blur image, IEK shows the sharp image that is recovered using the estimated kernel and IRK shows image estimated with true kernel.

5.2 Results of Synthetically Blurred Image Data

The SEBR method is evaluated on a standard benchmark dataset developed by Levin et. al. [35] for synthetically blur images. The images restored with the proposed technique are compared to the outputs of other up-to-date techniques used for image deblurring. This dataset contains a total of 32 blur images, consisting of 4 ground truth images of size of 255 × 255, 8 images are the ground truth kernels, whose sizes range from kernel size of 13 × 13 to size of 27 × 27.

Table 1 shows the obtained deblurring results of SEBR and other state-of-the-art techniques used for deblurring [11, 15, 19, 25, 28, 35, 46]. We have used the images from dataset of Levin et al. [35]. Output images are compared based on average values of PSNR, SSIM, error ratio, and the computational time used by deblurring process. Our proposed technique Scharr Edge Based Regularization (SEBR) outperforms the existing blind image deblurring techniques in terms of higher values of PSNR and SSIM values except for the method in [15]. However, the proposed SEBR is able to improve the computational cost as compared to the technique proposed in [15]. The ability of SEBR to preserve both the narrow and wider boundary lines during the kernel estimation helps in better recovery of edges from the blurred image that results in far better quality deblur image.

Table 1 Evaluation of output images on dataset of Levin et al.’[35] using mean values of PSNR, SSIM, Error Ratios and time of computation
Full size table

In Fig. 3, visual results obtained by the proposed technique SEBR are compared against the several up-to-date techniques used for image restoration [11, 15, 19, 25] According to the visual evaluations, it is inferred that the SEBR method has not just only deblurred the degraded image but it has also sharpened the weak edges of objects in an image. Also, the time of deblurring with SEBR is lower in comparison to other image recovery algorithms. The images in (b) and (c) are better but it has some regions having high level of blurriness. It can also be detected from the output that the image recovered by (d) is still blurred and has also distorted the significant information at the top portion of restored image. However, the deblurred image by (e) is a little improved but it has over sharped regions and high ringing artifacts been shown. The image (f) shows that the restored image by SEBR appears to be clearer and its PSNR value is also high as compared to the other methods.

Fig. 3
figure 3

Results of deblurring an image from the dataset of Levin et al. [35]with 27 × 27 sizes of Kernel a Original blur image, b Deblur by Jinsha et al. [15], c deblur by Yang et al. [25], d deblur by Cho and Lee [11], e deblur by Xu et al.[19] and f deblur by SEBR

Full size image

5.3 Results of Real Blur Data

The proposed deblurring technique SEBR is also evaluated on the real blur data for comparing its performance with other baseline techniques used for deblurring images from the real world. Real blur image datasets do not have ground truth images; therefore, they are only evaluated on the basis of the visual quality of recovered images. It is also significant to mention that the same parameters are set for all the algorithms for a fair comparison of results. SEBR technique is applied to real-world images having a uniform or non-uniform blur and images taken in low lightning having saturated regions.

5.3.1 Deblurring Results on Uniformly Blur Images

In Fig. 4, the proposed framework of SEBR is evaluated on an example image of a toy having uniform blur, and the resultant deblurred outputs are compared with other state-of-the-art techniques [11, 15, 19, 25]. It is observed that images deblurred by techniques in (b) and (e) are fine enough but it has ringing artifacts near the strong edges. Similarly, the deblurred image in (d) is still blurry. The image in (c) is fully distorted as it has over-smoothed the local textures of objects. However, images deblurred with SEBR have very small effects of ringing and it has also preserved the local texture and contrast of the image during the elimination of blurriness.

Fig. 4
figure 4

Results of deblurring on toy image a Real blur image b deblur with Cho and Lee [11], c deblur image with Xu et al. [19], d deblur with Yang et al. [25], e deblur with Jinsha et al. [15], f deblur with SEBR

Full size image

In Fig. 5, the deblurring results of SEBR are compared with the results of [11, 15, 19, 46] for a uniform blur image. As it is shown that images restored by the deblurring technique proposed in (b) produce results that is better than other methods, but it has minor fake texture and some regions having blur. The resultant images in (c) and (e) seem to be fine to some extent but the local textures in the objects are over smoothed and the geometrical structure and edges of objects are also not preserved properly. The resultant image in (d) is over smoothed and is also not deblurred effectively. However, the image deblurred with SEBR has fewer ringing artifacts as compared to the other deblurring methods. Interestingly, SEBR is equally able to preserve the fine textures in images while enhancing the edges during the process of deblurring.

Fig. 5
figure 5

Results of deblurring an example building image a real uniform blur image, b deblur with by Jinsha et al. [15], c deblur with Yang et al. [25], d deblur with Xu et al. [19], e deblur with Cho and Lee [11], f deblur with SEBR

Full size image

In Fig. 6, the deblurring results of SEBR are compared with other image deblurring methodologies [11, 15, 19, 25] used for uniform blur removal. The visual results of the kernel estimated by other deblurring techniques have shown the usefulness of SEBR. The image deblurring technique used in Jinsha et. al [15] produces a distorted output and the content of the image is also not clear as indicated in (b). The results of deblurring displayed in fig (c) and (d) have strong ringing artifacts. The image deblur in (e) has ignored the smaller textures areas during the image restoration process and results in an over-smoothed and a bit blurry image. The blur convolution kernel estimated with SEBR has compact neighboring pixels and is also comparatively less sparse which helps in preserving the salient edges and boundary lines of objects in images and recovering a sharper image with very few ringing effects.

Fig. 6
figure 6

Results of deblurring on example Mukta image a image with uniform blur b deblur with Jinsha et al. [15], c deblur with Xu et al. [19], d deblur with Choo and Lee. [11], e deblur with Yang et al. [25], f deblur image with SEBR

Full size image

5.3.2 Deblurring Results on Images With Non-uniform Blur

For evaluating SEBR on images having non-uniform blur, the deblurred results are compared with other deblurring techniques used for non-uniformly blurry images [28, 31, 53, 64] as shown below in Fig. 7. It is observed that the image in fig (b) and (c) have strong effects of ringing. The image in (d) is still very blur and not clear. The deblur result of (d) is better but its computational time is large while the image recovered with SEBR generates results that are comparable to deblurring results in (d) having rich textures and very sharp boundaries.

Fig. 7
figure 7

Results of deblurring on image with non-uniform blur a image having non-uniform blur, b deblur with krishnan et al. [28], c deblur with Hirsh et al. [31], d deblur Whyte et al. [64]. e deblur with Zhao et al. [53], f Deblur with SEBR

Full size image

Another example of real blur image having non-uniform blur is shown in Fig. 8. The proposed method of SEBR is compared with other techniques of non-uniform deblurring [28, 31, 53, 64]. It is shown from the results that the deblur images in (b) and (c) are not clear. The deblur result in (c) is a bit oversmoothed and lacks sharp edges. The deblurring result in (d) has blur regions in the center parts. The image recovered with the proposed SEBR as shown in fig (e) has fine textures with better contrast and it has very few ringing artifacts in just some areas of the output image.

Fig. 8
figure 8

Deblurring results on the real image with non-uniform blur a image with non-uniform blur, b deblur with krishnan et al. [28], c deblur with Hirsh et al. [31], d deblur Whyte et al. [64]. e deblur with Zhao et al. [53], f Deblur with SEBR

Full size image

Figure 9 shows an example image of real blur RGB image having non-uniform blur. The deblurring results of SEBR are compared with other methodologies used for uniform blur removal from real blur images [28, 31, 53, 64]. The deblurring results in figure (b) and (c) shows the kernel fails to recover the image. The result displayed in (d) is still very blurry. The deblurred results shown in fig. (e) have ringing artifacts near the boundary lines but the image recovered with the estimated kernel of SEBR is clearer and doesn’t have any artifacts near the strong boundary lines.

Fig. 9
figure 9

Results of deblurring on example image having non-uniform blur a image with non-uniform blur, b deblur with krishnan et al. [28], c deblur with Hirsh et al. [31], d deblur Whyte et al. [64]. e deblur with Zhao et al. [53], f Deblur with SEBR

Full size image

5.3.3 Deblurring Real Blur Images with Saturated Regions

The proposed Scharr Edge based regularization (SEBR) based deblurring gives favorable deblurring results on images captured in the real world having saturated blur areas. It is very hard to estimate the convolution kernel from the blur images having saturated areas. The already proposed state-of-the-art deblurring methodologies [11, 38, 65, 66] does not give quality results on real-world data having lots of saturated regions or images taken in low illumination. In sharp images the saturated regions are sparse but when the image becomes blurred these regions extend and look like blobs or streaks. The images taken in low lightning often have saturated regions.

Figure 10 shows a real blur image captured in low lightning having many saturated regions. For validation, the deblurring results of proposed SEBR technique is compared with other state of the art methodologies in [11, 38, 65, 66]. As the priors of these deblurring methods were basically developed to exploit the prominent edges in images in case of motion deblurring, these methods were not able to give good results. The technique in [38] was developed to handle a large amount of Gaussian noise in images so it is not effective on images having saturated areas as shown in fig. ( c). It is observed in fig. (b) and (d) that the method has limitations in estimating the latent image and shows many ringing artifacts. The image restored with SEBR based technique is clear and sharp. We notice that the deblurring technique in [65, 66] performs well because it was developed for detecting lightning streaks that come in images captured in low lightning images. The proposed method estimated the blur kernel well and gives favorable results without performing any additional preprocessing steps for the detection of light streaks.

Fig. 10
figure 10

Results of deblurring on images having saturated areas a image taken in low lightning, b deblur with Cho and Lee [11], c deblur with Xu et al. [38], d deblur with Hu et al. [66], e deblur with Chen et al. [65], f deblur with SEBR

Full size image

Figure 11 shows another real image taken in low illumination with many saturated regions. The output of SEBR deblurring technique is compared with other methodologies [11, 38, 65, 66] used for blur removal. As it is clearly seen in fig. (b) and fig.(c) that the saturated areas are very large, and the estimated kernel is not good enough to detect the light streaks properly. The deblurring results in fig. (d) and (e) performs well but still have a little blurry effect that’s dilating the white regions in image whereas the proposed SEBR method accurately estimates the kernel and gives better deblurring results which is sharper and clearer.

Fig. 11
figure 11

Deblurring results on images having lot of saturated areas a blurry image, b deblur with Cho and Lee [11], c deblur with Hu et al. [66], d deblur with Xu et al. [38], e deblur with Chen et al. [65], f deblur with SEBR

Full size image

6 Conclusion

An effective and efficient blind image deblurring technique based on Scharr edge-based regularization is proposed for estimating the blur convolution kernel that can restore a high-quality sharp image with minimum cost function. The Scharr edge based (SEB) gradient operator is used in combination with the normalized regularization that highly optimizes the salient edge extraction from blurry images because of its perfect rotation symmetry property and scale invariance, and results in an accurate estimation of blur kernel. It is concluded from the results that the kernel function estimated by SEBR is precise and preserves all the salient edges and local textures of objects with very few ringing artifacts as compared to other state-of-the-art deblurring methodologies. The effectiveness of SEBR is proved in terms of values of PSNR, SSIM, Error ratio, and costs of computation on synthetically blurred images. The comparison of experimental results demonstrates that SEBR is also very robust and equally effective in terms of visual quality on real blur image data having a uniform or non-uniform blur and low illumination images. The robustness of SEBR is the major improvement over existing techniques as it does not need to adjust the parameters for different types of blur data. In our future research, both global and local textures of blur image will be considered for better estimation of the convolved kernel and restoration of image.