Image Interpolation Based on Weighted and Blended Rational Function

Abstract

Conventional linear interpolation methods produce interpolated images with blurred edges, while edge directed interpolation methods make enlarged images with good quality edges but with details distortion for some cases. An adaptive rational-based algorithm for the interpolation of digital images with arbitrary scaling factors is proposed. In order to remove artifacts, we construct a new interpolation model with weight and blend, which are used for preserving the clear edge and detail. The proposed model is blended by basic rational interpolation model and three rotated rational models. The weight coefficients are determined by the edge information from different scale based on point sampling. Experimental results show that the proposed method produces images with high objective quality assessment value and good visual quality.

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

Image interpolation has a wide range of applications which aims to reconstruct a high resolution (HR) image from the low-resolution (LR) image. The most common interpolation methods are bilinear, bicubic, cubic spline algorithm, etc. [3, 6]. These conventional methods are the approximation of sinc function which corresponds to ideal filtering [14]. These methods have a relatively low complexity, but suffer from several types of visual degradation around “edges”.

To solve these problems, many adaptive interpolation algorithms have been developed [2, 4, 7–10, 13, 15–18]. These algorithms can be broadly divided into two categories: discrete method and continuous method. In discrete method, new edge-directed interpolation (NEDI) [8] estimates high resolution covariances form low resolution image based on the geometric duality; In [17], for a pixel to be interpolated two observation sets are defined in two orthogonal directions, and then fuse the directional interpolation results by minimum mean square-error estimate; Zhang and Wu [18] develop a soft-decision interpolation method which is able to estimate missing pixels by groups instead of by pixels. These discrete algorithms which consider more adaptive image information can improve the visual effect. However, these methods deliver not a continuous function but a set of subpixel values which are not suitable for resampling after, for example, rotation [10], and they have a much higher computational complexity than conventional methods. Besides, these methods sometimes generate speckle noise or distortion of edges [18].

Once a digital image is converted into an interpolating continuous function, we can resample it to obtain resized and rotated images at a better precision [10]. In fact, the continuous methods create a HR image though constructing a interpolating patch. Hu and Tan [4] presents a method for preserving the contours or edges based on adaptive osculatory rational interpolation kernel function. Zhang et al. [16] constructs a fitting surface by using image data as constraints to reverse sampling process for improving fitting precision. However, these continuous methods all suffer form blurred edge in some ways. Recently, a bivariate rational interpolation with parameters based on the function values is studied in [12, 19]. The rational function has a simple and explicit expression, and compared with other methods, it can keep the natural attributes of image better. Because the rational model is suitable for resizing natural image, it performs the details well and a relative clear edge. Unfortunately, the single bivariate rational model(basic model (Fig. 2a)) is asymmetric and does not meet the structures of natural image. So it can produce zigzagging artifacts around the edge regions.

In this paper, we construct an adaptive interpolation function with weight and blend based on rational function model. To reduce the zigzagging edge generated by the basic model, we rotate the basic rational model 3 times to get 4 interpolating functions. The proposed model is weighted and blended by them. The weight coefficients which contain edge information are adaptively calculated by distance, gradient and difference quotient based on point sampling, and can be used to keep the edge attributions. Experiments show that the proposed approach performs better than conventional and discrete methods in preserving edge.

This paper addresses the problem of constructing an adaptive weighted rational function such that the resized image has better precision and visual quality. We use blending model not only to maintain the image natural attribution but also to preserve the structure of image, and the adaptive weight coefficients can preserve edge information from different aspects. Furthermore, point sampling can ensure that more information in a cell can be used to determined the weight coefficients.

The paper is arranged as follows. In Sect. 2, the proposed interpolated model based on basic rational model is introduced. Section 3 shows the performance of the method.

2 Description of Proposed Method

In this section, the interpolation function with unknown weight coefficients is proposed, and the key problem is to determine the weight coefficients. Then the weight coefficients are determined by different scale edge information of local image.

2.1 The Interpolation Model

Thought rational function has good features to maintain details, it suffers from some visual problems around edge region. The proposed model can preserve edge region well as well as detail region. Let $[i,i+1;j,j+1]$ be the interpolated region. The proposed interpolation model based on rational spline function is showed in Fig. 1. The rectangle region surrounded by 4 black points is the interpolated region. All 16 points within the interpolation model are involved in the interpolation. And Fig. 2 shows the decomposition of the proposed model. The proposed model is weighted and blended by the 4 submodels. In Fig. 2, (a) means the basic rational spline interpolation model; (b) represents the $90^{\circ }$ counterclockwise rotation of (a) in Fig. 1 model, and the rotation center is the interpolation region; in the same way, (c) and (d) are 180$^{\circ }$ and 270$^{\circ }$ counterclockwise of (a). Figure 3 is another expression of Fig. 2, and (a), (b), (c) and (d) correspond respectively. Let the basic rational spline interpolation model (a) denotes $P_{1}(x,y)$, then (b), (c) and (d) are denoted $P_{2}(1-y,x)$, $P_{3}(1-x,1-y)$ and $P_{4}(y,1-x)$ respectively. All these four submodels can produce the same patch $[i,i+1;j,j+1]$. The proposed weighted and blended rational function $P_{i,j}(x,y)$ is expressed in Eq. 1.

$$\begin{aligned} \begin{array}{l} P_{i,j}(x,y)=aP_{1}(x,y)+bP_{2}(1-y,x)\\ \quad +cP_{3}(1-x,1-y)+dP_{4}(y,1-x), \end{array} \end{aligned}$$

(1)

where $a$, $b$, $c$ and $d$ are unknown weight coefficients.

Fig. 1.

The interpolation model

Fig. 2.

The rotation model: (a) Basic model; (b), (c) and (d) are 90, 180 and 270$^{\circ }$ counterclockwise rotation of (a) respectively.

Fig. 3.

The rotation model: (a) $P_{1}(x,y)$; (b) $P_{2}(1-y,x)$; (c) $P_{3}(1-x,1-y)$; (d) ${P_{4}(y,1-x)}$.

Now the basic rational spline function (Fig. 2(a)) is given. A bivariate rational interpolation with parameters based on the function values is constructed in [12, 19]. Let $P_{i,j}(x,y)$ be a bivariate function defined in the region $[i,i+1;j,j+1]$. Denote the pixel value by $I_{i,j}$. For any point $(x,y)\in [i,i+1;j,j+1]$, the bivariate rational interpolating function $P_{i,j}(x,y)$ can be expressed as

$$\begin{aligned} P_{i,j}(x,y)=\sum _{r=0}^2\sum _{s=0}^2 \omega _{r}(x,\alpha _{i})\omega _{s}(y,\beta _{j})I_{i+r,j+s}, \end{aligned}$$

(2)

where

$$\begin{aligned} \omega _0(t,\delta )= & {} \frac{(1-t)^2(\delta +t)}{(1-t)\delta +t},\\ \omega _1(t,\delta )= & {} \frac{t(1-t)\delta +3t^2-2t^3}{(1-t)\delta +t},\\ \omega _2(t,\delta )= & {} \frac{-t^2(1-t)}{(1-t)\delta +t}. \end{aligned}$$

Considering the basic model, 9 points $I_{i+r,j+s}$,$(r,s=0,1,2)$ are used to construct the patch $P_{1}(x,y)$ which crosses the 4 black points, and these 9 points have different basis functions. However, it would suffer from blurred edges. There are two main reasons. On the one hand, for a nature image, it will result some artifacts around edges because of its asymmetry; on the other hand, the function is constructed by $x$-direction first and then $y$-direction, which results the advantage on $x$-direction [19]. The proposed weighted and blended interpolation model can refrain from these two factors. Obviously, Fig. 1 contains 16 points and the interpolated region is located in the center of the model. And it is easy to know that the disadvantage of $y$-direction is eliminated through the rotation. For example, there is a horizontal direction edge marked in red as shown in Fig. 1. And Fig. 3 shows the changes of the direction of the red edge during rotation. We can see that in (a) and (c), the red edge is still horizontal, while it rotates to vertical direction in (b) and (d) models. It means that the proposed interpolation model balances the effect of different edge directions.

Then the edge information is used as constraints to determine the weight coefficients. It would not only be able to ensure good visual perception of detail areas, but also make the edge regions avoid zigzagging.

2.2 Adaptive Weights

From discussed above, we know that the weight coefficients are vital to the interpolation effect. And the weight coefficients should reflect local structure information and natural attributions of image. Following, the determination of the unknown in Eq. 1 is discussed. Adaptive interpolation means that the way the neighboring pixels influence the value of the interpolated pixel depends on local properties [11]. Thus the weight coefficients should be different when the construction of near pixels is different. In the proposed model, the $4\times 4$ pixels region is divided into 4 overlapping subregions, and the contribution of each subregion to the intermediate patch determines the weight coefficients. The $1\times 1$ rectangular region composed of four pixels can be seen as a basic cell. It can be seen from Fig. 4 that every subregion can be regraded as four basic cells, then the whole interpolation model contains 9 cells denoted as $f_{s}, (s=1,\cdots ,9)$. If we know the relationship between the 8-connected neighbors cells and intermediate cell $f_{5}$, it is easy to determine the four weight coefficients in Eq. 1.

For all $s=1,2,3,4,6,7,8,9$, let $w_{s}$ represent the impact factors between the patch $f_{s}$ and $f_{5}$. In Fig. 4, intuitively, $f_{5}$, $f_{6}$, $f_{8}$ and $f_{9}$ constitute the subregion $P_{1}$, and $f_{5}$ is the interpolated region. Thus we consider the relationship between $f_{6}$, $f_{8}$, $f_{9}$ and $f_{5}$ determines the contribution of the subregion $P_{1}$ to the intermediate interpolated region. And other subregions are in similar way. Then the $a$, $b$, $c$ and $d$ can be expressed as follows:

(3)

where $W=w_{1}+w_{2}+w_{7}+w_{9}+2w_{2}+2w_{4}+2w_{6}+2w_{8}$, which means normalization of the weight coefficients.

Fig. 4.

Every cell can be regard as a point.

Now we discuss how to determine $w_{s}$. It is not easy to measure the relationship between two patches, while there are more approaches to measure the relationship between two points. Therefore a cell is regarded as a point such that the relationship between two cells can be approximately replaced by two points. Usually the pixel value can be regarded as the sample value of a continuous patch [16]. The approximate point sampling value of a cell is expressed as

$$\begin{aligned} \iint _{A_{ij}}{P_{i,j}(x,y)dxdy}=\overline{P_{ij}}\times S_{ij}, \end{aligned}$$

where $A_{ij}$ is a cell region, and $S_{ij}$ is the area of $A_{ij}$, $P_{i,j}(x,y)$ represents function of a patch. Thus the point sampling value of patch can be approximately achieved by $\overline{P_{ij}}$. And the point values of $f_{1}$ to $f_{9}$ can be calculated. It is converted to a problem of computing the relationship between the intermediate point and the 8-connected neighbors points respectively.

In a natural image, distance, gradient and different quotient can measure the relationship between different pixels from different attributions. The distance measures the pixel space relationship, and the local gradient shows the edge information in a cell, and the difference quotient indicates the edge information among the whole model region. Thus, the local gradient and difference quotient can describe the edge from different scale. We focus on three factors distance, gradient, and difference quotient to determine the $w_{s}$, as shown in Eq. 4

$$\begin{aligned} w_{s}=F(\textit{distance},\textit{gradient},\textit{difference quotient}). \end{aligned}$$

(4)

Based on the weight expression of bilateral filter which contains distance and gray value, we construct a trilateral weighted expression. For all $s=1,2,3,4,6,7,8,9$, the Eq. 4 can be represented as

$$\begin{aligned} w_{s}=e^{-\frac{w_{s}^{1}}{{h_{1}^2}}}\times e^{-\frac{w_{s}^{2}}{{h_{2}^2}}} \times e^{-\frac{w_{s}^{3}}{{h_{3}^2}}}, \end{aligned}$$

(5)

where $w_{s}^{1}$, $w_{s}^{2}$ and $w_{s}^{3}$ represent Distance, Gradient, and Difference Quotient respectively, and $h_{1}$,$h_{2}$ and $h_{3}$ are adjusting parameters. Then the unknown $w_{s}^{1}$, $w_{s}^{2}$ and $w_{s}^{3}$ are calculated.

First, the weight coefficients depend on the distance between each point and the intermediate point. If $f_{s}$ is closer to the intermediate point, the weight coefficient will be greater. It is shown as

$$\begin{aligned} w_{s}^{1}=(x_{5}-x_{s})^2+(y_{5}-y_{s})^2, \end{aligned}$$

(6)

where $x$ and $y$ are the local coordinates of these points.

Second, the weight coefficients also depend on the local gradients. The local gradient is expressed as

$$\begin{aligned} w_{s}^{2}=\left| f'_{x}\right| ^2+\left| f'_{y}\right| ^2, \end{aligned}$$

(7)

where the $f'_{x}$ and $f'_{y}$ are the local gradient of a cell around the interpolation patch. In essence the smaller the local gradient of a pixel is, the more influence it should have on the intermediate pixel [11]. Obviously, the small scale edge information is considered due to the gradients as one of the factors in a cell.

Third, the second difference quotient is taken into account. If there is an edge along the vertical direction, $f_{2}$ and $f_{8}$ should have the closest connection to $f_{5}$. They are defined as

$$\begin{aligned} w_{2}^{3}=\left| 2f_{5}-f_{2}-f_{8}\right| ^2, \end{aligned}$$

(8)

$w_{1}^{3}$, $w_{3}^{3}$ and $w_{4}^{3}$ are defined in the same way. And $w_{6}^{3}=w_{4}^{3}$, $w_{7}^{3}=w_{3}^{3}$, $w_{8}^{3}=w_{2}^{3}$, $w_{9}^{3}=w_{1}^{3}$. This factor reflects the large scale edge information in the whole region of $4\times 4$, and the smaller the $w_{s}^{3}$ is, the more effect it should have on the $f_{5}$.

All the unknown factors are calculated. Substituting Eq. 5 into Eq. 3 gets the adaptive weights. Since the quality around the edges plays an important role in the visual effect of an image, $P_{i,j}(x,y)$ should reflect the characteristics around the edges as well as possible. In Eq. 7, the local gradient which infers the local small scale edge is involved, and in Eq. 8 all the pixels in the whole window are contained to determine weight coefficients which means that the large scale edge is considered. Thus the final weight coefficients $a$, $b$, $c$ and $d$ are adaptive by edge.

Fig. 5.

Benchmark images (Color figure online).

Table 1. PSNR and SSIM results of the reconstructed HR images.

Fig. 6.

Comparison on Fence. (a) Original image, (b) NEDI, (c) SME, (d) SAI, (e) RSAI, (f) Proposed method.

Fig. 7.

Comparison on Wall. (a) Original image, (b) NEDI, (c) SME, (d) SAI, (e) RSAI, (f) Proposed method.

3 Experiments

The proposed method is compared with recent interpolation algorithms: new edge-directed interpolation (NEDI) [8], soft-decision interpolation (SAI) [18], sparse mixing estimators (SME) [9] and robust soft-decision interpolation (RSAI) [5]. All experiments are performed with softwares provided by the authors of these algorithms¹. We have used 7 images as our benchmark images (Fig. 5). We downsmaple these HR images to get the corresponding LR images. Table 1 gives the PSNRs and SSIMs generated by all algorithms for the images in Fig. 5. It can be seen that the proposed method has a highest average PSNR and SSIM among all the algorithms.

Fig. 8.

Comparison on Girl. (a) Original image, (b) NEDI, (c) SME, (d) SAI, (e) RSAI, (f) Proposed method.

Fig. 9.

Comparison on Barbara. (a) Original image, (b) NEDI, (c) SME, (d) SAI, (e) RSAI, (f) Proposed method.

Figures 6,  7, 8 and 9 compare the interpolated images obtained by different algorithms. These images are cropped by red rectangle in Fig. 5. Figure 8 shows the edge information, and the others show details. For Fig. 8, all the algorithms perform similar results in edge region. We can see that NEDI suffers from some noisy interpolation artifacts (Figs. 7, 9, 6(b) because of the fixed interpolation window. And SAI method also suffers from noisy artifacts in Figs. 9(d) and 6(d). RSAI performs better than SAI but produces some unconnected stripes (Figs. 7(e) and 6(e)). Although SME has similar visual quality with the proposed method, the objective quality assessment value is lower than the proposed algorithm. Therefore, the proposed method can keep the edge region well, and it can perform better detail areas than other algorithms. Moreover, we also compared the proposed method with the methods in papers DFDF [17], KR [13], INEDI [1], the proposed method has better vision quality and objective quality assessment value as well.

4 Conclusions

We propose an adaptive image interpolation method using rational function. The rational function is weighted and blended to remove artifacts. The edge information is used as constraints to determinate the weights adaptively. The new method has the advantage in that it can easily zoom the image into multiples. Our method can perform well on PSNRs and SSIMs. Furthermore, the proposed method produces clean edges and fine details.