1 Introduction

The problem of image interpolation consists of reconstructing a function u(x, y), \(x, y \in {\mathbb {R}}\) that agrees with the known samples on a uniform square grid s(m, n), \(x, y \in {\mathbb {Z}}\). One of the basic operations in image processing and computer graphics, interpolation is required when the sampling points do not match the original pixel grid, image upscaling and texture sampling being two prime examples. Implementations of various interpolation algorithms can be found both in software (image processing applications and libraries) and hardware (graphics processing units). The computer graphics industry demonstrates significant interest in high-quality, high-performance interpolation methods [18].

Linear interpolation is an important class of methods that reconstruct u(x, y) by convolving the image s(m, n) with an interpolation kernel \(\psi (x, y)\):

$$\begin{aligned} u(x, y) = \sum _{m = -\infty }^{+\infty } \sum _{n = -\infty }^{+\infty } s(m, n) \psi (x - m, y - n). \end{aligned}$$

The kernel can be constructed as a product of two one-dimensional kernels: \(\psi (x, y) = \psi (x) \psi (y)\). Such separable kernels are often preferred by virtue of their computational convenience since in this case interpolation can be performed in two one-dimensional steps along each axis. Many established methods such as bicubic [15] and B-spline interpolation [4] belong to this class. Non-separable kernels have also been studied in [26].

Typical criteria employed for the design of interpolation kernels include smoothness, an exact representation of Taylor expansion terms, similarity to the ideal low-pass filter in Fourier space, or performance for a particular model of the Fourier spectrum [23]. However, these properties only indirectly correlate with the perceived image quality. We explore another avenue by trying to directly quantify the perceived artifacts.

Linear interpolation methods produce several types of undesirable effects, the most notable of which are the following.

  • Blurriness. Overly smooth transitions in areas where the sharp transitions were present in the original image, usually around object edges. This artifact type is especially noticeable for linear interpolation (\(\psi (x) = 1 - |x |, |x |\le 1\)).

  • Ringing. Oscillating kernels produce noticeable halos around hard edges.

  • Staircasing or blocking. The square pixel lattice coupled with the kernel separation process introduces anisotropic effects. For example, the isolevel contours of diagonal edges on the interpolated image form meandering, staircase-like curves instead of straight lines.

Blurriness and ringing artifacts are inevitable within the framework of linear interpolation. Reducing blurriness typically increases ringing as the kernel becomes more oscillating in order to increase edge acuity. We focus on the third type of artifact, staircasing or blocking. The importance of contours in visual perception is universally acknowledged in the field of human vision [2, 24]. Image quality assessment methods also make heavy use of edge information in the form of gradients [17, 32] or phase congruency [37]. Minimizing the distortions of edge contours is therefore very important for high-quality image interpolation.

A number of edge-directed interpolation techniques have been proposed [5, 10, 12, 16, 31, 36, 39] in order to supress the artifacts arising near sharp edges. While effective to varying degrees, these techniques are more complex and computationally expensive than linear interpolation as they include special steps in order to take edge directions into account. Four edge-directed interpolation methods benchmarked in [33] were slower than bicubic interpolation by factors ranging from 119 to 932.

The main contribution of this work is demonstrating that the staircase effect can be greatly reduced while staying within the linear interpolation framework. This goal is achieved by optimizing the kernel with respect to an appropriately defined quality metric. Most calculations are performed symbolically using the Mathematica computer algebra system [29].

2 Proposed Approach

We start by postulating a set of conditions that a good kernel should satisfy.

  • Interpolation: in order to agree with the existing samples, \(\psi (x)\) must be zero at any integer x except at \(x = 0\) where \(\psi (0) = 1\).

  • Continuity.

  • Partition of unity: \(\sum _{k = -\infty }^{+\infty } \psi (x - k) = 1\).

  • Exact representation of the linear signal term: \(\sum _{k = -\infty }^{+\infty } k \psi (x - k) = x\).

These conditions can be further strengthened by requiring the continuity of derivatives or the exact representation of higher-order terms of the Taylor expansion of the underlying continuous signal. While these additional properties are considered desirable from a theoretical perspective, it will be shown in Sect. 3 that they do not necessarily translate to improved image quality in practice. We include the continuity of the first derivative as an optional constraint.

We have chosen a separable piecewise-polynomial kernel form for simplicity and computational efficiency. The kernels come in even and odd variants corresponding to integer and half-integer interval endpoints respectively:

$$\begin{aligned} \psi (x) = \sum _{j = 0}^{p} c_{i, j} (|x |- i)^j, i = \lfloor |x |+ \Delta \rfloor , \end{aligned}$$

where \(\Delta \) is zero for even and 1/2 for odd kernels. The kernels are defined on the interval \((-r, r)\) and are zero elsewhere. We shall denote our kernels with given r and p by \(K_{(r, p)}\) and those satisfying the additional smoothness constraint (\(C^1\)-continuity) by \(K_{(r, p)_S}\).

The first two constraints translate to the following equations.

  • Interpolation: \(c_{i, 0} = \llbracket i = 0 \rrbracket \), where \(\llbracket P \rrbracket \) is the Iverson bracket taking the value 1 if the statement P is true and zero otherwise. Since \(c_{i, 0}\) are identical for all kernels satisfying the interpolation constraint, we shall omit their values when presenting \(c_{i, j}\) matrices.

  • Continuity: \(\sum _{j = 0}^{p} c_{i, j} = 0\) for any i.

The equations for the last two constraints are obtained by evaluating

$$\begin{aligned} \sum _{k = 1 - r}^{r} \psi (x - k) \quad \mathrm {and} \quad \sum _{k = 1 - r}^{r} k \psi (x - k) \end{aligned}$$

for \(x \in [0, 1)\) or \(x \in [0, 1/2)\) depending on the kernel type using the Simplify command and collecting the coefficients for each polynomial term. A general solution of the linear system incorporating all constraints can then be found by Solve. The number of free variables for different general solutions is reported in Table 1. The case \(r = 1\) has a unique solution corresponding to linear interpolation (\(\psi (x) = 1 - |x |\)) for any p (this follows from the linear term condition alone). The unique solution \(K_{(2, 3)_S}\) corresponds to Keys’ cubic kernel [15], and \(K_{(3/2, 2)}\) to Dodgson’s kernel [8].

Table 1 The number of free variables for different kernels. 0 corresponds to a unique solution and dash denotes an overconstrained system
Full size table

The general solutions for various values of r and p without the smoothness constraint are:

$$\begin{aligned}&K_{(2, 2)}:&K_{(3/2, 2)}: \\&\begin{bmatrix} c_{0, 2} \\ c_{1, 1} \\ c_{1, 2} \\ \end{bmatrix} = \begin{bmatrix} -1 &{} -1 \\ -1 &{} -1 \\ \quad 1 &{} \quad 1 \\ \end{bmatrix} \begin{bmatrix} 1 \\ c_{0, 1} \\ \end{bmatrix},&\begin{bmatrix} c_{0, 1} \\ c_{0, 2} \\ c_{1, 1} \\ c_{1, 2} \\ \end{bmatrix} = {1 \over 2} \begin{bmatrix} \quad 0 \\ -4 \\ -1 \\ \quad 2 \\ \end{bmatrix}, \end{aligned}$$
$$\begin{aligned}&K_{(2, 3)}:&K_{(3/2, 4)}: \\&\begin{bmatrix} c_{0, 3} \\ c_{1, 1} \\ c_{1, 2} \\ c_{1, 3} \\ \end{bmatrix} = {1 \over 3} \begin{bmatrix} -3 &{} -3 &{} -3 \\ -4 &{} -4 &{} -1 \\ \quad 3 &{} \quad 3 &{} \quad 0 \\ \quad 1 &{} \quad 1 &{} \quad 1 \\ \end{bmatrix} \begin{bmatrix} 1 \\ c_{0, 1} \\ c_{0, 2} \\ \end{bmatrix},&\begin{bmatrix} c_{0, 1} \\ c_{0, 3} \\ c_{0, 4} \\ c_{1, 1} \\ c_{1, 2} \\ c_{1, 3} \\ c_{1, 4} \\ \end{bmatrix} = {1 \over 2} \begin{bmatrix} \quad \,\,0 &{} \quad 0 \\ \quad \,\,0 &{} \quad 0 \\ \!-16 &{} -8 \\ \,\,-1 &{} \quad 0 \\ \quad \,\,0 &{} -1 \\ \quad \,\,0 &{} \quad 0 \\ \quad \,\,8 &{} \quad 4 \\ \end{bmatrix} \begin{bmatrix} 1 \\ c_{0, 2} \\ \end{bmatrix}, \end{aligned}$$
$$\begin{aligned}&K_{(2, 4)}:&K_{(5/2, 2)}: \\&\begin{bmatrix} c_{0, 4} \\ c_{1, 1} \\ c_{1, 2} \\ c_{1, 3} \\ c_{1, 4} \\ \end{bmatrix} = {1 \over 3} \begin{bmatrix} -3 &{} -3 &{} -3 &{} -3 \\ -5 &{} -5 &{} -2 &{} -1 \\ \quad 6 &{} \quad 6 &{} \quad 3 &{} \quad 3 \\ -4 &{} -4 &{} -4 &{} -5 \\ \quad 3 &{} \quad 3 &{} \quad 3 &{}\quad 3 \\ \end{bmatrix} \begin{bmatrix} 1 \\ c_{0, 1} \\ c_{0, 2} \\ c_{0, 3} \\ \end{bmatrix},&\begin{bmatrix} c_{0, 1} \\ c_{1, 1} \\ c_{1, 2} \\ c_{2, 1} \\ c_{2, 2} \\ \end{bmatrix} = {1 \over 4} \begin{bmatrix} \quad 0 &{} \quad 0 \\ -6 &{} -2 \\ \quad 4 &{} \quad 0 \\ \quad 2 &{} \quad 1 \\ -4 &{} -2 \\ \end{bmatrix} \begin{bmatrix} 1 \\ c_{0, 1} \\ \end{bmatrix}, \end{aligned}$$
$$\begin{aligned}&K_{(3, 2)}:&K_{(5/2, 3)}:\\&\begin{bmatrix} c_{0, 2} \\ c_{1, 2} \\ c_{2, 1} \\ c_{2, 2} \\ \end{bmatrix} = \begin{bmatrix} -1 &{} -1 &{} \quad 0 \\ \quad 0 &{} \quad 0 &{} -1 \\ -1 &{} -1 &{} -1 \\ \quad 1 &{} \quad 1 &{} \quad 1 \\ \end{bmatrix} \begin{bmatrix} 1 \\ c_{0, 1} \\ c_{1, 1} \\ \end{bmatrix},&\begin{bmatrix} c_{0, 1} \\ c_{0, 3} \\ c_{1, 2} \\ c_{1, 3} \\ c_{2, 1} \\ c_{2, 2} \\ c_{2, 3} \\ \end{bmatrix} = {1 \over 4} \begin{bmatrix} \quad \,\,0 &{} \quad 0 &{}\quad \,\, 0 \\ \quad \,\,0 &{}\quad 0 &{}\quad \,\, 0 \\ \quad \,\,4 &{} \quad 0 &{}\quad \,\, 0 \\ -24 &{} -8 &{} -16 \\ \,\,-1 &{} \quad 0 &{} \,\,-2 \\ \,\,-4 &{} -2 &{} \quad \,\,0 \\ \quad 12 &{} \quad 4 &{} \quad \,\, 8 \\ \end{bmatrix} \begin{bmatrix} 1 \\ c_{0, 2} \\ c_{1, 1} \\ \end{bmatrix}, \end{aligned}$$
$$\begin{aligned}&K_{(3, 3)}: \\&\begin{bmatrix} c_{0, 3} \\ c_{1, 3} \\ c_{2, 1} \\ c_{2, 2} \\ c_{2, 3} \\ \end{bmatrix} = {1 \over 5} \begin{bmatrix} -5 &{} -5 &{} -5 &{} \quad 0 &{} \quad 0 \\ \quad 0 &{}\quad 0 &{}\quad 0 &{} -5 &{} -5 \\ -7 &{} -7 &{} -2 &{} -6 &{} -1 \\ \quad 1 &{}\quad 1 &{}\quad 1 &{}\quad 3 &{}\quad 3 \\ \end{bmatrix} \begin{bmatrix} 1 \\ c_{0, 1} \\ c_{0, 2} \\ c_{1, 1} \\ c_{1, 2} \\ \end{bmatrix}, \end{aligned}$$
$$\begin{aligned}&K_{(5/2, 4)}: \\&\begin{bmatrix} c_{0, 3} \\ c_{0, 3} \\ c_{1, 3} \\ c_{1, 4} \\ c_{2, 1} \\ c_{2, 2} \\ c_{2, 3} \\ c_{2, 4} \\ \end{bmatrix} = {1 \over 4} \begin{bmatrix} \quad \,\,0 &{} \quad 0 &{} \quad 0 &{}\quad \,\, 0 &{}\quad \,\, 0 \\ \quad \,\,0 &{} \quad 0 &{}\quad 0 &{} \quad \,\, 0 &{}\quad \,\, 0 \\ \!-24 &{} -8 &{} -2 &{} -16 &{} \quad \,\, 0 \\ \,\,\,16 &{}\quad 0 &{} \quad 0 &{} \quad \,\, 0 &{} \!-16 \\ \,\,-1 &{} \quad 0 &{} \quad 0 &{} \,\,-2 &{} \quad \,\, 0 \\ \quad \,\,0 &{} -2 &{} \quad 0 &{} \quad \,\, 0 &{} \,\,-4 \\ \,\,\,12 &{}\quad 4 &{}\quad 1 &{} \quad \,\, 8 &{} \quad \,\, 0 \\ \!-16 &{} \quad 0 &{} -2 &{}\quad \,\, 0 &{} \,\,\,16 \\ \end{bmatrix} \begin{bmatrix} 1 \\ c_{0, 2} \\ c_{0, 4} \\ c_{1, 1} \\ c_{1, 2} \\ \end{bmatrix}. \end{aligned}$$

The general solutions with the smoothness constraint are:

$$\begin{aligned}&K_{(2, 3)_S}:&K_{(2, 4)_S}:\\&\begin{bmatrix} c_{0, 1} \\ c_{0, 2} \\ c_{0, 3} \\ c_{1, 1} \\ c_{1, 2} \\ c_{1, 3} \\ \end{bmatrix} = {1 \over 2} \begin{bmatrix} \quad 0 \\ -5 \\ \quad 3 \\ -1 \\ \quad 2 \\ -1 \\ \end{bmatrix},&\begin{bmatrix} c_{0, 1} \\ c_{0, 3} \\ c_{0, 4} \\ c_{1, 1} \\ c_{1, 2} \\ c_{1, 3} \\ c_{1, 4} \\ \end{bmatrix} = {1 \over 2} \begin{bmatrix} \quad 0 &{}\quad 0 \\ -7 &{} -4 \\ \quad 5 &{}\quad 2 \\ -1 &{} \quad 0 \\ -3 &{} -2 \\ \quad 9 &{}\quad 4 \\ -5 &{} -2 \\ \end{bmatrix} \begin{bmatrix} 1 \\ c_{0, 2} \\ \end{bmatrix}, \end{aligned}$$
$$\begin{aligned}&K_{(3, 3)_S}:&K_{(3, 4)_S}:\\&\begin{bmatrix} c_{0, 1} \\ c_{0, 3} \\ c_{1, 1} \\ c_{1, 2} \\ c_{1, 3} \\ c_{2, 1} \\ c_{2, 2} \\ c_{2, 3} \\ \end{bmatrix} = {1 \over 4} \begin{bmatrix} \quad \,\,0 &{}\quad 0 \\ \,\,-4 &{} -4 \\ \!-12 &{} -4 \\ \quad 19 &{}\quad 6 \\ \,\,-7 &{} -2 \\ \quad \,\,5 &{}\quad 2 \\ \!-10 &{} -4 \\ \quad \,\,5 &{}\quad 2 \\ \end{bmatrix} \begin{bmatrix} 1 \\ c_{0, 2} \\ \end{bmatrix},&\begin{bmatrix} c_{0, 1} \\ c_{0, 4} \\ c_{1, 1} \\ c_{1, 3} \\ c_{1, 4} \\ c_{2, 1} \\ c_{2, 2} \\ c_{2, 3} \\ c_{2, 4} \\ \end{bmatrix} = {1 \over 4} \begin{bmatrix} \quad \,\,0 &{}\quad \,\, 0 &{}\quad \,\, 0 &{}\quad 0 \\ \!-16 &{} \,\,-8 &{} \,\,-4 &{} \quad 0 \\ \,\,\,\,41 &{} \,\,\,\,20 &{} \,\,\,\,10 &{} -8 \\ \!-25 &{}-12 &{} \,\,-6 &{} \quad 4 \\ \quad \,\, 7 &{}\quad \,\, 4 &{}\quad \,\, 2 &{}\quad 0 \\ \,\,\,\,15 &{}\quad \,\, 8 &{}\quad \,\, 6 &{} -4 \\ \!-51 &{}-28 &{}-18 &{} \quad 0 \\ \,\,\,\,29 &{} \,\,\,\,16 &{} \,\,\,\,\,10 &{} -4 \\ \end{bmatrix} \begin{bmatrix} 1 \\ c_{0, 2} \\ c_{0, 3} \\ c_{1, 2} \\ \end{bmatrix}, \end{aligned}$$
$$\begin{aligned}&K_{(3/2, 4)_S}:&K_{(5/2, 3)_S}:\\&\begin{bmatrix} c_{0, 1} \\ c_{0, 2} \\ c_{0, 3} \\ c_{0, 4} \\ c_{1, 1} \\ c_{1, 2} \\ c_{1, 3} \\ c_{1, 4} \\ \end{bmatrix} = {1 \over 2} \begin{bmatrix} \quad 0 \\ -6 \\ \quad 0 \\ \quad 8 \\ -1 \\ \quad 3 \\ \quad 0 \\ -4 \\ \end{bmatrix},&\begin{bmatrix} c_{0, 1} \\ c_{0, 2} \\ c_{0, 3} \\ c_{1, 1} \\ c_{1, 2} \\ c_{1, 3} \\ c_{2, 1} \\ c_{2, 2} \\ c_{2, 3} \\ \end{bmatrix} = {1 \over 32} \begin{bmatrix} \quad \,\,0 \\ -56 \\ \quad \,\,0 \\ \!-18 \\ \,\,\,\,32 \\ \,\,-8 \\ \quad \,\,1 \\ \,\,-4 \\ \quad \,\,4 \\ \end{bmatrix}, \end{aligned}$$
$$\begin{aligned}&K_{(5/2, 4)_S}: \\&\begin{bmatrix} c_{0, 1} \\ c_{0, 3} \\ c_{1, 1} \\ c_{1, 2} \\ c_{1, 3} \\ c_{1, 4} \\ c_{2, 1} \\ c_{2, 2} \\ c_{2, 3} \\ c_{2, 4} \\ \end{bmatrix} = {1 \over 48} \begin{bmatrix} \quad \quad 0 &{}\quad \quad 0 &{}\quad \quad 0 \\ \quad \quad 0 &{} \quad \quad 0 &{}\quad \quad 0 \\ -208 &{} \,\,-88 &{} \,\,-26 \\ \quad 208 &{} \quad \,\,64 &{} \quad \,\,32 \\ \quad 256 &{} \,\,\,\,160 &{} \quad \,\,56 \\ -448 &{} -256 &{} -128 \\ \quad \,\,\,80 &{} \quad \,\,\,44 &{} \quad \,\,\,13 \\ -208 &{} -112 &{} \,\,-32 \\ -128 &{} \,\,-80 &{} \,\,-28 \\ \quad 448 &{} \,\,\,\,256 &{} \quad \,\,80 \\ \end{bmatrix} \begin{bmatrix} \quad 1 \\ c_{0, 2} \\ c_{0, 4} \\ \end{bmatrix}. \end{aligned}$$

The next step of our approach is optimizing the independent kernel coefficients with respect to an objective function measuring the severity of the staircasing effect. In order to define this function, we consider a sharp edge with a \(45^\circ \) orientation separating two half-spaces with values 0 and 1 (see Fig. 1). The pixel at location (i, j) in the corresponding rasterized image has one of the four different values that depend on the position of the grid:

$$\begin{aligned} d(i, j) = {\left\{ \begin{array}{ll} 0 &{} i - j < -1, \\ \theta ^2 / 2 &{} i - j = -1, \\ 1 - (1-\theta )^2/2 &{} i - j = 0, \\ 1 &{} i - j > 0. \end{array}\right. } \end{aligned}$$
Fig. 1
figure 1

Rasterization of a diagonal edge

Full size image

The resulting interpolant is

$$\begin{aligned} u(x, y) = \sum _{i = -\infty }^{+\infty }\sum _{j = -\infty }^{+\infty } d(i, j) \psi (x - i, y - j). \end{aligned}$$

A perfect staircasing-free interpolation should have straight isolines. A measure of staircasing penalizing isoline distortions was therefore introduced:

$$\begin{aligned} E_g^2(\theta ) := \int \int (\nabla u(x, y) \cdot (1, 1))^2 dx dy \end{aligned}$$

For a staircasing-free interpolation \(E_g = 0\) since \(\nabla u(x, y)\) is always orthogonal to edge direction. The integration is performed over unit squares with a nonzero gradient. The boundaries of the square regions correspond to piecewise intervals, i.e. integers for even kernels and half-integers for odd kernels:

$$\begin{aligned}{}[k - \Delta , k + 1 - \Delta ] \times [0 - k - \Delta , 1 - k - \Delta ] \cup [k + 1 - \Delta , k + 2 - \Delta ] \times [0 - k - \Delta , 1 - k - \Delta ], \end{aligned}$$

where k is the integer region index and \(\Delta = 0\) for even kernels and \(\Delta = 1/2\) for odd ones. The compound region thus defined covers a single period of u(x, y). The integration is performed separately for each square region after simplifying the piecewise interpolant into a polynomial. The degree of \(E_g^2\) polynomials is at most 4.

We have considered two options for the choice of \(\theta \):

  • \(\theta = 1/2\), corresponding to the worst case scenario. This choice produces the sharpest edge and the maximal value of \(E_g(\theta )\) for all kernels tested.

  • Averaging across all values of \(\theta \): \(\langle E_{g} \rangle ^2 := \int _0^1 E_g^2(\theta ) d\theta \).

In all our experiments, the kernels obtained with \(E_{g}(1/2)\) and \(\langle E_{g} \rangle \) quality metrics are nearly identical, with the maximal absolute deviation between the two variants never exceeding 0.006. Therefore we chose the \(E_{g}(1/2)\) metric, as it leads to less complex algebraic manipulations.

It is worth noting that the normalized \({\text {sinc}}\) kernel \({\text {sinc}}(x) = \sin (\pi x) / (\pi x)\) has \(E_g(\theta ) = 0\). This can be seen by performing the summation along a single diagonal:

$$\begin{aligned} \sum _{k=-\infty }^{+\infty } {\text {sinc}}(x - k) {\text {sinc}}(y - k) = {\text {sinc}}(x - y). \end{aligned}$$

Despite the absence of staircasing and theoretical optimality for band-limited signals, the \({\text {sinc}}\) kernel is a poor choice for image interpolation. It is computationally inconvenient because of infinite support, and its prominent and slowly fading oscillations produce severe ringing artifacts.

We have also experimented with a simpler measure of staircasing, the squared deviation of the interpolant from 1/2 integrated along the edge:

$$\begin{aligned} E_d(\theta ) = \int _0^1 (u(t, \theta + t) - 1 / 2)^2 dt. \end{aligned}$$

A staircasing-free interpolation implies \(E_d = 0\) since the central isoline always has the value 1/2. However, the converse is not true – the interpolant may have wavy isolines at values other than 1/2. This measure thus performed poorly, often producing pathologically oscillating kernels.

The kernel coefficients are then optimized with respect to \(E_{g}(1/2)\). We start by analytically differentiating \(E_g^2(1/2)\) with respect to free kernel coefficients in order to obtain the zero partial derivative conditions. The resulting systems of polynomial equations are solved with the Solve command. Internally, Solve employs cylindrical algebraic decomposition for dealing with such systems [30]. All the real critical points found by Solve are then checked using a second partial derivative test to find the local minima. The Hessian of the objective function is calculated and the PositiveDefiniteMatrixQ command is employed to verify its positive-definiteness. In all cases, a single real critical point that is also a local minimum has been found.

We have applied our optimization procedure to kernels with r ranging from 3/2 to 3 and p from 2 to 4. The complexity of the stationary point equations increases with the number of free coefficients v. The maximal number of terms of a \(E_{g}\) polynomial is given by multinomial coefficient (v, 4)!, and the maximal number of terms in each of the corresponding equations is \((v, 4)! - (v - 1, 4)!\). These upper bounds are usually reached. The largest system we were able to solve symbolicaly corresponds to \(K_{(3, 3)}\) and consists of 4 equations with 35 terms each. We were unable to obtain a symbolic solution for kernel \(K_{(3, 4)}\) (6 equations, 84 terms each) in a reasonable time (8 h on a Xeon 6146-based workstation) and only report the numeric results for this instance. The obtained kernels are plotted in Fig. 2. Below we provide four examplar polynomials along with the equations whose real roots correspond to the minima:nnnn

$$\begin{aligned}&K_{(2, 2)}: E_g^2(1/2) = (752 + 2611 c_{0,1} + 3192 c_{0,1}^2 + 1334 c_{0,1}^3 + 196 c_{0,1}^4) / 1440, \\&\qquad 2611 + 6384 c_{0,1} + 4002 c_{0,1}^2 + 784 c_{0,1}^3 = 0; \\&K_{(2, 4)_S}: E_g^2(1/2) = (9318135 + 7949688 c_{0,2} + 3041872 c_{0,2}^2 + 323456 c_{0,2}^3 + 12544 c_{0,2}^4) / 33868800, \\&\qquad 993711 + 760468 c_{0,2} + 121296 c_{0,2}^2 + 6272 c_{0,2}^3 = 0; \\&K_{(3, 2)}: E_g^2(1/2) = (4373 + 760 c_{0,1}^4 + 5 c_{0,1}^3 (1051 + 276 c_{1,1}) + 3 c_{0,1}^2 (4381 + 6 c_{1,1} (405 + 62 c_{1,1})) \\&\quad + 3 c_{1,1} (2146 + c_{1,1} (1099 + 4 c_{1,1} (53 + 13 c_{1,1}))) + c_{0,1} (12826 + 3 c_{1,1} (4176 \\&\quad + c_{1,1} (1299 + 152 c_{1,1})))) / 1440, \end{aligned}$$
$$\begin{aligned} {\left\{ \begin{array}{ll} 12826 + 3040 c_{0,1}^3 + 15 c_{0,1}^2 (1051 + 276 c_{1,1}) + 6 c_{0,1} (4381 + 6 c_{1,1} (405 + 62 c_{1,1})) + \\ 3 c_{1,1} (4176 + c_{1,1} (1299 + 152 c_{1,1})) = 0,\\ 1073 + 230 c_{0,1}^3 + 3 c_{0,1}^2 (405 + 124 c_{1,1}) + c_{1,1} (1099 + 2 c_{1,1} (159 + 52 c_{1,1})) + \\ 3 c_{0,1} (696 + c_{1,1} (433 + 76 c_{1,1})) = 0; \end{array}\right. } \end{aligned}$$
$$\begin{aligned} K_{(3, 3)_S}: E_g^2(1/2)= & {} (92669325 + 117493344 c_{0,2} + 52220952 c_{0,2}^2 + 9325760 c_{0,2}^3 + 598096 c_{0,2}^4) / 25804800, \\&7343334 + 6527619 c_{0,2} + 1748580 c_{0,2}^2 + 149524 c_{0,2}^3 = 0. \end{aligned}$$

Since many other polynomials and their stationary points in the symbolic form are too cumbersome to reproduce here, we only list the selected numeric kernel coefficients. The coefficients of \(K_{(2, 4)_S}\) admit a simple rational approximation resulting in a nearly identical kernel (maximal deviation \(1.2 \cdot 10^{-4}\)).

$$\begin{aligned} K_{(2, 2)}:&\begin{bmatrix} -0.621913 &{} -0.378087 \\ -0.378087 &{} \quad 0.378087 \\ \end{bmatrix},\\ K_{(2, 4)_S}:&\begin{bmatrix} \!\,0 &{} -1.751899 &{} 0.003798 &{} \quad 0.748101 \\ -0.5 &{} \,\,\,\,\,\,\!0.251899 &{} 0.996202 &{} -0.748101 \\ \end{bmatrix} \approx {{1} \over {4}} \begin{bmatrix} \,\,0 &{} -7 &{} 0 &{} \quad \!3 \\ \!\!-2 &{} \quad 1 &{} 4 &{} \!-3 \\ \end{bmatrix},\\ K_{(5/2, 3)}:&\begin{bmatrix} 0 &{} -1.581352 &{} \,0 \\ \!-0.825153 &{} \,1 &{} \quad 0.463315 \\ \quad 0.162576 &{} -0.209324 &{} -0.231657 \\ \end{bmatrix},\\ K_{(3, 3)}:&\begin{bmatrix} -0.435330 &{} -0.753337 &{} \quad 0.188667 \\ -0.548062 &{} \quad 0.379468 &{} \quad 0.168595 \\ \quad 0.092578 &{} \quad 0.046312 &{} -0.138890 \\ \end{bmatrix},\\ K_{(3, 3)_S}:&\begin{bmatrix} 0&{} -2.067867 &{} \quad 1.067867 \\ -0.932133 &{} \quad 1.648200 &{} -0.716067 \\ \quad 0.216067 &{} -0.432133 &{} \quad 0.216067 \\ \end{bmatrix},\\ K_{(3, 4)_S}:&\begin{bmatrix} 0&{} -1.851913 &{} \quad 0.542139 &{} \quad 0.309774 \\ -0.838313 &{} \quad 0.693843 &{} \quad 0.958096 &{} -0.813626 \\ \quad 0.169156 &{} \quad 0.165539 &{} -0.838547 &{} \quad 0.503852 \\ \end{bmatrix}. \end{aligned}$$

Some of the obtained kernels are nearly identical, namely

$$\begin{aligned} |\psi _{(2, 3)}(x) - \psi _{(2, 2)}(x) |< & {} 2.7 \times 10^{-4}, \\ |\psi _{(2, 4)}(x) - \psi _{(2, 2)}(x) |< & {} 1.4 \times 10^{-3}, \\ |\psi _{(3, 4)}(x) - \psi _{(3, 3)}(x) |< & {} 6.7 \times 10^{-4}, \\ |\psi _{(5 / 2, 4)}(x) - \psi _{(5 / 2, 4)_S}(x) |< & {} 3.2 \times 10^{-3}. \end{aligned}$$
Fig. 2
figure 2

Plots of interpolation kernels

Full size image

3 Kernel Evaluation

In addition to our optimized kernels, we have also tested several interpolators proposed in the literature. Apart from the widely used cubic kernel \(K_{(2, 3)_S}\), Keys also derived a unique \(C^1\)-smooth piecewise-polynomial interpolator \(Ks_{(3, 3)}\) with higher interpolation order. Its \(c_{i, j}\) coefficients are

$$\begin{aligned} {1 \over 12} \begin{bmatrix} \quad 0 &{} -28 &{} \,\,16 \\ -8 &{} \,\,\,\,15 &{} -7 \\ \quad 1 &{} \,\,-2 &{} \quad 1 \\ \end{bmatrix}. \end{aligned}$$

The Lanczos kernel is the most popular member of the windowed \({\text {sinc}}\) family of interpolators:

$$\begin{aligned} Ls_r(x) = {\text {sinc}}(x) {\text {sinc}}(x / r) \llbracket |x |< r \rrbracket \end{aligned}$$

According to Turkowski [27], it provides the best compromise between sharpness and ringing among several tested windowed \({\text {sinc}}\) filters. We have tested the commonly employed values \(r = 2\) and \(r = 3\). The Lanczos kernel does not satisfy the partition of unity condition. This resulting ripple is noticeable for \(r = 2\) (maximal deviation from unity 0.019) but is tolerable for \(r = 3\) (maximal deviation 0.0057). In the case of 2D kernel \(Ls_r(x) Ls_r(y)\), the deviation increases by a factor of nearly two.

Lagrange interpolation is a classical method that interpolates the given data with a polynomial of the lowest possible degree. When applied globally, it is susceptible to large oscillations. However, it can be applied locally to 2r points around the current x. For uniformly-spaced data, this technique is equivalent to convolution with a piecewise-polynomial kernel. In the case of integer r, the kernel can be found by noting that a subpolynomial on the interval \([i, i + 1)\) must evaluate to \(\delta (x)\) for integer x, \(i - r + 1 \le x \le i + r\). The coefficients for \(r = 2\) and \(r = 3\) are

$$\begin{aligned} Lg_{(2, 3)}(x): c_{i, j} = {1 \over 6} \begin{bmatrix} -3 &{} -6 &{} \quad 3 \\ -2 &{} \quad 3 &{} -1 \\ \end{bmatrix}, \end{aligned}$$
$$\begin{aligned} Lg_{(3, 5)}(x): c_{i, j} = {1 \over 120} \begin{bmatrix} \!-40 &{} \!-150 &{} \,\,\,\!50 &{} \quad \,\!30 &{} -10 \\ \!-60 &{} \quad \,80 &{} -5 &{} \!-20 &{} \quad \,\,5 \\ \quad \,\,6 &{} \,\,\,\,-5 &{} \,-5 &{} \quad \,\,5 &{} \quad \,\,1 \\ \end{bmatrix}. \end{aligned}$$

Lagrange kernels are \(C^0\)-smooth in case of an integer r and discontinuous in case of a half-integer r (the latter thus being excluded from the tests). They converge to \({\text {sinc}}(x)\) as \(r \rightarrow +\infty \) (see [19] for the proof), optimal in the low frequency region of the spectrum [23] and have the minimal support for a given interpolation order among the functions satisfying the interpolation constraints [3].

Schaum studied the performance of interpolators for different models of power spectrum \(|f(\nu )|^2\) [23]. For \(|f(\nu )|^2 \sim 1 / \nu ^4\), the optimal interpolator supported on \([-2, 2]\) is a piecewise-polynomial kernel

$$\begin{aligned} Sc_{(2, 3)}(x) = {{1} \over {15}} {\left\{ \begin{array}{ll} 3 (1 - |x |) (5 + 4 |x |- 5 |x |^2) &{} 0 \le |x |< 1, \\ (2 - |x |) (1 - |x |) (12 - 5 |x |) &{} 1 \le |x |< 2, \\ 0 &{} |x |\ge 2. \end{array}\right. } \end{aligned}$$

\(Sc_{(2, 3)}(x)\) satisfies the partition of unity and linear term representation constraints and is \(C^0\)-smooth.

B-splines are piecewise-polynomial functions defined recursively as

$$\begin{aligned} \beta _{0}(x) = {\left\{ \begin{array}{ll} 1 &{} |x |< 1/2 \\ 1/2 &{} |x |= 1/2 \\ 0 &{} |x |> 1/2 \end{array}\right. },\quad \beta _{p+1} = \beta _{p} *\beta _{0}, \end{aligned}$$

where \(*\) denotes the convolution operator. For \(p > 1\), \(\beta _{p}\) are non-interpolating, so an additional prefiltering step is required to satisfy the interpolation condition (see [4, 9] for details). The prefilter can be combined with \(\beta _{p}\) to get the actual interpolation kernel \(\beta ^{*}_p\). In particular, it can be shown (see [6]) that

$$\begin{aligned} \beta ^{*}_2(x)= & {} \sum _{k = -\infty }^{+\infty } \sqrt{2}(2 \sqrt{2} - 3)^{|k |} \beta _2(x - k), \\ \beta ^{*}_3(x)= & {} \sum _{k = -\infty }^{+\infty } \sqrt{3} (\sqrt{3} - 2)^{|k |} \beta _3(x - k). \end{aligned}$$

As the spline order increases, \(\beta ^{*}_p(x)\) converges to \({\text {sinc}}(x)\). B-spline interpolation is generally considered to be one of the highest quality linear methods, but the fact that the kernel is not compactly supported complicates the implementation.

The Mitchell–Netravali kernel introduced in [20] is given by

$$\begin{aligned} MN_{(2, 3)}(x) = {{1} \over {18}} {\left\{ \begin{array}{ll} 16 - 36 |x |^2 + 21 |x |^3 &{} 0 \le |x |< 1, \\ 32 - 60 |x |+ 36 |x |^2 - 7 |x |^3 &{} 1 \le |x |< 2, \\ 0 &{} |x |\ge 2. \end{array}\right. } \end{aligned}$$

It is a linear combination of \(\beta _{3}(x)\) and \(K_{(2, 3)_S}\) with weights 1/3 and 2/3 respectively. Unlike all other kernels in our tests, \(MN_{(2, 3)}(x)\) is non-interpolating. This property makes it a poor choice if the target sample rate is close to that of the original image. Nonetheless, we have included it in the comparison, as it was identified in [20] as the optimal \(C^1\)-smooth cubic kernel supported on \([-2, 2]\) interval in terms of perceived image quality.

To evaluate the performance of the optimized kernels on image features of various sizes and orientations we employed a zone plate function given by

$$\begin{aligned} I(x, y) = (1 + \cos (2 \pi F (x^2 + y^2))) / 2, F = 6. \end{aligned}$$

I(x, y) was sampled in the region \([0, 1] \times [0, 1]\) with sampling interval \(\Delta x = \Delta y = 1 / 30\) and then resampled with \(\Delta x = \Delta y = 1 / 360\) using various kernels. The resulting images are reproduced in Appendix 5. Table 2 lists the interpolation errors and the staircasing metrics of the new and existing kernels. Our results compare favorably with those of the existing kernels for \(r \ge 2\). The optimized interpolators outperform the popular Keys’ kernels with the same r in terms of both the staircasing magnitude and RMSE even at lower polynomial orders. The reduction of anisotropy is also apparent in the plots of the gradients of interpolant u(x, y) (Figs. 3 and 4). Odd kernels are less effective at reducing staircasing even when compared to even kernels with smaller support. The \(E_{g}\) data offers some justification for the Mitchell–Netravali kernel, as it has the lowest staircasing among the kernels of its size. However, \(K_{(2, 2)}\) has only 6% higher \(E_{g}\) while being interpolating.

Table 2 Staircasing metrics \(E_g(1/2)\) and zone plate root mean square interpolation errors of various kernels
Full size table
Fig. 3
figure 3

Gradients of u(x, y) for even kernels. Isolines of u(x, y) at levels 0, 1/4, 1/2, 3/4, 1 are shown in gray

Full size image
Fig. 4
figure 4

Gradients of u(x, y) for odd kernels. Isolines of u(x, y) at levels 0, 1/4, 1/2, 3/4, 1 are shown in gray

Full size image

We have also performed resampling tests on twelve benchmark images from the collection [1] (“apples”, “billiard balls a”, “cards a”, “coins”, “ducks”, “flowers”, “keyboard a”, “lion”, “garden table”, “tomatoes b”, “tools b”, “wood game”). The reduced \(300 \times 300\) images provided as a part of the image set were resampled to their original size (\(2400 \times 2400\)) and compared with the ground truth. Since the root mean square error (RMSE) is not a reliable indicator of perceived image quality, we employed 12 additional full-reference image quality assessment (IQA) methods implemented in the PIQ library [14]: SSIM [40], MS-SSIM [28], VIFp [25], FSIM [37], GMSD [32], VSI [35], HaarPSI [22], MDSI [41], MS-GMSD [34], LPIPS [38], PieAPP [21], DISTS [7]. In order to specifically assess the reconstruction of gradients, we have added gradient cosine similarity (GCS) to the set of IQA methods. The gradients were calculated using the maximally isotropic Scharr operator [11]:

$$\begin{aligned} g_x = {1 \over {2 (2 + \sqrt{12})}} \begin{bmatrix} -1 &{} 0 &{} 1 \\ -\sqrt{12} &{} 0 &{} \sqrt{12} \\ -1 &{} 0 &{} 1 \\ \end{bmatrix}, g_y = g_x^\intercal \end{aligned}$$

The gradient cosine similarity was then calculated as

$$\begin{aligned} GCS = {{\sum _i \sum _j {G_{i, j} \cdot G_{i, j}'}} \over {\sqrt{\sum _i \sum _j {G_{i, j}^2}} \sqrt{\sum _i \sum _j {{G'}_{i, j}^2}}}}, \end{aligned}$$

where \(G_{i, j}\) and \(G_{i, j}'\) are the per-pixel gradients of the original and the iterpolated images.

Since different IQA methods have different scales, the results of each method were rescaled into the [0, 100] range with 0 corresponding to the worst interpolation kernel and 100 to the ground truth image. The resulting standardized quality scores averaged across all images are listed in Table 3.

Table 3 Averaged standardized quality scores for various kernels and IQA methods
Full size table

The kernels ranked best by various IQA methods are \(K_{(3, 4)_S}\) (MS-SSIM, GMSD, MS-GMSD, HaarPSI, GCS), \(K_{(3, 3)_S}\) (RMSE, FSIM, MDSI), \(\beta _3\) (SSIM, LPIPS), \(\beta _2\) (VIF), \(K_{(2, 4)_S}\) (VSI), \(Ls_3\) (DISTS), \(Ls_2\) (PieAPP). The only case where our optimized kernels demonstrate no improvement is \(r = 3 / 2\). For larger values of r, the new kernels \(K_{(2, 4)_S}\), \(K_{(3, 3)_S}\) and \(K_{(3, 4)_S}\) outperform the existing interpolators with identical support according to the vast majority of IQA methods. The kernel \(K_{(5/2, 3)}\) compares favorably even with the larger Keys, Lagrange, and Lanczos interpolators. Remarkably, the kernels \(K_{(5/2, 3)}\), \(K_{(3, 3)_S}\) and \(K_{(3, 4)_S}\) outperform the significantly more costly cubic B-spline interpolation according to the majority of quality metrics (9, 10 and 10 metrics respectively). Increasing the support size and polynomial degree gives diminishing returns, so the potential improvements arising from kernels with \(r > 3\) or \(p > 4\) are likely marginal.

As Fig. 5 demonstrates, the new kernels with small \(E_g\) provide a noticeable reduction of staircasing around diagonal edges when compared to the popular Keys’ interpolators. However, for a given support size the kernels with the smallest \(E_g\) are usually not the ones preferred by IQA methods (the mean correlation between \(E_g\) and IQA scores is \(-0.54\)). There are two reasons for this inconsistency. First, optimization of \(E_g\) alone does not take into account image sharpness. Even though \(K_{(3/2, 2)}\) has the worst \(E_g\) among the tested kernels, it ranks above the linear interpolation according to all but one IQA method by virtue of producing sharper images. Second, the IQA methods themselves are imperfect and usually are not specifically designed for the distortion types introduced by interpolation.

Fig. 5
figure 5

A test image interpolated with Keys’ (a, b) and minimal \(E_g\) (c, d) kernels

Full size image

4 Conclusion

We have constructed several new high-quality separable piecewise-polynomial interpolation kernels for image resampling. The kernel coefficients were obtained by minimizing a specifically defined measure of the magnitude of staircasing artifacts around diagonal edges. By using the Mathematica computer algebra system we were able to evaluate the resulting polynomials in symbolic form. This is an important prerequisite for the optimization process, since the gradient of the objective function can be easily calculated. In most cases, we were able to find the stationary points and obtain the optimal kernel coefficients also in symbolic form. Although the optimization could have been performed numerically using Newton’s method, a symbolic approach had the advantage of proving the solution’s uniqueness.

The reduction of staircasing comes at a cost of increased kernel oscillations. Nonetheless, when compared to other popular interpolating kernels our results provide a noticeable improvement of subjective image quality in areas around sharp transitions. Depending on the desired computational cost and subjective preferences between sharpness and blocking, we recommend selecting a kernel from the following set: \(K_{(2, 2)}\), \(K_{(2, 4)_S}\), \(K_{(5/2, 3)}\), \(K_{(3, 3)}\), \(K_{(3, 3)_S}\), and \(K_{(3, 4)_S}\).

We note the discrepancy between theory and practice. By the standards of interpolation theory, our kernels are inferior to many other piecewise-polynomial interpolators proposed in the literature as they have low interpolation order and are not necessarily continuously differentiable. Nonetheless, they demonstrate superior performance both subjectively and according to various image quality assessment methods. We conclude that the theoretical considerations pertaining to one-dimensional interpolation are insufficient for the design of high-quality image resampling kernels. The often neglected anisotropic artifacts arising from the kernel separation process are a major factor determining the subjective image quality.

Optimization with respect to a single type of artifact is a limitation of the present work. A more complex objective function incorporating other artifact types could further improve the subjective image quality. The use of separable kernels is another limitation. With nonseparable kernels, the staircasing error metrics could be reduced further, but we decided not to pursue this direction for two reasons. First, it would greatly increase the number of coefficients and would lead to reduced speed and higher complexity. Second, the approach would introduce further complications since the vertical and horizontal edges would no longer be staircasing-free.

Since our artifact reduction technique stays within the linear interpolation framework, it retains the simplicity and computational efficiency of the linear methods. High-quality texture filtering on modern graphical processors is therefore a possible application. The use of the proposed kernels as a basis of more complex nonlinear methods is a promising direction for future work.

All Mathematica code used for kernel construction can be found on the author’s GitHub page [13].