Correlative Study of Image Magnification Techniques

Abstract

A digital image is often required to be rescaled. When the pixels in the image are mapped to a larger grid, new coordinate positions will be formed. The interpolation procedures are used to approximate the value for the unknown sample points set between the initial sample points. The concept is to replace the missing values using the known values of the initial sample points. The interpolation techniques are intended to effectively preserve the characteristic features of an image. There is an indeed greater number of interpolation algorithms available which have their strengths and challenges. In this study, interpolation techniques based on the nearest neighbour, bilinear interpolation, and high-quality magnification (hq3x) scaling were examined. The qualitative and quantitative properties of these three interpolation techniques were analysed. The parameters considered are the SSIM and SNR measure. In the experiment, the performance of the three methods was compared and measured from objective and visual assessments. The aim of this study is to analyse the visual quality of the output to determine the suitable method. From the analysis, it is observed that hqx scaling performs better than nearest neighbour interpolation and bilinear approach performs fairly closer to hqx scaling.

1 Introduction

Interpolation is a method of estimating the values at any given position between the range of distinct sets of points [1]. For several image processing applications, this is one among the basic procedures [2]. In digital imaging, scaling refers to expanding or reducing the number of pixels in the image. When the image dimension is extended, an additional number of sample points is created. Most commonly, magnifying the image involves interpolation procedures to approximate the numerical values of unknown sample points. A technique called image panning is related to image magnification in a way how the images are viewed in the display. In the context of image display, panning allows viewing of the image horizontally or vertically in the display system, whereas image magnification provides a deeper insight into the image or a wider perspective of the viewed object. The issue with inspecting the zoomed image, however, will only reveal a portion of the image. Panning makes it possible to look at the different parts of the zoomed image by shifting the focus inside the image to the desired area. Similarly, contrast enhancement is done to maximize the intensity values present in the image to give a better visual perception. This process differentiates objects and the background in a more distinguishable way [3, 4]. Image rescaling is often necessary for many image operations such as zooming or shrinking images for digital display devices, geometric transformations and subpixel registration, and decompression [5,6,7]. Interpolation algorithms are structured to achieve optimal results with a trade-off between maximum performance, edge smoothness and sharpness [8]. Interpolation, however, is meant to retain the image the details as the image is extended, adding new details would never make the original image. Image interpolation is the problem of approximating the value for the new sample point with the known points located around the unknown point. There are a number of practical methods for upscaling images which obtain the desirable result, but it is valuable to find the appropriate method of significantly lower distortion. In this study, to evaluate an efficient approach to achieving optimum performance, the efficiency of the image interpolation methods is tested from objective criteria along with visual criteria.

This paper is structured as follows. The second section provides a review of relevant literature on methods of image interpolation. The third section describes the three interpolation techniques. The fourth section presents the experimental analysis and performance comparisons and the conclusion is drawn in the final section.

2 Related Work

The classical interpolation approaches do not consider the local details of the image and the unknown pixel values are interpolated in the same way across the whole image. Dianyuan Han presents a comparison of widely used interpolation methods for image upscaling. The results show the bicubic interpolated image scores better both in SNR measures as well as the visual quality and take less computation time compared to cubic B spline method. The nearest neighbour interpolation is fast however it produces uneven edges. According to the author, the bilinear interpolation method makes an effective option with respect to the processing speed and visual quality [9]. There has been many study addresses interpolation based on the differences in local intensity features to preserve sharp edges. An edge-directed interpolation employed by Allebach et al. proposes expanding the image with the high-resolution edge map generated based on subpixel estimation of the original image and the edge map is used as a direction to interpolate the smooth regions with bilinear interpolation and the visual quality is achieved through iteration [10]. Another approach by Li et al. proposed a model to preserve edges using local covariance measures to estimate the covariance pixels of the high-resolution image. This estimation of covariance is done based on the geometric duality of the pixels of the low-resolution and high-resolution image. To minimize the complexity, a hybrid method is adopted to interpolate exactly the edge region with covariance-based interpolation and to interpolate smooth regions with bilinear interpolation [11]. Zhang presents a nonlinear interpolation method to retain the edge structure using directional filter and data fusion. The directional filter estimates two values along with the orthogonal directions of a missing sample from a local window and the directional estimated values are adaptively merged together to calculate a more robust value with linear MMSE [12]. To reduce the computational complexity and to achieve high-resolution image, the method focuses on isolating edge pixels from non-edge pixels. To do so, it measures the local variance of the unknown pixel’s nearest pixels. With bilinear interpolation, the non-edge pixels are interpolated and the edge pixels are adaptively interpolated.

3 Image Interpolation

Image interpolation is the process by which the unknown points are estimated using the known points. It generates high-resolution image from the low-resolution image without affecting the visual information present in the original image. Unknown sample points are created to extend the original image grid according to the interpolation ratio. Accuracy of the image being constructed depends on the number of identified samples. In interpolation, it is presumed that the closest known points have a greater impact when measuring value at the unknown point than those farther away [13].

The brightness values of points P1, P2, P3, P4, for example, represent the pixels in the original image as shown on the left in Fig. 1, and its original dimension is magnified three times as seen in Fig. 1 on the right. The number of unknown sample points generated is determined by the desired magnification ratio. The values of original points P1, P2, P3, P4 are mapped to the new position in the enlarged grid, respectively. Notice the number of unknown points such as X, between these four known sample points. Now the values of those unknown samples are to be estimated by interpolation procedure using the brightness values of original points P1, P2, P3, P4 [9].

Fig. 1

Image interpolation

3.1 Nearest Neighbour Interpolation

The nearest neighbour interpolation is the simplest approach which assigns each sample with the value of a sample closest to the point. It is a piecewise polynomial. It is a single point approach; therefore, the function g(x) at any point is interpolated based on just one nearest sample value. The nearest neighbour approximation kernel is described by a box filter kernel

$$ s\left( t \right) = \left\{ {\begin{array}{*{20}c} 1 & {{\text{if}}\;\left| t \right| < 0.5} \\ 0 & {{\text{otherwise}}} \\ \end{array} } \right. $$

(1)

This filter kernel is the first-order family of B-splines. The samples are interpolated by convolving the sampled signal with the box filter in the spatial domain. This is equivalent to multiplication of the transformed function with a sinc function in the frequency domain [5]. Since sinc function has endless side lobes on both sides of the main lobe, it is not an optimal filter for interpolation. If the sinc function is approximated with the box window function defined in Eq. (1), the inverse transform of this filter displays ringing effect as shown in Fig. 2. This is due to the poor absorption of the high-frequency elements which often contributes to the discontinuity of the intensity surface of the image. This method is preferred, however, because of its simple operations which make implementation easier with little computing time.

Fig. 2

a Nearest neighbour interpolation box kernel, b frequency response

As shown in Fig. 3, the pixel P(x,y) is at non-integer location and is interpolated using the closest single known neighbour, if, for example, the input coordinates of the original sample points P(x₁,y₁), P(x₁,y₂), P(x₂,y₁), P(x₂,y₂) are the four nearest neighbours to the new sample point. Now to find the value for the new coordinate point P(x.y), the distance between P(x,y) and the four neighbouring points P(x₁, y₁), P(x₁,y₂), P(x₂,y₁), P(x₂,y₂) are computed. The closest point is determined and its intensity value is assigned to the new sample point P(x,y) [9].

Fig. 3

Illustration of nearest neighbour interpolation

The nearest neighbour method, if scaled to an integer coordinate, will reproduce the original image. This approach can induce distortion if it is mapped partially within the original location. However, the arrangement of pixels which form the image becomes visible as the scale factor is increased. Olivier et al. proposed a new nearest neighbour value interpolation to overcome this problem. According to the proposed approach, determining the nearest value is guided by bilinear interpolation. The value for the missing pixel is estimated by taking the minimum difference between the values of four nearest cells and the value obtained by bilinear interpolation. The value which is almost equal to the nearest cells is set as the value for the missing pixel [14].

3.2 Bilinear Interpolation

The box filter in Eq. (1) is convolved with itself produces a triangular function and the triangular filter kernel is defined as [15]

$$ s(t) = \left\{ {\begin{array}{*{20}l} {1 - \left| t \right|} & {\left| t \right| < 1} \\ 0 & {{\text{otherwise}}} \\ \end{array} } \right. $$

(2)

It is a linear piecewise polynomial where straight lines are used to connect between the points. The sampled image is convolved with the triangular filter and the Fourier transform will be sinc function multiply with a sinc function. This filter is a low-pass filter and gives smoother frequency response as seen in Fig. 4.

Fig. 4

Bilinear interpolation a triangle kernel, b frequency response

The bilinear interpolation takes into account two closest sample points in each direction located near to the undefined sample point. The linear interpolation along with the horizontal directions and on the vertical direction is computed. The final value for the undefined sample point is the weighted average value of the closest neighbours. The weights are calculated based on the distance from the undefined sample point to the four nearest points and the points located close to an undefined point are assigned greater weights.

Figure 5 illustrates the process of bilinear interpolation. The sampling point at P(x,y) is to be interpolated. This point is surrounded by two nearest neighbour on the bottom row denoted by P(1,1) and P(2,1) and two nearest neighbour on the top row denoted by P(1,2), P(2,2). The value for sample point Q is determined, by performing two linear interpolations: one in the x-direction of the bottom row to estimate the value for intermediate point P(x,y₁) and top row to estimate the point.

Fig. 5

Illustration of bilinear interpolation

P(x,y₂) and one in the y-direction using the previously estimated points P(x,y₁), P(x,y₂) to interpolate P(x,y).

To calculate the value for the sample point P(x,y₁) and P(x,y₂), the bottom row is linearly interpolated followed by top row in the x-direction

$$ P\left( {x,y_{1} } \right) = P\left( {x_{1} ,y_{1} } \right) + \left( {P\left( {x_{2} ,y_{1} } \right) - P\left( {x_{1} ,y_{1} } \right)} \right)*\frac{{x - x_{1} }}{{x_{2} - x_{1} }} $$

(3.1)

$$ P\left( {x,y_{2} } \right) = P\left( {x_{1} ,y_{2} } \right) + \left( {P\left( {x_{2} ,y_{2} } \right) - P\left( {x_{1} ,y_{2} } \right)} \right)*\frac{{x - x_{1} }}{{x_{2} - x_{1} }} $$

(3.2)

and one linear interpolation on the y-direction to estimate the point)(x,y).

$$ P\left( {x,y} \right) = P\left( {x,y_{1} } \right) + \left( {P\left( {x,y_{2} } \right) - P\left( {x,y_{1} } \right)} \right)*\frac{{y - y_{1} }}{{y_{2} - y_{1} }} $$

(4)

3.3 High Quality Magnification (Hqx) Interpolation

HQX stands for high quality magnification [16]. It was created by Maxim Stepin. It is created for pixel art scaling algorithm. This interpolation approach checks for lines in the image and gives smooth interpolation. The hqx scaling algorithm compares the color of the 8 immediate neighbours to the color of each sample point. The difference in colour between the source and each neighbour is calculated based on the YUV threshold and further, the Y component has a greater weight assigned compared to U and V components. The pixels compared is either classified as close or distant which gives 256 possible combinations. Further, If the colour difference in any one of the three planes is above the threshold, that bit is 1 otherwise 0. The magnification of each input sample points to the corresponding 9 output sample points.

The interpolation pattern for each 3 × 3 is searched in the preconstructed lookup table. This table directs which pixel color is assigned to a particular pattern. For each close or distant combination, there is an entry in the lookup table. Each entry states how to blend the colors of the source pixels and its neighbours to interpolate nine output pixels. The hqx works well when the input is not anti-aliased. This algorithm produces an image of better visual quality and retains the edges without much blurriness.

4 Experiment and Discussion

The widely used algorithm for image interpolation is the nearest neighbour and bilinear interpolation. The high-quality magnification algorithm considered in pixel art scaling is compared with these approaches. The consistency of each of these methods is analysed. To determine the accuracy of the magnified images, a test image taken is reduced to a third of its original size and using different interpolation algorithms the image is extended to its original arrangement. The original size of the test image is compared with the reconstructed image for similarity measure.

4.1 Objective Assessment

In this study, the performance of the selected image magnification algorithms is compared. To evaluate the fidelity of the interpolated images, signal-to-noise ratio (SNR) and structure similarity index (SSIM) of those scaled images are measured. The SSIM measures the structure content variation between the original image and the magnified image and the SNR computes the difference between two images. These methods are tested in 20 images and the results of the different interpolation algorithms are shown in Table 1.

Table 1 Evaluation of SSIM and SNR of different interpolated method

The SNR measure between two images produces larger value, represents better quality of an image and the mean SSIM value should be close to one. The results of SSIM and SNR values of different interpolation algorithms from the table show the bilinear interpolation and hqx scaling performed better than nearest neighbour interpolation. Importantly, if the image is not antialiased the hqx scaling provides better image quality. The SSIM and SNR values of bilinear interpolated images are greater than the nearest neighbour approach but fairly close to hqx scaling algorithm.

4.2 Visual Quality Assessment

Generally, the quality of the magnified images by various interpolation methods can be determined by performing a visual examination of the results. For visual evaluation, the test images are enlarged to a scale factor of 3 to illustrate the visual contrast between nearest neighbour, bilinear and hqx scaling methods. An example of a magnified image for visual comparison is shown in Fig. 6. The nearest neighbour interpolation displays the presence of serration near the borders, curves and lines which can be realized from Fig. 6, since this filter suppresses the stopband frequencies which results in unwanted deviations in the frequency response. It also generates significant blockiness over the entire surface of the image. Hence the accuracy of the image is reduced.

Fig. 6

Interpolated images for visual contrast a shrunken image, b original image, c nearest neighbour, d bilinear, e HQX

On the other hand, the bilinear interpolation eliminates the serrated edges which are more apparent in the nearest neighbour approach. Due to the improved stopband attenuation performance, this filter gives a smoother frequency response. But this is achieved by blurring the high-frequency data present in the original image. This produces extensive smoothing on the image surface which makes the sharp edges and other features appear smudged. The hqx approach is a pattern matching algorithm which interpolates pixels from the pregenerated table. It begins to search for edges in the image and smoothly interpolate those edges. However, it has slightly visible serrated edges in the image. It retains much of the sharp features contained in the original image.

5 Conclusion

The nearest neighbour interpolation can be used for its speed and ease of implementation. It replicates the closest sample point intensity. This possibly brings a stair step pattern across the image surface, specifically around the edges. This method, however, does not add any new information to the image. The bilinear interpolation which works in the opposite way to the latter approach in computing time as well as many sample points involved for an interpolating point. It requires additional calculation as it stretches around interpolating point to four nearest pixels. It is a low-pass filter with a decent frequency response that softens the high-frequency features such as image borders which looks indistinct. Unlike the nearest neighbour method, the bilinear interpolation provides an image of continuous intensity. As a consequence, the quality of the image is more satisfactory for viewing. The hqx algorithm can get reasonably better image quality with fast computing time except the difficulty involved in constructing the custom lookup table for interpolation. The hqx produces an image of very good in quality. All the edges and lines contained in the image appears sharper. It performs better in the case of antialiased image. However, extending the image size past a certain level cannot be done since interpolating a large number of pixels would become unrealistic. In this scenario, opting for high quality sensor would be useful.