Keywords

1 Introduction

Corner detectors are very popular tools used for the description of visual scene. They are widely used to characterise the content of a digital image by pointing at details of the visual scene. They are also used as feature points, based on which the descriptors are extracted.

In this paper we focus on classical solutions which generic principle of feature points detection using corners is based on the three-step scheme. At first the image gradients are computed. Next, the gradients are combined together in order to obtain a cornerness function that describes, for each image pixel, the possibility of finding the corner in the particular image point. Finally, from the cornerness function, a list of feature points is extracted. We investigate in this paper the application of various diagonal gradients as well as various formulas of cornerness function in view of their efficiency and invariance to basic image transformations.

Based on this scheme several popular corner detectors has been proposed [2, 3]. Some improvements of this idea was introduced in [8, 9] where authors proposed has more accurate algorithm for corner and edge detections using different quadratic polynomial to approximate the variation around pixels. In [4, 5] authors propose Harris-like scale and affine invariant detectors using local extrema over scale of normalized derivatives.

However it is possible to propose other combinations of particular methods that also allow to detect corners within the image [1, 6]. The main goal of the research described in these paper have been comparing various combinations of directional gradients and defining the cornerness function. It will be shown that some of those combinations surpasses the classical methods in terms of the accuracy of pointing at meaningful image details as well as invariance to some image transformations.

The paper is divided as follows: In Sect. 2 we describe classic corner detector methods. Then in Sect. 3 we propose some modifications of original version which have been tested. Results of these tests are described in Sect. 4. Section 5 concludes the paper.

2 Classic Corner Detectors

Harris corner detector [2] is commonly used for finding of feature points. It is an improvement of Moravec’s corner detector [7] based on the convolution operator with simple linear gradient masks:

$$\begin{aligned} H= \left[ \begin{array}{ccc} -1&\underline{0}&1 \end{array} \right] , V= \left[ \begin{array}{ccc} 1&\underline{0}&-1 \end{array} \right] ^T{.} \end{aligned}$$
(1)

The above masks are used to get two directional linear gradients using convolution operator (denoted as \(\otimes \)): \(g_H = I\,\otimes \, H\), \(g_V = I\,\otimes \,V\). The directional response obtained (Eq. 1) is used to construct the covariance matrix:

$$\begin{aligned} M = \left[ \begin{array}{cc} I_{HH} &{} I_{HV}\\ I_{HV} &{} I_{VV} \end{array} \right] {,} \end{aligned}$$
(2)

where:

$$\begin{aligned} I_{HH} = g_H^2 \otimes W \;,\; I_{HV} = (g_H \cdot g_V) \otimes W \; , \; I_{VV} = g_V^2 \otimes W, \end{aligned}$$
(3)

are second order moments convoluted using Gaussian mask W. The final cornerness function is defined by:

$$\begin{aligned} r = det(M) - k \cdot trace(M)^2 = \lambda _1 \lambda _2 - k (\lambda _1 + \lambda _2)^2, \end{aligned}$$
(4)

where \(\lambda _1\) and \(\lambda _2\) are eigenvalues of matrix M. Parameter k is chosen manually, often set up as \(k = 0.04\).

In order to obtain the list of corner pixels, the regional maxima of cornerness function r are computed. These maxima may suffer from two defects. Their values could be insignificantly low. It means that either the corner described by this maximum is poorly visible or it is not a real corner of a visual scene, but a kind of artefact being the result of e.g. noise. In order to solve this problem, the cornerness function is thresholded at given level t, and only maxima that are higher than t are considered. The second problem is the fact that single maximum may consist of several connected pixels. To solve it, the reduction of connected set of pixels into single pixel is usually performed, by means of e.g. thinning.

There exist many modifications of original Harris algorithm. One of them is Kanade-Lucas-Tomasi detector—KLT [3] where the authors proposed to modify cornerness function, so that it is computed as minimum of eigenvalues of matrix M: \( r = \min (\lambda _1,\lambda _2)\). Other approximation was proposed in [8, 9] where authors show other way to compute cornerness function.

3 Modifications of Classic Scheme

Classic detectors consist of three principal steps, that refer to computation of gradient, cornerness function and finding points. The gradients used in first two of these stages may be replaced by some other ones. In our research we investigated some of alternatives in view of the quality of corner detection. We assumed that directional gradients, apart from vertical and horizontal versions, may also be formulated in two diagonal variants. The choice of diagonal gradients was motivated by the assumption that vertical and horizontal gradients may not be enough to achieve high quality of corner detection. Considering the diagonal variants may increase accuracy of finding local corner points. Using both normal and diagonal gradients provide us with four different edge responses.

In case of the second issue we investigated gradient masks covering larger than two-pixel only (comparing to described by the Eq. 1) neighborhood. Larger (in terms of the number of pixels belonging to) neighborhoods allow to consider wider context in which the corner pixel may be detected.

In our research we examined various directional gradients as well as methods of combining obtained directional responses into single cornerness function.

The following gradient masks have been investigated:

  1. 1.

    Classic masks:

    $$\begin{aligned} H_1\,{=}\, \left[ \begin{array}{ccc} -1&\underline{0}&1 \end{array} \right] \;, \; V_1\,{=}\, \left[ \begin{array}{ccc} 1 &{} \underline{0} &{} -1 \\ \end{array} \right] ^T \;, \; D_1\,{=}\, \left[ \begin{array}{ccc} 0 &{} 0 &{} 1 \\ 0 &{} \underline{0} &{} 0 \\ -1 &{} 0 &{} 0 \end{array} \right] \;, \; D^{\prime } _1\,{=}\, \left[ \begin{array}{ccc} 1 &{} 0 &{} 0 \\ 0 &{} \underline{0} &{} 0 \\ 0 &{} 0 &{} -1 \end{array} \right] . \end{aligned}$$
    (5)
  2. 2.

    Prewitt masks:

    $$\begin{aligned} H_2= \left[ \begin{array}{ccc} -1 &{} 0 &{} 1 \\ -1 &{} \underline{0} &{} 1 \\ -1 &{} 0 &{} 1 \end{array} \right] \;, \; V_2= \left[ \begin{array}{ccc} 1 &{} 1 &{} 1 \\ 0 &{} \underline{0} &{} 0 \\ -1 &{} -1 &{} -1 \end{array} \right] \;, \; D_2= \left[ \begin{array}{ccc} -1 &{} -1 &{} 0 \\ -1 &{} \underline{0} &{} 1 \\ 0 &{} 1 &{} 1 \end{array} \right] \;, \; D^{\prime } _2= \left[ \begin{array}{ccc} 0 &{} -1 &{} -1 \\ 1 &{} \underline{0} &{} -1 \\ 1 &{} 1&{} 0 \end{array} \right] . \end{aligned}$$
    (6)
  3. 3.

    Sobel masks:

    $$\begin{aligned} H_3= \left[ \begin{array}{ccc} -1 &{} 0 &{} 1 \\ -2 &{} \underline{0} &{} 2 \\ -1 &{} 0 &{} 1 \end{array} \right] \;, \; V_3= \left[ \begin{array}{ccc} 1 &{} 2 &{} 1 \\ 0 &{} \underline{0} &{} 0 \\ -1 &{} -2 &{} -1 \end{array} \right] \;, \; D_3= \left[ \begin{array}{ccc} -2 &{} -1 &{} 0 \\ -1 &{} \underline{0} &{} 1 \\ 0 &{} 1 &{} 2 \end{array} \right] \;, \; D^{\prime } _3= \left[ \begin{array}{ccc} 0 &{} -1 &{} -2 \\ 1 &{} \underline{0} &{} -1 \\ 2 &{} 1&{} 0 \end{array} \right] . \end{aligned}$$
    (7)
  4. 4.

    Three-pixel wide neighborhood:

    $$\begin{aligned} \begin{array}{c} H_4= \left[ \begin{array}{ccccccc} -1 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} 1 \\ -1 &{} 0 &{} 0 &{} \underline{0} &{} 0 &{} 0 &{} 1 \\ -1 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} 1 \end{array} \right] \;, \; V_4= \left[ \begin{array}{ccccccc} 1 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} -1 \\ 1 &{} 0 &{} 0 &{} \underline{0} &{} 0 &{} 0 &{} -1 \\ 1 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} -1 \end{array} \right] ^T \;, \; \\ \\ D_4= \left[ \begin{array}{ccccccc} 0 &{} 0 &{} 0 &{} 0 &{} 1 &{} 0 &{} 0 \\ 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} 1 &{} 0 \\ 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} 1 \\ 0 &{} 0 &{} 0 &{} \underline{0} &{} 0 &{} 0 &{} 0 \\ -1 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 \\ 0 &{} -1 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 \\ 0 &{} 0 &{} -1 &{} 0 &{} 0 &{} 0 &{} 0 \\ \end{array} \right] \;, \; D^{\prime } _4= \left[ \begin{array}{ccccccc} 0 &{} 0 &{} 1 &{} 0 &{} 0 &{} 0 &{} 0 \\ 0 &{} 1 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 \\ 1 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 \\ 0 &{} 0 &{} 0 &{} \underline{0} &{} 0 &{} 0 &{} 0 \\ 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} -1 \\ 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} -1 &{} 0 \\ 0 &{} 0 &{} 0 &{} 0 &{} -1 &{} 0 &{} 0 \\ \end{array} \right] . \end{array} \end{aligned}$$
    (8)

The four directional gradients are combined together in order to obtain single cornerness function. They are used to compute the second-order matrix as defined in Eq. 2. Since two pairs of perpendicular gradients are used here, two matrices are computed: M based on gradients with masks H and V, and \(M'\) based on D and \(D'\). The elements of these matrices are filtered using the lineal low-pass filter with the mask W (Eq. 3). Type of such mask has influence on pixel neighbor context taken while processing cornerness response. We have examined the impact of the change of a filter mask on the quality of detected corners. The following masks ware investigated:

  1. 1.

    Gaussian

    $$\begin{aligned} W_1= \left[ \begin{array}{c c c c c}0.12&0.23&0.29&0.23&0.12\end{array}\right] ^T\cdot \left[ \begin{array}{c c c c c}0.12&0.23&0.29&0.23&0.12\end{array}\right] . \end{aligned}$$
    (9)
  2. 2.

    Box filter

    $$\begin{aligned} W_2= \left[ \begin{array}{c c c c c}0.2&0.2&0.2&0.2&0.2\end{array}\right] ^T\cdot \left[ \begin{array}{c c c c c}0.2&0.2&0.2&0.2&0.2\end{array}\right] . \end{aligned}$$
    (10)
  3. 3.

    Triangle-shaped filter

    $$\begin{aligned} W_3= \left[ \begin{array}{c c c c c}0.1&0.2&0.3&0.2&0.1\end{array}\right] ^T\cdot \left[ \begin{array}{c c c c c}0.1&0.2&0.3&0.2&0.1\end{array}\right] . \end{aligned}$$
    (11)

Matrices M and \(M'\) was used to compute the cornerness function. We assumed here the following variants:

  1. 1.

    Smallest of eigenvalues of M (similar to KLT method, further referred to as klt):

    $$\begin{aligned} r = \min \{\lambda _1,\lambda _2\}, \end{aligned}$$
    (12)

    where \(\lambda _1\),\(\lambda _2\) are the eigenvalues of the matrix M.

  2. 2.

    Smallest of eigenvalues of M and \(M'\) (normal and diagonal, minsv)

    $$\begin{aligned} r = \min \{\lambda _1,\lambda _2,\lambda '_1,\lambda '_2\}, \end{aligned}$$
    (13)

    where \(\lambda '_1\) and \(\lambda '_2\) are eigenvalues of matrix \(M'\). This estimation takes into account the smallest of all eigenvalues to improve corner response detection.

  3. 3.

    Mean of smallest eigenvalues of M and \(M'\) (meansv)

    $$\begin{aligned} r = \frac{\min \{\lambda _1,\lambda _2\} + \min \{\lambda '_1,\lambda '_2\}}{2}. \end{aligned}$$
    (14)

    This estimation takes into account both responses (classical and diagonal gradient) to compute more accurate result.

  4. 4.

    Determinant of the matrix M (determinant)

    $$\begin{aligned} r = det(M) = I_{HH}\cdot I_{VV} - {I_{HV}}^2. \end{aligned}$$
    (15)

    This is simplified, low computational version of cornerness function.

We did not modify in our research the final step of processing i.e. extraction of points from the cornerness function—this step is performed using the approach used in the Harris method (see previous chapter).

4 Tests

4.1 Methodology of Testing

Evaluation of the quality of corner detectors requires two principal factors to be taken into account: correctness and stability. The first factor—correctness is usually verified by comparing, for a particular image, automatically detected corners with the indication of a human observer.

The second factor used to determine the quality of the detector—stability—refers to insensitivity of the detector to typical image transforms that may happen in case of images presenting the same visual scene from various points of view. In our testing scheme we’ve chosen several image transforms that may simulate the real distortions that may occur in case of various images of the same scene. Among these transforms are: blurring, speckle noise, gamma correction, illumination change, zoom and rotation. Moreover, to validate the level of dependence the invariance from the strength of the given transform, we prepared series of transforms with increasing strength. For example, a series of blur transforms consists of Gaussian blurs computed for increasing standard deviation value. The possible image transforms are of two kinds—with and without the geometric transformation. The presence of the geometric transformation is important, because in this case, the position of the corner points changes.

In order to measure the invariance to particular image transform we compare the corner points before and after applying this transform on the input image. This comparison is made by computing the factor equal to the number of corner points matched divided by the total number of them.

In Fig. 1, the position of corner points before and after the geometric transform of rotation is shown. Cross mark indicates the corner found on original image (a) and the same corner transformed to coordinates of rotated image (b). Circle mark represents corners found on transformed image (b).

Fig. 1
figure 1

Detected corners before a and after b image rotation

Full size image

Let P be a set of corner points on image I: \(P = \left\{ p_{1}, p_{2}, \ldots , p_{n}\right\} \), where n is the number of points detected on image I. Let \(T_{A}\) is the operator of geometric transformation with parameters described by the matrix A, and \(P'\) is set of corner points transformed by \(T_{A}\):

$$\begin{aligned} P' = T_{A}(P) = \left\{ p_{i}^{\prime }: \forall _{i\in (1, \ldots , n)} \, p_{i}^{\prime }=T_{A}(p_{i})\right\} . \end{aligned}$$
(16)

We also define the \(I'\) as the image I transformed by \( T_{A}\):

$$\begin{aligned} I' = T_{A}(I) \end{aligned}$$
(17)

The size of image is same as transformed image, which means that after geometric transform the image is cut to original size (see Fig. 1).

Let Q be a set of corner points on image \(I'\): \(Q = \left\{ q_{1}, q_{2}, \ldots , q_{m}\right\} \), where m is the number of points detected on image \(I'\).

Since the corner points of \(P'\) and Q are in the same coordinate system, it is possible to compute the Euclidean distance D between them. A pair \((p_{i}^{'} \in P^{'}, q_{j} \in Q)\) represents corresponding corner points (point \(p_{i}^{'}\) has the counterpart of \(q_{j}\)) in the context of the transformation \(T_{A}\) if and only if the following stability conditions are met:

$$\begin{aligned} D(p_{i}^{\prime }, q_{j}) \le D(p_{i}^{\prime }, q_{k}), \, \forall _{k}: k\in (1, \ldots , m) \wedge k\ne j \; ; \; D(p_{i}^{\prime }, q_{j}) \le \epsilon . \end{aligned}$$
(18)

Therefore, for each “original” point \(p_i\) we are looking for points being closest to “new” corner point \(q_i\) from transformed image such that its distance to \(p'_i\) is shorter than given \(\epsilon \) value. Consequently it is possible that some of the “original” corner points do not have a corresponding point in the transformed image. It may happen that the “original” point has several equally distanced corresponding points in the transformed image.

To measure the overall stability rate of detectors, the function S is introduced:

$$\begin{aligned} S(P') = \frac{k}{n'}, \end{aligned}$$
(19)

where \(k\le n\) is the number of points \(p'\in P'\) which has corresponding points in Q and \(n'\le n\) is the number of points in \(P'\) with coordinates that are located in transformed image range.

To measure the quality of detectors we use \(S(P^{'})\) functions for sequence of image transformations.

4.2 Results

Considering all 4 types of gradients (Eqs. 5–8), 4 formulas of cornerness function (Eqs. 12–15) and 3 low-pass filters (Eqs. 9–11), the total number of 48 detectors has been tested. Classic Harris detector, KLT and some proposed by us have been examined against minimal and mean values of \(S(P^{'})\) of the same sequence of affine transformations. Sequence consists of 170 image transforms including 72 rotations (every 5\(^{\circ }\)) of image, 20 different scale factors of image (from 95 to 105 %), 18 blurring factors, 20 levels of noise, 40 brightness values (60–150 %). Some of the chosen results (taking into account mean value of \(S(P^{'})\) results) are presented in Table 1. The measures included in the table refers to mean and minimum value of \(S(P^{\prime })\). Some of the results performs better than classic Harris and KLT method.

Table 1 Chosen results of tests (best methods are indicated in bold) and comparison with classic Harris and KLT methods
Full size table

5 Conclusions

The aim of a study presented in this paper was to compare various combinations of directional gradient cornerness functions and linear filters within the classic three-step schema of corner detection. As the results of experiments have shown some of these combinations provide us with methods witch perform better than classic Harris and KLT method. Taking into consideration diagonal gradient responses gives more accurate information about possible edge/corner presence. Also, using determinant version of cornerness function allows improving computational efficiency.