1 Introduction

In the field of image processing, image corners[1, 2] are important structural points with local image features. They usually refer to the interest points in the image that suddenly change the intensity and/or contour in all directions. Corner detection is an important image processing method[3,4,5], which is widely used in image feature extraction, image match and retrieval, high-level face (emotion) recognition, semantic image understanding, target detection and recognition, motion tracking, robot navigation and other fields. Corner detection is very natural for the human eye to recognize the corners in the images. The human eye can distinguish the real and false corners, accurately identify the position of the corners, be robust to noise, and can distinguish the importance of the corners. A corner detector has human eye characteristics that should also meet these standards. Many corner detection methods have been proposed in the past few decades. Most of these methods can be generalized to three categories: methods based on intensity change, methods based on contour angle, and methods based on intensity model. The method based on intensity change usually uses the first derivative, second derivative and wavelet operator of intensity to calculate the intensity change areas of the image as the corners. The method based on contour angle usually calculates the image contour first and then takes the areas with large turning angle in the contour as the corner points. The method based on intensity model locates the corner areas through the specific regional intensity models. At present, these three kinds of methods still have some main problems and have great room for improvement in detection performance. They cannot meet the standards that a corner detector with human eye features should meet. Additionally, these methods have their own advantages in technology, but they have not integrated various features such as intensity change, contour angle, intensity mode.

In this work, we focus on the fusion of multiple cues, such as intensity, pixel pattern, curvature, and scale, to come up with a high-performance corner detector, which provides a considerable solution to the main problems of the existing corner detection methods. We noticed that the most of the existing corner detection methods do not utilize the multiple cues, such as intensity, pattern, curvature, and scale, to measure the corners. Therefore, we introduce a composite model of both intensity, pattern, curvature, and scale as a possible solution for these problems. Firstly, a corner measurement function that reflects both the intensity, pattern, and curvature difference is formulated based on the 8-neighbor pixel blocks. Secondly, some scale-based global scale importance factors are formulated based on the contour distribution and corner distribution. Thirdly, based on the corner measure and the importance factors, a high-performance corner detector (IPCS) is derived. This method is a straightforward approach to the human eye like corner detector; it tries to identity the true and false corners, to compute the accurate position of the corners, to be robust to noise, to provide an importance evaluation metrics for corners. The experiments based on both the ground truth and the standard image set are conducted to evaluate the correctness and repeatability of the proposed detector. The experiment results come up with that the proposed detector has remarkable performance advantages among the comprising state-of-the-art detectors. Our contributions include: firstly, an effective model of intensity changes, named as 8-neighbors corner block intensity difference, that utilizes both first derivative and second derivative; secondly, a novel pixel pattern model that is robust to different types of corner conjunctions and noise; thirdly, a novel curvature model that can compute the contour curvature directly from intensity no considering the contour map; fourthly, a global importance evaluation model of the corners that utilizes multiple cues of global feature such as scale line factor and texture factor; at last, a fusion method of these individual measures that exhibit remarkable performance improvement. The paper is organized as follows: In Sect. 2, the related works are introduced; in Sect. 3, the individual local and global measures of the corners are introduced; in Sect. 4, the method of measure fusion is first introduced, and then the complete corner detection algorithm is proposed; in Sect. 5, the performance evaluation and experiment results are listed; in Sect. 6, the conclusion and future work are made and discussed.

2 Related work

The first widely used corner detection algorithm is the Harris algorithm [1] in 1988. The authors investigated the different patterns of corner area, edge area, and flat area in the images and came up with a statistics theorem about the corner points: The intensity of the around area of a corner point will change significantly along with any direction. Based on this theorem, they presented a rigid mathematic method to evaluate the corner feature. Because of its solid mathematic backbone, their algorithm is very concise, effective, and efficient. In the later days, many revised methods based on the Harris algorithm are presented, such as [6,7,8,9], and [10,11,12] which apply second-order derivative representation to identify corners. Different from the algebra and statistics-based methods, the methods based on pixel patterns make use of the unique pixel patterns of a corner area to identify the corner points. The early well-known pattern-based algorithm is the SUSAN algorithm [13] in 1997, and a newly published successor is the FAST algorithm [14] in 2010, which achieves a highly optimized computational efficiency. The FAST algorithm is immediately used in the mobile devices and embedded systems after its publication because it is the only method that can do the online detection in the video frames with acceptable delays in a resource-limited device at that time. Another well-constructed corner detection theory is the contour curvature-based methods, such as [15,16,17,18,19], and [20]. In their viewpoint, a corner point is a point that is in a contour line and has large curvature. Therefore, these methods mainly have two steps: firstly, extract and polish the contours from the image; secondly, compute the curvature measure of the points along the contours. Because these methods depend on the contour extraction results, on the one hand, they have relatively high accuracy and repeatability when there are high-quality contours even in a slightly noised condition; on the other hand, their performance is constrained because of that current contour extraction methods, such as [21,22,23] cannot always provide accurate results in condition that the image is with varying light, noise, and corner patterns. There are some main problems in the existing corner detection algorithms [10]. Firstly, the intensity derivative (gradient) operators may be failed to extract planar curves or corner measure metrics in some conditions[24]. For example, the T type corners in the ‘Geometric’ image are the joints of a strong edge and a weak edge. The large derivative of the strong edge interrupts the small derivative of the weak edge; thus, the large gaps are occurred and make the T type corners are wrongly identified as edges in the most of existing detectors. Secondly, the fragmental definition of pixel patterns makes the detector hard to be analyzed and designed. For example, the existing contour-based methods [15,16,17,18,19, 25], and [20] define mainly 5 types of corners: L type, T type, Y type, X type, and star type. The model-based methods [13, 14, 26], and [27] have even more types of pixel pattern definitions. Thirdly, the existing contour-based methods have problems in curvature calculation with local variation and noise [20]. Therefore, these methods usually have many missed corners that are not detected. Fourthly, the classic derivative measure-based methods and pixel pattern-based methods have only local measures, which makes them have many false corners caused by the noises or the wrong scales[8].

3 The proposed measures of the corners

There are mainly three kinds of functions to evaluate corners. Firstly, the intensity-based methods, such as [1, 6, 8], and [10], use the intensity changes, i.e., first-order derivative, second-order derivative, to evaluate corners; secondly, the model-based methods, such as [13, 14], and [27], use the pixel pattern to evaluate corners; thirdly, the contour-based methods, such as [15,16,17,18,19, 25], and [20], use the contour curvature to evaluate corners. In this section, the individual local and global corner measures in our method that involves both intensity, pattern, curvature, and scale are introduced.

3.1 Intensity measure

As we know that the corner points are in the cross edge between two or more flat areas in different levels of intensity, the two points that have the least intensity changes from the corner points are on the two nearest cross edges. The candidate corner area is divided into eight square pixel block sub-areas, the center block is the location of the corner point, it has the middle average intensity between the average intensities of the two flat areas, the one area has high intensity, and the other area has low intensity. There are two surrounding blocks that are both in the edge area, and the two blocks have the most similar average intensities as the average intensity of the center block. In this candidate corner area, if the least intensity changes of the 8-neighbors blocks are considerably large, it means that the intensity will be largely changed from the center block along with any direction. According to the analysis of [1], this area can be considered as a corner point location. Therefore, the intensity measure, i.e., corner block intensity difference CB, is defined as:

$$\begin{aligned} \begin{aligned} CB(x,y)&=\sum ((B(x,y)-B(x_1,y_1 ))^2 )\\&\quad +\sum ((B(x,y)-B(x_2,y_2 ))^2 ) \end{aligned} \end{aligned}$$
(1)

where B(x, y) is a \(w \times w\) point block; \(B(x_1,y_1 )\) and \(B(x_2,y_2 )\) are the two neighbor blocks with least intensity difference to B(x, y). Furthermore, the corner intensity difference CI is defined as:

$$\begin{aligned} CI(x,y)=(I(x,y)-I(x_1,y_1))^2+(I(x,y)-I(x_2,y_2))^2, \end{aligned}$$
(2)

where \(I(x_1,y_1)\) and \(I(x_2,y_2)\) are the average intensities of the two blocks in the 8-neighbors blocks with lest intensity difference to I(x, y). The corner intensity second-order difference C2I is defined as:

$$\begin{aligned} \begin{aligned} C2I(x,y)&= (I(x,y)-2*I(x_1,y_1)+I(x_3,y_3))^2\\&\quad + (I(x,y)-2*I(x_2,y_2)+I(x_4,y_4))^2, \end{aligned} \end{aligned}$$
(3)

where \((x_2,y_2)\) is the neighbor point of \((x_1,y_1)\) along the (x, y) to \((x_1,y_1)\) direction; \((x_4,y_4)\) is the neighbor point of \((x_2, y_2)\) along the (x, y) to \((x_2,y_2)\) direction.

Therefore, the contour intensity difference CTI is defined as:

$$\begin{aligned} \begin{aligned} CTI(x,y)&= \sqrt{CBI(x,y)*C2I(x,y)}+CBI(x,y),\\ CBI(x,y)&= \sqrt{CB(x,y)*CI(x,y)}.\\ \end{aligned} \end{aligned}$$
(4)

In the target area, the neighbor area of a point is divided into nine blocks. By applying a Gaussian filter as follows:

$$\begin{aligned} \begin{aligned} h_g(x,y)&= \exp {\left( \frac{-(x^2+y^2)}{2\sigma ^2}\right) },\\ h(x,y)&= \frac{h_g(x,y)}{\sum {h_g(x,y)}},\\ I_g(x,y)&= I(x,y)\otimes h(x,y).\\ \end{aligned} \end{aligned}$$
(5)

Considering the first-order derivative CBI of a block with its two neighbor blocks with the most equal intensities, the more the points close to the corner point, the larger CBI will be detected. In the same way, the more the points close to the corner point, the larger the second-order derivative C2I will be detected.

3.2 Pattern measure

It is known that the pixels of a corner have certain patterns. However, there are no uniform pattern measures for different types of corners. For example, the existing contour-based methods define mainly 5 types of corners: L type, T type, Y type, X type, and star type, and they use the different formulas for different corner types to calculate curvature. The fragmental definition of corner patterns makes the detectors hard to be analyzed and designed.

To cope with this problem, a corner pattern factor based on a local binary pattern is developed. In this definition, each of the 8 neighbors is a four squares block that has the width w and height w, (\(w=1,2,...\)). The 8 neighbor blocks of a point are ordered clockwise and marked in this way: a neighbor is marked as 1 if the neighbor block has larger intensity than the intensity of the center block; otherwise, it is marked as 0. By scanning the markers, the maximum length of a continues set of all 1 or all 0 is denoted as chr(x, y). Therefore, the corner pattern factor \(\alpha (x,y)\) of a point (x, y) is defined as:

$$\begin{aligned} \begin{aligned} \alpha (x,y)&= 1+min(chr2(x,y), 4-chr2(x,y)),\\ chr2(x,y)&= min(chr(x,y), 8-chr(x,y)), \end{aligned} \end{aligned}$$
(6)

where chr(x, y) is the boundary line distance between two connected 0 part and 1 part. Necessarily, the sub-pixel boundary line distance is interpolated by the intensity difference between the neighbor blocks and the center block.

According to the definition of the corner pattern factor \(\alpha \), the corner patterns can be uniformly measured. Because the chr is the length of the max connected 0/1 blocks, it not only reflects the main character of a junction but also avoids to measuring the patterns of all the blocks.

3.3 Curvature measure

3.3.1 Average contour curvature

The average contour curvature curv provides an initial curvature measure, and it is used to locate candidate corner area, evaluate the image quality, and choose the proper parameter set. It is defined as:

$$\begin{aligned} \begin{aligned} curv(x,y)&= \frac{1}{n} \sum _{i=1}^{n}{a_i},\\ a_i&= min(b_i, \pi - b_i), \\ b_i&= |atan(dx,dy) - atan(dx_i,dy_i)|,\\ \end{aligned} \end{aligned}$$
(7)

where \((x_i, y_i)\) are the contour points in the \(w\times w\) neighbor area, e.g., \(w = 5\), of point (x, y), dx, dy are the first-order derivative (gradients) of intensity alone the x- and y-axis.

3.3.2 Intensity-based measure

Different from the existing contour curvature-based method, in our method, the curvature is calculated through the gradient vectors of two points \((x_1, y_1)\) and \((x_2, y_2)\), corresponding to \(B(x_1,y_1 )\) and \(B(x_2,y_2)\) that are the two neighbor blocks with least intensity difference to B(x, y). It is known that the corner points are usually having much different gradient direction from its neighbor points.

Suppose the gradients of (x, y), \((x_1, y_1)\), and \((x_2, y_2)\) are \(dx_1\), \(dy_1\), \(dx_2\), \(dy_2\), \(dx_3\), \(dy_3\), respectively, the cosine angle between (x, y) and \((x_1, y_1)\) and cosine angle between (x, y) and \((x_2, y_2)\) can be computed from the angle between their gradient vectors \(v = [dx,dy], v_1 = [dx_1,dy_1], v_2 = [dx_2,dy_2]\).

Therefore, the contour curvature difference CTV is defined as:

$$\begin{aligned} \begin{aligned} CTV(x,y)&=\frac{(w_1* sinn(a_1)+w_2* sinn(a_2))}{(w_1+w_2)},\\ sinn(a_1)&=\sqrt{1-cos(a_1)^2 }+\sqrt{(1-cos(a_1))/2},\\ sinn(a_2)&=\sqrt{1-cos(a_2)^2}+\sqrt{(1-cos(a_2))/2},\\ w_1&=AB_{max}-AB|a_1,\\ w_2&=AB_{max}-AB|a_2,\\ AB_{max}&=max(AB|a=0,\frac{\pi }{8},\frac{\pi }{4},\frac{3\pi }{8},\frac{\pi }{2},\frac{5\pi }{8},\frac{3\pi }{4},\frac{7\pi }{8})\\ AB|a&= CB|a + CI|a \end{aligned} \end{aligned}$$
(8)

where CB|a and CI|a are the block intensity difference and the point intensity difference in the 8 directional neighbors as defined in the intensity measure section.

3.3.3 Contour-based measure

In the corner refining procedure, the contour curvature is measured more accurately in a larger \(w \times w\) neighbor area, e.g., \(w = 15\), and in a contour-based manner. In this measure, the corner area is separated from two sub-areas (center area and surround area), and the contour is departed from serval departing branches by excluding the contour in the center area. Give a candidate corner point (x, y), and its \(w\times w\) neighbor area, the refined contour curvature difference CTV2 and average contour curvature curv2 is defined as:

$$\begin{aligned} \begin{aligned} CTV2(x,y)&= max(2*sin(0.5*A)), \\ curv2(x,y)&= mean(B), \\ A&= \{\arccos (\cos (a_i))|i=1,...,n\},\\ B&= \{\arccos (\cos (b_i))|i=1,...,n\},\\ \end{aligned} \end{aligned}$$
(9)

where \(a_i\) is an angle between two contours branches in the \(w \times w\) neighbor area excluding the inner centrum area; \(b_i\) is angle between a contour point in the centrum area and a contour branch in the \(w \times w\) neighbor area.

3.4 Global scale importance factors

The most of existing corner measures provide the local weights of the corners and provide no global weights in a large scale. In following section, 5 global scale importance factors are derived from the corner measures CTV, curv, the contour distribution, and the corner points distribution. Given a contour map C, a connected contour map C2 is produced by labeling each connected contour component a unique number; a connected line map C3 is produced by labeling each straight part of a connected contour component a unique number.

[CTV factor]

$$\begin{aligned} \begin{aligned} GT(x,y)&= \frac{1}{3}\left( \frac{1}{2}\left( mean\left( CTV\left( x_i,y_i\right) \right) \right) \right) \\&\quad +std\left( CTV\left( x_j,y_j\right) \right) \\&\quad +mean\left( CTV\left( x_k,y_k\right) \right) \\&\quad +mean\left( CTV\left( x_{ii},y_{ii}\right) \right) \\&\quad +mean\left( CTV\left( x_{jj},y_{jj}\right) \right) \\&\quad -std\left( CTV\left( x_{ii},y_{ii}\right) \right) , \end{aligned} \end{aligned}$$
(10)

where \((x_i,y_i)\) are the points in the same connected contour component as (x, y), \((x_j,y_j)\) are the points in the same connected contour component and have lower CTV than the mean CTV of this component, \((x_k,y_k)\) are the points in the same connected contour component and have higher CTV than the mean CTV of this component, \((x_{ii},y_{ii})\) are the points in the same connected contour component of C2 as (x, y) of the \(w\times w\) neighbor area, \((x_{jj},y_{jj})\) are the points in the same connected contour component of the \(w\times w\) neighbor excluding the inner centrum area.

[Curv factor]

$$\begin{aligned} \begin{aligned} GV(x,y)&= \frac{1}{3}\left( \frac{1}{2}\left( mean\left( curv\left( x_i,y_i\right) \right) \right) \right) \\&\quad +std\left( curv\left( x_j,y_j\right) \right) \\&\quad +mean\left( curv\left( x_k,y_k\right) \right) \\&\quad +mean\left( curv\left( x_{ii},y_{ii}\right) \right) \\&\quad +mean\left( curv\left( x_{jj},y_{jj}\right) \right) \\&\quad -std\left( curv\left( x_{ii},y_{ii}\right) \right) , \end{aligned} \end{aligned}$$
(11)

where \((x_i,y_i)\) are the points in the same connected contour component as (x, y), \((x_j,y_j)\) are the points in the same connected contour component and have lower curv than the mean curv of this component, \((x_k,y_k)\) are the points in the same connected contour component and have higher curv than the mean curv of this component, \((x_{ii},y_{ii})\) are the points in the same connected contour component of C2 as (x, y) of the \(w\times w\) neighbor area, \((x_{jj},y_{jj})\) are the points in the same connected contour component of the \(w\times w\) neighbor excluding the inner centrum area.

[Line factor]

$$\begin{aligned} \begin{aligned} GN(x,y)&= \frac{Num(connected(curv(x_i,y_i)<Nb))}{Num(uniuqe(C3(x_i,y_i)))}, \end{aligned} \end{aligned}$$
(12)

where \((x_i,y_i)\) is the points in the same connected contour component of C2 as (x, y); the numerator of GN is the number of the connection points that have lower curv than the curvature bound Nb, Nb is usually a considerable small value, e.g., 0.3; the denominator of GN is the number of connected straight-line components of C3 in the same connected contour component of C2 as (x, y).

[Texture factor]

$$\begin{aligned} \begin{aligned} GT(x,y)&= max(sort(I(x_j,y_j))|a - sort(I(x_i,y_i)))\\ a&=0,\frac{\pi }{8},\frac{\pi }{4},\frac{3\pi }{8},\frac{\pi }{2},\frac{5\pi }{8},\frac{3\pi }{4},\frac{7\pi }{8}. \end{aligned} \end{aligned}$$
(13)

where \(sort(I(x_i,y_i))\) is the sorted intensities of the center block of (x, y); \(sort(I(x_j,y_j))\) is the sorted intensities of the 8-neighbors blocks of the center block of (x, y).

[Composition factor]

Given the raw corner map CC, the corner distribution map G is produced by counting the number of corners in the \(w \times w\) neighbor areas, and w should be considerably large. Given the intensity map I and a large enough \(\sigma \), e.g., \(\sigma =5\), a texture suppressed gradient magnitude map mag2 is generated. Based on the corner distribution map G, the texture suppressed gradient magnitude map mag2, the corner evaluation map F, and the corner intensity map CI, a composition factor \(\beta \) for global scale importance can be derived to refine the corners.

$$\begin{aligned} \begin{aligned} \beta (x,y)&= \sqrt{\frac{mean(G)*GN(x,y)*vfn(x,y)}{G(x,y)*mean(GN)}},\\ vfn(x,y)&= \frac{vf(x,y)}{mean(vf)},\\ vf(x,y)&= \root 3 \of {F(x,y)* CI(x,y)^2*mag2(x,y)} \end{aligned} \end{aligned}$$
(14)

where \((x_i,y_i)\) is the detected corner points in the raw corner map CC.

4 The new corner detector

4.1 The fusion methods

The measures of our method are categorized into two types: local measure and global measure. The local measures include the contour intensity difference CTI, the corner pattern factor \(\alpha (x,y)\), the refined contour curvature difference CTV2, and average contour curvature curv2. The global scale importance factors include CTV factor, curv factor, Line factor, Texture factor, Composition factor. The first fusion includes the three local measures formulated by composing the first- and second-order intensity derivative-based contour intensity difference CTI(x, y), the intensity-based contour curvature difference CTV(x, y), and the pattern-based alpha factor \(\alpha (x, y)\) as follows:

$$\begin{aligned} F(x,y)=(0.5+\alpha (x,y))\times CTI(x,y)\times CTV(x,y) \end{aligned}$$
(15)

The local fusion function F(x, y) is used to locate corner candidates that are the locally maximum points in the F value map, and then the global importance factors are computed only for those candidate corner areas. By applying the importance threshold rules for each global factor, the accurate and relatively important corners that is considered similar to the human eye standards will be filtered out.

4.2 The detection algorithm

Based on the definition of the new corner evaluation function F(x, y), a complete corner detection method is organized as three parts: the first pre-processing part that evaluates the noise rate of the image and does the Gaussian smooth filter and median de-noising filter; the second local measure computing part that computes the local measures and identifies the corner candidates by the local thresholds; the last global measure computing part that computes the refined corner measures and the global factors for the corner candidates, and locates the final corner points. For the sake of efficiency, the algorithm needs to address some problems. Firstly, to avoid doing the F(x, y) calculation for every pixel, the new method only scans the candidate areas whose edge pixels have large curvature within a 5x5 window. Secondly, to avoid manually tuning the parameters, the new method calculates the average curvature, or noise rate, (\(nsr=mean(curv)\)) in advance. If an image whose average curvature is high, i.e., \(nsr > 0.5\), the image may be a noised image. Thus, the method convolves the image by a median filter and adopts a relatively larger sigma and filter window to do the Gaussian smoothing. Thirdly, to identify those local important corner points, the initial candidate corners are generated from the function F(x, y) and the F thresh parameter (th), CTI thresh (th2), CTV thresh th3. Fourthly, to identify those globally important corner points, the initial candidate corners need to be refined according to the refined curvature and the global importance factors. The detailed algorithm is listed as follows.

figure h

5 Performance evaluation and experiment results

In this section, the performance of the proposed corner detector is evaluated and compared with the state-of-the-art corner detectors in a serial of experiments [17, 28], and the experiment results are reported. The 5 state-of-the-art corner detectors (ATCSS[16], ARCSS[17], CDPA[18], ANDD[19], and MTCSS[20]) are compared with the proposed detector in our experiment. To avoid the use of posterior knowledge, each of the detectors uses a default fixed parameter set suggested by the authors’ implementation codes if no automatic procedures to optimize the parameters are provided. There are 5 evaluation indexes (Missed corners (Mn), False corners (Fn), Average repeatability (R), Average match error (Me), and Average localization Error (Le)) are used in our experiment. The first index is defined as the number of missed corners. Missed corners are the corners of ground truth that are not detected by a detector in their k-neighborhood; k is usually between 5 and 8 pixels. The second index is defined as the number of false corners. False corners are the corners detected by a detector that are not in the N-neighborhood of ground truth corners. The third index is Average repeatability. It is defined as

$$\begin{aligned} R = \frac{Nr}{2*(1/No+1/Nt)} \end{aligned}$$
(16)

where No and Nt denote the number of detected corners from the original and transformed images by a detector, and Nr denotes the number of matched corner pairs in the original and transformed images. In case of that multiple transformed corners are matched to a single original corner, it can only be counted once. The fourth index is average match error Me; it is defined as \(Me = 1 ... R\). The fifth index is average localization error Le; it is defined as:

$$\begin{aligned} Le = \frac{1}{N}\sum {\sqrt{(tx_i-x_i)^2+(ty_i-y_i)^2)}} \end{aligned}$$
(17)

where \((tx_i,ty_i)\) is the coordination of the matched transformed corners, \((x_i,y_i)\) is the coordination of their corresponding original corners, N is the number of matched corner pairs.

5.1 Evaluation of detection performance based on ground truth

Firstly, the images with labeled ground truth corners are used to evaluate the correction of corner detection in normal and noisy circumstances. Figure 1 lists the four commonly used ground truth images (Geometric, Chessboard, Block, and Lab), which are used in this experiment.

Fig. 1
figure 1

Images with labeled ground truth corners

Full size image

The image ‘Geometric’ has 60 corners in varying intensity and curvature; the image ‘Chessboard’ has 80 corners with camera distortion and noise; the image ‘Block’ has 57 corners in varying light and shadow; the image ‘Lab’ has 249 corners in varying scale and pattern. The 6 detectors are applied on the four test images, and the proposed detector (short for P) executed in five modes, the four modes are in fixed thresholds (0.5, 1.0, 2.0, 4.0), and the one mode is in automatically selected threshold by experimental formula. The detection results are compared with the ground truth, and the missed corners (Mn), false corners (Fn), average match error (Me), and average localization error (Le) are listed in Table 1:

Table 1 Detection correctness of the 6 corner detectors for ground truth (max Le = 5)
Full size table

As shown in Table 1, the proposed detector has better performance in the most of test subjects. Specifically, for missed corner subject, the proposed detector finds more correct corners in the all four images than the comparisons; for the false corner subject, the proposed detector has slightly more false corners in ‘Block’ and ‘Lab’ than the best records; for the match error subject, the proposed detector has less match error in the all four images than the comparisons; for the localization error subject, the proposed detector has slightly more localization error in ‘Geometric’ and ‘Lab’ than the best records. For different thresholds of the proposed detector, a small threshold trends to lead a sensitive result that has less missed corners and more false corners; in contrary, a large threshold trends to lead a insensitive result that has more missed corners and less false corners. The automatic threshold leads to a general balanced good result.

To investigate the performance of the individual measures (Intensity, Pattern, Curvature, Scale), we conduct a test on the ground truth figures for the individual measures. The test results are listed in Table 2

Table 2 Detection correctness of the individual measures of IPCS for ground truth (max Le = 5)
Full size table

As shown in Table 2, the intensity measure (P-I) misses least true corners and reports least false corners, least measure, and location error among the three individual measures (P-I, P-P, P-C). The combination of intensity, pattern, curvature measures (P-IPC) has overall better performance than the individual measures. The scale importance factor improves the overall performances of the individual measures and combination measure, while the full combination P-IPCS achieves the best performance.

The detection result images of the 6 corner detectors on the four ground truth images are listed as follows.

Fig. 2
figure 2

ATCSS corner detection results on the ground truth images

Full size image
Fig. 3
figure 3

ARCSS corner detection results on the ground truth images

Full size image
Fig. 4
figure 4

CPDA corner detection results on the ground truth images

Full size image
Fig. 5
figure 5

ANDD corner detection results on the ground truth images

Full size image
Fig. 6
figure 6

MTCSS corner detection results on the ground truth images

Full size image
Fig. 7
figure 7

Proposed corner detection results on the ground truth images

Full size image

As shown in Fig. 7, the proposed detector has better results in human vision because it is both locally and globally weighted to select final corners. The significant corners that in large scale and strong measure response are more possible to be found and remained, while the corners in small scale and weak measure response are less possible to be remained. Specifically, in the ‘Geometric’ image, the proposed detector finds the most of the T type corners that are the junctions of a strong edge and a weak edge, while none of the comparing detectors find these corners; in the ‘Chessboard’ image, the proposed detector not only has the 100% match rate but also has the least localization error; in the ‘Block’ image, the proposed detector finds the most true corners even 3 detected corners near to the ground truth corners are labeled as false corners because they exceed the distance threshold; in the ‘Lab’ image, the proposed detector finds the most true corners and slight more false corners than the best records.

To evaluate the detection robustness in noisy circumstances, the four original test images are added with zero-mean Gaussian white noise with variance from 1 to 20, and then the six detectors identify corners in these noised images. The average match errors and average localization errors are listed in Fig. 8.

Fig. 8
figure 8

Average detection repeatability of the 6 corner detectors with Gaussian noise

Full size image

As shown in Fig. 8, the proposed detector has better average match performance among the 6 detectors, and its’ average localization error is also better than the most of the comparisons.

5.2 Evaluation of detection performance based on average repeatability

The USC-SIPI image database is a commonly used image processing test set. In this experiment, the volume 3 (Miscellaneous) of USC-SIPI is used for the evaluation of detection repeatability because this representative volume contains both artificial images and natural images, noise-free images and noisy images, indoor images and outdoor images, and many other scenes.

We had a total of 8,268 transformed test images which were obtained by applying the following six different transformations on the 39 original images:

Rotations: The original image was rotated at 18 different angles within \([-\pi /2,\pi /2]\) with a \(\pi /18\) interval.

Uniform scaling: The original image was scaled with scale factors \(Sx = Sy\) in [0.5, 2.0] with 0.1 interval, excluding 1.0.

Non-uniform scaling: The scale factors Sx and Sy were chosen by sampling the ranges [0.7, 1.5] and [0.5, 1.8] with a 0.1 interval.

Shear transformations: The shear factor c was chosen within the range [-1, 1] with a 0.1 interval, excluding 0.

JPEG compression: A JPEG quality factor was chosen within the range [5, 100] with an interval of 5.

Gaussian noise: Zero mean white Gaussian noise was added to the original image at the standard deviations chosen within [1, 20] with an interval of 1.

The results of average repeatability with different rotations, uniform scalings, non-uniform scalings, shear transformations, JPEG compressions, and Gaussian noises are shown in Fig. 9.

Fig. 9
figure 9

Average detection repeatability of the 6 corner detectors (max Le = 5)

Full size image

As shown in Fig. 9, the proposed detector has better performance in all the six transformations. Specifically, because the 8-neighbors blocks or 8-directional first and second derivative provide more accurate direction and corner measure, the remarkable improvements are achieved on the rotation and shear transformations; benefit from the scalable 8-neighbors block, the noticeable improvements are achieved on the uniform/non-uniform scaling transformations; due to the robustness of the corner importance factors, the moderate improvements are achieved in the JPEG compression and the Gaussian noising transformations.

Additionally, the proposed detector is examined in 5 different threshold modes (0.5, 1.0, 2.0, 4.0, auto) to test its average repeatability. The test result is listed in Fig. 10.

Fig. 10
figure 10

Average detection repeatability of the proposed detector in 5 threshold modes

Full size image

As shown in Fig. 10, the proposed detector usually has better repeatability with a lower threshold, while it usually has more false corners at the same time. The automatic threshold selection usually achieves similar repeatability as the best records of the fixed threshold mode while remaining better detection correctness.

The results of the general average match error and average localization errors of the all transformed test images in the average repeatability experiments are listed in Table 3:

As shown in Table 3, the proposed detector has both the better repeatability and the better localization performance.

5.3 Average runtime

The six detectors are implemented in MATLAB R2015, and their average running time is evaluated in a PC with 3.5GHz CPU and 16GB Memory. The time of the six detectors used to perform the average repeatability experiments on the USC-SIPI test images and their transformed images are used to evaluate the computational efficiency. The test images are categorized into 3 types (\(256\times 256\), \(512\times 512\), \(1024\times 1024\)) by the sizes of images. The average time usage of the six detectors is listed in Table 4:

As shown in Table 4, the proposed detector usually needs 2 to 5 times time of the best records of the comparisons. It is remarkable that the time usage of the proposed detector is becoming closer to the best records when the image size is increasing.

Table 3 Average repeatability and localization error
Full size table
Table 4 Average runtime
Full size table

6 Conclusion and future work

In this work, we proposed a corner detection approach that is a fusion of multiple cues, such as intensity, pixel pattern, curvature, and scale. Different from the most of existing corner measure functions, a new corner measure that reflects the influences of intensity, pixel pattern, and curvature in the 8-neighbors blocks is introduced. In this measure, the intensity difference is an 8-directional invariant derivative operator; the uniform pattern factor is a local binary pattern operator, which can suppress the noise and edge influence; the 8-neighbors curvature calculation needs no trace of curves along the contour. In addition, a refined curvature calculation method based on contour curvature and some global importance factors is used to refine the results by eliminating substantial corners and remaining major corners according to both local and global weights. Therefore, a high-performance corner detector is developed based on the proposed measure. In this detector, before the measure calculation, the average contour curvature is used to limit the candidate corner area, rate the image quality, and choose the proper parameter set. After the corner measure calculation and the local extreme localization, a global refine procedure is conducted to identify the final corners. A set of experiments based on both ground truth and standard test image sets are conducted to evaluate the correctness and repeatability of the proposed detector. The experiment results confirm that the proposed detector has an overall better performance comparing with the 5 state-of-the-art detectors. The main reasons for this advantage include: Firstly, the general model of pattern, intensity, curvature, and scale provides a more accurate corner measure; secondly, the usage of both local and global weights makes the detection results are more reasonable in human vision. At last, although the proposed method achieves considerable advances, the problem still remains, especially in the global corner refining procedure. We found that the proposed detector can always find sufficient raw corners efficiently, but the refining procedure is not always effective and efficient. Therefore, a more advanced corner importance measure may be a possible way to make the detection more correct, more reasonable, and faster.