1 Introduction

Content-based image retrieval (CBIR) has been a widely used image retrieval technique in various computer vision applications such as in medical field for getting the past patient details, E-commerce for finding the required products, information retrieval for retrieving images from the large database and so on. Unlike other retrieval techniques, CBIR uses image descriptor such as texture pattern, contour, local pattern, color and other image features to represent the visual contents of an image. The success of image retrieval system depends on the effectiveness of the features that are used to represent these image descriptors. The descriptors for CBIR can either be derived from spatial data [1], in which the features are extracted from image pixel values directly or from transformed image [2], in which features are extracted after transforming the image using image transform.

The spatial features are widely used in real-time applications for its simplicity and less complexity; however, it is more prone to external factors such as noise, lighting, gray level variance etc. Some examples of spatial features are LBP [3], HOG [4], spatial Co-occurrence [5], SIFT [6] etc. In contrast, transform domain features are less disturbed by such external factors but the complexity is quite high. Different transforms such as Fourier transform [7], discrete cosine transform [8], wavelet transform [9], ridgelet transform [10], contourlet transform [11], shearlet transform [12, 13] etc., are used to derive the image descriptors. The success of transform domain depends on how efficiently the image discontinuities (point, line and curve) are captured by the transformed data and also the effectiveness of the descriptors that are derived from the transformed data. Statistical features are most commonly used descriptor for the transform domain which captures the global variations in an image [14, 15]. Recently, lots of interest are shown by researchers to use the local features in the transformed domain such as local wavelet pattern [16], Local Shearlet-Based Energy Histograms [17], Discrete Cosine Transform in LBP space [18] etc.

The success of local binary pattern (LBP) in image processing applications led to lot of research in feature extraction based on spatial techniques. Ojala et al. proposed LBP [19] based on the assumption that the texture has local pattern and strength, LBP tries to capture these information as features. The main advantages of LBP are

  1. 1.

    Easy to understand the technique

  2. 2.

    Less complex to implement

  3. 3.

    Invariance to illumination changes

Due to the above advantages, LBP has been effectively used in several computer vision applications such as content-based image retrieval [20], recognition of faces [21, 22], texture recognition [23], recognition of genders [24], investigation of facial paralysis [25], investigation of medical images [26], Detection of abnormalities in mammogram images [27] and so on. Recently, deep learning (DL) has been a tremendous success in various computer vision applications [28, 29] including content-based image retrieval [30]. Convolutional neural network (CNN) [31] based architectures are becoming popular in deep learning and are used in many real-time applications due to the availability of dedicated hardware accelerators [32]. LBP has been used in deep learning to improve the overall system performance; Meng et al. [33] replaced the trainable kernels in CNN with LBP descriptor to achieve high recognition accuracy without requiring any costly model learning approach; Haijun Zhang et al. [34] proposed a technique for video advertising by combining CNN (for gender recognition) and HOG/LBP (for cloth identification).

To improve the performance, different versions of LBP have been proposed such as local ternary pattern (LTP) [35] which includes more detailed information by extending the LBP from two symbol codes to three symbol codes; Dominant LBPs [36], which uses the most frequently occurred pattern to make LBP less sensitive to image rotation and also more robust to noise; Completed LBPs [37], which is a generalized form the LBP with more completed information by expressing the sub-images with its center pixel and neighbor difference signed magnitude transform; LBP variance with global matching (LBPV) [38], which estimates the principal orientation and uses them to align the LBP histograms thereby including the spatial information to the local features of the LBP and so on.

Similarly, many derivatives are proposed to improve the robustness of LBP for the degraded images due to blurring and noise. Some of the techniques are adaptive hybrid pattern (AHP) [39] a noise robust hybrid pattern for texture analysis, Median Binary Pattern [40]—a binary texture metric based technique for texture categorization, Local Phase Quantization [41]—a descriptor based on Fourier transform by quantizing its phase information to overcome the image blurriness, fuzzy local binary pattern [42]—a variant of LBP to overcome the texture noise using fuzzy modeling technique, noise tolerant local binary pattern [43]—a LBP operator to extract the structural information for better texture analysis, and noise-resistant local binary pattern [44]—LBP with inbuilt error rectification technique to overcome the image noise.

In order to effectively capture the image details into the binary pattern, direction/gradient based patterns are being used by many researchers in different applications. Ahmed [45] uses the gradient directional pattern formed by quantizing the directional angles of the gradient for recognizing the facial expression. Local directional edge patterns formed from the edge directional magnitude are used for face recognition by Santosh et al. [46]. Structural binary gradient patterns are formed from the multidirectional image gradients and are used for face recognition by Weilin et al. [47]. Directional wavelet LBP by combining the idea of LBP with stationary wavelet transform was used by Maryam et al. [48] for human action recognition. Directional wavelet pattern [49] is used by Murala et al. for biomedical image indexing and retrieval. Gabor directional binary pattern [50] is used by Hongzhi for gaze estimation.

Most of these techniques rely on the directions or gradients estimated from the local regions over the predefined set of pixels for deriving the local patterns. In this work, a modified local pattern technique is proposed for image retrieval called local curve pattern (LCP) in which the patterns are derived from the pixels that are selected in runtime based on the image characteristics. Here, the pixel lies over the curves and lines are selected and are used to form the local pattern. This approach will capture more accurate information on curves and edges that are the main perceptual component of images. The proposed technique is compared with the conventional local pattern techniques and the result shows good improvement over the existing local pattern techniques.

The organization of paper is as follows: Overview of local pattern techniques is given in Sect. 2. Section 3 discusses the proposed local curve pattern in detail. The proposed image retrieval system is presented in Sect. 4, and the experimental setup and results are discussed in Sect. 5.

2 Review of local patterns

2.1 Local binary pattern (LBP)

LBP technique was proposed by Ojala et al. [19] for image classification and is successful in many image processing applications such as identification of faces [21], content-based image retrieval [51], object recognition [52], pedestrian detection [53] etc.. LBP is an operator that converts the local distribution of an image into a binary/integer pattern. The success of LBP is based on the fact that it is able to represent the local variation of an image more efficiently.

The formation of LBP pattern is very simple and straightforward which is an added advantage of LBP. For a given center pixel, LBP is derived by comparing its pixel value with its N-point neighbors, where N = 8 in our case. Each pixel is compared with its neighbors in a 3 × 3 neighborhood by subtracting the center pixel value, if the result is negative, it is encoded as “Zero”, else “One”. For each pixel, a pattern is derived by concatenating all these binary labels in clockwise direction. The equivalent decimal value of the binary pattern is then used for labeling the given pixel and is called LBP. For a given center pixel (x, y), LBP can be expressed as given below.

$${\text{LBP}}_{{P,R}} \left( {x_{c} ,y_{c} } \right) = \sum\limits_{{p = 0}}^{{P - 1}} S \left( {g_{p} - g_{c} } \right)2^{p}$$
(1)

where \(g_{c}\) and \(g_{p}\) are respectively the values of the center pixel and P equally spaced circle neighbors with radius R. The LBP transfer function S(x) is defined as

$$S\left( x \right) = \left\{ {\begin{array}{*{20}l} {1 ;} \hfill & \quad {x \ge 0} \hfill \\ {0 ;} \hfill & \quad {x < 0} \hfill \\ \end{array} } \right.$$
(2)

2.2 Local tetra pattern

LBP uses only two directions to capture the local gray level difference of an image, whereas LTrP [54] showed that the performance can be improved by considering more directional information and it uses four directions (45, 90, 135 and 180) to represent the local gray level variations. Given a center pixel in an image, LTrP is computed by finding the four different directional numbers (Dr = 1, 2, 3, 4) using the horizontal (x-axis) and vertical (y-axis) gradients and then forming the tetra patterns using the directional numbers. Let \(H_{\text{g}}\) and \(V_{\text{g}}\) be the gradients, respectively, in x-axis and y-axis directions with respect to the center pixel; then, the directional number is given as follows,

$${\text{Dr}} = \left\{ {\begin{array}{*{20}l} {1,} \hfill & \quad {H_{\text{g }} \ge 0 \;{\text{and}}\; V_{\text{g}} \ge 0} \hfill \\ {2,} \hfill & \quad {H_{\text{g }} < 0 \;{\text{and}}\; V_{\text{g}} \ge 0} \hfill \\ {3,} \hfill & \quad {H_{\text{g}} < 0 \;{\text{and}}\; V_{\text{g}} < 0} \hfill \\ {4,} \hfill & \quad {H_{\text{g}} \ge 0\;{\text{and}}\; V_{\text{g}} \ge 0} \hfill \\ \end{array} } \right.$$
(3)

With Eq. (3), the image will be converted into four directional values. Let the directional number of center pixel be “1”; then, the local tetra pattern can be derived by separating it into three patterns as given below:

$${\text{LTrP}}|_{{{\text{Dr}} = 2,3,4}} = \mathop \sum \limits_{p = 1}^{P} 2^{p - 1} \times f\left( t \right)|_{{{\text{Dr}} = 2,3,4}}$$
(4)
$$f\left( t \right)|_{t = \emptyset } = \left\{ {\begin{array}{*{20}l} {1,} \hfill & \quad {{\text{if}}\; t = = \emptyset } \hfill \\ {0,} \hfill & \quad {\text{else}} \hfill \\ \end{array} } \right.$$
(5)

Similarly other patterns are formed for the center pixel with other directional numbers and there will be a 12 (4 × 3) binary patterns for the local tetra patterns.

3 Local curve pattern (LCP)

3.1 Motivation

Although conventional local patterns are attractive for its simplicity, the performance is limited since it captures the local structure only on the predefined pixels that restricts the representation of local structures by these patterns. Also, lines and curves which are the major contributor of an image are not efficiently captured by the conventional local pattern techniques. To overcome this, we derive the pattern by navigating over the pixel beyond its neighbors through a predefined criterion so that it will capture more detailed local information along with local line and curve patterns effectively. Figure 1 depicts the comparison of the pixels used by LBP and LCP techniques.

Fig. 1
figure 1

Representation of LBP and LCP formation. a Pixels used for LCP. b Pixels used for LBP

3.2 Modeling

The idea of local curve pattern is to capture the local curve/line patterns by navigating over their structures. Without loss of generality, we assume the pixel under consideration is on the line or curve of an image. Then, the curve/line pattern can be captured by traversing over its smoothness and the direction of the traverse is determined by its neighboring pixels. Let \(I\) be the input image and \(I\left( {x,y} \right)\) be the pixel under consideration for deriving the local pattern, then the curve direction can be found by traversing the direction with minimum absolute difference over the pixels in all the directions as shown in Fig. 2. The dark arrow line in the figure shows the direction of pixel with minimum absolute difference when compared with other neighbors.

Fig. 2
figure 2

Traversing over the curve

With this approach, the Center pixel is moved to the direction of the curve for deriving each symbol of the local pattern and will capture the local curve/line pattern with an 8 symbol pattern. Mathematically local curve pattern at a point \(\left( {x_{c} ,y_{c} } \right)\) is defined as

$${\text{LCP}}\left( {x_{c} ,y_{c} } \right) = \mathop \sum \limits_{p = 0}^{L - 1} {\text{DN}}\left( {C_{p} } \right)B^{p}$$
(6)

where \(L\) is the length of the curve, B is the number of unique symbols, \(C_{p}\) is the center pixel of pth symbol and \({\text{DN}}\left( p \right)\) is the direction number of the pth pixel on the curve. Unlike the conventional local patterns, the center pixel for deriving each symbol will vary in LCP and the next center pixel is selected based on the curve direction estimated as given below.

$$C_{p + 1} = C_{n} |_{{\left\{ { \hbox{min} \left( {abs\left( {C_{p} - C_{n} } \right)} \right)} \right\}, \quad n = 0 \;{\text{to}}\;N - 1}}$$
(7)

where \(N\) is the total number of neighbors. As per Eq. (7), the direction of curve is estimated based on the minimum distance from the current center pixel, i.e., the next center pixel is the one which is more close to the current center pixel. The direction number \({\text{DN}}\left( p \right)\) is derived based on the number of directions the curve can navigate. In this work, we considered 3 × 3 window with 8 directions so that the direction number will be from zero to seven as in Eq. (8)

$${\text{DN}}\left( p \right) = n ;\quad {\text{for}}\;{\text{which}} \left\{ {abs\left( {C_{p} - C_{n} } \right)} \right\} \;{\text{is}}\;{\text{minimum}}\;| n = 0\; {\text{to}}\; N - 1$$
(8)

With N = 8, there will be eight unique symbols and the length of the curve is also eight, so there will have \(8^{8} = 16777216\) unique bins in the local pattern. This is very high considering that the histogram is used to derive the feature vectors. To overcome this we divide the eight directions into two four directions as in Fig. 3 so that there will be \(4^{8} = 65536\) bins in each four directions and the total of 131,072 bins in the local pattern. To further reduce the bins, we use the technique which was used in local tetra pattern [54] to represent the four directions.

Fig. 3
figure 3

Splitting eight directions into two directional patterns

After splitting into two four directions, assume we have four direction numbers (1, 2, 3 or 4) for a given center pixel \(C_{p}\) and the local curve pattern at a point \(\left( {x_{c} ,y_{c} } \right)\) is defined as

$${\text{LCP}}^{i} \left( {x_{c} ,y_{c} } \right) = \mathop \sum \limits_{p = 0}^{L/2 - 1} {\text{DN}}\left( {C_{p} } \right)B^{p}$$
(9)

where \(i = 0 \;{\text{and}} \;1\). As a result, the image will be converted into two images of tetra patterns and these tetra pattern images can be represented into the binary pattern as follows.

  1. 1.

    Based on the directional number, separate the tetra pattern into four patterns for a given center pixel as given below

    $${\text{i}} .\quad {\text{LCP}}^{i} \left( 1 \right)\left| {{}_{{L_{\text{Dir}}^{i} \left( {C_{p} } \right) = 1}}^{ } } \right.\quad {\text{ii}} .\quad {\text{LCP}}^{i} \left( 2 \right)\left| {{}_{{L_{Dir}^{i} \left( {C_{p} } \right) = 2}}^{ } } \right.$$
    $${\text{iii}} . \quad {\text{LCP}}^{i} \left( 3 \right)\left| {{}_{{L_{\text{Dir}}^{i} \left( {C_{p} } \right) = 3}}^{ } } \right.\quad {\text{iv}} .\quad {\text{LCP}}^{i} \left( 4 \right)\left| {{}_{{L_{\text{Dir}}^{i} \left( {C_{p} } \right) = 4}}^{ } } \right.$$
  2. 2.

    Convert each part into three binary patterns based on the direction numbers. Let the direction number of the center pixel be ‘one’ and its pattern can be formed by separating it into three different binary patterns as follows.

    $${\text{LCP}}^{i} |_{{{\text{direction}} = 2,3,4}} = \mathop \sum \limits_{p = 1}^{8} 2^{p - 1} \times h\left( {{\text{LCP}}^{i} \left( {C_{p} } \right)} \right) |_{{{\text{direction}} = 2,3,4 }}$$
    (10)
    $$h\left( {{\text{LCP}}^{i} \left( {C_{p} } \right)} \right) = \left\{ {\begin{array}{*{20}l} {1,} \hfill & {{\text{if}}\;{\text{LCTrP}}\left( {C_{p} } \right) = {\text{direction}}} \hfill \\ {0,} \hfill & {\text{otherwise}} \hfill \\ \end{array} } \right.$$
    (11)

This is extended to other directional numbers so that 12 different binary patterns are formed from the LCP directional numbers. For local patterns with 8 point neighbor, there will be \(2^{8} = 256\) bins which are quite high considering the computational cost and storage requirements. To reduce the feature dimension, uniform patterns are proposed in [55]. A local pattern (LBP) is called uniform if the total number of bitwise transitions from zero to one or vice versa is less than three when the bit pattern is considered as circular. For 8 point neighborhood, there are 58 uniform bins and other symbols are combined to a single value so that there are 59 bins in the uniform pattern. Similar approach is applied here for the local curve pattern to reduce the size of the bins so that each binary pattern of the tetra pattern will have 59 bins.

After forming the local curve pattern, feature vectors are extracted by building the histogram. Let \({\text{LCP}}\left( {x,y} \right)\) be the local curve pattern of size \(M \times N\), the histogram is calculated as given below

$${\text{LHist}}^{i} \left( k \right) = \mathop \sum \limits_{m = 0}^{M - 1} \mathop \sum \limits_{n = 0}^{N - 1} \delta \left( {{\text{LCP}}^{i} \left( {m,n} \right) - k} \right)$$
(12)

where \(k = 1\;{\text{to}}\; 59\) and \(\delta \left( x \right)\) is Dirac delta function and is given by

$$\delta \left( k \right) = \left\{ {\begin{array}{*{20}l} {1;} \hfill & \quad {x = 0} \hfill \\ {0;} \hfill &\quad {x \ne 0} \hfill \\ \end{array} } \right.$$
(13)

3.3 Advantages of LCP over other local pattern techniques

The advantages of LCP over the local tetra pattern can be justified by the following points.

  1. 1.

    Conventional local pattern techniques use predefined pixels which capture the local discontinuities of an image whereas it fails to capture the smoothness over the edges/lines. In contrast, LCP use pixels which are selected based on the local image characteristics in runtime that will have more accurate and effective information than the conventional techniques. Also the curves and lines which are the dominant features of the image are effectively captured by the LCP. In addition, LCP captures the smoothness over the curves and lines which make LCP an effective feature for image retrieval applications.

  2. 2.

    Existing local pattern techniques capture the pattern over its neighboring pixels, whereas the LCP captures the distinctive spatial relationships in a local region beyond the neighbors by traversing over the direction of curves/lines in an image. This results in more accurate representation of the local pattern by LCP especially the lines and curves of an image.

3.4 Proposed image retrieval system

The two phases of image retrieval system are training and classification. In training phase, the database is created using the LCP features. In the classification phase, the LCP features are derived from the input image and are compared with the features in the database to retrieve the closest top M images. Figure 4 shows the image retrieval system using local curve pattern during the retrieval phase. As illustrated in Fig. 4, following steps are used by the LCP-based image retrieval system.

Fig. 4
figure 4

Block diagram for LCP-based image retrieval

  1. 1.

    Load the query image and convert it to gray scale if required.

  2. 2.

    Find the direction numbers for the LCP as in Eq. (8) for two four directional patterns

  3. 3.

    Form the tetra patterns using the curve directions as in (9).

  4. 4.

    Form the binary patterns from the curve tetra patterns using Eqs. (10, 11)

  5. 5.

    Calculate the image histogram of the local curve pattern

  6. 6.

    Derive the uniform histogram from the image histogram

  7. 7.

    Form the features by concatenating the uniform histograms of two different directions.

  8. 8.

    Match the features of the query image with the features in the database.

  9. 9.

    Retrieve top N the best-matched images from the database.

Let \(I\left( {x,y} \right)\) be an input query image of size \(M \times N\) to the image retrieval system and let \(R\) be the feature of the input query image of size \(B\). Let \(T_{i}\) be the feature vectors in the database, where \(i\) is the feature vector of the ith image. To find the closest image to the query image from the database, we use χ2 distance [56] and is given in Eq. (14). The top best images are retrieved from the database which has minimum χ2 distance.

$$\chi_{i}^{2} \left( {R,T} \right) = \mathop \sum \limits_{b = 1}^{B} \frac{{\left( {R_{b} - T_{ib} } \right)^{2} }}{{R_{b} + T_{ib} }}$$
(14)

4 Experimental results and discussion

To evaluate the performance of proposed method, experiments were done using three different databases, viz Corel 1K [57], Corel 10K [58] and Brodatz [59]. Images from these databases are categorized into different classes based on their contents and are used for the different trails. Database for trail-I is constructed using the images from Corel-1K database which contain 1000 images of size 384 × 256 and are divided into 10 groups based on their characteristics. Images in this database were named sequentially such that every 100 images belong to the same group, i.e., images 0 to 99 belongs to group-1, 100-199 belongs to group-2 etc. Database for trail-II is constructed using Corel-10K dataset which contain 10,000 images of size 120 × 280 and are already divided into 80 different groups. (ex. art_1, art_antiques, art_cybr etc). Database for trail-III is constructed using Brodatz database which contain 112 images of size 515 × 512. Image groups are formed from each texture images by extracting the sub-images. The sub-images of size 128 × 128 and 256 × 256 are extracted with 50% overlapping regions so that there will be forty nine 128 × 128 and nine 256 × 256 sub-images. By adding the original image of size 515 × 512 into the group, there are 59 images in a group and total of 6608 (59 × 112) images in the database. Table 1 lists the details of the databases used in this work.

Table 1 Image databases used

Three different trails were done by giving each image from the database as a query to the retrieval system and retrieving top N images from the database, where N = 10, 20, 30, 40 and 50. For each query image, the local curve pattern features are extracted and are compared with the features in the database using (14) to retrieve the top best match images. The performance of the proposed retrieval system is measured in terms of average retrieval precision (ARP) and average retrieval rate (ARR). ARP is computed by finding the mean of precisions and ARR is computed by finding the mean of Recalls. The precision and recall are calculated as follows:

$${\text{Precision}} = \frac{{\# {\text{RIQ}}}}{{\# {\text{IR}}}}\quad \& \quad {\text{Recall}} = \frac{{\# {\text{RIQ}}}}{{\# {\text{IQD}}}}$$
(15)

where \(\# {\text{RIQ}}\) is the number of retrieved images belong to the query image class, \(\# {\text{IR}}\) is the total number of images retrieved and \(\# {\text{IQD}}\) is the total number of images in the database that belongs to query image class. To compare the performance of the proposed technique, three different techniques are used: viz LTrP (local tetra pattern) [54], BOF-LBP (local binary pattern with bag-of-filters) [60] and DBWP (directional binary wavelet pattern) [49]. The following section discusses the details of the experimental results for different trails.

In trail-I, Corel 1K database is used which covers wide range of images that exist in the real world. Table 2 lists the ARP (%) that is measured for LCP and other techniques using Corel 1K database. As per the table, performance of LCP outperforms other conventional techniques. The average improvement in ARP for LCP is 11.2% with LTrP, 8.3% with DBWP and 5.4% with BOF-LBP. Figure 5a, b depicts the performance comparison with Corel 1K database using ARP (%) and ARR (%), respectively, for different number of top images retrieved. Here, LCP curve is right on the top on both the figures which proves the effectiveness of LCP for image retrieval applications.

Table 2 ARP for Corel 1K database
Fig. 5
figure 5

Performance comparison with Corel 1K database using a ARP (%) and b ARR (%)

In trail-II, Corel 10K database is used which contains 10,000 images from 80 different image groups. The performance of the proposed technique is measured and listed in Table 3 for Corel 10K database. As per the table average improvement of ARP for LCP is 5.6% with LTrP, 4.9% with DBWP and 3.2% with BOF-LBP. Also from Fig. 6a, b, it is evident that the proposed LCP technique outperforms the conventional techniques.

Table 3 ARP of Corel 10K database
Fig. 6
figure 6

Performance comparison with Corel 10K database using a ARP (%) and b ARR (%)

In trial III, Brodatz database is used which consists of 112 texture groups. Table 4 shows the improvement in ARP with LCP technique and the average improvement is 4.7% with LTrP, 3.3% with DBWP and 2% with BOF-LBP. The effectiveness of LCP technique is clearly visible in Fig. 7a, b, which shows the pictorial representation of the ARP(%) and ARR(%), respectively, for the Brodatz database. Figure 8 shows the improvement in retrieval performance for Corel 1K, Corel 10K and Brodatz databases. The chart clearly shows the performance of local curve pattern is much better than other techniques (local tetra pattern, bag-of-words LBP and directional binary wavelet pattern). The improvement in the performance of LCP is due the following factors

Table 4 ARP of Brodatz database
Fig. 7
figure 7

Performance comparison with Brodatz database using a ARP (%) and b ARR (%)

Fig. 8
figure 8

Retrieval performance chart for different databases

  1. 1.

    Conventional local pattern techniques relay on the information present in the neighboring pixels, whereas local curve pattern captures information beyond the neighbors which allows LCP to gather more detailed local information in its pattern.

  2. 2.

    Since LCP selects the pixel to derive the local pattern in run time, it can represent the local lines and curves of an image more efficiently than other conventional local patterns

5 Conclusion

In this work, a new approach for local pattern called local curve pattern (LCP) is presented. In order to improve the performance, LCP derives the local pattern from the pixels selected in run time which allows it to represent the lines and curves more effectively. The performance of the proposed technique is validated using three different databases, and the experimental results show good improvement in the retrieval performance of LCP when compared with the other conventional techniques. This work can be extended for improving the performance of LCP by combining with suitable color feature and also by increasing the pattern length. Also analysis can be done for combining LCP with deep learning to improve the overall performance of the system.