1 Introduction

Traditional authentication and identification methods are based on features such as ID card, tokens, passwords and PIN codes. These features can be easily duplicated, cracked or stolen. An alternative to such systems is a biometric system that facilitates the process to recognize an individual by his/her unique physiological and behavioral characteristics. The advantages of a biometric system are: no prerequisite to remember whatever, not easy to manipulate or steal, difficult to forge or share, and testify the occurrence of genuine user at the time of enrollment. However, not every physiological or behavioral characteristic meet the requirements of being a biometric trait. A characteristic has to satisfy certain properties such as uniqueness, universality, collectivity, circumvention and permanence [14, 15]. Some features such as color of eye/skin/hair, age, scars, gender and height are known as soft biometrics. It has been observed that individuals cannot be distinguished on the basis of soft biometrics because these features may not be unique and stable. However, they improve the precision of the system when used in conjunction with physiological or behavioral biometrics traits. Illustrations of a few biometric traits such as face, signature, fingerprint, iris, speech, and palmprint are shown in Fig. 1. They have been presented in numerous civilian or forensic applications such as border crossing, e-banking, health care, and law enforcement [15]. The choice of a biometric characteristic depends on the exact requirements of the anticipated application [14]. It has been claimed that no biometric feature is superior to others or can replace any other but each one has its own pros and cons [4, 14] as mentioned in Table 1.

Fig. 1
figure 1

Biometric Traits

Full size image
Table 1 Trait-wise Challenges and Issues
Full size table

A biometric system can constitute of the following subsystems:

  • Enrollment: Extracted features are saved in a database to register an individual.

  • Verification: One-to-One matching that depends on a threshold (authentication).

  • Identification: One-to-Many matching that reports top matching scores and corresponding top probable subjects (recognition).

Hand based biometric traits have been intensively studied to develop a consistent authentication system with higher precision, usability, and acceptance [4, 17]. Several hand biometric traits such as palm print [9], Inner Knuckle Print (IKP)[18], and Finger Knuckle Print (FKP) [5] have unique anatomical structures that can be captured with low cost and small size imaging devices (such as a webcam) without mounting extra hardware [5, 17]. These traits are present in various access control applications and captures around 60 % of market share of biometric systems [17].

1.1 Motivation

In several Asian countries, like India, more than 70 % of the population resides in rural areas. The laborers, and cultivators do substantial work and use their hands very roughly. This causes plenty of damage to their fingerprints permanently [32]. It has been observed that their quality of fingerprint is not very good. In such a scenario, the quality of FKP is unaffected because they are not used for any other purpose. Hence, it is less prone to injuries [6].

1.1.1 Knuckle print

The convex shape skin patterns which form around finger joints namely Meta Carpo Phalangeal (MCP), Proximal Inter Phalangeal (PIP), and Distal Inter Phalangeal (DIP), consist of highly rich lines and creases called as finger dorsal knuckle print (FKP) [18, 19]. The skin patterns are believed to be very unique, universal, and permanent [6]. They lie on the outer side of the finger and survive longer. Minutia and singular points of the fingerprint are known to fade over time in the hands of cultivators and laborers as shown in Fig. 2a. In such scenarios, FKP is a viable trait to be used for human identity recognition.

Fig. 2
figure 2

FKP motivation and challenges: (a) Quality based comparison of fingerprint (top) and knuckleprint (bottom) images of a person (b) Knuckleprint images of the same finger with different nonlinear distortions due to knuckle bending

Full size image

Quality comparison between FKP and fingerprint

The prime goal of the proposed system is to come up with a solution applicable to rural areas. In this context, fingerprint and FKP based biometric systems are more suitable. A quality analysis of both of them reveals the superiority of FKP images in rural scenarios. In this work, efforts are being made to assess the image quality of FKP and fingerprint over a small in-house dataset of 10 individuals. Among them, 4 subjects were 55-64 years old and others were 40-45 years old. All of them belonged to rural Indian villages and were involved in very hard labor. To estimate the FKP quality, our previous work [32] has been used, while for fingerprint quality estimation publicly available NIST based nfiq tool [2] has been used objectively. Our observation over such images clearly suggests that the fingerprint quality in this scenario lags behind FKP.

Likewise, palm provides a bigger ROI region, with more detailed features. However, there is a risk of size complexity [9]. Also, the geometrical features of hand/finger may not be unique for identification because they can change over time due to illness and other environmental factors [11]. On the other hand, dorsal hand veins have strong anti-forgery characteristics as the skin patterns underneath are actually unique and remain relatively stable through the adult age [22]. The FKP can be collected easily using ordinary, contact-less cameras with lesser user co-operation [39]. The user failure to enroll (FTE) rate is lower as it is not associated with any fraud and criminal studies [32, 33]. Moreover, the dorsal knuckle patterns cannot be easily duplicated, and the possibility of information loss from this region is also less. But, there exist a big challenge to track the middle knuckle line for FKP registration because there might be deviations in spatial location of fingers during acquisition. Hence, it is concluded that a specific pre-processing, feature extraction and matching schemes are required to fully trace the curved shape features for higher FKP performance.

1.1.2 Deep-matching: Multi-scale image matching technique

The multi-scale architecture of Deep-matching technique is inspired by functioning of Convolution Neural Network (CNN). The CNN models are made up of neurons with learn-able weights/biases and have shown tremendous improvements in image classification [10]. However, the Deep-matching technique does not create NN models to learn but this is a beautiful hierarchically inspired image matching paradigm. Basically, a local descriptor focuses only at the salient feature locations but match the deformable objects poorly. It has been observed that the Scale Invariant Feature Transform (SIFT) like features are unable to compute their pointwise dense correspondence. On the contrary, Deep-matching technique utilizes a quad tree based multilayer network that filters an image cell at different scales and improves the feature matching of non-rigid deformable as well as large displacement surfaces very accurately [41].

1.1.3 Contribution

The non-rigid and weakly textured regions of a FKI provides intricacies to FKP recognition. They are the major contributor towards false rejection in FKP based matches. Moreover, we argue that the central line tracking for FKP registration may not have uniformity and leads to the requirement of flexible ROI extraction. The deviations in spatial location of the same finger at the time of acquisition can be clearly seen as depicted in Fig. 2b. This type of intuition degrades the pointwise correspondence of deformable FKP patches. To handle such non-rigid deformation and large displacement in FKP image, we need to estimate correlation at several patch scales. The quad tree patch subdivision based Deep-matching algorithm can be seen as an ultimate solution. The contribution of this work can be summarized as follows: The ROI of raw FKP image is cropped by considering central knuckle point as reference and further its contrast is enhanced using Contrast Limited Adaptive Histogram Equalization (CLAHE). In the image enhancement, the additive noise is now assumed as multiplicative and has shown significant improvement. The novel image transformation schemes based on the ordinal relationship of two adjacent pixel gradient values, named as Bubble Ordinal Pattern (BOP) and Star Ordinal Pattern (SOP) are proposed to obtain the robust edge information. The resultant image is further processed using ray optics based Image Ray Transform (used first time to best of our knowledge) to strongly highlight the curved shape tubular features present in FKP image. Eventually, a high performance Deep-matching algorithm (which is a Graphics Processing Unit (GPU) based parallelized algorithm) has been used to address dense and non-rigid deformations in FKP images. The proposed system has been tested over a publicly available benchmark – The Hong Kong Polytechnic University (PolyU) FKP database [1] that consists of 7920 FKP images. Its performance has been measured in terms of computation time, discriminating index, Equal Error Rate (EER) and Correct Recognition Rate (CRR). The block diagram representation of the proposed FKP biometric system is shown in Fig. 3. To the best of our knowledge, there are very few related works existing in the literature which address the problem of large displacement and non linear distortion in FKP images as shown in Fig. 2b.

Fig. 3
figure 3

The proposed framework for FKP based biometric system

Full size image

The rest of the article is organized into the following six main sections. Section 2 describes the FKP state-of-art work. In Section 3, ROI extraction, image enhancement, and image transformation schemes have been discussed. Section 4 highlights the role of feature extraction. The proposed matching strategy is discussed in Section 5. Then, experiment results are described in Section 6. Finally, the conclusions are summarized in the last section of this paper.

2 Related work

The skin patterns such as creases, lines, and texture over finger knuckle surface have a lot of discrimination ability to be broadly recognized as a biometric identifier. Initially, due to lack of benchmark datasets and poor recognition rates, this area did not receive much attention and thus a limited number of works were reported till 2007. After that, immense efforts have been made in the area of FKP based personal authentication system. Majority-wise, region of interest (ROI) detection, feature extraction, and classification play an important role in FKP based personal recognition system. There are some additional requirements too, such as the size of the template, power intake, memory storage etc., but these cannot be achieved together. In this section, a state of art related to ROI extraction as well as feature extraction/classification techniques are presented.

2.1 ROI extraction

To segment and extract fixed size ROI from an original FKP image is a crucial stage and it really affects the final recognition accuracy. The existing methods extract ROI based upon local convexity characteristics of the skin patterns near major finger joint (PIP). Most of the existing techniques are tested against two publicly available datasets i.e., PolyU FKP dataset and IIT Delhi Finger Knuckle Image dataset.

In [29], a local coordinate system is established to align the images (PolyU FKP) and ROI is cropped for feature extraction. In this method, the curved lines on the two sides of the PIP joint are observed and then pixels over lines are encoded as a two tuple (1,-1) as per their convex directions. The convexity magnitude is measured to find the strength of principal curve. It is minimum around the center of PIP joint while maximum on left and right sides lines because FKP image is positioned horizontally. On that basis, the center position is considered to fix the Y- axis of the coordinate system and ROI of size (160×80) is cropped empirically. In [20], authors designed a novel ROI extraction algorithm by using contrast enhanced and corrected skewed images. Subsequently, a center point based ROI detection and localization method for FKP images is given in [38]. This method is robust against finger displacement and rotation in the horizontal direction. In [32], the middle knuckle line and middle knuckle point are determined to extract the ROI of FKP image. The authors proposed a modified Gabor filter by adding curvature parameter to check the response of curved convex line features present around the major finger joint. It has been observed that the pixels over middle knuckle line have a maximum filter response because FKP structure is vertically symmetric. Finally, ROI is extracted on the basis of central knuckle point and found 95 % of accuracy over PolyU FKP dataset.

2.2 Feature extraction and classification

The conventional knuckle print recognition approaches can be categorized into the following groups: subspace methods, coding methods, texture analysis methods, and other image processing methods. In [7] an apparatus was developed to identify or verify individuals using their knuckle contours. In [18], the authors considered creases of finger inner surface for biometric identification. The matching was carried out using the normalized correlation function. In [36] the folds and creases on the finger outer surface were examined. The authors had employed Minolta 900/910 sensor to collect 3D finger samples, but it was quite an expensive, bulky, and slow processing device which limited its use in commercial or real time systems. In the following year, knuckle code based coding scheme for texture feature extraction was conceptualized [29]. During the same time, the authors [33] clubbed both the knuckle texture and geometrical features of hand, which were acquired by a peg-free and non-contact imaging setup. In [40], 2-D Gabor filter was employed to extract local orientation information, and saved it in a feature vector named competitive code. Likewise, the authors [24], modified the knuckle codes approach by applying radon transform on enhanced knuckle images and resulted in 1.14 % EER and 98.6 % rank one recognition rate. Additionally, [23] suggested a multi algorithmic approach based on matching scores of principal component analysis (PCA), linear discriminant analysis (LDA) and independent component analysis (ICA) with EER of 1.39 %. In [41], another significant method called as monogenic code (3 bit vector) which reflected the phase and orientation information in knuckle images was framed. Likewise, [42] improved their previous work by combining the magnitude (magnitude code) and the orientation information (improved competitive code). In another work [43], the authors have used a weighted sum rule to fuse local and global information to achieve optimum results. In [30], SIFT key points from Gabor filter based enhanced FKP images were extracted. In [6], knuckle texture had been used as an identifier in smartphone applications. In [31], efforts were made to compute FKP image quality attributes. In [11], score level fusion was adopted to integrate the obtained matching scores caused by fragility masks. In [37], the authors implemented a well-known local feature descriptor called as LBP over ROIs based FKP images. In [26], the authors performed matching over recovered minutiae samples by using minutiae cylindrical code, minutiae triangulation, and spectral minutiae based approaches. This section presented the various author’s research work and highlights the following issues: The FKP is still a less examined biometric trait for a wide range of applications. There are few related works that exist in literature which tackle the problem of large displacement and non-rigid deformation in FKP images, as this leads to false rejections in the matching process. There is a lack of FKP databases in which images incorporate the real world situations such as variation in bending of fingers.

3 Pre-processing approach

In this section, a sub region of finger knuckle image near Proximal Inter Phalangeal joint (PIP) is extracted using a ROI extraction algorithm [32]. The raw ROI images are then enhanced using the CLAHE scheme to stabilize the effects of poor contrast and non-uniform illumination. Although, finger dorsal surface is rich with texture patterns but the curve shape lines are not very clear. So, ROI samples are transformed into a better visual representation by gradient ordinal relationship based encoding schemes. Subsequently, its ray tracing features are extracted with the help of IRT which gives more discriminative information.

3.1 FKP ROI extraction

The basic idea behind ROI extraction is to apply a curvature Gabor filter with modification in X coordinate to estimate the middle knuckle line M kl , and middle knuckle point M kp as described in (1). The raw FKP image is thresholded to get binarized image from which knuckle boundary is measured as shown in Fig. 4b. The knuckle region is taken out using Sobel edge detector and a strongly attached component assuming the solid knuckle boundary (horizontal axis) as shown in Fig. 4c. The knuckle area is retrieved back from the pixels within the defined knuckle boundary as given in Fig. 4d. To estimate the Middle Knuckle Line M kl , the knuckle region can be considered symmetric around the middle knuckle line of phalangeal joint. Since, the convex shape lines are observed to be protuberant on left and right side of knuckle area, while curves near middle knuckle line do not possess the exact convex shape. This makes the efforts easy to spot middle knuckle line and midpoint. To visualize such a convex shaped texture, a knuckle filter using additive curvature parameter is designed, as shown in Fig 4e. The filter response can be seen as a modified version of classic Gabor filter, with X and Y modulated as:

$$ X = x \times cos\theta+y \times sin \theta+c (-x \times sin\theta+y \times cos \theta)^{2}; Y = -x \times sin\theta+y \times cos \theta $$
(1)
Fig. 4
figure 4

FKP ROI extraction (a) Original, (b) Binary Sample, (c) Boundary, (d) FKP area, (e) Knuckle filter, (f) Filter response, (g) Localized ROI (h) Final ROI

Full size image

Several curved Gabor filters can be obtained by changing the value of curvature parameter (c) and distance parameter (d) between two flipped curved filters. Now, the knuckle area image is convolved with the specific curvature Gabor filter (F 0.01,30) which provided a better idea to measure the strength of convex shaped lines over the knuckle region. The column wise filter response is found maximum at the middle column and that position is considered as a middle knuckle line (vertical axis) as shown in Fig. 4f. To estimate the Middle Knuckle Point M kp , the point at the center of the middle knuckle line with respect to the knuckle area is specified as the middle knuckle point. For locating a midpoint, top and bottom points of the middle knuckle line are assumed over knuckle area. Then, a ROI of fixed size is cropped from knuckle area image with consideration of M kl and M kp as shown in Fig. 4g. Finally, the knuckle ROI is normalized as given in Fig. 4h. Algorithm 1 summarizes the ROI extraction method.

figure d

3.2 FKP ROI enhancement

The textured region of the knuckle print can be used for identification as it is unique for every individual. The finger dorsal signifies a fairly deformable surface, which results in uncertain reflections and illuminations. A ROI sample is partitioned into non overlapping fixed size cells (10×10). The cell size is selected empirically which ensures that mean of every cell almost indicates the coarse illumination of it. This mean of coarse illumination is expanded to the original ROI sample. The estimated illumination of every cell is divided from the corresponding cell of the original ROI and get a uniformly brightened ROI sample, as shown in Fig. 5b. Instead of using the full illumination value for this purpose, we have scaled down the value using f scale (between 0 and 1). Now, the resulting ROI image is enriched using CLAHE, which improves the contrast in the texture without adding noise and increases the discriminative strength as shown in Fig. 5d. Further, the blocking effect is reduced using bi-linear interpolation. Finally, Weiner filtering is applied to smooth the boundaries between the blocks, and to minimize the additive noise as depicted in Fig. 5e.

Fig. 5
figure 5

Proposed Image Enhancement for FKP images (Difference between Additive Vs Multiplicative noise can be observed from Fig. 5c and d)

Full size image

Justification

Here, our assumption is that noise is multiplicative in nature that increases brightness. The increase in overall performance adds merit to our assumption.

3.3 FKP ROI transformation

A key issue to efficiently measure the texture properties is to get the best texture representation. Recently, many robust methods such as LBP [3] have been developed for texture representation which threshold the neighborhood pixel with respect to a central pixel. In this work, the enhanced ROI sample is transformed using novel image transformation schemes namely Bubble Ordinal Pattern (BOP) and Star Ordinal Pattern (SOP). We propose BOP/SOP over first order image gradient which shows high level discriminative features. They provide robust transverse and longitudinal representations against variable illumination. This is due to the fact that gradient values are strong indication of the local edge information. Further, a ray tracing based mechanism has been used to enhance the detection of knuckle structural features relying strongly on their intensity. Image Ray Transform (IRT) basically highlights the knuckle convex hull strongly due to presence of high intensity epidermal texture around major finger joint. Algorithm 2 concludes the image transformation steps.

figure e

Comparison of proposed transformation with various existing transformation

LBP [3] and CS-LBP [13] are originally defined over gray scale images (as shown in Fig. 6c, d), but we have observed that gray values as well as their locally computed ordinal relationship are very fragile, especially in the region where the texture is very elusive. On the other hand, they are not robust to illumination variation. Hence, in order to cater such issues, we have used the first order derivatives computed by difference operator. We propose the Bubble Ordinal Pattern (BOP) over first order gradient which can extract the high level discriminative features. To best of our knowledge, such Bubble pattern code (BOP) has never been considered previously. As compared to LBP, the BOP does not threshold with respect to center pixel. While, SOP is related to CS-LBP, but instead of 4-bit codes we have redundant them into 8 bit as they were not properly scaled into image representation. Thus, the key point to note here is that SOP/BOP computations are over the image gradients.

Fig. 6
figure 6

Visual features based appearance analysis. Images are persented before and after applying various existing transformations. First row: Original and IRT applied images, Second row: Transformations over gray images, Third and Fourth row : Transformations over gradient images. Last row shows rich and enhanced FKP tabular structure extracted using IRT

Full size image

Our motivation to work over gradient was that gradients are robust to variable illumination. Any gradient based ordinal relationship can be considered more robust as compared to any equivalent gray scale version. Since gradient values are involved, we have observed that automatically edges and their relationship gets most of the emphasis and finally got encoded into the proposed pattern (as shown in Fig. 6g-p). This encoding scheme not only becomes robust for variable illumination, but also very discriminative. Moreover in our current experimentation, we have monitored the behavior as well as visual image appearance as shown in Fig. 6, along with the performance of the proposed BOP/SOP codes presented in Section 6.6.

3.3.1 Bubble ordinal pattern (BOP)

The enhanced ROI samples are down-sampled to a fixed size in order to reduce computation time. The ROI samples are further processed by the proposed encoding schemes – Bubble Ordinal Pattern (BOP) and Star Ordinal Pattern (SOP). The computational steps related to BOP are mentioned in Algorithm 3 and transformed images are shown in Fig. 6h, i, m and n.

figure f

Transformation basis

The opted Sobel kernel (9×9) can estimate thick and discriminative edges. It is a discrete differentiation operator which is a result of quantification of the longitudinal and the transverse derivative approximations of the gradient of the image intensity function. The BOP computes the ordinal relationship of two adjacent pixel gradient values, specifically when neighbors have similar values. The gradient of any pixel will be + ve or −ve that is encoded into a bit either 1 or 0 coding value. Hence, such ordinal relationship encodes the type of edge that is passing trough the corresponding pixels. The proposed encoding mechanism utilizes Sobel longitudinal kernel to assign an 8-bit code (l BOP ) to every pixel based on the derivatives of eight neighbors. It has been observed that the obtained transverse derivatives are not very robust primarily due to the prominent longitudinal features present in the FKP images. Hence, after rigorous experiments, we have dropped transverse gradient (t BOP ) and other derived features from it.

Transformation

The bubble ordinal pattern based encoding evaluates BOP Code for every pixel based on transverse and longitudinal derivatives of its eight neighbors as shown in Fig. 7a. Suppose that E u,v defines spatial location of (u,v)th pixel in an enhanced FKP ROI sample. The obtained BOP Code response for each pixel is an 8-bit binary number whose k th bit is stipulated in (2):

$$ BOP_{Code}(u,v)[k] = \left\{\begin{array}{ll} 1, \text{ if } G_{k} > G_{k+1} \\ 0, \text{ otherwise} \end{array}\right. $$
(2)

where, G K , k=1,2,3....8 represent the gradients of eight adjacent pixels positioned around E u,v with the use of transverse or longitudinal Sobel kernels. Therefore, l BOP or t BOP are basically BOP Code based representation of entire pixels in a ROI sample as computed in step 2 of Algorithm 2.

Fig. 7
figure 7

Neighborhood pattern considered for image transformation

Full size image

Justification

The basic assumption is that the pattern of edges within eight neighborhood of any pixel does not change abruptly because gradient values are robust in nature. Our motivation was not only to achieve illumination invariance (so using gradient) but also to get discrimination that has been achieved because the edges get more and more emphasis. The related BOP codes are shown in Fig. 8b, c.

Fig. 8
figure 8

BOP and SOP based Image Transformation and Encoding Schemes (Transversed components are discarded)

Full size image

3.3.2 STAR ordinal pattern (SOP)

Another encoding scheme, known as star ordinal pattern (SGORP), has been used [32]. In this scheme, gradient of a pixel is given either positive, negative or zero code based on its edge position. An 8 bit encoding scheme for every pixel is performed which considers diagonally opposite neighbors (as shown in Fig. 7b) together with upper and lower ones. The SOP Code can be computed using the Algorithm 4. The related SOP codes are shown in Fig. 8d, e.

figure g

3.3.3 Image ray transform (IRT)

The transformed ROI knuckle image (I T ) consists of prominent curved lines, and stable structures than original ROI sample. The detection of specific structural features within an image is a challenging task in biometrics. The image ray transform (IRT) has been received significant consideration for structural feature detection empirically in medical imaging, and retinal vessel extraction. Based on the principal of ray optics, IRT technique uses the concept of tracing light rays through an image as depicted in Fig. 9. Hence, it mainly highlights the curvilinear structures such as tubes or curves as clearly evident from Figs. 6l-p. In this work, IRT is introduced for the first time, to best of our knowledge, to emphasize the curved (lines and creases) features from transformed FKP images as explicitly shown in Figs. 6 and 10. The transform treats an image pixel as a set of 2-D glass blocks with refractive indexes linked to intensity of the pixel and then casts a large number of rays through the image. The sequence of these rays is accumulated into an output image which emphasizes certain structural features. The route that a ray follows is often changed when propagating from one to another medium with different refractive indexes. Refraction of light at the surface of separation of two media can be better understood by the Snell’s Law.

Fig. 9
figure 9

A patch in the transformed image has been enlarged to explain Image Ray Tracing (IRT). Reflection, refraction and TIR happens in accordance to Snell’s law

Full size image
Fig. 10
figure 10

Image Ray Transformation (Transverse components are discarded)

Full size image

Justification

FKP images have a significant amount of longitudinal textured patterns. These are thick curved features obtained after BOP or SOP transformations. One can observe from Fig. 10, that such features can be easily highlighted using proposed IRT transformation.

IRT implementation

The IRT creates a magnified image as the way light reflects off curved features [8]. Take a transformed knuckle image (I T ) of size (m×n). Then, a refractive index (n i ) value for the i th pixel is computed as a function of its intensity (g).

$$ n_{i}= 1+\left( \frac{g}{255}\right)(n_{max}-1) $$
(3)

whereas, n max represents the maximum refractive index for a linear model. Next, the computation of the refractive index for randomly selected pixels is described. The algorithm traces a path through the pixels by emulating refraction and total internal reflection. Figure 9 visually explains the working of IRT. At each pixel the direction of normal is vertical. The blocks of distinct shades represent media with different refractive index.

  • At point M, ray is initialized and travels towards B.

  • At point N (n 1 = n 2), the ray continues to travel along the same direction.

  • At point O (n 1<n 2) or at point P (n 1>n 2), refraction occurs and the light ray bends towards or away from the normal accordingly.

  • At point Q (𝜃 i >𝜃 c ), hence Total Internal Reflection (TIR) takes place and the ray reflects back into the same medium.

  • At point R and S, the ray is an inverse replica of trace occurred at points O and P.

  • At Q, eventually the ray exits. The tracing is allowed to continue until d reflections or refractions have undergone for each ray.

For each pixel, during the path tracing, if the refractive index of the next pixel is different, the 𝜃 i (angle of incidence) and 𝜃 r (angle of reflection) are calculated as:

$$ \cos\theta_{i} = N.V $$
(4)
$$ n=\frac{n_{1}}{n_{2}} $$
(5)
$$ \cos\theta_{r}= \sqrt{1-n^{2}(1-N.V)} $$
(6)

where, N is the normal direction at each pixel, and V is a vector function of the initial direction vector at each pixel. The refraction vector, for the path to pass across boundary of pixel with different refractive index, is given as:

$$ R_{r}=nV+(n(N.V)-\cos\theta_{r})N $$
(7)

Likewise, if 𝜃 i >𝜃 c and n 1>n 2, internal reflection will take place and hence the reflection vector is given by:

$$ R_{l}=V-2(N.V)N $$
(8)

The initial direction ϕ, to start the path tracing from each pixel, is randomly drawn from a uniform distribution. The overall methodology is outlined in Algorithm 5. Once the random rays from all the different points are cast, they start to trace in the whole image. The algorithm works in such a way that regions consisting of pixels with similar intensity values are emphasized. Thus, the tracing path of all those rays will finally converge and highlights the curved lines and creases from the rest of the image. Finally, when each ray has undergone a fixed number of refractions or reflections, the tracing of rays is ceased. The IRT implementation is mainly affected by maximum refractive index n max (affects the number of segmented pixels), number of rays traced N (impacts the smoothness), and depth of ray tracing d (responsible for sharpness of the extracted features). Here, we set n max value to 100, increased N value to 10000, and assume d value to 256. All parameters used in this work, have been listed and their default values with descriptions, as well as operating ranges are presented in Table 2.

figure h
Table 2 Description and nature of parameters
Full size table

4 Feature extraction

To perform image matching by using the set of local interest points is an important aspect of local texture descriptor based approaches. They perform compact vector representations of a local neighborhood, and are building blocks of many computer vision algorithms. Local features such as points, edges, blobs, and small image patches are distinctive and enable these methods to better handle scale change, rotation, and variable illumination [16]. The popular scale and rotation invariant local texture detectors are: Scale invariant feature transform (SIFT), Speeded up robust transform (SURF), Hessian Laplace, Harris detector, Local Energy based shape histogram, Local binary pattern (LBP), Local direction pattern (LDP), HOG, etc. The SIFT operator provides a robust solution to compare image patches with rich textures [27]. It performs a cascaded operation to match the robust key-points of the corresponding regions. But SIFT fails to match non-rigid deformable regions and image patches with weak or repetitive textures [34]. This is because the 4 cell grid in SIFT is completely rigid and it is not able to detect the non-linear deformations present in FKP images. In this section, the problem to match FKP images with non-rigid deformation and large displacement is highlighted.

5 DeepKnuckle : a novel knuckle identification framework

In contrast to descriptor matching approaches a dense, hierarchical algorithm to compute dense correspondence between images is introduced which is known as Deep Matching [34]. Deep Matching algorithm is based on a multi-stage architecture with 6 layers (depending on the image size), interleaving convolutions and max-pooling, a construction similar to deep convolution nets. Here, convolution is done at patch-level, which proceeds in a multi-layer fashion. In this multi-layered architecture, the images are subdivided into patches like a quad-tree pattern, but the quadrants in the target image are not in fixed positions. Hence, these sub-blocks are allowed to move locally to re-optimize their position, irrespective of the position of other sub-patches. In this way, each of the 16 sub-patches in a 4*4 grid can move and match to the position from which it finds a closest match.

5.1 Building of the approach

This algorithm starts to generate the correlation maps in a bottom up fashion and later computes the correspondence in a top down fashion. The SIFT/HOG descriptor which is a popular approach to match regions between images, with 4×4 spatial cells generates a real vector in 128 dimensional space V∈R 4×4×8. Now the SIFT patch can be split into 4 quadrants represented as I=[I 0,I 1,I 2,I 3] with I n∈R 2×2×8. Suppose, there are two SIFT descriptors – one reference and another target. In the target descriptor, the quadrants of 4×4 sized grids have not been kept fixed. Their positions can be optimized by maximizing \(Sim(I,I^{\prime }(p))= \frac {1}{4}{\sum }_{i=0}^{3} \max _{p_{i}}sim(I_{i},I^{\prime }(p_{i}))\), where I ′(p)∈R 2×2×8 is the descriptor of a single quadrant extracted at position p. The similarity can be estimated efficiently with the assumption that each of these quadrants can move independently (upto some extent), which gives a coarse non-rigid matching. This method can perform reasonable non-rigid matching with explicit pixel-wise correspondences, if it is applied in recursive nature, which motivates to use Deep Matching in FKP matching problem. For the sake of clarity, the matching algorithm is divided into two main steps.

  • Correlation maps are computed 23using a bottom-up algorithm.

  • A top-down method estimates the motion of atomic patches starting from matches of large patches.

5.2 DeepKnuckle : matching algorithm

In this section, we introduce a matching algorithm based on correlations at patch-level. Two given images I R and I T are compared using algorithm 6. Initially, the algorithm works in a bottom-up fashion (fine level to coarse-level) which include convolution, max-pooling and sub-sampling as depicted in Fig. 11. It starts with the computation of correlation maps of small sized patches and proceeds up to larger patches by aggregating the smaller patches. Finally, a top-down method is implemented to estimate the motion of atomic patches starting from top level correlation maps. Each response map has maxima which alone cannot explain the full set of pixel-wise correspondences between the images so that the correspondences extracted from all local maxima are merged in order to better estimate the global flow. In this way, after using deep matching the number of feature points matched correctly can be used as a matching score. In this work, we have used its inverse as a dis-similarity score. Finally, a top-down method is implemented to estimate the motion of atomic patches starting from top level correlation maps as depicted in Fig. 12. All parameters used in DeepMatching, have been listed and their default values with descriptions, as well as operating ranges are presented in Table 2.

figure i
Fig. 11
figure 11

Correlation maps obtained while matching reference image with target image using DeepMatching [34]

Full size image
Fig. 12
figure 12

Top down correspondence pruning over correlation map pyramid

Full size image

6 Experimental results

Any identification or verification algorithm is tested to evaluate its performance over a data set to check its adaptability in a variety of applications. In this work, the largest benchmark publicly available, PolyU FKP dataset [1] has been used for performance evaluation. The rigours tests are performed to determine the practicality of FKP images for a biometric system.

6.1 Database specification

The database was developed by Biometric Research Center (UGC/CRC) at the Hong Kong Polytechnic University and is freely available for academic, non-commercial use. It contains 7920 images in BMP image format with resolution 110×220. The 165 (125:40) individuals participated in the enrollment process, including males and females. For each subject, 6 images per index/middle finger are acquired in two different sessions (time gap between 14 to 96 days).

6.2 Parametric analysis

The proposed system employs multiple techniques that can module wise perform several essential tasks for effective and efficient FKP matching. Proposed techniques like BOP, SOP, IRT, Deep-Matching, ROI extraction, and local enhancement, multiple key parameters to be determined initially so as to let this system work. We have generated a small validation data set consisting of 80 subjects by considering first 20 subjects from each finger LI, LM, RI, RM. The best performing parametric set has been chosen in terms of CRR, and EER as shown in Table 2.

In Table 2, each and every parameter used in this work has been described. Its default value and operating range has been also been specified. Based on the operating range and its default value, a semantic label from the set {High,Moderate,Low}, has also been allotted. These labels can significantly enhance user’s qualitative parametric understanding and reproducibility of results over different databases and scenarios.

6.3 Testing protocol

The performance metrics used for analysis of the proposed system are Equal Error Rate(EER), Correct Recognition Rate (CRR), Error under ROC Curve EUC, and Decidability Index (DI) along with computation time. The strategy is tested over inter-session matching that considers first six images as training and last six images for testing. Based on the matching score, a matching is declared genuine if both the images belong to the same class else considered as an impostor.

6.4 ROC based analysis

For a better justification of the proposed method, experiments related to recognition and verification are thoroughly conducted. The results for ROC based analysis are presented below:

Test 1: :

In this experiment, four categories of PolyU FKP database – Right Index (RI), Right Middle (RM), Left Index (LI), Left Middle (LM) – are considered independently. The corresponding ROC characteristics for each category of database are shown in Fig. 13. Based on the testing protocol of our system, total of 5,940 genuine and 9,74,160 impostor matches are reported. The main conclusions from the first test are as follows. The KCP lBOP based schemes gave significant improvement in results over KCP lSOP transformation because FKP image patterns are better justified in BOP based longitudinal gradient. The multi-feature fusion outperforms individual feature based schemes when tested over individual FKP data sets as mentioned in Table 3. Among all four datasets, one can observe that the optimum EER of 0.96 % and CRR of 100 % are achieved with LM (165 subjects) images.

Test 2: :

In the second test, all subjects (660) and their corresponding poses (660∗12) are included for performance evaluation. A similar methodology is adopted for selection of training and test images per class. Thus, a total number of 15,657,840 impostor and 23,760 genuine matching scores are computed. The results for complete FKP data set are described in Table 3 and ROC plots are shown in Fig. 14a. The proposed multi-feature fusion scheme achieves 0.92 % EER over full FKP data-set which is significantly higher than four individual feature based schemes. Hence, it can be concluded that our experimental results on PolyU FKP database using proposed multi-feature fusion with Deep Matching are highly robust for FKP based biometric systems. The genuine against impostor score distribution has been presented in Fig. 14b for fusion scheme over full FKP database. A well discriminating characteristic can be observed, as also reported in terms of discriminating index (DI=3.304) in Table 3.

Fig. 13
figure 13

Test-1 : ROC Curve based Experimental Analysis

Full size image
Table 3 ROC based Performance Analysis over individual FKP databases
Full size table
Fig. 14
figure 14

Test-2 : ROC curve based experimental analysis (Full FKP Database)

Full size image

6.5 Comparative analysis with other state-of-the-art FKP systems

The performance of our proposed system is compared against some well-known systems and results has been presented in Table 4. Overall, the proposed fusion scheme has performed very well over all type of FKP images. The results presented in Table 4 strengthened our belief of suitability of FKP images for high end security purpose. The results presented for comparison are taken from [32] as public implementations are not available. One can observe that Compcode [40] and fusion of MoriCode and MtexCode [11] have been performing consistently well. But the proposed scheme has achieved superior performance. Fusion of multiple features as reported in [11] and [43] has also been surpassed by our proposed fusion, which highlights the effectiveness of our proposal. The overall performance of left hand images is slightly worse than right handed, as most of the subjects were right handed individuals.

Table 4 Comparative Analysis over FKP Database (Results as reported in [32])
Full size table

6.6 Comparison among prevalent akin transformations

In this work, we have only considered transformations over gradient values as the performance over transformed gray values turned out to be poor as well as unfair comparison. We conclude that the vertical gradient information is more useful than the horizontal gradient for a FKP due to more prominent vertical edge structure. We have empirically observed that a 4-bit CS-LBP [13] does not achieve the high performance. Thus, we generate an 8 bit code by bit duplication. Bit redundancy have scaled the bit-code value of each pixel. It resulted in a better performance because we compute and match features over images obtained after transformation, rather than using hamming distance between codes as dis-similarity measure.

The detailed ROC based comparative analysis among prevalent akin transformations has been presented in Fig. 15. It has been observed that BOP followed by IRT has shown superior performance among all and closely followed by SOP + IRT. Therefore, we have fused their longitudinal components. CS−LBP, which is closely related to SOP + IRT, falls well short in performance but better then several others such as GORP + IRT transformation. LBP transformation’s performance has been observed to be bad as compared with others. Remaining transformations without applying IRT have also shown very similar ROC behaviour as shown in Fig. 15. Hence, such comparison helps us to conclude the superiority of the proposed DeepKnuckle based FKP framework, in which transformation BOP/SOP is followed by IRT and later matched using DeepMatching.

Fig. 15
figure 15

Comparative analysis of various transformations such as LBP, CS-LBP etc. Results are obtained by applying DeepMatching over first 100 LI subjects using default parameters.

Full size image

6.7 Time analysis

The proposed prototype system is implemented in C/C++ language which uses open source computer vision library OpenCV along with Torch,Atlas and several other libraries. The machine is equipped with four Intel(R) Core(TM) 2 Quad CPU Q9550@2.83GHz processors with a RAM of 8GB and open source Ubuntu 10.10 as its operating system. The detailed time analysis of the system is reported in Table 5. The proposed system can perform single verification in less than a second (i.e. 756.82 ms) in our prototype system, which is fast enough for real-time applications.

Table 5 Timing Analysis
Full size table

7 Conclusion

This work has been carried out with the aim to develop a FKP recognition system for a rural community that can be used for applications such as financial inclusion schemes (NREGA and MNREGA). In this work, the proposed algorithm is based on score level fusion of multiple texture features. Initially, the ROI’s of enhanced FKP images are extracted using modified Gabor filter method. The extracted ROI samples are further transformed using two novel BOP and SOP schemes to obtain robust image representations. The SIFT image features are matched using a hierarchical Deep matching algorithm. Deep matching performed in top down and bottom up fashion in order to handle weak texture and non-rigid deformable regions robustly. To put merit to the work, the performace is evaluated over the largest FKP benchmark dataset. The results are then compared with other state of the art methods. Our proposed methods are better in terms of CRR (99.39 %), EER (0.92 %) and computation time (756.82 ms). It has been observed that transverse features are not very useful and robust in FKP based recognition system. Also, instead of additive, a multiplicative noise assumption is better suited for such applications.