Score Level Fusion of Multibiometrics Using Local Phase Array

Abstract

Local phase array for biometric recognition have demonstrated efficient performance in face, palmprint and finger knuckle recognition. If the matching score for each trait is calculated by one matcher using local phase array, the size of the system can be reduced and the simple score level fusion can be used to exhibit good performance for person authentication. In this paper, we consider the score level fusion of face, iris, palmprint, and finger knuckle whose matching scores are calculated using local phase array. Through a set of experiments using public databases, we demonstrate effectiveness of local phase array for multibiometric recognition compared with the combination of the state-of-the-art recognition algorithm for each trait.

1 Introduction

Person authentication systems that use various biometric traits such as fingerprint, iris, and vein, are now becoming extensively used as the applicability of biometric authentication expands [8]. Biometric systems that utilize a single trait do not always exhibit high quality performance because one trait is no longer universal in applications with a large number of users, and the noise levels in sensed data increase due to the imperfect conditions during acquisition. To overcome these limitations within unibiometric systems, person authentication systems that make use of multiple biometric traits have recently attracted considerable interest [10].

Multibiometric systems improve performance by the complementary use of multiple traits, and exploiting distinctive advantages such as the capacity to: (A) address limited population coverage, (B) hinder spoofing by impostors, and (C) assess noise in sensed data, which are previously unmanageable with unibiometric systems. Fusion levels for multbiometric systems can be classified into five categories: (i) sensor level, (ii) feature level, (iii) score level, (iv) rank level, and (v) decision level. In this paper, we focus on score level fusion of multiple biometric traits, since matching scores are accessible and relatively simple to combine regardless of the algorithms or traits used.

In general, the matching score for each trait is calculated by using its dedicated recognition algorithm. Therefore, the increase in the number of traits results in a large-scale system as shown in Fig. 1 (a). In contrast, the unified recognition algorithm for multibiometric recognition is expected to realize a compact system as shown in Fig. 1 (b).

Fig. 1.

Multibiometric systems: (a) conventional system using multiple matcher, and (b) proposed system using local phase array.

In a previous study, we proposed a biometric recognition algorithm using local phase array, and demonstrated its efficiency for face, palmprint and finger knuckle [5]. If the matching score for each trait is calculated by one matcher using local phase array, the size of the system can be reduced and the simple score level fusion can be utilized to exhibit high quality performance for person authentication. In this paper, we consider the score level fusion of face, iris, palmprint and finger knuckle whose matching scores are calculated using local phase array. Through a set of experiments using public databases, we demonstrate the effectiveness of local phase array for multibiometric recognition compared with the combination of the state-of-the-art recognition algorithms for each trait.

2 Biometric Recognition Using Local Phase Array

This section describes the fundamentals of biometric recognition using local phase array [5].

In general, biometric recognition systems normalize the position and illumination of images according to the type of biometric traits. For example, in the case of face recognition, we detect the face region, extract feature points such as eyes, nose, mouth, etc., and then normalize the position of the face according to feature points.

To perform accurate similarity evaluation taking into consideration nonlinear deformation of normalized images, we employ local phase array extracted from multi-scale image pyramids with 3 layers. Figure 2 shows a flow of biometric recognition using local phase array which consists of 2 steps: (i) feature extraction and (ii) matching. In the following, we describe the detailed procedure of each step.

Fig. 2.

Flow of biometric recognition using local phase array.

2.1 Feature Extraction

The hierarchical local phase array consists of the phase feature of the entire image in the top layer and the phase features of local block images in the middle and bottom layers. Phase features in the middle and bottom layers are extracted according to the position of reference points. The feature extraction step consists of (i) reference point placement, (ii) hierarchical image generation and (iii) phase feature extraction.

(i) Reference Point Placement

The reference points are the center coordinates of each local block image. Let a reference point be ${\varvec{p}} = (p_1,p_2)$ $(={\varvec{p}}^0)$ and the registered image be $I \ (=I^0)$, respectively.

(ii) Hierarchical Image Generation

For $l=\{1, 2\}$, we generate the l-th layer images $I^l(n_1,n_2)$, which are $1 / 2^l$ times the size of $I^0(n_1,n_2)$. Also, we calculate coordinate ${\varvec{p}}^1 = (p^1_1,p^1_2) = \left( \left\lfloor p^0_1/2 \right\rfloor , \left\lfloor p^0_2/2 \right\rfloor \right)$, corresponding to $\varvec{p}^0$ on $I^1(n_1,n_2)$.

(iii) Phase Feature Extraction

In the top layer, we calculate 2D DFT of $I^2$ and its phase components. In the middle and bottom layers, we extract $W_1 \times W_2$-pixel local block images with its center on ${\varvec{p}}^1$ and ${\varvec{p}}^0$ from $I^1$ and $I^0$, respectively. Then, we calculate 2D DFTs of all the local image blocks and their phase components. To reduce the size of local phase array, we can eliminate the meaningless high frequency components which are not required for calculating the Band-Limited Phase-Only Correlation (BLPOC) function [6]. We also reduce the size of phase information based on the symmetry property of DFT. In addition, the amount of registered data can be reduced by phase quantization. Refer to [5] for more details on phase quantization.

2.2 Matching

The matching step consists of (i) hierarchical image generation of the input image, (ii) global image registration in the top layer, (iii) local image block registration in the middle layer, (iv) similarity evaluation in the bottom layer and (v) matching score calculation.

(i) Hierarchical Image Generation of the Input Image

Let $J \ (=J^0)$ be the input image. For $l=\{1, 2\}$, we generate the l-th layer images $J^l(n_1,n_2)$, which are $1 / 2^l$ times the size of $J^0(n_1,n_2)$.

(ii) Global Image Registration in the Top Layer

In the top layer, we estimate the translational displacement between $I^2$ and $J^2$ using BLPOC. We denote the estimated global translations as ${\varvec{\delta }_\mathrm{global}}=(\delta _{\mathrm{global},1}, \delta _{\mathrm{global},2})$.

(iii) Local Image Block Registration in the Middle Layer

In the middle layer, we estimate the translational displacement between local block images of $I^1$ and $J^1$. We extract the $W_1 \times W_2$-pixel image blocks with its center on ${\varvec{q}}^1={\varvec{p}}^1+2{\varvec{\delta }}_\mathrm{global}$ from $J^1$. Using BLPOC for each local block image pair of $I^1$ and $J^1$, we estimate the local translations ${\varvec{\delta }}_\mathrm{local}$.

(iv) Similarity Evaluation in the Bottom Layer

We evaluate the similarity between each local block image pair in the bottom layer. We extract the $W_1 \times W_2$-pixel local block images with its center on ${\varvec{q}}^0=2({\varvec{q}}^1+{\varvec{\delta }}_\mathrm{local})$ from $J^0$. Then, we calculate the BLPOC function between each local block image pair of $I^0$ and $J^0$ and obtain the correlation peak value $\alpha$.

(v) Matching Score Calculation

We evaluate the matching score between I and J according to the correlation peak values obtained in the step (iv). In this paper, we employ the matching score S defined by

$$\begin{aligned} S = \frac{\sum _i \alpha _i}{N_\mathrm{block}}, \end{aligned}$$

(1)

where $N_\mathrm{block}$ is the number of local image block pairs, and $\alpha _i$ ($i = 1, 2, \cdots , N_\mathrm{block}$) is the correlation peak value of i-th local image block pair.

3 Score Fusion Approaches

This section describes score fusion approaches considered in this paper.

Score fusion approaches are broadly classified into 3 approaches: (i) density-based approach, (ii) classifier-based approach, and (iii) transformation-based approach [10]. The density-based approach estimates the Probability Density Function (PDF) of the matching scores for both genuine and impostor pairs of each trait, then, this approach calculates the combined matching score according to the relation between genuine and impostor PDFs. Given an accurate estimation of these PDFs, the density-based approach exhibits the best performance in score fusion approaches, however, an accurate estimation is not always possible under practical situations where the amount of training data is limited. Transformation-based approach allows us to approximate easily the relation between the PDFs compared with classifier-based approach. Therefore, we employ the density-based and transformation-based approaches in this paper. From this point, the matching scores of face, iris, palmprint and finger knuckle are denoted by $S_x \left( x \in T = \left\{ \mathrm{F}, \mathrm{E}, \mathrm{P}, \mathrm{K} \right\} \right)$, and the matching score vector is denoted by ${\varvec{S}} = [S_\mathrm{F}, S_\mathrm{E}, S_\mathrm{P}, S_\mathrm{K}]$, where the high value of $S_x$ indicates the high possibility of genuine match. The set of biometric traits to be fused is indicated by $T^\prime$ as

$$\begin{aligned} T^\prime \in \mathfrak {P}(T) \backslash \{\phi ,\{\mathrm{F}\},\{\mathrm{E}\},\{\mathrm{P}\},\{\mathrm{K}\}\}, \end{aligned}$$

(2)

where $\mathfrak {P}(T)$ is a power set of T.

3.1 Density-Based Approach

This approach uses the PDF of matching score S to combine matching scores calculated from different traits. In the training stage, the PDFs $p_x(S\mid \omega )$ of each $x \in T^\prime$ for $\omega \in \left\{ \mathrm{genuine}, \mathrm{impostor} \right\}$ are estimated from the training data set, in this paper the PDF $p_x(S\mid \omega )$ is modeled by a Gaussian mixture. In the testing stage, the values of $p_x (S_x\mid \mathrm{genuine})$ and $p_x (S_x\mid \mathrm{impostor})$ for ${\varvec{S}}$ of the input data. Then, we calculate the combined matching score $S_\mathrm{fusion}$ as a Likelihood Ratio (LR) between genuine and impostor defined by

$$\begin{aligned} S_\mathrm{fusion} = \frac{p ({\varvec{S}} \mid \mathrm{genuine})}{p ({\varvec{S}} \mid \mathrm{impostor})} = \frac{\prod _{x \in T^\prime } p_x (S_x \mid \mathrm{genuine}) }{\prod _{x \in T^\prime } p_x (S_x \mid \mathrm{impostor}) }. \end{aligned}$$

(3)

3.2 Transformation-Based Approach

This approach employs simple fusion rules to calculate the combined matching scores by transforming input matching scores of different traits into a common domain. The parameters for score transformation, i.e., score normalization, are calculated from the training data set. In this paper, we employ 3 normalization techniques [7]: (i) Min-max, (ii) Double sigmoid, and (iii) tanh-estimators. Using the parameters, each element $S_x$ of the matching score vector ${\varvec{S}}$ is transformed into $S^\prime _x$. We then calculate the combined matching score $S_\mathrm{fusion}$ from the normalized matching score vector ${\varvec{S}}^\prime = [S^\prime _\mathrm{F}, S^\prime _\mathrm{E}, S^\prime _\mathrm{P}, S^\prime _\mathrm{K}]$ using the simple fusion rules:

$$\begin{aligned} \begin{array}{ll} \bullet \mathbf{Average}: &{} S_\mathrm{fusion} = \frac{\sum _{x \in T^\prime } S^\prime _x}{\left| T^\prime \right| } \\ \bullet \mathbf{Mean \ Square \ (MS)}: &{} S_\mathrm{fusion} = \frac{\sum _{x \in T^\prime } S^{\prime 2}_x}{\left| T^\prime \right| } \\ \bullet \mathbf{Residuals}:&{} S_\mathrm{fusion} = 1 - \prod _{x \in T^\prime } \left( 1 - S^\prime _x \right) \end{array} \end{aligned}$$

4 Experiments and Discussion

This section describes a set of experiments to evaluate performance of the proposed multibiometric recognition system using local phase array.

4.1 Virtual Multibiometric Databases

We make a virtual multibiometric database generated from public biometric databases to evaluate performance of score level fusion of face, iris, palmprint and finger knuckle recognition algorithms.

The virtual multibiometric database consists of chimera subjects created by pairing together face, iris, palmprint and finger knuckle images from unimodal databases. In the experiments, the number of chimera subjects is 100 with 4 images for each biometric trait. As for face images, we use 144 subjects with 4 images from FERET database [9]. As for iris images, we use 175 subject with 4 images from Iris Challenge Evaluation 2005 (ICE 2005) database [2], where we have assumed the left eye and right eye of the same person as different subjects. As for palmprint images, we use 600 subjects with 4 images from CASIA Palmprint database [1], where we have assumed the left and right hand of the same person as different subjects. As for finger knuckle images, we employ PolyU FKP database [3] which consists of 7920 images with 165 subjects and 6 different images for each of the left index finger, the left middle finger, the right index finger and the right middle finger in 2 separate sessions. We assume each finger knuckle of the same person as different subjects, i.e., a total of 660 subjects ($= 165$ subjects $\times 4$ fingers). We use 660 subjects with 4 images from PolyU FKP database, where 2 images are from the first session and the remaining 2 images are from the second session. Subsequently, for all these subjects, we calculate matching scores of all the possible combinations of genuine pairs using local phase array. Then, we made a database for each trait by selecting first 100 subjects in order of increasing the average of those matching scores. In the following experiments, we combine these 100 subjects to make chimera subjects of virtual multibiometric databases. Figure 3 shows examples of images for each database.

For performance comparison, we employ the following conventional algorithms: Local Phase Quantization [4] for face recognition, Ordinal Code [11] for iris recognition, SIFT [13] for palmprint recognition, and Local-Global Information Combination [12] for finger knuckle recognition. These algorithms are known to belong to the state-of-the-art algorithms for the corresponding biometric trait.

For each database, we evaluate Equal Error Rate (EER) of conventional and local phase array algorithms using $100 \times _4\mathrm{C}_2 = 600$ genuine pairs and $_{100}\mathrm{C}_2 \times 4 \times 4 = 79,200$ impostor pairs. Figure 3 shows a summary of EERs of conventional algorithms and local phase array for each database. The proposed algorithm exhibits comparable performance with the conventional algorithms specialized for each biometric trait despite selecting worst subjects for the local phase array algorithm.

Fig. 3.

Examples of ROI images in each database and EERs [%] of each recognition algorithm: (a) face, (b) iris, (c) palmprint, and (d) finger knuckle.

4.2 Performance Evaluation

We evaluate the error rates statistically by using the bootstrap technique, which is a non-parametric method to estimate the confidence interval by random data sampling.

We create 100 chimera subjects with 4 biometric traits by randomly combining subjects from each database generated in Sect. 4.1. For each trait, the number of all the possible combinations of genuine matching scores is $_4\mathrm{C}_2 = 6$, while the number of all the possible combinations of impostor matching scores between different 2 chimeras is $4 \times 4 = 16$, since one chimera subject has 4 images of each trait. In this paper, 2 genuine scores and 6 impostor scores are used in the training step and remaining scores are used in the testing step. We then generate a set of virtual score vectors using the same procedure in [7]. Thus, the total number of genuine combinations of score vectors ${\varvec{S}}$ for one chimera subject is $4^4 = 256$, since there are 4 genuine matching scores for each trait. On the other hand, the total number of impostor combinations of score vectors ${\varvec{S}}$ between 2 different chimeras is $10^4 = 10,000$, since there are 10 impostor matching scores for each trait. Among the above score vectors, we randomly select 128 genuine pairs and 512 impostor pairs to generate a set of virtual score vectors. Then, for $\left| T^\prime \right| = 2$, we apply the score fusion approaches described in Sect. 3 to $128 \times 100 = 12,800$ genuine pairs and $512 \times _{100}\mathrm{C}_2 = 2,534,400$ impostor pairs and evaluate EERs calculated from combined matching scores. We repeated the experiment described above 100 times for different combinations of subjects and scores in the virtual database.

Table 1 summarizes EERs for each fusion rule when using all the possible combinations of biometric traits, where EER indicates an average of 100 trials. “Conventional” indicates fusing scores calculated by conventional algorithms, “LPA” indicates fusing scores calculated by the algorithm described in Sect. 2. “Best single modality” indicates lower EER of the respective 2 traits. “Similarity” indicates a simple combination of the matching score between 2 images, which is given as a value within [0, 1] calculated by each recognition algorithm.

Table 1. EERs [%] for each combination rule.

From the EERs of Conventional, we can make 3 observations. First, most of them are higher than those of single modality cases. In particular, the combination of face, whose EER is the highest, and finger knuckle, whose EER is the lowest, shows this tendency. Second, specifically “Similarity + MS” exhibits significantly high EER for face-palmprint and iris-palmprint compared with other fusion rules. This is because distribution of matching scores for palmprint is significantly different from other algorithms. Third, EERs of “Double sigmoid + MS” and “Double sigmoid + Residuals” improves the EERs of other fusion rules. These observations indicate that conventional algorithms have to employ normalization to use simple combination rules and to ensure high performance.

For the result of LPA, we can make three observations. First, EERs are lower than those of single modality cases, except for the combination of iris and palmprint for some fusion rules. Second, “Similarity + MS” shows the best EER between simple combinations. Third, in this case also, the EERs of “Double sigmoid + MS” and “Double sigmoid + Residuals” improve the EERs of simple combination and the other normalization methods. These observations indicate that LPA does not always need normalization since it can employ simple combinations for almost all the combination of traits.

In both Conventional and LPA, LR exhibits the lowest EERs for most cases. As mentioned in Sect. 3, LR is expected to show the best performance, if the PDFs for genuine and impostor pairs were estimated accurately. However, the transformation-based approaches, “Double sigmoid + MS” and “Double sigmoid + Residuals,” also exhibit efficient performances comparable to the ones of LR. Therefore, these score fusions are robust against limited training data and the diversity of their score distributions compared with LR.

Focusing on “Similarity + Average” and “Double sigmoid + MS/Residuals,” EERs of LPA are significantly improved compared with those of Conventional. This result indicates that multiple recognition algorithms require a complex optimization for score normalization and combination approaches to exhibit the efficiency of score fusion observed in LPA, since the optimal score normalization method and optimal fusion rule might be different for each recognition algorithm.

As observed above, successful score fusion with LPA does not depend on normalization methods and fusion rules as it does with the conventional method. Hence, the use of LPA for multi-modal biometric systems makes it possible to improve the performance only with simple combination.

5 Conclusion

This paper proposed a biometric recognition algorithm with score level fusion of multiple matching scores calculated by local phase array. Through a set of experiments, we demonstrate that simple score fusion approach is enough to exhibit good recognition performance for local phase array. Hence, the use of local phase array makes it possible to realize simple and compact multibiometric person authentication systems, since only one matching module and simple score fusion approach are employed. In future, we will consider other types of multibiometric fusion for local phase array. Also, we will develop a fast multibiometric identification system using local phase array.