1 Introduction

Situational awareness systems became very popular recently due to increased amount of information required to make decision on the access permissions as well as increased complexity of decision-making process. The analysis of appearance, physiological data, behavioral data, soft biometrics and newly emerged social behavioral patterns in collaborative and virtual environments has led to the need to develop new type of situation awareness systems that combine information fusion with pattern recognition of humans or their avatars.

Recently, the meta-model for situation awareness system was developed and presented in Cyberworlds 2012 [32], which identified future challenges in designing such systems. Specifically, authors pointed out that risk monitoring, treatment and communication as an operational tool for proactive prevention/mitigation in real-time and virtual collaborative environments is important, and that new methods and quantifiable metrics for assessment of the vulnerabilities of the system capacities of building/structure are necessary. In this paper, we propose a multibiometric approach to decision-making in situation awareness systems, based on random projection and random fusion method. One of the unique features of this approach is the ability to fully protect the identity of an individual and the privacy of collected data, while at the same time having very high discriminative capabilities for identifying individuals and threats.

Multimodal biometric system is a relatively new alternative to unimodal biometric system [6, 9, 33, 36]. Multimodality can be achieved in different ways: such as combining multiple biometric traits, selecting different feature sets from the same source of biometric, using different sensors, fusing the decision of individual biometric system, etc. [7, 34, 35]. In our system, we have used different feature sets from different biometric traits. From the literature, it is found that multimodal biometric system often outperforms a unimodal biometric system in terms of accuracy and reliability [3, 17, 18]. It can solve some common unimodal biometric system such as intra-class variability, interclass similarity, non-universality, sensitivity to noise and other issues. Multimodal biometric system can improve the performance of a biometric system in a number of aspects, including accuracy, circumvention, resistance to errors and spoof attacks [3]. Multimodal biometric systems are more secure compared with unimodal systems in terms of authentication accuracy [4].

The concept of cancelable biometric or cancelability has become popular very recently [8, 9, 33], because in traditional biometric system, original biometric template can be reconstructed using reverse engineering [5]. This is a new trend that focuses on how to transform a biometric data or feature into a new one so that users can change their single biometric template in a biometric security system. Up until now, multimodal system cancelability has not been considered. However, this can be argued that template protection is even more crucial in such systems. Multimodal biometric system uses numbers of biometric credentials so it is cooperative to the attackers to get more evidences if they manage to break the system. Once templates from multimodal system are compromised, the individual loses all the sensitive data stored in the current security system and all other systems related to that individual. This is why it is crucial for a multibiometric system to provide the template security and cancelability. It can be claimed that using cancelability for each biometric trait separately in multimodal biometric system can solve the problem. However, this is not as easy as it looks: the solution may be costly in term of computation efforts and performance. If one of the traits gets compromised, similar method can be used to break other traits. Another concern can be key protection and storage; system needs to issue key for each biometric trait. In this paper, we tackle the above problems and present a novel solution for cancelable biometrics in multimodal system. We develop a new cancelable biometric template generation algorithm using random cross folding of multiple biometric traits, random projection and transformation-based feature extraction and selection. Performance of the proposed algorithm is validated on a virtual multimodal face and ear database. Specifically, algorithm security is validated by issuing different original and fake keys for different subjects of the database. Algorithm discriminability remains high because of the similar cross-fold indices and random projection matrix (vectors) for a class. Similarly, revocability and diversity are ensured by issuing different sets of keys for training and testing process. This paper is an extended version of the paper multimodal cancelable biometrics [9]. In contrast to [9], which uses twofold random cross-folding, with distance-based clustering, in this paper we utilize a threefold with linear discriminant features and combination with situation awareness. We introduced a new methodology in this paper to reduce the risk to the system and improve the performance. It is to the best of our knowledge, the first cancelable multimodal system developed using this novel methodology.

Proposed methodology is briefly described as follows. At the first step, the threefold random selections of raw features are made for each biometric trait. It gives system random mixtures of raw feature from face and ear biometrics. The process can be named as random biometric fusion. The outcome of this process is similar to watermarking of face on ear template and vice versa. In the second step, features are extracted from each fold separately using Fisherface (Fisher linear discriminant analysis) method. Features of each fold are then projected using orthogonally transformed cross-fold indexes in the third step. Distance-based feature extraction method is applied on each pair of features from three folds. In the fourth step, features from orthogonal cross-fold indexes are projected using random projection method. Finally, to enhance the discriminability, the linear discriminant analysis (LDA) is applied to the obtained features from random projection. These features are then used in a classifier to get the final authentication performance. The system is tested on virtual multimodal databases for face and ear biometrics, considering both cancelability and performance. The results presented in experimentation section show that the developed random projection method for cancelable biometric satisfies the template protection requirements.

2 Related works

There are different levels of attacks that may take place in a biometric system in both real and virtual environment at sensor level, application level, database level, etc., [2, 29–31]. Therefore, biometric systems should be intelligent in computing and processing. There are variety of methods for intelligent computing that were developed very recently [10, 27]. Cancelable biometric system is a type of intelligent system that can protect the database from template-level attack. Biometric system with cancelability or template protection scheme can protect the biometric database from security threats.

From the literature review, we have summarized the categories of cancelable biometric system. Based on the number of biometrics used, cancelable biometric system can be divided into unimodal and multimodal cancelable biometric system. In unimodal system, single biometric traits are used for cancelability. The multimodal cancelable biometric system uses multiple biometric traits. Based on the order of cancelability, both multimodal and unimodal cancelable biometric system can be divided into two main categories: first-order cancelable biometric system and second-order cancelable biometric system.

In this type of system, biometric data are transformed once into cancelable feature from original or extracted biometric feature. Cancelable biometric template can be generated using either biometric cryptosystem or transformation-based system. In biometric cryptosystem, a secret key is linked with the biometric data [1, 12, 13]. This key can be selected randomly or can be generated from biometric data. Two most popular biometric cryptosystems are fuzzy commitment scheme [11] and fuzzy vault scheme [12]. Goh et al. 2003 presented cancelable face biometric using biometric cryptosystem. In his work, he generated key from face template. Research work presented in the paper [14], is an example of transformation-based system where authors transformed fingerprint into another domain using Cartesian, radial and functional transformation. They are not combining transformation to obtain the cancelable template, thus this is first-order system.

In second order cancelability, biometric traits are transformed into cancelable template twice. In each level, both biometric cryptosystem and transformation-based systems can be used. It is also possible to repeat the same scheme twice for cancelability. Ratha et al. 2007 first proposed the concept of cancelable biometrics (or cancelable template). He provided the basics of the cancelable biometrics, but did not address the discriminability issues. Later salting (cryptosystem) of biometrics was introduced based on combining a user-defined key or password to increase the between-class variation and enhance the discriminability. However, a transform-based approach also takes the original biometric template and the user-specific key to enhance the discriminability of the transformed templates [15]. The advantage of the transformed template is cancelability. Second order of cancelable system can also be designed based on biometric cryptosystem and transformation-based approach [8]. Their proposed approach retains the advantages of both the transform-based approach and biometric cryptosystem approach. Recently, Paul and Gavrilova [9] presented a second order multimodal cancelable biometric system using random projection. Three levels of random cross-folding is used in the proposed method that ensure more cancelability, hence the security. There are also group of methods that can exploit multi-order cancelability. In this type of system, several cancelable mechanisms are applied on biometric data. The method presented in this paper belongs to multi-order cancelable biometric system.

In this method, multimodal approach is presented using cross-folding random indexes of template to achieve the multimodal cancelable biometric template. There are a number of benefits to the proposed method. Applying the multimodal technique at feature level, the proposed cancelable multibiometric system can enhance the interclass variability and thus improve the performance of the multimodal system. Furthermore, the main complications of multimodal biometric system are memory and computational complexity during the training and testing. In the traditional multibiometric system, all of the biometric traits are stored in the database and used for computation. The proposed cancelable multibiometric system is able to fuse all the templates for different biometrics into one single multimodal biometric trait that can reduce the database size and computation during the identification and verification process.

3 Proposed method

3.1 Situation awareness of the system

Situation awareness system can analyze the situation and react differently to environment based on the change of a system state or a system variable [32]. These types of systems are also capable of performance monitoring and real-time risk analysis [32]. Biometric security system can be designed as a situation awareness system to better mitigate security threats and risks, as well as to respond to changes in environment. We propose a new concept of biometric cancellable situation awareness system that can ensure the security of the biometric template and monitor the performance of the system over time. This unique combination has not been studied previously in the literature. The proposed system is capable of automatic decision-making in a high-risk situation. On the other hand, based on the behavior of the system, user can estimate the risk to the system. Figure 1 shows the general architecture of a situation awareness system developed to support cancelability.

Fig. 1
figure 1

Situation awareness system which supports cancelability

Full size image

Authors of the paper [32] established the links between risk monitoring, treatment and communication as an additional obligatory task of disaster and emergency management system [32]. Figure 2 shows the meta model of their system. This model also works for shared virtual environment.

Fig. 2
figure 2

Meta modal of situation awareness for emergency management system [32]

Full size image

Biometric security system is frequently utilized in a shared environment such as airport, border control and public access control. A feature often not considered in such environment is ability of a biometric security system to estimate and predict the risks. The following section describes the architecture of the biometric system for enhanced security and recognition performance.

3.2 Proposed architecture

The proposed cancelable biometric system is divided into four parts: (a) random cross-folding for biometric fusion; (b) Fisher’s linear discrimination of distance-based feature; (c) double random projection; and (d) applying LDA to enhance the interclass variability. Finally, the cancelable template is used to model the k-NN classifiers for performance analysis. Figure 3 shows the block diagram of the random transformation-based cancelable biometric template generation. In the first stage of the transformation is random cross-folding. The outcome of the process is two sets of feature that are cancelable. Random indexes are used to generate three folds. Distance-based features are calculated from Fold 1, Fold 2 and Fold 3. Distance features are then projected using random projection that transforms the original m-dimensional data to n-dimensional. Random indexes are transformed using Gram–Schmidt transformation to an orthogonal random matrix. This matrix is used for first random projection. In the second random projection, a random matrix from random seed is transformed using Gram–Schmidt transformation. Transformed matrix is used as random projection matrix. To enhance the discriminability of the feature, LDA is used to find discriminant feature from randomly projected features. Fisherface and principal component analysis are other possibilities, but have lower discriminability thus is less preferable. Finally, LDA features are then classified using k-nearest neighbor classifier. Projected feature using LDA are cancelable because it comes from two levels of random projection and initial cross-folding between face and ear template. If the indices of cross-folding and random projection matrix are changed, the cancelable template can be achieved for multiple biometrics. Furthermore, by changing the random projection matrix new cancelable template can be generated for different applications. Multi-order of cancelability is achieved using both random cross-folding and two levels of random projection. This system is aware of the state/situation of the biometric security system. It may exercise the following situations over time.

Fig. 3
figure 3

Block diagram of the proposed system: cancelable template generation algorithm using transform-based approach

Full size image

Situation 1-attack attempt: Attack attempt: When the system is aware that some attacker is trying to attack the system, it moves into this state and can take actions based on the situation. System can change the intermediate cancelable transformation so that user does not require any change in key. At this awareness, system can block some of the attack threats. In this situation, system start minimize true acceptance rate and system alert could be issued.

Situation 2-attack: System can block the user and record the event. Later, when user tried to login, the notification with the warning about the attack using his/her credentials could be issued. System is able to reissue the template and keys. At this situation, system will be online but some users may be interrupted from the service.

Situation 3-compromised: Once the system found that the template is compromised, it can reissue the template for the user and check other individuals credentials for authentication. This level of situation may cause emergency awareness, where service outage may be triggered.

3.3 Random biometric fusion

In this paper, we proposed to achieve the cancelability using random cross-folding and random projection method. We furthermore applied it for the first time to multimodal biometric system to achieve cancelability in the presence of multiple biometrics. Feature fusion of multiple biometric traits is a highly important step for multifold random selection. Raw biometric features randomly are divided into three parts. A pseudorandom number algorithm is used to split the raw features into two parts. Face biometric template is divided into Face-Fold 1, Face-Fold 2 and Face-Fold 3. Similarly, ear template is divided into Ear-Fold 1, Ear-Fold 2 and Ear-Fold 3. Finally, Face-Fold 1, Ear-Fold 2 and Ear-Fold 3 are combined to achieve the Face-Ear Fold 1, Face-Ear Fold 2 and Face-Ear Fold 3. Random indexes are selected such a way that all three selections are almost similar number. It may differ by one to three pixels. To make the feature set similar in size for classification and consistency of the system, we have discard those pixels. From the experiment, we found that discarding a maximum of three pixels does not have any effect on feature space.

Cross-folding fuses the face and ear biometrics into two template of Face-Ear template. First template is watermark of face on ear. Similarly, the second template is watermark of ear on face. However, this process is not fixed in term of raw feature selection. Raw features are randomly selected to generate the Face-Ear template so the system is able to generate multiple templates from one set of face and ear biometric. Another advantage of this process is it creates new relationship among biometric traits.

3.4 Feature extraction using Fisherface

Fisherface method is a class-specific linear projection method for face recognition to maximize the interclass variation and minimize the intra-class similarity [19], whereas Eigenface method finds total variation of data regardless of class specification [16]. The Fisherface allows us to identify the discriminative class-specific feature extraction. Because of the class specification and discriminability of feature extraction, we have used Fisherface method instead of Eigenface or PCA as a tool to find the features from cross-folded cancelable biometric data. The three folds of biometric feature are used to calculate the distance feature. We have taken absolute distance between features of two folds then distance between the remaining fold. It is also possible to use Euclidean distance, but the random projection is already in Euclidean space. Taking Euclidean distance may cause overlap of large number of classes. Finally, these features are projected using random projection in two levels.

3.5 Random projection

Features found from previous section are used to project using random projection. Johnson and Lindenstrauss [20] first developed the idea of random projection. A number of researchers have used random projection for cancelable biometric system [8, 15, 21]. Random projection technique is used as an alternative of PCA for dimension reduction of data [22]. The main goal of random projection is to project vector on to a reduced dimensional space called Euclidian space [20]. The main property of the random projection is to keep the Euclidean distance similar in some extent before and after the projection of vector. Random projection changes the vector and transform into new vector, but it keeps the statistical property of the original [21]. In the proposed method, random projection matrix is calculated in two steps. In the first step, a random matrix is generated based on the random seed. In the second step, random matrix is transformed in to orthogonal matrix using Gram–Schmidt orthogonal transformation. The Gram–Schmidt transformation is used to orthonormalize a set of vectors (2D matrix) in Euclidean space Rn [23]. This transformation allows the projection to keep the distance of projected feature same in Euclidean space.

4 Experiments

To validate the proposed methodology, we have designed comprehensive test settings involving five biometric databases. The tests were intended to examine system behavior with respect to cancelability, specifically on recognition performance of the cancellable system (the higher the better) and on possibility to reconstruct original template from cancellable template (the lower the better). To ensure successful training, database selection and pre-processing is necessary and crucial. For testing our method, we have used a virtual database that contains data from two different unimodal biometric databases for face and ear. For face, FERET [24] VidTIMIT [25] and Olivetti Research Lab Database [28] were chosen. We have taken variety of databases and randomly combined them to generate virtual database. Figure 4 shows the virtual database setup from five face and ear databases.

Fig. 4
figure 4

Randomly sampled 10 virtual multimodal database from five face and ear databases. Each database contains 858 face and ear images of 143 individuals

Full size image

A subject selection from the face database was random and different sets of virtual database were generated. Two databases called University of Science and Technology Beijing (USTB) Image Database I & II [26] for ear are selected to generate virtual multimodal Face-Ear. To generate the virtual database all the images of the ear databases are used. Sample of virtual multimodal biometric database are shown in Fig. 5.

Fig. 5
figure 5

Some samples from virtual multimodal database

Full size image

4.1 Experimental setup

We have designed cancelable multibiometric system using MATLAB 2009b and C# on Intel Core i7 2.2GHz Windows 7 Enterprise workstation. Developed system is menu driven graphical user interface (GUI) that support both 32-bit and 64-bit version of Windows. Multimodal virtual database is preprocessed and saved as MATLAB standard database file with mat extension. Each biometric trait is scaled into 75x50-resolution grayscale bitmap image. GUI is designed using C# that includes a button to selection of database for connection. As soon as database is connected, it automatically retrieves all the dimension information and number of samples form the database. Developed system has the capability of processing biometrics of different resolution and this processing is automatic. User can also input number of fold for cross-validation process. To improve the training and testing process tenfold cross-validation of the dataset is used. All the results presented in this paper are from tenfold cross-validation and system can automatically create the dataset for tenfolds to use them in training and testing. In addition, random indices and random projection matrix can be changed using another configuration GUI module.

4.2 Experimental results

In the experiment, the goal was to check the performance of multimodal cancelable biometric system using the proposed method. Performance of cancelability depends on both recognition accuracy and the cancelability of the biometric system. To achieve this goal, the following scheme is designed.

The system is tested on variety of measures such as improvement of classification accuracy, improving and keeping the performance of multimodal biometric system. The result of multimodal and unimodal cancelable biometric system is also compared. For the final performance, tenfold cross-validation is used on our virtual multimodal database using k-NN classifier. Properties of cancelable biometric are tested, such as keeping interclass variability (improved performance), issuing new template, reverse processing to generate original template. As a result, it is shown that using cancelable biometric template achieved a better performance than the matching performed on original image. We also tested the level of random projection for cancelability. We have found that first and second level of random projection keeps similar recognition performance. However, for third and fourth level of random projection performance degraded (See Fig. 6). For cancelable biometric template and the original face template, similar attribute is observed. The similar comparative result for cancelable ear and original ear template is also shown. Finally, the performance of multimodal cancelable template and cancelable unimodal cancelable biometric is shown in Fig. 7. Multimodal cancelable system provides us with better performance.

Fig. 6
figure 6

Recognition performance over different levels of random projection. Increased level of random projection degrade the performance

Full size image
Fig. 7
figure 7

Recognition performance for cancelable unimodal and cancelable multimodal biometric system. Multimodal cancelable system outperforms unimodal system

Full size image

From the result, it has been found that using cancelable biometric template from unimodal system preserves the interclass and intraclass variability. On the other hand, it can be also seen that multimodal cancelable system improves the performance compared with unimodal biometric system. Thus, the proposed multimodal cancelable biometric system preserves the cancelable property. Assuming memory is sufficient for the system, LDA is capable of correctly classifying features might degrade 2–5 %.

4.3 Results of cancelability

If the randomly selected feature indexes were not available, cancelable biometric system would not correctly recognize an individual class. We have tested the system using other randomly selected indexes to split the features. The test was under the same experimental conditions, with modifications in random selection of features. It is found that if the random selection of feature is changed, classifier is unable to recognize the person. This performance ensures cancelability of the template. If those indices of cross-folding are available to the attacker, it will be computationally too hard to reproduce the original face or ear template. Figure 8 show the result of changed random selection and projection. In first and second level of random projection system security is extremely high. Since higher level of random projection decrease the recognition performance, it is better to take up to second level of random projection. In this optimal level of projection, system keeps very good recognition accuracy and security of the template.

Fig. 8
figure 8

Recognition performance over different levels of random projection. Increased level of random projection degrade the performance

Full size image

5 Conclusion

Situation awareness cancellable biometric system is presented in this paper. Biometric system may be often compromised. Database of a biometric system normally stores privacy information and credentials for an individual. System behavior in a real and virtual environment can be established by incorporating the situation awareness. Performance of a system is also important for security, thus a new cancelable biometric template generation algorithm is presented using random fusion and projection of biometric traits. A number of situation awareness scenarios are considered. Experimentation was carried out on virtual multimodal system, for face and ear templates. The results showed that the proposed method can effectively produce cancelable template for multimodal biometric system without sacrificing recognition accuracy. In the future, different fusion methods can be tested to comprehend their influence on the performance. Detailed situation analysis for the cancelable system is another direction of future research.