Face Recognition Through Symbolic Data Modeling of Local Directional Gradient

Abstract

Face recognition systems are vulnerable to changes in expression, light, and occlusion factors. This paper presents a robust face recognition algorithm that effectively deals with these variations. The algorithm is based on the local directional gradient features that exploit the edge information in multiple directions, modeled as symbolic data object . First, face parts are detected and cropped from the face images, then the cropped images are resized to 64p × 64p grayscale images. Further, local directional gradient-based features are computed based on co-relation between pixel elements in multiple directions. The extracted directional gradient features (GC) are represented as a symbolic data object. For classification , a new symbolic similarity measure based on content is devised and employed. The experimental results on AR database show that proposed algorithm obtains an average recognition accuracy of 93.50% in the presence of illumination variations, expression changes, and occlusion variations.

1 Introduction

Face is one of the widely used universal biometric features of human beings. Due to its passive nature and ease of image acquisition, several face biometric systems/techniques have been developed and proposed by a large community of researchers. Such biometric systems have advantage over other biometrics systems requiring subjects’ cooperation [1], such as iris, finger print, and palm print, etc. In general, most of face biometric recognition systems can be classified into feature and appearance-based systems depending upon the features considered for recognition [2, 3]. In feature-based face recognition systems, features of mouth, nose, eyes are detected and the relation between these parts are established. The characteristics and relations between the detected parts are used as feature descriptors. In contrast, appearance-based face recognition systems use pixel intensity values of face images for recognition.

In feature-based recognition systems, the shape and relation between facial parts are considered as feature descriptors. These systems utilize prior knowledge of human face images and most of them are based on graph matching or dynamic links. Many graph construction-based algorithms are presented which use the feature points of an image as vertices, and their corresponding feature descriptors as labels [4]. The graph constructed for face image can be used as a whole [4] or it can be used to divide the facial parts into two or more subgraphs which are further used to extract features for recognition and classification [5]. For example, the distance between facial parts and their relative positions are derived and stored in a feature space matching. During the matching process, these features are used as feature descriptors. But such representation requires reliable and accurate face detection and tracking methods, which are difficult in many situations [6].

In appearance-based recognition systems face features are extracted from whole image or some parts of the face image. More recently, Principal Component Analysis (PCA), Independent Component Analysis (ICA), Local binary pattern (LBP) [7], the local directional pattern (LDP)-based linear methods have been proposed [1]. The PCA retains most important features about original samples during the projection of samples to a feature space in order to reduce the space complexity. The ICA uses second-order and high-order statistics of input face image and projects extracted face features onto the basis vectors which are statistically independent. The LDA captures class-related information and finds a set of feature vectors that minimize the difference within a class while maximizing the difference between classes. In order to take the advantage of PCA and LDA, the KPCA plus LDA [7] method is proposed; however, it considered only lower order statistics and neglected the higher order statistical relations such as relation between three or more pixels for classification . In order to take advantage of higher statistics for classification task LBP method has been proposed. Even though, the LBP descriptor is more robust and efficient compared to other appearance-based face recognition systems to uniform illumination changes, however, it is sensitive to the presence of noise and nonuniform illumination changes in multiple directions. The gradient map gives more information about the edges or corners of an image at multiple directions [8, 9]. The gradient of adequate strength is considered important in a given task of classification [10]. The gradient-based LDP and variation of the LDP methods has been developed to handle issues of LBP features. LDP feature is obtained by computing edge information in multiple directions at each element of the image plane [3]. In contrast to linear methods many nonlinear methods such as kernel PCA have been proposed [6]. Even though much work has been done on face recognition, higher accuracy with minimal set of face features still remains a great challenge [3, 11]. However many appearance-based methods are developed which use minimum set of features for recognition but the performance of such methods is based on the type of distance/similarity measure used for classification and are computationally expensive [5]. Hence, there is a need of development of robust and efficient appearance-based system that uses minimum set of face features for person identification.

The proposed work describes an appearance-based robust and efficient person identification system which uses minimal set of features. The work also explores the advantages of symbolic similarity measures through symbolic modeling of face features [6, 12,13,14,15] during recognition and classification process. The following are some of the major contributions made in this work:

• A new directional gradient magnitude feature of a face image, using symbolic data modeling and analysis approach for face recognition .

• Issues related to changes in illumination, expression and presence of occlusion during face recognition.

A new appearance-based symbolic data modeling approach to face recognitions is proposed in this work. Initially, face parts are detected from the face images using Viola-Jones algorithm. Later, the cropped face images are resized to 64 × 64 pixel size and converted into grayscale images. Further from the training samples the sum of cardinal gradient magnitude (GC) in multiple directions is calculated for each class of face images and represented as a symbolic data object. Similarly for the test image, multidirectional gradient feature is calculated and represented as symbolic data object . The gradient value in multiple directions gives the valuable information about the edges or corners of the face images and plays significant role in the task of face classification . In recent works the local directional gradient features are calculated along $0^{ \circ } \,{\text{to}}\, 9 0^{ \circ }$ or $0^{ \circ } \,{\text{to}}\, 1 8 0^{ \circ }$ or $0^{ \circ } \,{\text{to}}\, 3 1 5^{ \circ }$ [16], however there exists many other directional gradient values in an image plane. The proposed work explores new way for the computation of local gradient features in multiple directions in the empirically chosen range. The work also proposes new content-based symbolic similarity analysis methodology for recognition of test object. The symbolic object in the trained knowledge base which yields maximum similarity is considered as identified person face class. The proposed method is tested on freely available AR [17] database that consists of face images which vary in expression, illumination, and have occlusions. The methodology outperforms and produces an average recognition accuracy rate of 93.50%.

The paper is organized into six sections. Section 2 presents an overview of the proposed approach, the technique used for feature extraction and symbolic representation is described in Sects. 3 and 4 describes the symbolic similarity analysis used for classification , further experimental results and discussions are illustrated in Sect. 5. Section 6 concludes the work and enlists the future directions.

2 Symbolic Data Modeling Approach to Face Recognition

A new approach to face recognition using symbolic modeling of multidirectional gradient features is proposed in this work. Each person class of face images is represented as a symbolic face (object) i.e. face features of images belonging to a person(class) are represented by a face object of quantitative interval-valued variables and a vector of symbolic objects. Further, a new symbolic similarity measure is devised and used for classification of test image which is also represented as a symbolic object. The different steps involved in the proposed system are illustrated in Fig. 1.

Fig. 1

Overview of symbolic data modeling approach to face recognition

Face detection and preprocessing: In this phase, all the training and testing face samples are cropped by detecting the face using Viola-Jones algorithm [18]. All the cropped face images are down sampled to 64 × 64 pixels and converted into grayscale images.

Symbolic Modeling and Generation of symbolic knowledge base: This phase is carried out in two stages namely feature extraction and symbolic data modeling of face features as symbolic data objects . In feature extraction stage the sum of cardinal gradient vector magnitude is computed from each class of face images and represented as symbolic data object which is used as symbolic knowledge bases for the development of symbolic data modeling approach to face recognition .

Similarity Analysis: Using content-based similarity measure , symbolic similarity between symbolic model of training samples and symbolic data model of test face images are computed. Further, index of the most similar symbolic data object is found, i.e., symbolic data object which represents the maximum similarity is found and the result is output.

The overall process of face recognition through symbolic data modeling can be described as follows:

The face database consists of N number of classes (persons) and each class have M number of samples (each person having M number of sample images). Then, the proposed methodology involves cropping of detected face part from the original face image, extraction of face features and representation of these face features as symbolic knowledge base and symbolic vector. Further symbolic data analysis of trained face objects (symbolic knowledge base and symbolic vector) with features of test image + represented as symbolic object is computed. Further, the test object which is most similar to the person class object that belongs to the symbolic knowledge base is identified and output as the recognized face class. The different phases involved in the proposed model are shown in Fig. 2.

Fig. 2

Phases of symbolic data modeling approach to face recognition

3 Symbolic Modeling

This section describes symbolic modeling process, which includes feature extraction and representation of face features as Symbolic objects.

3.1 Features for Symbolic Representation

In the work, directional gradient magnitude features of a grayscale face image are considered for face recognition. The work proposes a method that uses local correlation between neighbouring pixels (elements) in an image block for the calculation of sum of cardinal gradient vector magnitude. The method utilizes second-order statistics, such as spatial and orientational correlation in horizontal and vertical directions. The major steps involved in the process are; computing image gradient, magnitude, edge orientation, and calculating sum of cardinal gradient vector magnitude, i.e., GC. The process involved in the computation is shown in Fig. 3.

Fig. 3

Steps in computation of GC

Let Ѱ (x, y) be a function that represents gray scale image of size n × n. Then, the gradient of a function of two variables Ѱ (x, y) in horizontal directions can be defined as:

$$h_{xy} = \frac{\partial f}{\partial x}$$

(1)

Equation (1) represents the difference in x values (horizontal direction).

In general, gradient change in horizontal direction at xyth pixel indicates change in intensity in x direction. The horizontal gradient change for all pixels of the image can be represented as in Eq. (2):

$$H = [h_{xy} ]\,\forall \,{\text{x}} = 1\,{\text{to}}\,{\text{n}} - 1\,{\text{and}}\,\forall \,{\text{y}} = 1\,{\text{to}}\,{\text{n}} - 1$$

(2)

The gradient of a function of two variables Ѱ (x, y) in vertical directions indicates change in intensity of y variable and can be defined as:

$$v_{xy} = \frac{\partial f}{\partial y}$$

(3)

Equation (3) represents the difference in y values (vertical direction).

In general, gradient change in vertical direction at xyth pixel indicates change in intensity in y direction. The vertical gradient change for all pixels of the image can be represented as in Eq. (4):

$$V = [v_{xy} ]\,\forall \,{\text{x}} = 1\,{\text{to}}\,{\text{n}} - 1\,{\text{and}}\,\forall \,{\text{y}} = 1\,{\text{to}}\,{\text{n}} - 1$$

(4)

Now, using the Eqs. (1) and (3) cardinal gradient magnitude mag at xyth pixel can be computed as:

$$mag_{xy} = \sqrt {v_{xy}^{2} + h_{xy}^{2} } \,\forall \,{\text{x}} = 1\,{\text{to}}\,{\text{n}} - 1\,{\text{and}}\,\forall \,{\text{y}} = 1\,{\text{to}}\,{\text{n}} - 1$$

(5)

In general, cardinal gradient matrix of image Ѱ (x, y) can be represented by MAG and is computed as in the Eq. (6)

$$MAG = [mag_{xy} ]\,\forall \,{\text{x}} = 1\,{\text{to}}\,{\text{n}} - 1\,{\text{and}}\,\forall \,{\text{y}} = 1\,{\text{to}}\,{\text{n}} - 1$$

(6)

Further, using the Eqs. (1) and (3) gradient orientation $\theta xy$ at xyth pixel can be computed as:

$$\theta_{xy} = { \tan }^{ - } \left( {h_{xy} ,v_{xy} } \right)\,\forall {\text{x}} = 1\,{\text{to}}\,{\text{n}} - 1\,{\text{and}}\,\forall \,{\text{y}} = 1\,{\text{to}}\,{\text{n}} - 1$$

(7)

In general, gradient orientation matrix $\Theta$ of image Ѱ can be represented as

$$\Theta = [\theta_{xy} ]\,\forall \,{\text{x}} = 1\,{\text{to}}\,{\text{n}} - 1\,{\text{and}}\,\forall \,{\text{y}} = 1\,{\text{to}}\,{\text{n}} - 1$$

(8)

Further using the gradient orientation matrix $\Theta$; the binary pattern matrix at xyth pixel in kth direction can be computed as described in Eq. (9)

$$bpxy^{{(k)}} = \left\{ \begin{gathered} 1{\text{ if}} \theta xy > = \alpha {\text{ and }}\theta xy{\text{ < }}\beta \hfill \\ 0{\text{ otherwise}} \hfill \\ \end{gathered} \right.{\mkern 1mu} \forall {\mkern 1mu} {\text{x}} = 1{\mkern 1mu} {\text{to}}{\mkern 1mu} {\text{n}} - 1{\mkern 1mu} {\text{and}}{\mkern 1mu} \forall {\mkern 1mu} {\text{y}} = 1{\mkern 1mu} {\text{to}}{\mkern 1mu} {\text{n}} - 1{\text{ }}$$

(9)

In the Eq. (9) $\alpha {\text{ and }}\beta$ represents empirically chosen specified range of directional values of a image block, k = 1 to 4 represents the four image blocks in four directions namely left, top, right and bottom directions of a plane (Fig. 4).

Fig. 4

Directional computation of GC

The content of the face image block may vary due to illumination, expression, and partial occlusion in four directions (i.e., left, right, top, and bottom of an image plane), i.e., the presence of edges and corners shows higher response values in some particular directions [3]. From the literature survey it is also found that, for the calculation of local gradient of an image the only considered gradient directions are in the range of $0^\circ {\text{ to 90}}^\circ$ or $0^\circ {\text{ to 180}}^\circ$ or $0^\circ {\text{ to 315}}^\circ$ [16]; however, there exist many other ways to calculate multidirectional gradients of an image to accumulate more gradient information which is essential for the development of robust and efficient face recognition system. Hence in the proposed model in order to compute total change in all four directions of a face image block, the range of θ (0, 2 π) is divided into four equal blocks. Figure 4a, b illustrate the range of values of $\alpha {\text{ and }}\beta$ in four directions and are listed in Eq. (10)

$$\left. {\begin{array}{*{20}l} {\alpha = \{ - 7.5^\circ ,82.5^\circ ,172.5^\circ , - 97.5^\circ \} } \hfill \\ {\beta = \{ 7.5^\circ ,97.5^\circ , - 172.5^\circ , - 82.5^\circ \} } \hfill \\ \end{array} } \right\}$$

(10)

In this work of face recognition after thorough experiments the range of $\theta$ (0, 2 π) is divided into k directions using empirically chosen range of values for $\alpha \,{\text{and}}\,\beta$. It is found that four equal division of $\theta$ (0, 2 π) with the given range of $\alpha \,{\text{and}}\,\beta$ as shown in Fig. 4 a, b gives the better performance. Hence in the proposed work four equal division of $15^\circ$ degree in the four directions as shown in the Fig. 4a, b is employed.

In general, prominent binary pattern matrix in kth direction can be computed as in Eq. (11).

$$BP^{(k)} = [bp_{xy}^{(k)} ]\,\forall \,{\text{x}} = 1\,{\text{to}}\,{\text{n}} - 1,{\text{y}} = 1\,{\text{to}}\,{\text{n}} - 1\,{\text{and}}\,{\text{k}} = 1\,{\text{to}}\, 4$$

(11)

Now the full binary pattern matrix in all four directions of a image plane can be computed by logically OR ing in the four matrices BP ¹, BP ², BP ³ and BP ⁴ as in Eq. (12).

$$FBP = BP^{1} \vee BP^{2} \vee BP^{3} \vee BP^{4}$$

(12)

where $\vee$ represent logical OR operation.

Further spatial convolution of $FBP$ and MAG can be computed and represent the directional gradient matrix DG as in Eq. (13).

$$DG = \left( {FBP.*MAG} \right)$$

(13)

In general, the sum of cardinal gradient vector magnitude (GC) of a face image Ѱ in four directions can be computed as in Eq. (14).

$$GC(\Psi ) = \sum\limits_{x = 1}^{n - 1} {\sum\limits_{y = 1}^{n - 1} {DG} }$$

(14)

For each element of image block, the proposed work considered all the neighbouring elements and computed directional gradients using the Eq. (14). The main purpose of calculating GC is to obtain total edge weight of different edge information by considering gradients in appropriate directions. More edge information reflects higher variation [3, 8, 10] of the face image. So, the calculation of sum of cardinal gradient magnitudes is supposed to play an important role in the task of face image classification in the presence of variation in expression, illumination, and occlusion. Such variations of face images are found in freely available AR database. The proposed work uses GC as a feature descriptor and is employed in symbolic data modeling of face image classes which is described in Sect. 3.2.

3.2 Symbolic Data Modeling of Face Image

Let the face image database consists of N number of face image classes and each class of face image consists of M number of face images that vary in expression, occlusion, and illuminations (Images are drawn from AR database).

Let Ω = {Ѱ _1, Ѱ _2, Ѱ ₃ …. Ѱ _M} be a set that represents M images of a face class, which are first-order objects. In general, all face image classes can be represented as

$${\text{U}} = \left\{ {\Omega _{\text{i}} } \right\}\,\forall \,{\text{i}} = 1\,{\text{to}}\,{\text{N}}$$

(15)

The vector representing the symbolic data model of face images belonging to the ith class SV⁽ⁱ⁾ is represented in Eq. (16)

$${\text{SV}}^{{({\text{i}})}} = \{ GC(\varPsi_{1}^{i} ),GC(\varPsi_{2}^{i} ), \ldots ,GC(\varPsi_{M}^{i} )\} \forall \,{\text{i}} = 1\,{\text{to}}\,{\text{N}}$$

(16)

Further, the symbolic interval data representation of ith class using maximum and minimum value of SV ⁽ⁱ⁾ is computed as:

$${\text{GC}} - {\text{MM}}^{(i)} = \{ max(SV^{{({\text{i}})}} ),min(SV^{{({\text{i}})}} )\} \,{\text{where}}\,\forall \,{\text{i}} = 1\,{\text{to}}\,{\text{N}}$$

(17)

Now using the Eq. (17) symbolic knowledge base of N classes is represented as follows:

$$SKB = \{ {\text{GC}} - {\text{MM}}^{(i)} \} \,\forall \,{\text{i}} = 1\,{\text{to}}\,{\text{N}}$$

(18)

The matrix SKB represent symbolic knowledge base of the all face classes. The symbolic similarity analysis can be done by considering SKB, SV of trained objects and test symbolic object.

4 Symbolic Similarity Analysis

The symbolic features can be classified into quantitative, qualitative, and structured variables depending upon nature of data they represent. The similarity analysis of such components can be carried out employing position, span, and content [13] measures. The choice of similarity measure by position, span, and content depends upon type of symbolic features considered for representing the object. In this work, each symbolic object is represented using quantitative interval data variable representing the person face image (Eq. 17) and symbolic vector class (Eq. 16). Hence best choice of symbolic similarity measurement is content similarity. The symbolic similarity measure used in this work is a modified form the content-based symbolic similarity measurement proposed in the earlier works namely [13]. The procedure used in this work for employing content-based symbolic similarity measurement is described as follows:

Let

Tobj = {GC} be a GC feature extracted from the test face sample Ѱ.

SO ⁽ⁱ⁾ be the ith symbolic object in the knowledge base representation SKB/SV⁽ⁱ⁾.

Then, the similarity measure S between two symbolic objects Tobj and an object in symbolic knowledge base represented by SKB/SV ⁱ.

Now consider: SO ⁽ⁱ⁾ ϵ SKB/SV ⁽ⁱ⁾ and Tobj are two qualitative interval type feature vectors, $\forall \, i = 1\, {\text{to}} \,N$.

Let

al =:

lower limit of interval GC-MM ⁽ⁱ⁾(min value), belonging to SO ⁽ⁱ⁾

au =:

upper limit of interval GC-MM ⁽ⁱ⁾(max value), belonging to SO ⁽ⁱ⁾

bl =:

lower limit of interval Tobj (min value)

bu =:

upper limit of interval Tobj (max value)

In test object Tobj both upper bound and lower bound are equal, i.e., bl == bu

Let inters = number of common elements between SV ⁽ⁱ⁾ and Tobj

l _s  = span length of GC-MM ⁽ⁱ⁾ and Tobj

= | max (au, bu) − al|

The M + 2 symbolic features of SO can be represented as:

$$SO^{(i)} = \{ SV^{(i)} ,GC - MM^{(i)} \}$$

(19)

where SV ⁽ⁱ⁾ is 1 × M vector of GC(Ѱ _j ⁱ ) values, $\forall \,{\text{i}} = 1\,{\text{to}}\,{\text{N}}\,{\text{and}}\,\forall \,{\text{j}} = 1\,{\text{to}}\,{\text{M}} .$

Then qualitative interval type similarity component due to content between SO ⁽ⁱ⁾ and Tobj features can be defined as:

$$S^{(i)} \left( {SO^{(i)} ,Tobj} \right) = inters^{(i)} /l_{s}$$

(20)

where inters ⁽ⁱ⁾ is computed using overlap measure method. Overlap measure assigns a similarity of 1 if the values are approximately equal otherwise assigns similarity as 0. Hence, Overlap measure between SO ⁽ⁱ⁾ and Tobj at i th class can be computed as:

$${\text{inters}}^{(i)} = \sum\limits_{j = 1}^{M} {CMP(SV_{j}^{i} ,Tobj)\,\forall \,{\text{i}} = 1\,{\text{to}}\,{\text{N}}}$$

(21)

where $CMP(SV_{j}^{i} ,Tobj) = \left\{ {\begin{array}{*{20}l} 1 \hfill & {{\text{if}}\,GC(\Psi _{j}^{i} ) \cong GC(Tobj)} \hfill \\ 0 \hfill & {\text{otherwise}} \hfill \\ \end{array} } \right.$ and $\forall \,{\text{i}} = 1\,{\text{to}}\,{\text{N}}\,{\text{and}}\,\forall \,{\text{j}} = 1\,{\text{to}}\,{\text{M}}$.

Further, the net similarity of test object Tobj with all the N face classes can be represented as a 1 × N vector.

$$SIM = \left[ {S^{(i)} } \right],\forall \,{\text{i}} = 1\,{\text{to}}\,{\text{N}}$$

(22)

The find the identify of person class, class with max similarity is found in Eq. (23)

$${\text{ID - person - class}} = {\text{max}}\left( {SIM} \right)$$

(23)

where max(SIM) returns the index of the identified person class.

The results obtained from the proposed symbolic data modeling for face recognition are discussed in the results and performance analysis section.

5 Results and Performance Analysis

The experiments are conducted using the AR database. The AR database consists of 120 classes of face images, having 26 samples per class. The face images vary in illumination, expression and have partial occlusion and are taken in two sessions. The AR database has been used to evaluate the efficiency and robustness of the proposed algorithm with reference to different variations. The results of experiments show that, the performance of the proposed algorithm is superior and the recognition speed is fast because face symbolic objects are represented with minimum number of features. Hence, there is a minimum computation involved in comparison between test feature vector and trained features vectors represented as symbolic data objects . In the initial stage, face parts are detected using Viola-Jones algorithm [18] and complete face part is cropped. The cropped images are resized into 64 × 64 pixel size and are converted into grayscale image. Some of the images employed for training and testing are shown in Fig. 5.

Fig. 5

Samples of original (a), and corresponding cropped images (b) of AR database

In order to verify the effectiveness of the proposed methodology, 1200 images of 120 persons from AR database is considered to build the image gallery. 1200 face images are partitioned into five unequal subsets, S1 through S5. The subset S1 contains some of images that vary in left light, right light, and both side lights, S2 contains some of the face images that vary in expressions such as neutral expression, smile, anger, and scream. S3 contains face images of persons wearing scarf and sunglass. S4 contains face images of persons wearing scarf in uniform light and in varying light conditions. S5 contains face images of persons wearing sunglass in uniform light and in varying light conditions. The results obtained by experimentation on S1–S5 set are reported in Table 1.

Table 1 Composition of AR database and recognition results

It is observed that symbolic similarity measurement method with proposed symbolic data modeling technique for face recognition system achieves higher average recognition rate of 93.50% (Table 1).

The implementation of proposed model is carried out in Matlab R2013b platform on PC with Intel®B960 processor, 2 GB DDR3 memory. The total time taken by the proposed methodology to construct symbolic knowledge base and symbolic vector is 83.489 s. The average execution time for recognition is 0.108 s per test face images is obtained.

6 Conclusion

In this paper, a new approach for face recognition that uses directional gradients features between neighbouring pixels is presented. For each face image a sum of cardinal gradient is calculated by integrating more optimal number of directional gradients. The proposed work accumulates more edge information of an image, which achieves more efficient and robust image similarity within the class and differences amongst different classes of face images. The method employs symbolic modeling to represent the face image classes and this has minimized the number of features required for representation. The experiments show that the work is efficient and robust and accurately handles.