1 Introduction

With the development of the science and technology, biometrics technology is applied widely. Comparing with the finger print and iris recognition, face recognition has better application prospects because it is contactless and not compulsive. Feature extraction is a key step in face recognition. Face feature extraction approaches can be categorized into two classes. The first class is to find reliable features against realistic conditions by combining several visual features in a pre-defined way; the second class is to learn a metric from training data to enhance strong inter-class differences and intra-class similarities [11, 18]. Subspace feature extraction method which can be roughly divided into linear subspace method and nonlinear subspace method is used commonly. The linear subspace methods mainly include the principal component analysis [14], the linear discriminant analysis [2], the singular value decomposition [6] and the independent components analysis [1] methods. The nonlinear subspace methods mainly include the kernel principal component analysis [13], the kernel fisher discriminant analysis [9] and the manifold learning method [12], the multi-task linear discriminant analysis [19], among which the manifold learning method is proved to present face image information better. He et al. [5] proposed Locality Preserving Projection (LPP) algorithm which is a linear approximation of Laplace eigenmap. The LPP approach can reflect the nonlinear manifold character of the high dimensional data [23], but it cannot preserve the space structure information of the Two-Dimensional (2D) face image because it needs to transform the 2D face image matrix into a dimensional (1D) vector. It is known that not only the differences occurred in eyes, nose and mouth areas between faces play a key role in determining the identity of the person, but also their relative positions are very important too. The 2DPCA, and 2DLDA and their improved method [4, 16] have been proposed in recent years. However, they cannot still solve the problem of small samples. The 2DLPP algorithm proposed by Chen et al. [3] could deal with the 2D face image directly since it considered the space correlation of the face image and reduced the dimension effectively while maintaining the structure information of a face image. Zhang et al. [20] directly used the 2DLPP approach to solve the problems with small samples. Zhi et al. [21] directly used the 2DLPP approach to solve the problems with different facial expressions. Wang et al. [15] proposed the bilateral 2DLPPs approach and achieved some good results. However, these methods extracted the features of face image using the 2DLPP method directly and did not consider the characters of the face image. Lu et al. [8] analyzed the Discrete Cosine Transform (DCT) features of the face image and proposed a face recognition approach based on the DCT low frequency information of the face image and the 2DLPP algorithm. Li et al. [7] proposed the face recognition using wavelet and 2DLPP. Wang et al. [17] used wavelet and improved 2DPCA to recognize face. However, since the DCT and wavelet transforms are performed to the whole image in these methods, it destroyed the relative position information of the original face image and weakened the recognition effect using the 2DLPP method to some extent. The Local Binary Pattern (LBP) operator proposed by Ojala et al. [10] can be used to extract the local texture information of a face image directly, which can preserve the space structure of the face image well at the same time.

Motivated by above analysis, a face recognition method based on LBP and 2DLPP is proposed, which can enhance the local texture information and preserve the relative spatial structure simultaneously. First, the LBP mapping features of the face image are extracted as the elementary features to enhance the main information of face image and reduce the effects of the illumination variation. Then the LBP features are processed further using the 2DLPP method to obtain the final 2D Laplace features, which can preserve the spatial information. At last, the face recognition is performed using the Nearest Neighborhood (NN) method.

The rest of this paper is organized as follows. In Section 2, we give a review of the LBP and 2DLPP face feature extraction approaches. The proposed detailed face recognition method is given in Section 3. In Section 4, we do some experiments on different benchmark datasets and analyze the results. Finally, some conclusions are concluded in Section 5.

2 Proposed face feature extraction

2.1 LBP elementary feature extraction

The basic idea of LBP operator [10] is as follows. A pixel point (xc, yc) in an m × n image is taken as a central point of a 3 × 3 window, where 1 < xc < m, 1 < yc < n, as shown in Fig. 1a. The pixel value of the central point (xc, yc) is gc, the values of the adjacent eight pixels are g0-g7. The texture value T of the central pixel can be described as a function of its adjacent pixel values.

$$ T\left({x}_c,{y}_c\right)=t\left({g}_c,{g}_0,\cdots, {g}_7\right) $$
(1)
Fig. 1
figure 1

A basic LBP operator process

Full size image

Every adjacent pixel value is compared with the central pixel value. The result is denoted as 1 if gi > gc, 0 ≤ i ≤ 7, else 0. Then the expression of the texture value T can be described as follows.

$$ T\left({x}_c,{y}_c\right)=t\left(s\left({g}_0-{g}_c\right),\cdots, s\left({g}_7-{g}_c\right)\right) $$
(2)

where \( s(x)=\Big\{{\displaystyle \begin{array}{c}1\kern0.5em x>0\\ {}0\kern0.5em x\le 0\end{array}} \). Thus, we can obtain eight binary values for all adjacent points. Let us take the window in the Fig. 1b as an example. The eight binary results are shown in Fig. 1c. The LBP value of the central pixel in the window can be computed by transforming the eight binary values lined in clockwise (Fig. 1d) to a decade value (Fig. 1e) using the following equation.

$$ T\left({x}_c,{y}_c\right)=\sum \limits_{i=0}^7s\left({g}_i-{g}_c\right){2}^i $$
(3)

We take six face images of a person from the Extended Yale B (EYB) face database (Fig. 2a) as an example and perform the LBP operation. The result is shown in Fig. 2b. For comparing with the Ref. [8], we do 2DDCT as well, as shown in Fig. 2c, in which the gray background is used to present image contents better. From Fig. 2b, we can find that the LBP image can not only reflect texture feature of all typical regions more clearly, but also weaken the feature of some smooth regions comparing with the original image. In addition, the illumination effects can be removed significantly. From Fig. 2c, we can see that the main information of the original images is distributed in their low frequency domains. Although the 2DDCT is used to keep their space characteristics, it is still a global transform. Thus, the space structure of the original image cannot be preserved. Furthermore, we reconstruct the images from their 2DDCT low frequency components (the front 30 × 30) as shown Fig. 2d. It can be seen that they can represent the information of the original images well, but they still contain the illumination effects and cannot decrease the variable illumination effects. Therefore, the LBP feature is chosen as the elementary feature of the face image in this paper.

Fig. 2
figure 2

Original image, its LBP, 2DDCT image and reconstruction image using low frequency components

Full size image

2.2 2DLPP final feature extraction

The LPP approach can effectively reduce the dimension in face recognition. However, it may cause the space information loss of the 2D image data because it needs to transform the 2D face image to 1D vector. For solving this, the 2DLPP approach [3] is proposed, which can preserve the space structure information of the 2D data because it extracts 2D features from the 2D image. The basic idea of the 2DLPP approach is as follows.

Suppose there are N face images Ai ∈ Rm × n(i = 1, 2, ⋯, N) forming training samples set A = {A1, A2, ⋯, AN}. The samples belong to the G classes and the gth (g = 1, 2, ⋯, G) class includes ng samples. The main idea of the 2DLPP algorithm is to obtain the feature set with less dimensions by projecting the training set A to projection matrices L ∈ Rn × r and R ∈ Rm × c using Y = LTAR and preserve the structure characteristics better at the same time. Therefore, the key problem is how to find the optimum projection matrices L and R. For solving such an optimization problem, we define the objective function of 2DLPP as follows.

$$ \min \left(J\left(\mathbf{L},\mathbf{R}\right)\right)=\sum \limits_{i<j}^N{\left\Vert {\mathbf{Y}}_i-{\mathbf{Y}}_j\right\Vert}_F^2{W}_{i,j} $$
(4)

where \( {\left\Vert \ast \right\Vert}_F^2 \) is the square of Frobenius norm. The weighting Wij is defined as

$$ {W}_{i,j}=\Big\{{\displaystyle \begin{array}{cc}\exp \left({\left\Vert {\mathbf{A}}_i-{\mathbf{A}}_j\right\Vert}^2\right)/t,& {\mathbf{A}}_i\ \mathrm{and}\ {\mathbf{A}}_j\mathrm{belong}\ \mathrm{to}\ \mathrm{the}\ \mathrm{same}\ \mathrm{class}\\ {}0,& \mathrm{otherwise}\end{array}} $$
(5)

where t > 0 is a constant. In general, a similarity weighting Wij are put if Ai and Aj belong to the same class, otherwise the weighting is set to 0. The weighting Wij is set to 1 if Ai and Aj are in the same class in this paper.

In general, the projection matrices L and R can be solved using the iteration method [4]. Thus, the 2DLPP features Yi of a face image Ai can be computed according to the following equation.

$$ {\mathbf{Y}}_i={\mathbf{L}}^T{\mathbf{A}}_i\mathbf{R} $$
(6)

where Yi ∈ Rr × c.

3 Proposed face recognition method

The proposed face recognition approach combines the merits of the LBP operation and the 2DLPP method. The LBP operation enhances the face image differences between different persons, and 2DLPP can not only better represent the essential face manifold structure and reduce the dimension of the face features, but also preserve the space structure information of a face image. The block diagram of the proposed approach is shown in Fig. 3. Firstly, the LBP features of the training samples are extracted by performing the LBP operation to the original face image. Then the projection matrices are solved using the iteration algorithm [21]. Thirdly, the 2DLPP features are extracted further to get the final LBP + 2DLPP features of the face images. Finally, the classification is performed for testing face using the NNC. For simplicity, we denoted the proposed method as LBP + 2DLPP. The detailed process of the LBP + 2DLPP approach is as follows.

  1. Step 1:

    Perform LBP operation to every face image Ai with the size of m × n to obtain its LBP pattern image Bi, where 1 ≤ i ≤ N and N is the number of the training samples.

  2. Step 2:

    Solve projection matrices L and R according to the iteration algorithm [4]. The detailed process is as follows.

    1. Step 2.1:

      Set the sizes of the projection matrices L and R are r and c, the maximum iteration number is K, and the objection function error threshold is ε.

    2. Step 2.2:

      Set the initial value of the projection matrix L is L0 = (Ic, 0)T, where Ic is a c × c identity matrix.

    3. Step 2.3:

      Set k = 1. The projected matrix can be computed by the Eq. (6).

Fig. 3
figure 3

The face recognition block diagram based on LBP + 2DLPP

Full size image
$$ {\mathbf{Y}}_i={\mathbf{L}}^T{\mathbf{B}}_i\mathbf{R} $$
(7)
  1. Step 2.4:

    Compute the projection matrix Rk. The Eq. (4) can be written as follows according to Eq. (7).

$$ \min \left(J\left(\mathbf{L},\mathbf{R}\right)\right)=\sum \limits_{i<j}^N{\left\Vert {\mathbf{Y}}_i-{\mathbf{Y}}_j\right\Vert}_F^2{W}_{i,j}=\sum \limits_{i<j}^N{\left\Vert {\mathbf{L}}^T{\mathbf{B}}_i\mathbf{R}-{\mathbf{L}}^T{\mathbf{B}}_j\mathbf{R}\right\Vert}_F^2{W}_{i,j} $$
(8)

where Bi and Bj are the LBP features of training samples. For solving the objection function defined in Eq. (8), two functions are defined as follows referring to [3].

$$ {\mathbf{S}}_{\mathbf{R}1}\left(\mathbf{L}\right)=\sum \limits_{i,j}^N{\left({\mathbf{B}}_i-{\mathbf{B}}_j\right)}^T{\mathbf{L}}_{k-1}{\mathbf{L}}_{k-1}^T\left({\mathbf{B}}_i-{\mathbf{B}}_j\right){W}_{i,j} $$
(9)
$$ {\mathbf{S}}_{\mathbf{R}2}\left(\mathbf{L}\right)=\sum \limits_{i,j}^N{\mathbf{B}}_i^T{\mathbf{L}}_{k-1}{\mathbf{L}}_{k-1}^T{\mathbf{B}}_j{W}_{i,j} $$
(10)

Thus, the question solving the Eq. (8) can be transformed to compute c eigenvectors corresponding to c minimal eigenvalues for

$$ {\mathbf{S}}_{\mathbf{R}1}\left(\mathbf{L}\right)\mathbf{r}={\lambda}_R{\mathbf{S}}_{\mathbf{R}2}\left(\mathbf{L}\right)\mathbf{r} $$
(11)

The c eigenvectors forms the projection matrix Rk = [r1, r2, ⋯, rc].

  1. Step 2.5:

    Compute the projection matrix Lk. Like step 2.4, two functions can be defined as follows for solving Eq. (8).

$$ {\mathbf{S}}_{\mathbf{L}1}\left(\mathbf{R}\right)=\sum \limits_{i,j}^N{\left({\mathbf{B}}_i-{\mathbf{B}}_j\right)}^T{\mathbf{R}}_k{\mathbf{R}}_k^T\left({\mathbf{B}}_i-{\mathbf{B}}_j\right){W}_{i,j} $$
(12)
$$ {\mathbf{S}}_{\mathbf{L}2}\left(\mathbf{R}\right)=\sum \limits_{i,j}^N{\mathbf{B}}_i^T{\mathbf{R}}_k{\mathbf{R}}_k^T{\mathbf{B}}_j{W}_{i,j} $$
(13)

The question solving the Eq. (8) can be transformed to solve r eigenvectors corresponding to r minimal eigenvalues for

$$ {\mathbf{S}}_{\mathbf{L}1}\left(\mathbf{R}\right)\mathbf{l}={\lambda}_L{\mathbf{S}}_{\mathbf{L}2}\left(\mathbf{R}\right)\mathbf{l} $$
(14)

The solved r eigenvectors form the projection matrix Lk = [l1, l2, ⋯, lr].

  1. Step 2.6:

    Compute J(Lk, Rk).

  2. Step 2.7:

    Stop the iteration and go to Step 2.8 if ∣J(Lk, Rk) − J(Lk − 1, Rk − 1) ∣  < ε or k > K, else k = k + 1 and go to Step 2.4.

  3. Step 2.8:

    The projection matrices meeting Eq. (4) are obtained by R = Rk, L = Lk.

  1. Step 3:

    Compute the 2D Laplace features of every face LBP image with the size of r × cYi by projecting its LBP features Bi(1 ≤ i ≤ N) to the projecting matrices L and R according to Eq. (6).

  2. Step 4:

    Compute the features Yj of the testing face image according to step 1 and 3.

  3. Step 5:

    Classify using the NNC. The testing sample can be determined to be the gth class if \( l=\underset{i}{\mathrm{argmin}}d\left({\mathbf{Y}}_j,{\mathbf{Y}}_i\right) \) and the lth sample belongs to the gth class, where d(Yj, Yi) can be computed as follows.

$$ d\left({\mathbf{Y}}_j,{\mathbf{Y}}_i\right)=\sum \limits_{k=1}^c{\left\Vert {\mathbf{y}}_k^{(j)}-{\mathbf{y}}_k^{(i)}\right\Vert}_2 $$
(15)

where \( {\mathbf{y}}_k^{(j)} \) presents the kth column vector of Yj, \( {\mathbf{y}}_k^{(i)} \) presents the kth column vector of Yi.

4 Experimental results and analysis

In this paper, four benchmark datasets are used to evaluate the proposed approach. One dataset is the Yale face database, which includes 165 images of 15 individuals (each person has 11 different images) under various facial expressions, gaits and lighting conditions. The second dataset is the EYB database, which contains 38 people and everyone has 64 images with different facial expressions and illumination conditions. The third dataset is CMU PIE C09 face database, which contains 1632 images in all and 24 images for every person with various facial expressions and lighting conditions. For using the proposed method better, we first do some experiments on how to choose the 2DLPP dimensions. And then some comparison experiments are performed. At last, the overall experimental analysis is given.

4.1 2DLPP feature dimensions analysis

The size of the 2DLPP dimensions is significant to the storage space of the training samples, the training and testing time. Although some experiments are performed [8], more detailed analysis is necessary. Therefore, some experiments on these three databases are designed to find the dimension selection rule in this paper.

4.1.1 The EYB database

We randomly choose 5, 8 and 10 face images for every person as training samples separately and adjust their size to 64 × 64. The iteration number is set to 100, error threshold is 0.001. The dimensions of the 2DLPP changes from 4 × 4 to 43 × 43. Ten serial numbers are chosen randomly from 1 to 64 as the training sample numbers. The front 5, 8 and all of the 52th, 17th, 64th, 58th, 46th, 32nd, 14th, 25th, 55th and 16th face images of every person are taken as training samples separately in this paper. The corresponding training face image examples of the first and second people are shown in Fig. 4. For convenience, the Recognition Accuracies (RA) for every situation are denoted as RA# according to the number of the selected front training sample. Thus, they are denoted as RA5, RA8 and RA10 respectively, as shown in Fig. 5. For observing more clearly, we give their recognition results corresponding to the 2DLPP dimensions from 4 × 4 to 12 × 12 in Table 1. From Fig. 5 and Table 1, we can see that the recognition results are stable when the dimensions are larger than 8 × 8. The proposed method does not get a higher recognition accuracy when the dimensions are taken a larger size, and the training and testing time are increased at the same time.

Fig. 4
figure 4

The corresponding training face image examples from the EYB Database

Full size image
Fig. 5
figure 5

The recognition results on the EYB dataset with the image size 64 × 64

Full size image
Table 1 The recognition results on the EYB dataset with different 2DLPP dimensions
Full size table

For observing the relation between the size of the 2DLPP dimensions and the size of the image further, we resize the original image to 32 × 32 and perform the same experiments. The recognition results are shown in Fig. 6. The recognition results can obtain a better stable status when the dimensions are larger than 7 × 7.

Fig. 6
figure 6

The recognition results on the EYB dataset with the image size 32 × 32

Full size image

4.1.2 The CMU PIE C09 database

We randomly choose 5, 8 and 10 face images for every person as training samples separately and adjust their size to 64 × 64, too. The iteration number, error threshold and the dimensions of the 2DLPP ranges are same with Section 4.1.1. The ten random integers are {1, 19, 16, 6, 22, 2, 17, 23, 11, 4}. The front 5, 8 and the 1st, 19th, 16th, 6th, 22nd, 2nd, 17th, 23rd, 11th and 4th face images of every person are taken as training samples separately in this paper. The corresponding training face image examples of the first and second people are shown in Fig. 7. The RAs are shown in Fig. 8 for these three situations. For observing more clearly, we give the recognition results denoted as RA5, RA8 and RA10 corresponding to the 2DLPP dimensions 4 × 4, 6 × 6, …, 20 × 20 in Table 2. From Fig. 8 and Table 2, we can see that the recognition results can obtain a better stable status when the dimensions are larger than 8 × 8 for every situation and does not get a higher recognition accuracy when the dimensions are taken a larger size.

Fig. 7
figure 7

The corresponding training face image examples from the CMU PIE C09 Database

Full size image
Fig. 8
figure 8

The recognition results on PIE dataset with the image size 64 × 64

Full size image
Table 2 The recognition results on PIE database with different 2DLPP dimensions
Full size table

For observing the relation between the size of the 2DLPP dimensions and the size of the image better, we resize the original image to 32 × 32 and perform the same experiments. The recognition results are shown in Fig. 9. The similar conclusion on the PIE database and the EYB database can be obtained.

Fig. 9
figure 9

The recognition results on PIE dataset with the image size 32 × 32

Full size image

4.1.3 The Yale database

We randomly choose 3, 5 and 8 face images for every person as training samples separately and adjust their size to 64 × 64 because the number of the face images for every person is less than the other datasets. The iteration number, error threshold and the dimensions of the 2DLPP ranges are same with Section 4.1.1. The eight random integers are {11, 10, 1, 7, 3, 2, 4, 9}. The front 3, 5 and all of the 11th, 10th, 1st, 7th, 3rd, 2nd, 4th and 9th face images of every person are taken as training samples separately in this paper. The corresponding training face image examples of the first and second people are shown in Fig. 10. The RAs are shown in Fig. 11 for these three situations, denoted as RA3, RA5 and RA8. From Fig. 11, we can see that the recognition results can obtain a better stable status when the dimensions are larger than 8 × 8 for every situation and doesn’t get higher recognition accuracy when the dimensions are larger.

Fig. 10
figure 10

The corresponding training face image examples from the Yale Database

Full size image
Fig. 11
figure 11

The recognition results on the Yale dataset with the image size 64 × 64

Full size image

For observing the relation between the size of the 2DLPP dimensions and the size of the image better, we resize the original image to 32 × 32 and perform the same experiments. The recognition results are shown in Fig. 12. The similar conclusion on the Yale database with two above databases can be obtained.

Fig. 12
figure 12

The recognition results on the Yale dataset with the image size 32 × 32

Full size image

Overall, the 2DLPP dimensions D can be taken to meet the following condition for m × m original image.

$$ \left\lfloor {\log}_2^m\right\rfloor +2\le D\le 2\left(\left\lfloor {\log}_2^{m\Big)}\right\rfloor +2\right) $$
(16)

The dimensions can be taken as 8 × 8~16 × 16 for the 64 × 64 original image, and 7 × 7~14 × 14 when the size of the original image is 32 × 32.

4.2 Experimental results and analysis using different method

For convenience, the size of the face image is adjusted to 64 × 64 in the following experiments. We randomly choose 3~10 images as training samples for every person on the experimental Yale, EYB, and CMU PIE benchmark datasets separately and the rest as the testing samples. For evaluating the performance of the proposed method, every experiment is performed 10 runs, and the Average Recognition Accuracy (ARA) is calculated as the criterion. The dimensions of the 2DLPP method are set r = c = 14, and the iteration number is set 30 for solving the projection matrices. The dimensions of the 2DDCT method are 30 × 30.

4.2.1 The Yale database

The recognition results using the face recognition method based on 2DLPP [3], 2DDCT, 2DDCT + 2DLPP [8], wavelet + Improved 2DPCA [7] and wavelet + 2DLPP [17] are computed for comparison. The experimental results are shown in Table 3. The proposed approach can recognize the face images better than other methods and has better robustness for various illuminations, facial expressions, gait and with glasses or not.

Table 3 The ARAs on the Yale database (%)
Full size table

4.2.2 The CMU PIE Pose C09 database

The recognition results using the face recognition method based on 2DLPP [3], 2DDCT, 2DDCT + 2DLPP [8], wavelet + Improved 2DPCA [7] and wavelet + 2DLPP [17] are computed for comparison. The experimental results are shown in Table 4. The proposed approach can recognize the face images better than other methods and has better robustness for various illuminations, facial expressions.

Table 4 The ARAs on the CMU PIE Pose C09 database (%)
Full size table

4.2.3 The EYB database

The recognition results using the face recognition method based on 2DLPP [3], 2DDCT, 2DDCT + 2DLPP [8], wavelet + Improved 2DPCA [7], wavelet + 2DLDA, wavelet + 2DLPP [17] and multi-wavelet and sparse (MLL + sparse) [22] are computed to compare, where the GHM multi-wavelet and approximation order prefilter are selected in this paper. The experimental results are shown in Table 5. The proposed approach can recognize the face images more effective than other methods and has better robustness for various illuminations, facial expressions.

Table 5 The ARAs on the EYB database (%)
Full size table

According to Tables 1, 2, 3, 4 and 5 and Figs. 5, 6, 8, 9, 11 and 12, it is proved that the proposed face recognition method based on LBP + 2DLPP in this paper has better recognition results than the DLPP, 2DDCT, 2DDCT + 2DLPP, wavelet + Improved 2DPCA, wavelet + 2DLPP and MLL + sparse methods and has better robustness for various illumination, facial expressions and shields, especially for the small number of training samples. In addition, an effective 2DLPP dimension selection method is given by experiments, which proves the proposed method can effectively decrease the feature dimension.

5 Conclusions

In this paper, an approach to face recognition is proposed by using LBP and 2DLPP, in which LBP enhances the face image detailed features and 2DLPP preserves not only the effective information of a face image with less dimensions, but also the space structure information of a face image. The experimental results on the Yale dataset, the EYB dataset and the CMU PIE C09 dataset show that the proposed methods can effectively achieve better recognition results and has higher robustness than the face recognition methods based on 2DLPP, 2DDCT, 2DDCT + 2DLPP and wavelet + I2DPCA and wavelet + 2DLPP. However the proposed method cannot obtain a high recognition rate for the database with different poses, so our next work is to study the relative method to solve it.