Abstract
Facial expressions are direct means of communication of human’s emotional state. Hence facial expression recognition (FER) has always been a topic of great interest of researchers specially for smart applications. Numerous approaches have been proposed for FER using Local binary patterns (LBP). This chapter presents an analysis of LBP feature descriptor for FER. State of the art approaches have been discussed that use LBP as their main component. Here, basic LBP operator along with several variants and their main properties are described that have been proved useful for FER. A general framework for FER is described which includes four consecutive modules. These modules are preprocessing, feature extraction, dimensionality reduction and classification. LBP based FER results have been reported on three benchmark datasets JAFFE, CK+ and Yale. Experimentation demonstrates usefulness of LBP in FER.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Bousmalis K, Mehu M, Pantic M (2013) Towards the automatic detection of spontaneous agreement and disagreement based on nonverbal behaviour: a survey of related cues, databases, and tools. Image Vis Comput 31(2):203–221
Bartlett MS, Littlewort G, Frank M, Lainscsek C, Fasel I, Movellan J (2005) Recognizing facial expression: machine learning and application to spontaneous behavior. In: 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05), vol 2, pp 568–573
Tian YL, Kanade T, Cohn JF (2005) Facial expression analysis. Handbook of face recognition. Springer, New York, pp 247–275
Rudovic O, Pantic M, Patras I (2013) Coupled Gaussian processes for pose-invariant facial expression recognition. IEEE Trans Pattern Anal Mach Intell 35(6):1357–1369
Pantic M, Rothkrantz LJM (2000) Automatic analysis of facial expressions: the state of the art. IEEE Trans Pattern Anal Mach Intell 22(12):1424–1445
Bettadapura V (2012) Face expression recognition and analysis: the state of the art. arXiv preprint arXiv:1203.6722, http://arxiv.org/pdf/1203.6722
Fasel B, Luettin J (2003) Automatic facial expression analysis: a survey. Pattern Recogn 36(1):259–275
Ekman P, Friesen WV (1978) Facial action coding system: a technique for the measurement of facial movement. Consulting Psychologists Press, USA
Zhang Z (1999) Feature based facial expression recognition: sensitivity analysis and experiment with a multi-layer perceptron. Int J Pattern Recognit Artif Intell 13(6):893–911
Guo G, Dyer CR (2003) Simultaneous feature selection and classifier training via linear programming: a case study for face expression recognition. In: 2003 IEEE computer society conference on computer vision and pattern recognition (CVPR’03), vol 1, pp 346–352
Valstar M, Pantic M (2006) Fully automatic facial action unit detection and temporal analysis. In: IEEE computer society computer vision and pattern recognition workshop (CVPRW’06), pp 149–149
Valstar MF, Patras I, Pantic M (2005) Facial action unit detection using probabilistic actively learned support vector machines on tracked facial point data. In: IEEE computer society computer vision and pattern recognition workshop (CVPRW’05), vol 3, p 76
Franco L, Treves A (2001) A neural network facial expression recognition system using unsupervised local processing. In: 2nd international symposium on image and signal processing and analysis (ISPA’01), pp 628–632
Uddin MZ, Lee JJ, Kim TS (2009) An enhanced independent component-based human facial expression recognition from video. IEEE Trans Consum Electron 55(4):2216–2224
Lekshmi VP, Sasikumar M (2009) Analysis of facial expression using Gabor and SVM. Int J Recent Trends Eng 1(2):47–50
Owusu E, Zhan Y, Mao QR (2014) A neural-AdaBoost based facial expression recognition system. Expert Syst Appl 41(7):3383–3390
Kumbhar M, Jadhav A, Patil M (2012) Facial expression recognition based on image feature. Int J Comput Commun Eng 1(2):117–119
Shinohara Y, Otsu N (2004) Facial expression recognition using fisher weight maps. In: Sixth IEEE international conference on automatic face and gesture recognition, pp 499–504
Ahmed F, Bari H, Hossain E (2014) Person-independent facial expression recognition based on compound local binary pattern (CLBP). Int Arab J Inf Technol 11(2):195–203
Nigam S, Khare A (2015) Multiscale local binary patterns for facial expression-based human emotion recognition. In: Computational vision and robotics. Springer, New Delhi, pp 71–77
Darwish A, Hassanien AE, Elhoseny M, Sangaiah AK, Muhammad K (2017) The impact of the hybrid platform of internet of things and cloud computing on healthcare systems: opportunities, challenges, and open problems. J Ambient Intell Humanized Comput. First Online: 29 Dec 2017. https://doi.org/10.1007/s12652-017-0659-1
Elhoseny M, Abdelaziz A, Salama A, Riad AM, Sangaiah AK, Muhammad K (2018) A hybrid model of internet of things and cloud computing to manage big data in health services applications. Future Gen Comput Syst. Available online 15 Mar 2018 (In Press). https://doi.org/10.1016/j.future.2018.03.005
Abdelaziza A, Elhoseny M, Salama AS, Riad AM (2018) A machine learning model for improving healthcare services on cloud computing environment. Measurement 119:117–128. https://doi.org/10.1016/j.measurement.2018.01.022
Zhang J, Tan T (2002) Brief review of invariant texture analysis methods. Pattern Recogn 35(3):735–747
Pietikäinen M (2005) Image analysis with local binary patterns. In: Image analysis. Springer, Berlin, pp 115–118
Mäenpää T, Pietikäinen M (2005) Texture analysis with local binary patterns. In: Handbook of pattern recognition and computer vision, vol 3, pp 197–216
Mäenpää T (2003) The Local binary pattern approach to texture analysis: extensions and applications. Doctoral dissertation, University of Oulu
Turtinen M (2007) Learning and recognizing texture characteristics using local binary patterns. Doctoral dissertation, University of Oulu
Ilea DE, Whelan PF (2011) Image segmentation based on the integration of color–texture descriptors—a review. Pattern Recogn 44(10):2479–2501
Nigam S, Khare A (2016) Integration of moment invariants and uniform local binary patterns for human activity recognition in video sequences. Multimedia Tools Appl 75(24):17303–17332
Wang A, Wang S, Lucieer A (2010) Segmentation of multispectral high-resolution satellite imagery based on integrated feature distributions. Int J Remote Sens 31(6):1471–1483
Burçin K, Vasif NV (2011) Down syndrome recognition using local binary patterns and statistical evaluation of the system. Expert Syst Appl 38(7):8690–8695
Nigam S, Khare A (2015) Multiresolution approach for multiple human detection using moments and local binary patterns. Multimedia Tools Appl 74(17):7037–7062
Mattivi R, Shao L (2010) Spatio-temporal dynamic texture descriptors for human motion recognition. In: Intelligent video event analysis and understanding. Springer, Berlin, pp 69–91
Nanni L, Brahnam S, Lumini A (2011) Combining different local binary pattern variants to boost performance. Expert Syst Appl 38(5):6209–6216
Nanni L, Lumini A, Brahnam S (2012) Survey on LBP based texture descriptors for image classification. Expert Syst Appl 39(3):3634–3641
Pietikäinen M, Ojala T, Xu Z (2000) Rotation-invariant texture classification using feature distributions. Pattern Recogn 33(1):43–52
Ojala T, Pietikäinen M, Mäenpää T (2000) Gray scale and rotation invariant texture classification with local binary patterns. In: European conference on computer vision (ECCV’2000). Springer, Berlin, pp 404–420
Zhao G, Ahonen T, Matas J, Pietikainen M (2012) Rotation-invariant image and video description with local binary pattern features. IEEE Trans Image Process 21(4):1465–1477
Guo Z, Zhang L, Zhang D (2010) Rotation invariant texture classification using LBP variance (LBPV) with global matching. Pattern Recogn 43(3):706–719
Ojala T, Pietikainen M, Maenpaa T (2002) Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans Pattern Anal Mach Intell 24(7):971–987
Lahdenoja O, Poikonen J, Laiho M (2013) Towards understanding the formation of uniform local binary patterns, vol 2013, Article ID 429347, 20 p. https://doi.org/10.1155/2013/429347
Chan CH, Kittler J, Messer K (2007) Multi-scale local binary pattern histograms for face recognition. Springer, Berlin, pp 809–818
Liao S, Zhu X, Lei Z, Zhang L, Li SZ (2007) Learning multi-scale block local binary patterns for face recognition. In: Advances in biometrics. Springer, Berlin, pp 828–837
Guo Z, Zhang D (2010) A completed modeling of local binary pattern operator for texture classification. IEEE Trans Image Process 19(6):1657–1663
Heikkilä M, Pietikäinen M, Schmid C (2009) Description of interest regions with local binary patterns. Pattern Recogn 42(3):425–436
Tan X, Triggs B (2010) Enhanced local texture feature sets for face recognition under difficult lighting conditions. IEEE Trans Image Process 19(6):1635–1650
Wang L, He DC (1990) Texture classification using texture spectrum. Pattern Recogn 23(8):905–910
Viola P, Jones MJ (2004) Robust real-time face detection. Int J Comput Vision 57(2):137–154
Elhoseny M, Ramírez-González G, Abu-Elnasr OM, Shawkat SA, Arunkumar N, Farouk A (2018) Secure medical data transmission model for IoT-based healthcare systems. IEEE Access 6:20596–20608. https://doi.org/10.1109/access.2018.2817615
Shehab A, Elhoseny M, Muhammad K, Sangaiah AK, Yang P, Huang H, Hou G (2018) Secure and robust fragile watermarking scheme for medical images. IEEE Access 6:10269–10278. https://doi.org/10.1109/access.2018.2799240
Jolliffe I (2005) Principal component analysis. Wiley, New York
Bengio Y, Courville A, Vincent P (2013) Representation learning: a review and new perspectives. IEEE Trans Pattern Anal Mach Intell 35(8):1798–1828
Gang L, Xiao-hua L, Ji-Liu Z, Xiao-gang G (2009) Geometric feature based facial expression recognition using multiclass support vector machines. In: 2009 IEEE international conference on granular computing (GRC’09), pp 318–321
Burges CJ (1998) A tutorial on support vector machines for pattern recognition. Data Min Knowl Disc 2(2):121–167
Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20(3):273–297
Dailey MN, Joyce C, Lyons MJ, Kamachi M, Ishi H, Gyoba J, Cottrell GW (2010) Evidence and a computational explanation of cultural differences in facial expression recognition. Emotion 10(6):874
Kanade T, Cohn JF, Tian Y (2000) Comprehensive database for facial expression analysis. In: Proceedings of the fourth IEEE international conference on automatic face and gesture recognition, 2000. IEEE, pp 46–53
Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE computer society conference on computer vision and pattern recognition workshops (CVPRW). IEEE, pp 94–101
Yale Facial Expression Database. http://vision.ucsd.edu/content/yale-face-database
Pantic M, Valstar M, Rademaker R, Maat L (2005) Web-based database for facial expression analysis. In: ICME 2005. IEEE international conference on multimedia and expo, 2005. IEEE, 5 pp
Acknowledgements
This work is supported by Science and Engineering Research Board, Department of Science and Technology, Government of India under grant number PDF/2016/003644.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Nigam, S., Singh, R., Misra, A.K. (2019). Local Binary Patterns Based Facial Expression Recognition for Efficient Smart Applications. In: Hassanien, A., Elhoseny, M., Ahmed, S., Singh, A. (eds) Security in Smart Cities: Models, Applications, and Challenges. Lecture Notes in Intelligent Transportation and Infrastructure. Springer, Cham. https://doi.org/10.1007/978-3-030-01560-2_13
Download citation
DOI: https://doi.org/10.1007/978-3-030-01560-2_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-01559-6
Online ISBN: 978-3-030-01560-2
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)