Abstract
Facial expression recognition is one of the most active areas of research in computer vision since one of the non-verbal communication methods by which one understands the mood/mental state of a person is the expression of face. Thus, it has been used in various fields such as human-robot interaction, security, computer graphics animation, and ambient assistance. Nevertheless, it remains a challenging task since existing approaches lack generalizability and almost all studies ignore the effects of facial attributes, such as age, on expression recognition even though the research indicates that facial expression manifestation varies with age. Recently, a lot of progress has been made in this topic and great improvements in classification task were achieved with the emergence of deep learning methods. Such approaches have shown how hierarchies of features can be directly learned from original data, thus avoiding classical hand designed feature extraction methods that generally rely on manual operations with labelled data. However, research papers systematically exploring the performance of existing deep architectures for the task of classifying expression of ageing adults are absent in the literature. In the present work a tentative to try this gap is done considering the performance of three recent deep convolutional neural networks models (VGG-16, AlexNet and GoogLeNet/Inception V1) and evaluating it on four different benchmark datasets (FACES, Lifespan, CIFE, and FER2013 ) which also contain facial expressions performed by elderly subjects. As the baseline, and with the aim of making a comparison, two traditional machine learning approaches based on handcrafted features extraction process are evaluated on the same datasets. Carrying out an exhaustive and rigorous experimentation focused on the concept of “transfer learning”, which consists of replacing the output level of the deep architectures considered with new output levels appropriate to the number of classes (facial expressions), and training three different classifiers (i.e., Random Forest, Support Vector Machine and Linear Regression), VGG-16 deep architecture in combination with Random Forest classifier was found to be the best in terms of accuracy for each dataset and for each considered age-group. Moreover, the experimentation stage showed that the deep learning approach significantly improves the baseline approaches considered, and the most noticeable improvement was obtained when considering facial expressions of ageing adults.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
Zeng Z, Pantic M, Roisman G I, Huang T S. A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2009, 31(1): 39-58.
Pantic M, Rothkrantz L J M. Automatic analysis of facial expressions: The state of the art. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000, 22(12): 1424-1445.
Fasel B, Luettin J. Automatic facial expression analysis: A survey. Pattern Recognition, 2003, 36(1): 259-275.
Carroll J M, Russell J A. Do facial expressions signal specific emotions? Judging emotion from the face in context. Journal of Personality and Social Psychology, 1996, 70(2): 205-218.
Rolls E T, Ekman P, Perrett D I et al. Facial expressions of emotion: An old controversy and new findings: Discussion. RSPTB, 335(1273): 69.
Shbib R, Zhou S. Facial expression analysis using active shape model. International Journal of Signal Processing, Image Processing and Pattern Recognition, 2015, 8(1): 9-22.
Cheon Y, Kim D. Natural facial expression recognition using differential-AAM and manifold learning. Pattern Recognition, 2009, 42(7): 1340-1350.
Soyel H, Demirel H. Facial expression recognition based on discriminative scale invariant feature transform. Electronics Letters, 2010, 46(5): 343-345.
Gu W, Xiang C, Venkatesh Y V, Huang D, Lin H. Facial expression recognition using radial encoding of local Gabor features and classifier synthesis. Pattern Recognition, 2012, 45(1): 80-91.
Shan C, Gong S, McOwan P W. Facial expression recognition based on local binary patterns: A comprehensive study. Image and Vision Computing, 2009, 27(6): 803-816.
Chen J, Chen Z, Chi Z, Fu H. Facial expression recognition based on facial components detection and HOG features. In Proc. the Scientific Cooperations International Workshops on Electrical and Computer Engineering Subfields, Aug. 2014, pp.884-888.
Guo G, Guo R, Li X. Facial expression recognition influenced by human ageing. IEEE Transactions on Affective Computing, 2013, 4(3): 291-298.
Wang S, Wu S, Gao Z, Ji Q. Facial expression recognition through modeling age-related spatial patterns. Multimedia Tools and Applications, 2016, 75(7): 3937-3954.
Malatesta C Z, Izard C E. The facial expression of emotion: Young, middle-aged, and older adult expressions. In Emotion in Adult Development, Malatesta C Z, Izard C E (eds.), Sage Publications, 1984, pp.253-273.
Malatesta-Magai C, Jonas R, Shepard B, Culver L C. Type A behavior pattern and emotion expression in younger and older adults. Psychology and Aging, 1992, 7(4): 551-561.
Malatesta C Z, Fiore M J, Messina J J. Affect, personality, and facial expressive characteristics of older people. Psychology and Aging, 1987, 2(1): 64-69.
Lozano-Monasor E, López M T, Vigo-Bustos F, Fernández- Caballero A. Facial expression recognition in ageing adults: From lab to ambient assisted living. Journal of Ambient Intelligence and Humanized Computing, 2017, 8(4): 567-578.
LeCun Y, Bengio Y, Hinton G. Deep learning. Nature, 2015, 521(7553): 436-444.
Yu D, Deng L. Deep learning and its applications to signal and information processing [Exploratory DSP]. IEEE Signal Processing Magazine, 2011, 28(1): 145-154.
Li S, Deng W. Deep facial expression recognition: A survey. arXiv:1804.08348, 2018. https://arxiv.org/abs/1804.08348, Dec. 2019.
Ginne R, Jariwala K. Facial expression recognition using CNN: A survey. International Journal of Advances in Electronics and Computer Science, 2018, 5(3): 13-16.
Goodfellow I J, Erhan D, Carrier P L et al. Challenges in representation learning: A report on three machine learning contests. In Proc. the 20th International Conference on Neural Information Processing, Nov. 2013, pp.117-124.
Kahou S E, Pal C, Bouthillier X et al. Combining modality specific deep neural networks for emotion recognition in video. In Proc. the 15th ACM on International Conference on Multimodal Interaction, Dec. 2013, pp.543-550.
Liu M, Wang R, Li S, Shan S, Huang Z, Chen X. Combining multiple kernel methods on Riemannian manifold for emotion recognition in the wild. In Proc. the 16th International Conference on Multimodal Interaction, Nov. 2014, pp.494-501.
Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556, 2014. https://arxiv.org/abs/1409.1556, Dec. 2019.
Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks. In Proc. the 26th Annual Conference on Neural Information Processing Systems, Dec. 2012, pp.1106-1114.
Szegedy C, Liu W, Jia Y et al. Going deeper with convolutions. In Proc. the 2015 IEEE Conference on Computer Vision and Pattern Recognition, June 2015, pp.1-9.
Viola P, Jones M J. Robust real-time face detection. International Journal of Computer Vision, 2004, 57(2): 137-154.
Zuiderveld K. Contrast limited adaptive histogram equalization. In Graphics Gems IV, Heckbert P S (ed.), Academic Press Professional, 1994, pp.474-485.
Hubel D H, Wiesel T N. Receptive fields and functional architecture of monkey striate cortex. The Journal of Physiology, 1968, 195(1): 215-243.
Pan S J, Yang Q. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 2010, 22(10): 1345-1359.
Russakovsky O, Deng J, Su H et al. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 2015, 115(3): 211-252.
Lin M, Chen Q, Yan S. Network in network. arXiv:1312.4400, 2013. https://arxiv.org/abs/1312.4400, Dec. 2019.
Milborrow S, Nicolls F. Active shape models with SIFT descriptors and MARS. In Proc. the 9th International Conference on Computer Vision Theory and Applications, Jan. 2014, pp.380-387.
Shan C, Gong S, McOwan P W. Facial expression recognition based on local binary patterns: A comprehensive study. Image and Vision Computing, 2009, 27(6): 803-816.
Ebner N. C, Riediger M, Lindenberger U. FACES — A database of facial expressions in young, middle-aged, and older women and men: Development and validation. Behavior Research Methods, 2010, 42(1): 351-362.
Minear M, Park D C. A lifespan database of adult facial stimuli. Behavior Research Methods, Instruments, & Computers, 2004, 36(4): 630-633.
Li W, Li M, Su Z, Zhu Z. A deep-learning approach to facial expression recognition with candid images. In Proc. the 14th IAPR International Conference on Machine Vision Applications, May 2015, pp.279-282.
Goodfellow I J, Erhan D, Carrier P L et al. Challenges in representation learning: A report on three machine learning contests. In Proc. the 20th International Conference on Neural Information Processing, Nov. 2013, pp.117-124.
Wu T, Turaga P, Chellappa R. Age estimation and face verification across ageing using landmarks. IEEE Transactions on Information Forensics and Security, 2012, 7(6): 1780-1788.
Giannopoulos P, Perikos I, Hatzilygeroudis I. Deep learning approaches for facial emotion recognition: A case study on FER-2013. In Advances in Hybridization of Intelligent Methods: Models, Systems and Applications, Hatzilygeroudis I, Palade V (eds.), Springer, 2018, pp.1-16.
Georgescu M I, Ionescu R T, Popescu M. Local learning with deep and handcrafted features for facial expression recognition. arXiv:1804.10892, 2018. https://arxiv.org/pdf/1804.10892.pdf, Dec. 2019.
Abadi M, Barham P, Chen J et al. TensorFlow: A system for large-scale machine learning. In Proc. the 12th USENIX Symposium on Operating Systems Design and Implementation, Nov. 2016, pp.265-283.
Caroppo A, Leone A, Siciliano P. Facial expression recognition in ageing adults: A comparative study. In Ambient Assisted Living, Leone A, Caroppo A, Rescio G et al. (eds.), Springer, 2018, pp.349-359.
Li W, Tsangouri C, Abtahi F, Zhu Z. A recursive framework for expression recognition: From web images to deep models to game dataset. Machine Vision and Applications, 2018, 29(3): 489-502.
Wang X, Wang X, Ni Y. Unsupervised domain adaptation for facial expression recognition using generative adversarial networks. Computational Intelligence and Neuroscience, 2018, Article No. 7208794.
Ionescu R T, Popescu M, Grozea C. Local learning to improve bag of visual words model for facial expression recognition. In Proc. the 2013 ICML Workshop on Challenges in Representation Learning, June 2013.
Benitez-Quiroz C F, Srinivasan R, Feng Q, Wang Y, Martinez A M. EmotioNet challenge: Recognition of facial expressions of emotion in the wild. arXiv:1703.01210, 2017. https://arxiv.org/abs/1703.01210, Dec. 2019.
Mollahosseini A, Hasani B, Mahoor M H. AffectNet: A database for facial expression, valence, and arousal computing in the wild. IEEE Transactions on Affective Computing, 2019, 10(1): 18-31
He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In Proc. the 2016 IEEE Conference on Computer Vision and Pattern Recognition, June 2016, pp.770-778.
Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z. Rethinking the inception architecture for computer vision. In Proc. the 2016 IEEE Conference on Computer Vision and Pattern Recognition, June 2016, pp.2818-2826.
Szegedy C, Ioffe S, Vanhoucke V, Alemi A A. Inception-v4, Inception-ResNet and the impact of residual connections on learning. In Proc. the 31st AAAI Conference on Artificial Intelligence, February 2017, pp.4278-4284.
Chollet F. Xception: Deep learning with depthwise separable convolutions. In Proc. the 2017 IEEE Conference on Computer Vision and Pattern Recognition, July 2017, pp.1800-1827.
Author information
Authors and Affiliations
Corresponding author
Electronic supplementary material
ESM 1
(PDF 1681 kb)
Rights and permissions
About this article
Cite this article
Caroppo, A., Leone, A. & Siciliano, P. Comparison Between Deep Learning Models and Traditional Machine Learning Approaches for Facial Expression Recognition in Ageing Adults. J. Comput. Sci. Technol. 35, 1127–1146 (2020). https://doi.org/10.1007/s11390-020-9665-4
Received:
Revised:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11390-020-9665-4