Abstract
Emotions are becoming increasingly important in human-centered interaction architectures. Recognition of facial expressions, which are central to human-computer interactions, seems natural and desirable. However, facial expressions include mixed emotions, continuous rather than discrete, which vary from moment to moment. This paper represents a novel method of recognizing facial expressions of various internal states via manifold learning, to achieve the aim of human-centered interaction studies. A critical review of widely used emotion models is described, then, facial expression features of various internal states via the locally linear embedding (LLE) are extracted. The recognition of facial expressions is created with the pleasure-displeasure and arousal-sleep dimensions in a two-dimensional model of emotion. The recognition result of various internal state expressions that mapped to the embedding space via the LLE algorithm can effectively represent the structural nature of the two-dimensional model of emotion. Therefore our research has established that the relationship between facial expressions of various internal states can be elaborated in the two-dimensional model of emotion, via the locally linear embedding algorithm.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
Nelson C A. The recognition of facial expressions in the first two years of life: Mechanisms of development. Child Development, 1987, 58(4): 889–909.
Bartlett M, Viola P, Sejnowski T, Larsen J, Hager J, Ekman P. Classifying Facial Action. Advances in Neural Information Processing Systems 8. Touretzky D et al. (eds.), Cambridge: MIT Press, MA, 1996.
Essa I, Pentland A. Coding, analysis, interpretation, and recognition of facial expressions. IEEE Trans. Pattern Analysis and Machine Intelligence, 1997, 19(7): 757–763.
Lien J. Automatic recognition of facial expressions using hidden Markov models and estimation of expression intensity [Ph.D. Dissertation]. Carnegie Mellon University, 1998.
Oliver N, Pentland A, Berard F. LAFTER: A real-time face and lips tracker with facial expression recognition. Pattern Recognition, 2000, 33(8): 1369–1382.
Bartlett M. Face Image Analysis by Unsupervised Learning. Kluwer Academic Publishers, 2001.
Cohen I, Sebe N, Garg A, Chen L S, Huang T S. Facial expression recognition from video sequence. In Proc. Int. Conf. Multimedia and Exp (ICME), Lausanne, Switzerland, Aug. 26–29, 2002, pp.121–124.
Yang P, Liu Q, Metaxas D N. Boosting coded dynamic features for facial action units and facial expression recognition. In Proc. CVPR, Minneapolis, USA, June 18–23, 2007, pp.511–518.
Zhao G, Pietikainen M. Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans. Pattern Analysis and Machine Intelligence, 2007, 29(6): 915–928.
Ekman P. Universal and cultural difference in facial expressions of emotions. Nebraska Symposium on Motivation, 1971, Cole J K (ed.), Lincoln: University of Nebraska Press, 1972, 19: 207–283.
Scimmack U. Response latencies of pleasure and displeasure ratings: Further evidence for mixed feeling. Cognition and Emotion, 2005, 19(5): 671–691.
Lang P J. The emotion probe: Studies of motivation and attention. American Psychologist, 1995, 50(5): 372–385.
Russell J A. Evidence of convergent validity on the dimension of affect. J. Personality and Social Psychology, 1978, 30(38): 1152–1168.
Peter C, Herbon A. Emotion representation and physiology assignments in digital systems. Interacting with Computers, 2006, 18(2): 139–170.
Donato G, Bartlett M, Hager J, Ekman P, Sejnowski T. Classifying facial actions. IEEE PAMI, 1999, 21(10): 974–989.
Schmidt K, Cohn J. Dynamics of facial expression: Normative characteristics and individual difference. In Proc. Int. Conf. Multimedia and Expo, Tokyo, Japan, Aug. 22–25, 2001, pp.547–550.
Tong Y, Wang Y, Zhu Z, Ji Q. Robust facial feature tracking under varying face pose and facial expression. Pattern Recognition, 2007, 40(11): 3195–3208.
Kong H, Wang L, Teoh E K, Li X. Generalized 2D principal component analysis for face image representation and recognition. Neural Network, 2005, 18(5/6): 585–594.
Penev P S. Local feature analysis: A statistical theory for information representation and transmission [Ph.D. Dissertation]. Rockefeller University, 1998.
Seung H S, Lee D D. The manifold ways of perception. Science, 2000, 290(12): 2268–2269.
Roweis S T, Saul L K. Nonlinear dimensionality reduction by locally linear embedding. Science, 2000, 290(5500): 2323–2326.
Carrol J M, Russell J A. Do facial expressions signal specific emotions? Judging emotion from the face in context. Journal of Personality and Social Psychology, 1996, 70(2): 205–218.
Russell J A. Is there universal recognition of emotion from facial expression? A review of the cross-cultural studies. Psychological Bulletin, 1994, 115(1): 112–141.
Russell J A. Culture and categorization of emotion. Psychological Bulletin, 1991, 110: 426–450.
Osgood C E, MayWH, Miron M S. Cross-Cultural Universals of Affective Meaning. Urbana: Univ. Illinois Press., 1975.
Russell JA, Lewicka M, Nitt T. A cross-cultural study of a circumplex model of affect. Journal of Personality and Social Psychology, 1989, 57(5): 848–856.
Russell J A, Ridgeway D. Dimension underlying children’s emotion concepts. Developmental Psychology, 1983, 19: 795–804.
Kim Y A, Kim J K, Park S K, Oh K J, Chung C S. The study of dimension of internal states through word analysis about emotion. Korean Journal of the Science of Emotion and Sensibility, 1998, 1(1): 145–152.
Kim J K, Mun H S, Oh K J. Stability of two-dimension structure of emotion. Korean Journal of the Science of Emotion and Sensibility, 1999, 2(1): 43–52.
Bahn S B, Han J H, Chung C S. Facial expression database for mapping facial expression onto internal state. In Proc. Emotion Conf. Korea, Seoul, Korea, Nov. 1997, pp.215–219.
Author information
Authors and Affiliations
Corresponding author
Additional information
This study was supported by research funds from Chosun University, 2008.
Electronic Supplementary Material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Shin, YS. Facial Expression Recognition of Various Internal States via Manifold Learning. J. Comput. Sci. Technol. 24, 745–752 (2009). https://doi.org/10.1007/s11390-009-9257-9
Received:
Revised:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11390-009-9257-9