Abstract
For effective interaction between humans and socially adept, intelligent service robots, a key capability required by this class of sociable robots is the successful interpretation of visual data. In addition to crucial techniques like human face detection and recognition, an important next step for enabling intelligence and empathy within social robots is that of emotion recognition. In this paper, an automated and interactive computer vision system is investigated for human facial expression recognition and tracking based on the facial structure features and movement information. Twenty facial features are adopted since they are more informative and prominent for reducing the ambiguity during classification. An unsupervised learning algorithm, distributed locally linear embedding (DLLE), is introduced to recover the inherent properties of scattered data lying on a manifold embedded in high-dimensional input facial images. The selected person-dependent facial expression images in a video are classified using the DLLE. In addition, facial expression motion energy is introduced to describe the facial muscle’s tension during the expressions for person-independent tracking for person-independent recognition. This method takes advantage of the optical flow which tracks the feature points’ movement information. Finally, experimental results show that our approach is able to separate different expressions successfully.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
ISO/IEC IS 14496-2 (1999) Visual A compression codec for visual data
Bassili J (1979). Emotion recognition: The role of facial movement and the relative importance of upper and lower areas of the face. J Personality Soc Psychol 37: 2049–2059
Breazeal C (2003). Toward sociable robots. Robot Auton Systems 42: 167–175
Cowie R, Douglas-Cowie E, Tsapatsoulis N, Votsis G, Kollias S, Fellenz W and Taylor J (2001). Emotion recognition in human– computer interaction. IEEE Signal Process Magaz 18(1): 32–80
Donato G, Bartlett MS, Hager JC, Ekman P and Sejnowski TJ (1999). Classifying facial Actions. IEEE Trans Pattern Anal Mach Intell 21(10): 974–989
Essa I, Pentland A (1997) Coding, analysis, interpretation, and recognition of facial expressions. IEEE Trans Pattern Anal Mach Intell 19(7)
Essa IA, Pentland A (1995) Facial expression recognition using a dynamic model and motion energy. In: International conference on computer vision (ICCV), pp 360–367
Feng GC, Yuen PC, Lai J (2000) Virtual view face image synthesis using 3d spring-based face model from a single image. In: Proceedings of fourth IEEE international conference on automatic face and gesture recognition, pp 530–535
Ge SS (2007) Social Robotics: Integrating advances in engineering and computer science. In: Proceedings of electrical engineering/ electronics, computer, telecommunications and information technology international conference. Chiang Rai, Thailand, pp xvii–xxvi
Ge SS, Guan F, Loh A, Fua C (2006) Feature representation based on intrinsic structure discovery in high dimensional space. IEEE International Conference on Robotics and Automation, pp 3399–3404
Ge SS, Loh AP, Guan F (2003) Sound localization based on mask diffraction. In: Proceedings of IEEE international conference on robotics and automation, pp 1972–1977
Ge SS, Loh AP and Guan F (2005). Robust sound localization using lower number of microphones. Int J Info Acqui 2(1): 1–22
Ge SS, Yang Y, Lee TH (2006) Hand gesture recognition and tracking based on distributed locally linear embedding. In: Proceedings of 2nd IEEE international conference on robotics, automation and mechatronics. Bangkok, Thailand, pp 567–572
Guan F, Li LY, Ge SS and Loh AP (2007). Robust huamn detection and identification by using stereo and thermal images in human robot interaction. Int J Inf Acquis 4(2): 161–183
Harris C, Stephens M (1988) A combined edge and corner detector. In: Proceedings of the 4th Alvey vision conference, pp 147–151
Intel Corporation: OpenCV Reference Manual (2001) http://www.intel.com/technology/computing/opencv/index.htm
Izard CE (1990). Facial expressions and the regulation of emotions. J Personality Soc Psychol 58(3): 487–498
Kahler K, Haber J, Seidel H (2001) Geometry-based muscle modeling for facial animation. In: Proceedings of graphics interface
Kotropoulos C, Pitas I (1997) Rule-based face detection in frontal views. In: Proceedings of IEEE international conference on acoustics, speech and signal processing (ICASSP 97), vol IV, pp 2537–2540
Loh AP, Guan F, Ge SS (2004) Motion estimation using audio and video fusion. In: Proceedings of the 8th international conference on control, automation, robotics and vision (ICARCV), pp 1569–1574
Lucas B, Kanade T (1981) An iterative image registration technique with an application to stereo vision. In: Proceedings of the 7th international joint conference on artificial intelligence (IJCAI ’81), pp 674–679
Lyons MJ, Akamatsu S, Kamachi M, Gyoba J (1998) Coding facial expressions with gabor wavelets. In: Proceedings of the third IEEE international conference on automatic face and gesture recognition, pp 200–205. http://kasrl.org/jaffe.html
Matsuno K, Tsuji S (1994) Recognizing human facial expressions in a potential field. In: International conference on pattern recognition, pp. 44–49
McKenna S, Gong S and Raja Y (1998). Modelling facial colour and identity with gaussian mixtures. Pattern Recogn 31(12): 1883–1892
Nedel LP, Thalmann D (1998) Real time muscle deformations using mass-spring systems. In: Proceedings of the computer graphics international, pp 156–165
Otsuka T, Ohya J (1996) Recognition of facial expressions using HMM with continuous output probabilities. In: Proceedings 5th IEEE international workshop on robot and human communication (RO-MAN), pp 323–328
Padgett C and Cottrell G (1997). Representing face images for classifying emotions, vol 9. MIT Press, Cambridge
Padgett C, Cottrell G, Adolps B (1996) Categorical perception in facial emotion classification. In: Proceedings of cognitive science conference, vol 18, pp 249–253
Paiva A (ed) (2000) Affective interactions: towards a new generation of computer interfaces. Springer, New York
Pantic M and Rothkrantz LJM (2000). Automatic analysis of facial expressions: the state of the art. IEEE Trans Pattern Anal Mach Intell 22(12): 1424–1445
Roivainen P, Li H and Forcheimer R (1993). 3-D motion estimation in model-based facial image coding. IEEE Trans Pattern Anal Mach Intell 15: 545–555
Roweis ST and Saul LK (2000). Nonlinear dimensionality reduction by locally linear embedding. Science 290: 2323–2326
Tapus A and Mataric MJ (2006). Towards socially assistive robotics. Int J Robot Soc Jpn (JRSJ) 24(5): 576–578
Tapus A and Mataric MJ (2007). Methodology and themes of human-robot interaction: A growing research field. Int J Adv Robot Systems 4(1): 103–108
Tapus A, Mataric MJ and Scassellati B (2007). The grand challenges in socially assistive robotics. IEEE Robot Autom Magaz (RAM), Spec Issue Grand Chall Robot 14(1): 35–42
Tenenbaum JB, de Silva V and Langford JC (2000). A global geometric framework for nonlinear dimensionality reduction. Science 290: 2319–2323
Terzopoulos D and Waters K (1993). Analysis and synthesis of facial image sequences using physical and anatomical models. IEE Trans Pattern Anal Mach Intell 15(6): 569–579
Tsapatsoulis N, Raouzaiou A, Kollias S, Cowie R and Douglas- Cowie E (2002). MPEG-4 facial animation, chap. Emotion recognition and synthesis based on MPEG-4 FAPs. Wiley, New York
Vapnik V (1995). The nature of statistical learning theory. Springer, New York
Wallhoff F: Facial expressions and emotion database. http://www.mmk.ei.tum.de/~waf/fgnet/feedtum.html
Waters K, Frisbie J (1995) A coordinated muscle model for speech animation. In: Proceedings of graphics interface, pp 163–170
Yacoob Y and Davis LS (1996). Recognizing human facial expressions from long image sequences using optical flow. IEEE Trans Pattern Anal Mach Intell 18: 636–642
Zhang Y and Ji Q (2005). Active and dynamic information fusion for facial expression understanding from image sequences. IEEE Trans Pattern Anal Mach Intell 27: 699–714
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Yang, Y., Ge, S.S., Lee, T.H. et al. Facial expression recognition and tracking for intelligent human-robot interaction. Intel Serv Robotics 1, 143–157 (2008). https://doi.org/10.1007/s11370-007-0014-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11370-007-0014-z