Abstract.
Information extraction of facial expressions deals with facial-feature detection, feature tracking, and capture of the spatiotemporal relationships among features. It is a fundamental task in facial expression analysis and will ultimately determine the performance of expression recognition. For a real-world facial expression sequence, there are three challenges: (1) detection failure of some or all facial features due to changes in illumination and rapid head movement; (2) nonrigid object tracking resulting from facial expression change; and (3) feature occlusion due to out-of-plane head rotation. In this paper, a new approach is proposed to tackle these challenges. First, we use an active infrared (IR) illumination to reliably detect pupils under variable lighting conditions and head orientations. The pupil positions are then used to guide the entire information-extraction process. The simultaneous use of a global head motion constraint and Kalman filtering can robustly track individual facial features even in condition of rapid head motion and significant expression change. To handle feature occlusion, we propose a warping-based reliability propagation method. The reliable neighbor features and the spatial semantics among these features are used to detect and infer occluded features through an interframe warping transformation. Experimental results show that accurate information extraction can be achieved for video sequences with real-world facial expressions.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
Abd-Almageed W, Fadali MS, Bebis G (2002) A non-intrusive Kalman filter-based tracker for pursuit eye movement. In: Proceedings of the 2002 American Control conference
Ahlberg J (2000) Real-time facial feature tracking using an active model with fast image warping. http://citeseer.ist.psu.edu/538871.html
Bookstein F (1989) Principal warps: thin-plate splines and the decomposition of deformations. IEEE Trans Pattern Anal Mach Intell 11(6):567-585
Bookstein F (1991) Morphometric tools for landmark data. Oxford: Cambridge University Press, Cambridge, UK
Belongie S, Malik J, Puzicha J (2002) Shape matching and object recognition using shape contexts. IEEE Trans Pattern Anal Mach Intell 24(4):509-522
Beier T, Neely S (1992) Feature-based image metamorphosiss. In: Computer Graphics (Proceedings of SIGGRAPH 92) 26(2):35-42
Bourel F, Chibelushi CC, Low AA (2000) Robust facial feature tracking. Proceedings of the 11th British machine vision conference, 1:232-241
Bretzner L, Lindeberg T (1998) Feature tracking with automatic selection of spatial scales. Comput Vis Image Understand 71(3):385-392
Chetverikov D, Verest’oy J (1998) Tracking feature points: a new algorithm. In: Proceedings of the international conference on pattern recognition, pp 1436-1438
Choi CS, Takebe T (1994) Analysis and synthesis of facial image sequences in model-based image coding. IEEE Trans Video Technol 4(6):257-275
Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20:273-297
Daugman J (1988) Complete discrete 2-d gabor transforms by neural networks for image analysis and compression. IEEE Trans ASSP 36:1169-1179
Eisert P, Girod B (1998) Analyzing facial expressions for virtual conferencing IEEE Comput Graph Appl 18(5):70-78
Fieguth P (1997) Color-based tracking of heads and other mobile objects at video frame rates. In: Proceedings of the conference on computer vision and pattern recognition
Gorodnichy D (2002) On importance of nose for face tracking. In: Proceedings of the international conference on automatic face and gesture recognition (FG’2002)
Haro A, Flickner M, Essa I (2000) Detecting and tracking eyes by using their physiological properties, dynamics, and appearance. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 163-168
Huang J, Ii D, Shao X, Wechsler H (1998) Pose discrimination and eye detection using support vector machines (svms), In: Proceedings of NATO-ASI on face recognition: from theory to applications, pp 528-536
Hwang V (1998) Tracking feature points in time-varying images using an opportunistic selection approach. Pattern Recog 22:247-256
Ji Q, Yang X (2001) Real time visual cues extraction for monitoring driver vigilance. In: Schiele B, Sagerer G (eds) Lecture notes in computer science, vol 2095. Springer, Berlin Heidelberg New York, p 107
Kalman RE (1960) A new approach to linear filtering and prediction problems. Trans ASME J Basic Eng 85:35-45
Kapoor A, Picard RW (2002) Real-time, fully automatic upper facial feature tracking. In: Proceedings of the 5th IEEE international conference on automatic face and gesture recognition, pp 10-16
Kass M, Witkin A, Terzopoulos D (1987) Snakes: active contour models. Int J Comput Vis 1:321-332
Lee T (1996) Image representation using 2d Gabor wavelets. IEEE Trans Pattern Anal Mach Intell 18(10):959-971
Li H, Roivainen P, Forchheimer R (1993) 3-D motion estimation in model-based facial image coding. IEEE Trans Pattern Anal Mach Intell 15(6):545-555
Lucas B, Kanade T (1981) An iterative image registration technique with an application to stereo vision. In: Proceedings of the international joint conference on artificial intelligence, pp 674-679
Luettin J, Thacker NA, Beet SW (1996) Locating and tracking facial speech features. In: Proceedings of the international conference on pattern recognition, pp 652-656
Malciu M, Preteux F (2000) Tracking facial features in video sequences using a deformable model-based approach. http://citeseer.nj.nec.com/394625.html
Manjunath B, Chellappa R, Malsburg C (1992) A feature based approach to face recognition. Proceedings of IEEE international conference on computer vision and pattern recognition, pp 373-378
Maurer T, Malsburg C (1996) Tracking and learning graphs on image sequences of faces. In: Proceedings of the international conference on artificial neural networks, pp 373-378
McKenna S, Gong S, Wurtz R, Tanner J, Bannin D (1997) Tracking facail feature points with gabor wavelets and shape models. In: Proceedings of the international conference on audio-and video-based biometric person authentication, pp 35-42
Meyer F, Bouthemy P (1994) Region-based tracking using affine motion models in long image sequences. CVGIP Image Understand 60:119-140
Morimoto C, Koons D, Amir A, Flickner M (1999) Framer-ate pupil detector and gaze tracker. In: Proceedings of IEEE ICCV, frame-rate workshop
Moriyama T, Kanade T, Cohn J, Xiao J, Ambadar Z, Gao J, Imanura M (2002) Automatic recognition of eye blinking in spontaneously occurring behavior. In: Proceedings of the international conference on pattern recognition (ICPR ‘2002)
Pantic M, Rothkrantz L (2000) Automatic analysis of facial expressions: The state of the art. IEEE Trans Pattern Anal Mach Intell 22(12):1424-1445
Petajan E, Graf H (1996) Robust face feature analysis fo automatic speech-reading and character animation. In: Proceedings of the international conference on automatic face and gesture recognition, pp 357-362
Rangarajan K, Shah M (1991) Establishing motion correspondence. CVGIP: Image Understanding 54:56-73
Salari V, Sethi IK (1990) Feature point correspondence in the presence of occlusion. IEEE Trans Pattern Anal Mach Intell 12:87-91
Sethi IK, Jain R (1987) Finding trajectories of feature points in a monocular image sequence. IEEE Trans Pattern Anal Mach Intell 9:56-73
Shapiro L,Wang H, Brady J (1992) A matching and tracking strategy for independently-moving, non-rigid objects. In: Proceedings of the 3rd British machine vision conference, pp 306-315
Shi J, Tomasi C (1994) Good features to track. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 593-600
Sorenson HW (1970) Least-squares estimation: from Gauss to Kalman. IEEE Spectrum 7:63-68
Tomasi C, Kanade T (1991) Detection and tracking of point features. Technical Report, Carnegie Mellon University, Pittsburgh, PA
Thompson W, Lechleider P, Stuck E (1993) Detecting moving objects using the rigidity constraint. IEEE Trans Pattern Anal Mach Intell 15:162-166
Torresani L, Bregler C (2002) Space-time tracking. In: Proceedings of ECCV, pp 801-812
Tomasi C, Kanade T (1992) Shape and motion from image streams: a factorization method. Carnegie Mellon CMU-CS-92-104, pp 1-36
Turk M, Pentland A (1991) Eigenfaces for recognition. J Cogn Neurosci 3(1):71-86
Yang J, Stiefelhagen R, Meier U, Waibel A (1998) Real-time face and facial feature tracking and applications. In: Proceedings of the international conference on auditory-visual speech processing (AVSP’98)
Yuille A, Cohen DS, Hallinan PW (1989) Feature extraction from faces using deformable templates. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 104-109
Yuille A, Hallinan P (1993) Deformable templates. In: Blake A, Yuille A (eds) Active vision. MIT Press, Cambridge, MA, pp 21-38
Zhang L (1998) Automatic adaptation of a face model using action units for semantic coding of videophone sequences. IEEE Trans Circuits Syst Video Technol 8(6):781-795
Zheng Q, Chellappa R (1995) Automatic feature point extraction and tracking in image sequences for arbitrary camera motion. Int J Comput Vis 15:31-76
Zhang Z (1998) Feature-based facial expression recognition: Experiments with a multi-layer perception. Technical report INRIA, no 3354
Zhong Y, Jain AK, Dubuisson-Jolly M (2000) Object tracking using deformable templates. IEEE Trans Pattern Anal Mach Intell 22(5):544-549
Zhu Z, Ji Q, Fujimura K, Lee K (2002) Combining Kalman filtering and mean shift for real time eye tracking under active IR illumination. In: Proceedings of the international conference on pattern recognition, pp 373-378
Wang H, Brady J (1992) Corner detection with subpixel accuracy. Technical Report OUEL, Department of Engineering Science, University of Oxford, no 1925/92
Wiskott L, Fellous J, Krieger N, Malsburg C (1995) Face recognition and gender determination. In: Proceedings of the international workshop on automatic face and gesture recognition, pp 92-97
Fleet D, Jepson A (1993) Stability of phase information. IEEE Trans Pattern Anal Mach Intell 15(12):1253-1268
Wei X, Zhu Z, Yin L, Ji Q (2004) A real time face tracking and animation system In: 1st IEEE workshop on face processing in video, in conjunction with IEEE international conference on computer vision and pattern recognition
Gu H, Ji Q (2004) An automated face reader for fatigue detection, In: 6th international conference on automatic face and gesture recognition, 17-19 May 2004, Seoul, Korea
Li X, Ji Q (2003) Active affective state detection and assistance with dynamic Bayesian networks. 3rd workshop on affective and attitude user modeling assessing and adapting to user attitudes and affect: why, when and how?, in conjunction with 10th international conference on user modeling, Pittsburgh, PA
Wiskott L, Fellous J-M, Kruger N, von der Malsburg C (1997) Face recognition by elastic bunch graph matching. IEEE Trans Pattern Anal Mach Intell 19(7):775-779
Hutchinson TE (1990) Eye movement detection with improved calibration and speed. US patent 4,950,069
Cristinacce D, Cootes TF (2003) Facial feature detection using ADABOOST with shape constraints. In: Proceedings of BMVC, 1:231-240
Cristinacce D, Cootes TF (2004) A comparison of shape constrained facial feature detectors. In: Proceedings of the international conference on face and gesture recognition, pp 375-380
Cristinacce D, Cootes TF, Scott I (2004) A multistage approach to facial feature detection. In: Proceedings of the British machine vision conference, 1:277-286
Cootes TF, Edwards GJ, Taylor CJ (1998) Active appearance models. In: Proceedings of the European conference on computer vision 2:484-498
Author information
Authors and Affiliations
Corresponding author
Additional information
Received: 16 August 2003, Accepted: 20 September 2004, Published online: 20 December 2004
Correspondence to: Qiang Ji
Rights and permissions
About this article
Cite this article
Gu, H., Ji, Q. Information extraction from image sequences of real-world facial expressions. Machine Vision and Applications 16, 105–115 (2005). https://doi.org/10.1007/s00138-004-0161-6
Issue Date:
DOI: https://doi.org/10.1007/s00138-004-0161-6