Abstract
We introduce a novel framework for automatic 3D facial expression analysis in videos. Preliminary results demonstrate editing facial expression with facial expression recognition. We first build a 3D expression database to learn the expression space of a human face. The real-time 3D video data were captured by a camera/projector scanning system. From this database, we extract the geometry deformation independent of pose and illumination changes. All possible facial deformations of an individual make a nonlinear manifold embedded in a high dimensional space. To combine the manifolds of different subjects that vary significantly and are usually hard to align, we transfer the facial deformations in all training videos to one standard model. Lipschitz embedding embeds the normalized deformation of the standard model in a low dimensional generalized manifold. We learn a probabilistic expression model on the generalized manifold. To edit a facial expression of a new subject in 3D videos, the system searches over this generalized manifold for optimal replacement with the ‘target’ expression, which will be blended with the deformation in the previous frames to synthesize images of the new expression with the current head pose. Experimental results show that our method works effectively.
Access provided by Autonomous University of Puebla. Download to read the full chapter text
Chapter PDF
Similar content being viewed by others
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Blanz, V., Vetter, T.: A Morphable Model for the Synthesis of 3D Face. In: ACM SIGGRAPH, Los Angeles, CA, pp. 187–194 (1999)
Zhang, Y., Prakash, E.C., Sung, E.: A New Physical Model with Multi-layer Architecture for Facial Expression Animation Using Dynamic Adaptive Mesh. IEEE Transactions on Visualization and Computer Graphics 10(3), 339–352 (2004)
Wang, Y., Huang, X., Lee, C., Zhang, S., Li, Z., Samaras, D., Metaxas, D., Elgammal, A., Huang, P.: High Resolution Acquisition, Learning and Transfer of Dynamic 3-D Facial Expressions. In: Proc. Eurographics 2004, Grenoble, France (2004)
Vieira, M.B., Velho, L., Sá, A., Carvalho, P.C.: A Camera-Projector System for Real-Time 3D Video. In: IEEE Int. Workshop on Projector-Camera Systems, San Diego, CA (2005)
Roweis, S., Saul, L.: Nonlinear Dimensionality Reduction by Locally Linear Embedding. Science 290, 2323–2326 (2000)
Tenenbaum, J.B., Silva, V., de Langford, J.C.: A Global Geometric Framework For Nonlinear Dimensionality Reduction. Science 290, 2319–2323 (2000)
Tenebaum, J.B., Freeman, W.T.: Separating Style and Content with Bilinear Models. Neural Computation J. 12, 1247–1283 (1999)
Vasilescu, A.O., Terzopoulos, D.: Multilinear Subspace Analysis for Image Ensembles. In: Proc. Computer Vision and Pattern Recognition, Madison, WI (2003)
Elgammal, A., Lee, C.: Separating Style and Content on a Nonlinear Manifold. In: Proc. Computer Vision and Pattern Recognition, Washington (2004)
Sumner, R., Popovic, J.: Deformation Transfer for Triangle Meshes. In: ACM SIGGRAPH, Los Angeles, CA (2004)
Chang, Y., Hu, C., Turk, M.: Probabilistic Expression Analysis on Manifolds. In: Proc. Computer Vision and Pattern Recognition, Washington (2004)
Vacchetti, L., Lepetit, V., Fua, P.: Stable Real-time 3D Tracking Using Online and Offline Information. IEEE Trans. on Pattern Analysis and Machine Intelligence 26, 1385–1391 (2004)
Goldenstein, S.K., Vogler, C., Metaxas, D.: Statistical Cue Integration in DAG Deformable Models. IEEE Trans. on Pattern Analysis and Machine Intelligence 25, 801–813 (2003)
Zhang, Z., Lyons, M., Schuster, M., Akamatsu, S.: Comparison Between Geometry-based and Gabor-wavelets-based Facial Expression Recognition Using Multi-layer Perceptron: IEEE Conf. on Automatic Face and Gesture Recognition, Nara, Japan (1998)
Chuang, E., Deshpande, H., Bregler, C.: Facial Expression Space Learning. Pacific Graphics (2002)
Kanade, T., Cohn, J., Tian, Y.: Comprehensive Database for Facial Expression Analysis. In: IEEE Conf. on Automatic Face and Gesture Recognition, pp. 46–53 (2000)
Sim, T., Baker, S., Bsat, M.: The CMU Pose, Illumination, and Expression Database. IEEE Trans. on Pattern Analysis and Machine Intelligence 25, 1615–1618 (2003)
Noh, J., Neumann, U.: Expression Cloning. In: ACM SIGGRAPH, Los Angeles, CA (2001)
Lee, J., Shin, S.Y.: A Hierarchical Approach to Interactive Motion Editing for Human-like Figures. In: ACM SIGGRAPH, Los Angeles, CA, pp. 39–48 (1999)
Zhang, Q., Liu, Z., Guo, B., Shum, H.: Geometry-Driven Photorealistic Facial Expression Synthesis. In: SIGGRAPH Symposium on Computer Animation (2003)
Pyun, H., Kim, Y., Chae, W., Kang, H.W., Shin, S.Y.: An Example-Based Approach for Facial Expression Cloning. In: Siggraph Symposium on Computer Animation (2003)
Sá, A., Carvalho, P.C., Velho, L.: (b,s)-BCSL: Structured Light Color Boundary Coding for 3D photography. In: Int. Fall Workshop on Vision, Modeling, and Visualization (2002).
Sederberg, T.W., Parry, S.R.: Free-Form Deformation of Solid Geometric Models. In: ACM SIGGRAPH, Dallas, TX, pp. 151-159 (1986)
Bourgain, J.: On Lipschitz Embedding of Finite Metric Spaces in Hilbert Space. Israel J. Math. 52, 46–52 (1985)
Blanz, V., Scherbaum, K., Vetter, T., Seidel, H.: Exchanging Faces in Images. In: Proc. Eurographics, Grenoble, France (2004)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Chang, Y., Vieira, M., Turk, M., Velho, L. (2005). Automatic 3D Facial Expression Analysis in Videos. In: Zhao, W., Gong, S., Tang, X. (eds) Analysis and Modelling of Faces and Gestures. AMFG 2005. Lecture Notes in Computer Science, vol 3723. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11564386_23
Download citation
DOI: https://doi.org/10.1007/11564386_23
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-29229-6
Online ISBN: 978-3-540-32074-6
eBook Packages: Computer ScienceComputer Science (R0)