Abstract
We present an efficient face reconstruction and real-time facial expression for VR interaction driven by RGB-D videos. A RGB-D camera is first used to capture depth images, and a coarse face model is then rapidly reconstructed. The user’s personalized avatar is generated using pre-defined face model template and shape morphing techniques. We track the user’s head motion and face expression using a RGB camera. A set of facial features are located and labelled on the colour images. Corresponding facial features are automatically labelled on the reconstructed face model. The user’s virtual avatar is driven by the set of facial features using Laplacian deformation. We demonstrate that our algorithm is able to rapidly create a personalized face model using depth images and achieve realtime facial expression for VR interaction using live RGB videos. Our algorithm can be used in online learning environments that allow learners to interact with simulated and controlled virtual agents.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Baltrušaitis, T., Robinson, P., Morency, L.P.: 3D constrained local model for rigid and non-rigid facial tracking. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2610–2617 (2012)
Baltrušaitis, T., Robinson, P., Morency, L.P.: Openface: an open source facial behavior analysis toolkit. In: IEEE Winter Conference on Applications of Computer Vision, pp. 1–10 (2016)
Bassili, J.N.: Facial motion in the perception of faces and of emotional expression. J. Exp. Psychol. Hum. Percept. Perform. 4(3), 373–379 (1978)
Bernardini, F., Mittleman, J., Rushmeier, et al.: The ball-pivoting algorithm for surface reconstruction. IEEE Trans. Visual Comput. Graphics 5(4), 349–359 (1999)
Bickel, B., Botsch, M., Angst, R., et al.: Multi-scale capture of facial geometry and motion. ACM Trans. Graph. 36(3), 33–41 (2007)
Blanz, V., Vetter, T.: A morphable model for the synthesis of 3D faces. In: ACM SIGGRAPH, pp. 187–194 (1999)
Bouaziz, S., Wang, Y., Pauly, M.: Online modeling for realtime facial animation. ACM Trans. Graph. 32(4), article 40 (2013)
Cao, C., Weng, Y., Lin, S., et al.: 3D shape regression for real-time facial animation. ACM Trans. Graph. 32(4), article 41 (2013)
Cao, C., Wu, H., Weng, Y., et al.: Real-time facial animation with image-based dynamic avatars. ACM Trans. Graph. 35(4), article 126 (2016)
Casas, D., Alexander, O., Feng, A. W., et al.: Rapid photorealistic blendshapes from commodity RGB-D sensors. In: ACM Symposium on Interactive 3D Graphics and Games, pp. 134–134 (2015)
Chai, J.X., Xiao, J., Hodgins, J.: Vision-based control of 3D facial animation. In: ACM SIGGRAPH/Eurographics Symposium on Computer Animation, pp. 193–206 (2003)
Chen, Y.L., Wu, H.T., Shi, F.H.: Accurate and robust 3D facial capture using a single RGBD camera. In: IEEE International Conference on Computer Vision, pp. 3615–3622 (2013)
Chuang, E., Bregler, C.: Performance driven facial animation using blendshape interpolation. Technical Report, Stanford University (2002)
Deng, Z., Chiang, P.Y., Fox, P., et al.: Animating blendshape faces by cross-mapping motion capture data. In: ACM Symposium on Interactive 3D Graphics and Games, pp. 43–48 (2006)
Deng, Z., Noh, J.: Computer facial animation: a survey. In: Data-Driven 3D Facial Animation, pp. 1–28. Springer, London (2008)
Goto, T., Escher, M., Zanardi, C., et al.: MPEG-4 based animation with face feature tracking. In: Computer Animation and Simulation, pp. 89–98. Springer, Wien (1999)
Huang, H., Chai, J.X., Tong, X., et al.: Leveraging motion capture and 3D scanning for high-fidelity facial performance acquisition. ACM Trans. Graph. 30(4), article 74 (2011)
Ichim, A.E., Bouaziz, S., Pauly, M.: Dynamic 3D avatar creation from hand-held video input. ACM Trans. Graph. 34(4), article 45 (2015)
Lin, I.C., Yeh, J.S., Ouhyoung, M.: Extracting 3D facial animation parameters from multiview video clips. IEEE Comput. Graph. Appl. 22(6), 72–80 (2002)
Liu, Z.C., Zhang, Z.Y., Jacobs, C., et al.: Rapid modeling of animated faces from video. Comput. Animation Virtual Worlds 12(4), 227–240 (2001)
Ramasubramanian, V., Paliwal, K.K.: Fast k-dimensional tree algorithms for nearest neighbor search with application to vector quantization encoding. IEEE Trans. Signal Process. 40(3), 518–531 (1992)
Sifakis, E., Neverov, I., Fedkiw, R.: Automatic determination of facial muscle activations from sparse motion capture marker data. ACM Trans. Graph. 24(4), 417–425 (2005)
Sorkine, O., Cohen-Or, D., Lipman, Y., et al.: Laplacian surface editing. In: Eurographics/ACM SIGGRAPH Symposium on Geometry Processing, pp. 175–184 (2004)
Sturm, J., Bylow, E., Kahl, F., et al.: CopyMe3D: scanning and printing persons in 3D. In: German Conference on Pattern Recognition, pp. 405–414. Springer, Berlin (2013)
Sumner, R.W., Popović, J.: Deformation transfer for triangle meshes. ACM Trans. Graph. 23(3), 399–405 (2004)
Thomas, D., Taniguchi, R.I.: Augmented blendshapes for real-time simultaneous 3D head modeling and facial motion capture. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3299–3308 (2016)
Tulyakov, S., Vieriu, R.L., Semeniuta, S., et al.: Robust real-time extreme head pose estimation. In: IEEE International Conference on Pattern Recognition, pp. 2263–2268 (2014)
Wan, X., Jin, X.G.: Data-driven facial expression synthesis via Laplacian deformation. Multimedia Tools Appl. 58(1), 109–123 (2012)
Wang, K.K., Zhang, G.F., Bao, H.J.: Robust 3D reconstruction with an RGB-D camera. IEEE Trans. Image Process. 23(11), 4893–4906 (2014)
Waters, K.: A muscle model for animation three-dimensional facial expression. ACM SIGGRAPH 21(4), 17–24 (1987)
Weise, T., Bouaziz, S., Li, H., et al.: Realtime performance based facial animation. ACM Trans. Graph. 30(4), article 77 (2011)
Weng, Y.L., Cao, C., Hou, Q.M., et al.: Real-time facial animation on mobile devices. Graph. Models 76(3), 172–179 (2014)
Zollhöfer, M., Martinek, M., Greiner, G., et al.: Automatic reconstruction of personalized avatars from 3D face scans. Comput. Animation Virtual Worlds 22(2–3), 195–202 (2011)
Zollhöfer, M., Nießner, M., Izadi, S., et al.: Real-time non-rigid reconstruction using an RGB-D camera. ACM Trans. Graph. 33(4), article 156 (2014)
Acknowledgements
This research work was partially supported by NSFC (No. 61631166002 and No. 61572196).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this chapter
Cite this chapter
Ren, H., Zhang, X. (2021). Efficient Facial Reconstruction and Real-time Expression for VR Interaction Using RGB-D Videos. In: Cai, Y., van Joolingen, W., Veermans, K. (eds) Virtual and Augmented Reality, Simulation and Serious Games for Education. Gaming Media and Social Effects. Springer, Singapore. https://doi.org/10.1007/978-981-16-1361-6_14
Download citation
DOI: https://doi.org/10.1007/978-981-16-1361-6_14
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-16-1360-9
Online ISBN: 978-981-16-1361-6
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)