Abstract
We present a complete framework designed for film production requiring live (pre) visualization. This framework is based on a famous game engine, Unity®. Actually, game engines possess many advantages that can be directly exploited in real-time pre-vizualization, where real and virtual worlds have to be mixed. In the work presented here, all the steps are performed in Unity: from acquisition to rendering. To perform real-time compositing that takes into account occlusions that occur between real and virtual elements as well as to manage physical interactions of real characters towards virtual elements, we use a low resolution depth map sensor coupled to a high resolution film camera. The goal of our system is to give the film director’s creativity a flexible and powerful tool on stage, long before post-production.
Access provided by Autonomous University of Puebla. Download to read the full chapter text
Chapter PDF
Similar content being viewed by others
References
Bartczak, B., Schiller, I., Beder, C., Koch, R.: Integration of a time-of-flight camera into a mixed reality system for handling dynamic scenes, moving viewpoints and occlusions in real-time. In: Proceedings of the 3DPVT Workshop, Atlanta, GA, USA, June 2008
Blinn, J.F.: Compositing. 1. theory. IEEE Computer Graphics and Applications 14(5), 83–87 (1994)
Boisson, G., Kerbiriou, P., Drazic, V., Bureller, O., Sabater, N., Schubert, A.: Fusion of kinect depth data with trifocal disparity estimation for near real-time high quality depth maps generation. In: IS&T/SPIE Electronic Imaging, pp. 90110–90110. International Society for Optics and Photonics (2014)
Briand, G., Bidgolirad, F., Zlapka, J.F., Lavalou, J.M., Lanouiller, M., Christie, M., Lvoff, J., Bertolino, P., Guillou, E.: On-set previsualization for vfx film production. In: International Broadcasting Convention (IBC). Netherland, Amsterdam (2014)
de Goussencourt, T., Bertolino, P.: Using the unity game engine as a platform for advanced real time cinema image processing. In: ICIP, Québec, Canada, September 2015
Debevec, P.E., Malik, J.: Recovering high dynamic range radiance maps from photographs. In: ACM SIGGRAPH 2008 classes, p. 31. ACM (2008)
Ferstl, D., Reinbacher, C., Ranftl, R., Rüther, M., Bischof, H.: Image guided depth upsampling using anisotropic total generalized variation. In: 2013 IEEE International Conference on Computer Vision (ICCV), pp. 993–1000. IEEE (2013)
Gandhi, V., Cech, J., Horaud, R.: High-resolution depth maps based on TOF-stereo fusion. In: 2012 IEEE International Conference on Robotics and Automation (ICRA), pp. 4742–4749. IEEE (2012)
Garcia, F., Mirbach, B., Ottersten, B., Grandidier, F., Cuesta, Á.: Pixel weighted average strategy for depth sensor data fusion. In: Proceedings - International Conference on Image Processing, ICIP, pp. 2805–2808 (2010)
Gokturk, S.B., Yalcin, H., Bamji, C.: A time-of-flight depth sensor-system description, issues and solutions. In: CVPRW 2004 Conference on Computer Vision and Pattern Recognition Workshop, pp. 35–35. IEEE (2004)
Hach, T., Steurer, J.: A novel RGB-Z camera for high-quality motion picture applications. In: Proceedings of the 10th European Conference on Visual Media Production, CVMP 2013, pp. 1–10. ACM, New York (2013)
Hahne, U., Alexa, M.: Combining time-of-flight depth and stereo images without accurate extrinsic calibration. International Journal of Intelligent Systems Technologies and Applications 5(3), 325–333 (2008)
Izadi, S., Kim, D., Hilliges, O., Molyneaux, D., Newcombe, R., Kohli, P., Shotton, J., Hodges, S., Freeman, D., Davison, A., et al.: Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera. In: Proceedings of the 24th annual ACM symposium on User interface software and technology, pp 559–568. ACM (2011)
Klein, G., Murray, D.: Compositing for small cameras. In: Proceedings of the 7th IEEE/ACM International Symposium on Mixed and Augmented Reality, pp 57–60. IEEE Computer Society (2008)
Knecht, M., Traxler, C., Purgathofer, W., Wimmer, M.: Adaptive camera-based color mapping for mixed-reality applications. In: 2011 10th IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp 165–168. IEEE (2011)
Kopf, J., Cohen, M.F., Lischinski, D., Uyttendaele, M.: Joint bilateral upsampling. ACM Transactions on Graphics 26(3), 96 (2007)
Natron, INRIA (2015). https://natron.inria.fr/ (Accessed May 14, 2015)
Northam, L., Istead, J., Kaplan, C.S.: A collaborative real time previsualization tool for video games and film. In SIGGRAPH Posters, p. 121. ACM (2012)
Nuke, the Foundry (2015). https://www.thefoundry.co.uk/products/nuke/ (Accessed May 14, 2015)
OpenEXR (2015). http://www.openexr.com/ (Accessed May 14, 2015)
Patra, S., Bhowmick, B., Banerjee, S., Kalra, P.: High resolution point cloud generation from kinect and hd cameras using graph cut. VISAPP (2), 311–316 (2012)
Shotton, J., Sharp, T., Kipman, A., Fitzgibbon, A., Finocchio, M., Blake, A., Cook, M., Moore, R.: Real-time human pose recognition in parts from single depth images. Communications of the ACM 56(1), 116–124 (2013)
Shpunt, A., Zalevsky, Z.: Three-dimensional sensing using speckle patterns, US Patent 8,390,821, March 5, 2013
Solidanim SolidTrack system (2015). http://www.solid-track.com/ (Accessed May 14, 2015)
Song, Y., Glasbey, C.A., van der Heijden, G.W.A.M., Polder, G., Dieleman, J.A.: Combining stereo and time-of-flight images with application to automatic plant phenotyping. In: Heyden, A., Kahl, F. (eds.) SCIA 2011. LNCS, vol. 6688, pp. 467–478. Springer, Heidelberg (2011)
Unity Game Engine (2015). https://unity3d.com/ (Accessed May 14, 2015)
Zhang, Z.: Flexible camera calibration by viewing a plane from unknown orientations. In: Proceedings of the Seventh IEEE International Conference on Computer Vision, 1999, vol. 1, pp. 666–673. IEEE (1999)
Zhang, Z.: A flexible new technique for camera calibration. Pattern Analysis and Machine Intelligence, IEEE Transactions on 22(11), 1330–1334 (2000)
Zhu, J., Wang, L., Yang, R., Davis, J.: Fusion of time-of-flight depth and stereo for high accuracy depth maps. In: IEEE Conference on Computer Vision and Pattern Recognition CVPR 2008, pp. 1–8. IEEE (2008)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
de Goussencourt, T., Dellac, J., Bertolino, P. (2015). A Game Engine as a Generic Platform for Real-Time Previz-on-Set in Cinema Visual Effects. In: Battiato, S., Blanc-Talon, J., Gallo, G., Philips, W., Popescu, D., Scheunders, P. (eds) Advanced Concepts for Intelligent Vision Systems. ACIVS 2015. Lecture Notes in Computer Science(), vol 9386. Springer, Cham. https://doi.org/10.1007/978-3-319-25903-1_76
Download citation
DOI: https://doi.org/10.1007/978-3-319-25903-1_76
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-25902-4
Online ISBN: 978-3-319-25903-1
eBook Packages: Computer ScienceComputer Science (R0)