Abstract
Due to the demand for depth maps of higher quality than possible with a single depth imaging technique today, there has been an increasing interest in the combination of different depth sensors to produce a “super-camera” that is more than the sum of the individual parts. In this survey paper, we give an overview over methods for the fusion of Time-of-Flight (ToF) and passive stereo data as well as applications of the resulting high quality depth maps. Additionally, we provide a tutorial-based introduction to the principles behind ToF stereo fusion and the evaluation criteria used to benchmark these methods.
Access provided by Autonomous University of Puebla. Download to read the full chapter text
Chapter PDF
Similar content being viewed by others
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Park, J., Kim, H., Tai, Y., Brown, M., Kweon, I.: High quality depth map upsampling for 3d-tof cameras. In: IEEE Proc. ICCV (2011)
Huhle, B., Fleck, S., Schilling, A.: Integrating 3d time-of-flight camera data and high resolution images for 3dtv applications. In: Proc. 3DTV Conf. IEEE (2007)
Castaneda, V., Mateus, D., Navab, N.: Stereo time-of-flight. In: 2011 IEEE International Conference on Computer Vision (ICCV), pp. 1684–1691. IEEE (2011)
Kuhnert, K., Stommel, M.: Fusion of stereo-camera and pmd-camera data for real-time suited precise 3d environment reconstruction. In: Int. Conf. on Intelligent Robots and Systems, pp. 4780–4785. IEEE (2006)
Fischer, J., Arbeiter, G., Verl, A.: Combination of time-of-flight depth and stereo using semiglobal optimization. In: Int. Conf. on Robotics and Automation (ICRA), pp. 3548–3553. IEEE (2011)
Eisemann, E., Durand, F.: Flash photography enhancement via intrinsic relighting. ACM Transactions on Graphics (Proc. of SIGGRAPH) 23 (2004)
Petschnigg, G., Szeliski, R., Agrawala, M., Cohen, M., Hoppe, H., Toyama, K.: Digital photography with flash and no-flash image pairs. ACM Trans. Graph. 23(3), 664–672 (2004)
Chen, J., Paris, S., Wang, J., Matusik, W., Cohen, M., Durand, F.: The video mesh: A data structure for image-based three-dimensional video editing. In: Proc. of the International Conference on Computional Photography (ICCP) (2011)
Lo, W.Y., van Baar, J., Knaus, C., Zwicker, M., Gross, M.: Stereoscopic 3d copy & paste. ACM Trans. Graph. 29(6), 147:1–147:10 (2010)
Zitnick, C.L., Kang, S.B., Uyttendaele, M., Winder, S.A.J., Szeliski, R.: High-quality video view interpolation using a layered representation. ACM Trans. Graph. 23(3), 600–608 (2004)
Devernay, F., Beardsley, P.: Stereoscopic Cinema. In: Ronfard, R., Taubin, G. (eds.) Image and Geometry Processing for 3-D Cinematography. Geometry and Computing, vol. 5, pp. 11–51. Springer, Heidelberg (2010)
Fei, Y., Yu, M., Shao, F., Jiang, G.: A color correction algorithm of multi-view video based on depth segmentation. In: International Symposium on Computer Science and Computational Technology (ISCSCT 2008), vol. 2, pp. 206–209 (2008)
Wilkes, L.: The role of ocula in stereo post production. The Foundry. Whitepaper (2009)
Templin, K., Didyk, P., Ritschel, T., Myszkowski, K., Seidel, H.P.: Highlight microdisparity for improved gloss depiction. ACM Transactions on Graphics (Proc. SIGGRAPH) 31(4) (2012)
Mcmillan, L., Gortler, S.: Image-based rendering: A new interface between computer vision and computer graphics. SIGGRAPH Comput. Graph. 33, 61–64 (2000)
Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. IJCV 47(1), 7–42 (2002)
Chiu, W.C., Blanke, U., Fritz, M.: Improving the kinect by cross-modal stereo. In: British Machine Vision Conf. BMVA, pp. 116–1 (2011)
Hrkać, T., Kalafatić, Z., Krapac, J.: Infrared-visual image registration based on corners and hausdorff distance. In: Ersbøll, B.K., Pedersen, K.S. (eds.) SCIA 2007. LNCS, vol. 4522, pp. 383–392. Springer, Heidelberg (2007)
Bilodeau, G., Torabi, A., Morin, F.: Visible and infrared image registration using trajectories and composite foreground images. Image and Vision Computing 29(1), 41–50 (2011)
Toet, A., Van Ruyven, L.J., Valeton, J.M.: Merging thermal and visual images by a contrast pyramid. Optical Engineering 28(7), 287789–287789 (1989)
Wu, C.: Visualsfm: A visual structure from motion system (2011)
Newcombe, R.A., Lovegrove, S.J., Davison, A.J.: Dtam: Dense tracking and mapping in real-time. In: 2011 IEEE International Conference on Computer Vision (ICCV), pp. 2320–2327. IEEE (2011)
Horn, B.K., Brooks, M.J.: Shape from shading. MIT Press (1989)
Barron, J.T., Malik, J.: Shape, albedo, and illumination from a single image of an unknown object. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 334–341. IEEE (2012)
Sturmer, M., Penne, J., Hornegger, J.: Standardization of intensity-values acquired by time-of-flight-cameras. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW 2008), pp. 1–6. IEEE (2008)
Kim, Y., Theobalt, C., Diebel, J., Kosecka, J., Miscusik, B., Thrun, S.: Multi-view image and tof sensor fusion for dense 3d reconstruction. In: ICCV Workshops, pp. 1542–1549. IEEE (2009)
Zhu, J., Wang, L., Gao, J., Yang, R.: Spatial-temporal fusion for high accuracy depth maps using dynamic mrfs. IEEE Transactions on Pattern Analysis and Machine Intelligence 32(5), 899–909 (2010)
Ghobadi, S.E., Loepprich, O.E., Lottnera, O., Ahmadov, F., Hartmann, K., Weihs, W., Loffeld, O.: Analysis of the personnel safety in a man-machine-cooperation using 2d/3d images. In: Proceedings of the EURON/IARP International Workshop on Robotics for Risky Interventions and Surveillance of the Environment, Benicassim, Spain (January 2008)
Bradski, G.: The OpenCV Library. Dr. Dobb’s Journal of Software Tools (2000)
Bouguet, J.Y.: Camera calibration toolbox for matlab (2004)
Hahne, U., Alexa, M.: Combining time-of-flight depth and stereo images without accurate extrinsic calibration. IJISTA 5(3), 325–333 (2008)
Nair, R., Lenzen, F., Meister, S., Schäfer, H., Garbe, C., Kondermann, D.: High accuracy TOF and stereo sensor fusion at interactive rates. In: Fusiello, A., Murino, V., Cucchiara, R. (eds.) ECCV 2012 Ws/Demos, Part II. LNCS, vol. 7584, pp. 1–11. Springer, Heidelberg (2012)
Dal Mutto, C., Zanuttigh, P., Cortelazzo, G.M.: A probabilistic approach to tof and stereo data fusion. In: 3DPVT, Paris, France, vol. 2 (2010)
Schiller, I., Beder, C., Koch, R.: Calibration of a pmd-camera using a planar calibration pattern together with a multi-camera setup. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences 37, 297–302 (2008)
Zhu, J., Wang, L., Yang, R., Davis, J.: Fusion of time-of-flight depth and stereo for high accuracy depth maps. In: Proc. CVPR, pp. 1–8. IEEE (2008)
Kim, Y.M., Chan, D., Theobalt, C., Thrun, S.: Design and calibration of a multi-view tof sensor fusion system. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW 2008), pp. 1–7. IEEE (2008)
Horn, B.K.: Closed-form solution of absolute orientation using unit quaternions. JOSA A 4(4), 629–642 (1987)
Guan, L., Pollefeys, M., et al.: A unified approach to calibrate a network of camcorders and tof cameras. In: Workshop on Multi-Camera and Multi-Modal Sensor Fusion Algorithms and Applications (M2SFA2 2008) (2008)
Hartley, R.I., Zisserman, A.: Multiple View Geometry in Computer Vision, 2nd edn. Cambridge University Press (2004)
Dal Mutto, C., Zanuttigh, P., Mattoccia, S., Cortelazzo, G.: Locally consistent toF and stereo data fusion. In: Fusiello, A., Murino, V., Cucchiara, R. (eds.) ECCV 2012 Ws/Demos, Part I. LNCS, vol. 7583, pp. 598–607. Springer, Heidelberg (2012)
Zhang, L., Curless, B., Seitz, S.M.: Spacetime stereo: Shape recovery for dynamic scenes. In: Proceedings of the 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2, pp. II–367. IEEE (2003)
Yang, Q., Tan, K.H., Culbertson, B., Apostolopoulos, J.: Fusion of active and passive sensors for fast 3d capture. In: MMSP (2010)
Reynolds, M., Dobos, J., Peel, L., Weyrich, T., Brostow, G.J.: Capturing time-of-flight data with confidence. In: 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 945–952. IEEE (2011)
Zhu, J., Wang, L., Yang, R., Davis, J., et al.: Reliability fusion of time-of-flight depth and stereo for high quality depth maps. TPAMI (99), 1 (2011)
Song, Y., Glasbey, C.A., van der Heijden, G.W.A.M., Polder, G., Dieleman, J.A.: Combining stereo and time-of-flight images with application to automatic plant phenotyping. In: Heyden, A., Kahl, F. (eds.) SCIA 2011. LNCS, vol. 6688, pp. 467–478. Springer, Heidelberg (2011)
Gudmundsson, S., Aanaes, H., Larsen, R.: Fusion of stereo vision and time-of-flight imaging for improved 3d estimation. IJISTA 5(3), 425–433 (2008)
Bartczak, B., Koch, R.: Dense depth maps from low resolution time-of-flight depth and high resolution color views. In: Bebis, G., Boyle, R., Parvin, B., Koracin, D., Kuno, Y., Wang, J., Pajarola, R., Lindstrom, P., Hinkenjann, A., Encarnação, M.L., Silva, C.T., Coming, D. (eds.) ISVC 2009, Part II. LNCS, vol. 5876, pp. 228–239. Springer, Heidelberg (2009)
Hahne, U., Alexa, M.: Depth imaging by combining time-of-flight and on-demand stereo. In: Kolb, A., Koch, R. (eds.) Dyn3D 2009. LNCS, vol. 5742, pp. 70–83. Springer, Heidelberg (2009)
Dal Mutto, C., Zanuttigh, P., Cortelazzo, G.M.: A probabilistic approach to tof and stereo data fusion. In: 3DPVT, Paris, France (May 2010)
Beder, C., Bartczak, B., Koch, R.: A combined approach for estimating patchlets from PMD depth images and stereo intensity images. In: Hamprecht, F.A., Schnörr, C., Jähne, B. (eds.) DAGM 2007. LNCS, vol. 4713, pp. 11–20. Springer, Heidelberg (2007)
Ruhl, K., Klose, F., Lipski, C., Magnor, M.: Integrating approximate depth data into dense image correspondence estimation. In: Proc. European Conference on Visual Media Production (CVMP) (August 2012)
Gandhi, V., Cech, J., Horaud, R.: High-resolution depth maps based on tof-stereo fusion. In: 2012 IEEE International Conference on Robotics and Automation (ICRA), pp. 4742–4749. IEEE (2012)
Van Meerbergen, G., Vergauwen, M., Pollefeys, M., Van Gool, L.: A hierarchical symmetric stereo algorithm using dynamic programming. International Journal of Computer Vision 47(1-3), 275–285 (2002)
Gallup, D., Frahm, J.M., Mordohai, P., Yang, Q., Pollefeys, M.: Real-time plane-sweeping stereo with multiple sweeping directions. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2007), pp. 1–8. IEEE (2007)
Kolmogorov, V., Zabih, R.: Computing visual correspondence with occlusions using graph cuts. In: Proceedings of the Eighth IEEE International Conference on Computer Vision (ICCV 2001), vol. 2, pp. 508–515. IEEE (2001)
Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence 23(11), 1222–1239 (2001)
Sun, J., Zheng, N.N., Shum, H.Y.: Stereo matching using belief propagation. IEEE Transactions on Pattern Analysis and Machine Intelligence 25(7), 787–800 (2003)
Ihler, A.T., Mcallester, D.A.: Particle belief propagation. In: International Conference on Artificial Intelligence and Statistics, pp. 256–263 (2009)
Birchfield, S., Tomasi, C.: A pixel dissimilarity measure that is insensitive to image sampling. IEEE Transactions on Pattern Analysis and Machine Intelligence 20(4), 401–406 (1998)
Hirschmüller, H., Scharstein, D.: Evaluation of cost functions for stereo matching. In: Proc. CVPR, pp. 1–8. IEEE (2007)
Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracy optical flow estimation based on a theory for warping. In: Pajdla, T., Matas, J(G.) (eds.) ECCV 2004. LNCS, vol. 3024, pp. 25–36. Springer, Heidelberg (2004)
Zach, C., Pock, T., Bischof, H.: A duality based approach for realtime TV-L1 optical flow. In: Pattern Recognition: 29th DAGM Symposium, vol. 29, pp. 214–223 (2007)
Nocedal, J.: Updating quasi-newton matrices with limited storage. Mathematics of Computation 25, 773–782 (1980)
Hirschmüller, H.: Stereo processing by semiglobal matching and mutual information. TPAMI 30(2), 328–341 (2008)
Mattoccia, S.: A locally global approach to stereo correspondence. In: 2009 IEEE 12th International Conference on Computer Vision Workshops (ICCV Workshops), pp. 1763–1770. IEEE (2009)
Cech, J., Sara, R.: Efficient sampling of disparity space for fast and accurate matching. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2007), pp. 1–8. IEEE (2007)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Nair, R. et al. (2013). A Survey on Time-of-Flight Stereo Fusion. In: Grzegorzek, M., Theobalt, C., Koch, R., Kolb, A. (eds) Time-of-Flight and Depth Imaging. Sensors, Algorithms, and Applications. Lecture Notes in Computer Science, vol 8200. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-44964-2_6
Download citation
DOI: https://doi.org/10.1007/978-3-642-44964-2_6
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-44963-5
Online ISBN: 978-3-642-44964-2
eBook Packages: Computer ScienceComputer Science (R0)