Abstract
Double-sided DIBR can generate a perfect virtual view image. However, most of the data are the same in the two candidates. It is a waste cost. This paper proposed a simpler and more effective method to reduce the cost called forward and reverse wrapping DIBR (FR-DIBR). The FR-DIBR utilizes a main color image with associated depth map and an assistant color image. First, it wraps the main reference image to the virtual plane, and then reversely wraps the region of the disocclusion in the virtual view to the assistant reference image to get the lost information of the holes. The efficiency is improved by only wrapping one image and the disocclusion region. A depth map inpainting method called BBWF is also proposed. The experimental results show that the FR-DIBR can reduce the computation cost by 33 % and generate a clearer virtual image compared with the double-sided DIBR.
Access provided by Autonomous University of Puebla. Download conference paper PDF
Similar content being viewed by others
Keywords
1 Introduction
Arbitrary view generation method can be classified into geometry-based information called GBR and image-based information called IBR. The GBR renders a virtual view by building a 3D geometry model. However, it is hard even impossible to construct all the models of the objects in the scene because of the high complex to reconstruct a model of an object, especially there are many objects in the scene with some unexpected modes. Contrast to GBR, the IBR method does not need the 3D details, since it renders the virtual views only by using the reference images. By doing so, naturalness of the virtual view image is preserved. Another advantage is that it can render virtual image in real time. It is more important for the 3D display with huge data to visit.
Depth Image-Based Rendering (DIBR) [1] is the most important and latest method in many IBR methods which has several attractive features. DIBR can theoretically render arbitrary view using a reference color image and the associated depth map. However, there are some problems like visibility, re-sampling, and disocclusion which make it hard to create a high quality virtual image [1]. Among those problems, disocclusion is considered as the most important and difficult one. Several approaches have been proposed to solve the problem.
One method is to fill the holes by using the information of the remainder part of the reference image. Simple way is to fill the holes with the background of the image [2]. Complex way is to do that using image-inpainting [3] [4–6]. Layered-depth-image (LDI) is another approach [7]. Recently, depth map pre-processing [8, 9] and bi-warping methods [10–14] are proposed to fill the holes.
However, most of the methods above cannot achieve the best result between image quality and low algorithm cost. The approach utilizing the information from the remainder part will cause artificial impairments and might involve some time-consuming schemes like inpainting. As to LDI, it needs to store the multi-layer depth value, it is a computationally complex and badly bandwidth consuming for transmission. Pre-processing methods would bring serious geometrical distortion. Bi-warping method needs two color image and two depth map and it also needs to wrap two reference images to the virtual plane.
Considering the image quality and the cost, we propose a simple and effective DIBR method. The method can get a better quality virtual image compared with the way using the data of the reminder of the reference image, depth map pre-processing, and inpainting. And the algorithm only costs 67 % compared with the double-sided DIBR. The virtual views generated with our approach have a more satisfying image quality at a low algorithm cost compared with other approaches.
We organize the reminder of the paper as follows. Section 2 briefly describes traditional double-sided DIBR technique. Section 3 is the details of our proposed method. Section 4 is the results of experiment. And Sect. 5 is the conclusion.
2 Double-Sided DIBR
Double-sided DIBR [12] can generate a better quality virtual image since it can use the information of both sides of the virtual view. So the virtual image has no apparent distortion and artificial noise without pre-processing the depth map and using inpainting approach. First, it uses two reference color images and two depth maps to get two candidate images. Second, it synthesizes the two candidate virtual images to get the final virtual image after interpolation and inpainting for the cracks and other tiny holes. The best advantage of double-sided DIBR is the complementary of the information in the disocclusion region. Figure 80.1 is the main framework of the double-sided DIRB.
A warping function [1] can theoretically generate arbitrary view by projecting the reference image to the objective plane with the depth map corresponding. The warping function can be expressed as follows.
where s and z is a scale parameter equaled to the depth value in the world space, m and m′ are coordinates of the reference image and the virtual image, M is the world coordinates. A and A′ is the 3 × 3 Matrix presents the intrinsic parameters, R and t are the external parameters of the reference camera. R is the rotation parameters, t is the translation parameters. The s is the depth map value. R′ and T are the external parameters of the virtual camera. In this paper, we assume they are known.
3D wrapping can get the coarse virtual image. And image emerging can handle the big problem of the disocclusion. Figure 80.2 shows the results of the tradition double-sided DIBR. It must wrap the whole left and right reference image to the virtual view plane. But we can see that most of data between the left and the right candidate image are the same, while the holes need to fill is only a little proportion. And most of the cost is caused by the wrapping algorithm in the DIBR. So we can reduce the cost if we can reduce the wrapping data. Considering the cost, we proposed the FR-DIBR for the real time of two views to multi-view system.
3 FR-DIBR
Figure 80.2 illustrates the procedure of our approach FR-DIBR. The input files are two color images as the reference image and one depth map corresponding to one of the reference image. The output file is the arbitrary virtual view image in the middle of the two reference image: the color image with the depth map is considered as the main color image, another image as the assistant image.
Step 1. Forward Wrapping. Wrap the main color image to the objective plane to get the coarse virtual mage and the depth map in the virtual view using the wrapping function with the depth map corresponding to the reference image.
Step 2. Use the OPFD and Z-BUFFER [11] algorithm to handle the overlap, visibility, and re-sampling problems.
Step 3. Virtual depth map processing: after 3D wrapping of the main color image, we get the virtual view color image. The virtual view depth map is obtained at the same time. We named the depth map as the virtual depth map. From Eq. (80.4), we can know that when project the M to the m’, we get the u’, v’ as the coordinate and the depth map value z at the same time in the virtual plane. It does not cause another cost in the whole algorithm. And then a method called block-based water filled (BBWF) is used to fill the holes of the depth map, a block as 8 × 8, only use the depth with the lower value to fill the hole like the water stream. Figure 80.3 shows the BBWF algorithm, and Fig. 80.4 shows the results (Fig. 80.5).
Step 4. Reverse wrapping: find the holes in the virtual image and wrap the position to the assistant reference image with the virtual depth map. The procedure follows:
Search the holes used in the block 8 × 8. Sum the holes in the block.
Wrap the coordinate (u’, v’) to the assistant image, get the corresponding (u1, v1) in the assistant reference image when the sum of the holes is more than 50.
Use block 4 × 4 pixels to search the best match block in the assistant image, and get the pixels in the block to fill the holes in the block of the virtual image.
At this moment, we have handled the disocclusion problem in the virtual image, but there are still some little holes and cracks in the virtual image. We use the simple interpolate to handle them. At last we get the final virtual color image.
4 Experimental Results
We used the MSR standard test sequence and breakdancing sequence to test our proposed view synthesis approach. In our experiments, camera 2 is chosen as the main reference viewpoint, camera 4 as the assistant viewpoint to get the virtual viewpoint at the position of camera 3. The virtual image quality of the proposed algorithm and other algorithms is compared in Fig. 80.6, and the computational complexity is analyzed in Table 80.1. We can see that the virtual image is clearer and the cost is reduced by 33 %.
5 Conclusion
We have proposed a new double-sided DIBR. It is simpler and effective. In our algorithm we choose two viewpoints as the reference viewpoints to generate the intermediate viewpoint.
We can generate a good quality view image. Overlap problem is handled by OPFD method [11]. The disocclusion is filled by getting the information in the assistant image. The cracks and tiny holes have been removed completely but simple interpolation. The most important, we reduce the algorithm cost by 33 % through introducing the reverse wrapping which wraps the virtual point to the reference viewpoint. And it is important for the real time DIBR system.
References
Fehn C (2004) Depth-image-based rendering (DIBR), the compression and transmission for a new approach on 3DTV. In: Proceedings of SPIE steroscopic displays and virtual reality systems, vol 1, pp 93–104
Siddharth J (2003) Hole filling in images. In: M S thesis U C Berkeley vol 5, pp 45–67
Oh K-J, Yea S, Ho Y-S (2009) Hole filling method using depth based in-painting for view synthesis in free viewpoint television and 3-D video. In: Proceedings of picture coding symposium (PCS), vol 5. Chicago, pp 1–4
Wang Q, Cao N (2011) The hole-filling method for view synthesis in free view point television (FTV) and 3D video. In: IEEE international conference on electric information and control engineering ICEICE 2011, vol 5. Wuhan, pp 1805–1807
Ismael D, Beatrice P-P (2010) Depth-aid image in painting for novel view synthesis. In: IEEE international workshop on multimedia signal processing, vol 1. Saint Malo, pp 167–170
Yang L et al (2010) Probabilistic reliability based view synthesis for FTV. In: IEEE international conference on image processing ICIP 2010, vol 9. Hong Kong, pp 26–29
Yea S, Vetro A (2009) Multi-layered coding of depth for virtual view synthesis. In: Proceedings of picture coding symposium (PCS’09), vol 5, Chicago, pp 5–9
Chen W-Y, Chang Y-L, Lin S-F, Ding L-F, Chen L-G (2005) Efficient depth image based rendering with edge dependent depth filter and interpolation. In: Multimedia and Expo ICME 2005 IEEE international conference, vol 1, pp 1314–1317
Lee P-J., Effendi (2010) Adaptive edge-oriented depth image smoothing approach for depth image based rendering. In: IEEE international symposium on broadband multimedia systems and broadcasting (BMSB) vol 1, pp 56–63
Liu Z-W, An P, Liu S-X, Zhang Z-Y (2007) Arbitrary view generation based on DIBR. In: Proceedings of international symposium on intelligent signal processing and communication systems vol 1, pp 168–171
Feng Y-M, Li D-X, Luo K, Zhang M (2009) Asymmetric bidirectional view synthesis for free viewpoint and three-dimensional video. In IEEE Trans Consum Electron 55(4):2349–2355
Song G et al (2011) A new double-sided DIBR method for virtual view generation for 3DTV. In: IEEE international conference on communication technology ICCT 2011 Jinan (25–28), pp 1099–1102
Xue J-B, Song G, Li H (2011) Virtual view generation method without pre-processing depth-image for 3DTV. In: IEEE international conference on multimedia and signal processing CMSP, vol 5. Guilin, pp 362–365
Yu L, Xiang S, Deng H, Zhou P (2011) Depth based view synthesis with artifacts removal for FTV. In: IEEE international conference on image and graphics, vol 1. Hefei, pp 506–510
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Gui, H., Pang, Z., Chen, D., Chen, M., Tan, H. (2013). A Forward and Reverse Wrapping Depth Image-Based Rendering (FR-DIBR) Method for Arbitrary View Generation. In: Yang, Y., Ma, M. (eds) Proceedings of the 2nd International Conference on Green Communications and Networks 2012 (GCN 2012): Volume 1. Lecture Notes in Electrical Engineering, vol 223. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-35419-9_80
Download citation
DOI: https://doi.org/10.1007/978-3-642-35419-9_80
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-35418-2
Online ISBN: 978-3-642-35419-9
eBook Packages: EngineeringEngineering (R0)