Keywords

1 Introduction

Arbitrary view generation method can be classified into geometry-based information called GBR and image-based information called IBR. The GBR renders a virtual view by building a 3D geometry model. However, it is hard even impossible to construct all the models of the objects in the scene because of the high complex to reconstruct a model of an object, especially there are many objects in the scene with some unexpected modes. Contrast to GBR, the IBR method does not need the 3D details, since it renders the virtual views only by using the reference images. By doing so, naturalness of the virtual view image is preserved. Another advantage is that it can render virtual image in real time. It is more important for the 3D display with huge data to visit.

Depth Image-Based Rendering (DIBR) [1] is the most important and latest method in many IBR methods which has several attractive features. DIBR can theoretically render arbitrary view using a reference color image and the associated depth map. However, there are some problems like visibility, re-sampling, and disocclusion which make it hard to create a high quality virtual image [1]. Among those problems, disocclusion is considered as the most important and difficult one. Several approaches have been proposed to solve the problem.

One method is to fill the holes by using the information of the remainder part of the reference image. Simple way is to fill the holes with the background of the image [2]. Complex way is to do that using image-inpainting [3] [4–6]. Layered-depth-image (LDI) is another approach [7]. Recently, depth map pre-processing [8, 9] and bi-warping methods [10–14] are proposed to fill the holes.

However, most of the methods above cannot achieve the best result between image quality and low algorithm cost. The approach utilizing the information from the remainder part will cause artificial impairments and might involve some time-consuming schemes like inpainting. As to LDI, it needs to store the multi-layer depth value, it is a computationally complex and badly bandwidth consuming for transmission. Pre-processing methods would bring serious geometrical distortion. Bi-warping method needs two color image and two depth map and it also needs to wrap two reference images to the virtual plane.

Considering the image quality and the cost, we propose a simple and effective DIBR method. The method can get a better quality virtual image compared with the way using the data of the reminder of the reference image, depth map pre-processing, and inpainting. And the algorithm only costs 67 % compared with the double-sided DIBR. The virtual views generated with our approach have a more satisfying image quality at a low algorithm cost compared with other approaches.

We organize the reminder of the paper as follows. Section 2 briefly describes traditional double-sided DIBR technique. Section 3 is the details of our proposed method. Section 4 is the results of experiment. And Sect. 5 is the conclusion.

2 Double-Sided DIBR

Double-sided DIBR [12] can generate a better quality virtual image since it can use the information of both sides of the virtual view. So the virtual image has no apparent distortion and artificial noise without pre-processing the depth map and using inpainting approach. First, it uses two reference color images and two depth maps to get two candidate images. Second, it synthesizes the two candidate virtual images to get the final virtual image after interpolation and inpainting for the cracks and other tiny holes. The best advantage of double-sided DIBR is the complementary of the information in the disocclusion region. Figure 80.1 is the main framework of the double-sided DIRB.

Fig. 80.1
figure 1

Double-sided DIBR

Full size image

A warping function [1] can theoretically generate arbitrary view by projecting the reference image to the objective plane with the depth map corresponding. The warping function can be expressed as follows.

$$ {\rm{s}}\!{\cdot}m = A[R|t]M $$
(80.1)
$$ M = R^{ - 1} A^{ - 1} ({\rm{s}}\!{\cdot}m) $$
(80.2)
$$ {\rm{z}}\!{\cdot}m' = A'R'M + A'T $$
(80.3)
$$ {\rm{z}}\!{\cdot}m' = A'R'R^{ - 1} A^{ - 1} ({\rm{s}}\!{\cdot}m - At) + A'T $$
(80.4)

where s and z is a scale parameter equaled to the depth value in the world space, m and m′ are coordinates of the reference image and the virtual image, M is the world coordinates. A and A′ is the 3 × 3 Matrix presents the intrinsic parameters, R and t are the external parameters of the reference camera. R is the rotation parameters, t is the translation parameters. The s is the depth map value. R′ and T are the external parameters of the virtual camera. In this paper, we assume they are known.

3D wrapping can get the coarse virtual image. And image emerging can handle the big problem of the disocclusion. Figure 80.2 shows the results of the tradition double-sided DIBR. It must wrap the whole left and right reference image to the virtual view plane. But we can see that most of data between the left and the right candidate image are the same, while the holes need to fill is only a little proportion. And most of the cost is caused by the wrapping algorithm in the DIBR. So we can reduce the cost if we can reduce the wrapping data. Considering the cost, we proposed the FR-DIBR for the real time of two views to multi-view system.

Fig. 80.2
figure 2

The framework of the FR-DIBR

Full size image

3 FR-DIBR

Figure 80.2 illustrates the procedure of our approach FR-DIBR. The input files are two color images as the reference image and one depth map corresponding to one of the reference image. The output file is the arbitrary virtual view image in the middle of the two reference image: the color image with the depth map is considered as the main color image, another image as the assistant image.

Step 1. Forward Wrapping. Wrap the main color image to the objective plane to get the coarse virtual mage and the depth map in the virtual view using the wrapping function with the depth map corresponding to the reference image.

Step 2. Use the OPFD and Z-BUFFER [11] algorithm to handle the overlap, visibility, and re-sampling problems.

Step 3. Virtual depth map processing: after 3D wrapping of the main color image, we get the virtual view color image. The virtual view depth map is obtained at the same time. We named the depth map as the virtual depth map. From Eq. (80.4), we can know that when project the M to the m’, we get the u’, v’ as the coordinate and the depth map value z at the same time in the virtual plane. It does not cause another cost in the whole algorithm. And then a method called block-based water filled (BBWF) is used to fill the holes of the depth map, a block as 8 × 8, only use the depth with the lower value to fill the hole like the water stream. Figure 80.3 shows the BBWF algorithm, and Fig. 80.4 shows the results (Fig. 80.5).

Fig. 80.3
figure 3

BBWF algorithm process

Full size image
Fig. 80.4
figure 4

The result of the virtual depth map inpainting

Full size image
Fig. 80.5
figure 5

The procedure of the FR-DIBR

Full size image

Step 4. Reverse wrapping: find the holes in the virtual image and wrap the position to the assistant reference image with the virtual depth map. The procedure follows:

Search the holes used in the block 8 × 8. Sum the holes in the block.

Wrap the coordinate (u’, v’) to the assistant image, get the corresponding (u1, v1) in the assistant reference image when the sum of the holes is more than 50.

Use block 4 × 4 pixels to search the best match block in the assistant image, and get the pixels in the block to fill the holes in the block of the virtual image.

At this moment, we have handled the disocclusion problem in the virtual image, but there are still some little holes and cracks in the virtual image. We use the simple interpolate to handle them. At last we get the final virtual color image.

4 Experimental Results

We used the MSR standard test sequence and breakdancing sequence to test our proposed view synthesis approach. In our experiments, camera 2 is chosen as the main reference viewpoint, camera 4 as the assistant viewpoint to get the virtual viewpoint at the position of camera 3. The virtual image quality of the proposed algorithm and other algorithms is compared in Fig. 80.6, and the computational complexity is analyzed in Table 80.1. We can see that the virtual image is clearer and the cost is reduced by 33 %.

Fig. 80.6
figure 6

Results of different algorithm without holes filling a Real image b Classic Gaussian filter c Traditional bi-directional DIBR algorithm d Proposed algorithm

Full size image
Table 80.1 The computational complexity
Full size table

5 Conclusion

We have proposed a new double-sided DIBR. It is simpler and effective. In our algorithm we choose two viewpoints as the reference viewpoints to generate the intermediate viewpoint.

We can generate a good quality view image. Overlap problem is handled by OPFD method [11]. The disocclusion is filled by getting the information in the assistant image. The cracks and tiny holes have been removed completely but simple interpolation. The most important, we reduce the algorithm cost by 33 % through introducing the reverse wrapping which wraps the virtual point to the reference viewpoint. And it is important for the real time DIBR system.