Keywords

1 Introduction

A panorama contains a wide field-of-view (FOV) of the scene surrounding an observer and therefore, has more feature and information as compared to the normal input images of a scene. Earlier in 1900, panoramic cameras were used for general purpose panoramic photography for aerial and architectural panoramas. However, the introduction of computer technology and digital images simplified the process and digital panoramas achieved wide acceptance. Panoramic image mosaicing aims at reconstructing the whole scene from the limited images or video frames in order to give better visualization and complete view of the environment. There are many applications such as computer vision/graphics [1], image based rendering [2], virtual travel, virtual environment [3], scene change detection, video indexing/compression [4] and satellite/aerial imaging [5]. Anyone can generate a panorama these days having a computer and a camera with the help of available software.

A panorama can be captured in many possible ways, either single camera rotated around its optical centre or multiple cameras covering different part of the scene. Sometimes, large FOV lens e.g. fish-eye lens or parabolic mirror [6] are used to capture panoramic images but these are very expensive as well as hardware intensive means of panorama generation. Mosaicing algorithms [7–9] provide simplified solution to overcome the above mentioned problems.

The main emphasis of this work is on panoramic image mosaic generation. It has been implemented using feature based method of alignment and optimized graph-cut blending to generate high quality panoramic mosaic from a sequence of input images.

The paper organization is as follows: Sect. 2 gives an overview of the fundamental steps required for panoramic image mosaicing. The proposed method is described in Sect. 3 with the help of different steps and flow chart. Section 4 demonstrates the results obtained on implementation of the proposed algorithm on scenery images. The paper concludes in Sect. 5.

2 Panoramic Image Mosaicing

Panoramic mosaic generation process can be described in three main steps: image capturing system, pre-processing of the images and image mosaicing.

Panoramic image from multiple images can either be generated using a camera mounted on a tripod and rotated about its axis or using multiple cameras mounted together covering different directional views of the scene. In some cases, omnidirectional image sensors with fisheye lenses or mirrors are used, but their short focal length, high cost and lens distortion limits their access.

The images captured using panoramic imaging system are pre-processed before mosaicing for removing noise or any other intensity variations in the image. The processed images are then mosaiced by using the fundamental steps of mosaicing to get a seamless panoramic image.

3 Proposed Method of Panoramic Image Mosaicing

The images acquired using panoramic image capturing systems are pre-processed before stitching them together and then features are extracted from the images. The images are then re-projected over global frame which may result in creation of seam in the overlapping region of the images. A new optimized graph-cut method is employed for blending the individual images to create a single seamless panorama.

3.1 Feature Detection and Matching

In the proposed method of mosaicing, SIFT (Scale invariant feature transform) features [10] have been used for feature detection. SIFT features are stable, accurate, invariant to scaling and rotation which provides robust matching.

After feature point detection, next step is selection of corresponding features between the images. RANdom SAmple Consensus (RANSAC) [11] is used for robust estimation of homography between images for matching inliers.

3.2 Transformation of Images and Re-Projection

Based on the presence of different type of geometric distortion, transformations such as rigid, affine or projective are used. In case of panoramic image mosaics, the transformation used is projective transform between the images [12]. The projective model using a small number of parameters is most efficient and accurate parametric model for panoramic mosaic generation [13]. The images are transformed so as to align them on a common compositing surface or re-projection manifold. There are mainly three types of compositing manifold planar, cylindrical and spherical. In the proposed algorithm, planar manifold has been considered.

3.3 Blending of Images

After projecting the images over the compositing manifold, the next operation performed over it is blending of images. Various factors such as change in illumination or exposure artifacts cause intensity differences over the region of overlap resulting in the generation of visible edges in the final mosaic. These regions are handled differently from the other regions of the mosaic by applying an algorithm that helps in removing the seam while preventing loss of information. Image blending algorithms are categorized into two classes which are transition smoothing and optimal seam finding. In the proposed method, an optimized graph-cut algorithm is used for blending of images. It is based on optimal seam finding approach. Optimized graph-cut performs better for seam smoothing as compared to simple graph-cut and other existing blending techniques.

3.4 Optimized Graph-Cut

In graph cut technique, a seam is searched in such a way that the region between the images has a minimum variation of intensities. The search is in terms of minimization of energy of certain energy function. Earlier, algorithm such as dynamic programming [14] could not explicitly improve the existing seams as it was memoryless. However, graph-cut overcomes this limitation by treating each pixel uniformly and placing patches on the existing images. A graph is constructed for the overlapping region with edge weight corresponding to error across the seam cutting that edge and thus, creating a seam with least error.

In the algorithm, matching quality measure calculates the intensity difference between the pairs of pixels. Let ‘s’ and ‘t’ be the position of two adjacent pixels in the overlapping region, \( A\left( s \right) \) and \( B\left( s \right) \) be the pixel intensities at ‘s’ for old and new patch. Similarly \( A\left( t \right) \) and \( B\left( t \right) \) be the pixel intensities at position ‘t’. The matching quality [15] cost can now be defined as,

$$ M\left( {s,t,A,B} \right) = \left\| {A\left( s \right) - B\left( s \right)} \right\| + \left\| {A\left( t \right) - B\left( t \right)} \right\| $$
(1)

where \( \left\| \bullet \right\| \) denotes the appropriate norm. All the steps of the proposed panoramic mosaicing algorithm have been summarized in Fig. 1.

Fig. 1
figure 1

Proposed algorithm for panoramic image mosaic generation

Full size image

4 Results and Discussions

Different set of panoramic images have been considered and the algorithm is employed over these sets for generation of seamless panoramic mosaics. The first set consists of three input images as shown in Fig. 2a, b, c. In the first step, SIFT features are extracted and matched using RANSAC. Figure 2d shows the corresponding matches between the two images. The images are re-projected over a planar compositing surface for mosaic generation. The mosaic of two input images is shown in Fig. 2e which becomes an input image for the upcoming steps. The third image is matched with the generated mosaic for feature correspondence and alignment (Fig. 2f).

Fig. 2
figure 2

Panoramic mosaicing. a, b, c Input images. d Matching between image (a) and (b). e Mosaic of (a) and (b). f Matching between generated mosaic (e) and input image (c)

Full size image

The aligned images are finally projected over the compositing manifold to get the final mosaic. There is a creation of artificial edge or seam between the images in the transition area as can be clearly seen in Fig. 3a. The red boxes show the seam in this mosaic. If the images are not blended for removing the seam, then the final mosaic will contain visible artifacts. Blending ensures a high degree of smoothness in the transition region of the mosaic making it seamless and visually appealing. Results using graph-cut is shown in Fig. 3b and the zoomed view in Fig. 3c of a selected part of the scene shows the artifacts remaining in the image after simple graph-cut blending. The algorithm uses optimized graph-cut for the blending of images. The results of optimized graph-cut are shown in Fig. 3d. The horizontal line artifacts Fig. 3c have been removed using proposed technique.

Fig. 3
figure 3

Seam visibility. a Red boxes show the seam generated between the image in the transition region, b seam smoothing using graph-cut showing horizontal discrepancies, c zoomed view to show the horizontal marks (artifacts) of the mosaic in (b). d Panoramic mosaic using proposed method, e zoomed view to show the removed horizontal marks (artifacts) of the mosaic

Full size image

The zoomed view of the selected portion of the image is shown in Fig. 3e. It can be observed that there is no horizontal artifact in the image mosaic as in the case of graph-cut.

5 Conclusion

Camera motion, illumination difference, sensor noise and parallax cause degradation in the quality of panoramic image mosaic making it visually distorted. An efficient feature based panoramic image mosaicing method is proposed which attempts to alleviate the above mentioned problems during panorama creation. The algorithm uses feature based registration for aligning the images. The seam created in the area of overlap is removed using optimized graph-cut method which provides a feasible solution to the problem of seamless panoramic image mosaicing. The results show the effectiveness of the proposed algorithm.