Abstract
A fusion based seamless mosaic method for stitching remote sensing images is introduced in this paper. The proposed method focuses on one major problem in the process of mosaic: how to generate visually pleasant stitching result in the cases of misalignment, global and local intensity differences between images. First, two partially overlapped images are decomposed into high-frequency components and low-frequency components with Gaussian low-pass filter. Second, by considering the information characteristics contained in both separated components, different mosaic schemes are designed to accomplish stitching process accordingly. For the low-frequency components consisting of coarse shape and illumination information, two-dimension weighted blending rule is utilized to achieve smoothing transition. For the high-frequency components including rich details, an improved seam searching strategy based on dynamic programming is introduced. With the obtained stitching seam guiding the stitching process, visible structural break can be avoided. Finally, the mosaic result is produced by linearly composing both stitching results of different components together. Experimental results demonstrate the effectiveness of the proposed method in generating seamless mosaic results without introducing any unexpected blurring or artifacts.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
1 Introduction
Remote sensing techniques can provide an abundance of useful information, which plays an important role in both military and civilian domain, such as battlefield investigation and marine resources surveillance. Due to the limited field-of-view of sensors, the area of interest always cannot be occupied in one obtained image. Therefore, it is essential to generate large-scale composition image with a sequence of overlapped remote sensing images, which is usually referred to as image mosaic technique [1–3].
Generally, the following two desirable properties are required in generating visually pleasant mosaic result [4]. First, the result should be as similar as possible to the input images in terms of intensity and geometry. Second, the stitching line should be invisible. However, the overlapped images obtained from different time, angles and sensors always present global/local intensity difference. Besides, the misalignment, caused by the inaccurate registration or the presence of moving objects and motion parallax, is another common issue in the process of mosaic. Both the intensity difference and misalignment will introduce visible stitching seam, blurring effect and structure inconsistence. Assuming that images with partially overlapped regions have already been aligned by some image registration methods [5–7], image mosaic methods can be categorized into two methods: weighted blending based method and optimal seam based method [8].
The weighted blending based methods, commonly called as alpha blending or feathering [9–11], aim at achieving smooth transition in the overlapped region. Specifically, the mosaic result is considered as a weighted combination of both images, where the weight coefficient is calculated according to the distance from the stitching boarders or the centers. More advanced weighted blending techniques are developed by using variational approach [12] or Poisson-based gradient-domain compositing approach [13]. However, the blending based mosaic methods have the drawback that ghost and blurring artifacts may be introduced if overlapped images are misaligned [4, 14].
The optimal seam based methods intend to search a curve in the overlapping area, on which the two registered images have the minimum difference. With the overlapping area being divided into two parts by the searched curve, each part can then be filled with the pixels in the corresponding image. Many optimal seam searching methods have been developed in the literature, including Dijkstras algorithm based methods [15–17], dynamic programming based methods [18–20], and graph-cuts based methods [21, 22]. However, the optimal seam based methods fail in generating visually pleasant mosaic results when it comes to the case of intensity difference. To avoid the visual inconsistence caused by unbalanced intensity, histogram matching based pre-processing has been extensively considered [23–25]. As one statistic based intensity balancing method, the histogram matching is effective in adjusting the global intensity distribution of the target image to fit the reference image through nonlinear transformation. But in the case that the overlapped images have quite different scenery distribution (e.g. 10 % field and 90 % river in one image, while 90 % field and 10 % river in another one), histogram based balancing method is no longer appropriate. Meanwhile, the local intensity difference cannot be eliminated via histogram matching.
Considering the merits and drawbacks of the weighted blending based method and the optimal seam based method, both methods are integrated appropriately into one framework to improve the mosaic performance. First, low-pass filter is applied to decompose each overlapped image into high-frequency component and low-frequency component. Then different mosaic schemes are designed for stitching each component. For low-frequency components consisting of uneven illumination information, the weighted blending rule based mosaic scheme is introduced to achieve smooth intensity transition. With no detail information in the low-frequency components, the weighted blending rule can be effective in avoiding ghost or blurring effect. For high-frequency components consisting of rich details (edges, structures and texture, etc.), a modified optimal seam searching method is designed to determine the seam line and guide the stitching process within the overlapping area [18–20]. With no uneven intensity information in the high-frequency components, the modified optimal seam based mosaic method can work well in preserving structure and texture consistence. Finally, the mosaic result can be obtained via linearly composing both mosaic results corresponding to the components of high-frequency and the low-frequency together.
The rest of the paper is organized as follows. In Sect. 2, we described the proposed seamless mosaic method in detail. Then we show the experimental mosaic results on real remote sensing images by our method in Sect. 3. Meanwhile, the comparison results of different mosaic methods are also given in Sect. 3. Finally, the conclusions are drawn in Sect. 4, and the focus of the future related work is also referred in this section.
2 The Proposed Method
Let \(I_1\) and \(I_2\) represent two overlapped remote sensing images respectively, the proposed remote sensing image mosaic method consists of four parts: low-pass filtering based image decomposition, weighted blending based mosaic for low-frequency components, optimal seam based mosaic for high-frequency components, and the composition of both mosaic results of different components. The first step aims at decomposing each registered image into high-frequency component \(I_{iH}(i=1,2)\) and the low-frequency component \(I_{iL}(i=1,2)\), denoted as \(I_i=I_{iH}+I_{iL}(i=1,2)\). Then, we can generate two mosaic results with different mosaic schemes, denoted as \(I_{H-mosaic}\) and \(I_{L-mosaic}\), respectively. Finally, the mosaic result can be obtained via composing mosaic results of different components, mathematically denoted as \(I_{mosaic}=I_{L-mosaic}+I_{H-mosaic}\). The framework is shown in Fig. 1.
2.1 Low-Pass Filtering Based Image Decomposition
The process of image decomposition aims at separating the original images \(I_1\) and \(I_2\) into coarse component and the detail component. The coarse component mainly consists of profile and illumination information, and the detail component only contains details such as edges, structures and textures. To fulfill this decomposition process, related works have been done in [26–28], varying from easy low-pass/high-pass filtering [26] to wavelet transformation [27] and the morphological component analysis [28]. In our method, the Gaussian low-pass filtering is adopted for the sake of low computational cost, as recommended in [26]. The Gaussian low-pass filtering is defined as:
where x is the distance from the origin in the horizontal axis, y is the distance from the origin in the vertical axis, and D 0 is the cut-off frequency. One example of Gaussian low-pass filtering based decomposition result is given in Fig. 2. It is clear from Fig. 2 that such decomposition method is effective in generating the coarse component (low-frequency component, \(I_L\)) and detail component (high-frequency component, \(I_H\)).
2.2 Weighted Blending Based Low-Frequency Components Mosaic
For the low-frequency components \(I_{1L}\) and \(I_{2L}\), the weighted blending rule is applied to generate the mosaic result \(I_{L-mosaic}\). Let \(\varPhi _{0}\) denotes the overlapping region, \(\varPhi _{1}\) and \(\varPhi _{2}\) denote the non-overlapping regions belonging to image \(I_{1L}\) and \(I_{2L}\), respectively. Traditionally, the 1-D weighted blending rule is defined as:
where λ is weight coefficient calculated by λ = d/W, in which d represents the distance from the current point to the border of the overlapping area and W represents the width of the overlapping region. One example is given in Fig. 3. It is clear that the two images in Fig. 3a, b have difference in local intensity, which would lead to visible seam if pasting both images together directly. Instead, with the 1-D blending rule defined as Eq. (2), applied to stitch \(I_{1L}\) and \(I_{2L}\), visually smooth transmission can be achieved.
When it comes to the case of 2-D shift transformation (as shown in Fig. 4a), the weight coefficient λ is adjusted to:
where H and W respectively represents the height and width of the overlapping regions, y and x respectively denote the relative vertical and horizontal location to the borders, as illuminated in Fig. 4. One example of applying 2-D blending rule in stitching low-frequency components is given in Fig. 4c, and it is clear that smooth transition can be achieved from left to right and from up to down.
2.3 Optimal Seam Based High Frequency Components Mosaic
The two registered images of high-frequency components, \(I_{1H}\) and \(I_{2H}\), mainly consist of rich structures and textures. Considering the weighted blending based mosaic method would fail in avoiding blurring and double-edge in the case of misalignment, an optimal seam searching strategy is utilized to guide the process of stitching \(I_{1H}\) and \(I_{2H}\). First, by using sum of squared differences (SSD), one energy function is defined as [29]:
where \(Block_1(x,y)\) and \(Block_2(x,y)\), respectively, represent image patches surrounding each pixel (x, y) in images \(I_{1H}\) and \(I_{2H}\), and the block size is \(N\times {N}\). Then we will search for a path in the overlapping area along which the two textures can match best, i.e. where the SSD error is the lowest. The optimal mosaic path C is a curve crossing from top to bottom of the overlapping region with exactly one point on each row. Following the dynamic programming based optimal seam searching methods [29, 30], one general cumulative minimum energy is defined as:
With the energy values for all pixels in the overlapping regions obtained, the next step is to trace back the process of cumulative minimum energy calculation from the pixel with the minimum energy in the last row. Then the optimal seam C (marked with red line in Fig. 5) with cumulative minimum energy is decided. Finally, the mosaic result in the overlapping area \(\varPhi _0\) can be divided into \(\varPhi _{01}\) and \(\varPhi _{02}\) by the curve C, where the region \(\varPhi _{01}\) and \(\varPhi _{02}\) are respectively filled with pixels belonging to image \(I_{1H}\) and \(I_{2H}\). The illustration is given in Fig. 5, where the red line represents the determined optimal line. The traditional dynamic programming based methods can work effectively in the case of 1-D shift transformation.
However, the restriction in seam searching direction (either from top to down or from left to right) easily leads to the failure in dealing with the case of 2-D shift transformation in case of intensity difference. It is clear from Fig. 6a that the searched red line by 1-D searching scheme could not connect the specified point A and point B. As a result, two visible horizontal seams respectively originated from point A and point B are unavoidable if two overlapped images have intensity difference. Considering 2-D shift transformation is a more general situation in stitching remote sensing images, it is of great importance to design a better stitching seam searching scheme to avoid obvious artifacts. In our method, one modified optimal seam searching scheme is designed based on the dynamic programming algorithm. First, one optimal seam (marked with the red line in Fig. 6a) from the upmost to the downmost of the overlapping area is searched with the traditional dynamic programming algorithm [30]. Second, two another seams are searched in a similar way from the leftmost to the rightmost (as marked with the blue line in Fig. 6) and from the rightmost to the leftmost (as marked with the green line in Fig. 6). Differently, the blue (yellow) seam, we firstly calculate the energy of all the paths from right to left (left to right) with Eq. (6). Then we trace back this process from the point A (B) to determine the blue (yellow) seam. Finally, the seam connecting the point A and B can be determined to separate the overlapping area \(\varPhi _0\) into two parts, denoted as \(\varPhi _{01}\) and \(\varPhi _{02}\) (as shown in Fig. 6). By filling the areas \(\varPhi _{01}\) and \(\varPhi _{02}\) with the corresponding pixels in the left image \(I_1\) and the right image \(I_2\) respectively, the mosaic result of high-frequency components can be obtained.
One example is shown in Fig. 7. It is clear that obvious structural break (seen more clearly in the zoomed area in the red box) can be seen in the aligned result (Fig. 7a) due to the misalignment. By applying the modified seam searching scheme in stitching the high-frequency components, consistent structure can be achieved in the area marked with blue box in Fig. 7b.
3 Experiments
3.1 Experimental Setup
In this section, the performance of the proposed fusion based mosaic method is tested on three groups of remote sensing image pairs. As shown in Fig. 8, the test image pairs consist of one Landsat image pair and two optical aerial image pairs. For the sake of convenience, three groups of test images are respectively named by “Landsat” (Fig. 8a), “Aerial 1” (Fig. 8b) and “Aerial 2” (Fig. 8c). These test image pairs are selected as varieties of sceneries are contained (e.g. mountain, field, city and river, etc.).
To evaluate the performance in generating seamless mosaic result, our method is compared with both the weighted blending rule based and the optimal seam based mosaic methods. The blending rule is achieved by Eqs. (3) and (4), while the dynamic programming based optimal seam searching scheme is implemented according to [30]. In our method, two necessary parameters are needed to be set: the cut-off frequency D 0 in Eq. (1) and the patch size N × N in Eq. (5). To demonstrate the effect of these two parameters in mosaic results, we repeat the mosaic experiments with only one parameter (D 0 or N) being varied and the other one being settled. Two groups of test experimental results are given in Figs. 9 and 10.
Firstly, the influence of D 0 is tested with N = 3. As shown in Fig. 9a, broken structure can be seen in the case of D 0 = 0. This is because small value of D 0 leads more illumination information to be separated in the high-frequency component, however, the optimal seam based method for mosaicing high-frequency components fails in determining an optimal stitching seam if there is illumination difference. On the other hand, more structural information would be separated in low-frequency component as the value of D 0 increases, which may generate varying degrees of blurring effect, as shown in Fig. 9d–f (D 0 ≥ 0.1). This blurring effect is caused by applying the blending rule in mosaicing the low-frequency components containing structural information. Therefore, the selection of D 0 should be neither too high nor too low, which is suggested to be values between [0.03, 0.07] according to the experimental results. In our experiments, D 0 is set to be 0.03.
Then, with the D 0 settled to be 0.03, the value of N is varied to test the influence of N. In case of inaccurate registration, unexpected optimal seam can be determined and it would lead to failure in structure preservation which can be seen more clearly in the zoomed blue box of Fig. 10a (N = 1). To generate better stitching seam,the related pixels in a neighbored window are taken into considered. However, large size (N ≥ 5) of neighboring window cannot well eliminate structure inconsistence, as shown in Fig. 10c–e.Therefore, N is set to be three to generate better seamless mosaic results.
3.2 Experimental Results
In this section, the mosaic performance is tested on three remote sensing image pairs (shown in Fig. 8), and the visual comparison results of various mosaic methods are given in Figs. 11, 12 and 13.
Figure 11 shows the comparison results of stitching image pair “Landsat”. The image pair “Landsat”(Fig. 8a) presents no intensity difference, and it is used to test the performance in handling the case of misalignment. Practically, we misalign the images “Landsat” with five pixels respectively in vertical and horizontal directions. It is obvious from Fig. 11a, b that both the 2-D weighted blending rule based and the optimal seam based methods are affected by the misalignment. Specifically, blurring and ghost phenomenon appears in the mosaic result obtained via blending rule based method, which can be noticed more clearly in the area marked with red box in Fig. 11a. Though the optimal seam based mosaic method is free of blurring effect, obvious structural break is introduced in the overlapping area, as shown in the red box of Fig. 11b. By contrast, our method can achieve visually smoothing transition in terms of intensity and structure simultaneously (Fig. 11c), meanwhile, no blurring effect is introduced.
Comparison results of another two groups of test image pairs, the “Aerial 1” and the “Aerial 2”, are respectively shown in Figs. 12 and 13. It is clear that local color tone variance (bright spot in the overlapping area in Fig. 8b) exists in “Aerial 1”, meanwhile the “Aerial 2” has global variance in terms of color tone. These two image pairs are utilized to evaluate the mosaic performance in dealing with both the problems of intensity variance and misalignment.
Figure 12 shows the comparison mosaic results corresponding to image pairs “Aerial 1”. It is clear from the blue box in Fig. 12b that visible seam can be seen in the mosaic result by the optimal seam based method. By contrast, both the blending rule based method and our method can achieve smooth transition in the lake area, as shown in blue boxes in Fig. 12a, c. However, the blending rule applied in the original images leads to serious artifact of double-edge in the overlapping area due to the misalignment, which can be seen more clearly in the red box of Fig. 12a. By integrating both the 2-D blending rule and the modified optimal seam searching scheme into one framework, our method can successfully avoid both artifacts and eliminate obvious seam simultaneously (Fig. 12c).
Figure 13 shows the comparison results of stitching image pair “Aerial 2”. In accordance with the result shown in Fig. 12a, b, the weighted blending rule cannot avoid the blurring effect due to misalignment (red box in Fig. 13a), meanwhile, the mismatched alignment leads to the blurring road in the optimal seam based mosaic result (in the blue box of Fig. 13b). In addition, the restricted stitching seam searching direction of traditional dynamic programming algorithm makes the visible vertical seam unavoidable, as shown in the red box of Fig. 13b. Regardless of the negative effect caused by local/global uneven intensity and misalignment, our method can generate visually pleasant seamless mosaic result (Fig. 13c), which is free of blurring effect and structural break.
4 Conclusions and Discussions
In this paper, a fusion based seamless mosaic method is proposed to generate visually pleasant stitching result with partially overlapped remote sensing images. Considering the merits and drawbacks of both the weighted blending rule and the optimal seam searching strategy, they are integrated into one framework to ensure the intensity and structural consistence simultaneously. This seamless mosaic process is implemented by firstly decompose each aligned image into two sub-components, followed by various mosaic schemes are designed to stitch different sub-components. For low-frequency components consisting of coarse shape and illumination information, the 2-D weighted blending rule is utilized to achieve smooth intensity transition. For the high-frequency components containing important structure and texture information, a modified optimal seam searching scheme is proposed to guide the stitching process, and as a result, structural break can be avoided. The excellent performance in the experimental section demonstrates the effectiveness of the proposed mosaic method in avoiding visible stitching seam and blurring effect, especially in the cases of misalignment and intensity difference.
This paper mainly focuses on generating seamless mosaic result from two overlapped images with the proposed fusion based framework. In practice, to generate seamless mosaic image of large area, quantities of remote sensing images or videos are needed. Meanwhile, more complicated transformation may exist between images, including shift, rotation and scaling. Therefore, the emphasis of our future work will be laid on adjusting the proposed fusion based mosaic framework to stitch series of images or aerial/satellite videos automatically and quickly.
References
Grandi, G. D., Bouvet, A., Lucas, R. M., Shimada, M., Monaco, S., & Rosenqvist, A. (2011). The K&C PALSAR mosaic of the african continent: Processing issues and first thematic results. IEEE Transactions on Geoscience and Remote Sensing, 49(10), 3593–3610.
Collings, S., Caccetta, P., Campbell, N., & Wu, X. L. (2011). Empirical models for radiometric calibration of digital aerial frame mosaics. IEEE Transactions on Geoscience and Remote Sensing, 49(7), 2573–2588.
Yang, A., Li, X., Xie, J., & Wei, Y. (2013). Three-dimensional panoramic terrain reconstruction from aerial imagery. Journal of Applied Remote Sensing, 7(1), 073497.
Zomet, A., Levin, A., Peleg, S., & Weiss, Y. (2006). Seamless image stitching by minimizing false edges. IEEE Transactions on Image Processing, 15(4), 969–977.
Peng, Y. X., Ying, S. H., Qin, J., & Zeng, T. Y. (2013). Trimmed strategy for affine registration of point sets. Journal of Applied Remote Sensing, 7(1), 073468.
Sibiryakov, A., & Bober, M. (2006). Image registration using RST-clustering and its application in remote sensing. SPIE Proceedings, 6365, 63650G.
Zaragoza, J., Chin, T. J., Brown, M. S., & Suter, D. (2013). As-projective-as-possible image stitching with moving DLT. In Proceedings of IEEE conference on computer vision and pattern recognition, Portland (pp. 2339–2346).
Szeliski, R. (2006). Image alignment and stitching: A tutorial. Foundations and trends in computer graphics and Computer Vision, 2(1), 1–104.
Uyttendaele, M., Eden, A., & Szeliski, R. (2001). Eliminating ghosting and exposure artifacts in image mosaics. In Proceedings of IEEE conference on computer vision and pattern recognition, Kauai, USA, pp. 509–516.
Shum, H. Y., & Szeliski, R. (2000). Construction of panoramic image mosaics with global and local alignment. International Journal of Computer Vision, 36(2), 101–130.
Schechner, Y. Y., & Nayar, S. K. (2003). Generalized mosaicing: High dynamic range in a wide field of view. International Journal of Computer Vision, 53(3), 245–267.
Wang, W., & Ng, M. K. (2013). A variational approach for image stitching I. SIAM Journal on Imaging Sciences, 6(3), 1318–1344.
Tao, M. W., Johnson, M. K., & Paris, S. (2013). Error-tolerant image compositing. International Journal of Computer Vision, 103(2), 178–189.
Levin, A., Zomet, A., Peleg, S. & Weiss, Y. (2004). Seamless image stitching in the gradient domain. In Proceedings of European conference on computer vision, Prague, Czech Republic, pp. 377–389.
Dijkstra, E. W. (1995). A note on two problems in connexion with graphs. Numerische Mathematik, 1(1), 269–271.
Davis, J. (1998). Mosaics of scenes with moving objects. In Proceedings of IEEE conference on computer vision and pattern recognition, Santa Barbara, USA, pp. 354–360.
Chon, J., Kim, H., & Lin, C. (2010). Seam-line determination for image mosaicking: A technique minimizing the maximum local mismatch and the global cost. ISPRS Journal of Photogrammetry and Remote Sensing, 65(1), 86–92.
Avidan, S., & Shamir, A. (2007). Seam carving for content-aware image resizing. ACM Transanctions on Graphics, 26(3), 10.
Pan, J., Wang, M., Li, D. R., & Li, J. (2009). Automatic generation of seamline network using area Voronoi diagrams with overlap. IEEE Transactions on Geoscience and Remote Sensing, 47(6), 1737–1744.
Wan, Y., Wang, D., Xiao, J., Wang, X., Yu, Y., & Xu, J. (2012). Tracking of vector roads for the determination of seams in aerial image mosaics. IEEE Geoscience and Remote Sensing Letters, 9(3), 328–332.
Jia, J. Y., & Tang, C. K. (2005). Eliminating structure and intensity misalignment in image stitching. In Proceedings of IEEE international conference on computer vision, Beijing, China, vol. 2, pp. 1651–1658.
Gracias, N., Mahoor, M., Negahdaripour, S., & Gleason, A. (2009). Fast image blending using watersheds and graph cuts. Image and Vision Computing, 27(5), 597–607.
Mehtre, B. M., Kankanhalli, M. S., Narasimhalu, A. D., & Man, G. C. (1995). Color matching for image retrieval. Pattern Recognition Letters, 16(3), 325–331.
Kalpoma, K. A., & Kudoh, J. I. (2007). Image fusion processing for IKONOS 1-m color imagery. IEEE Transactions on Geoscience and Remote Sensing, 45(10), 3075–3086.
Holtkamp, D. J., & Goshtasby, A. A. (2009). Precision registration and mosaicking of multicamera images. IEEE Transactions on Geoscience and Remote Sensing, 47(10), 3446–3455.
Wang, M., Pan, J., Chen, S. Q., & Li, H. (2005). A method of removing the uneven illumination phenomenon for optical remote sensing image. In Proceedings 2005 IEEE international geoscience remote sensing symposium (IGARSS), Seoul, Korea, pp. 3243–3246.
Xu, Z. P. (2014). Medical image fusion using multi-level local extrema. Information Fusion, 19, 38–48.
Elad, M., Starck, J.-L., Querre, P., & Donoho, D. L. (2005). Simultaneous cartoon and texture image inpainting using morphological component analysis. Applied and Computational Harmonic Analysis, 19(3), 340–358.
Kwatra, V., Schödl, A., Essa, I., Turk, G., & Bobick, A. (2003). Graphcut textures: Image and video synthesis using graph cuts. ACM Transactions on Graphics, 22(3), 277–286.
Efros, A. A., & Freeman, W. T. (2001). Image quilting for texture synthesis and transfer. In Proceedings of ACM SIGGRAPH, New York (pp. 341–346).
Acknowledgments
This paper is supported by the National Natural Science Foundation of China (No. 61172161) and the National Natural Science Fund for Distinguished Young Scholars of China (No. 61325007).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Lu, T., Li, S. & Fu, W. Fusion Based Seamless Mosaic for Remote Sensing Images. Sens Imaging 15, 101 (2014). https://doi.org/10.1007/s11220-014-0101-0
Received:
Revised:
Published:
DOI: https://doi.org/10.1007/s11220-014-0101-0