1 Introduction

Remote sensing techniques can provide an abundance of useful information, which plays an important role in both military and civilian domain, such as battlefield investigation and marine resources surveillance. Due to the limited field-of-view of sensors, the area of interest always cannot be occupied in one obtained image. Therefore, it is essential to generate large-scale composition image with a sequence of overlapped remote sensing images, which is usually referred to as image mosaic technique [1–3].

Generally, the following two desirable properties are required in generating visually pleasant mosaic result [4]. First, the result should be as similar as possible to the input images in terms of intensity and geometry. Second, the stitching line should be invisible. However, the overlapped images obtained from different time, angles and sensors always present global/local intensity difference. Besides, the misalignment, caused by the inaccurate registration or the presence of moving objects and motion parallax, is another common issue in the process of mosaic. Both the intensity difference and misalignment will introduce visible stitching seam, blurring effect and structure inconsistence. Assuming that images with partially overlapped regions have already been aligned by some image registration methods [5–7], image mosaic methods can be categorized into two methods: weighted blending based method and optimal seam based method [8].

The weighted blending based methods, commonly called as alpha blending or feathering [9–11], aim at achieving smooth transition in the overlapped region. Specifically, the mosaic result is considered as a weighted combination of both images, where the weight coefficient is calculated according to the distance from the stitching boarders or the centers. More advanced weighted blending techniques are developed by using variational approach [12] or Poisson-based gradient-domain compositing approach [13]. However, the blending based mosaic methods have the drawback that ghost and blurring artifacts may be introduced if overlapped images are misaligned [4, 14].

The optimal seam based methods intend to search a curve in the overlapping area, on which the two registered images have the minimum difference. With the overlapping area being divided into two parts by the searched curve, each part can then be filled with the pixels in the corresponding image. Many optimal seam searching methods have been developed in the literature, including Dijkstras algorithm based methods [15–17], dynamic programming based methods [18–20], and graph-cuts based methods [21, 22]. However, the optimal seam based methods fail in generating visually pleasant mosaic results when it comes to the case of intensity difference. To avoid the visual inconsistence caused by unbalanced intensity, histogram matching based pre-processing has been extensively considered [23–25]. As one statistic based intensity balancing method, the histogram matching is effective in adjusting the global intensity distribution of the target image to fit the reference image through nonlinear transformation. But in the case that the overlapped images have quite different scenery distribution (e.g. 10 % field and 90 % river in one image, while 90 % field and 10 % river in another one), histogram based balancing method is no longer appropriate. Meanwhile, the local intensity difference cannot be eliminated via histogram matching.

Considering the merits and drawbacks of the weighted blending based method and the optimal seam based method, both methods are integrated appropriately into one framework to improve the mosaic performance. First, low-pass filter is applied to decompose each overlapped image into high-frequency component and low-frequency component. Then different mosaic schemes are designed for stitching each component. For low-frequency components consisting of uneven illumination information, the weighted blending rule based mosaic scheme is introduced to achieve smooth intensity transition. With no detail information in the low-frequency components, the weighted blending rule can be effective in avoiding ghost or blurring effect. For high-frequency components consisting of rich details (edges, structures and texture, etc.), a modified optimal seam searching method is designed to determine the seam line and guide the stitching process within the overlapping area [18–20]. With no uneven intensity information in the high-frequency components, the modified optimal seam based mosaic method can work well in preserving structure and texture consistence. Finally, the mosaic result can be obtained via linearly composing both mosaic results corresponding to the components of high-frequency and the low-frequency together.

The rest of the paper is organized as follows. In Sect. 2, we described the proposed seamless mosaic method in detail. Then we show the experimental mosaic results on real remote sensing images by our method in Sect. 3. Meanwhile, the comparison results of different mosaic methods are also given in Sect. 3. Finally, the conclusions are drawn in Sect. 4, and the focus of the future related work is also referred in this section.

2 The Proposed Method

Let \(I_1\) and \(I_2\) represent two overlapped remote sensing images respectively, the proposed remote sensing image mosaic method consists of four parts: low-pass filtering based image decomposition, weighted blending based mosaic for low-frequency components, optimal seam based mosaic for high-frequency components, and the composition of both mosaic results of different components. The first step aims at decomposing each registered image into high-frequency component \(I_{iH}(i=1,2)\) and the low-frequency component \(I_{iL}(i=1,2)\), denoted as \(I_i=I_{iH}+I_{iL}(i=1,2)\). Then, we can generate two mosaic results with different mosaic schemes, denoted as \(I_{H-mosaic}\) and \(I_{L-mosaic}\), respectively. Finally, the mosaic result can be obtained via composing mosaic results of different components, mathematically denoted as \(I_{mosaic}=I_{L-mosaic}+I_{H-mosaic}\). The framework is shown in Fig. 1.

Fig. 1
figure 1

The framework of the proposed seamless image mosaic method

Full size image

2.1 Low-Pass Filtering Based Image Decomposition

The process of image decomposition aims at separating the original images \(I_1\) and \(I_2\) into coarse component and the detail component. The coarse component mainly consists of profile and illumination information, and the detail component only contains details such as edges, structures and textures. To fulfill this decomposition process, related works have been done in [26–28], varying from easy low-pass/high-pass filtering [26] to wavelet transformation [27] and the morphological component analysis [28]. In our method, the Gaussian low-pass filtering is adopted for the sake of low computational cost, as recommended in [26]. The Gaussian low-pass filtering is defined as:

$$\begin{aligned} g(x,y) = exp({-(x^2+y^2)/{2{D_0}^2}}) \end{aligned}$$
(1)

where x is the distance from the origin in the horizontal axis, y is the distance from the origin in the vertical axis, and D 0 is the cut-off frequency. One example of Gaussian low-pass filtering based decomposition result is given in Fig. 2. It is clear from Fig. 2 that such decomposition method is effective in generating the coarse component (low-frequency component, \(I_L\)) and detail component (high-frequency component, \(I_H\)).

Fig. 2
figure 2

One example of the low-pass filtering based image decomposition

Full size image

2.2 Weighted Blending Based Low-Frequency Components Mosaic

For the low-frequency components \(I_{1L}\) and \(I_{2L}\), the weighted blending rule is applied to generate the mosaic result \(I_{L-mosaic}\). Let \(\varPhi _{0}\) denotes the overlapping region, \(\varPhi _{1}\) and \(\varPhi _{2}\) denote the non-overlapping regions belonging to image \(I_{1L}\) and \(I_{2L}\), respectively. Traditionally, the 1-D weighted blending rule is defined as:

$$\begin{aligned} I_{mosaic}(x,y)={\left\{ \begin{array}{ll} I_{1L}(x,y),&{} (x,y)\in {\varPhi _{1}} \\ \lambda \cdot {I_{2L}(x,y)}+(1-\lambda )\cdot {I_{1L}(x,y)},&{} (x,y)\in {\varPhi _{0}} \\ I_{2L}(x,y),&{} (x,y)\in {\varPhi _{2}} \end{array}\right. } \end{aligned}$$
(2)

where λ is weight coefficient calculated by λ = d/W, in which d represents the distance from the current point to the border of the overlapping area and W represents the width of the overlapping region. One example is given in Fig. 3. It is clear that the two images in Fig. 3a, b have difference in local intensity, which would lead to visible seam if pasting both images together directly. Instead, with the 1-D blending rule defined as Eq. (2), applied to stitch \(I_{1L}\) and \(I_{2L}\), visually smooth transmission can be achieved.

Fig. 3
figure 3

1-D weighted blending rule based mosaic method for low-frequency components. a The low-frequency component of image 1. b The low-frequency component of image 2. c 1-D weighted blending rule based mosaic result of (a, b)

Full size image

When it comes to the case of 2-D shift transformation (as shown in Fig. 4a), the weight coefficient λ is adjusted to:

$$\begin{aligned} \lambda = \mu _1/(\mu _1+\mu _2) \end{aligned}$$
(3)
$$\begin{aligned} \mu _1=(H-y)/H, \mu _2=(W-x)/W \end{aligned}$$
(4)

where H and W respectively represents the height and width of the overlapping regions, y and x respectively denote the relative vertical and horizontal location to the borders, as illuminated in Fig. 4. One example of applying 2-D blending rule in stitching low-frequency components is given in Fig. 4c, and it is clear that smooth transition can be achieved from left to right and from up to down.

Fig. 4
figure 4

2-D weighted blending rule based mosaic method for low-frequency components. a Illustration of 2-D weighted blending rule. b Aligned result. c 2-D weighted blending rule based mosaic result

Full size image

2.3 Optimal Seam Based High Frequency Components Mosaic

The two registered images of high-frequency components, \(I_{1H}\) and \(I_{2H}\), mainly consist of rich structures and textures. Considering the weighted blending based mosaic method would fail in avoiding blurring and double-edge in the case of misalignment, an optimal seam searching strategy is utilized to guide the process of stitching \(I_{1H}\) and \(I_{2H}\). First, by using sum of squared differences (SSD), one energy function is defined as [29]:

$$\begin{aligned} e(x,y)=\sum (Block_1(x,y)-Block_2(x,y))^2 \end{aligned}$$
(5)

where \(Block_1(x,y)\) and \(Block_2(x,y)\), respectively, represent image patches surrounding each pixel (x, y) in images \(I_{1H}\) and \(I_{2H}\), and the block size is \(N\times {N}\). Then we will search for a path in the overlapping area along which the two textures can match best, i.e. where the SSD error is the lowest. The optimal mosaic path C is a curve crossing from top to bottom of the overlapping region with exactly one point on each row. Following the dynamic programming based optimal seam searching methods [29, 30], one general cumulative minimum energy is defined as:

$$\begin{aligned} E(x,y)=e(x,y)+min(E(x-1,y-1),E(x-1,y),E(x-1,y+1)) \end{aligned}$$
(6)

With the energy values for all pixels in the overlapping regions obtained, the next step is to trace back the process of cumulative minimum energy calculation from the pixel with the minimum energy in the last row. Then the optimal seam C (marked with red line in Fig. 5) with cumulative minimum energy is decided. Finally, the mosaic result in the overlapping area \(\varPhi _0\) can be divided into \(\varPhi _{01}\) and \(\varPhi _{02}\) by the curve C, where the region \(\varPhi _{01}\) and \(\varPhi _{02}\) are respectively filled with pixels belonging to image \(I_{1H}\) and \(I_{2H}\). The illustration is given in Fig. 5, where the red line represents the determined optimal line. The traditional dynamic programming based methods can work effectively in the case of 1-D shift transformation.

Fig. 5
figure 5

Optimal seam based mosaic method for high-frequency components in the case of 1-D shift transformation

Full size image

However, the restriction in seam searching direction (either from top to down or from left to right) easily leads to the failure in dealing with the case of 2-D shift transformation in case of intensity difference. It is clear from Fig. 6a that the searched red line by 1-D searching scheme could not connect the specified point A and point B. As a result, two visible horizontal seams respectively originated from point A and point B are unavoidable if two overlapped images have intensity difference. Considering 2-D shift transformation is a more general situation in stitching remote sensing images, it is of great importance to design a better stitching seam searching scheme to avoid obvious artifacts. In our method, one modified optimal seam searching scheme is designed based on the dynamic programming algorithm. First, one optimal seam (marked with the red line in Fig. 6a) from the upmost to the downmost of the overlapping area is searched with the traditional dynamic programming algorithm [30]. Second, two another seams are searched in a similar way from the leftmost to the rightmost (as marked with the blue line in Fig. 6) and from the rightmost to the leftmost (as marked with the green line in Fig. 6). Differently, the blue (yellow) seam, we firstly calculate the energy of all the paths from right to left (left to right) with Eq. (6). Then we trace back this process from the point A (B) to determine the blue (yellow) seam. Finally, the seam connecting the point A and B can be determined to separate the overlapping area \(\varPhi _0\) into two parts, denoted as \(\varPhi _{01}\) and \(\varPhi _{02}\) (as shown in Fig. 6). By filling the areas \(\varPhi _{01}\) and \(\varPhi _{02}\) with the corresponding pixels in the left image \(I_1\) and the right image \(I_2\) respectively, the mosaic result of high-frequency components can be obtained.

Fig. 6
figure 6

The modified optimal seam based mosaic method for high-frequency components in the case of 2-D shift transformation

Full size image

One example is shown in Fig. 7. It is clear that obvious structural break (seen more clearly in the zoomed area in the red box) can be seen in the aligned result (Fig. 7a) due to the misalignment. By applying the modified seam searching scheme in stitching the high-frequency components, consistent structure can be achieved in the area marked with blue box in Fig. 7b.

Fig. 7
figure 7

Example of the modified optimal seam based mosaic method for high-frequency components. a Aligned result. b Modified optimal seam based mosaic result

Full size image

3 Experiments

3.1 Experimental Setup

In this section, the performance of the proposed fusion based mosaic method is tested on three groups of remote sensing image pairs. As shown in Fig. 8, the test image pairs consist of one Landsat image pair and two optical aerial image pairs. For the sake of convenience, three groups of test images are respectively named by “Landsat” (Fig. 8a), “Aerial 1” (Fig. 8b) and “Aerial 2” (Fig. 8c). These test image pairs are selected as varieties of sceneries are contained (e.g. mountain, field, city and river, etc.).

Fig. 8
figure 8

Three groups of test remote sensing image pairs. a Image pair, “Landsat”. b Image pair, “Arial 1”. c Image pair, “Arial 2”

Full size image

To evaluate the performance in generating seamless mosaic result, our method is compared with both the weighted blending rule based and the optimal seam based mosaic methods. The blending rule is achieved by Eqs. (3) and (4), while the dynamic programming based optimal seam searching scheme is implemented according to [30]. In our method, two necessary parameters are needed to be set: the cut-off frequency D 0 in Eq. (1) and the patch size N × N in Eq. (5). To demonstrate the effect of these two parameters in mosaic results, we repeat the mosaic experiments with only one parameter (D 0 or N) being varied and the other one being settled. Two groups of test experimental results are given in Figs. 9 and 10.

Fig. 9
figure 9

Influence of parameter D 0 in mosaic results of image pair “Arial 2”. a D 0 = 0. b D 0 = 0.03. c D 0 = 0.07. d D 0 = 0.1. e D 0 = 0.15. f D 0 = 0.3

Full size image

Firstly, the influence of D 0 is tested with N = 3. As shown in Fig. 9a, broken structure can be seen in the case of D 0 = 0. This is because small value of D 0 leads more illumination information to be separated in the high-frequency component, however, the optimal seam based method for mosaicing high-frequency components fails in determining an optimal stitching seam if there is illumination difference. On the other hand, more structural information would be separated in low-frequency component as the value of D 0 increases, which may generate varying degrees of blurring effect, as shown in Fig. 9d–f (D 0 ≥ 0.1). This blurring effect is caused by applying the blending rule in mosaicing the low-frequency components containing structural information. Therefore, the selection of D 0 should be neither too high nor too low, which is suggested to be values between [0.03, 0.07] according to the experimental results. In our experiments, D 0 is set to be 0.03.

Then, with the D 0 settled to be 0.03, the value of N is varied to test the influence of N. In case of inaccurate registration, unexpected optimal seam can be determined and it would lead to failure in structure preservation which can be seen more clearly in the zoomed blue box of Fig. 10a (N = 1). To generate better stitching seam,the related pixels in a neighbored window are taken into considered. However, large size (N ≥ 5) of neighboring window cannot well eliminate structure inconsistence, as shown in Fig. 10c–e.Therefore, N is set to be three to generate better seamless mosaic results.

Fig. 10
figure 10

Influence of parameter N in mosaic results of image pair “Arial 2”. a N = 1. b N = 3. c N = 5. d N = 7. e N = 9

Full size image

3.2 Experimental Results

In this section, the mosaic performance is tested on three remote sensing image pairs (shown in Fig. 8), and the visual comparison results of various mosaic methods are given in Figs. 11, 12 and 13.

Fig. 11
figure 11

Comparison mosaic results of test image pair “Landsat”. a 2-D weighted blending based method. b Optimal seam based method. c The proposed fusion based method

Full size image
Fig. 12
figure 12

Comparison mosaic results of test image pair “Aerial 1”. a The blending rule based method. b The optimal seam based method. c The proposed fusion based method

Full size image
Fig. 13
figure 13

Comparison mosaic results of test image pair “Aerial 2”. a The blending rule based method. b The optimal seam based method. c The proposed fusion based method

Full size image

Figure 11 shows the comparison results of stitching image pair “Landsat”. The image pair “Landsat”(Fig. 8a) presents no intensity difference, and it is used to test the performance in handling the case of misalignment. Practically, we misalign the images “Landsat” with five pixels respectively in vertical and horizontal directions. It is obvious from Fig. 11a, b that both the 2-D weighted blending rule based and the optimal seam based methods are affected by the misalignment. Specifically, blurring and ghost phenomenon appears in the mosaic result obtained via blending rule based method, which can be noticed more clearly in the area marked with red box in Fig. 11a. Though the optimal seam based mosaic method is free of blurring effect, obvious structural break is introduced in the overlapping area, as shown in the red box of Fig. 11b. By contrast, our method can achieve visually smoothing transition in terms of intensity and structure simultaneously (Fig. 11c), meanwhile, no blurring effect is introduced.

Comparison results of another two groups of test image pairs, the “Aerial 1” and the “Aerial 2”, are respectively shown in Figs. 12 and 13. It is clear that local color tone variance (bright spot in the overlapping area in Fig. 8b) exists in “Aerial 1”, meanwhile the “Aerial 2” has global variance in terms of color tone. These two image pairs are utilized to evaluate the mosaic performance in dealing with both the problems of intensity variance and misalignment.

Figure 12 shows the comparison mosaic results corresponding to image pairs “Aerial 1”. It is clear from the blue box in Fig. 12b that visible seam can be seen in the mosaic result by the optimal seam based method. By contrast, both the blending rule based method and our method can achieve smooth transition in the lake area, as shown in blue boxes in Fig. 12a, c. However, the blending rule applied in the original images leads to serious artifact of double-edge in the overlapping area due to the misalignment, which can be seen more clearly in the red box of Fig. 12a. By integrating both the 2-D blending rule and the modified optimal seam searching scheme into one framework, our method can successfully avoid both artifacts and eliminate obvious seam simultaneously (Fig. 12c).

Figure 13 shows the comparison results of stitching image pair “Aerial 2”. In accordance with the result shown in Fig. 12a, b, the weighted blending rule cannot avoid the blurring effect due to misalignment (red box in Fig. 13a), meanwhile, the mismatched alignment leads to the blurring road in the optimal seam based mosaic result (in the blue box of Fig. 13b). In addition, the restricted stitching seam searching direction of traditional dynamic programming algorithm makes the visible vertical seam unavoidable, as shown in the red box of Fig. 13b. Regardless of the negative effect caused by local/global uneven intensity and misalignment, our method can generate visually pleasant seamless mosaic result (Fig. 13c), which is free of blurring effect and structural break.

4 Conclusions and Discussions

In this paper, a fusion based seamless mosaic method is proposed to generate visually pleasant stitching result with partially overlapped remote sensing images. Considering the merits and drawbacks of both the weighted blending rule and the optimal seam searching strategy, they are integrated into one framework to ensure the intensity and structural consistence simultaneously. This seamless mosaic process is implemented by firstly decompose each aligned image into two sub-components, followed by various mosaic schemes are designed to stitch different sub-components. For low-frequency components consisting of coarse shape and illumination information, the 2-D weighted blending rule is utilized to achieve smooth intensity transition. For the high-frequency components containing important structure and texture information, a modified optimal seam searching scheme is proposed to guide the stitching process, and as a result, structural break can be avoided. The excellent performance in the experimental section demonstrates the effectiveness of the proposed mosaic method in avoiding visible stitching seam and blurring effect, especially in the cases of misalignment and intensity difference.

This paper mainly focuses on generating seamless mosaic result from two overlapped images with the proposed fusion based framework. In practice, to generate seamless mosaic image of large area, quantities of remote sensing images or videos are needed. Meanwhile, more complicated transformation may exist between images, including shift, rotation and scaling. Therefore, the emphasis of our future work will be laid on adjusting the proposed fusion based mosaic framework to stitch series of images or aerial/satellite videos automatically and quickly.