Keywords

1 Introduction

CCD (Charge Coupling Device) is broadly used in photoelectric detection and imaging fields, since its low noise, large dynamic range, high quantum efficiency and high charge conversion efficiency [1]. Different types of CCD are used in remote sensing application, such as TDI (Time Delayed and Integration) CCD, area array CCD and etc. In this chapter, frame transfer area array CCD is used in remote sensing camera, and the traditional push broom imaging mode of TDI CCD system is applied. In each exposure time, the camera is pushed forward the distance of one pixel for exposure imaging. Therefore, the image data of two adjacent frames has a one pixel difference in the TDI push broom direction.

To overcome the image quality degradation problem caused by the mismatch between pixel shift rate and transfer rate and earth rotation in traditional TDI technical, a digital TDI method based on curve fitting algorithm is proposed in this chapter. Firstly, displacement of adjacent two frame images output by area array CCD is estimated. Secondly, displacement is corrected used Lagrange interpolation polynomial. Finally, the corrected image data is output after digital time delay integration. In low luminance conditions, this method can gain high sensitivity, high output rate and large dynamic range. The negative factor of low SNR caused by the harsh environmental conditions is greatly improved.

2 The Working Principle of TDI CCD

TDI CCD is a special linear CCD. It adopts time delay integration (TDI) technique [2] and can obtain high sensitivity, high output rate and large dynamic range in low luminance condition. The working process of traditional analog TDI CCD is shown in Fig. 1. In the first integration period, the object is exposed and integrated in the first pixel of one column. The Photo-generated charge is not read out as ordinary CCD but transferred to the next pixel of this column. In the second integration period, object is moved, exposed and integrated in the second pixel of this column. The Photo-generated charge newly gained adding the charge transferred from previous pixel then transfer to the next pixel in this column, and so on. In the Nth integration period, object has moved to Nth pixel of this column to expose and integrate. After this integration time, photo-generated charge of the Nth pixel added the charge transferred from previous (N-1) pixels transfer to readout resister, and the whole charge is read out as the ordinary linear CCD.

Fig. 1
figure 1

Charge transfer schematic of TDI CCD

Full size image

In this imaging mode, pixel shift rate should match charge transfer rate and pixel shift direction must follow the TDI direction. Otherwise, the image degradation in TDI direction (column direction) and its vertical direction (row direction) is introduced. On the one hand, due to the multistage accumulation integration character of TDI CCD, the clear image can be obtained by ensuring the match of pixel shift rate and charge transfer rate in a long time. On the other hand, because of earth rotation, the satellite moving direction is not same as the actual imaging direction which will cause image motion in one integration time and reduce image quality.

3 Digital TDI Working Mode

In this paper, frame transfer area array CCD is used in remote sensing camera, and the traditional push broom imaging mode used in TDI CCD system is applied. The imaging results when pixel shift rate and transfer rate is matching are given in Fig. 2, which assumes the size of area array CCD is 4 × 4. It can be seen from the figure that the image data of two adjacent frames has a one pixel difference in TDI direction. In this condition, the digital TDI working mode of area array CCD can be realized by adding the data corresponding to the same scene in different frame image.

Fig. 2
figure 2

The imaging results when pixel shift rate and transfer rate is matching

Full size image

As discussed in the previous section, the image degradation in push broom direction is introduced by the mismatch between pixel shift rate and transfer rate. When pixel shift rate is faster than transfer rate, the moving distance in one exposure time is larger than one pixel. This will cause an overlapping area between the adjacent two lines in each frame image. As shown in Fig. 3, the lower border of the first row of pixels and the upper border of the second row of pixels will sweep the gray area. Therefore, the second line in the first frame image and the first line in the second frame image are not corresponding to the same scene. In contrast, when pixel shift rate is slower than transfer rate, the moving distance in one exposure time is smaller than one pixel. This will cause an interspaced area between the adjacent tow lines in each frame image. As shown in Fig. 4, the lower border of the first row of pixels and the upper border of the second row of pixels both will not sweep the gray area.

Fig. 3
figure 3

The imaging results when pixel shift rate is faster than transfer rate

Full size image
Fig. 4
figure 4

The imaging results when pixel shift rate is slower than transfer rate

Full size image

And furthermore, due to the earth rotation, there is a displacement in the horizontal direction (which is vertical to the push broom direction) between adjacent frame images during remote sensing camera imaging.

Figure 5 gives the flow chat of digital TDI working mode. To realize the digital TDI working mode of area array CCD, the displacement in push broom direction and its vertical direction between two adjacent frames of image is estimated firstly. Secondly, the image is interpolated in accordance with the calculated displacement. Finally, the data corresponding to the same scene in different frame image is added.

Fig. 5
figure 5

The flow chart of digital TDI working mode

Full size image

  • A. Displacement Estimation

Theoretically, the variance between two images corresponding to the same scene at different time reflects their relative movement. A displacement estimation algorithm is proposed based on this idea. The flow chat of displacement estimation algorithm is given in Fig. 6.

Fig. 6
figure 6

The flow chart of displacement estimation algorithm

Full size image
  1. (1)

    Mean Filter

There is a variety of noises in the output signal of CCD [2], such as shot noise, dark current noise, fix pattern noise (FPN), thermal noise, read noise, and so on. The thermal noise and read noise are signal independent and like the white noise which mean value is 0. Therefore, in order to reduce the influence of random noise and improve the algorithm reliability, the mean filter is implemented to the image, the formula is as follows:

$$ g(x,y) = \frac{1}{(2a + 1)(2b + 1)}\sum\limits_{s = - a}^{a} {\sum\limits_{t = - b}^{b} f (x + s,y + t)} $$
(1)

where, x is the TDI direction (row direction), y is the CCD direction (column direction), f(x,y) is the input image, g(x, y) is the image after mean filter, (2a + 1) and (2b + 1) is the size of mean filter window. In order to save hardware resource, the mean filter of row and column direction is separated and executed respectively.

  1. (2)

    Variance Calculation

Theoretically, the variance between two images of different time corresponding to the same scene reflects their relative movement. When these two images are complete coincide, the variance is minimal. Therefore, the displacement of two image can be determined by calculated its variance in different phases.

The push broom method is used in this chapter camera. So the image data of adjacent frames have a one pixel difference in TDI push broom direction (shown in Fig. 2). The variance of adjacent frame images when they are completely overlap is calculated as follow:

$$ Var_{c0l0} = \frac{1}{(M - 1)N}\sum\limits_{x = 2}^{M} {\sum\limits_{y = 1}^{N} {(g_{i} (x,y) - g_{i - 1} (x + 1,y))^{2} } } $$
(2)

Similarly, the variance in TDI direction in different phase of adjacent frame images after mean filter can be calculated respectively. The formula is as follows:

$$ Var_{cpml0} = \frac{1}{{(M - m_{1} )N}}\sum\limits_{x = 1}^{{M - m_{1} }} {\sum\limits_{y = 1}^{N} {(g_{i} (x,y) - g_{i - 1} (x + m_{1} ,y))^{2} } } $$
(3)
$$ Var_{cnml0} = \frac{1}{{(M - m_{2} )N}}\sum\limits_{{x = m_{2} + 1}}^{M} {\sum\limits_{y = 1}^{N} {(g_{i} (x,y) - g_{i - 1} (x - m_{2} ,y))^{2} } } $$
(4)

In (3), m 1 = 2, 3, 4 are corresponding to the variance of phase difference 1, 2, 3 respectively. In (4), m 2 = 0, 1, 2 are corresponding to the variance of phase difference −1, −2, −3 respectively.

The variance in CCD direction of different phase can be calculated as follows:

$$ Var_{c0lpn} = \frac{1}{(M - 1)(N - n)}\sum\limits_{x = 2}^{M} {\sum\limits_{y = 1}^{N - n} {(g_{i} (x,y) - g_{i - 1} (x + 1,y + n))^{2} } } $$
(5)
$$ Var_{c0lnn} = \frac{1}{(M - 1)(N - n)}\sum\limits_{x = 2}^{M} {\sum\limits_{y = n}^{N} {(g_{i} (x,y) - g_{i - 1} (x + 1,y - n))^{2} } } $$
(6)

In (5), n = 0, 1, 2 are corresponding to the variance of phase difference 1, 2, 3 respectively. In (6), n = 0, 1, 2 are corresponding to the variance of phase difference −1, −2, −3 respectively.

  1. (3)

    Polynomial Fitting

Analysis shows the variance in different phase should satisfy some relationship. In this chapter, six-time polynomial is used to describe this relationship:

$$ V = c_{0} + c{}_{1}P + c_{2} P^{2} + c_{3} P^{3} + c_{4} P^{4} + c_{5} P^{5} + c_{6} P^{6} = \sum\limits_{i = 0}^{6} {c_{i} P^{i} } $$
(7)

where, the coefficients of six-time polynomial are calculated using the least squares algorithm [3].

  1. (4)

    Displacement Estimation

The minimum point of six-time polynomial showed in (7) is the displacement of two adjacent frame images. Since the six-time polynomial does not have numerical solution, the iterative approach is used to calculate its approximate solution.

During the motion of the satellite, the change of the displacement is a slow process. So the following iterative method is used in this chapter. Assuming the displacement calculated in the ith time is delt i . The value of six-time polynomial is calculated form delt i  − 0.005 to delt i  + 0.005, and the step is 0.0001. The displacement corresponding to the minimum point in this 101 points is adopted as the displacement delt i+1 calculated in (i + 1)th time.

  • B. Displacement Correction

Before executing the digital TDI, in order to make the image data of the same ground feature corresponding to the same region exactly, the image data need to be interpolated according to the displacement of row and column direction calculated above. The displacement correction along the push broom direction is shown in Fig. 7. Supposing the displacement of column direction is delt y , the processing for the situation that pixel shift rate is faster than transfer rate is as follows: the first line of each image remains intact, the second line is shifted delt y downward, the third line is shifted 2 × delt y downward, and the fourth line is shifted 3 × delt y downward. After this processing, the image data will correspond to the same region exactly.

Fig. 7
figure 7

The schematic diagram of displacement correction along push broom direction

Full size image

In this chapter, the seven-order Lagrange polynomial is used to shift and register the image. The calculation is as follows:

$$ lx_{i} = \prod\limits_{i = 0,i \ne j}^{7} {\frac{{x \times delt_{y} - j}}{i - j}} $$
(8)
$$ z_{x} (x,y) = \sum\limits_{i = 0}^{7} {lx_{i} f} (x + i - 3,y) $$
(9)

where, i, j = 0, 1, 2,…,7, x = 4, 5, \( \ldots \), M-4, f(x, y) is the input image.

From the foregoing depiction, there is a displacement between two adjacent frames of image in the horizontal direction. Supposing the displacement of this direction is delt x . The displacement between the first frame image and the second frame image is delt x . The displacement between the first frame image and the third frame image is 2 × delt x . So the displacement correction in row direction is same as column direction. The first line of each image remains intact, the second line is shifted delt x in lateral, the third line is shifted 2 × delt x in lateral, and the fourth line is shifted 3 × delt x in lateral. The shift and register using the seven-order Lagrange polynomial is as follows:

$$ ly_{i} = \prod\limits_{i = 0,i \ne j}^{7} {\frac{{y \times delt_{x} - j}}{i - j}} $$
(10)
$$ z(x,y) = \sum\limits_{j = 0}^{7} {ly_{i} z_{x} } (x,y + i - 3) $$
(11)

where, \( y = 1,2, \ldots ,N \). To make the image after processing has the same number in row direction, even extension is executed for the boundary pixel.

  • C. Digital TDI realization method

After the displacement correction in row and column direction completed, the digital TDI working mode can be realized by adding the data corresponding to the same scene in different frame image by using FPGA. The formula is as follows:

$$ T(x,y) = \frac{1}{N}\sum\limits_{i = 1}^{N} {z_{i + x - 1} } (N - i + 1,y) $$
(12)

where, N is the TDI stage, z i+x−1 is the image after correction.

4 Experiment Result

In this chapter, the Digital TDI operation method of array CCD is tested on an imager developed by our institute. There is a relationship between the SNR of TDI CCD and its stage. The signal of N stage TDI-CCD increase N times while the noise only increases \( \sqrt N \) times, which indicates the SNR improve \( \sqrt N \) [4]. Table 1 shows the Mean Value, Noise, SNR (dB) results of the images in different TDI stages. Because the image data after digital TDI is divided by N in formula (12), so the output image has the same mean value in different TDI stage. It is not difficult to find that image noise (root-mean-square error) is reduced to half of its origin when the TDI stage is increased 4 times, so the SNR is almost improved 6 dB.

Table 1 The SNR of different TDI stage
Full size table

5 Conclusion

In this chapter, on the basis that frame transfer area array CCD can work properly, according to the working principle of TDI CCD, a digital TDI operation mode of area array CCD based on curve fitting using FPGA is presented. This method can obviously improve the image degradation caused by the mismatch between pixel shift rate and charge transfer rate and the earth rotation. It can be broadly used in push broom imaging system using area array CCD. The experimental results show the digital TDI operation method proposed in this chapter can improve SNR of image significantly, and enhance the imaging performance of remote sensing camera.