Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Image registration is the process of spatially aligning two images of a scene so that corresponding points assume the same coordinates. This process enables finding, for each point in the first image, the corresponding point in the second image. In this monograph, the first image will be referred to as the reference image and the second image will be referred to as the sensed image. The reference image is kept unchanged and the sensed image is transformed to take the geometry and spatial coordinates of the reference image.

If the images represent different views of a 3-D scene, or if the scene represents a dynamic environment, it may not be possible to find correspondence between all points in the images. Image registration aims to find correspondence between points that are present in both images.

An example of image registration is given in Fig. 1.1. The reference image is a Landsat multispectral (MSS) image of Kalkaska County, Michigan, and the sensed image is a Landsat thematic mapper (TM) image of the same area. Registration involves spatially aligning the TM image with the MSS image. By registering the images, it becomes possible to fuse information in the images or identify differences between the images.

Fig. 1.1
figure 1

(a) A Landsat MSS image and (b) a Landsat TM image of Kalkaska County, Michigan. (c) The TM image is geometrically transformed to spatially align the MSS image

Full size image

Another example of image registration is given in Fig. 1.2. Reference and sensed images show downtown Honolulu, Hawaii. By registering the images using their overlap area, an image mosaic is created that contains areas covered by both images.

Fig. 1.2
figure 2

(a), (b) Images of downtown Honolulu, Hawaii. (c) Registration of the images and creation of an image mosaic

Full size image

The word registration can be traced back to the year 1900 in a US patent by Becker [5]. In this patent, Becker discloses a focusing camera that uses a half-mirror and a full mirror to create two images of a scene on the viewer’s retina. By changing the orientation of one mirror with respect to the other, images from the two mirrors are aligned on the viewer’s retina. The mechanism that changes the orientation of one mirror with respect to the other also changes the distance of the image plane to the lens. The mechanism is designed in such a way that when images from the two mirrors perfectly align on the viewer’s retina, the film also moves to the right distance to the lens, enabling creation of a sharp image of the scene on the film.

The next advancement in image registration is observed in the film industry in the production of a double-coated color film. In an apparatus disclosed by Kelley and Mason in a US patent in 1917 [15], a color film is created by recording the blue band on one side of the film and a combination of red and green bands on the opposite side of the film. The challenge in creating a color film is to print two images of the same scene taken separately in perfect alignment on the film. The disclosed apparatus achieved alignment of different color bands with high precision using two registration pins that fit into the perforations in the edges of the films.

The next natural use of image registration happened in printing. Seaman, in a US patent in 1937 [29], disclosed an apparatus for printing overlapping images of a scene captured separately into a more complete image. The invention not only allowed the creation of an image mosaic from two or more overlapping images, a masking mechanism was introduced that made it possible to add figures captured from one scene to the photograph of another scene.

Image registration as we know it today emerged as an electronic image comparison device. Dressler, in a US patent in 1956 [7], disclosed an electronic device for comparing and combining two overlapping images. The images were provided on films. Through the use of a half mirror and a full mirror, the images were projected to an image subtraction circuitry that produced an output with an amplitude proportional to the degree of match between the images. By providing the means to translate and rotate one film with respect to the other and by observing the output of the subtraction circuitry, the capability to align and combine images from two films into a larger image was provided.

The electronic image comparison idea of Dressler was later extended to an image correlator. Link and Smith, in a US patent in 1962 [18], disclosed an electronic tube that could store a reference image and correlate it with a sensed image, producing an output signal with its strength a function of the correlation between the images. Steiner, in a US Patent in 1965 [31], further advanced the tube to make it possible to move the sensed image with respect to the reference image and align the images. Johnson, in a US Patent in 1969 [14], further improved the apparatus to normalize the correlation measure so that the output amplitude would be independent of the amplitude of the sensed image, thereby measuring the degree of match between the reference and sensed images more accurately.

The first example of digital image registration is traced back to the work of Roberts at MIT in 1963 [27]. By aligning the projections of edges of model polyhedral solids with image edges, Roberts developed a computational method for locating and recognizing predefined polyhedral objects in an image.

Methods for registering full digital images first appeared in the remote sensing literature. Anuta [1, 2] developed an automatic method for registering images with translational differences. Efforts to speed up the method and make it less sensitive to intensity differences between the images were made by Leese et al. [16], Barnea and Silverman [4], and Pratt [26] shortly after. The use of image registration in robot navigation was pioneered by Mori et al. [23], Levine et al. [17], and Nevatia [24]. Image registration found its way to medical imaging as data from medical scanners became digitally available [3, 30, 35].

While registration activities during 1970s focused on alignment of satellite images using rigid, similarity, and affine transformation functions, increased production of low-altitude aerial images during 1980s was the driving force behind invention of methods that could spatially align images with local geometric differences [9–13]. Due to the increased use of medical images during 1990s and the need for spatially aligning multimodality image, considerable advances were made in formulation of information theoretic similarity/dissimilarity measures that could compare and register multimodality images [8, 19, 20, 22, 25, 32, 34].

During the last decade, due to the increased use of videos in surveillance and other real-time applications, image registration became a necessary step in analysis of single and multi source videos. Advances in the imaging technology to increase resolution and quality of images, also increased the complexity of methods that can register such images. Intensity-based registration methods gradually lost ground to point-based registration methods that could accommodate local geometric differences between images.

During the past several years, considerable efforts have gone into locating unique points in images [28, 33] and finding the correspondence between them [6, 21]. Locally unique points in two images that are used to register the images will be referred to as control points in this monograph.

The general steps involved in image registration and the relation between them are shown in Fig. 1.3. An image registration system can be considered a black box that receives a reference image and a sensed image and resamples the sensed image to spatially align with the reference image. This operation assigns the same coordinates to corresponding points in the images, defining both images in the same coordinate system.

Fig. 1.3
figure 3

Components of an image registration system and the relation between them

Full size image

Depending on the severity of the intensity and geometric differences between the reference and sensed images, different steps may be needed to register the images. If the images can be treated as rigid bodies, they will have only translational and rotational differences. By translating and rotating the sensed image with respect to the reference image, the images can be registered. Early methods achieved image registration in this manner. The process is simple but it is not efficient. It is also limited to images that have only translational and rotational differences. Moreover, the presence of outliers in one or both images can break down the registration process.

A more robust approach will select a number of control points in the reference image and will try to locate them in the sensed image. This is achieved by selecting small windows centered at the points in the reference image and searching for the windows in the sensed image. If the images have only translation and rotational differences, each correct match will produce the same rotational parameter. By matching multiple windows in the images, the rotation parameter shared by two or more matches can be used to register the images even in the presence of outliers.

If the geometries of the images are not related by a rigid transformation, a sequence of steps as shown in Fig. 1.3 are needed to register the images. First, a number of control points is selected in each image. Then, features describing the neighborhoods of the points are calculated and the most informative features are selected. Initial correspondence is established between the points using the features of the points. Additional information about the images is then used in the form of constraints to distinguish the correct correspondences from the incorrect ones. Once a set of corresponding points in the images is found, the parameters of a nonlinear function to transform the space of the sensed image to that of the reference image are determined. The chapters containing the details of each step in image registration are also included in Fig. 1.3.

Chapters 2–10 describe the tools needed to design image registration methods for various applications and the underlying principles. Well-known registration methods that are designed by these tools are reviewed in Chap. 11.