Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Introduction

Medical imaging is of vital importance in modern medicine and of special interest in diagnostic medicine. Current medical imaging consists of a wide range of instrumentation including but not limited to computed tomography (CT), magnetic resonance imaging (MRI), ultrasonography (US), fluoroscopy, plain film radiography, positron emission tomography (PET), and single-photon emission computed tomography (SPECT). Each of these imaging modalities has certain advantages and limitations.

Clear advantages of one imaging system can be a limitation of another. Hence fusion of these imaging technologies may provide an approach for maximizing imaging information, as illustrated in Fig. 28.1.

Fig. 28.1
figure 00281

Information content from two imaging modalities typically overlaps in certain aspects. Image fusion can merge complementary information from two or multiple imaging modalities

Full size image

We will first consider briefly advantages and limitations of common medical imaging technologies. Since the 1970s, CT usage has increased rapidly, and it is estimated that today more than 72 million scans are performed annually in the USA alone [1]. Although CT imaging is user-friendly and allows rapid image acquisition, concerns have been voiced recently regarding repetitive usage and radiation exposure resulting in a possible increase of life-time cancer risk [2]. Additionally, contrast agents that are required for some studies can lead to renal impairment and hence renal function should always be considered in image modality selection.

MR imaging provides excellent soft-tissue contrast and is free of ionizing radiation. MRI utilization is rapidly growing and today approximately 30 million annual scans are performed in the USA [3]. However, availability of MRI systems is limited and imaging costs are high compared to those of CT. The major contraindications to MRI are metallic implants in the body such as aneurysm clips, cochlear devices, spinal nerve stimulators, pacemaker, implantable cardioverter-defibrillator (ICD), and deep brain stimulators. Patients are frequently imaged using Gadolinium-based contrast agents. Gadolinium chelates are extravascular MRI contrast agents that are cleared by the renal system. Individuals with severe renal impairment may therefore experience adverse effects with the use of these contrast agents, and hence clinically, patients are screened for their kidney function by estimating the glomerular filtration rate (GFR).

PET is a functional imaging modality involving a radionuclide such as fluorodeoxyglucose (18F-FDG) which is intravenously injected after which the location of the tracer throughout the body can be determined for diagnostic purposes [4]. PET imaging has been very successful, especially in oncology [5]. One of the concerns regarding PET/CT imaging studies is the level of radiation exposure that can be in the order of ten to above 30 mSv per scan, depending on imaging protocols [6]. Moreover, at present the availability of PET imaging is limited and scanning costs are high. One of the major technical limitations of PET imaging is poor resolution when compared with other imaging modalities. The fusion of PET/CT systems alleviated the limitation of image resolution in part and also increased availability to approximately 2,000 installed systems in 2010 [7].

Ultrasound imaging is among the safest of imaging modalities with several advantages including portability and excellent temporal resolution. Moreover, US is a widely available versatile medical imaging technology. US has no known long-term side effects, and the modality can be used to image soft tissue, vasculature, blood flow, muscle, and bone surfaces (Fig. 28.2). US technology is based on sound waves that are generated and detected by piezoelectric elements inside the US transducer. Sound waves travel in a beam originating from the transducer that also acts as receiver for the reflected and scattered echo signals that are subsequently converted into an image. Clinical US imaging systems utilize acoustic waves in the low MHz range resulting in limited tissue penetration depth and inability to penetrate bone. US imaging is, however, more operator dependent than CT or MR imaging and hence the sonographer training and experience are important factors of image quality. Another limitation of ultrasound imaging is its “lateral drop out” or loss of detail in the lateral segments of the image, i.e., the segments away from the ultrasound beam—which is a consideration in assessing atherosclerosis [8].

Fig. 28.2
figure 00282

US images of the left common carotid artery (LCCA) of one subject from different angles (views points). The white arrows indicate the same location in the LCCA illustrating changes in speckle patterns

Full size image

Image Fusion

Image fusion presents the clinician and researcher with the opportunity to merge content from multiple imaging modalities, thereby possibly alleviating limitations of individual scanning technologies (Fig. 28.3). Image fusion can be implemented at the hardware or the software level. Table 28.1 lists clinically available image fusion systems such as, for example, PET/CT [4, 9] and recently available PET/MRI systems [10, 11]. Although image fusion is commonly understood as an inter-imaging modality technology, it can also be applied to intra-modality fusion; for instance to analyzing pre- and postoperative imaging data.

Fig. 28.3
figure 00283

Axial images of a carotid endarterectomy (CEA) sample. Panel (b) shows a proton density weighted MRI slice of the internal and external carotid artery (arrows). Note the hypointense region in the external carotid artery indicating the absence of plaque. Panel (a) depicts a fused image composed of the MRI slice and the corresponding ultrasound image

Full size image
Table 28.1 Image fusion systems
Full size table

Table 28.2 shows a summary of important characteristics of major imaging technologies. Broadly speaking, image fusion between one or more imaging modalities can be accomplished by hardware or software fusion.

Table 28.2 Summary of important parameters for the most commonly used clinical imaging modalities
Full size table

Image fusion can be broadly classified into two categories, namely hardware fusion and software fusion.

Hardware Image Fusion

In the hardware-based approach to image fusion, data are acquired simultaneously by different imaging modalities. The advantage of these systems is that the resulting imaging data are co-registered, and data fusion is performed in real time. The disadvantages include the possible requirement of larger equipment.

Ultrasonography

Compound imaging or the fusion of information obtained through multiple scans of the same anatomy or region of interest has been one of the main approaches toward improving information obtained through ultrasound-based imaging. Compound imaging systems fuse images from separate sequential scans of the area of interest [24–28]. Jeong and Kwon [29] obtained US scans of human breast tissue using two opposing array transducers. Although these efforts improved the image quality and resolution, imaging was done sequentially and hence resulted in long scan times or suboptimal co-registration, i.e., the ability to obtain information in the same plane. The simultaneous usage of multiple transducers might exacerbate the known US limitation of lateral dropout leading to potential loss of information.

The issue of lateral dropout in US imaging has been addressed by fusing images obtained from different angles using a technique known as spatial compounding which has been an active field of ultrasound research aimed at reducing intensity variations due to interference patterns from tissue echoes known as speckles [28]. Jespersen et al. [27] developed a scanning method known as multi-angle compound imaging (MACI) that uses a linear phased array to create iteratively a beam at one of n-angles at a time producing a set of acquisitions from different angles. MACI averages all n-images resulting in better tissue contrast and reduced speckle noise. The basic concept of MACI was extended by Behar et al. [28] to improve lateral resolution and speckle contrast by simultaneous image acquisition using three laterally separated transducers with only one acting as transmitter resulting in a compound image. Spatial compounding and MACI reduce speckle and improve tissue contrast at the cost of a reduced image frame rate.

PET/CT/MRI

PET/CT is a very successful imaging technology that combines functional imaging with anatomical imaging [9]. Although technical designs and specifications differ among vendors, the PET detectors and CT components are mounted typically inside a single gantry resulting in a co-registered data acquisition of both modalities [30, 31]. The major CT components such as detectors, readout electronics, and the X-ray tube are mounted on a rotating ring. The PET detectors and electronics are mounted on a separate ring, partial ring, or are integrated in a stationary setup, depending on vendor specifications. Technical details on PET/CT scanner instrumentation can be found in an article by Alessio et al. [32].

In 2011 the US Food and Drug Administration approved the first commercial PET/MRI system for US market. PET/MRI is a promising technology, given its capability to simultaneously image function with increased soft tissue contrast at a significantly lower radiation burden [33, 34]. PET/MRI can reduce ionizing radiation by approximately 70 % when compared to state of the art PET/CT systems. Hence, this new technology might be especially of interest to vulnerable populations such as individuals receiving multiple scans and children.

Software Image Fusion

The software-based approach to image fusion is routinely used for the co-registration of images obtained with two or more modalities (Fig. 28.4). The process of image fusion involves several algorithmic steps that can vary substantially in number depending on the application at hand. Image fusion techniques integrate algorithms from the broad areas of computer vision and object recognition [35, 36]. Common problems that arise in image fusion are due to inherent differences in the underlying imaging technologies. For instance, in-plane resolution is quite different between US and MRI images. The image fusion algorithms need to be able to tackle these differences in a robust fashion. Figure 28.5 shows the typical steps used in software-based image fusion techniques.

Fig. 28.4
figure 00284

Carotid endarterectomy (CEA) sample imaged with μCT and MRI. (a) Coronal view obtained with μCT for the CEA sample. Note the heavily calcified regions (bright areas). The horizontal red line indicates the location of the axial images in panels (b)–(e). (b) Axial slice at bifurcation of the μCT scan showing the internal and external carotid arteries with calcified nodules and increased wall thickness. (c) Co-registered MRI image acquired with a proton-density weighted turbo spin echo sequence on 3.T Siemens Verio system. (d) Euclidean transformed MRI slice in order to align with the μCT image. The black and gray arrows indicate calcified areas (hyperintense) serving as fiducial markers for image fusion. Note that calcified regions appear hypointense in MRI. (e) Representation of the fused μCT-MRI slices using transparency (best viewed in color) (Color figure online)

Full size image
Fig. 28.5
figure 00285

Multi-modality image fusion workflow

Full size image

One of the most important steps in the image fusion workflow is the co-registration of the multi-modality imaging data. This aspect requires the identification of anatomical landmarks that are present in all of the source images. Depending on the approach, these landmarks, which are commonly referred to as fiducial markers, need to be extracted automatically or manually. Clinically, semiautomatic approaches have proven superior since automatic algorithms may perform poorly in the presence of imaging artifacts. Subsequently, these fiducial markers are used to extract features which in turn play a vital role in the quality of the final image fusion output.

Features are a compact representation of an image’s content and of central importance in software-based image fusion systems. In the case of image fusion, the complexity of features can extend from simple coordinates of a set of fiducial markers to wavelet descriptors of selected regions of interest or include a set of features combining texture, structure, shape, motion, or entropy information[37–43]. The construction of features is of central importance in image analysis and image registration algorithms. It is beyond the scope of this chapter to discuss feature extraction in detail. Therefore, we will focus on the conceptual aspects, and the interested reader is referred to specialized literature on feature computation [35, 41, 44–52]. An important property of features for image fusion applications is invariance. Invariant features remain unchanged in case the image content is transformed according to a group action, i.e., the features obtained for an unaltered or a transformed image are mapped to the same point in feature space. A simple example is the color histogram of an image that remains identical under any permutation of the image pixels. However, a slight change in illumination, i.e., changing the actual values of pixels, may significantly change a simple color histogram. The concept of invariance considerably simplifies semi-automatic or fully automatic image co-registration. Instead of comparing images in all transformed instances, only one comparison has to be performed.

The next step in the workflow outlined in Fig. 28.5 is the actual registration algorithm that uses the features extracted from both modalities to establish correspondence between the imaging data sets. The most commonly used registration approach assumes a rigid scenario meaning that a set of rotations, translations, and perhaps a uniform scaling operation results in a correspondence between the imaging data. These operations are also known as a similarity transformation that consists of 4 degrees of freedom (DoF) in the case of a two-dimensional image, i.e., translation in x and y direction (2 DoF), in plane rotation (1 DoF), and uniform scaling (1 DoF). In order to solve the similarity transformation, a total of 2 points is sufficient. In the general 3D case such as a volumetric US or MRI study, the similarity transformation has a total of 7 DoF (3 translation, 3 rotation, 1 scaling) and requires at least 3 fiducial markers to obtain a co-registration between the data sets. The rigid registration approach works favorably for many applications. However, in cases where a patient’s anatomy changes such as, for example, when scans were taken pre and post a surgical or endovascular intervention, the rigid registration approach delivers suboptimal results. In these cases image fusion can be attempted by using nonrigid registration algorithms. There is a broad spectrum of nonrigid registration methods that basically allow to transform elastically the content of one image to match the content of another one. In general, nonrigid registration algorithms are complex and time consuming as the number of degrees of freedom can be very large [53, 54]. In order to alleviate this limitation, nonrigid registration algorithms commonly incorporate expert knowledge of the underlying anatomical change that improves performance and accuracy.

US-CT and US-MRI Fusion

Ultrasound is a real time cost-effective and widely available imaging technology that can be fused with other modalities such as CT or MRI. US is of special interest in intraprocedural and postoperative imaging due to its noninvasive nature. Any other technique would require image viewing, processing pauses, or prolonging the procedure. Crocetti et al. [55] examined in a recent study the feasibility of a commercial multimodality fusion imaging system (Virtual Navigator System, Esaote SpA, Genoa, Italy), for real-time fusion of preprocedure CT scans with intraprocedure US. The study was conducted ex vivo using calf livers prepared with radiopaque internal targets to simulate liver lesions. Subsequently, acquired CT scans were fused with real time US images resulting in mean registration errors of 3.0 ± 0.1 mm.

Nakano et al. [56] used a commercially available image fusion system (Real-time Virtual Sonography, Hitachi Medical, Tokyo, Japan) to perform breast imaging. The system was tested in 51 patients who presented with 63 lesions. Patients underwent MR imaging on a 1.5 T imager followed by a sonographic evaluation of the same lesions. Lesion size measured by real-time virtual sonography and MRI was similar (r = 0.848, p < 0.001). Similarly, positioning errors for the sagittal and transverse planes and relative depth from the skin were small (6.9 mm, 7.7 mm, and 2.8 mm).

Wein et al. [57] developed an automatic CT-US registration framework for diagnostic imaging. Liver and kidney CT and US scans from 25 patients were fused to assess registration errors of the proposed algorithm. One expert defined ground truth data by manually locating fiducial landmarks (lesions) in both imaging modalities. Subsequently, registration errors were compared between the automatic algorithm and the fiduciary point-based registration method. The point-based method using manually identified lesions yielded more accurate results than the automatic method with respective fiducial registration errors of 5.0 mm and 9.5 mm. However, the point-based method involved up to 10 min of identifying fiducial markers, whereas the automatic method required approximately 40 s. Although the automatic method is not readily usable in the clinical setting, it could provide a means to reduce the time necessary to fuse CT and US data sets.

In another study, Caskey et al. [58] developed an US–CT fusion system with the capability to combine real-time US images with pre-acquired CT images. The system was tested using Met-1 tumors in the fat pads of 12 female mice. The CT data were used to identify the Hounsfield units of the tumor which in turn were validated histologically. The US and CT data were fused using fiducial markers with an accuracy of approximately 1 mm.

Clinical Research Applications

As multi-modality imaging becomes more prevalent, attention will be directed toward systematic, reproducible methods for inter-modality comparison of image sets for a given biological system. The fusion imaging techniques discussed above allow for such comparisons to be conducted.

With regard to ultrasound imaging, one can envision fusion ultrasound protocols using baseline ultrasound scans as a reference for true serial comparisons (i.e., ultrasound–ultrasound comparisons) for monitoring cardiac function and wall motion abnormalities. Fusion with more detailed structural scans such as MRI or CT may also allow for overlaying of functional information from real-time ultrasounds. We describe our experience for ex vivo and in vivo imaging toward development of these protocols in the subsequent sections, using a commercially available ultrasound system capable of fusion imaging.

Figure 28.6 illustrates the four main image fusion steps after uploading data from a secondary imaging modality:

Fig. 28.6
figure 00286

(a–f) Carotid duplex and neck MR fusion imaging

Full size image
  1. 1.

    Locking the plane (Fig. 28.6a)

  2. 2.

    Presenting markers and orientation, zoom out/in to view area of interest (Fig. 28.6b)

  3. 3.

    Co-registration: Choosing at least three reference points on each imaging modality (Fig. 28.6c)

  4. 4.

    Displaying the fused images in cross section and longitudinal views (Fig. 28.6d–f)

In Fig. 28.6, the carotid bifurcation and calcified plaques were used as intrinsic landmarks for co-registration. Below, we describe in detail the various considerations both for in-vivo and ex-vivo fusions.

Registration of Landmarks

As previously stated, one of the main challenge with inter-modality fusion of image sets comes from linking recognizable image landmarks through co-registration. Techniques for hardware- and software-based co-registration have been described in depth in the previous sections. Image landmarks, or fiducial markers, chosen for co-registration can be intrinsic or extrinsic to the biological system of interest.

The simplest co-registration techniques with intrinsic landmarks use anatomic features [59]. Anatomy common to a given biological system (e.g., carotid bifurcations) also allows for standardization of in vivo imaging protocols. When extrinsic landmarks are introduced into a biological system, compatibility of the chosen markers between various imaging modalities should be considered. For ex vivo imaging, spatial features of these landmarks aid with identifying spatial orientation within an imaging modality, since the native anatomic orientation (e.g., left–right, anterior–posterior, cranial–caudal) may no longer be present. We encountered these issues in our ex vivo experiments with carotid endarterectomy tissue specimens and have found plastic intravenous 3-way stopcocks to be adequate 3D markers for this system (Fig. 28.7).

Fig. 28.7
figure 00287

Schematic of carotid tissue specimen embedded with an extrinsic landmark (intravenous 3-way stopcock) in an agarose gel matrix to preserve spatial configuration

Full size image

Preparation for Inter-Modality Imaging

Maintaining spatial orientation of the biological system of interest has become so important for inter-modality comparisons that entire medical imaging fields have been developed to address this issue [59]. Again, hardware and software processing techniques have been covered in detail in the above sections. Steps may still be taken with preparing a given biological system to standardize quality assessment protocols of inter-modality comparisons and to assist with such comparisons.

In the ex vivo setting, soft tissues such as vessels are easily deformable and thus do not maintain their shape. Furthermore, the ultrasound imaging modality requires a substrate/media through which the sound beam can travel, ideally a substrate/media with acoustic properties resembling those of soft tissue. Agarose gels maintain the spatial arrangement of embedded tissues and can be mixed to have such desired acoustic properties. Additionally, agarose has minimal chemical interaction with biological tissue. Such a gel has been used for ultrasound imaging of carotid endarterectomy tissues ex vivo [60]. Furthermore, these constructs are compatible with other imaging modalities such as MRI and CT and have also been used for imaging phantoms for these modalities. We have used an agarose gel media (3 % by mass [g]–volume [mL]) for our studies [61].

Carotid endarterectomy tissues were obtained 1–3 h after surgical resection and preserved in phosphate-buffered saline/50 % glycerol at −20 °C to maintain the ultrastructural properties of the tissue. Prior to use, specimens were dialyzed for 24 h against phosphate-buffered saline to remove the glycerol and then embedded in an agarose mixture. The low-melting agarose mixture was created by heating to 60 °C and then degassing under vacuum. Degassing minimized the presence of bubbles in the mixture, which are hyperechogenic on ultrasound and interfere with ultrasound imaging. Each tissue was suspended with an intravenous plastic 3-way stopcock as an extrinsic fiducial marker (Fig. 28.8).

Fig. 28.8
figure 00288

Carotid tissue specimen embedded in an agarose gel matrix for MR imaging. Thread was used to suspend the specimen in the matrix

Full size image

For MRI, tissues and markers were suspended in empty 50 mL tubes, then filled with agarose and placed in a specially constructed MRI compatible holder (Fig. 28.9) [61]. After MRI each tissue was extruded intact as an agarose cylinder from the tube and then transferred to a plastic box (10.5 × 2.5 × 4.0 cm), and additional degassed molten agarose was added to form an agarose bed (Fig. 28.10). This procedure was performed to provide adequate surface contact for the ultrasound probe.

Fig. 28.9
figure 00289

Custom-designed holder for MR imaging of tissue specimens

Full size image
Fig. 28.10
figure 002810

Carotid endarterectomy tissue embedded in an agarose gel bed for ultrasound imaging

Full size image

For in vivo experiments, we observed and recorded the subject’s head position for the MRI scan. Having the subject reproduce this positioning for the 3D ultrasound scan helps minimize MRI-US co-registration issues. In theory, extrinsic landmarks may be employed for in vivo multi-modality imaging. However, such landmarks would either be limited to the skin surface, interfering with the contact by the ultrasound probe, or be necessarily invasive, requiring needle insertion of the marker. Other external reference systems, such as the Meijer’s arc, may also be considered. However, their reliability for co-registration should be compared with that of the above markers.

Three-Dimensional Ultrasound Imaging

MR Imaging

For in vivo experiments involving subjects with known carotid atherosclerosis, individuals were positioned supine in a Signa Excite 3.0 T MRI scanner (GE Healthcare, Wauwatosa, WI). The carotid arteries were imaged using a 6-cm phased array 4-channel carotid coil (Pathway Med Tech, Redmond, WA). A standard 3-plane localizer was used to identify the carotid arteries. Subsequently, a 2D time of flight (TOF) sequence was applied as a localizer to identify both the right and left common carotid bifurcation (flow divider) and to obtain high quality blood flow and vessel wall imaging. Three 2D fast spin echo scans were acquired using proton-density weighting, T2-weighting, and T1-weighting. The longitudinal coverage of this set of images was centered at the carotid bifurcation and covered a large part of the carotid artery below and above the bifurcation.

For ex vivo experiments, carotid endarterectomy tissue specimens were imaged using the same 3.0 T scanner and phased array coil as for the subjects. Serial axial proton-density weighted (PDW), T1-weighted, and T2-weighted images were acquired (2-mm slice thickness, matrix 512 × 512, field of view 100 × 100 mm) using a fast-spin echo sequence, providing 10–31 slices with an in-plane resolution of ∼0.195 mm. Correction algorithms adjusted for magnetic field strength gradients across the sample image. More recently, we also imaged the carotid tissue specimens using a 3.0 T Siemens Verio system with a 32-channel head coil. The CEA samples were imaged using a turbo spin echo sequence with the following parameters: repetition time (TR) = 3010 ms; echo time (TE) = 6.1 ms; number of averages = 4; slice thickness = 2 mm; echo train length = 7; pixel bandwidth = 521 Hz/pixel; flip angle = 123°; x–y pixel spacing = 0.364 mm; and number of slices = 63.

D Ultrasound Imaging

3D ultrasound images can be acquired using probes (a) with motors set to move elements over a certain length, (b) with two-dimensional arrays of transducer elements, or (c) with freehand techniques. With free-hand techniques, the probe is moved in a direction perpendicular to the scan plane or rotated in place, and internal or external reference systems are used to combine a series of B-mode ultrasound images acquired during the sweep. Internal reference systems may consist of (a) timing through clocks or (b) image processing algorithms that recognize large movement in the scan plane. These internal reference systems are beyond the scope of this chapter, but the interested reader is directed to relevant citations for more detail [63]. External reference systems have also been used. Some ultrasound systems have included accelerometers with the transducer probe, capable of detecting probe motion and direction [63]. Others have acquired position information through electromagnetic transmit–receive setups.

For our experiments, we have used one such commercially available position-sensing system as the external reference. This system consists of a mid-range DC magnetic transmitter for generating a weak magnetic field. Sensor arrays attached to the two-dimensional vascular ultrasound probes sensed this field and transmitted the information to an in-built circuit system. Thus, information on the position and orientation of the ultrasound probe can be acquired with the system from the signal detected by the attached sensor arrays. The system can also be used independent of a three-dimensional image data set to generate a three-dimensional reference “map” or image set.

After calibration through a semiautomated co-registration process, real-time B-mode ultrasound images were mapped to the co-registered three-dimensional image sets in a manner similar to that used by the Global Positioning System (GPS) navigation devices to map the device location to stored maps. For our experiments, we used a plane-point co-registration technique. First, the corresponding image set (e.g., MRI scans) for a biological system that was imaged was loaded into the ultrasound system, which performed multiplanar reconstruction of the image data set to form a three-dimensional “map.” The reconstruction was used to navigate to a landmark of interest (e.g., an extrinsic landmark such as a plastic marker for ex vivo experiments or the chin of a volunteer for in vivo experiments). Then, the plane for the loaded three-dimensional image set was locked. The corresponding plane on the live B-mode ultrasound scan was found and locked on the system, completing the plane-lock step. The system then tracked motion of the live ultrasound scan with freehand navigation through the loaded three-dimensional image set. Next, an intrinsic landmark (e.g., calcification in tissue or carotid bifurcation) was located, and the three-dimensional image set was rotated in-plane to align the corresponding images. Finally, the intrinsic landmark was marked with a point on both the live ultrasound scan to refine the real-time co-registration.

Ultrasound Imaging Fusion Experiments

For ex vivo experiments, we have validated use of the system for three-dimensional co-registration against manual co-registration based on anatomic landmarks with 13 carotid endarterectomy tissue specimens [64]. We found that on average, 13.92 (standard error [SE] 1.95) MRI slices each 2 mm thick and 265.77 (SE 28.58) ultrasound frames were necessary to image the tissue samples, translating to 19.66 (SE 2.07) ultrasound scan frames per MRI slice [64]. There were excellent inter-reader agreements between semi-automated GPS-like system and two different readers (intraclass correlation coefficients [ICC] > 0.99) for 33 landmarks (Fig. 28.11) [64]. Further experiments with additional specimens have suggested modestly better agreement between ultrasound and volumes measured by water displacement (ICC 0.85) than between MRI and water volumes (ICC 0.81) [65].

Fig. 28.11
figure 002811

Real-time B-mode ultrasound scan (left) of a carotid tissue specimen co-registered to corresponding MR image set (right). The faint circular pattern on the ultrasound scan corresponds to the agarose cylinder used to hold the specimen for MR imaging, also seen on the right

Full size image

We have also examined if in vivo repeatability of carotid intima-media thickness (CIMT) measurements can be improved using the same system. CIMT measures using the Meijer’s arc and the GPS-like system were 0.61 (SE 0.03) mm and 0.63 (SE 0.03) mm at the initial visit, respectively [64]. On the second visit (∼2 days apart), CIMT measures were 0.64 (SE 0.03) mm and 0.64 (SE 0.04) mm, respectively. There was good agreement (ICC > 0.7) between the two methods (ICC 0.92). Overall, we found greater repeatability C-IMT measures with the registered images (ICC 0.91) than those with the Meijer’s arc (ICC 0.84) (Fig. 28.12) [64]. However, because of the 3-D map, the image quality was not as good as when performed with regular 2D approaches.

Fig. 28.12
figure 002812

Real-time B-mode ultrasound scan (left) of a carotid bifurcation co-registered to a three-dimensional ultrasound image set (right) taken 2 weeks previously. Green markers identify corresponding points. A cross at position 1 signifies a marker within the scan plane. A box at position 2 signifies a marker outside the scan plane (i.e., in a neighboring scan plane or in a position coming out or going into the screen)

Full size image

We have also used the same co-registration technique for our in vivo study of carotid atherosclerosis, using carotid bifurcations as the intrinsic landmark (Fig. 28.13).

Fig. 28.13
figure 002813

Real-time B-mode ultrasound scan with color Doppler (left) of a carotid bifurcation co-registered to corresponding MR image set (right). Arrows indicate the internal carotid artery, and the asterisk indicates the internal jugular. A difference in scale between panes exists due to zooming with the ultrasound window, and hence the slight offset between modalities. The system maintained the scale for the MR image sets because of the native MRI resolution

Full size image

So far, our three-dimensional evaluations have been limited to structural measurements of thicknesses and volumes. Future work may include more complex analyses such as physiologic functional assessment. In fact, others have begun work to incorporate color Doppler information from echocardiography with cine cardiac MRI, paving the way for four-dimensional co-registration [66]. One group has extracted color Doppler blood flow from transthoracic echocardiograms in ten volunteers, co-registered the flow information with corresponding cardiac MR images in time and space using the mitral annular root and root of the aortic valve as intrinsic landmarks, and fused the two data sets into one image set. Registration quality was assessed in this study using the variation in distance of a landmark between echocardiograms and cardiac MRI.[66] The fusion process was conducted off-line using in-house built algorithms, but one can already envision fusion of real-time ultrasonography with cine cardiac MRI using electrocardiography gating with existing technologies [66]. Challenges with algorithm development and temporal resolution would need to be overcome to bring these advanced analyses to research, and eventually clinical, settings.

Clinical Applications and the Future

Clearly the capability to fuse imaging technologies will have significant value in both diagnostic and therapeutic medicine. With respect to ultrasound the major advantage of portability, safety and providing real time functional and anatomical information can be harnessed to complement the excellent anatomic tomographic information that can be provided by MRI and CT.

Ultrasound-based fusion is already beginning to show value both in the clinical and research arenas. Ultrasound on its own is already used to guide biopsies. However, the limitations of ultrasound (drop out, tissue penetration) have to be circumvented by the physician. Although this is clearly possible, it is not optimal. The ability to get in addition the excellent anatomical information as can be provided by a CT scan or MRI will allow the physician performing the procedure more confidence and accuracy in performing the procedure. For example, during a biopsy, images can be fused and an ultrasound be used to guide the biopsy in real time. With image overlays, the physician will be able to simultaneously look at the CT scan/MRI to know if he is in the right anatomical location. These approaches have been already shown to improve prostate biopsies [67]. Similarly, ultrasound-based radiofrequency ablation can be used to treat some tumors. However, ultrasound drop out, isoechoic tumors, prior treatment, etc. may make it difficult for ultrasound to adequately identify these tumors for therapy. The fusion with CT/MRI may allow increased accuracy in the identification of these [55, 68]. In a recent animal study, ultrasound-ultrasound fusion was applied to improve the delivery of radiofrequency ablative therapy. In this study, gold pellets were inserted into the renal parenchyma in a canine model and then 3D ultrasound images were obtained. These 3D ultrasound images were used to plan ablative therapies. Then, using real time, 2D ultrasound, registered to the 3D data set, radiofrequency ablative therapy was delivered, and this overall improved the accuracy of delivery of therapy [69]. From a cardiovascular imaging perspective, in addition to the improved anatomical information, fusion could be an excellent educational tool when we try to resolve artifactual findings.

Ultrasound, on the other hand, can be used to complement other modalities as well. The functional information that can be obtained through ultrasound can be fused with the anatomical information on a MRI/CT to allow better interpretation of a MRI finding. For example, once a stent is placed, MR images in the region of the stent may not be helpful and functional information may be difficult to get. Fusing the images could allow ultrasound to get functional (Doppler) based information related to the same. Figure 28.14 shows an example of an MRA of the middle cerebral artery in the brain that was fused with ultrasound (transcranial Doppler) information to gain knowledge about how the stent was functioning. In such cases, fusion imaging can abrogate the need for a follow-up angiogram (Fig. 28.14).

Fig. 28.14
figure 002814

Transcranial duplex and brain MRA fusion imaging

Full size image

Several potential uses of ultrasound–ultrasound fusion (similar to our efforts to fuse CIMT data and improve reliability) can be thought of as well including, for example, monitoring abdominal aortic aneurysms for expansion, cardiac chambers, and change in lumen dimensions in arterial segments. This may allow meaningful comparisons and a better appreciation of the progression/stability of disease. Clearly, there are innumerable clinical situations where one can think of possible uses of fusion.

However, one must be cognizant of limitations as well [64]. The time taken for the procedure will clearly increase. Further there is a learning curve: the advantages offered by the fusion technology will be limited by the accuracy of registration that the sonographer performs. There may be occasions where a small error in registration may not matter. However, there may equally be situations (such as during a biopsy/therapy) where any error will be critical. Then, when one images a dynamic structure such as the heart, changes in heart rate and rhythm between studies may all affect the ability to fuse the images without additional processing.

Conclusion

Image fusion, be it hardware or software has the potential to enable us to improve imaging-based assistance in diagnostics and therapeutics. Image fusion may be of particular value to ultrasound imaging given its clear differences (vis a vis advantages and disadvantages) with other imaging modalities. Although additional work is required, this developing application offers great promise.