Keywords

3.1 Introduction

Multimodal imaging has the ability to provide enhanced descriptions of organs by combining descriptors related to anatomy, function, and metabolism. Considering cardiac imaging, many different modalities are clinically used, including computed tomography (CT) imaging, magnetic resonance imaging (MRI), ultrasound (US) imaging, and nuclear imaging. All these modalities provide complementary information, opening the way towards a combined analysis of heart multiphysics that may be useful to improve the diagnosis of cardiac pathologies, plan and assist their treatment, and perform its follow-up.

Chronic heart failure (HF), affecting an estimated 26 million people worldwide [1], corresponds to the heart’s inability to pump enough blood to fulfill the body’s needs. Cardiac dyssynchrony, i.e., spatial inhomogeneity in timing of contractions, is frequently observed in these patients and leads to a poor outcome if left untreated. Cardiac resynchronization therapy (CRT) is a non-pharmacological treatment based on the implantation of a pacemaker with leads stimulating the heart. Generally, a biventricular pacing is used, with the implantation of three leads: two endocardial leads in the right atrium and ventricle, and one transvenous lead to pace the left ventricle (LV) [19]. CRT has shown its efficiency to treat patients with left-sided heart failure associated with dyssynchrony despite pharmacological therapy [23]. However, the percentage of non-responders remains high: about 30% [37]. A suboptimal position of the LV lead, that is the most difficult one to implant, may be one factor explaining non-response. Indeed, in current practice, the LV stimulation site is selected during the implantation, based on the per-operative angiographic images providing a very poor description of the left ventricle and of the coronary venous access.

On the other hand, it has been shown that cardiac sites with well-defined specific characteristics are better responders to the therapy [4, 10]. Especially, stimulating the site associated with the latest mechanical contraction improved patient outcome [28]. On the contrary, stimulating a fibrotic tissue (i.e., scarred tissue with degraded contractility) is associated with non-response to CRT [35]. In this context, generating an enhanced description of the heart would allow the clinicians to select the most appropriate pacing sites in order to plan the implantation. This description should include the mechanical contraction, the tissular properties but also a description of the LV anatomy and of the coronary veins which are used to implant the LV lead. These descriptions result from different imaging modalities, which have thus to be registered and fused.

Few works of the literature addressed the combination of multiple pre-operative modalities to plan CRT. This combination has mostly been done by using, independently for each modality, a division of the cardiac wall in segments (i.e., a 17-segment bulls-eye representation [11]) [2, 5, 13], thus providing a limited spatial accuracy. Some teams proposed to use a landmark-based registration to combine 3D echography and CT [33], or 2D echography, CT, and SPECT [31]. It provides more accurate characterizations, but is dependent on the selection of well-defined landmarks in the images. Performing an automatic registration method is challenging since the considered modalities are highly different in terms of dimension (2D vs 3D, static vs dynamic), resolution (e.g., 0.4 × 0.4 × 0.6 mm in CT versus 1.8 × 1.8 × 4 mm in cine-MRI in our case), represented information, and since the considered organ, the heart, is submitted to fast deformations, related to the cardiac contraction but also to respiration. It therefore needs the development of specific methods adapted to the characteristics of both the organ and the studied modalities.

We thus have developed a workflow to fuse multimodal cardiac images (i.e., CT, MRI and US) to generate, pre-operatively, a patient-specific model of the LV including different characteristics useful to plan CRT interventions. This model includes the anatomy of the LV and of the coronary veins, the mechanical contraction of the wall, and its tissular characteristics in terms of macroscopic fibrosis.

The global workflow is presented in Fig. 3.1. It is based on multimodal data processing, including the segmentation of CT images to extract the LV and veins, fibrosis characterization based on LGE-MRI, and processing of echographic data using speckle tracking echography to characterize the mechanical activation. Then, all these images are registered. These different steps have been previously presented in different publications [7,8,9, 32]. In this chapter, we summarize these processing and show how they can be integrated to generate a patient-specific 3D model which can help the clinician to select the optimal site(s) by considering the sites localized close to a vein, associated with a large mechanical delay and with the absence of fibrosis. The different processing steps are described below.

Fig. 3.1
figure 1

Multimodal image analysis workflow

Full size image

3.2 Multimodal Image Processing

The first step of the proposed workflow is to extract the descriptors of interest by processing the acquired images, namely the anatomy in CT images, mechanical activation in echography, and fibrosis in MRI. Table 3.1 summarizes the main characteristics of the acquired images.

Table 3.1 Main characteristics of the considered images (img/CC: images per cardiac cycle;∗: in STE, the considered resolution is the average distance between the landmarks)
Full size table

3.2.1 Segmentation of LV and Veins in CT Images

CT imaging, because of its high spatial resolution, remains the reference modality for the anatomical description of the heart, especially considering the coronaries.

The considered CT images were acquired with a multislice computed tomography scanner (General Electric, LightSpeed VCT 64- slice system). Iodine contrast product was injected to enhance the left heart inner cavities (100 and 150 cm3 being at 300 mg/ml or 350 mg/ml). The acquisitions were retrospectively synchronized on the ECG, providing dynamic 3D+t images (10 or 20 phases describing the cardiac cycle). Since the contrast of the cavities is high, a simple region growing enables to segment the left cavities (atrium and ventricle). In case of segmentation leaks due to limited contrast, a manual correction was used to separate the cavities from neighboring structures.

The segmentation of the veins is much more challenging, but required since the LV lead implantation is done through the venous network. However, these structures are small, with a low contrast and with a variable anatomy among patients. Moreover, they are very close to arteries, making them difficult to identify. Many works have been proposed in the literature for the segmentation of vessels in MR or CT imaging [21]. However, most of them are focused on the arteries, which present a higher contrast than the veins. Moreover many methods fail to segment the vessels of medium and low diameter, which are of interest for the implantation. We thus used a semi-automatic process [9] including, for each slice containing a specific vein (coronary sinus, great cardiac vein, middle cardiac vein, posterior vein, or left marginal vein): the manual selection of a region of interest; a filtering to enhance tubular structures; a thresholding to identify the tubular objects; a connexity based object identification considering either the largest object or the one related an object previously segmented in the neighboring slices. In case of missing segments, the vein was finally interpolated.

Manual Selection of a Region of Interest

For each vein, the user defined a 2D region of interest on a slice selected such that the vein is visible. It is then extended to a volume of interest by including the three superior and inferior slices.

Tubular Structures Filter

The VOI was filtered using the Frangi filter [12], a filter based on the Hessian matrix of the image intensity, intended to enhance the tubular structures. In order to detect vessels of different sizes, a multi-scale approach was used by filtering the VOI with a set of Gaussian filters of different sizes (standard deviation in the range [1, 3] pixels). For each pixel, the highest value resulting from the different Gaussian smoothing was retained.

Thresholding and Connexity Based Object Identification

The resulting filtered ROI was thresholded using an interactively selected threshold T. The resulting binary region was analyzed to keep either the largest region (if the current slice is the first one to be processed), or the region connected to the vein segmented in the previous slice.

Interpolation of the Vein

In some cases, the whole vein could not be extracted using the previous process and some intermediate parts were missing between two segmented parts. Then, an interpolation was used to estimate the missing segment, based on the discretization of the contour using regularly spaced points and their interpolation with cubic splines.

3.2.2 Characterization of the Mechanical Activation Using Speckle Tracking Echography

Despite the increased attention gained by MRI for cardiac function assessment [14], the measurement of dyssynchrony remains mainly performed using echography, and especially with speckle tracking [10]. By tracking acoustic markers (speckles) in ultrasound images, speckle tracking echography (STE) provides a mechanical description of the LV contraction in terms of strain curves.

Transthoracic echocardiography (Vivid7, GE Healthcare, Horten, Norway) was acquired at rest. 2D acquisitions were realized using two- and four-chamber apical views over a cardiac cycle. On each view, the endocardium wall was manually segmented on diastole by the expert and tracked using speckle tracking (General Electric ECHOPAC workstation). The strain curves, computed on regularly spaced points on the endocardium, were then automatically computed by the speckle tracking software, and exported. Different descriptors can then be extracted from these strain curves, especially the maximal strain value, to identify areas with low contractility, and the delay between the peak of the QRS complex and the maximal strain (electro-mechanical delay) to identify the delayed segments.

3.2.3 Characterization of Fibrosis in LGE-MRI

LGE-MRI enables to identify macroscopic fibrosis (i.e., replacement of myocytes after cell damage) by using a gadolinium contrast agent injection and by imaging the increased amount of gadolinium in fibrous tissue [25].

LGE- and cine-MRI were acquired using cardiac SENSE coils (Philips Achieva®, Philips Medical Systems, Best, The Netherlands). Breath-hold cine (bTFE sequence) and LGE (TFE) were acquired for SAX and long axis views (2CH,4CH), using retrospective ECG synchronization. Both inversion recovery (IR) and phase-sensitive IR (PSIR) techniques were used to acquire LGE. 12 slices were acquired for SAX and 3–5 for 2CH/4CH. Multiview LGE was acquired 5–10 min after gadolinium injection (0.1 mmol/kg of Dotarem Guerbet, Roissy, France), at mid-diastole. 16 slices were acquired for SAX and 12 for 2CH/4CH.

If needed, motion artifacts were corrected using an in-house developed software presented in [7].

In order to segment fibrosis areas in LGE images, the inner and outer contours of the myocardium were firstly manually roughly segmented. Then, a fuzzy c-means algorithm was applied to the myocardium and blood pool to associate, to each voxel, a probability to belong to the gadolinium-enhanced class [18]. The membership probability was then automatically thresholded on the myocardium by using the threshold value providing the most stable output [3].

3.3 Multimodal Image Registration

Once all the images have been processed, they were registered and fused to generate the patient-specific 3D model incorporating all the descriptors. The CT image, providing the best spatial resolution, was considered as the anatomical reference. MRI and STE were therefore registered towards it. Since these three modalities embed dynamic data’s acquisition (cine sequence in MRI), the proposed registration process takes advantage of all the available information by combining a temporal synchronization with the spatial registration, thus enabling to register all the phases of the dynamic images in a single process.

3.3.1 STE to CT Registration

Few works have been proposed in the literature for cardiac multimodal registration considering 2D [24] or 3D US data [22, 27]. The goal of these works is mainly to compare pre- and per-operative data during surgical gestures guided by US [36], or to combine complementary descriptors to help in diagnosis [34]. The temporal synchronization of data is often the first problem to solve. Some works use the ECG, by considering only one instant related to the same cardiac phase [20], whereas others use a dynamic registration relying on an interpolation process [16, 34]. Then, to perform the geometrical matching, some authors considered either iconic methods based on mutual information [17], or geometrical methods based on geometrical landmarks [15, 30], or modelling methods based on the simulation of US images from CT data [34].

We developed a method based on Fourier descriptors to handle differences in spatial resolution and dynamic time warping (DTW) [29] to deal with differences of cardiac rates and ECG sampling frequencies.

The dynamic geometry of the ventricle was available in both STE and CT with a 2D dynamic contour for STE and a 3D dynamic surface for CT. The proposed process combined, in an iterative way, synchronization and registration [7, 32] (cf. Fig. 3.2a). At each iteration, the position of the echographic acquisition plane, expressed in the CT image coordinate system, was refined. This position enabled to generate a cut of the CT dynamic surface, resulting to a dynamic contour. The registration metric, used to express the quality of the registration, was thus defined between the two dynamic contours, from CT and from STE. In order to handle the differences between both contours (i.e., differences in smoothness), they were expressed in terms of Fourier descriptors (maximum order: 50). They were then synchronized by using DTW applied to the Fourier decompositions of both 2D dynamic contours. The final registration metric was, after synchronization, the sum of the differences of the Fourier descriptors.

Fig. 3.2
figure 2

Multimodal image registration; (a) STE to CT registration; (b) LGE-MRI to CT registration

Full size image

3.3.2 LGE-MRI to CT Registration

LGE-MRI was registered to CT images by considering cine-MRI as an intermediate (cf. Fig. 3.2b). Indeed, cine-MRI is consistent with LGE-MRI and, by including the temporal dimension, provides more information than LGE-MRI for the registration towards CT images. LGE-MRI was rigidly registered towards the closest phase in cine-MRI by using normalized mutual information (NMI).

The registration of cine-MRI and CT is challenging mainly because of the temporal synchronization. If the ECG is used during the acquisition of both images, it may be deformed by the MRI magnetic field and is not always available after the acquisition.

Thus, cine-MRI and CT images were registered in two steps [8]: (1) temporal synchronization based on image information; (2) rigid registration of the synchronized sequences using a multi-image metric.

The temporal synchronization was performed by computing normalized cross-correlation (NCC) curves of input image sequences. One NCC curve was firstly computed for each image sequence, defined by the normalized cross-correlation between the first image of the sequence and each image of the same sequence. The two resulting curves have proven to globally describe the cardiac dynamics in dynamic CT and cine-MRI [6, 26]. The nonlinear synchronization of corresponding cardiac dynamics was then estimated by warping the time axes of these curves, using a DTW procedure. A multi-image rigid registration procedure, using normalized mutual information (NMI) as the metric, was finally used to align dynamic CT and cine-SAX (short axis) with synchronized cardiac dynamics.

3.4 Patient-Specific 3D Model Generation

The patient-specific model results from the registration of STE and LGE-MRI towards the planning CT image. Thus, the anatomical description (LV endocardium and veins) is enhanced by the mechanical activation (delay to the peak of strain and peak amplitude, defined on the 2D ultrasound acquisition planes) from STE and by segmented fibrosis provided by LGE-MRI.

The 2D mechanical descriptors have been represented by spheres, positioned on the endocardium along the US acquisition planes, and whose color represents the associated value. Considering the fibrosis, it is represented using a 3D volume corresponding to the localization and extent of fibrosis in the myocardium.

3.5 Results

The different steps presented above have been evaluated:

  • on simulated data: for STE to CT registration [32];

  • on real data:

    • with manually positioned fiducial markers: for cine-MRI to CT registration [8];

    • with reference geometrical transform: for LGE-MRI to cine-MRI registration [7];

    • with an expert qualitative assessment: for CT segmentation [9], for fibrosis segmentation [7], and STE to CT registration [32].

Further evaluations could be performed, especially considering CT and fibrosis segmentations, by comparing the results with manual delineations from experts.

Hereafter, the results obtained on the real data of two patients undergoing a CRT procedure are presented. All the processing results, including CT and MRI segmentations, STE processing and images registrations have been validated by an clinical expert. All the images have been acquired 2 days before the CRT implantation (day D-2) and processed the day before (D-1), in order to be presented to the cardiologist before the implantation.

Figures 3.3, 3.4, and 3.5 illustrate the workflow’s different steps for the first patient. Figure 3.3 shows one slice of the CT image and the segmentation result with the left cavities (ventricle and atrium) and the veins. The veins of interest for the LV lead implantation are represented in green, the others in blue. Figure 3.4 presents the echographic image and the speckle tracking result. Figure 3.5 illustrates the MRI data (cine-MRI and LGE-MRI) and the result of the fibrosis segmentation process.

Fig. 3.3
figure 3

CT segmentation: example of one original slice and segmentation result (same slice and in 3D). Top-right: the segmented cavities are represented in red and the segmented vein in purple. Bottom: the veins of interest for the LV lead implantation are represented in green, the others in blue

Full size image
Fig. 3.4
figure 4

Echographic data: example of one 2D image and of the associated strain curves estimated with STE. Left: acquired image (four-chamber apical view) with the associated manual segmentation and, in color, cardiac segments; Right: strain curves associated with the segments

Full size image
Fig. 3.5
figure 5

(a) Example of one cine-MRI slice; (b) example of one LGE-MRI slice, with the associated manual delineation of the myocardium and resulting segmented fibrosis displayed in 3D

Full size image

The developed registration workflow results in the illustrated models in Figs. 3.6 and 3.7 for the two patients. While the first patient has a large area of fibrosis, which has to be avoided for the left lead implantation, the second patient shows no tissular defects. The model thus enables the clinician to identify the cardiac sites accessible through a venous access and associated with no fibrosis and with a delayed mechanical contraction.

Fig. 3.6
figure 6

Generated model for the first patient. It includes: the left cavities and the veins (green: veins of interest for the LV lead implantation); the fibrosis (in dark red); the electro-mechanical delays (EMD, with spheres colored according to the colorbar)

Full size image
Fig. 3.7
figure 7

Generated model for the second patient. It includes: the left cavities and the veins (green: veins of interest for the LV lead implantation); the electro-mechanical delays (EMD, with spheres colored according to the colorbar); no fibrosis was detected in LGE-MRI

Full size image

3.6 Conclusion

Multimodal cardiac imaging, by combining complementary descriptors (e.g., anatomical, mechanical, and tissular), has the ability to provide an improved characterization of the heart which may be useful in the management of cardiac diseases. However, some specific processes including segmentation, characterization, registration, and fusion have to be developed, evaluated, and combined to provide data management workflows adapted to the clinical needs and requirements.

In this work, we developed a workflow to assist the clinicians in planning CRT procedures. It involves the processing and registration of CT, MRI, and STE images, and the generation of a pre-operative patient-specific model. This model provides different information needed to define cardiac sites likely to respond positively to the stimulation: anatomical description of the left ventricle and the veins to assess the venous access, mechanical contraction to assess the delayed sites, and fibrosis detection to avoid the sites with poor tissular characteristics for CRT.

This process has been implemented in clinical conditions. It should allow the clinician to target the best pacing sites and to evaluate their accessibility through the venous network. This project needs further clinical evaluation, e.g., by evaluating if the proposed planning platform enables to reduce the intervention time and the rate of non-responding patients.

In the future, this workflow may be tuned and enriched to be extended to other pathologies and interventional gestures, such as the ablation of ventricular tachycardia or of atrial fibrillation.