Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

[F-18] fluoromisonidazole positron emission tomography (FMISO-PET), perfusion computed tomography (pCT) and dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) are imaging techniques that can extract relevant quantitative information from tumours. Understanding the tumour micro-environment can improve its characterisation and enable more effective personalised treatment for patients. Recent research effort focused on exploring these imaging techniques to identify hypoxia in tumours. Having a reliable non-invasive method to quantify hypoxia will help to predict chemoradiotherapeutic treatment, and to develop and administer hypoxic sensitisers on patients undergoing treatment [15]. Determining whether and how these imaging modalities can be used to determine hypoxia is inherently difficult. FMISO-PET uses a tracer that binds to hypoxic regions and it has been shown to be effective in some regions of the body [8]. However, for rectal cancers, the excretions present in the rectal region generate a very high background uptake. For this reason, clearly identifying hypoxic regions is difficult and studies must be performed to determine whether and at what level of enhancement these can be distinctively found. DCE-MRI and pCT do not directly measure hypoxia, but the perfusion information provided by these techniques could possibly be used as a surrogate measure of hypoxia and to segment tumour regions [7, 13]. Determining ground truth is challenging and requires a more reliable imaging method. Histopathological imaging is a generally accepted choice to provide a localised ground truth for hypoxia. Hypoxia can be identified in these images with Pimonidazole staining.

Naturally, image registration is an appropriate way to obtain a mapping between these different images and to enable a localised comparison between them. The large amount of deformation and other distortions caused by the process of resecting, slicing and preparing tissues for histological acquisitions present the main difficulty in registering these images. The most common approach taken for the registration of in vivo radiological images to ex vivo histology is to have an intermediate radiological ex vivo acquisition of the resected tissue before slicing.

Previously, functional radiological imaging of cancer has been validated using histological findings in a pre-clinical study [3, 13]. Rat prostate cancers were imaged using FMISO-PET, DCE-MRI and hypoxia-marked histological imaging. However, these images were only rigidly registered using fiduciary markers inserted in the animals before image acquisition, a method which cannot be used in human trials. Several works have explored ex vivo MRI/CT to histology registration, but the different tissue locations configure them as very different problems to the one addressed here. Most of the research to-date has focused on aligning brain images, where there is a relevant amount of coherence of the main types of brain tissue (white matter, grey matter and cerebrospinal fluid) which allows for segmentation-driven registration or histogram matching between the images [1]. Similarly, rigid registration of \(\mu \)CT to histology has been studied [2], which presents tissues with very well defined structures, similar representation across both modalities and very little nonrigid deformation. Ex vivo MRI to histology nonlinear registration was performed for prostate cancer, for which both segmentation-based and intensity-based approaches have been investigated [4].

In this work we present a new method to register ex vivo MRI to histology images of rectal cancer. It differs from previous works by showing an automated solution to register images with large amounts of nonlinear deformation on different imaging representations by applying non-specific methods which could be employed for different body parts and modalities (as opposed to segmentation-based and histogram matching approaches). Another relevant contribution is the application of two patch-based techniques to extract features which can be used as similarity metrics for image registration: dense SIFT and MIND [5, 9].

This paper is structured as follows. In Sect. 2 the methods developed to register ex vivo MRI to histology are presented. Section 3 describes the images used in this work, as well as the experiments conducted to evaluate the proposed framework and their results. Finally, Sect. 4 discusses and concludes this work.

Fig. 1.
figure 1

Example of rectal cancer images before registration: (a) histology, (b) corresponding ex vivo MRI slice and (c) blockface photograph.

Full size image

2 Methods

The challenge addressed in this work can be described as finding a transformation \(T^i\) for each histology image \(I_H^i\) to an MRI volume \(I_M\), where the grouping of all the histology images form a histology stack \(I_H\). To solve this problem we have designed a three-stage solution. Firstly, a global transformation \(T_Z\) to axially (Z-direction) align \(I_M\) to \(I_H\) is estimated. Then, after applying this transformation we rigidly register each \(I^i_H\) to its corresponding MRI slice \(I^i_M\), thus estimating \(T^i_r\). This is followed by nonlinear registration, which ultimately computes \(T^i_{nl}\) for each of the histology slices. These steps are described in Sects. 2.1, 2.2 and 2.3.

As can be seen from Fig. 1, MRI and histology tumour images show different types of tissues, but the main feature characteristics are preserved (albeit under severe deformation). Hence, we opted to use similarity metrics with strong edge responses. Two similarity metrics with such property were explored in this work: the Scale Invariant Feature Transform, both its original form and dense variant (SIFT [10] and dense SIFT [9]), and the Modality Independent Neighbourhood Descriptor (MIND) [5]. Both of these metrics extract descriptor vectors based on the intensity relationships between voxels in regions of the images.

SIFT is a method to localise points of interest (POIs) in images and extract high dimensional discriminative descriptor vectors based on the gradient profile on the regions around these salient points [10]. Dense SIFT is a variant of this method where an image is converted to a vector-valued image by computing these descriptors at each voxel without the POI detection of classic SIFT [9].

MIND is a transform that computes a feature vector for each voxel of an image [5]. This feature vector is determined by the similarity of nonlocal patches around each voxel. Its descriptors have much smaller dimensions than SIFT and have already shown to generate good results in medical image registration tasks.

Both dense SIFT and MIND can be used as a similarity measure. Each of the images being registered (\(I_1\), \(I_2\)) is transformed using these methods generating the vector-valued images (\(\text {SIFT}(I(\mathbf {x}))\) or \(\text {MIND}(I(\mathbf {x}))\)) with d descriptors. Then the sum-of-squared differences (SSD) can be computed between these images resulting in a similarity measure (SIM). For the SIFT case this is expressed as:

$$\begin{aligned} \text {SIM}_{\text {SIFT}}(I_1(\mathbf {x}),I_2(\mathbf {x})) = \Big |\Big |\text {SIFT}(I_1(\mathbf {x}))-\text {SIFT}(I_2(\mathbf {x}))\Big |\Big |_{2} \end{aligned}$$
(1)

An analogous equation can be used for \(\text {SIM}_{\text {MIND}}\).

2.1 Histology to MRI Axial Registration

Initialisation is one of the main difficulties in registering 2D histological slices to 3D MRI. In general, the specimens are sliced axially before the histology image acquisition, hence finding corresponding axial MRI slices for each histology image is a good first step towards image registration.

SIFT is a fitting solution for this problem as it reduces the large MR volume space to a small number of POIs which can be matched to the histology slices ones. However, due to the large amount of deformations found in the histology, the POIs from the histology may not always be similar to the MRI ones. Thus, to robustly perform this step we opted to jointly register all the histology slices from a tumour to the corresponding MRI volume. This is done by first generating a histology stack by placing each of the slices on their expected relative axial positions (which can be determined with the aid of the blockface images Fig. 1(c)). This initial axial translational transformation (\(T_Z\)) is obtained with the following sequence of steps:

  • SIFT is applied to the histology stack, generating a set of POIs and descriptors \(P_H\). Where \(P^n=[\mathbf {x},\mathbf {D}]\), \(\mathbf {x}\) denoting the interest point’s coordinates and \(\mathbf {D}\) its descriptor vector.

  • SIFT is applied to the MRI volume, finding a set of POIs \(P_M\).

  • For each POI of the histology stack (\(P^n_H\)) the most similar point in the MRI volume (\(P^k_M\)) is found by minimising the SSD between descriptors:

    $$\begin{aligned}{}[P^n_H,P^k_M] = \text {arg } \underset{P^m_M}{\text {min}} \Big (\big |\big |P^n_H[\mathbf {D}] - P^m_M[\mathbf {D}] \big |\big |_{2}\Big ) \end{aligned}$$
    (2)

  • Each correspondence pair denotes an axial translation \(T^n_Z = P^n_M[z] - P^k_H[z]\) between the histology and MRI stacks.

  • A final axial transformation (\(T_Z\)) is obtained as the mode of all these transformations: \(T_Z = \text {Mode}\{T^n_Z\}\).

Reconstruction of a 3D histology volume is not possible in our case, as the histology images are not contiguous and sometimes incomplete, but individual slices can be matched to the MR volume. For that reason, we can only perform a translation in the axial direction in this first step. Note that, a similar approach was proposed to estimate rigid transforms for histology to \(\mu \)CT registration of bone structures [2].

2.2 2D MRI Rigid Registration

After finding a corresponding MRI slice (\(I^i_{M}\)) for each histology image (\(I^i_{H}\)), the next step is to perform 2D to 2D rigid registration between this image pair (estimating \(T^i_r\)). We investigated both dense SIFT and MIND similarity metrics for this step (using Eq. 1), by iteratively minimising:

$$\begin{aligned} \hat{T_r} = \text {arg } \underset{T_r}{\text {min}} \left( \int _{\mathbf {x}=0}^{\varOmega }\text {SIM}(I_1,I_2)(\mathbf {x}) d\mathbf {x}\right) \end{aligned}$$
(3)

where \(\varOmega \) is the image domain and SIM denotes Eq. 1 using either dense SIFT or MIND. This minimisation was performed using a Levenberg-Marquandt optimizer from the alglib libraryFootnote 1.

The results for this step using dense SIFT were very unsatisfactory (and thus will not be reported in the results section). The cause for this is that regions with very low match values with dense SIFT dominate its response as a global similarity metric. Thus, its use as a global measure requires some adaptations (such as mutual saliency weighting [11]).

2.3 2D MRI Nonlinear Registration

Histological images show high levels of nonlinear deformation, especially as some regions shrink while others expand, in addition to tissue tearing and shearing. We performed nonlinear motion correction in this work using logDemons [14], which provides invertible diffeomorphic transformation fields (\(T_{nl}\)). Even though the deformation present in these images are not diffeomorphic, such approach is chosen as a one-to-one correspondence is relevant for future data analysis between functional and histology images. A multi-resolution framework was applied for this registration and we compared the use of dense SIFT, MIND (minimising Eq. 1) and local cross-correlation (LCC) [6], a standard similarity metric.

3 Experiments and Results

3.1 Data Acquisition, Preparation and Landmark Selection

T2-weighted MRI images were acquired from 2 resected rectums with adenocarcinomas while in formalin for 12 to 24 h. The images were acquired using 1.5 T scanner (TwinSpeed, GE Healthcare) with 3 mm oblique axial sections with 0.3 mm spacing. The specimens were then sectioned in 3 mm sections when possible and 6 mm at more friable sections of the tumour. Haematoxylin and eosin stains were applied and histological imaging acquisitions were performed. A total of 8 histological slices with axial resolution of 2 \(\upmu \)m were obtained this way.

A number of steps were taken to prepare the images for the proposed method. The ex vivo MRI images of the resected cancer possessed a much larger field of view than the histology, thus, to facilitate the registration algorithm a region around the tumour was cropped. For each of the histology images, an expert histologist and oncologist jointly annotated a set of two to five landmarks (25 in total) in both the histology and the ex vivo MRI using the software VV [12].

Fig. 2.
figure 2

Registration of two cases from different patients (row 1 and 2, respectively). (a,e) Histology and (b,f) corresponding MRI slice after rigid registration, (c,g) after nonlinear registration using dense SIFT and (d,h) after nonlinear registration with yellow overlay of the histology. Landmarks are shown by the crosses: histology (blue), MRI after rigid registration (green) and MRI after nonlinear registration (red) (Color figure online).

Full size image

3.2 Framework Parameters and Results

The parameters for the registration framework were empirically determined. For the initial slice estimation, an off-the-shelf implementation of SIFT, ezSIFT Footnote 2, was used along with its standard parameters: 36 bins, threshold set to 8 and number of layers set to 3. For nonlinear registration, the logDemons framework was applied with three resolution levels with 40 iterations at each level and transformation field smoothing \(\sigma _{\text {diff}}=4\) pixels. The SIFT Flow library was used for dense SIFT with the standard parameters: cell size = 2 and 8 bins [9]. For MIND, the search region was \(R=2\) and the Gaussian weighting \(\sigma _{\text {MIND}}=0.5\).

The proposed method was quantitatively evaluated using the annotated landmarks. For each of the histological images, the obtained transformations were applied to the landmarks and the mean distance to the MRI landmarks were computed. The mean registration error for each of the images was evaluated after the rigid registration steps and after nonlinear registration, Table 1 presents these results. It can be observed that nonlinear registration was able to compensate some of the deformation in these images and considerably decrease the distance between the landmarks, in the best case, with dense SIFT, an average improvement of 60 % was observed which corresponds to a mean error of 1.08 mm. Figure 2 presents some registered images using this similarity metric. As expected, the similarity metrics with a strong edge response, dense SIFT and MIND, could better characterise and compare the images being registered, leading to better registration results. However, despite MIND being very good at identifying edges, it is not as good as dense SIFT in discriminating them, this is observed in Fig. 3, a case where MIND matches noncorresponding edges.

Table 1. Average mean squared error and standard deviation of the landmarks after rigid and nonlinear registration with different similarity metrics.
Full size table
Fig. 3.
figure 3

Histology slice (right) and MRI image (left) after nonlinear registration using (a) MIND and (b) dense SIFT. A close inspection shows that both methods are good at matching edges, but MIND is not very discriminative and consequently registers noncorresponding edges, while dense SIFT is more accurate.

Full size image

4 Discussion and Conclusions

We have presented a comprehensive framework that addresses the challenging tasks of aligning histological slices through a tumour with volumetric ex vivo MRI of the tumour. Despite it being developed for a rectal cancer application, the proposed method does not use any specific characteristic of the region of the body being analysed and thus could be employed for other similar histology registration tasks. The framework was evaluated through landmarks, and visual inspection also confirmed its validity of this solution for this problem.

For this, we have investigated the potential of dense SIFT as a patch-based similarity metric within the logDemons framework, achieving good results compared to other multi-modality and feature vector based similarity metric. Due to some limiting factors dense SIFT has not been widely used yet for medical image registration: its descriptors have very high dimensions and were originally developed for 2D images, two aspects that need to be adapted to be used with the very large 3D volumes found in medical imaging.

To-date, the validation was performed in a small dataset, which is one of the difficulties when working with histology images and we intend to increase the number of cases as part of an ongoing trial. There are several aspects that can be explored in future works. Dense SIFT was not a viable option as a global similarity metric and could be further developed for such task. Moreover, this application should be extended to also account for nonlinear deformation across MRI volume slices. This work will follow by registering ex vivo to in vivo MRI volumes and then to the functional acquisitions (DCE-MRI, pCT and FMISO-PET), obtaining a correspondence between the histology and these acquisitions.