Brain Tumor Segmentation by Variability Characterization of Tumor Boundaries

Abstract

Automated medical image analysis can play an important role in diagnoses and treatment assessment, but integration and interpretation across heterogeneous data sources remain significant challenges. In particular, automated estimation of tumor extent in glioblastoma patients has been challenging given the diversity of tumor shapes and appearance characteristics due to differences in magnetic resonance (MR) imaging acquisition parameters, scanner variations and heterogeneity in tumor biology. With this work, we present an approach for automated tumor segmentation using multimodal MR images. The algorithm considers the variability arising from the intrinsic tumor heterogeneity and segmentation error to derive the tumor boundary and produce an estimate of segmentation error. Using the MICCAI 2015 dataset, a Dice coefficient of 0.74 was obtained for whole tumor, 0.55 for tumor core, and 0.54 for active tumor, achieving above average performance in comparison to other approaches evaluated on the BRATS benchmark.

Funded by the National Institutes of Health (NIH) under the award number R01CA1575533.

1 Introduction

Quantitative measurement and assessment of medical images can play an important part in diagnosis of a disease, treatment planning, and clinical monitoring. As imaging technology and standards have been rapidly changing and increasing in complexity within the field of neuro-oncology, it has become extremely burdensome for clinicians to manually review imaging studies. In addition to increased labor and expense, manual measurements can have a high degree of measurement variability [1] due to the inconsistency and diversity of MRI acquisition parameters (e.g. echo time, repetition time, etc.) and strategies (2D vs. 3D) along with hardware variations (e.g. field strength, gradient performance, etc.) that change the appearance characteristics of the tumor [2]. The increased variability in measurement from multiple imaging sources, combined with the need for faster interpretation, may potentially result in errors with treatment decisions or conclusions about of potential therapeutic benefits.

Simplistic two-dimensional measurements used to characterize therapeutic changes in the Response Assessment in Neuro Oncology (RANO) criteria [3] have been used for several years. Despite the need for an automated characterization, an accurate classification of brain tumors remains challenging for automated approaches as it has also proven difficult for expert neuroradiologists as well [4].

In this paper, we hypothesize the inherent variability in tumor volume measurements can be leveraged to provide a more accurate assessment of tumor burden and produce an estimate of tumor segmentation variability. While multiple automated segmentation techniques are being actively developed [3], a method that accounts for the variability in tumor burden estimation has not been entirely investigated [5]. We explored a different perspective towards the identification of tumor boundaries and developed a knowledge-based approach that considers a series of brain tissue probability distribution maps as prior information to inform the location and boundaries of brain tumors. This algorithm uses superpixel-based morphological features and the prior statistical maps to generate a preliminary tumor region. The areas of highest variation inside this preliminary region are iteratively measured to create a Tumor Variability Map (TVM), which represents the image heterogeneity along the tumor boundary (measure of uncertainty).

2 Methods

We developed a processing pipeline to automate the segmentation from the raw MRI images to create tumor variability maps that indicate tumor extent. This pipeline is illustrated on Fig. 1 and PseudoCode 1.

Fig. 1.

Overall segmentation process from input multimodal MRI, feature extraction, preliminary tumor ROI calculation by supervoxel classification, and a tumor histogram variability analysis to generate segmentation error estimates for the overall tumor boundary and different tumor components. The process is repeated for all time-points available for an input subject. The output variability maps are a graphical representation that reflect the likely location of a heterogeneous tumor boundary.

The system is divided into a series of preprocessing strategies followed by the proposed tumor segmentation algorithm. This approach finds an approximate tumor ROI by using the knowledge-based approach proposed in this paper. Afterwards, the intensity variation observed on the approximate tumor ROI is analyzed to find the possible tumor boundaries for the TVM. This approach was evaluated using the 2015 Multimodal Brain Tumor Image Segmentation Benchmark (BRATS) dataset [6].

2.1 Tumor Segmentation

As first step, the algorithm selects all MR modalities of interest including pre-contrast T1-weighted images, post-contrast T1-weighted images (T1+C), T2-weighted images, and T2-weighted fluid attenuated inversion recovery (FLAIR). Then, a series of preprocessing steps are performed as preliminary step before running the proposed segmentation approach, including intra-subject image registration [7], skull stripping [8], intra-subject intensity normalization (z-scores), and image denoising (bias-field correction and soft Gaussian smoothing) [8]. Note that the data for the BRATS benchmark has already been preprocessed.

Afterwards, a series of tissue probability masks are generated to provide context/knowledge about the approximate distribution of normal cerebral tissues including gray matter, white matter and cerebrospinal fluid (CSF), using this information for tumor identification [10]. The tissue distribution information models image intensities as a mixture of k Gaussians, modelled by a mean (μ_k), standard deviation (σ_k) and a mixing proportion. Following this, Bayes rule is employed to produce the posterior probability of each tissue class. Using this model, the probability of observing an element with intensity y_i on the k ^th Gaussian is given by:

$$ P\left( {y_{i} |k = \mu_{k} ,\sigma_{k} } \right) = \frac{1}{{\sqrt {2\pi \sigma_{k}^{2} } }}\exp \left( { - \frac{{\left( {y_{i} - \mu_{k} } \right)^{2} }}{{2\sigma_{k}^{2} }}} \right) $$

(1)

Finally, the probability of obtaining the pixel y on a Gaussian is maximized with respect to μ, σ and γ by the minimization of the cost function [10]:

$$ \varepsilon = - \log P\left( {y |\mu ,\sigma , \gamma } \right) = - \mathop \sum \limits_{i = 1}^{I} \log \left( {\mathop \sum \limits_{k = 1}^{K} \frac{{ \gamma_{k} }}{{\sqrt {2\pi \sigma_{k}^{2} } }}\exp \left( { - \frac{{(y_{i} - \mu_{k} )^{2} }}{{2\sigma_{k}^{2} }}} \right)} \right) $$

(2)

where K is the total number of Gaussian distributions (one for each tissue), and I is the total number of image elements. The update of the mixture proportion (γ_k) is performed by the expectation maximization (EM) algorithm and generates pixel-wise probability maps for cerebral gray matter, white matter and cerebrospinal fluid. These maps (obtained using SPM [9]) let us devise the likely tumor distribution so subsequent analysis and statistics can be performed only on the image patches that are the most likely to correspond to the tumor.

After this step is completed, using the information provided by the tissue distribution probability maps as well as imaging features from each MR volume an initial tumor ROI is obtained. This process involves the partition of the images into superpixels by using the SLIC algorithm [11], a popular method that implements an adaptation of the k-means clustering approach that provides and efficient and fast segmentation of an input image while combining color and spatial proximity to generate the superpixels. By default, in a simple implementation of the SLIC algorithm, only one parameter has to be set (the number of superpixels to be extracted) before being able to use it. In this work, we modified the approach to automatically select the number of clusters based on the histogram distribution of the input imaging volume (3D), setting an initial histogram partition parameter k at 10, roughly based on the type of normal and tumor tissues present on the input images [12].

Afterwards, the preliminary tumor ROI is obtained by selecting the superpixels that represent the regions with the lowest probability of being normal brain tissue according to the information provided by the described distribution probability maps for normal cerebral tissues (including gray matter, white matter and cerebrospinal fluid) [13]. This process is iterated under different orientations (axial, coronal, sagittal) to increase the accuracy of the initial ROI, hypothesizing that different tumor shapes may be easier for the algorithm to identify if visualized under different perspectives (e.g., a u-shaped tumor might be visualized as two different small structures on the axial view but as a continuous and more defined mass on the coronal view, as shown in Fig. 2). Finally, the preliminary tumor ROI is then obtained by taking the union of all regions generated across different perspectives, resulting in a single volumetric ROI.

Fig. 2.

To have multiple estimates of the tumor boundaries, multiple rotations are found along the tumor major axes so at the end of this process these measurements can be aggregated into the measurement variability map. In this work, the segmentation occurs on the three main tumor axes and ten random rotations on each axis.

2.2 Multimodal Tumor Boundary Selection

The next step involves identifying a set of tumor boundaries for the total tumor mass as well as for the tumor sub-regions (i.e., edema, necrosis, enhancing and non-enhancing tumor) using the preliminary tumor ROI defined in the previous step. This approach intends to represent the tumor heterogeneous boundary by doing multiple measurements and then combine them into a TVM to quantify uncertainty associated with segmentation boundaries.

The specific tumor boundaries are obtained as follow: A single definition for T2 abnormality was used to define a “T2 abnormal ROI” using the preliminary ROIs found on the FLAIR and T2 contrast images. Regions of edema are extracted by ranking the intensity rate of change on the preliminary tumor ROI histogram, defining as boundaries the locations where the highest total variation across the histogram corresponding to the tumor region are found. The tumor variability map is produced by aggregating the different binary ROIs obtained at each of these identified values on the tumor histogram.

The same process is followed to locate the enhancing and necrotic regions but using the post-contrast T1 sequence or a subtraction map (defined as T1+c - T1 volumes) [13]. Similarly, the TVM for these regions is then obtained by aggregating these different approximations of the tumor boundary (Fig. 3). A binary representation of the tumor mask is obtained by using the following approach on the TVM:

Fig. 3.

Box-plots showing the proposed method’s performance for the BRATS 2015 dataset. It plots the Dice similarity coefficient on the left (higher is better) and Hausdorff distance on the right (lower is better) when comparing a binarized tumor mask generated by the proposed approach and a segmentation gold standard. Label “Total” refers to all tumor components (edema, enhancing, necrosis and non-enhancing tumor), “Core” refers to the tumor core (excluding regions of vasogenic edema), and ‘Enhancing’ refers to active tumor cells with microvascular proliferations.

$$ I = \left\{ {\begin{array}{*{20}c} {P_{i,j} \ge \frac{n}{2} \therefore I_{i,j} = 1} \\ {P_{i,j} < \frac{n}{2} \therefore I_{i,j} = 0} \\ \end{array} } \right. $$

(3)

where I is the output binary image, P _i,j is the intensity at pixel location i,j of the TVM P and n is the number of discrete probability levels defined in the variability map. The output is a set of masks that represent the tumor extent and the different sub-regions with the possibility to calculate variability metrics (e.g., agreement ratio, standard deviation, statistical change measurement, and others).

3 Results

This proposed approach was tested on all 220 cases in the BRATS 2015 dataset and evaluated on three components: whole tumor, tumor core (enhancing and necrotic components) and active tumor (enhancing component). The Dice coefficient for total tumor mass of 0.74 (median: 0.77, 1^st quartile: 0.66, 3^rd quartile: 0.84), 0.54 for the tumor core (median: 0.57, 1^st quartile: 0.37, 3^rd quartile: 0.75) and 0.54 for the active tumor (median: 0.60, 1^st quartile: 0.29, 3^rd quartile: 0.76). Figure 3 shows the Dice coefficient as well as the Hausdorff distance metric for this dataset.

Fig. 4.

Tumor segmentation results for three different subjects. First and second rows represent the input post-contrast T1-weighted and FLAIR scans respectively for each subject (column), rows three to five show the result of the variability analysis to find different boundary estimations on each tumor (edema, enhancing and necrosis respectively) (non-enhancing tumor is not shown in this example). The color bar represents the pixel-wise probability for each tumor tissue. Finally, the bottom row displays a color coded binary mask that represents the total abnormality (all components) and subclasses (enhancing shown in red, vasogenic edema in blue and necrosis in green). (Color figure online)

On Fig. 4 some examples of input images and the output tumor variability maps for edema, enhancing and necrotic regions of the tumor as well as the representation of these tumor compartments overlaid on the image as binary masks. This binary representation of the TVMs (as previously described) is done with the purpose of similarity computations against binary gold standards as well as an easy integration with other processing pipelines (e.g. evaluation of clinical variables, genetic algorithms, etc.) or statistical approaches that require a binary input (Fig. 5).

Fig. 5.

Examples of brain tumor segmentation results for total tumor mass on three different cases. The first row shows one of the four imaging sequences that are used to perform the tumor segmentation (FLAIR is used on this example for the examples on the left and right and T1+c is used for the example on the center). The tumor variability map is displayed for total tumor on the second row, highlighting on red the regions where the algorithm is most certain that there is an abnormality in that pixel region and showing other color intensities for decreased belief in tumor abnormality according to the color bar located on the right. Finally, the third row shows the binarization of the variability map, according to a majority agreement of the different estimates (that is using a threshold of 0.5), this enables comparison with binary gold standards to evaluate accuracy and also allows for other more standard metrics and use the results in other processing pipelines. (Color figure online)

4 Discussion

We proposed a multimodal framework for automated, probabilistic brain tumor segmentation by using variability in estimates of the tumor boundary. By exploiting tumor heterogeneity from different imaging sources, this algorithm is able to automatically generate tumor probability maps or alternatively add a measurement of uncertainty to binary tumor segmentations. As the proposed approach iteratively measures the tumor boundaries, it is able to better detect and capture the heterogeneity found on brain tumors (e.g., being able to capture the shape of tumors with eccentric outlines). By explicitly quantifying the error associated with any given segmentation, we believe that this added information is critical to understand and judge the actual tumor extent by a radiologist or neuro-oncologist when interpreting the follow-up imaging data in the clinical setting.

When evaluating the results with other approaches proposed in previous years, our results are comparable to or surpass the mean performance of other algorithms [6] (Reza, Meier, Cordier, Bauer, Festa, Geremia, Buendia, Taylor, Shin). As a method to improve our results, we are also developing a classifier based on Convolutional Neural Networks (CNN) [14] to help in the definition of the preliminary tumor ROI and also to help reduce the number of false positives during the tumor boundary selection. Combining the result of our knowledge-based approach and the result of the CNN (trained to classify whether an individual voxel is part of a brain tumors using an independent dataset) might contribute towards better results on the different tumor contours (some examples where the proposed approach did not segment the tumor accurately are shown in Fig. 6).

Fig. 6.

Examples where our approach did not segment the tumor appropriately. The first column shows an example where the tumor is under segmented, we believe this is because a miscalculation on the prior information that is obtained as a preprocessing step and only the hyper intense edema was selected as part of the tumor, leaving the darker parts unselected. The second column shows a resection cavity on which the full area of enhancement is not captured and the third column shows how image artifacts can also cause problems while trying to segment the tumor.

The inclusion of variability calculations into segmentation methodologies can lead to better results and ultimately provide more meaningful data to clinicians as the knowledge of a measurement variation is fundamental to make more objective decisions. Future work includes the evaluation of variability on tumor biomarkers (such as tumor volume, thickness of enhancing margin, necrosis proportion, etc.) and impact on medical decision making to provide automated evaluations of tumor progression over time (as shown in Fig. 7) to be able to better evaluate treatment effectiveness and increase the radiologist’s efficiency at evaluating imaging studies.

Fig. 7.

Example of tumor progression over time. Showing volume measurement for each time-point with its respective error estimate. As time passes it becomes critical to evaluate if the action taken at a given point in time (e.g., chemotherapy, radiotherapy) had a significant effect on the tumor characteristics observed on the following MRI scans. The proposed segmentation method with analysis of tumor boundary variability enables clinicians to have different estimates of tumor characteristics (e.g., tumor volume, grow rate) and statistically define if there has been significant change over time or not (e.g., analysis of variance), essential for subsequent treatment planning.