Introduction

Radiomics [1,2,3,4,5,6,7] is science that studies the extraction and quantification of explicit image features from imaging studies (e.g., CT, MR, ultrasound, optical coherence tomography, mammography, microscopy) and their relationship to other clinical data (e.g., survival, molecular phenotype, co-morbidities). It involves delineation of a volume of interest (VOI) (e.g., surrounding a tumor or a region of interstitial lung disease), followed by computation of features of the VOI and the voxels within it (e.g., shape, margin sharpness, intensity distribution, and texture). Many researchers are engaged in radiomics studies, and most often, they use their own software to compute the image features. The use of disparate computation engines leads to difficulties in comparing results across research groups and in aggregating them for better-powered studies. In response to this, we developed a free, open-source, modular, feature extraction framework that executes a set of tested feature-extraction algorithms that can be used in radiomics research. It is also designed to run on a server so that research groups can use the same exact computations, allowing their results to be compared and aggregated.

While there have been other efforts in the academic and industrial worlds to create frameworks for collaborative computational research [8,9,10,11,12,13], there are few available tools for image feature computation and analysis. One such tool is IBEX [13], which provides an end-to-end solution with a graphical interface for researchers to annotate image datasets, compute radiomics features, and analyze the resulting feature data. A more recently released software package is Pyradiomics [14], which provides a set of python libraries that can be invoked to process one or multiple image series and compute radiomics features. Our system, the Quantitative Image Feature Engine (QIFE), is a locally run or server-based system exclusively for radiomics feature computation. Our tool differentiates itself from the others by focusing on out-of-the-box parallel processing, modularity, and its ability to be plugged into existing workflows. It accepts as input DICOM files containing volumetric data together with DICOM Segmentation Objects (DSOs) specifying the volumes of interest (objects) to be processed. Code is also provided to modify the images and/or segmentations to allow simulation of different segmentations, acquisition, and/or reconstruction conditions. We developed the QIFE using MATLAB 8.6 R2015b (Mathworks, Natick, MA), exploiting its ability to generate parallel threads for performance improvement on multiprocessor systems, and we distributed it in multiple formats, including an open-source code repository to allow collaboration and modification, and a self-contained Docker [15] image for easy tool sharing. Sharing code widely allows the QIFE to be continuously tested, and any identified bugs can be fixed and re-shared to the community [16,17,18]. Finally, we have put a strong emphasis on provenance and research attribution. As part of its output, the QIFE stores the date and time, the version that was run, its run-time configuration parameters, and a citation defining each feature computed, fostering reproducible science [19, 20].

“Quantitative Image Feature Engine (QIFE) Architecture” section presents the architecture of the engine describing its internal workings and organization. “Included Components” section lists and describes all components included in the engine at the moment of this publication. “Speed and Memory Tests” section describes methods and results of an experiment using a large cohort to analyze the time and memory requirements using different modes of parallel processing. “Feature Comparison” section describes how to obtain the engine, either in as source code or in a Docker image for execution. “Current Use” section discusses several use cases currently underway. “Limitations” section discusses some limitations of this software. Finally, “Conclusions” section summarizes and offers concluding remarks.

Materials and Methods

QIFE Architecture

The QIFE generates quantitative features from a volume of interest, defined by a DSO, from a DICOM series. It consists of a managing framework and four stages: input, pre-processing, feature computation, and output, as shown in Fig. 1. Each stage contains one or more components that are loaded at run time. The managing framework provides functions to facilitate the communication between components and stages. The output of each stage serves as the input for the next one. We designed the stages to be modular, such that each component can be swapped out for another one as long as the new one implements the stage-defined interface. Information is passed from the framework to each component on request. When a component finishes processing, it returns its output to the framework for collation.

Fig. 1
figure 1

The Quantitative Image Feature Engine (QIFE) architecture. The rounded rectangles represent stages, and the rectangles within represent components. Data flows occur from left to right. The Managing Framework (1) reads the configuration file and sets up all stages for processing. The input stage (2) loads the segmentation and image files into the engine, and then the pre-processing stage (3) applies its components sequentially to normalize or otherwise modify the input data appropriately. Next, the feature computation stage (4) runs each of its components independently and extracts quantitative descriptors of the images and segmentation. Finally, the output stage (5) generates files summarizing the results. At the end of execution, the managing framework outputs a log file with the information required to reproduce the results

Full size image

The Managing Framework (1) is the fixed structure of the engine, within which the stages and components communicate. It parses the configuration file at the beginning of the run, loads the necessary components, provides global helper functions for each component, facilitates communication between components, and collates the results from each stage.

The configuration file is a text file that defines which components to load, sets global configuration parameters, and overrides individual default parameters in each of the component. Appendix 1 lists all of parameters currently accepted by this configuration file, and Appendix 2 gives an example of such a configuration file.

The input Stage (2) loads the data that will be processed by the engine. This includes DICOM files containing one or more image series and one or more DSOs describing the location of the VOIs within each relevant DICOM image series. The components in this stage receive the path and filenames or Unique identifier (UID) [21] of the image series that need to be processed and loads them into memory. The output of this stage includes an array storing metadata for each slice (e.g., slice location, thickness, etc.) and two 3D matrices, one for the intensity values in the scan and one for the binary mask representing the volume of interest. The output of this stage is submitted to the Managing Framework.

The Managing Framework then submits the output of the Input Stage to the Pre-processing Stage (3), which then executes code to alter the original scan data. Components in the pre-processing stage can be invoked to apply filtration, add noise, and/or alter the VOI to facilitate experiments, e.g., to test stability of the features extracted with respect to imaging and segmentation conditions. The components of this stage are applied sequentially, using the processing order established in a configuration table, with the output of each component applied to the input of the following component. After all components have executed, the stage outputs its results to the Managing Framework.

The Managing Framework then submits the output of the Input Stage to the Feature Computation Stage (4), which contains components that compute quantitative features describing voxel data within the VOIs defined by the supplied DSOs. Each feature component can return a single value, a group of values, and/or an image. Finally, this stage collates all results into a structure and submits it to the Managing Framework.

Finally, the Managing Framework submits the results of the Feature Computation Stage to the Output Stage (5), which contains components to convert the outputs to user-readable formats. Components in this stage can generate multiple outputs depending on the configuration. For example, the CSV component groups results per patient and writes them into CSV files for easy import in other tools, and the cross-sectional image generator component outputs example images.

We next describe the specific components that we have implemented.

Included Components

In this section, we describe the components currently included in the QIFE. For an up-to-date list and description of all features in the current version of the engine included, please refer to our Github repository https://github.com/riipl/3dpipeline.

Input Stage

Components in the input stage load data from outside the engine and send it to the managing framework for further processing by the following stages. The engine currently implements a single component to load DICOM image files and DSOs for image and segmentation data, respectively.

The DSO/DICOM loader loads DSOs [22] and their referenced DICOM [21] image files. Parameters required by this component include the path of the directory containing the original DICOM images and the path of the directory containing the DSOs. This component consists of three different sub-components: indexing, segmentation-loading, and scan reading.

First, the indexing sub-component scans all the directories provided and checks the type of all files with the extension “DCM.” It then determines if the file is a valid DICOM image or DSO; if not, the file is logged and ignored. The component generates three hash tables: (1) mapping the DSOs’ UIDs to their file path, (2) mapping DICOM images’ UIDs to their file path, and (3) mapping DICOM Series’ UIDs and Instance number [21] to their file path. The first and second hash tables are used to minimize the time required to link the individual bit planes in the DSOs to the correct DICOM images. The third hash table is used to find DICOM images that are not referenced by the DSOs but might be needed for certain features that use values outside the segmentation (e.g., edge features). The hash tables can be stored in the root directories of the provided paths to avoid re-computation in future runs with the same dataset.

Next, the segmentation-loading sub-component loads the DSO corresponding to the scan being analyzed into memory. This sub-component sorts the segmentation planes by their slice location parameter in the DSO’s DICOM header.

Finally, the object-reading sub-component loads the corresponding DICOM images referenced by the DSO. As some computations (for example, edge features) will require images superior and inferior to the segmentation volume itself, a padding parameter (in mm) is provided. In this case, images superior and inferior to the most superior and inferior aspects, respectively, of the segmentation VOI that are closer than the distance provided by the padding parameter are loaded into the appropriate locations superior and inferior to the VOI. The intensity values of all loaded images are then normalized [23, 24] by their slope and intercept metadata values if present in the DICOM image files [21].

This component generates three outputs: (1) a 3D integer matrix containing the possibly normalized values from the DICOM images corresponding to those referenced by the DSO and possibly padded superior and inferior to it, (2) a 3D binary matrix same size as (1) with a value of 1 wherever the voxel is part of the segmentation and zero otherwise, and (3) a cell array with a cell per each DICOM image loaded containing all their image metadata.

Pre-processing Stage

The pre-processing stage receives the segmentation and image data loaded by the input stage and runs components that may modify these data. Each component in this stage runs sequentially using the output of each component as the input to the next. This stage then outputs the modified segmentation and intensity data. The engine currently implements components that can deform the DSO, bridge small gaps, eliminate all but the largest volume, and fill holes in the original segmentation. These can be enabled or disabled individually using the configuration file as desired.

Segmentation Deformation

This sub-component can be used to apply morphological operations [25] to a supplied segmentation. This component takes as input parameters the desired operation (i.e., erosion or dilation) and the size of the spherical structuring element in millimeters.

Topology Preservation

This sub-component merges disjoint components of segmentations that are within N voxels of each other using a morphological closing operation, with N being a user-defined parameter. This can be required, e.g., when the segmentation deformation sub-component splits a segmentation into two or more disjoint regions during erosion.

Maximum Connected Volume Selection

This sub-component groups all connected voxels and, if there is more than one group, eliminates all but the group with the largest volume.

Hole Filling

This sub-component finds unsegmented regions that are completely surrounded by segmented regions and includes them in the segmentation to be processed.

Feature Computation Stage

Components in the feature computation stage receive as inputs the (potentially) modified segmentation and image intensity values generated by the pre-processing stage and compute quantitative descriptors of the volumes of interest specified by the segmentation and the data they contain. Each component in this stage runs independently of the others, i.e., the output of each component does not affect any other component. The output of this stage is an aggregated list of all the descriptors generated by the components. The engine currently implements components to extract the following features: size distribution features, intensity distribution, edge sharpness, local volume invariant integral (a metric of shape), surface roughness, sphericity, and Haralick’s texture features.

Size Distribution Features

This component computes the size of the volume of interest. It reports its volume (mm3), its largest cross-sectional area (mm2), and its largest diameter (mm).

Intensity Distribution Features

This component characterizes the global distribution of intensity values by computing summary statistics [26] (mean, standard deviation, geometric mean, harmonic mean, minimum value, maximum value, kurtosis, trimmed mean, skewness) of the intensity value distribution in the segmentation.

Edge Sharpness Features

This component describes the statistics of the intensity changes between the interior and exterior of the segmentation. To define the surface of the segmentation, we use the MATLAB function isosurface [27] to create a triangle mesh that fits the segmentation. We then decimate the mesh using MATLAB’s reducepatch function [28] to a value set in the configuration file (defaults to 600 triangles). Next, the component calculates normal vectors at the centroid of each triangle of the mesh, as shown in Fig. 2, and then trilinearly interpolates the voxel intensities along the normals every 1 from 5 mm outside to 5 mm inside the segmented volume. Finally, we fit a sigmoid function [29, 30], characterized by scale, S, and window, W, parameters, to the interpolated intensity points using MATLAB’s nlinfit [31] (nonlinear regression) with the following parameters: max iterations = 300; TolFun = 1e-8; TolX = 1e-8; Display = off, DerivStep = eps^(1/3); FunValCheck = on; Robust = on; WgtFun = bisquare. The fit can fail when the values inside and outside the VOI are similar, e.g., when a lung tumor is next to the chest wall, or when the normal traverses noisy data or heterogeneous tissues; in these cases, the component does not record the S and W parameters. Finally, the component returns summary statistics of the distribution of S and W parameters, and the percentage of normals for which the fit failed.

Fig. 2
figure 2

a A mesh obtained by running the marching cubes algorithm on a segmented tumor. b The decimated mesh and the normal to each face

Full size image

Local Volume Invariant Integral (LVII) Feature

This component characterizes the local curvature of the volume of interest (VOI) by generating a sphere of size N (configuration options in Appendix 1) centered at each point along the boundary of the VOI and calculating the percentage of the sphere filled by the VOI. This percentage will vary depending on the local curvature of the VOI; the component returns summary statistics (mean, standard deviation, geometric mean, harmonic mean, minimum value, maximum value, kurtosis, trimmed mean, skewness) of the distribution over all points on the surface of the VOI. Figure 3 shows some examples of LVII illustrated in 2D for simplicity.

Fig. 3
figure 3

2D Example of LVII. The percentage of the circle filled by the VOI changes depending on the local curvature

Full size image

Roughness Feature

Surface roughness [32] characterizes the surface of the volume of interest (VOI) by computing surface deviations compared to a simulated smooth surface. To compute surface roughness, the QIFE generates a simulated smooth surface by first generating two new volumes using the morphological closing and opening operations, respectively, followed by combining them with a voxel-wise “and” operation. The size of the structuring element used in the morphological operations determines the cutoff frequency of this smoothing filter (configuration options found in Appendix 1). Second, the shortest distance between this smooth surface to each of the boundary voxel is calculated to find the deviations. This component returns summary statistics of the (average, standard deviation, kurtosis, skewness, root mean squared, maximum valley depth, maximum peak height, maximum height of profile of this difference (Japanese Industrial Standard (JIS) [33], and German Institute for Standardization (DIN) [34]).

Sphericity Feature

This component characterizes how similar the shape of the volume of interest (VOI) is to a sphere by calculating [35]:

$$ \psi =\frac{\pi^{\frac{1}{3}}{(6V)}^{\frac{2}{3}}}{A} $$

where ψ, V, and A are the sphericity, volume, and surface area of the segmentation, respectively.

Haralick’s Texture Features

To characterize texture features, this component generates gray-level co-occurrence matrices (GLCM) [36] from the raw intensity values for each voxel within the segmentation for 13 directions corresponding to the 26 connected voxels [37] and at the distances specified by the configuration file (configuration options found in Appendix 1). The Haralick’s Features from each of these 13 matrices are aggregated (average, standard deviation, minimum and maximum) by distance, making the reported values rotational invariant. Table 1 lists the texture features we compute; this component outputs the aggregated statistics generated for each distance.

Table 1 List of Haralick features extracted from the gray-level co-occurrence matrix. Details of the implementation of each of these features can be found in the references
Full size table

Output Stage

The components in the output stage receive lists of feature values from the feature computation stage and generate user-readable files and images to be used in further analysis. The engine currently implements components that produce the following outputs per segmented VOI: a comma-separated feature value file, a run information file, and example cross-sectional images/segmentations.

CSV Exporter

The CSV exporter writes the feature names (defined in each feature component configuration) and results generated by the feature-computation stage to a CSV file [38].

Run Information Exporter

This component creates a file detailing the run date and time, the UIDs of the files that were processed, the configuration, and the software version used to run the experiment. This file allows researchers to repeat the experiment, as it documents when it ran, the version of the QIFE and components it was using, the UID of the objects that were processed, and the references defining each computation performed.

Cross-sectional Image Generator

This component outputs an image file displaying the slice with the largest cross-sectional VOI area in each of three directions (sagittal, coronal, axial), overlaid with the outline of the DSO. This image is useful for debugging and demonstration purposes as it gives a summary of the volume used to generate the results. Figure 3. 2D Example of LVII. The percentage of the circle filled by the VOI changes depending on the local curvature

Figure 4 shows an example of a processed lung nodule from a CT image volume.

Fig. 4
figure 4

Output from the cross-sectional image generator showing, from left to right, axial, coronal, and sagittal cross-sections of a lung CT-scan and the tumor boundaries given by the supplied segmentations (solid red line). The location of each cross-section corresponds to the maximum cross-sectional area for each orientation

Full size image

Evaluation

To evaluate the computational efficiency of the QIFE as a function of parallelization choice, we extracted features from 108 CT scans presenting lung tumors ranging in size from 0.37 to 371.53 cm3 (mean = 31.35 cm3; s.d. = 66.78 cm3). We ran the QIFE with all features active at their default settings. We repeated the experiment for different parallelization levels (none, object, feature, internal) and stored the total running time and maximum memory usage for each run.

Results

Speed and Memory Tests

Table 2 shows the total run time and the peak memory usage for each parallelization level. Total run time was obtained using MATLAB’s profiler; peak memory usage was obtained using Sysinternals Process Monitor. Running without parallelization resulted in the longest running time with the least memory, while parallelizing at the object level took the least time but required the most amount of memory, as multiple copies of the engine must be memory-resident at the same time. Feature-level parallelization did not improve speed much compared to no parallelization. This is to be expected because (a) there is overhead, which is not reduced by parallelization, and (b) we used only four cores and of the nine feature classes executed in parallel; some are computationally intensive and others are significantly less so. In this scenario, it is unlikely that the computationally intense feature classes will always be processed in parallel, effectively limiting the expected performance increase.

Table 2 Total run-time (seconds) and peak memory usage (GB) for the QIFE while processing 5 CT lung scans presenting masses from 0.81 to 59.04 cm3 (mean 14.113, s.d. 25.21 cm3). Total run time was obtained using MATLAB’s profiler, peak memory usage was obtained using Sysinternals Process Monitor. We used a Clevo P370EM laptop, containing an Intel I7-3740QM (4 cores, all available to the engine) CPU clocked at 2.70GHz and 16.0 GB of RAM and a solid-state drive (Samsung EVO 850)
Full size table

Feature Comparison

To demonstrate the features of the QIFE, we processed four lung nodules of different in shapes, sizes, and compositions, as shown in Fig. 5. Nodules A and C are mostly solid, while nodules B and D present ground glass components (Table 3). Comparison of chosen features between the four nodules is shown in Fig. 5. Nodules A and C are solid tumors, while nodules B and D are part solid and part ground glass. As expected, solid tumors (nodules A and C) present a higher mean intensity and lower texture variance (mean GLCM variance) than the tumors with partial ground glass (nodules B and D). Nodules A and B show a high surface variation, giving them higher roughness values compared to nodules C and D. Finally, nodules A, B, and C are mostly convex, while nodule B presents multiple concavities, therefore having the lowest sphericity. It shows selected feature output by the QIFE for these four nodules. The volume feature shows that nodules A and B are similar in size, while nodule C and nodule D are much smaller. Nodules B and D present a lower mean intensity, as expected due to the lower density of the ground-glass portion of the tumor. The roughness feature characterizes high frequency surface variations; as Fig. 5 shows, Nodules A and B surfaces are less smooth, resulting in a higher roughness values. The sphericity feature is proportional to the ratio of surface area to volume and is therefore higher for more spherical objects, resulting in higher values for nodules A, C, and D. Finally, tumors with ground-glass components (nodules B and D) have more heterogeneous density and therefore have lower mean intensities and larger texture features (GLCM).

Fig. 5
figure 5

Four example nodules: Each row shows three cross-sectional views of a different nodule. The location of each slice corresponds to the maximum cross-sectional area for each orientation

Full size image
Table 3 Comparison of chosen features between the four nodules shown in Fig. 5. Nodules A and C are solid tumors while nodules B and D are part solid, part ground glass. As expected, solid tumors (nodules A and C) present a higher mean intensity, and lower texture variance (mean GLCM variance) than the tumors with partial ground glass (nodules B and D). Nodules A and B show high surface variation, giving them a higher roughness values compared to nodules C and D. Finally, nodules A, B, and C are mostly convex, while nodule B presents multiple concavities, therefore having lowest sphericity
Full size table

Deployment and Collaboration

Code Repository

The QIFE Version 1.0 has been released as a BSD-licensed [39] open-source project in Github, (https://www.github.com/riipl/3dpipeline). Collaboration from the research community is encouraged either by contributing code, creating pull requests, or by testing and reporting. The complete source code can be downloaded from the repository; Mathworks MATLAB 8.6 R2015b is necessary to run the engine.

Docker

We have also released a compiled version of the engine. The compiled version of the engine requires that the correct release (9.0.1 at the time of this publication) of MATLAB’s Compiler Runtime (MCR) be present in the host computer. To simplify deployment, we have provided a Docker container hosted in Dockerhub (https://hub.docker.com/r/riipl/3d_qifp/), containing the correct MCR and the compiled QIFE, making it completely portable, and avoiding any possible mismatch between the QIFE and the user’s computer.

Discussion

Current Use

The QIFE is currently deployed as part of an internal web application, known as the Quantitative Image Feature Pipeline (QIFP), at Stanford University. A web interface was created to act as a wrapper between the Docker module containing the QIFE and the users. Researchers can upload their scans and segmentations, and the application returns a CSV with their results after processing.

The QIFE was also used in a study of feature sensitivity to segmentation to evaluate whether digital biopsies could be used to speed up segmentation while retaining information [40]. In this study, the QIFE used pre-processing components that deformed the original 3D digital biopsies to simulate multiple manual-segmentations.

Finally, the QIFE was also used in a study to compare feature extraction implementations and showed reasonable correlations with features of the same type [41].

Limitations

One limitation of QIFE is that it does not use the latest advances in GPU technology to speed up computation. We decided to forego this optimization as access to computers with powerful GPUs is not always a possibility, and we wanted to make the QIFE universally compatible. This limitation can be addressed at the component level. While the engine logic does not use the GPU, our components could implement GPU functionality in the future. Components using the GPU should either have a fallback to use the CPU in case a GPU is not present or show in their documentation that a GPU is required for them to function.

Another limitation is that currently the QIFE only computes 3D features, while there are multiple studies in the literature showing the usefulness of 2D features. For example, in previous studies, our group created a 2D feature extraction engine and successfully used it for image retrieval [42] and to generate features that correlate with gene expression [43]. Integrating other features such as these could expand the applicability of the QIFE.

Conclusions

In this paper, we presented the QIFE, a portable, open-source feature extraction framework that focuses on modularity, standards, parallelism, provenance, and integration. We describe the different stages of QIFE and the components that can be loaded and executed at each stage.

Integration with existing research workflows can be implemented by changing the input-stage and output-stage components. By separating the input-output logic from feature processing, researchers can easily add MATLAB code to read other input formats, generate new features, and implement new output types.

The QIFE also outputs information about the configuration and components involved in generating a set of results. By storing these log files, provenance of result sets is recorded and experiments can be repeated or compared across datasets with confidence on the similarity of the processing engine.

Finally, we also tested the QIFE at different parallelization levels measuring memory and time taken for each setting. Speed-ups can be obtained by parallelizing execution at the cost of memory usage. Different parallelization levels provide different trade-offs, and the optimal setting will depend on the size and composition of the dataset to process and the number of processing cores available.