Keywords

1 Introduction

Lung cancer has the highest mortality rate in both males and females where the 3-year survival rate for patients with lung cancer is 25% [19]. There are no obvious symptoms at the beginning of lung cancer, and as a consequence, most patients seek treatment at the later stage, minimizing survival chances. Therefore, early detection and diagnosis of lung cancer is of the utmost importance [1]. The chest computed tomography (CT) imaging modality provides high-resolution images of nodules with lavish details; however, pulmonary nodules have inhomogeneous densities and lower contrast compared to blood vessel segments and other anatomical structures, increasing the complexity of nodule detection [2, 12]. To assist radiologists in automatically detecting nodules and replacing the time-consuming manual delineation of nodules, Computer-Aided Detection (CADe) and Diagnosis (CADx) systems are developed. The latest technologies use Artificial Intelligence (AI) to assist in the auxiliary diagnosis of the disease and improve the overall accuracy of the diagnosis while decreasing the detection time [10]. In recent literature, researchers have presented deep learning-based CAD systems with promising results. The convolutional neural network (CNN) framework has been used for the classification of nodules [7] and the reduction of false positive (FP) [20]. Shen et al. proposed a Multi-Crop CNN (MC-CNN) [18] and Setio et al. developed Multi-View CNN (MV-CNN) [16] to classify lung nodule. A 3D-CNN model based on Volumes of Interest (VOI) and a Fully Convolutional Network (FCN) was used to produce a score map for nodule classification [9]. Both CADe and CADx systems have been independently investigated [6], CADe are unable to provide lesion’s radiological characteristics, consequently missing crucial information, while CADx systems do not identify lesions and therefore do not possess high levels of automation. Therefore, a new and advanced CAD system is needed that incorporates the benefits of detection from CADe and diagnosis from CADx into a single system for better performance.

Fig. 1.
figure 1

Overview of our CAD system comprising of 3D residual U-Net and multi-Region Proposal Network for lung nodule detection and classification.

Full size image

1.1 Contribution

Our contribution is as follows:

  • 3D Residual U-Net Model A novel nodule detection method is proposed using 3D CT images for candidate nodule detection; compared to existing 2D U-Net models, our 3D residual model considers rich spatial features and therefore has more discriminative selection criteria.

  • Multi-Region Proposal Network (mRPN) We added four RPNs so that nodules with varying diameters can be detected with ease and efficiency. The RPN split-and-merge cascade network mitigates the problem of undetected small nodules.

  • Malignancy Score-Based Approach (MSBA) Malignancy score is calculated to classify each detected nodule into one of the four classes based on its aggregate malignancy score.

  • False-Positive Reduction Algorithm (FPRA) We proposed an algorithm for a false positive (FP) reduction rate. Comparative results are much better than the existing FP reduction algorithm. The performance of the proposed CAD system is evaluated with state-of-the-art CAD systems using various performance evaluation metrics. The experimental results showed that the proposed method can not only be used for detection but also performs well for the classification of pulmonary cancer nodules as malignant and benign.

1.2 Paper Organization

Section 2 describes our approach, Sect. 3 discusses the implementation details, Sect. 4 describes the experimental results, and Sect. 5 concludes this paper.

2 Our Approach

We improved three aspects of the lung cancer detection models. First, the datasets used by most authors do not consider the lung wall. We considered location as one of the most important features and therefore found that most nodules occur in close proximity to the lung wall. Therefore, our model avoided omitting the edge of the lung. Secondly, we used 3d U-Net to filter candidate nodules [22]. Finally, we applied CNNs for nodule classification.

2.1 Pre-processing

Our in-house dataset comprises 56 patients’ CT scans using a GE CT scanner (with contrast and 3mm slice thickness) in DICOM format. Each CT scan is composed of 80 to 200 distinct slices and the primary tumor was manually delineated using 3D-Slicer software. We resized each CT-scan data using average upsampling and average downsampling with bilinear interpolation by ImageJ software. Furthermore, we scaled the pixel value using the min-max scalar method and applied the CLAHE (Contrast Limited Adaptive Histogram Equalization) method. Data augmentation is necessary because deep learning-based models require large training datasets. The positive dataset to train our proposed model was insufficient, leading to the overfitting problem. We increased the positive dataset sample by using \(128\times 128\times 128\) window size. We used affine transformations (rotation [\(0^\circ \) to \(270^\circ \) around the center point], flipping, translation, and scaling) along with image enhancement by Gaussian High Pass filter with kernel size \(3\times 3\) to improve image quality and sharpening filter. For each image \(P_{n}(\textrm{z})\) shown in Eq. (1), and S in Eq. (2) where \(r_t\), \(z_t\) is the reset and update to apply the affine transformations, respectively. While \(\hat{h}_{t}\) is the final augmented state of the data. The affine transformations were standardized so that the average samples have variance=1 and mean=0.

$$\begin{aligned} P_{n}(\textrm{z})=\frac{1}{\sigma \sqrt{2\pi }}\textrm{e}^{-}\frac{(\textrm{z}-\mu )^{2}}{2\sigma ^{2}} \end{aligned}$$
(1)
$$\begin{aligned} \textrm{S}[\mathrm {r_t},\mathrm {z_t}]=\Sigma _{\textrm{k}=-\textrm{r}}^{\textrm{n}}\textrm{W}(\textrm{b},\textrm{W}+\textrm{hr}) \end{aligned}$$
(2)

2.2 RPN Split-Merge Cascade Network

For the detection of various nodules having different diameters, we used varying levels of \(RPN_{Lx}\) referring to different sizes of the nodule. We set the \(RPN_{L1}\) as small anchors to detect the diameter of the nodule \(\tau \) that ranges from 3 mm to 10 mm and has a volume \(\upsilon <\)= 80 mm\(^3\) while \(RPN_{L2}\), \(RPN_{L3}\) and \(RPN_{L4}\) have large anchors to detect nodules ranging from \(\tau \) = 10 mm–20 mm or \(\upsilon \) = 80–200 mm\(^3\), \(\tau \) = 20 mm–30 mm or \(\upsilon >\)= 200–300 mm\(^3\) and \(\tau >\)= 30 mm or \(\upsilon >\) 300 mm\(^3\), respectively. The motivation behind these RPNs is the four stages of lung cancer that are categorized by different diameters, while \(RPN_{L1}\) is for all the input nodules. The RPN split and merge cascade network starts with the \(RPN_{L1}\) which is further split to either \(RPN_{L2}\) or the rest and then in the next step it is input for the \(RPN_{L3}\) or \(RPN_{L4}\). Since each RPN level generates separate RoI sets, a merging layer is required that combines the RoI sets into one, the RPN levels merge layer \(RPN_m\) takes the input RoI sets from all the RPN levels (\(RPN_{L1}\), \(RPN_{L2}\), \(RPN_{L3}\), \(RPN_{L4}\)) and outputs an aggregate RoI set \(RoI_{agg}\). For the possibility of duplicate RoI or low objectiveness score RoI, we used the non-maxima suppression (Non-MS) when the intersection over union (IoU) overlap is above the threshold (threshold set at \(\rho _t = 0.5\)). After using the Non-MS, we selected the top hundred RoI with low objectiveness scores for further use. Nodule detection using different levels of RPN having various anchors improves the detection phase, as both diameter and volume are taken into consideration.

2.3 3D Residual U-Net Training Strategy and Architecture

Our proposed model relies heavily on exploiting the symmetries of the 3D space [16]. Therefore, the lung CT scan is converted into 3D fragments which are used as input for the 3D residual net. For 3D CT images containing lung nodules, the lung nodule regions were cropped to a size of \(128\times 128\times 64\). The 3D residual net detects the module malignancy based on the characteristics obtained from the input image, and the probability of cancer stage is estimated [5]. We used binary valued threshold (corrosion & expansion) and Laplacian of Gaussian to segment the lung nodules (including lung wall), morphologic closing was performed and connected component operations are labeled in order to remove background and noise. A hole-filling algorithm based on contour information was also used to reserve the nodules on the lung wall. We obtained a collection of the interval \([x_k,y_k]\) that contains all the intervals of \(C_n\). \(\textbf{M}\), so if n is large enough, \(\displaystyle \sum _{k=1}^n\left| y_k-x_k\right| <\eta \). But \(\displaystyle \sum _{k=1}^n\left| f_c(y_k)-f_c(x_k)\right| =1\). The segmentation issue is addressed by taking N partitions of the set of features represented by P of classes M, thus minimizing the cost term of the error function by assigning the pixel P in Eq. (3) and Eq. (4)

$$\begin{aligned} \min _{\textbf{M},\textbf{x}} \sum _{i=1}^{N}\Vert \textbf{y}_{i}-\textbf{P}\textbf{x}_{i}\Vert _{2}^{2} \quad \text{ s.t. } ~ \forall i ~ \Vert \textbf{x}_{i}\Vert _{0} \end{aligned}$$
(3)
$$\begin{aligned} E=\frac{1}{2}{\sum }_{k=1}^{N}{\sum }_{l=1}^{M}{R}_{kl}^{n}{V}_{kl}^{2} \end{aligned}$$
(4)

Another challenge for our proposed model was learning the complex inner spatial relationship between parameters using deeper CNN. We added multiple residual blocks in the middle of the 3D U-Net model, which is capable of producing higher-level packet information. Taking into account the complex anatomical structures surrounding the lung lesion, we needed an effective method to use contextual information at multiple levels [15]. An overview of the CAD system for the detection of lung nodules using the 3D residual U-Net and multi-Region Proposal Network (mRPN) is shown in Fig. 1.

Algorithm 24
figure a

. Malignancy Score-Based Approach

Full size image

2.4 Malignancy Score-Based Approach (MSBA)

For the classification of detected lung lesions in the nodule detection phase, the Malignancy Score-Based Approach (MSBA) is used to achieve the assessment of nodule malignancy of candidate lesions. For this phase, the regions of interest (RoIs) that are marked by the nodule detection phase are redefined from each marked location resulting from the last step. MSBA assigns the malignancy score to RoIs by considering the metastasis information provided in the data set to classify the candidate nodule into T0, T1, T2, and T3 stages. The details of MSBA are provided in Algorithm 1. The neighboring pixels in 3D surrounding the RoIs are taken into consideration in terms of intensity values and their eigenvalues (Hessian Matrix and the Gradient Matrix) to assign an aggregate score to each candidate-marked lesion. The result of this step is the allocation of the average score to all candidate nodules.

3 Implementation

In addition to our in-house lung cancer CT dataset, we used publicly available datasets namely the LIDC-IDRI [4], ANODE09 [8], and LUNA16 [17] for evaluation. The probability of nodules is calculated for the nodule candidates generated by the classification model. On the basis of this probability, we mark the nodules as benign and malignant in Fig. 2. To reduce false positive results, we have proposed an algorithm that considers the probability of the candidate nodule and further improves the classification of the nodules into different stages while omitting false positive results at each stage. The details of our proposed algorithm for false positive reduction are provided in Algorithm 2.

Table 1. Confusion matrix of lung cancer classification Using 3D Residual U-Net
Full size table
Fig. 2.
figure 2

Qualitative results of our proposed model on in-house dataset showing accurately detected benign nodules (T0) in (a),(b),(c); benign nodules (T1) in (d),(e),(f); malignant nodules (T2) in (g),(h),(i), malignant nodules (T3) in (j), (k) (l) and accurately detected negative nodules in (m),(n) and (o).

Full size image
Algorithm 25
figure b

. False-Positive Reduction Algorithm

Full size image

4 Experimental Results

Our results are obtained using the concept that if a detected nodule is very close to the annotated nodule, we gain a score and the score is related to the FROC curve on sensitivity at 1/8, 1/4, 1/2, 1, 2, 4, and 8 [21]. We obtained a score of 0.974 (MAX = 1) by randomly selecting data records as the test set (excluding training and validation dataset), and an accuracy of approximately 0.997 was recorded. The results are validated using the common performance metric of computer-aided detection and diagnosis systems, i.e., average accuracy, specificity, and sensitivity. For a detailed performance analysis of our CAD system, we applied the ROC curve, which presents the TPR (True Positive Rate) as the FPR function (False Positive Rate). We plotted the sensitivity (TPR) with their respective FPR to compare our CAD system with state-of-the-art CAD systems.

The effectiveness of our method 3D Residual U-Net is verified by comparing it with Convolutional Neural Networks (CNN), Massive training artificial neural networks (MTANNs), Fully Convolutional Networks (FCN), Region-based Fully Convolutional Networks (RFCN), and RNN (Recurrent Neural Networks), the performance results are depicted in Fig. 3. Table 2 summarizes the performance of our proposed CAD system in comparison to other CAD systems in terms of accuracy, sensitivity, specificity, False Positive (FP), and average FP (using FPRA). It is visible from the confusion matrix in Table 1 that the proposed model maintains a high sensitivity for the classification of types of lung cancer even in the region where FP per scan is low. Although the proposed model achieved a high sensitivity value for stage classification for T0, T2, and T3, the sensitivity value of T1 decreases as the FP per scan becomes small, which is impractical in the clinical environment [14]. A summary of our proposed CAD system in terms of stage classification is shown in Table 1. The experimental results demonstrate the superiority in the classification and class generalization of our proposed 3D Residual U-Net based CAD system. A comparison of nodule classification by CAD systems is shown in Fig. 4.

Table 2. Performance comparison of our proposed model with state-of-the-art
Full size table
Fig. 3.
figure 3

The plot illustrating the sensitivity of the 3D residual model on the Luna16 dataset with the state-of-the-art CAD systems. The x-axis denotes the average number of FP per scan while the y-axis represents the sensitivity.

Full size image
Fig. 4.
figure 4

Comparison of our 3D residual U-Net CAD system nodule classification with state-of-the-art CADe systems.

Full size image
Table 3. Comparison of our proposed model on nodule classification with the state-of-the-art CADe systems using CT dataset (subset of inhouse data, LIDC-IDRI [4], ANODE09 [8], LUNA16)
Full size table

A comparison of our model with the existing state-of-the-art CADe systems on the LIDC-IDRI dataset with varying nodule sizes is shown in Table 3. We have compared the detection accuracy of our CADe system with the detection accuracy of five other existing systems, which are evaluated on the dataset from the LIDC-IDRI database. The high accuracy of our proposed model with varying sizes signifies the detection capability of our 3D Residual U-Net model. Our method has promising results in discriminating cancer nodule types without compromising detection accuracy (Fig. 4).

5 Conclusion

We proposed a novel CAD system for the automatic detection and classification of lung nodules in CT images. Our CAD system comprises two models, the 3D residual U-Net and multi-Region Proposal Network (mRPN), which have demonstrated effective nodule detection results even with small-sized lesions. For the classification of nodules, the aggregate malignancy score is calculated for each detected nodule. Based on this score, detected nodules are classified into four classes; T0, T1, T2, and T3. Experimental results illustrate the efficacy of our proposed CAD system in comparison to state-of-the-art CAD systems that use various performance evaluation metrics. Our proposed CAD system is generic and therefore could possibly be extended to the detection of other cancers.