Keywords

1 Introduction

Chest X-rays reveal most of the disorders of the lungs including an outline of the heart and some of the major blood vessels. The prognosis of lungs is vital as the lungs are associated with some of the common diseases like asthma, chronic bronchitis, emphysema, chronic obstructive pulmonary diseases (COPD), pulmonary fibrosis, sarcoidosis, etc., which are a result of infection and pollution. The September 2018 report published by Indian Council of Medical Research (ICMR) and others [1] puts India with 32% of the world’s total COPDs. The respiratory diseases-related deaths are also increasing substantially. The top three contributors for these deaths are ambient air pollution, smoking, and household air pollution. Some research [2] had predicted rise in the deaths due to smoking, especially in the middle age adults. Respiratory diseases-related death will be around 6.5% in India [3].

An unknown genetic change happening in the basic building block of the human body, i.e., the cell, leads to a change in the normal life cycle of it. An uncontrollable growth of these cells forms a mass or a nodule commonly known as tumor. Such tumors are classified as benign or malignant; the latter is cancerous and can develop in any part of the human body. The medical field classifies the cancer based on the origin of such tumor. Carcinomas are the cancer prevalent in organs like lungs, breast, prostate, etc., and form solid masses, and can spread through metastasis. A diagnostician can conduct several tests to diagnose the condition of the patient with medical imaging is one of the non-invasive methods.

Of the 200 varieties of cancer, lung cancer affects millions of people around the world. A timely diagnosis and detection together with proper treatment can cure the lung cancer. But the detection of the cancer in the lungs could be accidental in many cases. A low cost non-invasive method of the diagnosis of the lung cancer as against the biopsy is the analysis of digital chest X-rays (CXR). The computed tomography (CT) is the expensive imaging alternative, seldom found in rural areas. The location, size, and other details of the nodules are visible on a CXR, and the diagnostician is trained in interpreting the images and classifying them. But during routine check-up, the nodules could be missed by a human observer, and there is interobserver variability in the interpretation, Qin et al. [4].

This gives rise to a requirement of a computer-aided detection of such nodules on the CXR. The computer can be trained to detect the nodules and classify the same. Early research work aimed at replacing the doctor with the computers, which could mostly be attributed to the early hype created by the artificial intelligence. But the later works have been focusing on making the computer generated information available to the doctor and assist him in his decision making. This could reduce the actual nodules being missed by the doctor. This use of computers in the diagnosis of the medical condition is known as computer-aided diagnosis (CAD).

There are multiple surveys on the performance and efficacy of CAD [5,6,7,8,9]. Few of the commercially available CAD systems are: iCAD SecondLook™ (for mammography), RapidScreen™ (for CXR), CADstream™ (by Confirma for breast MRI), ImageChecker™ (for mammography by R2 Technology), MeVis, Medicsight, Fujita, Zhang et al. [10].

The segmentation of the lungs from a CXR is still an unsolved problem. Hence, this paper presents a fusion method for segmenting the lungs from a given CXR in which the results given by active shape model (ASM) are refined with morphological operations to yield a better result, in a shorter time, overcoming the disadvantages of the ASM. In the further sections, a brief description of the dataset is given and the ASM is described. The next section is about morphological operations. Finally the segmentation results are presented in terms of standard performance indicators and then conclusions.

2 Material and Method

In computer-aided diagnostic systems for lungs, the computer analyzes the given CXR and returns its findings to the diagnostician. If there are any candidate nodules or lesions, the output of the CAD is further analyzed by the diagnostician. This reduces the burden on the human being and can enhance the productivity, thereby improving the patient care. An accurate determination of even the subtle nodules on the CXR at an early stage could be lifesaving.

A typical CAD system has the following sequence of operations for detecting the nodules on the lung images: Preprocessing, segmentation of the lungs, removal of the rib shadows, detection of the nodules, characterization and classification of the nodules as shown in Fig. 1.

Fig. 1
figure 1

A flowchart of the typical steps in a CAD system. Preprocessing is to prepare the image for the next stages and enhances the signal content in it

Full size image

Each stage of the above-mentioned crucial steps has a plethora of possible approaches [11, 12]. The choice and the implementation could be guided by the problem at hand. For the study purpose, the CXRs from the Japanese Society of Radiological Technology (JSRT) have been used [13]. The database has 247 images of which a number of nodule and non-nodule are 154 and 93, respectively.

Each image is of dimension 2048 × 2048 pixels, and the pixels are of 0.175 mm size. These images, which have also have patient specific information, have been used for training and testing. Table 1 summarizes the data distribution of JSRT database.

Table 1 Metadata of JSRT database
Full size table

There are a multitude algorithms and different methodologies to segment the lungs from the CXR. Few of them are mentioned here.

The interdependency of pulmonary abnormalities and lung region symmetry has been explored for screening the diseased population in PA radiographs by Santosh and Antani [14]. It is based on the hypothesis that asymmetry in lung region could indicate abnormality. An accuracy of 91% in abnormality detection has been reported. The method compares the left and right lungs by segmenting the two regions using graph cut algorithm as an atlas.

The saliency of the lung region has been used as the basis for its detection by Li et al. [15] on JSRT images. The work is based on pixel intensities and used a saliency map to classify the region of lungs. Active contour model is used to estimate edges and have reported 0.91 overlap score.

A deep learning-based method is used by Saidy and Lee [16] for lungs’ segmentation from the CXR suing the encoder–decoder convolutional network. The algorithm has been tested and results on five JSRT images have been reported.

Usage of convolutional neural networks to detect the nodules and an effort to reduce false positives has been shown by Li [17]. Apart from the shape model, the local gray-level distribution is also modeled in the statistical models in the work reported by Seghers [18]. The appearance model-based segmentation of the lungs is reported by Dieter et al. Another work reported is of Nagata et al. [19], which uses rule-based algorithm. The marking of the landmark points is dealt with considerable detail in the work of Starcevic et al. [20]. Yan et al. [21] also present a segmentation algorithm for the two-dimensional CXR.

The CXRs from the database pose various challenges to the segmentation algorithms, and hence in the following discussion, the same is highlighted. Each of the CXRs is having unique multimodal distribution of the gray-level intensities. Hence, a simple thresholding operation cannot produce a satisfactory result.

An example image is shown in Fig. 2a and its histogram in Fig. 2b. Figure 2c–h in the next rows shows the binary images obtained based on various threshold values, ‘T’. The operation is to classify the pixels as ‘1’ if the value of it in input image Im is above the T, else group it under ‘0’, shown mathematically as,

$$I_{o} \left( {i,j} \right) = \left\{ {\begin{array}{*{20}c} {0, I_{m} \left( {i,j} \right) < T} \\ {1, I_{m} \left( {i,j} \right) \ge T} \\ \end{array} } \right.$$
(1)
Fig. 2
figure 2

a A sample CXR from the database, b its histogram. c–h are the binary images obtained by thresholding the input image at various thresholds gray level ‘T’

Full size image

A satisfactory result is not given by any of those chosen values. This is because of the distribution of the gray-level values across the image, with great variability. The pixel value in the image depends on attenuation of the X-rays by various muscles internally and also depends on the position of the patient with respect to the imaging device. Hands and neck portion are visible in some images of the database, where as it not seen in the remaining. A generalized segmentation algorithm may fail to work in this case.

Variability in Shapes:

The 3D thorax is superimposed on a 2D image, and hence there is an overlap of the rib cage, lungs, heart and other organs like vertebrate, windpipes, etc. In spite of this, the lungs to be segmented are having a characteristic shape which is easily recognizable by a human observer. The diagnosticians are able to observe lungs alone by mentally subtracting the overlapping structures because of the training and experience. The same operation can be done by a computer but at some compromise in quality.

Hence, only one approach cannot solve the problem of segmentation and requires multiple operations either in series or parallel, to obtain a satisfactory result.

The ASM has been used for the segmentation of the lungs in CXRs and has given good results. But the limitation of these statistical models is its vulnerability to local intensity features. All the landmark points on the statistical model are annotated by an expert, and an average of all such shapes from each training image is the shape model. The shape model so built is initially placed on the target image. The model has to stretch itself to fit to an instance of the lung in it, if present. The movement of the shape, in or out, is guided by local gray-level profile. The gray-level profile is a one-dimensional sample of the image at a landmark point averaged from all the training images. This sample is searched along the normal to the boundary of the object, before settling at a best-fit location. At each of the iterations, the model adjusts itself to the available instance of the lung and an energy minimization function decides the number of further iterations. In case if the point deviates from the actual boundary, then it will never return to the correct edge of the object as there is no reset option in ASM as described in [22]. The results after first iteration of the ASM on the CXRs are depicted in Fig. 3. The image number has been retained as per the database.

Fig. 3
figure 3

Lungs boundary detection by ASM in one iteration, on CXRs

Full size image

3 Refinement of the ASM Results

The use of the ASM is restricted to only a coarse shape of the lungs. Further refinement of the results to obtain an accurate segmentation of the lungs which delineates its salient features like diaphragm, costophrenic (CP) angle, and the apex of the two lobes is done by the application of morphological operations. This is interspersed with rule-based approaches also.

This fusion method approach of fine tuning the segmentation outcomes has the following advantages: (i) The statistical model can be built with only few annotated points, and hence the complexities of computations are overcome. The lateral sides of the lungs are almost linear and hence few points are sufficient to describe them. The major curvature is at the CP angle, and a couple of landmark points are used in modeling it. (ii) The problem of the shape model being swayed away by a non-edge location is overcome. (iii) The method is faster. (iv) The operations are guided by the local gray-level distributions only. For processing a portion of the image, a suitable region is carved out and it makes the process independent of the gray-level distribution elsewhere in the image. (v) The two lungs are not modeled separately in the shape model, but are considered as a bell-shaped structure. This will remove the two lungs from the rest of the image in first stage and then separate them more easily. (vi) The accuracy of the method is higher. A flowchart of the proposed method is shown in Fig. 4.

Fig. 4
figure 4

Flowchart of the proposed fusion method

Full size image

The first iteration of the ASM gives an approximate location of the lungs on the target CXR. 30 landmark points have been used in building the ASM and these points stretch and modify according to the variability in the target shape as shown in Fig. 3. As indicated in the flowchart, a suitable rectangular region around these set of points is carved out for locating the diaphragm, shown in red color box in Fig. 5. Once the region is carved out, a simple thresholding operation suffices to segment the rectangular image content in to lungs and background, with diaphragm as the dividing line. Morphological operations like opening and closing help in removal of noise and obtaining an accurate detection of diaphragm.

Fig. 5
figure 5

A schematic representation of the prominent components

Full size image

A dynamic program then has to divide the original image in to smaller portions and then process them individually to get the side edges of the lungs. This is shown with the blue color boxes. The first box closer to the diaphragm represents the region which will be processed for detecting the edge of the lungs. As can be observed, the area that is carved out is a small region and contains a portion of the lungs which is distinct from the background. Hence, here too, a thresholding operation based on Otsu’s method is sufficient in highlighting the edges. The number of such vertical boxes for detecting the side edges of the lungs depends on the height of the lung, which in turn is directly obtained from the first iteration of the ASM. The first iteration gives an approximate coverage of the lungs in the target CXR (Fig. 3). Once the lateral edges are determined, a suitable rectangular region is dynamically identified by the program, which would possibly house the lobes of the lungs, depicted by yellow color box in Fig. 5. The upper lobes are not simple lines unlike diaphragm or lateral edges, but very complex structure as it contains prominent impressions of cervical bones, shadows of the collar bones and wind pipes. The cluttered nature of this region makes it imperative to use a mix of thresholding, morphological operations and a dynamic program to accurately decide the edges of the upper lobes.

It should be observed that till this step the lungs outline resembles an inverted convex hull, and has been determined. The right and left lungs are still fused with each other and have the heart silhouette in between. The gray-level distribution of the lungs is different from that of the heart, and hence can be separated by binarizing the image. Through this entire set of steps, the whole CXR is divided in smaller segments and processed separately, to look for a specific feature, i.e., edge.

The proposed fusion method has been tested on a select 55 CXRs from the same database, and these were not part of the set of training images. One of the authors has segmented the right and left lung regions manually for each of the test images, under the guidance of an expert. This has been considered as the gold standard for comparison of the automated segmentation [23]. The quantification of the results has been done based on the standard evaluation metrics: Sensitivity, Specificity and Accuracy. These are defined in Eqs. 2–4, in terms of four basic outcomes: True Positive, True Negative, False Negative, False Positive.

These cardinals are defined as follows: True Positive is the set of pixels rightly classified as belonging to the lungs and True Negative are the set of pixels rightly classified as not belonging to the lungs. On the same lines, False Positive is the total number of pixels wrongly classified as belonging to the lungs and False Negative is the total number of pixels wrongly marked as non-lung pixels (Fig. 6).

$${\text{Sensitivity}} = { }\frac{{{\text{TP}}}}{{{\text{TP}} + {\text{FN}}}}$$
(2)
$${\text{Specificity}} = \frac{{{\text{TN}}}}{{{\text{TN }} + {\text{FP}}}}$$
(3)
$${\text{Accuracy}} = { }\frac{{{\text{TP}} + {\text{TN }}}}{{{\text{TP}} + {\text{TN}} + {\text{FP}} + {\text{FN}}}}$$
(4)
Fig. 6
figure 6

The results of application of the fusion method on select CXRs. a The input images from the database. b The corresponding outermost edges detected by single iteration of ASM. c Separation of the right and left lungs using the proposed. d Color coded overlap region between the machine segmentation and the gold standard segmentation

Full size image

These parameters and the corresponding values have been tabulated in Table 2. The proposed algorithm was tested on three image sizes as depicted. These images were obtained by down-sampling the original image. Correspondingly the time requirement is also less for the segmentation for lower size image.

Table 2 Performance of the proposed method on nodule CXRs of three different sizes
Full size table

Yellow color pixels indicate the TPs, and blue color pixels are part of the lungs but omitted by the machine classifier during segmentation as not belonging to the lungs. Hence, these are the FN pixels. On the same lines, the red color pixels are the background pixels but incorrectly classified as lung pixels, and hence FPs. The white pixels in the same image are the background pixels and classified accordingly. These are the TNs. Random colors have been chosen for an effective representation and expected to give a good contrast on print or electronic medium.

All the above-mentioned performance indices have been shown in Table 3 for 2048 × 2048 images.

Table 3 Gist of the performance indices on the CXRs from JSRT database
Full size table

4 Conclusion

The problem of segmentation of the lungs can be approached in multiple ways, but each approach has its own limitations, and a hybrid approach is required to refine the results, to be of any clinical importance. All the objects of interest in a medical image are bound by a characteristic shape, but vary within a permissible variability. Modeling the shape and including in it the variability is a challenging task, as the model depends on the training dataset. In the proposed method, the model is dependent on the training dataset, but not its accuracy as it is a hybrid method. The simple framework for detection of shapes as explained in this paper can be extended to any shape on medical or other images.