Keywords

1 Introduction

Coronavirus disease 2019 (COVID-19), which was first reported in December 2019, has rapidly spread to more than 200 countries, becoming a serious threat to human lives. But now it goes on with some kinds of variation in 2021, and has already developed into a global health crisis. By May 13, 2021, more than 161 million cases of COVID-19 have been confirmed, including 3,300,000 deaths. The common symptoms of COVID-19 include fever, dry cough, and some respiratory problems. The control of COVID-19 largely depends on timely diagnosis. It is generally known that the reverse-transcription polymerase chain reaction (RT-PCR) test is the standard method for screening suspected cases [1]. However, the lab testing has some drawbacks. Firstly, the shortage of medical equipment and high demand of testing environments will limit the rapid screening of suspected cases. Secondly, the RT-PCR test is time-consuming because of laboratory processes, usually 24–48 h. Thirdly, comparing to RT-PCR test, chest X-ray and chest CT scan have a high sensitivity for COVID-19 infection [2], and in practice, their imaging equipment is more easily accessible and operated. Therefore, the listed advantages make medical imaging methods a necessary complement for early screening stage, providing a huge help to clinicians. The latest China’s diagnosis and treatment protocol for COVID-19 (trial version 8) also highlights the value of imaging for detecting COVID-19. For example, in Wuhan, if the image features of X-ray or CT scan are observed [3], many suspected cases can be classified as suspected cases of COVID-19 as soon as possible, thereby receiving timely treatment. The suspected patients, though without clinical symptoms like coughing, fever, dyspnea and muscle aches, also need to be quarantined for further lab tests.

However, there is the phenomenon of cross-infection in the actual chest X-ray or CT scans. This is likely to cause COVID-9 to infect doctors who are not sick or patients with other diseases. Not only is there no way to save the patient, but there is also the risk of sacrifice. Meanwhile, due to the increment of confirmed and suspected cases of COVID-19, it is a labor-intensive task for radiologists to manually contour lung lesions and analyze a large number of scanning reports timely and accurately, which will lead to a delay in the diagnosis of COVID-19, miss the best time for treatment. To speed up diagnosis, it is important to develop fast-automatic segmentation and diagnosis method for COVID-19 combined with deep learning and AI technique.

Fig. 1.
figure 1

[4] Chest CT images of laboratory-confirmed COVID-19 patient and one healthy subject. GGOS is highlighted with a red border in patients and compared with a green border in normal subjects. (a), (b) and (c) are lung CT images of COVID-19 patients in the early, progressive and absorption stages, respectively; (e), (f) and (g) are local magnification images of GGO in those three stages, respectively. (d) is a CT lung scan of a normal person, and (h) is the local magnification of a normal person. (Color figure online)

Full size image

Chest CT features of COVID-19 patients usually include ground glass opacity (GGO), consolidation, and other rare ones such as pleural or pericardial effusion [5], where GGO is a universal feature of all the chest CT findings. A study [6] points out identifying these common features is important for better identification of COVID-19.

However, CT imaging features of COVID-19 patients varied in different stages of the disease. CT images of COVID-19 patients with different disease stages and normal subjects are shown in Fig. 1. In the early stage, both lungs often exhibit patchy or diffuse GGO, with thickened small blood vessels in the lesions. Besides, there is nodular and patchy high-density opacity under the pleura of bilateral inferior lobes. In the progressive stage, the two lung diseases change quickly, multiple lesions fuse into huge sheets of consolidation, with the lesion density raising. In the absorption phase, the lesion area is lightly reduced and the density is slowly decreased [35]. However, due to the facts of age, autoimmunity of patients, the analysis and classification of COVID-19 medical image will also be influenced on a certain extent.

Fig. 2.
figure 2

[34] (a) (b) is the CXR image of normal people, (c) (d) is the CXR image of COVID-19 patients [10]. Images (c) (d) represent blurred lungs with a shadow of diffuse consolidation compared with those of normal people.

Full size image

For the chest X-ray images of COVID-19, the representative characteristic is large blurred lungs, shown in Fig. 2, which probably be accompanied by thickening of cracks at night and a small amount of pleural effusion. When the patient is badly ill, there is a shadow of diffuse consolidation in both lungs, white lungs may appear, at times with pleural effusion.

Through reading of the COVID-19 image diagnosis literatures, people discover that major COVID-19 image diagnosis be based on chest CT images, with fewer references to CXR. This paper starts from the progress of medical image segmentation technique, including segmentation methods and their applications, then followed by a discussion of COVID-19 diagnosis methods by chest CT and X-ray imaging combined with machine learning and artificial intelligence techniques. The last part is the summary and looking forward to the future of medical imaging diagnosis. It is hoped that this review will provide some help and guidance for doctors and researchers to carry out COVID-19 treatment. This review is primarily based on medical imaging studies related to COVID-19 prior to March 31, 2021.

2 Medical Image Segmentation of COVID-19 and Its Applications

Segmentation refers to the division of an image into different regions according to the adjacent similar features. Image segmentation is complex and challenging in biomedical engineering tasks, which is influenced by many aspects, including noise, low contrast, illumination, and object irregularities. The goal of segmentation is to separate the area or object of interest from the rest of the body for quantitative measurement. Now segmentation methods have been divided into three categories: manual, semi-automatic and automatic. It takes time and effort for manual segmentation. Semi-automated segmentation is prevalent, which is normally integrated with open software. The automatic segmentation method does not require user intervention. The three methods have their strengths and weaknesses. Even now, the segmentation problem remains challenging. The paper [10] points out that although a large number of AI systems have been used to assist diagnosis in clinical practice, there is yet little work on the segmentation of infection in chest CT and X-ray scans.

Table 1. COVID-19 applies age segmentation classification methods and some examples specifically discussed in this section, along with the results obtained from each method.
Full size table

In this section, Table 1 lists the latest existing medical image segmentation methods and summarizes some examples of COVID-19’s application of image segmentation classification methods in this section, as well as the results obtained from each method.

2.1 Performance Index of Medical Image Segmentation

In general, the method of image segmentation based on CT scanning technique detects its own performance through the statistical average value of performance indicators such as sensitivity, accuracy, robustness and special effect. Sensitivity represents the proportion of pairs among all positive examples, and it has the ability of measuring the algorithm model. The degree of special effect represents the proportion of pairs among all negative examples, which measures the capacity of the algorithm model to identify negative examples. Accuracy represents the proportion of positive examples in the examples divided into positive examples, which is the most common evaluation index. If the accuracy is higher, the performance of the method will be better. However, it is so far from comprehensive and scientific to estimate an algorithm model solely by the accuracy. Robustness has the ability to deal with missing values and outliers. The function of the algorithm model can be measured by these index values, which can aid doctors and researchers in selecting appropriate methods for CT image segmentation.

2.2 Segmentation Methods

In all the references, there are many lung segmentation techniques for different purposes. In paper [7], a dual branch combined network (DCN) for COVID-19 diagnosis is proposed. Individual level classification and lesion segmentation are achieved by DCN. Figure 3 reveals the general framework of the method. The whole method consists of three parts: 1) Lung segmentation based on U-NET; 2) Slice-level combinatorial segmentation and classification using DCN; 3) Individual level classification with three layers of fully connected network. This method realizes the simultaneous classification and segmentation of COVID-19, but it may be hard to distinguish from other organs because of the serious lesions in some patients. In consideration of the low accuracy of the experimental results, especially for the CT images of COVID-19 patients. The paper [12] proposes to segment the deep network with INF-NET infection, which applying a parallel partial decoder to aggregate advanced features and create a global map. Explicit edge attention and implicit reverse attention are used to model the boundary and enhance the representation. A semi-supervised segmentation framework based on random selection propagation strategy is also used to alleviate the deficiency of labeled data. The semi-supervised segmentation framework enhances ability and attains better performance. Compared with unsupervised detection and segmentation, semi-supervised model can better identify the target region, which is suitable for COVID-19 detection. The paper [9] presents a new deep neural network for automatic segmentation of the COVID-19-infected area and the entire lung from chest CT images, extending the compression and excitation units [14] to deal with the weakened or blurred boundaries. The shape changes of the COVID-19 infected zone are handled by a Progressive Atru Space Pyramid Pool (PASPP). In the paper [10], researchers develop a novel lightweight convolutional neural network based on deep learning, known as ANAM-NET. Design for abnormal COVID-19 segmentation in chest CT images, introduce a label-based weighting strategy of network cost function. Figure 4 demonstrates the key steps involved in segmenting COVID-19 anomalies. ANAM-NET is fully automated, ensuring full turnaround time for segmentation. In the paper [11], an anti-noise framework is proposed to learn segmentation tasks from noise labels. The COVID-19 pneumonia lesion segmentation network is used to treat lesions with different scales or appearance. But collecting perfectly clean labels for medical image segmentation is a challenge. The paper [16] constructs a large-scale COVID-19 classification and segmentation dataset, COVID-CS, developed a joint classification and segmentation system for COVID-19 diagnosis. Using this system, each suspected COVID-19 patient is diagnosed in 22 s by classification model and in 1 s for non-infected cases. Although the mean time is not as good as that in paper [10], the system data set can achieve up to around 93% specificity and 95% sensitivity, providing a convincing visual explanation. Medical image segmentation technology is supposed to be from multiple angles to improve performance and sensitivity. When patients run into a wide range of lesions, the shape of lesions is unsure, and the scanning boundary is fuzzy, it is evident that the multiple COVID-19 segmentation techniques U-NET-based has lots of drawbacks, and the sensitivity, stability and time performance will be greatly reduced.

Fig. 3.
figure 3

The frame composition of the lung segmentation method using CNN.

Full size image
Fig. 4.
figure 4

Key steps of the proposed approach for automated segmentation of abnormalities in chest CT images.

Full size image

2.3 The Application of COVID-19 Segmentation

Although the paper [8] achieves ideal results with INF-NET for segmentation of infected areas, there are a number of limitations in the current model. In clinical practice, it is frequently essential to classify COVID-19 patients and then segment the infected area to facilitate treatment. INF-NET mainly focuses segmentation of pulmonary infection. However, it is also good at monitoring longitudinal disease changes and performing large-scale screening treatments, and there is great potential for evaluating the diagnosis of COVID-19 by quantifying the area of infection. The paper [10] proposes ANAM-NET segmentation of abnormal COVID-19 images. Compared with other models, it has a very low computational complexity, therefore can be widely deployed in clinical Settings, enabling rapid evaluation of abnormal COVID-19 images [13]. A dual-channel CNN pipeline is implemented to automatically segment COVID-19 pulmonary infection tissue from CT images by making up three unequal input images. Compared with end-to-end learning, supervised and unsupervised methods, segmentation is more flexible and efficient, which is fully capable of detecting abnormal areas with low intensity contrast between diseased and healthy tissue. In the case of infection with fuzzy boundary or wide range of lesions, the utilization of dual CNN pipeline will have more benefits, which can reasonably simplify the procedure and decreases the time cost. The focus attention module LA is based on the development of a dual-branch composite network DCN [7]. LA module can amass the network on the infected sites, which can be applied in the early screening of COVID-19.

In short, the image segmentation is playing a very important role in automatically screening and detection of lesion area of COVID-19 CT or X-ray images, and is helping radiologists to make a quick judgement for the next step of the diagnosis and treatment in a short time.

3 Medical Image Diagnosis of COVID-19 Based on Machine Learning

At present, due to imaging technology’s advantages of low cost and high sensitivity, X-rays and chest CT images have been widely used in the diagnosis of pulmonary infection by radiologists. However, medical images, especially chest CT scans, contain hundreds of sections, which can be difficult for radiologists to handle. It’s better to use a machine than to waste human resources on it. Therefore, computer vision method based on deep learning has broad application prospects in medical images, and it is very necessary to use artificial intelligence to assist diagnosis. Machine learning is a method in which a computer uses existing data to train an algorithmic model, and then uses the model to predict the future. Usually, machine learning will use regression algorithm, neural network, support vector machine, clustering and other methods to build models. Although deep learning is a kind of machine learning, it complicates the model by using deep learning networks. Deep learning networks are mechanisms that mimic the human brain for interpreting images, sounds, text and data. Deep learning has driven the rapid development of artificial intelligence. Deep learning not only gives machine learning more practical applications, but also extends the overall scope of artificial intelligence, such as medical imaging.

3.1 Chest X-ray Diagnosis of COVID-19 with Deep Learning

Controlling COVID-19 largely depends on correct diagnosis. AI-based X-ray equipment is readily available around the world, and results are obtained soon. Although it is less sensitive than chest 3D CT imaging, it can be used for the early detection of COVID-19, minimize the amount of treatment time patients waste at the diagnosis stage. Applying artificial intelligence technology, chest X-ray images can be analyzed [16].

Because of the limited computing power and lack of available data for identifying cases of infection through lung X-ray images, it is likely to construct a custom convolutional neural network (CNN) from scratch using the large number of historical lung X-ray images provided by ChExpert, and retrain a deep learning CNN model to examine whether lungs are healthy, adjust the final L-layer using COVID-19 X-ray images. One of the biggest superiorities of CNN model is that data are immediately used to realize the model in the process of training [17].

The paper [18] proposes to use the RESNET-50 deep learning model to extract features from chest X-ray images. All the operations are displayed in the flow chart in Fig. 5. Three different virus types of COVID-19, normal pneumonia and viral pneumonia are studied using SVM algorithm. The experimental results prove that the deep learning-based model can directly extract characteristic and generate results even with limited COVID-19 data.

Fig. 5.
figure 5

Flow diagram

Full size image

In order to study the pattern of chest images of COVID-19 patients, it is a good choice to take CNN as a deep learning algorithm. CNN is a development of the traditional Neural Network proposed in 1968 [19]. In addition, the study uses a pre-trained model as a dataset [20], which included 100 COVID-19 chest radiographs and 100 normal chest radiographs. The COVID-19 chest radiograph dataset can be acquired through GitHub [21] and the normal chest radiograph data set also can be gained through Kaggle [22]. Contrast-Restricted Histogram Equalization (CLAHE) is used to improve image quality. Target recognition and image enhancement in medical images are realized by combining CNN and CLAHE, especially for COVID-19 virus detection in chest radiograph images. The purpose of the study is to compare and measure the accuracy of techniques used to improve image quality. Researchers carry out the experiment with the aid of HE, GC and CLAHE. The outcomes indicate that GC has prime sensitivity and CLAHE has prime accuracy. The key factors of Clahe are the clip limit (CL) and the number of textures (NT) [23]. The experimental data demonstrates that the accuracy rate of the rule set is 99% the verification rate is 97%, which signifies that the basic model test method of Clahe is improved by applying CNN. The implementation of CLAHE assists to boost the quality of the classified image and enhance the contrast of the image. On the side, the new CCSHnet [24] for COVID-19 diagnosis in CCTs uses a group of computer-processed X-ray images taken from disparate angles to generate a cross-sectional image of the scanned area. This way can offer three-dimensional volume data, highlighting additional spatial features and anomalies, can also is used to generate high-quality, detailed images.

The paper [25] derives wavelets from varying frequency and finite duration. An automatic diagnosis method of COVID-19 in chest X-ray images is proposed which based on wavelet theory. This way integrates multi-resolution analysis into the disease recognition network. The dataset consists of chest X-ray images of healthy individuals, COVID-19 and patients with viral pneumonia. In this dataset, there are 1439 images from these three categories, 678 normal cases, 132 COVID-19 cases and 629 viral pneumonia cases. The results will be compared with four latest technologies (DarkCovidNet, Flat-EfficientNet B3, Hierarchy-EfficientNet B3, Detate-Resnet18). The performance of the four different technologies is displayed in Table 2. These techniques use deep learning models to diagnose COVID-19 from chest X-ray images. DarkCovidNet uses YOLO network to detect COVID-19 in chest radiograph images [26] with overall accuracy of approximately 87%. The second and third methods are both based on efficiency networks [27], and plane method and hierarchical method are proposed. The combined accuracy of the two methods is around 93%. This makes known that the accuracy has been increased. In the future, it is envisaged to use the improved deep convolutional network to automatically detect chest radiography images.

Table 2. Performance comparison of four different deep learning net [28]
Full size table

On the whole, most studies use X-ray images to classify COVID-19 patients and non-COVID-19 patients. Aim at obtaining more accurate and reliable data, the researchers propose X-rays based on convolution and deep learning strategies to detect COVID-19 virus patients. As a result of the restricted image data of COVID-19, it is hard to gain a large amount of data and conduct more experimental tests, so its clinical suitability still has doubts. In the next, people will put emphasis on early detection of COVID-19 virus diagnosis [40].

3.2 Chest CT Diagnosis of COVID-19 with Deep Learning

Convolutional neural network is an extraordinary deep neural network model, generally divided into one dimensional, two dimensional, three dimensional. As the simplest, One-dimensional convolution is a linear space. In essence, it is the convolution of a word vector. One-dimensional convolutional neural network is usually used in low-dimensional space such as natural language processing and sequence model. Two-dimensional convolution is a spread of one-dimensional, which is often used in computer vision and image processing. The specific idea of three-dimensional convolution is the same as the previous two. However, three-dimensional convolution puts time dimension into the input of neural network which makes neural network extract temporal and spatial features and identifying behaviors. The training method of 3D convolutional neural network is similar to convolutional neural network. Therefore, 3D convolutional neural network is the most advanced, so it is frequently used in the medical field, such as CT image and video processing.

It is difficult for humans to distinguish COVID-19, because its appearance on CT images likes other types of viral pneumonia. Therefore, there are a lot of studies about how to distinguish COVID-19 patients from non-COVID-19 patients. This includes a new PSSPNN model [29] that can be used to classify COVID-19, secondary tuberculosis, community-captured pneumonia and healthy subjects, but the method still needs to be improved. In addition, Li Xu et.al have attempted to develop a CAD system for screening COVID-19 CT images. The paper [30] presents a 2DCNN, which takes a series of CT slices as input and 2D-RESNET50 as backbone to extract CNN features from each slice of the CT series. They then use the maximum pooling operation to combine these features, and the resulting map is provided to a fully connected layer, which can generate probability scores for each class. This paper [31] firstly uses a three-dimensional segmentation model, namely VNET [32], to segment lesion candidates from CT images. Then, the 2D-Resnet18 model is used to classify the COVID-19 or Influenza A viral pneumonia.

Besides direct segmentation and 3D networking, the authors also propose a new multi-task primitive attention learning strategy to realize COVID-19 screening in volume chest CT images. To be specific, by devising an up-front focus on residual learning (PARL) block, the aim is integrating two RESNET-based branches into an end-to-end training model framework. Within these data blocks, hierarchical attention information from the lesion area detection branch is transferred to the COVID-19 classification branch to learn a more discriminating representation. This approach is getting more accurate in locating the lesion area and allowing additional surveillance information to heighten the performance of the COVID-19 classification task. Experimental results manifest that this method exceeds other state-of-the-art COVID-19 screening methods [39].

With the rapid growth of COVID-19, chest CT images have become the main basis for radiologists to collect information about patients because of their high sensitivity and low cost. However, due to the high infectious nature of the epidemic, medical personnel face great risks when collecting COVID-19 CT data. Therefore, even though the image data is relatively common, it cannot meet the large amount of data required for the training of the deep learning model. To meet the data requirements of COVID-19 CT imaging, researchers propose a CT image synthesis method based on conditional generation antagonistic network, which can effectively generate high quality and realistic COVID-19 CT images for deep learning-based medical imaging tasks. Results reveal that this way is beneficial to the synthesis effort of COVID-19 CT images. For good measure, a new AVNC model [33] has emerged which unites attention mechanisms and improves multiplexing data enhancement to extend data sets.

Since COVID-19 is highly infectious [10], for the sake of accelerating the process of COVID-19 CT data acquisition in deep learning-based CT imaging and preventing possible infection of healthcare workers when they contact COVID-19 patients, researchers put forward novel CGAN structure that contains a global-local generator and a multi-resolution discriminator. Both the discriminator and the generator adopt dual network to plan to emulate the local and global information of CT images respectively. Simultaneously, this dual structure has a communication mechanism of information interchange, which is helpful to generate lively CT images with stable global structure and various local details [36].

4 Summary and Future Work

Medical imaging methods are widely employed in medical diagnosis, especially chest X-ray and chest CT. X-ray is the most extensively used diagnostic X-ray examination in medical practice, which is of great significance for early clinical research and diagnosis. The quantitative analysis of medical images demands the segmentation and diagnosis of the boundary of the object of interest. Traditionally, doctors complete medical image segmentation. But when the number of divisions is massive, the need for storage is very urgent. Given the shape and difficulty of various organs, it is hard for doctors to segment the right areas accurately and there is nothing they can do about the large number of images that need to be segmented. Therefore, advanced automated segmentation technology is especially needed to solve this problem.

This paper reviews COVID-19 image segmentation and diagnosis techniques, introduces medical image segmentation methods and applications in detail, as well as COVID-19 diagnosis based on CT scan and X-ray scan. It illustrates the significant role of rapid automatic segmentation and AI diagnosis technology in COVID-19 prevention and control management.

With the raising demand for COVID-19 diagnosis, medical image segmentation technology complemented by dual CNN channels, deep neural network (CoVIDSEGNET) or DCN model has greatly improved the accuracy and sensitivity of segmentation technology [38]. The intelligent diagnosis has the capacity of automatic screening and timely analysis, avoiding the possibility of misdiagnosis, and improving radiologists’ working efficiency. It is important to note that the current study still has many limitations. The CT or X-ray images of COVID-19 patients have blurred boundaries, large lesion area, or consolidation, which makes it difficult to segment medical images accurately. It is expected that in the future, the image scanning will be clearer and the resolution will be higher, enabling medical image segmentation technology to integrate AI technology into the process of image acquisition and synthesis. It will also attempt to apply AI to the overall segmentation and diagnosis of COVID-19, realize automatic screening, analysis and report generation to make quick judgments by using artificial intelligence [37]. In particular, images of small lesions such as COVID-19 can be flagged and drawn to conclusions. In the future, AI image acquisition and synthesis workflow will enable patients to obtain high-quality images even at low radiation levels.