Introduction

Ultrasonography (US) is a common and widespread modality in the screening and diagnosis of many diseases because of its noninvasiveness and convenience [1,2,3]. However, for operators, extensive experience of US examination should be required for the accurate diagnosis of the lesions because of the necessity of real-time recognition of abnormal lesions during examinations [4, 5].

A lack of specialists for US imagining and diagnosis is an urgent medical matter that needs to be solved. Recently, in addition to conventional machine learning, sophisticated deep learning technology is becoming available in this field. From this point of view, development of computer-aided detection and diagnosis will be an attractive solution for the US diagnosis, especially in underpopulated areas with insufficient medical services. Based on these backgrounds, several reports that applied machine learning technology for the diagnosis of diseases have been published [6,7,8]. However, because of the specific nature of US diagnosis, there are still several issues that need to be resolved for the application of computer-aided diagnosis (CAD) in this field [9]. In this review, we evaluate the current performance of CAD for liver lesions based on the previous publications, and discuss the advantages and problems to be solved for future establishment of deep learning-based CAD system as a screening tool for US examination of liver disease.

Advance in computer-aided diagnosis for medical imaging

Recently, several studies reported the impact of CAD using deep learning algorithm for medical imaging. Gulshan et al. [10] reported the performance of deep convolutional neural networks (CNNs) for automated detection of diabetic retinopathy and diabetic macular edema using a dataset of 128,175 photo images of retinal fundus. The performance of algorithm showed an area under the receiver operating curve (AUC) of 0.991, indicating that the use of this algorithm could show a positive impact on improved care and outcomes. Another study reported the diagnostic assessment of deep learning algorithms for detecting lymph node metastases of breast cancer through the training of whole slide image with and without nodal metastasis [11]. Reportedly, the performance was comparable with that of an expert pathologist without time constraints. In the field of dermatology, deep learning using a dataset of 129,450 clinical images of skin lesions by CNNs achieved performance on par with expert dermatologists for classifying skin cancers [12]. On the other hand, creating large dataset of medical image sometimes requires many skilled people and much time. From this point of view, efficacy of transfer learning algorithm was also reported for identifying the common treatable blinding retinal disease and pediatric pneumonia using smaller dataset of optical coherence tomography and chest X-ray images, respectively [13].

Current status of computer-aided US diagnosis for liver lesions

Compared to the image diagnosis using dataset of other photo images, US images are more heterogeneous for the CAD in terms of their image parameters, setting of equipment, and condition of examination [14,15,16,17,18]. To date, several studies have shown the effectiveness of machine learning for the diagnosis of focal liver lesions by a B-mode ultrasound image (Table 1). For example, Hwang et al. [19] tried to extract textural features of focal lesions such as cysts, hemangiomas, and malignant lesions for the diagnosis of a liver tumor; they demonstrated high diagnostic accuracy among all focal liver lesion groups. Contrast-enhanced US (CEUS) was also applied for more accurate diagnosis of liver tumors [20, 21]. Kondo et al. [22] reported the automatic classification of benign tumors, hepatocellular carcinoma, and metastatic tumors using machine learning of the CEUS image. They used a contrast agent, Sonazoid, and found that sensitivity, specificity, and accuracy that distinguish malignant lesions from benign were 94.0%, 87.1%, and 91.8%, respectively. Another report applied a multiple kernel learning-based framework, achieving accuracy of more than 90% for the different diagnosis of benign and malignant liver tumors using CEUS images [23]. Although, some studies applied neural network for CAD of focal liver lesions, their accuracies are similar to those by support vector machine, probably because of the limited size of training cohort. However, it is expected that development of deep neural network-based CAD using larger dataset should help to achieve more accurate and refined US diagnosis of focal liver diseases, such as that reported in CAD for retinopathy and skin lesions [10, 12].

Table 1 Performance for detection and diagnosis of focal lesions based on the machine learning using liver image of ultrasonography
Full size table

Further, application of CAD for diffuse liver lesions has also been reported (Table 2). It is also known that machine learning is useful for the discrimination of fatty liver disease (FLD) from normal liver [24,25,26]. These studies constructed the computer-aided diagnostic software for detecting FLD using a B-mode US image. More importantly, Acharya et al. [27] reported an algorithm to discriminate automatically the normal, FLD, and cirrhosis US image using a neural network classifier with an accuracy of 97.33%, specificity of 100% and sensitivity of 96%. Because a subset of FLD might progress to liver cirrhosis, such discrimination is informative in the clinical setting.

Table 2 Performance for detection and diagnosis of diffuse lesions based on the machine learning using liver image of ultrasonography
Full size table

For the diagnosis of liver cirrhosis, some studies applied liver stiffness by transient elastography for the algorithm of artificial neural networks and showed high diagnostic performance for the diagnosis of cirrhosis, portal hypertension, and esophageal varices [28, 29]. Other studies focused on the parameters of the B-mode image and the Doppler method, such as liver parenchyma, thickness of spleen, hepatic vein waveform and damping index, and hepatic artery pulsatile index, for the grading of liver fibrosis [30]. However, the number of images from previous machine learning trials is still small (< 1000 images); a larger US image database with reliable supervised data should be constructed for the development of a solid CAD algorithm. On the other hand, some reports also showed the efficacy of deep leaning algorithm for diffuse liver disease, such as detecting FLD using B-mode images [31], and staging of liver fibrosis using the share wave elastography images [32]. Interestingly, accuracy for detecting FLD is 100% by deep leaning algorithm; higher than that by conventional machine learning [31]. Deep learning-based CAD also showed detailed classification of liver fibrosis [32]. Using the dataset of US images from dog livers, Banzato et al. [33] developed a deep learning-based US image diagnosis for detecting degenerative hepatic disease. They also used transfer learning, and reported the high sensitivity and specificity for diagnostic accuracy compared to those by cytology and serum biochemical markers. Therefore, it is conceivable that deep learning using larger dataset should be a powerful tool for detecting diffuse liver lesions.

Future perspectives for the development of computer-aided US diagnosis applicable for clinical practice

US is a common and noninvasive imaging modality and, therefore, applicable for the diagnosis of many diseases in a wide range of organs including digestive, reproductive, urogenital, and endocrine organs, and the circulatory system. It is also a useful medical device for screening diseases because of its handiness. In addition, the development of new technology in the field of US diagnosis, such as US elastography, Doppler US, and CEUS, allows us to apply US imaging for detailed examination of abnormal lesions. Therefore, a large number of images with solid supervised data may be collectable for constructing a database essential for deep learning.

Firstly, for the development of computer-aided or artificial intelligence (AI)-aided US diagnosis, deep learning for disease screening, which can detect the pathological findings and distinguish them from the physiological image, should be required. Secondly, a function for the differential diagnosis of the lesion is also needed (Fig. 1). Because there is a complex normal architecture consisting of vessels and the bile duct in the liver with anatomical anomalies, the pattern of still images in each segment of the liver that are restricted by the position and direction of the probe should be unified among operators to distinguish pathological abnormalities from normal structures [34]. In addition, for the development of deep leaning algorithm for different diagnoses of hepatic lesions, supervised image data, where specific lesions and definitive diagnoses are linked, should be essential. After the learning of still image data and the diagnosis process of the disease, a database of movie images taken in the unified order throughout the entire part of the liver needs to be constructed. Video images of Doppler US and CEUS are also optical materials in the development of a robust AI algorithm for diagnosis.

Fig. 1
figure 1

Workflow for the development of deep learning-based diagnosis system in ultrasonography. The application of deep learning technology for ultrasonography (US) image diagnosis consists of two steps. Firstly, deep learning using supervised images data of pathological lesions is required. At this point, recognition of the lesions is performed by the operators. This step aims at achieving accurate diagnosis through deep learning; both focal (such as liver cancer and benign tumors) and diffuse liver lesions (such as fatty liver and liver cirrhosis) are required using still image (including B-mode and elastography-based images), and movie image (B-mode, contract-enhanced US, and color Doppler images). Secondly, for the development of deep learning-based screening system, a database that includes normal liver images and artifact images needs to be constructed. In this step, computer (or artificial intelligence: AI) is required to detect pathological lesions. The segments and detailed portion, where the lesions exist, also need to be determined. For this purpose, position of normal structure consists of many vessels and ducts should be recognized by AI; still and movie images database should be analyzed. Fusing these technologies will facilitate the next-generation AI-equipped US

Full size image

Current issues in the development of computer-aided diagnosis on ultrasonography in liver disease

Because of the unique characteristics of the US image, there could be some issues that need to be resolved for the development of CAD system. For example, compared to the other images, the quality of the US image is, currently, highly operator dependent in terms of detection and specification of the lesions. In addition, differences in the software and design of imaging systems among the machines, models, and vendors could also affect the image display. The unique function of US, such as image parameters including gain, sensitivity time control, and mechanical index, could also lead to the difference in image quality. Therefore, the examination condition should directly affect the differences in US images.

On the other hand, a recent report showed the efficacy of deep learning and transfer learning for detecting liver lesion, regardless of the difference of their US image parameters [33]. However, although some studies applied deep neural network algorithm for developing CAD system of US images, the cohort size is still too small to achieve an enough performance of deep learning (Tables 1, 2). Therefore, constructing a big database for deep learning-based CAD system should be essential, which could overcome the heterogeneity of imaging for computer-aided US image diagnosis. In addition, US images may be preferably linked to image parameters unique to each US examination. Further analyses should be conducted to clarify these matters.

In addition, as mentioned, complex normal liver architectures have also made the application of AI-based technology difficult for US diagnosis; the procedure of US examination needs to be unified among the operators for the construction of a well-qualified database of US images. From this point of view, several projects are currently ongoing that overcome the heterogeneity of US images and help to develop the operator-friendly system of computer-aided US diagnosis in the field of liver diseases. As AI-aided US diagnosis should have the potential for the management of liver disease, such as detecting early response to cancer treatment, which is difficult for conventional methods [35,36,37], its development is rapidly expanding in many fields, including the diagnosis of liver diseases [38,39,40].