Abstract
Purpose of the Review
This review discusses the recent advances in automated echocardiography using artificial intelligence and machine learning (ML) techniques. Specific emphasis is placed on the potential for machine learning-based methods to improve accuracy and reproducibility of echocardiographic assessment as well as early cardiovascular disease detection and personalized risk assessment.
Recent Findings
Echocardiography remains the first line imaging modality for evaluation of many cardiovascular diseases. The last few years have witnessed a rapid expansion and growth of ML-based automated analysis and interpretation of echocardiography. These ML algorithms have shown great promise for improving data reliability, accuracy, and reproducibility of echocardiographic results. We anticipate that the application of ML algorithms will further expand the indications of echocardiography to include diseases that are traditionally only diagnosed with the more advanced imaging modalities such as cardiac magnetic resonance imaging. The ability to leverage ML’s robust capability for processing large and complex datasets will result in improved diagnosis of cardiovascular disease at subclinical stages, enable prediction of disease progression and prognosis, and facilitate the characterization of disease phenotypes to allow more targeted therapies.
Summary
The paradigm is rapidly shifting in the field of echocardiography with the emergence of ML algorithms that are promising to improve data reliability, accuracy, reproducibility, and workflow. Current and emerging evidence suggests that these systems will undoubtedly revolutionize the diagnostic utility of echocardiography both at subclinical and clinical stages and are expected to improve personalized cardiovascular risk assessment. However, widespread implementation of this novel technology will need to overcome challenging regulatory body approval processes. At present, the technology shows promise in improving diagnostic pathways, but evidence of clinical utility is lacking. Large trials will be required to provide robust evidence of ML’s prognostic value in echocardiographic assessment before its implementation in routine clinical practice.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
Introduction
Echocardiography remains the most indispensable cardiac imaging modality for the diagnosis, prognostication, and follow-up of various cardiovascular diseases (CVD). It allows for a rapid and comprehensive noninvasive assessment and accurate diagnosis of myocardial and structural heart diseases, provides quantifiable measurements of left and right ventricular function and pulmonary pressures, and facilitates the assessment of valvular function and intracardiac shunts [1]. Echocardiography is recommended as the first line cardiac imaging modality due to its widespread availability, low costs, versatility, acceptability, and safety profile. The demand for echocardiography is predicted to rise over the next few years due to the projected increase in the prevalence of CVD [2].
Despite its prolific utilization in clinical practice, echocardiography has some well-recognized limitations that impact its usefulness. For example, an echocardiogram generates a large volume of data that must be subjectively interpreted, resulting in significant inter-observer and intra-observer variability. Large and complex datasets generated during an echocardiogram are difficult to interpret, and a significant portion of this data remains underutilized [3]. Echocardiography is further limited by operator-dependent data acquisition, variability in instrumentation, poor image quality, and variability in data analysis and interpretation which often require expert skills.
Artificial intelligence (AI) has emerged over the last few decades as a novel technique with great potential for enhancing the diagnostic utility of cardiac imaging. AI involves the use of machine or computer systems to perform complex tasks with good and reproducible accuracy [4, 5•]. This technology has existed since the 1950s, but its application to medical diagnostics, particularly in cardiology, is relatively new and continues to evolve. AI is expected to be incorporated into all forms of cardiac imaging including echocardiography, cardiac magnetic resonance (CMR), cardiac-computed tomography angiography, single-photon emission–computed tomography, and positron emission tomography [6,7,8,9,10]. AI will undoubtedly have an impact on medical diagnostics, clinical decision-making, and treatment choices [11].
In echocardiography, AI is expected to improve the workflow by fully automating echocardiographic analysis and reducing the time required to provide clinically meaningful diagnostic information to the treating clinicians. Existing studies have now demonstrated that ML-based analysis of echocardiogram significantly reduce inter-observer variability and improve reproducibility [10, 12]. These algorithms have ability to analyse complex datasets and identify subtle cardiac structural and functional patterns that could easily be missed by trained observers [13, 14]. Current data suggest that the incorporation of ML algorithms into routine echocardiography may allow clinicians to expand indications for echocardiography to include a full assessment of conditions that are traditionally diagnosed with the more advanced imaging modalities such as CMR [15•]. Furthermore, ML techniques combined with both clinical and imaging data variables will further enhance clinical decision-making by allowing for personalized cardiac risk assessment for individual patients [16]. Perhaps the greatest benefits of ML-based echocardiographic analysis could be the ability to detect CVD at subclinical stages, predict progression and prognosis of CVD, and achieve the goal of personalized medicine through characterization of disease phenotypes and subsequent targeted therapies.
This review summarizes the recent discoveries and updates in the field of ML-assisted automated echocardiography. Recent updates in the use of ML-based models in the identification of echocardiographic views, image segmentation, and image interpretation, as well as quantification of cardiac structure and function are discussed. We conclude the review by highlighting specific areas of ML-based echocardiography with the greatest potential to revolutionize modern echocardiography, such as early detection of CVD at subclinical stages, prognostication of CVD, and disease phenotyping.
The Basic Concepts of Artificial Intelligence
AI and ML are often used interchangeable although strictly speaking, ML techniques can be best described as methods for achieving AI [4]. For both AI and ML, statistical modelling and mathematical algorithms are used to seek, quantify, and interpret relationships between datasets [4]. As many reviews have previously described in great detail the basics of AI and ML and associated terminologies [4, 7, 8, 17•], our focus here is on the recent developments in the field of echocardiography and ML.
It is important, however, to appreciate the differences between the two types of ML that are commonly used to train AI models, namely “supervised learning” and “unsupervised learning”. An understanding of these two types of ML is important for the appreciation of recent advances made in the field of automated echocardiography.
Supervised ML is used to teach an AI system to rapidly, accurately, and efficiently analyze large quantities of data using computing and statistical algorithms to achieve accurate and reproducible classification of data [18]. This learning technique is commonly used in medical diagnostics, such as in the recognition of echocardiographic views and detection of cardiac abnormalities on echocardiography [5•]. The key to supervised learning is that it requires the input of pre-labelled data to train a model to enhance classification and regression of further data [19].
On the contrary, unsupervised ML focuses on discovering patterns and associations between variables and uses unlabelled datasets [4]. Unlike supervised ML, unsupervised learning does not attempt to fit data to an outcome; rather, it identifies consistent patterns within datasets and lends itself to data exploration and generation of new hypotheses [4, 20]. As such, unsupervised ML promises to play a significant role in the unmet areas of cardiovascular imaging and practice. As an example, using an unsupervised, hierarchical cluster analysis technique, Lancaster et al. have previously demonstrated that their clustering algorithms improved the prediction of event-free survival over conventional classification for all-cause mortality and cardiac mortality in patients being evaluated for diastolic dysfunction [21]. Limited evidence also suggests that unsupervised ML techniques could identify specific characteristics and phenotypes of heart failure with preserved ejection fraction that are not visible to a human observer [22].
For both types of learning, the performance of ML programs improves as the dataset increases in size [23]. Because supervised and unsupervised learning techniques have distinctive roles, the learning technique that is utilized to train an AI model should carefully consider the intended use of the model.
Deep learning (DL) is a subset of AI inspired by the workings of the human brain, commonly referred to as an artificial neural network (ANN). The main advantage of DL is its iterative ability to continue learning and thereby improving its processing speed and diagnostic accuracy over time without explicit guidance [24,25,26]. There are 3 types of nodes in an ANN: input nodes, hidden nodes, and output nodes. The hidden nodes are analogous to the interneurons and perform the processing of information [26]. There can be any number of interconnected layers within the network with each layer containing any number of nodes [24]. Each layer receives inputs from previous layers and sends outputs to subsequent layers until the output layer is reached [26].
A convolutional neural network (CNN) is a subtype of ANN that mimics the visual cortex. Its main function is to process two-dimensional (2D) image–based data using multiple layers [24, 25]. Typically, a CNN is structured so that the lowest layers recognize the simplest features of the image such as the edges, with subsequent layers recognizing combinations of simple features. As such, each layer provides increasing levels of abstraction while the top layer provides the output [4, 5•, 27].
DL methods play significant roles in echocardiography such as identification of echocardiographic views, and recognition of useful associations between images and pathologies [5•, 9, 27]. DL has already proven itself to be of value in the fields of radiology (detecting boundaries of organs in computed tomography and magnetic resonance imaging), pathology (alerting suspicious areas of tissue biopsies), and dermatology (interpreting skin lesions as benign or malignant) [28,29,30]. These qualities readily translate to echocardiography and are discussed in the following sections and summarized in Table 1.
Artificial Intelligence in Echocardiography
Image Acquisition and Recognition of Views
The accuracy of transthoracic echocardiography (TTE) relies upon accurate image acquisition, traditionally achieved with the standard windows, beginning with the parasternal views (long axis and short axis) followed by apical and subcostal views. These result in the acquisition of approximately 70 unlabelled videos collected from non-consecutively acquired views [5•]. The conclusion or diagnosis is achieved by integrating these acquisitions and by approximating three-dimensional (3D) objects from 2D cross-sectional images. As a result, the first step to interpretation of echocardiography is the ability to recognize and appreciate the limitations of each of the echocardiographic views [3]. Echocardiography can be limited by variability occurring during image acquisition, challenging boundary detection, as well as variability occurring during image interpretation [31]. Indeed, variability in image quality and interpretation are well-recognized limitations in echocardiography, resulting in poor reproducibility and significant controversies [31, 32].
Recent advances in state-of-the-art computer vision software and ML techniques have potential for reducing this variability and improve accuracy. Several studies have now demonstrated that using novel ML-based approaches can accurately and efficiently distinguish echocardiographic views. Khamis et al. used a multi-stage classification algorithm that employed spatio-temporal feature extraction (cuboid detector) and supervised dictionary learning (LC-KSVD) approaches to identify apical four, two, and three-chamber views with an accuracy of approximately 95% [33]. A CNN model tested by Zhang et al. had an accuracy of 84% in distinguishing between parasternal long axis, parasternal short axis, apical 2-chamber, apical 3-chamber, and apical 4-chamber views [5•]. On a larger scale of images, Madani et al.’s CNN was able to classify 15 major views (12 from B-mode and 3 from pulsed-wave Doppler, continuous-wave Doppler and M-mode recordings). Using cluster analysis, this method achieved an accuracy of 97.8% on videos from patients who had a variety of pathologies, including normal variants and suboptimal views [34]. On still images, the model performed better than board-certified echocardiographers with an accuracy of 91.7% compared with 79.4%, respectively [34]. The study by Madani and colleagues further demonstrated the potential mechanisms by which the CNN is able to classify echocardiographic views [34]. By performing exclusion experiments, the authors showed that the CNN perhaps works in a similar fashion to an expert echocardiographer when classifying echocardiographic views (Fig. 1).
These studies demonstrate the ability of trained AI-based models to identify standard echocardiographic views, a first step towards echocardiographic image analysis. Importantly, ML algorithms are able to recognize these views in less than a minute (21 ms in the case of Madani et al’s model), thereby demonstrating the potential for ML to improve speed and efficiency of echocardiographic workflow. Further work is still required before these innovations are incorporated into routine clinical practice. As demonstrated by Madani and colleagues, the accuracy of most ML algorithms used for identification of echocardiographic views is highest for views that are most highly represented in the training data, and lowest for views which are closely related, e.g., apical three-chamber vs apical two-chamber views and apical four-chamber vs apical five-chamber views [34]. Furthermore, real-world data are still needed to demonstrate that the incorporation of these ML-based view recognition algorithms into modern echocardiography translate into cost-efficient and effective strategies.
Image Analysis and Interpretation
Assessment of Left Ventricular Function
The interpretation of acquired echocardiography images requires expertise and can be challenging to a less experienced reader. The assessment of left ventricular function (both visually and quantitatively) is the most commonly performed measurement in echocardiography and has significant prognostic value [1, 32]. The assessment of left ventricular ejection function (LVEF) is subject to significant inter-observer variability and poor reproducibility [33].
There is good evidence that ML can improve the assessment of left ventricular function and reduce this variability. Knackstedt et al. demonstrated that LVEF and longitudinal strain could rapidly and reproducibly be analysed in 8 s using a ML algorithm, and this was not subject to the variability encountered when visually assessing or manually tracking [35]. Several other studies have also shown that automated software in 3D TTE can reproducibly interpret end-diastolic volume, end-systolic volume, and LVEF. Levy et al. studied 54 patients who had comprehensive 3D TTE and CMR performed on the same day and found significant correlations between CMR and ML-based 3D TTE assessment of end-diastolic volume (r = 0.93), end-systolic volume (r = 0.93), and LVEF (r = 0.91) [36]. Tsang et al.’s study calculated automated measurements of LVEF, left ventricular end-systolic volume, and left atrial volumes with ML-based methods and demonstrated strong agreement with CMR-derived values (r = 0.84 to 0.95) [37]. A similar technique for left chamber quantification in 88 patients with atrial fibrillation using single beat full-volume 3D TTE data correlated well with manual tracing while, taking significantly less time (5 min) than the manual tracing method (27 min) [38].
These data demonstrate the potential of ML techniques to revolutionize the echocardiographic assessment of cardiac function and disease detection by rapidly providing accurate and reproducible measurements with good diagnostic accuracy [4]. Furthermore, because the manual tracing required to achieve quantitative assessment of ventricular function (recommended by guidelines) is often not feasible in routine clinical practice, this rapid and reproducible assessment by ML algorithms will be increasingly important with the increasing numbers of echocardiograms being performed worldwide, both in cardiology and emergency departments.
Assessment of Right Ventricular Function
Many pathologies can cause right ventricular (RV) dysfunction, including chronic left-sided heart failure, acute coronary syndromes, pulmonary embolism, pulmonary arterial hypertension, and congenital heart diseases. Irrespective of the aetiology, patients with RV dysfunction experience excess morbidity and mortality [39]. Because of its complex anatomic features (crescent shaped) and retrosternal position, the assessment of RV with 2D echocardiography is challenging and is associated with many limitations such as foreshortening, geometrical assumptions, cumbersome acquisition, and long post-processing times [31]. As a result, although the RV carries invaluable prognostic information, it has traditionally received less attention in echocardiographic assessment.
Accurate and reproducible quantification of the RV size and function can be achieved with 3D echocardiography. However, the existing software programs used with 3D echocardiography are time-consuming and require specialized training in image acquisition and data analysis [40]. 3D echocardiography has therefore not been widely implemented in routine clinical assessment of the RV [40]. Could AI become the answer to optimal and reproducible echocardiographic evaluation of the RV? In an unselected cohort of 56 patients with a wide range of RV sizes and function who underwent 3D echocardiography and CMR imaging on the same day, Genovese et al. recently demonstrated that the RV size and function could be evaluated with ML-based methods with excellent reproducibility [16]. In this study, accurate and fully automated measurements of the RV size and function were possible in 32% of the patients while accurate measurements were obtained in the remaining patients with minimal boundary editing.
Although more studies are needed to demonstrate the reproducibility of ML-based methods for the assessment of RV, the study by Genovese and colleagues offer very promising results. With the rapid expansion of ML algorithms, it is anticipated that incorporation of ML-based techniques into 3D echocardiography could potentially become the standard assessment of the RV.
Valvular Function
Early diagnosis and treatment of primary valvular diseases such as primary mitral and tricuspid regurgitation are associated with improved outcomes [41]. However, current analysis of valvular diseases requires manual imputation of datasets or multiple echocardiographic measurements and is subject to inter-observer variability [42]. Limited data suggest that the assessment of valvular function and identification of pathology may be amenable to ML techniques. Jeganathan and colleagues have previously shown that automated ML techniques improve reproducibility of mitral valve assessment using 3D transesophageal echocardiography [43]. ML algorithms could also be used to assess the severity of mitral regurgitation (MR) with one study demonstrating that this can be achieved with a sensitivity of 99.38% and specificity of 99.63% for differentiating between normal, moderate, and severe MR [42]. Emerging data indicate that aortic stenosis can be predicted from other echocardiographic data without the need for any left ventricular outflow tract measurements (gradients, velocity, and diameter) [44]. This would have great potential for improving efficiency and reducing study duration and cost. However, the current data for using ML-based algorithms for valvular assessment remain limited, and more studies are needed. We are yet to see quality studies demonstrating the utility of ML-assisted automated analysis and quantification of tricuspid and pulmonic valves as well as mitral stenosis. Nonetheless, the smaller number of existing studies suggests that ML-assisted valvular assessment will become a reality in the next few years.
Diagnostic Utility
Emerging data suggest that ML-based echocardiographic models may improve the diagnosis of cardiac diseases that are traditionally challenging to diagnose with echocardiography, and that often require more advanced imaging modalities, such as CMR. Narula et al. combined the techniques of support vector machines, random forests, and ANNs to accurately differentiate pathologic from physiologic cardiomyopathy in athletes [10]. This ensemble model displayed higher sensitivity and specificity when compared with the standard quantitative measurements of early-to-late diastolic transmitral velocity ratio (p < 0.01), average early diastolic tissue velocity (e’) (p < 0.01), and strain (p = 0.04) [10]. Zhang et al. trained different CNNs to recognize the appearance of 3 chronic structural heart diseases: hypertrophic cardiomyopathy, cardiac amyloidosis, and pulmonary arterial hypertension with C statistics (area under the receiver operating characteristic curve, AUC) of 0.93, 0.87, and 0.85, respectively [5•]. Finally, Raghavendra et al. deployed an unsupervised learning model (double density-dual tree discrete wavelet transform to detect 12 echocardiographic features to distinguish patients with obstructive coronary artery disease from those without disease with accuracy of 96.05%, sensitivity of 96.12%, and specificity of 96.00% [45].
Furthermore, ML-assisted strain imaging and speckle tracking echocardiography (STE) could potentially be used to differentiate between commonly mistaken conditions. Sengupta et al. found that an ML algorithm was able to differentiate between restrictive cardiomyopathy and constrictive pericarditis using 15 STE variables (AUC of 89.2%) [13]. This algorithmic approach, referred to as an associative memory classifier, achieved an AUC of 96.2% after the addition of 4 echocardiographic variables (e’, E/e’, septal, and posterior left ventricular wall thickness). Furthermore, the model was superior to commonly used echocardiographic variables such as early diastolic mitral annular longitudinal velocity and global longitudinal strain. The study demonstrated the ability of an ML algorithm to be taught and then recall key components of a complex echocardiographic evaluation, and has, as the authors concluded, the “potential for standardizing and improving workflow in busy echocardiography laboratories” [12]. Further AI developments and improvements are expected over the next few years, and these will radically transform modern echocardiography.
Stress Echocardiography
Stress echocardiography is vulnerable to wide variations in sensitivity and specificity [46, 47]. ML techniques show promise in identifying and quantifying inducible wall motion abnormalities, thereby demonstrating an ability to overcome the limitations in accuracy. Mansor et al.’s automated model showed improved classification of wall motion abnormalities (84%) compared with individual rest (73%) and stress (68%) sequences [48]. Another study that compared random forests, support vector machines, and CNN on a 3D dobutamine stress echocardiography dataset demonstrated that these automated techniques could differentiate between normal and abnormal wall motions, and that this ability was slightly higher for deep learning approach [49].
3D Echocardiography
Despite the superior accuracy and reproducibility of 3D echocardiography compared with 2D echocardiography, widespread application of 3D echocardiography for quantitative assessment of left atrial and left ventricular volumes has not yet been implemented due to the increased time and expertise required to obtain these measurements [37]. As discussed above, current data indicate that the transition to 3D echocardiography may be hastened with incorporation of well-trained and validated ML algorithms [12, 50].
The Future of Artificial Intelligence in Echocardiography
Detection of CVD Disease at Subclinical Stage
CVD can progress at subclinical stages for many years before the onset of symptoms. Delayed diagnosis can lead to poor prognosis often related to irreversible pathophysiologic changes that occur over many years. With the increasing prevalence of obesity, hypertension, and metabolic diseases such as diabetes, there is a growing interest in the identification of disease at subclinical stages which may facilitate early interventions and improve outcomes. Many noninvasive techniques such as electrocardiograms and echocardiography, which are readily available in community settings, coupled with ML algorithms, could be used to achieve this goal [51].
LVEF has traditionally been used as a marker of LV dysfunction, with low LVEF being associated with poorer outcomes [1]. However, LVEF is highly dependent on hemodynamic conditions and is less sensitive to minor changes occurring over time. Global longitudinal strain (GLS), a measure of myocardial deformation, is a novel and relatively newer marker of subclinical LV dysfunction and has useful prognostic value [52, 53]. The prognostic value of GLS is observed in patients with heart failure irrespective of the LVEF [53, 54]. GLS can be measured with tissue Doppler imaging or 2D stress echocardiography. However, GLS is not routinely measured in clinical practice as it requires appropriate training. Zhang et al. demonstrated that fully automated assessment of an entire echocardiogram including the evaluation of GLS is possible with ML algorithms [5•]. Once incorporated into routine echocardiography, ML-based models may improve the detection of CVD at subclinical stages by enhancing the detection of subtle changes that are not detectable or feasible with the current methods.
Furthermore, there is a highly anticipated expectation that ML technology will improve access to echocardiography in rural and less developed parts of the world. This would facilitate early detection of CVD and help to close the gap in disease outcomes between these regions and the more affluent areas. Zhang et al. postulated that AI could help detect early signs of heart disease in a cost-efficient manner by having non-experts use handheld imaging devices at point-of-care locations and subsequently uploading images to a cloud-based automated interpretation system that would allow comparison with prior studies [5•]. Such an approach would promote early detection of CVD and potentially reduce the morbidity and mortality associated with CVD [55].
Prognostication and Risk Prediction
In an era of sophisticated noninvasive cardiovascular imaging modalities, molecular profiling, and ML technologies, there is a growing interest in achieving personalized risk assessment and prognostication. LVEF is currently the most commonly used echocardiographic prognostic marker despite its limited prognostic value [52, 54]. A search for improved prognostic markers is an area where application of ML-based methods may play a substantial role in improving prognostic assessment. Samud et al. have previously demonstrated that ML-based echocardiography combined with electronic health records could significantly improve the prediction of adverse outcomes with an accuracy of 96% [56•]. Unsupervised ML techniques using clustering methods are also promising to improve the identification of at-risk patients and the prediction of cardiovascular outcomes. These techniques have been demonstrated to identify a cluster of echocardiographic phenotypes associated with an increased risk of adverse cardiovascular outcomes in patients with type 2 diabetes [57], diastolic dysfunction [21, 58], and heart failure [59]. Using guideline-recommended echocardiographic parameters for assessment of left ventricular diastolic dysfunction, an unsupervised cluster analysis identified unique patterns that predicted outcomes better than conventional classifications recommended by the guidelines [60•].
These results suggest that ML techniques may uncover as-yet unknown, but significant, echocardiographic measurements that have a real impact on clinical outcomes. Zhang et al. previously demonstrated that an ML algorithm could play a significant role in longitudinal monitoring for cardiotoxicity in cancer survivors [5•]. Based on these preliminary findings, it is anticipated that ML-based techniques will improve the prognostic assessment of various cardiovascular diseases. The current and potential applications of ML algorithms in echocardiography are illustrated in Fig. 2.
Implications for Echocardiography Workforce
AI and ML algorithms are expected to radically change echocardiographic imaging practice. This has raised great concerns that “machines” may potentially replace echocardiographers in the future. This may not be the correct assumption at least based on current evidence. Instead, the echocardiographers will benefit from the support that will be offered by ML-assisted cardiac imaging. These technologies will significantly reduce the time that the imagers spend extracting and integrating the echocardiographic data and will therefore improve the workflow. The echocardiographers of the future will be expected to understand the basic principles of AI and possess abilities to integrate ML-based applications into routine echocardiography. Therefore, echocardiographers who embrace this innovative technology will stand to benefit rather than lose.
Challenges and Pitfalls
Despite the significant progress that has been made in ML-based echocardiography, there are still many challenges to overcome. First, our understanding of the inner workings of the hidden networks of ML models, often involving thousands of interconnections, remains limited [61]. Second, the training of ML models is restricted by the availability of large datasets covering a broad range of patients and pathologies/normal variants. As demonstrated in previous studies, these large datasets are required for optimal training and validation of clinically useful ML algorithms [34]. In particular, labelling of diagnostic images, required for supervised ML, is challenging, time-consuming, and requires appropriate training and expertise. Generative adversarial networks are now being explored as potential strategies for addressing this limitation. A generative adversarial network consists of a generator which generates an image, and a discriminator that predicts whether an image is a real or generated image [17•]. Generative adversarial networks were initially developed for use in unsupervised learning but can now be leveraged in semi-supervised ML to perform automated image analysis in data-limited settings [62].
A third challenge is the lack of data demonstrating clinical utility. Current studies investigating the use of ML in echocardiography have compared their results against expert readers rather than disease outcomes. For AI to be incorporated into clinical practice, it will need to prove its clinical utility beyond being diagnostically useful. Figure 3 demonstrates the stages in the development of a clinically useful AI model and highlights the limitations related to each stage. As shown in the figure, the current data demonstrate feasibility of ML-based echocardiography, with limited single-center studies suggesting potential diagnostic utility. However, there are currently no data to show that these ML-based models improve clinical outcomes. Therefore, studies evaluating the effects of ML applications on these outcomes will be required before the integration of ML into modern echocardiography practice. Linking electronic health records to existing imaging databases will perhaps be the first step towards achieving this goal, but this is yet to be fully accomplished [11, 63].
Finally, due to the potential legal liabilities and clinical governance issues, application of this technology will require extensive validation prior to regulatory approval. The success of ML-based models in echocardiography will rest upon the ability of computer scientists, echocardiographers, and cardiologists to collaborate in order to address these limitations.
Conclusion
AI is promising to transform all forms of cardiac imaging, including echocardiography. Current and emerging data suggest that these ML algorithms are able to recognize echocardiographic views, automatically evaluate left and right ventricular function, and quantify chamber and valvular measurements. The main clinical benefits of ML algorithms in echocardiography will include reduction in time spent extracting and integrating echocardiographic data and improvement in reproducibility of echocardiographic assessments. This revolutionary approach will allow patients to obtain a more objective assessment in a timely manner. It is also expected that improvement in workflow and efficiency will reduce the costs of echocardiography imaging. Other potential benefits will include incorporation of ML into risk prediction models and early detection of cardiovascular diseases at preclinical stages.
Although significant progress has been made over the last few years, we are currently still many steps behind the incorporation of ML technology into routine echocardiography. Clinical utility lags behind diagnostic utility as we are yet to demonstrate that ML-based diagnostic models improve clinical outcomes. Extensive validation and robust clinical benefits will need to be demonstrated before regulatory approval can be provided. Current evidence suggests we are still a long way from clinical application.
Nonetheless, the future of AI and echocardiography is bright, and with the rapidly expanding knowledge of ML algorithms, the application of ML-based echocardiography is within reach and is expected to radically transform modern echocardiography in the near future.
References
Papers of particular interest, published recently, have been highlighted as: • Of importance
Hillis GS, Bloomfield P. Basic transthoracic echocardiography. BMJ (Clinical research ed). 2005;330(7505):1432–6.
Boon N, Norell M, Hall J, Jennings K, Penny L, Wilson C, et al. National variations in the provision of cardiac services in the United Kingdom: second report of the British cardiac society working group, 2005. Heart. 2006;92(7):873–8.
Wharton G, Steeds R, Allen J, Phillips H, Jones R, Kanagala P, et al. A minimum dataset for a standard adult transthoracic echocardiogram: a guideline protocol from the British society of echocardiography. Echo Res Pract. 2015;2(1):G9–G24.
Johnson KW, Torres Soto J, Glicksberg BS, Shameer K, Miotto R, Ali M, et al. Artificial intelligence in cardiology. J Am Coll Cardiol. 2018;71(23):2668–79.
• Zhang J, Gajjala S, Agrawal P, Tison Geoffrey H, Hallock Laura A, Beussink-Nelson L, et al. Fully automated echocardiogram interpretation in clinical practice. Circulation. 2018;138(16):1623–35 This is a very interesting paper that utilises a large echocardiography dataset to fully automate echocardiographic analysis using ML techniques from view identification, image segmentation, quantification of structure and function to disease detection.
Alsharqi M, Upton R, Mumith A, Leeson P. Artificial intelligence: a new clinical support tool for stress echocardiography. Expert Rev Medical Devices. 2018;15(8):513–5.
Alsharqi M, Woodward WJ, Mumith JA, Markham DC, Upton R, Leeson P. Artificial intelligence and echocardiography. Echo Res Pract. 2018;5(4):R115–r25.
Gandhi S, Mosleh W, Shen J, Chow CM. Automation, machine learning, and artificial intelligence in echocardiography: a brave new world. Echocardiography. 2018;35(9):1402–18.
Krittanawong C, Tunhasiriwet A, Zhang H, Wang Z, Aydar M, Kitai T. Deep learning with unsupervised feature in echocardiographic imaging. J Am Coll Cardiol. 2017;69(16):2100–1.
Narula S, Shameer K, Salem Omar AM, Dudley JT, Sengupta PP. Machine-learning algorithms to Automate morphological and functional assessments in 2D echocardiography. J Am Coll Cardiol. 2016;68(21):2287–95.
Motwani M, Dey D, Berman DS, Germano G, Achenbach S, Al-Mallah MH, et al. Machine learning for prediction of all-cause mortality in patients with suspected coronary artery disease: a 5-year multicentre prospective registry analysis. Eur Heart J. 2017;38(7):500–7.
Sengupta PP, Huang YM, Bansal M, Ashrafi A, Fisher M, Shameer K, et al. Cognitive machine-learning algorithm for cardiac imaging: a pilot study for differentiating constrictive pericarditis from restrictive cardiomyopathy. Circ Cardiovasc Imaging. 2016;9(6).
Arsanjani R, Dey D, Khachatryan T, Shalev A, Hayes SW, Fish M, et al. Prediction of revascularization after myocardial perfusion SPECT by machine learning in a large population. J Nucl Cardiol. 2015;22(5):877–84.
Haro Alonso D, Wernick MN, Yang Y, Germano G, Berman DS, Slomka P. Prediction of cardiac death after adenosine myocardial perfusion SPECT based on machine learning. J Nucl Cardiol. 2018;26(5):1746–54.
• Genovese D, Rashedi N, Weinert L, Narang A, Addetia K, Patel AR, et al. Machine learning-based three-dimensional echocardiographic quantification of right ventricular size and function: validation against cardiac magnetic resonance. J Am Soc Echocardiogr. 2019;32(8):969–77 This is a recent paper demonstrating the use of ML techniques for automated assessment of right ventricular size and function.
Betancur J, Otaki Y, Motwani M, Fish MB, Lemley M, Dey D, et al. Prognostic value of combined clinical and myocardial perfusion imaging data using machine learning. J Am Coll Cardiol Img. 2018;11(7):1000–9.
• Litjens G, Ciompi F, Wolterink JM, de Vos BD, Leiner T, Teuwen J, et al. State-of-the-art deep learning in cardiovascular image analysis. JACC Cardiovasc Imaging. 2019;12(8 Pt 1):1549–65 This in-depth review provides an excellent overview of ML techniques with a focus on DL, including limitations associated with each DL technique.
Mayr A, Binder H, Gefeller O, Schmid M. The evolution of boosting algorithms. From machine learning to statistical modelling. Methods Inf Med. 2014;53(6):419–27.
Playford D, Jais P, Weerasooriya R, Martyn S, Bollam L, Turewicz M, et al. A validation study of automated atrial fibrillation detection using Alerte digital health’s artificial intelligence system. Heart Lung Circ. 2017;26:S279–S80.
Deo RC. Machine learning in medicine. Circulation. 2015;132(20):1920–30.
Lancaster MC, Salem Omar AM, Narula S, Kulkarni H, Narula J, Sengupta PP. Phenotypic clustering of left ventricular diastolic function parameters: Patterns and Prognostic Relevance. JACC Cardiovasc Imaging. 2018;2562(7 Pt 1):1149–61.
Sanchez-Martinez S, Duchateau N, Erdei T, Fraser AG, Bijnens BH, Piella G. Characterization of myocardial motion patterns by unsupervised multiple kernel learning. Med Image Anal. 2017;35:70–82.
Shrestha S, Sengupta PP. Machine learning for nuclear cardiology: the way forward. J Nucl Cardiol. 2018;26(5):1755–8.
Forsstrom JJ, Dalton KJ. Artificial neural networks for decision support in clinical medicine. Ann Med. 1995;27(5):509–17.
LeCun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015;521(7553):436–44.
Dilsizian ME, Siegel EL. Machine meets biology: a primer on artificial intelligence in cardiology and cardiac imaging. Curr Cardiol Rep. 2018;20(12):139.
Lee JG, Jun S, Cho YW, Lee H, Kim GB, Seo JB, et al. Deep learning in medical imaging: general overview. Korean J Radiol. 2017;18(4):570–84.
Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature. 2017;542:115.
Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. Jama. 2016;316(22):2402–10.
Litjens G, Kooi T, Bejnordi BE, Setio AAA, Ciompi F, Ghafoorian M, et al. A survey on deep learning in medical image analysis. Med Image Anal. 2017;42:60–88.
Lang RM, Badano LP, Mor-Avi V, Afilalo J, Armstrong A, Ernande L, et al. Recommendations for cardiac chamber quantification by echocardiography in adults: an update from the American Society of Echocardiography and the European Association of Cardiovascular Imaging. J Am Soc Echocardiogr. 2015;28(1):1–39 e14.
Pellikka PA, She L, Holly TA, Lin G, Varadarajan P, Pai RG, et al. Variability in ejection fraction measured by echocardiography, gated single-photon emission computed tomography, and cardiac magnetic resonance in patients with coronary artery disease and left ventricular dysfunction variability in left ventricular ejection fraction by cardiac imaging modality variability in left ventricular ejection fraction by cardiac imaging modality. JAMA Netw Open. 2018;1(4):e181456 e.
Khamis H, Zurakhov G, Azar V, Raz A, Friedman Z, Adam D. Automatic apical view classification of echocardiograms using a discriminative learning dictionary. Med Image Anal. 2017;36:15–21.
Madani A, Arnaout R, Mofrad M, Arnaout R. Fast and accurate view classification of echocardiograms using deep learning. NPJ Digit Med. 2018;1(1):6.
Knackstedt C, Bekkers SC, Schummers G, Schreckenberg M, Muraru D, Badano LP, et al. Fully automated versus standard tracking of left ventricular ejection fraction and longitudinal strain: the FAST-EFs multicenter study. J Am Coll Cardiol. 2015;66(13):1456–66.
Levy F, Dan Schouver E, Iacuzio L, Civaia F, Rusek S, Dommerc C, et al. Performance of new automated transthoracic three-dimensional echocardiographic software for left ventricular volumes and function assessment in routine clinical practice: comparison with 3 tesla cardiac magnetic resonance. Arch Cardiovasc Dis. 2017;110(11):580–9.
Tsang W, Salgo IS, Medvedofsky D, Takeuchi M, Prater D, Weinert L, et al. Transthoracic 3D echocardiographic left heart chamber quantification using an automated adaptive analytics algorithm. J Am Coll Cardiol Img. 2016;9(7):769–82.
Otani K, Nakazono A, Salgo IS, Lang RM, Takeuchi M. Three-dimensional echocardiographic assessment of left heart chamber size and function with fully automated quantification software in patients with atrial fibrillation. J Am Soc Echocardiogr. 2016;29(10):955–65.
Haddad F, Doyle R, Murphy Daniel J, Hunt SA. Right ventricular function in cardiovascular disease, Part II. Circulation. 2008;117(13):1717–31.
Rudski LG, Lai WW, Afilalo J, Hua L, Handschumacher MD, Chandrasekaran K, et al. Guidelines for the echocardiographic assessment of the right heart in adults: a report from the American Society of Echocardiography endorsed by the European Association of Echocardiography, a registered branch of the European Society of Cardiology, and the Canadian Society of Echocardiography. J Am Soc Echocardiogr. 2010;23(7):685–713 quiz 86–8.
Baumgartner H, Falk V, Bax JJ, De Bonis M, Hamm C, Holm PJ, et al. 2017 ESC/EACTS guidelines for the management of valvular heart disease. Eur Heart J. 2017;38(36):2739–91.
Moghaddasi H, Nourian S. Automatic assessment of mitral regurgitation severity based on extensive textural features on 2D echocardiography videos. Comput Biol Med. 2016;73:47–55.
Jeganathan J, Knio Z, Amador Y, Hai T, Khamooshian A, Matyal R, et al. Artificial intelligence in mitral valve analysis. Ann Card Anaesth. 2017;20(2):129–34.
Playford D, Bordin E, Talbot L, Mohamad R, Anderson B, Strange G. Analysis of aortic stenosis using artificial intelligence. Heart Lung Circ. 2018;27:S216.
Raghavendra U, Fujita H, Gudigar A, Shetty R, Nayak K, Pai U, et al. Automated technique for coronary artery disease characterization and classification using DD-DTDWT in ultrasound images. Biomed Signal Process Control. 2018;40:324–34.
Chykeyuk K, Clifton DA, Noble JA, editors. Feature extraction and wall motion classification of 2D stress echocardiography with relevance vector machines. 2011 IEEE International Symposium on Biomedical Imaging: From Nano to Macro; 2011 30 March-2 April 2011.
Geleijnse ML, Krenning BJ, van Dalen BM, Nemes A, Soliman OI, Bosch JG, et al. Factors affecting sensitivity and specificity of diagnostic testing: dobutamine stress echocardiography. J Am Soc Echocardiogr. 2009;22(11):1199–208.
Mansor S, Hughes NP, Noble JA. Wall motion classification of stress echocardiography based on combined rest-and-stress data. Med Image Comput Comput Assist Interv. 2008;11(Pt 2):139–46.
Omar HA, Domingos JS, Patra A, Upton R, Leeson P, Noble JA, editors. Quantification of cardiac bull’s-eye map based on principal strain analysis for myocardial wall motion assessment in stress echocardiography. 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018); 2018: IEEE.
Zhou SK, Guo F, Park J, Carneiro G, Jackson J, Brendel M, et al., editors. A probabilistic, hierarchical, and discriminant framework for rapid and accurate detection of deformable anatomic structure. 2007 IEEE 11th International Conference on Computer Vision; 2007: IEEE.
Attia ZI, Kapa S, Lopez-Jimenez F, McKie PM, Ladewig DJ, Satam G, et al. Screening for cardiac contractile dysfunction using an artificial intelligence-enabled electrocardiogram. Nat Med. 2019;25(1):70–4.
Hiemstra YL, Tomsic A, van Wijngaarden SE, Palmen M, Klautz RJM, Bax JJ, et al. Prognostic value of global longitudinal strain and etiology after surgery for primary mitral regurgitation. JACC Cardiovasc Imaging. 2019;3049.
Park JJ, Park JB, Park JH, Cho GY. Global longitudinal strain to predict mortality in patients with acute heart failure. J Am Coll Cardiol. 2018;71(18):1947–57.
Kalam K, Otahal P, Marwick TH. Prognostic implications of global LV dysfunction: a systematic review and meta-analysis of global longitudinal strain and ejection fraction. Heart. 2014;100(21):1673–80.
Braunwald E. The war against heart failure: the Lancet lecture. Lancet. 2015;385(9970):812–24.
• Samad MD, Ulloa A, Wehner GJ, Jing L, Hartzel D, Good CW, et al. Predicting survival from large echocardiography and electronic health record datasets: optimization with machine learning. JACC Cardiovasc Imaging. 2019;12(4):681–9 This study demonstrates the potential for ML algorithms to improve prognostic assessment in echocardiography when combined with clinical variables.
Ernande L, Audureau E, Jellis CL, Bergerot C, Henegar C, Sawaki D, et al. Clinical implications of echocardiographic phenotypes of patients with diabetes mellitus. J Am Coll Cardiol. 2017;70(14):1704–16.
Salem Omar AM, Lancaster MC, Narula S, Baiomi A, Narula J, Sengupta P. Computational unsupervised clustering of echocardiographic variables for the assessment of diastolic dysfunction severity. J Am Coll Cardiol. 2018;71(11 Supplement):A1519.
Omar AMS, Narula S, Abdel Rahman MA, Pedrizzetti G, Raslan H, Rifaie O, et al. Precision phenotyping in heart failure and pattern clustering of ultrasound data for the assessment of diastolic dysfunction. J Am Coll Cardiol Img. 2017;10(11):1291–303.
• Lancaster MC, Salem Omar AM, Narula S, Kulkarni H, Narula J, Sengupta PP. Phenotypic clustering of left ventricular diastolic function parameters: patterns and prognostic relevance. JACC cardiovascular imaging. 2019;12(7 Pt 1):1149–61 This is an interesting paper demonstrating the utility of unsupervised ML using clustering techniques to predict cardiovascular outcomes among patients undergoing echocardiographic assessment of diastolic function.
Khan S, Rahmani H, Shah SAA, Bennamoun M. A guide to convolutional neural networks for computer vision. Synthesis Lectures on Computer Vision. 2018;8(1):1–207.
Madani A, Ong JR, Tibrewal A, Mofrad MRK. Deep echocardiography: data-efficient supervised and semi-supervised deep learning towards automated diagnosis of cardiac disease. NPJ Digit Med. 2018;1(1):59.
Betancur J, Otaki Y, Motwani M, Fish MB, Lemley M, Dey D, et al. Prognostic value of combined clinical and myocardial perfusion imaging data using machine learning. JACC Cardiovasc Imaging. 2017;2406.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interest
All authors declare no conflict of interest.
Human and Animal Rights and Informed Consent
This article does not contain any studies with human or animal subjects performed by any of the authors.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This article is part of the Topical Collection on Echocardiography
Rights and permissions
About this article
Cite this article
Gahungu, N., Trueick, R., Bhat, S. et al. Current Challenges and Recent Updates in Artificial Intelligence and Echocardiography. Curr Cardiovasc Imaging Rep 13, 5 (2020). https://doi.org/10.1007/s12410-020-9529-x
Published:
DOI: https://doi.org/10.1007/s12410-020-9529-x