Keywords

Artificial Intelligence (AI) is a broad term commonly used to describe one of the highest growth industries worldwide, which is quickly finding new and exciting applications in healthcare, particularly for image-based diagnostic workup [1,2,3]. The concept of AI has evolved significantly since early work using model-driven algorithms in the 1940s to its current state where computational power allows observational learning of pattern by a computer algorithm (i.e., machine learning (ML)) from large amounts of now digitally available medical data, precluding the need for a priori knowledge of an underlying process and thus eliminating the requirement for human-driven modeling and feature engineering [1]. ML, a narrower subfield of AI, develops algorithms that learn to perform tasks, make decisions, or predictions automatically from data, rather than having a behavior explicitly programmed. ML can be broadly further subdivided in supervised and unsupervised types of learning. The latter techniques find patterns in data and provide structural learning (e.g., classes within a dataset) without the need for data annotation, making these suitable to learn from large and unlabeled datasets. Clustering analysis and Principal Component Analysis (PCA) are techniques typically used for these tasks. A recent study using the former technique identified four new different clinical phenotypes in septic patients, each with its corresponding host-response pattern and clinical outcome [2].

In contrast, supervised learning techniques may be applied in learning tasks using “cleaned” data. That is data that is organized and provided in the form of input examples (and its features) paired with those examples’ specific outputs which constitute the “ground truth” (or labels) to be predicted when building a certain inference model. Supervised learning techniques include regression and classification algorithms such as linear and logistic regression, discriminant analysis, decision tree, random forest, naïve Bayes, support vector machines (SVM), and neural networks. These type of ML models have been evolving for decades and continue to be the most commonly used in healthcare. An early example is DXplain, a medical decision support system developed at the Laboratory of Computer Sciences at Massachusetts General Hospital, that takes a set of clinical findings (signs, symptoms, laboratory data) and produces a ranked list of differential diagnoses [3].

Deep Learning (DL) started having an impact in the early 2000s and it took over an additional decade for its use in healthcare. It also encompasses methods in the family of ML, and most frequently referrers to supervised learning using Artificial Neural Networks (ANN), which use multiple layers of features from inputs to progressively extract higher level features in order to predict labeled outputs. Examples of ANN include convoluted neural networks (CNN), deep belief networks (DBN), and recurrent neural networks (RNN). DL may also be used in unsupervised or semi-supervised learning and can become competent in exceedingly complex relationships between features and labels, for which have shown similar or exceeding capabilities to humans in solving problems of computer vision (CV) in medicine [4,5,6]. Usually the process of preparing datasets with features and labels can be time consuming; however, once readily available, ML algorithms can rapidly be trained and tested. For instance, during a developing pandemic, when essential swab and serologic testing was lacking in the USA and around the globe, a pre-trained ANN showed high accuracy in diagnosing coronavirus disease 2019 (COVID-19) while differentiating this disease from other types of viral or bacterial pneumonias using plain chest X-rays; similar ANN validated chest CT against viral RNA RT-PCR (reverse transcription polymerase chain reaction) test as a sensitive modality for diagnosis of COVID-19 [7,8,9].

Other computational training tasks use unstructured data to build models that can, for example, diagnose from physician’s notes in medical records (a.k.a. as “free text”) in combination with the medical literature, using either supervised or unsupervised natural language processing (NLP) learning techniques. Recent examples in this domain have shown that complications from spinal surgery can be screened from operative reports, septic shock, suicide risk can be predicted from clinicians’ notes and patients that need follow-up imaging can be detected using previous radiology reports [10,11,12,13].

Each of these ML methods is uniquely suited to certain tasks, but their products—models predicting a class or outcome in a narrow specific area of healthcare or medical imaging—are the current essence of AI in medicine.

AI’s proposed applications in healthcare are diverse, from hospital finance to diagnostic and therapeutic applications, promising to improve efficiency by decreasing both the time that tasks take and the potential for medical error [14]. A key characteristic of AI’s current and proposed applications is expanding what is possible today, not only increasing the speed or accuracy of current processes.

1 AI Applications Support the Infrastructure and Interventions of Healthcare, Including Molecular Imaging

At the time of writing this chapter there were 114,002 results for “Artificial Intelligence” in PubMed and, while these entries date back to the year 1951, over fifty percent of the results correspond to the last 7 years. This highlights the importance of time in scientific breakthroughs. While models and theoretical concepts for the now call AI/Deep Learning revolution were first introduced over half-a-century ago by Frank Rosenblatt, a psychologist from Cornell University [15] and Marvin Minsky, a cognitive scientist from MIT [16], it was not until recently that these concepts were perfected and implemented at scale in various industries (including more recently healthcare) by computer and data scientists due to more recent explosive growth in the global digital datasphere and computational power.

1.1 Drug Development

In drug development AI models predict the chemical and pharmaceutical properties of small-molecule candidates for drug design and development, new applications of existing drugs, and new patients who can benefit from drugs, predict bioactivity and identify patient characteristics for clinical trials [17,18,19,20]. Moreover, AI in combination with sources of large amounts of data in the biosciences has been used to build in silico models of disease processes, such as cancer, to enable computer-aided design and testing of potential therapeutic compounds [21]. In the realm of infection, an approach using ML known as computational phenotyping, was capable to predict antibiotic resistance phenotypes in various bacterial pathogens and another showed it could facilitate rapid drug development against SARS coronavirus 2 (SARS-CoV-2) the causative organism in COVID-19 [22, 23].

1.2 Clinical Workflow

In clinical medicine, some of the tasks most amenable to being performed by a computer, as well as some that can only be performed with levels of computational ability beyond the human brain, reside in the clinical workflow itself. By streamlining patient experience and clinical operations, and improving patient flow through key points in the clinical experience including admissions, discharges, and ICU transfers, AI has the potential to significantly improve the efficiency of clinical care [24]. AI also extends the capability of clinical care implementation, for example, through AI’s incorporation into robotic surgical guidance systems [25, 26]. During the COVID-19 pandemic, AI has also been proposed as a real-time forecasting tool [27], and for early infection identification, monitoring, surveillance, and prevention as well as mitigation of the impact to healthcare indirectly related to COVID-19 [28].

AI is now imbedded in various day-to-day operations of many imaging departments including scheduling, image acquisition, dose reduction, image reconstruction and post-processing, prioritization for reporting, classification of findings for reporting, and the reporting task itself [1]. Beyond plugging AI into various pieces of the existing workflow, eventually molecular imaging workflows will need to be redesigned to take full advantage of AI, for example through merging data sources into a data model to enable easier data exploration and visualization [29].

2 AI’s Clinical Applications with a Focus on Molecular Imaging

From a clinical perspective, models developed using modern DL methods can, in certain circumstances, be generalized across diseases and imaging modalities and are typically less susceptible to errors in predictions secondary to noise. The interactions of various systems within a disease as well as complex dependencies of disease states on each other can be better understood with AI through DL because of its ability to aggregate multiple data streams from imaging, laboratory, genomics, proteomics, pathology, as well as data from the electronic medical record, social networks, wearable sensors, and other data sources to create integrated diagnostic systems [30]. One could envision multimodality DL modeling approaches integrating multiple data streams not only to compute disease prognosis but in every step of the diagnostic imaging workflow to involve both upstream and downstream applications. Such instances could include: planning (e.g., patient selection and scheduling based on a patient’s disease profile, previous and future interactions with the healthcare system) to scanning (e.g., reducing diagnostic study radiation dose and bettering image quality), to reading (e.g., automated detection and classification of pathologies), and reporting (e.g., automating reports with reproducible measurements, automating prediction of clinical outcomes) (Fig. 1.1).

Fig. 1.1
figure 1

Schematic representation of potentially useful AI applications along the image life cycle including potential applications in the upstream (planning through image quality control) and downstream (triage through image reporting) domains. Adapted with permission from Dr. Curt Langlotz

Full size image

2.1 Understanding Disease

More and more, the concept that diseases are a manifestation of interconnected organ systems is gaining traction. Understanding these systems and their sequalae of signs and symptoms is an area where combined molecular and anatomic imaging modalities can contribute. However, the connections within and between these systems is highly complex and models built on AI can help in pattern elucidation. For example, areas of the brain associated with specific cognitive changes typical of genetic disorders affecting the brain primarily or secondarily have been identified using machine learning approaches [31, 32]. In their study of the ability of 18F FDG PET to predict neuropsychological performance (NPP) in patients with neurofibromatosis Type 1 (NF 1), Schutze and colleagues built on the anatomical findings of MRI studies of NF 1, concluding that the accuracy in predicting NPP based on PET suggested an underlying metabolic pattern of cognitive function [32].

2.2 Diagnosis

AI is used in a plethora of diagnostic tasks using data from traditional and untraditional sources. In primary care, patients report their symptoms and concerns to chatbots that then route their care to the appropriate channel for further diagnosis or treatment [33]. Difficult diagnoses are aided by piecing together symptoms of the patient with those of millions of other patients for diagnosis [34]. Given its particularly complicated origins including genetic and environmental factors interacting with the immune system and other normal tissues in the body, cancer diagnosis is another area where AI is helping in diagnostic tasks using data from general health screening and diagnostic tests, including blood testing, imaging, and pathology [35]. In addition to the field of cancer, numerous AI applications have been developed in neurology and cardiology [36, 37].

In diagnostic imaging tasks, methods of using computational analysis to aid in the detection of lesions have evolved from early work in temporal subtraction methods and artificial neural networks performed in the 1960s–1980s to sophisticated DL methodologies of today [38, 39]. AI can recognize specific diagnoses on imaging, such as pathologic bacteria on microscopy of blood samples or findings on a radiograph [40].

In cancer detection and diagnosis, AI can facilitate the workflow efficiency and accuracy of imaging clinician expertise through precise determination of tumor volume and its change over time and tracking of multiple lesions [30]. Automated PET segmentation of nodules based on neural networks trained in the spatial and wavelet domains have been shown to be reproducible, volumetrically accurate, and demonstrate lower absolute relative error when compared to other automated techniques [41]. Other ML approaches have been useful in dealing with segmentation of larger and more complicated tumors of the head and neck, particularly in the setting of heterogeneous radiopharmaceutical uptake, in segmenting brain tumors and classifying brain scans [42,43,44]. In evaluating measures that are not typically detectable by an imaging physician, AI can help further guide additional testing and patient management [45].

The greatest strength of AI may be its ability to integrate far more factors than are possible for a single physician. For example, by analyzing images in tandem with blood and other laboratory testing, genomics, and unstructured data from patient medical records, AI algorithms are being used to make diagnoses in a more wholistic manner, decreasing the physician’s difficulty of integrating disparate results from numerous tests and the medical history [46]. This approach is known as multimodality deep learning. A recent study using this approach showed that combining clinical, pathological, and imaging information increased the predictive power of clinical outcomes in glioblastoma multiforme, where survival is poor and ranges from one to two years in most patients [47]. Another recent promising methodology in this domain could be useful for pancancer prognosis prediction using clinical data, mRNA expression data, microRNA expression data, and whole slide histopathology images [48].

2.3 Radiologic-Pathology Correlation

The power of AI over the experience of any single imaging physician or pathologist is the ability to cross-reference imaging or other data from individual tumors to databases of limitless cases for comparison, rather than limiting comparison to those cases seen over the physician’s career [30]. AI solutions for pathology have been shown to make diagnoses over tenfold faster than pathologists; while having obvious direct clinical applications, AI-based pathology has shown high value in applications in the pharmaceutical industry [49].

2.4 Characterization

Beyond connecting lesions on imaging with specific pathologic correlations, AI can assist with other areas of classification as well. For example, in neurology, Parkinson’s Disease severity can be classified with 99mTc-TRODAT-1 SPECT Imaging based on support vector machine models [50]. Several applications of AI have been published in the cancer field including detection, characterization, and monitoring response to treatment of various cancer types [30]. In the realm of hybrid imaging molecular imaging modalities such as PET or SPECT provide molecular characterization of lesions possibly seen on companion anatomic imaging, such as CT. Increasingly, work is focusing on predicting the data that would be produced by the functional modality using the traditional anatomic modality in combination with artificial intelligence. For example, uptake of 68Ga DOTATATE on PET has been used to label bone metastases as active on PET-CT, with subsequent development of AI models to predict activity using only radiomic data from the CT portion of the study [51]. This new paradigm may enable a greater global reach of the benefits of molecular imaging, allowing even those geographies that lack molecular imaging systems the ability to better characterize lesions using staple and inexpensive modalities.

AI has been key in the proliferation of the field of radiomics. Radiomic approaches aim to identify imaging phenotypes specific to diseases that can be used in their diagnosis, characterization, and treatment management, approaches that some have coined as “radiomic biopsy.” These imaging phenotypes can be defined by characteristics of the images measured or observed by an imaging specialist or features extracted based upon pre-defined statistical imaging features, of which over 5000 have been described. Such radiomic features can identify key components of tumor phenotype for multiple lesions at multiple time points over the course of treatment, potentially allowing enhanced patient stratification, prognostication, and treatment monitoring for targeted therapies, although care must be taken in evaluating the generalizability of the results of these approaches [52]. Radiogenomics, the translation of intratumoral phenotypic features to genotypes, has been most explored in cancer imaging. Making these types of correlations requires the development of new methodologies to summarize phenotypes of large heterogenous populations of cells within a single tumor and look for underlying genotypic similarities [53].

2.5 Treatment Planning

In the realm of treatment, AI has become a pillar of the concept of personalized healthcare whereby machine-based learning based on numerous and seemingly endless sources of medical data is anticipated to identify insights into patterns of disease and prognosis [54, 55]. Models predicting response based upon any given choice of therapy would greatly inform choices of drug therapy made by patients and their physicians.

In the delivery of radiation, AI has enabled dose distribution prediction for intensity-modulated treatment planning based on patient-specific geometry and prescription dose on CT of cancers of the head and neck [56, 57]. Similarly, AI models can predict radiation dose to normal organs for preemptive adjustment of technique [58]. Ideally, these types of techniques will also inform therapies based on radiopharmaceuticals as the field of theranostics grows.

In treatments involving radiation, AI has the potential to improve safety and quality. For example, in the realm of radiation oncology, DL with convolutional neural networks has been used to identify radiotherapy treatment delivery errors using patient-specific gamma images [59]. However, AI can’t be treated as a black box whose output should be trusted at face value, particularly when this output directly affects therapy. Rather, the fields of molecular theranostics and radiation therapy must recognize the fallibility of any technology that is misused with potentially significant consequences and that the workforce, including physicians, technologists, and radiation physicists, must become more conversant in various AI approaches and algorithm development [60].

2.6 Prediction of Response to Treatment

Many current pharmacologic therapies and radiotherapy approaches rely on indirect actions on disease, requiring the decoding of complicated molecular inter-relationships to define response to therapy, a task that is suited for AI. Such predictive models may require input of clinical factors; for example, one study using pretherapeutic clinical parameters to predict the outcome of 90Y radioembolization in patients with intrahepatic tumors [61]. Alternatively, models may focus on predictions made solely upon imaging, such as the prediction of radioresistant primary nasopharyngeal cancers from CT, MR, and PET imaging prior to IMRT using radiomics analysis combined with machine learning to identify the most predictive features [62]. Finally, models may rely on a combination of predictors, such as a study by Jin et al. investigating the ability to predict treatment response based on a machine learning model combining computed tomography (CT) radiomic features and dosimetric parameters for patients with esophageal cancer (EC) who underwent concurrent chemoradiation (CRT) [63].

2.7 Overall Prognosis

Evaluations of overall prognosis can be helpful to guide therapeutic choices in highly aggressive diseases or diseases that have a more chronic course with multiple therapeutic options. Just as in the case of models to predict response to therapy, overall prognostic models may incorporate clinical information, imaging data, or a spectrum of data from in vivo molecular imaging to ex vivo tissue analysis and patient characteristics.

For example, the ability to train neural network models on data from a single low-cost, widely available test such as bone scan to predict prognosis in patients with metastatic prostate cancer or breast cancer enables the development of a widely applicable prognostic model [64] By comparison, models developed from studies such as one using a combination of highly specialized inputs including 11C methionine (11C MET) PET, tumor grade, histology, and isocitrate dehydrogenase 1 R132H mutational status to predict survival in glioma patients may only be applied under very specialized circumstances [65].

2.8 Reporting

The ability to provide a timely, accurate and actionable imaging report is paramount to ensure providing quality of care and better clinical outcomes. It is known that medical image reporting errors are not rare [66, 67]. A plausible explanation for this may be the increasing number and complexity of clinical imaging studies with lagging in training of radiologist specialists, rendering attending radiologists overburdened. Hence, solutions to augment imaging specialists improve and expediate clinical reporting could be helpful (Fig. 1.2). Already an AI framework that could provide considerable benefits for patient safety and quality of care for busy emergency and trauma imaging services that press radiologists to meet the demand of increased imaging volume and provide accurate reports has been proposed [68]. Similarly, another AI framework could potentially increase threefold the measurements of target lesions in oncologic scans and provided faster notification of actionable findings to referring clinicians [69]. Likewise, a recent study by Rao and colleagues demonstrated an AI tool that can serve as peer reviewer to augment radiologists diagnosing intracranial hemorrhage and reducing error rates [70].

Fig. 1.2
figure 2

This screen displays a mockup depicting an AI automated clinical reporting workstation. On the left hand side an 18F-NaF PET/CT is displayed. Here, the back-end generates VOIs in overlapping PET/CT images. The AI system detected and segmented lesions and prompts a clinical reader to accept or deny the addition of these to the clinical report (right hand side)

Full size image

Finally, healthcare records, imaging, medical decision-making, and treatment data are now continuously recorded within the boundaries of healthcare systems in siloed fashion. In part due to this, most of current machine learning efforts in healthcare, including those in hybrid imaging and molecular imaging, are unfortunately only based on data from single institutions. AI and machine learning are inherently statistical methodologies, and as such, they benefit the most from large and heterogenous datasets, ideally from multiple institutions. Never before has the failure to build robust data-sharing systems for large-scale and near real-time analysis in healthcare has been more evident than with the outbreak of COVID-19 pandemic. Nevertheless, an international shared data model exist for information from intensive care units (ICUs): MIMIC database is such model, it is publicly available, deidentified, and widely used by investigators and engineers around the world, helping to drive research in clinical informatics, epidemiology, and machine learning [71]. Efforts like this that can enable global researchers generate AI applications that empower imaging specialists and other healthcare workers to make data-driven decisions, are sadly lacking in the hybrid and molecular imaging community. MIMIC or other established models that enable medical data sharing between institutions may serve as direction for the molecular imaging and other medical communities yet, while such endeavor could boost the task of modeling using retrospective medical data, prospective multicenter validation of developed ML models should be warrant before and during clinical deployment.

3 Conclusion

With all these abilities, the boundary between the job of AI and the role of the human imaging specialist will be debated. While some prognosticate an era of medical specialists like radiologists and pathologists being augmented by large-scale computation from AI-based applications, others foresee a future where traditional imaging physicians and pathologists cease to have a role, replaced by a physician “information specialist” trained less-so in radiology/pathology and more-so in the data sciences, statistics, and parallel fields that serve as information sources, such as genomics and proteomics [72]. Beyond its influence on medical diagnosis and therapy, AI will have effects throughout the healthcare continuum, including keeping people healthy [73]. As the role and influence of AI in healthcare continues to evolve, real and potential benefits become certain, and so far suggest AI will not replace radiologists and physicians, but radiologists and physicians who use AI will replace those who don’t [74].