Introduction

The diagnostic yield of invasive coronary angiography (ICA) has been a well-documented concern in cardiology. A major study in 2010 revealed that only 37% of patients receiving ICA were found to have stenosis of ≥ 50% [1]. Coronary CT angiography (CCTA) has emerged as a powerful tool for assessment of patients with suspected coronary artery disease (CAD) [2]. It has grown into an increasing role as a gatekeeper for referral to ICA [3] but further optimization is needed. The PROMISE trial showed that CCTA resulted in a higher rate of ICA than functional testing in symptomatic patients with suspected CAD. Furthermore, a coronary intervention was performed only half of the time despite stenosis of ≥ 50% on ICA being present in the majority of cases [4]. In order to improve CCTA’s role as a gatekeeper, a method to safely and effectively decrease the proportion of patients referred for ICA who do not receive intervention is essential.

An algorithm called the machine learning ischemia risk score (ML-IRS) has been previously developed [5]. It used automated feature selection followed by machine learning integration to predict lesion-specific ischemia defined by invasive fractional flow reserve (FFR) ≤ 0.8. The machine learning model incorporates quantitative CCTA factors and demographic information to generate a percent probability of pathological FFR on a per-vessel basis. It has been shown to improve prediction of lesion-specific ischemia over traditional measurements [5]; the ML-IRS is integrated into coronary plaque analysis research software for application to CCTA data following quantitative analysis of stenosis and plaque.

Patients receiving CCTA who are subsequently referred for cardiac catheterization, but do not receive revascularization, are a high-cost population with low yield from the invasive procedure; and it would therefore be beneficial to effectively identify these patients. The purpose of this study was to examine whether the ML-IRS from quantitative CT can predict revascularization in patients referred to ICA after CCTA, in a prospective dual-center registry.

Materials and methods

Study population

The study cohort included 352 consecutive patients (age 63 ± 10 years, 68% male) between 2013 and 2017 at two advanced cardiac care facilities within the USA (Cedars-Sinai Medical Center, Los Angeles, CA; Oklahoma Heart Institute, Tulsa, OK). The cohort selected patients who were referred for standard clinical CCTA followed by ICA due to decision by a primary treating physician within 3 months. Patients with incomplete clinical data, poor image quality, coronary calcium score (CCS) > 1000 Agatston units (AU), or prior revascularization by coronary artery bypass grafting or percutaneous coronary intervention were excluded. IRB approval was obtained at both institutions. Written informed consent was obtained for all patients. There was no overlap in subjects from any prior publications.

Clinical data including age, sex, body mass index, history of hypertension, hyperlipidemia, current tobacco use, family history of CAD, symptoms of chest pain or shortness of breath, and referral location (inpatient, emergency department, or outpatient) were obtained by database or chart review. Procedural reports from cardiac catheterization were extracted for all patients. The report was analyzed for operator-determined estimate of vessel stenosis, any intervention performed, fractional flow reserve measurement, and intravascular ultrasound measurement. Patients who received percutaneous coronary intervention or were referred for coronary artery bypass grafting as stated in the procedure report were counted as having received revascularization.

CT acquisition

CT scans were performed on multiple dual-source CT scanner platforms (Siemens Definition, Flash, and Force scanners, Siemens Healthineers), using standard clinical coronary imaging protocols as previously described [6]. In patients without contraindications, sublingual nitroglycerin was administered. Beta blockade with metoprolol was given prior to imaging with primary heart rate goal of ≤ 65 bpm. Iohexol (Omnipaque 350 mg/mL, GE Healthcare) was administered via large-bore intravenous catheter. Contrast volume (90 to 140 mL) and injection rate (5 to 9 mL/s) were adapted to body surface area and followed by saline flush. Timing bolus or automated bolus tracking at the descending aorta was used to trigger acquisition. Whole-volume image acquisition was completed in a single breath-hold.

CT analysis

Standard clinical measurement of CCS by Agatston score was performed and recorded at the time of CCTA. Quantitative image analysis was performed by two readers (A.K., over 3 years’ experience, and P.A.M., over 1.5 years’ experience with cardiac CT) who were blinded to clinical characteristics, ICA data, and previous clinical CCTA read. Plaque analysis was performed in all vessels with distal normal reference ≥ 2.0 mm using semi-automated software (Autoplaque version 2.5, Cedars-Sinai Medical Center). Readers identified regions with visible disease. Vessel lumen and wall contours were generated, followed by manual adjustment as needed. Plaque characterization was performed automatically using adaptive scan-specific attenuation thresholds. Low-density non-calcified plaque was defined as HU < 30, as described previously. Coronary segments with uncertainty in plaque presence were reviewed and adjudicated by two readers. Approximate time for analysis per case was 20 min.

Vessel parameters including maximal diameter stenosis, plaque volumes, plaque burdens, and contrast density difference (CDD) were calculated automatically after completion of quantitative semi-automated plaque analysis. Quantitative stenosis was calculated by dividing the narrowest luminal diameter by the mean of two normal non-diseased reference points. Significant stenosis was defined by quantitative diameter stenosis ≥ 70%. Plaque burden was defined as plaque volume normalized by total diseased vessel volume. CDD over a lesion was computed as follows: the luminal contrast density, defined as mean luminal attenuation per unit area, was computed automatically over 1-mm cross-sections of the arterial segment. The CDD was defined as the maximum percentage difference in contrast densities, relative to the proximal reference cross-section with no disease.

Machine learning ischemia risk score

Following plaque analysis, the ML-IRS is automatically generated as the probability (%) of lesion-specific ischemia (invasive FFR ≤ 0.8) for the left anterior descending, left circumflex, and right coronary arteries. The ML-IRS was previously developed in an independent population, using a supervised learning process by feature selection with information gain ranking and ensemble classification boosting with tenfold cross-validation to predict invasive lesion-specific ischemia. It includes factors based on quantitative CT variables (CDD, maximum quantitative stenosis, minimum luminal diameter, low-density non-calcified plaque volume, total non-calcified plaque volume, plaque length, total plaque volume, vessel volume, minimal luminal area, low-density non-calcified plaque composition, maximum area stenosis, low-density non-calcified plaque burden, myocardial mass, non-calcified plaque burden, total plaque burden, non-calcified plaque composition, and maximum remodeling index) plus age and gender of the patient obtained from the image DICOM header [5].

The primary endpoint in our study was revascularization during or as a direct referral after ICA. Secondary analysis was prediction of lesion-specific ischemia (FFR ≤ 0.80) and prediction of revascularization within the sub-cohort of patients who received invasive FFR during ICA.

Statistical analysis

Continuous variables were expressed as a mean and standard deviation when normally distributed and median and interquartile range when not. Univariable and multivariable analyses were performed on both a per-patient and per-vessel basis. Per-vessel analyses included vessel-specific quantitative plaque features, ML-IRS, and whether revascularization was performed in the vessel. Per-patient analyses included total quantitative plaque features, highest ML-IRS, and whether revascularization was performed in any vessel in the patient. Univariable analysis used two-sample Wilcoxon rank-sum tests for non-parametric values, chi-squared test for categorical variables, and Student’s t test for normally distributed numerical variables. Simple logistic regression was used for FFR subgroup analysis. Multivariable logistic regression was used to assess associations between clinical risk factors (age, sex, presence of symptoms, history of hypertension, hyperlipidemia, or diabetes, and family history of CAD), significant stenosis, patient referral location (emergency department or inpatient versus outpatient), and ML-IRS with revascularization. Optimized cutoff for ML-IRS was calculated on a per-vessel basis using the two-graph ROC analysis as the intersection of the sensitivity and specificity graphs [7]. Elements with missing data were excluded from logistic regression analysis.

Three multivariable models were created to assess the ability of the ML-IRS to improve prediction compared with other methods. Model 1 represented clinical risk assessment and included age, sex, referral location, symptoms, hypertension, hyperlipidemia, diabetes, current tobacco use, and family history. Model 2 represented a clinical risk assessment combined with traditional use of CCTA and combined the covariates in Model 1 plus the presence of significant stenosis by CCTA. In Model 3, we added ML-IRS to the covariates in Model 2 to assess whether ML-IRS could improve on traditional prediction models. ROC curves were generated for the models and AUC’s were compared. Incremental predictive value between Models 2 and 3 was also measured using continuous net reclassification improvement (NRI), as pre-existing established risk categories were not known for the ML-IRS [8, 9]. All statistical analyses were performed with Stata version 15.1. The NRI computation was performed using a SAS v9.4 module. All p values < 0.05 were considered significant.

Results

A total of 352 patients who received CCTA followed by ICA were identified and analyzed, with 1056 total vessels included in the analysis (Table 1). The mean age was 63.2 ± 10.4 years and 67.6% of the patients were male. Most patients (85.1%) presented with symptoms. The median body mass index (BMI) was 27.5 kg/m2 (24.7–31.8) and the median coronary calcium score by Agatston was 264.7 (82.0–533.1). The FFR sub-population consisted of 74 patients with 83 analyzable vessels and had similar patient demographics compared with the overall population (Table 1). Figures 1 and 2 show case examples from our patient population.

Table 1 Demographic information for total cohort and for sub-cohort of patients with fractional flow reserve
Full size table
Fig. 1
figure 1

Case example: Machine learning ischemia risk score (ML-IRS) and coronary plaque analysis in a 67-year-old male symptomatic patient undergoing coronary CT angiography in left anterior descending (LAD) artery. a Sample output of quantitative analysis and ML-IRS. b Multiplanar oblique views of LAD coronary atherosclerosis (red: non-calcified plaque, yellow: calcified plaque). Panel c Curved multiplanar reformat of LAD atherosclerosis. ML-IRS in the LAD was 60%; invasive fractional flow reserve measured in the LAD was 0.73 (ischemic)

Full size image
Fig. 2
figure 2

Case example: Machine learning ischemia risk score (ML-IRS) and coronary plaque analysis in a 57-year-old male symptomatic patient undergoing coronary CT angiography in left anterior descending (LAD) artery. a Sample output of quantitative analysis and ML-IRS. Left circumflex artery had no coronary plaque and therefore, risk was not estimated. b Multiplanar oblique views of LAD coronary atherosclerosis (red: non-calcified plaque, yellow: calcified plaque). c Curved multiplanar reformat of LAD atherosclerosis. ML-IRS in the LAD was 11%; invasive fractional flow reserve measured in the LAD was 0.83 (not ischemic)

Full size image

Per-patient univariable analysis

Revascularization was performed in 59% of the total subjects (Table 1). There was a higher proportion of male gender, symptoms, and family history of CAD in patients receiving revascularization in the total population. In the sub-population with FFR performed, there was a higher proportion of men, higher CCS, and a lower proportion of patients with diabetes. None of these differences met statistical significance and may be partially or wholly due to uncontrolled biases. There was no relation between site (Cedars-Sinai versus Oklahoma Heart Institute) and revascularization.

Per-vessel univariable analysis

In the per-vessel analysis, the ML-IRS ranged between 0 and 81% with a median of 18.7% (6.4–34.8) (Table 2). Revascularization was performed in 26% of the vessels. Univariable analysis found that vessels receiving revascularization had higher ML-IRS (33.6% (21.1–55.0) versus 13.0% (4.5–28.1), p < 0.0001). Revascularized vessels also had higher CDD, total plaque burden, non-calcified plaque burden, low-density plaque burden, and calcified plaque burden. Interquartile ranges of plaque components are noted to overlap between revascularization and non-revascularization groups despite statistical significance. This suggests that, while there are differences between groups, individual plaque features alone are insufficient to predict revascularization.

Table 2 Per-vessel plaque features in relation to revascularization in all vessels, and in the subgroup with measured FFR
Full size table

In the FFR sub-cohort (83 vessels, 74 patients), the average FFR was 0.80 ± 0.10, with an average of 0.74 ± 0.08 in vessels receiving revascularization versus 0.85 ± 0.07 in vessels not revascularized (p < 0.0001) (Table 2). Only ML-IRS and CDD were significantly different between patients who received revascularization versus those who did not in this sub-population. Overlap of interquartile range is again seen; this overlap may be reduced by sampling higher numbers. ML-IRS in patients with FFR ≤ 0.8 was 38.5% (25.6–50.6) versus 21.3% (12.9–31.8) for FFR > 0.8 (p = 0.0007 for difference). Univariable logistic regression for ML-IRS’s association with invasive FFR ≤ 0.8 was significant. Odds of FFR ≤ 0.8 was increased by 55% for every 10% increase in ML-IRS (p = 0.003). Similarly within the FFR population, odds of revascularization increased by 39% for every 10% increase in ML-IRS (p = 0.016).

Multivariable analysis

Per-vessel and per-patient logistic multivariable analyses were performed for prediction of revascularization (Table 3). Variables included were age, sex, symptoms (as defined by chest pain or shortness of breath), clinical risk factors (hypertension, hyperlipidemia, diabetes, current tobacco use, and family history), significant stenosis > 70% (present in any vessel for per-patient analysis), referral location (inpatient or emergency department versus outpatient), and ML-IRS (maximum score for per-patient analysis). In the per-vessel analysis, a 10% increase in ML-IRS resulted in a 46% increase in odds of revascularization (p < 0.0001). In the per-patient analysis, odds of revascularization increased by 20% per 10% increase in ML-IRS (p = 0.011). If significant stenosis was present, odds increased by 98% (p = 0.007) in the per-vessel and 94% (p = 0.038) in the per-patient analysis. Presence of family history of CAD increased odds of revascularization by 49% (p = 0.019) in per-vessel and 74% (p = 0.025) in per-patient analysis. Symptoms were significant in the per-vessel analysis but not in the per-patient analysis. Unexpectedly, the presence of symptoms appeared protective (OR 0.64 with symptoms, p = 0.039). Referral location was associated with an increased odds of revascularization by 76% for patients from inpatient or emergency department in the per-patient analysis (p = 0.042). Per-vessel ML-IRS had an optimized cutoff within the population at a value of 23.9%, which yielded an odds ratio for revascularization of 5.5 (4.0–7.4), a sensitivity of 73.2% and specificity of 68.0%, positive predictive value of 44.2%, and negative predictive value of 88.0%.

Table 3 Per-vessel and Per-patient multivariable analyses for predicting revascularization including clinical risk factors, significant stenosis, and Machine Learning Ischemia Risk Score (per 10% change)
Full size table

Multivariable logistic regression models to predict revascularization were compared. Model 1 (age, sex, referral location, symptoms, hypertension, hyperlipidemia, diabetes, current tobacco use, and family history) assessed clinical risk. The AUC for prediction of revascularization for Model 1 was 0.58 (0.54–0.62). Model 2 (Model 1 plus presence of significant stenosis) represented addition of traditional use of CCTA. The AUC for Model 2 was 0.69 (0.65–0.72). Model 3 (Model 2 plus ML-IRS) assessed the additional importance of the ML-IRS. The AUC for Model 3 was 0.78 (0.75–0.81). There was a significant improvement between each model (p < 0.0001) (Fig. 3). The continuous NRI for addition of ML-IRS to standard CCTA analysis (Model 3 over Model 2) was 0.636 (0.503–0.769 95% CI, p < 0.0001). This consisted of a 28% improvement in classification of patients receiving revascularization and a 36% improvement in those not receiving revascularization.

Fig. 3
figure 3

Per-vessel prediction of revascularization with ROCs from multivariable models. p < 0.0001 for difference between models. Model 1 (yellow): age, sex, symptoms, hypertension, hyperlipidemia, current smoking, diabetes, family history, and referral location; Model 2 (green): Model 1 + significant stenosis; and Model 3 (blue): Model 2 + ML-IRS

Full size image

Discussion

In this study, ML-IRS was significantly associated with revascularization in both univariable and multivariable analyses. Prediction of revascularization significantly improved when the ML-IRS was added to a model with clinical information and CCTA analysis including stenosis. This suggests that this on-site machine learning–based tool may be able to improve current practice patterns and predict which patients would benefit the most from referral to ICA. The continuous NRI measure also showed significant improvement with the integration of the ML-IRS with traditional methods of risk assessment; notably, there was a 36% improvement in predicting patients not receiving revascularization, indicating that addition of ML-IRS may help patients avoid unnecessary cardiac catheterization.

In our population, only patients who were referred for ICA after CCTA were included. This is a high-risk population, as shown by the high rate of revascularization. Expansion to a lower risk population of patients receiving CCTA may improve ability to avoid unnecessary referral. The analysis also showed low predictive ability for revascularization using traditional risk factors and symptoms. Presence of symptoms showed inverse association with revascularization, potentially due to the overall high prevalence of symptoms or patients without acute symptoms being specifically referred for revascularization as outpatients. The analysis of the FFR sub-population helps explain the method by which predictive ability may be applied to the general population. The ML-IRS was internally validated in the NXT trial population to predict invasive FFR [5], which was also consistent in a subset of our dual-center population. While invasive FFR may be the gold standard for assessing physiological significance of a coronary stenosis, in our study design, we recognize that the cost, need for medication administration, and logistics of performing FFR in all patients lead to a minority of patients receiving invasive assessment. Our study was structured to assess the patient-centered endpoint of revascularization: the clinical question facing patients with stenoses as well as their healthcare providers at the point of care. ML-IRS was more strongly associated with FFR than revascularization, which suggests that, if invasive FFR had been used more broadly, revascularization rate may have been even more consistent with our predictions.

There have been multiple other approaches to artificial intelligence–based prediction of invasive FFR [10,11,12,13,14], including commercially available noninvasive FFR (FFRct). FFRct uses computational fluid dynamics in a 3-D model of the coronary tree in order to provide an accurate prediction of invasive FFR from CCTA [10, 15]. While this approach has been successful, it incurs significant cost and the need to send studies to an external location for analysis. Beyond luminal measurements alone, CTA also allows measurement of other features such as quantitative coronary plaque characteristics [16, 17] and CDD [5], which are all included in the ML-IRS. Machine learning–based approaches have utilized quantitative CCTA to predict various outcomes including myocardial perfusion by PET [18] or CT [19] or invasive FFR [14, 20]. While these approaches have typically shown strong testing characteristics with relatively low processing times, to our knowledge, our study is the first to use a machine learning–based approach to predict revascularization. With Youden’s index cutoff of 23.9%, the ML-IRS exhibited strong negative predictive value of 88.0%, despite the studied population being clinically challenging enough that all patients included received ICA after CCTA. This further suggests that use of the ML-IRS may help patients avoid unnecessary cardiac catheterization.

There are limitations in our study. Our population of patients receiving ICA after CCTA results in selection bias, and results may be different in all patients receiving CCTA. Additionally, only a portion of patients (21%) received invasive FFR. This limits the interpretability within the FFR sub-population. We note that our FFR utilization is consistent with real-world utilization of FFR in intermediate lesions [21]. FFR utilization in all patients may help support the efficacy of the algorithm; however, this may not reflect “real-world” practice of revascularization as shown in our study [5]. The primary endpoint of revascularization does not capture long-term patient hard outcomes, which were not available to us at both sites. We did not directly compare our results to FFRct, since it was not available for all patients and could not be added for research purposes only. We recognize that this represents a separate, validated approach. While hypothesis-generating, our findings cannot make definitive statements regarding the ability of the ML-IRS to improve economic or clinical outcomes without prospective clinical studies.

In conclusion, ML-IRS from quantitative coronary CT angiography improved the prediction of future revascularization and can potentially identify patients likely to receive revascularization if referred to cardiac catheterization. This ML score is correlated with invasive FFR measurements providing dual-center external validation and improves on clinical risk prediction models.