FormalPara Clinician’s capsule

What is known about the topic?

Clinical decision support has been proposed as an effective intervention to facilitate evidence-based diagnostic imaging.

What did this study ask?

Can electronic decision support improve CT utilization and yield for ED patients with head injuries and suspected pulmonary embolism?

What did this study find?

Decision support effectiveness was limited by low physician engagement and did not significantly impact CT utilization or diagnostic yield.

Why does this study matter to clinicians?

While decision support can address many barriers to evidence-based imaging, effecting meaningful practice change requires demonstrating value to clinicians.

Introduction

Computed tomography (CT) is an essential diagnostic tool; however, utilization has grown rapidly over the past two decades [1, 2], leading to concerns about cost [3], radiation exposure and consequent cancer risk [4, 5], overdiagnosis of clinically insignificant disease [6], incidental findings leading to unnecessary testing and treatment [7], and variation among physicians [8, 9], suggesting opportunities for practice improvement exist. Consequently, evidence-based clinical decision rules have been developed to avoid unnecessary imaging for low-risk patients.

Two clinical scenarios for which validated decision rules exist to guide CT imaging are mild traumatic brain injury [10,11,12] and suspected pulmonary embolism [13,14,15,16], and Canadian and American Choosing Wisely recommendations suggest avoiding CT imaging for patients identified as low risk by these rules [17, 18]. However, despite robust evidence supporting their use, implementation efforts have met with limited success [19]. Commonly cited barriers include difficulty in remembering and applying decision rules, threats to physician autonomy, workflow concerns, uncertainty with regard to patient outcomes, patient/colleague expectations and potential medicolegal risks [20, 21]. While electronic clinical decision support has the potential to address many of these concerns; prior implementations have demonstrated modest reductions in CT utilization at best [22,23,24,25,26,27,28,29,30]. Moreover, the bulk of these studies have used non-randomized designs which are prone to bias and influence of temporal trends, tempering the conclusions that can be drawn from them [31].

This study used a randomized design to determine if electronic decision support can reduce unnecessary CT imaging for low-risk emergency department (ED) patients with head injuries or suspected pulmonary embolism. Our primary outcome was CT utilization, as defined by the proportion of eligible patients with CT performed. We hypothesized that the intervention would result in clinically meaningful absolute decreases in CT utilization of at least 5% (11% relative decrease) among head injury patients and 1% (15% relative decrease) among patients with suspected pulmonary embolism.

Methods

Study design, setting and time period

This prospective multicenter cluster-randomized controlled trial was conducted at five urban acute care sites (one level one trauma center, three community EDs and one urgent care center) in Calgary, Alberta, Canada from August 1, 2014 to August 30, 2017. Combined patient visits exceeded 390,000 annually during the study period. This study was approved by the University of Calgary Conjoint Health Research Ethics Board (REB14-0650) with a waiver of consent for physician participation and collection of patient-level data, was registered on ClinicalTrials.gov (NCT02410941) and was funded by a competitive research grant from Alberta Innovates Health Solutions.

Population

All emergency staff physicians at the five study sites were eligible for inclusion in the study. Physicians were excluded if they were trainees or had less than 10 eligible patient encounters in either the baseline or intervention periods (to ensure adequate sample size for data analysis and opportunity for exposure to decision support). Patient encounters were eligible for inclusion in the study if they were 18 years or older and attended one of the five study sites with a Canadian Triage and Acuity Scale (CTAS) 2–5 [32]. CTAS 1 (highest acuity) encounters were excluded given the higher likelihood of requiring emergent CT imaging and to avoid any inadvertent delays in care for critically ill patients. Encounters with a Canadian Emergency Department Information Systems (CEDIS) [33] chief complaint of “head injury” were eligible for the head injury analysis and encounters with chief complaints of “shortness of breath” or “chest pain” were eligible for the suspected pulmonary embolism analysis. These complaints were chosen to avoid inappropriate decision support triggers for other clinical scenarios (e.g., CT head to rule out stroke, D-dimer to rule out isolated deep venous thrombosis).

Intervention

A balanced incomplete block cluster-randomized design with physicians as the unit of randomization was employed. All physicians at the five study sites were randomized in a 1:1 ratio to receive decision support for either CT imaging of patients with head injury or suspected pulmonary embolism. Decision support tools were developed with a multidisciplinary team in accord with best practices [34, 35] and refined based on end-user testing. Head injury decision support automatically opened in a pop-up window whenever a non-contrast CT head was ordered for an eligible patient, prompted the user to complete the Canadian CT Head Rule, then provided an imaging recommendation, quantitative risk estimate of 7-day neurosurgical intervention and patient handout. Decision support for suspected pulmonary embolism triggered whenever a D-dimer or CT pulmonary angiogram was ordered for an eligible patient. The tool confirmed the indication for the test and prompted the physician to complete the 2-level Wells Score, Pulmonary Embolism Rule-out Criteria (PERC) and/or age-adjusted D-dimer as clinically indicated, then provided a quantitative estimate of 90-day venous thromboembolism risk, imaging recommendation and patient handout (Fig. 1). Performance reports quantifying individual physicians’ CT utilization rates and decision support adherence compared against their anonymized peers were distributed quarterly.

Fig. 1
figure 1

Screenshot of decision support for suspected pulmonary embolism

Full size image

Outcome measures

Eligible patient encounters and CT utilization were identified from administrative data for the 2 years prior to the intervention (to provide baseline data) and for the 1-year following implementation. Patient demographics including age, sex, CTAS score, mode of arrival, ED length of stay, disposition, CT imaging, return ED visits and admissions were also retrieved from administrative data. An index visit was defined as the first visit for an eligible patient during the intervention period. Return visits were defined as a repeat ED visit for any reason during the follow-up period (30 days for head injury and 90 days for suspected pulmonary embolism patients), in accord with prior literature.

The primary outcome of CT utilization by physicians, defined as the proportion of eligible encounters with either non-contrast head CT (head injury arm) or CT pulmonary angiography (suspected pulmonary embolism arm) performed on the index visit, was calculated from administrative data. For the suspected pulmonary embolism analysis, the secondary outcome of diagnostic yield (proportion of studies diagnostic for acute pulmonary embolism) was calculated based on manual review of diagnostic imaging reports by a trained research assistant blinded to the randomization status of the ordering clinician, with physician adjudication of indeterminate cases.

Data analysis

A generalized linear mixed effects model (binary distribution, logit link function) was used to assess whether CT utilization or diagnostic yield differed between intervention and control groups. The unit of analysis was the individual patient. We used the constrained baseline approach of Hooper et al. [36] to test the between-arm difference at post-intervention, controlling for baseline by including fixed effects for time (pre- and post-intervention), and group by time interaction, and adjusting for pre-specified patient covariates: age and sex. As the trial included all available physicians, we determined the detectable difference for each of the two primary trial outcomes, using the method of Teerenstra [37]. Because the Canadian CT head rule considers all head injury patients aged 65 years and older high risk, a pre-specified subgroup analysis was performed for patients aged 18–64 years and ≥ 65 years. Coefficients of variation and intracluster correlation coefficients were calculated to assess if there was any impact on inter-physician variability for CT utilization. Secondary outcomes were tabulated descriptively for the intervention period.

Sample size

For the head injury analysis, we determined that 100 physicians in each arm with an average of 25 patients per physician would achieve 80% power to detect a clinically meaningful 5% absolute (11% relative) CT utilization reduction from the baseline rate of ~ 42%, assuming an intracluster correlation coefficient of 0.1, a correlation with baseline of 0.8, and using a two-sided significance level of 5%. We applied a cluster size coefficient of variation of 50% to account for variability in cluster sizes across physicians. For suspected pulmonary embolism, 100 physicians in each arm with an average of 150 patients per physician would achieve 80% power to detect a clinically meaningful 1% absolute (15% relative) CT utilization reduction from the baseline rate of ~ 6.5%, assuming an intracluster correlation coefficient of 0.02, a correlation with baseline of 0.8, a coefficient of cluster size variation of 50%, and using a two-sided significance level of 5%. All statistical analyses were conducted using SAS v9.4 (SAS Institute Inc., Cary, NC, USA).

Results

A total of 225 physicians were randomized (109 to head injury and 116 to suspected pulmonary embolism interventions) during the study period. Data were collected for a 2-year baseline period prior to the study (August 1, 2014–August 1, 2016) and for 1-year following implementation (August 2, 2016–August 30, 2017). A total of 20 physicians were excluded because they had less than 10 eligible patient encounters, leaving 104 physicians randomized to head injury and 101 randomized to the suspected pulmonary embolism interventions available for the analysis. Patient demographics in the post-intervention period were similar between intervention and control groups (Table 1).

Table 1 Patient characteristics—Post-intervention period only
Full size table

Head injury

Intervention physicians triggered decision support 1071 times, and voluntarily interacted with it 407 times (38.0%). CT head utilization was similar for both intervention and control physicians in the baseline (41.5% vs 42.9%) and post-intervention periods (39.8% vs 42.7%), respectively, (Table 2; Fig. 2) and the between-arm difference in the intervention period was not statistically significant in the adjusted mixed effects model (OR 0.91, 95% CI 0.74–1.08, p = 0.31).

Table 2 CT utilization and diagnostic yield by clinical scenario
Full size table
Fig. 2
figure 2

CT utilization and CT pulmonary angiography diagnostic yield by month, pre- and post-decision support implementation

Full size image

The stratified subgroup analysis showed that for patients aged 18–64 years, CT head utilization was similar for both intervention and control physicians in the baseline (31.8% vs 32.4%) and post-intervention periods (28.3% vs 30.7%), respectively, (Table 2), and the between-arm difference in the intervention period was not statistically significant (adjusted OR 0.91, 95% CI 0.76–1.11). For patients aged 65-years and older, results were similar for intervention and control physicians in the baseline (73.4% vs 76.2%) and post-intervention periods (71.7% vs 73.7%), respectively, and the between-arm difference in the intervention period was not statistically significant (adjusted OR 0.94, 95% CI 0.69–1.27, p = 0.74).

Suspected pulmonary embolism

Intervention physicians triggered decision support 1829 times, and voluntarily interacted with it 823 times (45.0%). Of 32,144 eligible patient encounters during the intervention period, CT pulmonary angiography was performed in 1995 (6.2%) of cases. Utilization for eligible patients was similar between intervention and control physicians in the baseline period (6.3% vs 6.5%) and in the post-intervention period (6.2% vs 6.2%) respectively (Table 2; Fig. 2). No statistically significant difference in utilization between groups in the intervention period was noted in the mixed effects model (OR 0.98, 95% CI 0.87–1.11, p = 0.74). Of 1995 CT pulmonary angiography studies performed during the intervention period, 354 (17.7%) were diagnostic for acute pulmonary embolism. Diagnostic yield was similar between intervention and control physicians in both the baseline (from 18.5% vs 17.1%) and post-intervention periods (19.5% vs 16.1%), respectively, and the between-arm difference in the intervention period was not statistically significant (OR 1.23, 95% CI 0.96–1.65, p = 0.10).

Secondary outcomes

Patient outcomes are presented descriptively (Table 3). There were no clinically meaningful differences noted between intervention and control groups for any of the variables measured. Between-physician measures of variability were similar in both the baseline and post-intervention periods (head CT utilization: coefficient of variation 0.426 vs 0.455; intracluster correlation coefficient 0.094 vs 0.097; CT pulmonary angiography utilization: coefficient of variation 0.692 vs 0.706, intracluster correlation coefficient 0.02 in both periods).

Table 3 Patient outcomes (intervention period only)
Full size table

Discussion

Interpretation of findings

To our knowledge, this is the first randomized multicenter trial examining the impact of electronic decision support on CT utilization in the ED, and is important because the bulk of prior studies have used non-randomized designs prone to bias and confounding. Moreover, the comprehensive intervention addressed most of the commonly cited barriers to evidence-based imaging. Unfortunately, we were unable to demonstrate a statistically significant impact of decision support on CT utilization and diagnostic yield, although point estimates suggested a weak signal of effect. While we considered the possibility that increased utilization among low-users could have offset reduced utilization among high-users and concealed a benefit signal, we did not detect any reduction in inter-physician variability in our analysis. The limited effectiveness of the intervention is likely owing to low rates of voluntary physician interaction which appeared to attenuate further over time. These findings imply that simple provision of decision support is insufficient to meaningfully change imaging practices, and that future interventions must be perceived by clinicians as adding value to their practice to be successful.

Comparison to previous studies

While it is difficult to directly compare our CT pulmonary angiography utilization data with prior literature because of differing denominators used, the 17.7% overall diagnostic yield observed here is almost triple the 6.3% yield reported in a recent United States sample of over 300 hospital regions [38]. Similarly, head CT utilization appeared to be lower in this cohort, where only 43% of head injury patients underwent CT compared to 63–76% in prior studies [19]. These findings suggest that selective CT imaging was already being used prior to the intervention, and may have limited the potential for further improvements.

Strengths and limitations

The randomized multicentre design and comprehensive intervention are major strengths of this study. The primary limitation is the voluntary nature of the intervention, permitting physicians to neither interact with the decision support, nor follow its recommendation. Owing to technical limitations, decision support could not be integrated directly into the order entry software and opened in an external window which could be ignored by the physician. However, mandatory interaction has limitations. In fact, a prior study of decision support in the ED demonstrated that almost 10% of data entered by clinicians was erroneous, raising the possibility that clinicians were “gaming” mandatory data entry to avoid intrusive alerts or avoid appearing non-compliant with institutional guidelines [39]. In addition, the implementation of decision support at the time of CT ordering (when an imaging decision has already been made), likely further limited its impact. Finally, it is possible that because the decision rules used in the study have demonstrated limited impact in prior real-world implementations [19, 26], newer simplified tools such as the YEARS criteria [40] may be more easily incorporated into practice.

Clinical implications

These findings suggest that while decision support may have a role in facilitating evidence-based imaging, future interventions must demonstrate value for physicians to be widely adopted. Examples may include simplified tools integrated into charting to decrease cognitive load and improve documentation efficiency during patient assessment and/or incentives for adherence to evidence-based guidelines. Direct consultation with clinicians is recommended to design interventions that would be perceived as helpful, and determine how best to implement them in the local practice setting.

Research implications

Future efforts to facilitate evidence-based imaging might begin with end-user consultations and focus on upstream interventions that improve physician workflow, implementation in settings where opportunity for improvement is greatest and possibly incentives for evidence-based practice.

Conclusion

This randomized study used a comprehensive decision support intervention to address many of the previously cited barriers to evidence-based CT imaging, but was limited by low levels of participation and was unable to significantly impact CT utilization or diagnostic yield. Understanding local needs through direct consultation and demonstrating value to clinicians through interventions that improve workflow are likely necessary to effect meaningful practice change.