Abstract
Background
Clinical decision support (CDS) is a promising tool for reducing antibiotic prescribing for acute respiratory infections (ARIs).
Objective
To assess the impact of previously effective CDS on antibiotic-prescribing rates for ARIs when adapted and implemented in diverse primary care settings.
Design
Cluster randomized clinical trial (RCT) implementing a CDS tool designed to guide evidence-based evaluation and treatment of streptococcal pharyngitis and pneumonia.
Setting
Two large academic health system primary care networks with a mix of providers.
Participants
All primary care practices within each health system were invited. All providers within participating clinic were considered a participant. Practices were randomized selection to a control or intervention group.
Interventions
Intervention practice providers had access to an integrated clinical prediction rule (iCPR) system designed to determine the risk of bacterial infection from reason for visit of sore throat, cough, or upper respiratory infection and guide evidence-based evaluation and treatment.
Main Outcome(s)
Change in overall antibiotic prescription rates.
Measure(s)
Frequency, rates, and type of antibiotics prescribed in intervention and controls groups.
Results
33 primary care practices participated with 541 providers and 100,573 patient visits. Intervention providers completed the tool in 6.9% of eligible visits. Antibiotics were prescribed in 35% and 36% of intervention and control visits, respectively, showing no statistically significant difference. There were also no differences in rates of orders for rapid streptococcal tests (RR, 0.94; P = 0.11) or chest X-rays (RR, 1.01; P = 0.999) between groups.
Conclusions
The iCPR tool was not effective in reducing antibiotic prescription rates for upper respiratory infections in diverse primary care settings. This has implications for the generalizability of CDS tools as they are adapted to heterogeneous clinical contexts.
Trial Registration
Clinicaltrials.gov (NCT02534987). Registered August 26, 2015 at https://clinicaltrials.gov
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
INTRODUCTION
Despite decades of research, inappropriate prescribing rates for acute respiratory complaints remain high. For example, half of all antibiotic prescriptions for acute respiratory infections (ARIs) are inappropriate1; 10% of sore throats are estimated to be bacterial, but antibiotic-prescribing rates have remained at 60% since 20002; and 71% of acute bronchitis cases are prescribed antibiotics, despite repeated guidelines advising against this3.
Many studies have attempted to use clinical decision support (CDS) tools within the electronic health record (EHR) to reduce antibiotic-prescribing ARIs with mixed success.4, 5 One cluster randomized trial observed a significant decline in antibiotic prescribing for acute bronchitis with a CDS tool (from 74 to 61%, P = 0.01).6 Two other cluster randomized trials evaluating two different CDS tools in primary care practices found no change in antibiotic prescribing with low adoption of the intervention by providers.7, 8 Another study reported a 10% increase in antibiotic prescribing after implementing a CDS, despite a 19% decrease in antibiotic prescribing among the small number of visits where the CDS was used.9 Inappropriate antibiotic prescription was lower where primary care sites used peer comparison and accountability.10 These mixed results suggest that CDS tools’ ability to reduce inappropriate antibiotic prescribing is constrained by clinicians’ variable usage.
We previously developed and tested a novel tool with two integrated clinical predication rules (iCPRs) to reduce antibiotic prescribing for ARIs in primary care for the iCPR1 study. Specifically, the two iCPRs were the rule by Heckerling et al. for pneumonia and the rule by Centor et al. for streptococcal pharyngitis.11 The iCPRs were embedded directly into the EHR to help clinicians identify patients most likely to have a bacterial infection and increase appropriate antibiotic-prescribing rates. The study observed a 35% reduction in antibiotic prescribing for relevant ARIs. These results were supported by high adoption of the tool (58%).11 However, the study was limited by a single site design.
To assess the generalizability of the original findings in iCPR1, the tool was adapted and implemented across diverse primary care settings; effectiveness was tested in these new contexts12. Our approach promoted provider adoption of this complex CDS at the point of care by seamlessly blending CDS into provider workflow.
The iCPR2 study consisted of 2 phases. Phase 1 leveraged user-centered design and usability techniques used in the original CDS tool development to adapt to diverse workflows across the sites (as described in prior publication11).
This paper reports on phase 2, where we conducted a cluster randomized clinical trial (RCT) studying (1) adoption of the tool for two target conditions and (2) effects of the new CDS tool on provider diagnostic and treatment patterns.
METHODS
We evaluated the effectiveness of the user-centered design adapted iCPR tool in combination with academic detailing versus academic detailing alone to reduce antibiotic-prescribing rates among intervention practices. The study process was evaluated using the RE-AIM framework described below.
Setting
The institutional review board at all participating sites approved the study protocols. The CDS RCT was conducted within two large academic health system primary care networks in Wisconsin and Utah with a mix of internal and family medicine providers (attending physicians, residents, fellows, physician assistants, and nurse practitioners).
Exclusion and Inclusion Criteria
All internal medicine and family medicine primary care practices within each health system were invited to participate. If consent was provided by practice leadership, all providers were considered participants. The single site used in iCPR1 was excluded.
Study Design
Primary care practices were randomized using a random number generator to a control group (usual care + academic detailing) or an intervention group (iCPR2 CDS tool + academic detailing) (see Appendix A in the ESM).
Academic Detailing
Providers in both arms received a 30–45-minacademic detailing session during a regular practice meeting. Academic detailing included a review of the evidence and appropriate use of the Centor (McIsaac for children) and Heckerling CPRs used in the study, discussion of evidence-based diagnosis of streptococcal pharyngitis and pneumonia, and review of standard guidelines for treating the two diseases. Participants received handouts with the CPRs and treatment guidelines and links to online resources. Intervention providers received a live demonstration of the iCPR tool embedded in the EHR. Providers who were unable to attend were given access to printed and online training materials.
Adaptation of CDS Workflow
In the original CDS workflow for iCPR1, the tool was triggered by a matching reason for visit, diagnosis, or diagnosis in addition to antibiotic order related to an acute respiratory tract infection (Fig. 1); the participant then received an alert that, if accepted, would reveal the risk calculator.12 Following calculator completion, the participant was provided a link in an additional alert to bundled order sets that included diagnosis and next steps.12 Based on usability testing, the CDS tool was adapted for distinct workflows at each site (Fig. 1).13 There was a strong desire by providers to have the tool trigger earlier in their workflow. Therefore, the CDS tool triggered first upon entry of the reason for visit (cough, sore throat, URI) in the EHR and included a second trigger upon diagnosis entry (as done in the original iCPR1 study).14 The iCPR2 triggers are detailed in Appendix B in the ESM. We activated the CDS tool for providers in the intervention practices from October 22, 2015 through May, 2016, based on timing of academic detailing and collected data until June 30, 2018.
CDS Scoring and Recommendations
Based on chief compliant, the CDS prompted providers to complete an iCPR risk score calculator and gave management recommendations and bundled order sets with documentation based on the score. CPRs and recommendations for each score based on the risk of streptococcal pharyngitis and pneumonia are described in Table 1. Patient encounters with the study providers were monitored, and data were collected from the EHR for the intervention period. If dismissed, the iCPR alert would not appear again during that visit; providers could retrieve the iCPR order set anytime from the order entry workflow.
Outcomes
Provider demographics (age, gender, degree, training level, and specialty) were collected from available EHR data. Patient characteristics (age, gender, and race) were selected for analysis based on availability in the EHR and their potential to influence provider decisions.
Process Outcomes
This study process and measures of implementation are adapted from the RE-AIM framework12 and have been modified to reflect a clinical context. Reach pertains to percentages of participating primary care clinics at each site. Efficacy pertains to differences in antibiotic prescription rates between intervention and control groups as discussed in primary outcome. Adoption pertains to utilization rates of the tool in the intervention group for eligible visits in which (1) the provider opened the tool after it was triggered and (2) the provider completed the risk score calculator embedded in the tool (see Appendix C in the ESM). Implementation pertains to the fidelity of use of the intervention tools based on the rate at which the provider signed the associated bundled order set (see Appendix C in the ESM). Maintenance pertains to The comparison of adoption and efficacy trends between years 1 and 2 (see Appendix D in the ESM).
Primary Outcome
Efficacy of the tool was assessed based on the differences in provider antibiotic ordering. Antibiotic orders were evaluated by examining EHR data from patient encounters where patients had a visit reason that triggered the tool in the intervention group or would have triggered the tool in the control group. Positive antibiotic orders were identified as any antibiotic prescribed during the encounter. We measured the frequency, rates, and type of antibiotics prescribed for eligible encounters.
Secondary Outcomes
A secondary outcome of inappropriate antibiotic prescribing was calculated using a list of previously published diagnoses considered appropriate to prescribe antibiotics for an ARI.10 Any oral antibiotic prescribed during an iCPR ARI encounter that was not connected with an appropriate visit diagnosis was considered inappropriate. Antibiotic prescription appropriateness was considered a secondary outcome to be consistent with the original iCPR1 primary outcome (overall antibiotic prescribing), and because its determination of inappropriateness is dependent on EHR diagnosis coding which has reliability issues. Additional secondary outcomes measured were order frequencies of chest radiographs, rapid streptococcal tests, and pharyngitis throat cultures during eligible encounters.
Statistical Analysis
Differences in the characteristics of patients seen by providers in the two arms were measured using t test and χ2 test, as appropriate. However, due to a large amount of skewness, the number of patients per provider was compared with the Mann-Whitney U test. Patient characteristics were compared using similar methods. Process and primary and secondary outcomes were compared for all intervention and control visits and were stratified by pneumonia or streptococcal iCPR tool. Absolute risk (risk difference) and relative risk (RR) ratios of antibiotic prescribing in the intervention group versus the control group were compared using a generalized estimating equation model (with a logit link) with robust standard errors to account for clustering of patient visits (encounters) within practices and subsequent correlated outcomes. The outcomes were further stratified by primary care networks in Wisconsin and Utah in Appendix E in the ESM. Because the median age of patients differed in the intervention and control groups, we performed a secondary analysis adding age as a covariate in our age-adjusted models. Analyses were conducted with the R statistical software (R version 3.5.2) using 2-sided P values.
RESULTS
From October 22, 2015, through June 30, 2018, 33 primary care practices with 541 providers participated in the study. Two practices declined to be part of the study. There were no statistically significant differences in demographics between providers in the intervention and control groups. Over 100,573 patient visits occurred during the study period. Patient age, sex, race, and number of diagnosis recorded were statistically different but were small and likely clinically insignificant (Table 2).
Process Outcomes
Regarding reach, 94% of the practices approached to be part of the study in the two institutions participated (33 out of 35). The CDS tool was triggered 42,126 times among intervention practices (Table 3). This tool had low adoption rates, with providers completing the calculator in 6.9% of triggered encounters. Low utilization rates were observed in both health systems (Wisconsin 9.6%, Utah 2.5%) for strep (7.9%) and pneumonia (6.3%). Regarding implementation, low calculator completion rates were complemented by low rates of bundled order set completion overall (0.9%) and at each site (Wisconsin 1.3%, Utah 0.1%). Regarding maintenance, Appendix D in the ESM shows rates of antibiotic prescription from 0 to 33 months, where the completion rate of iCPR calculator decreased steadily over time.
Primary Outcomes
Regarding efficacy, there were no statistically significant differences in overall antibiotic prescriptions rates (Table 4), rapid streptococcal test orders, or chest X-ray orders between the control and intervention sites (Table 5). Providers in the intervention group were equally likely to order antibiotics (age-adjusted RR, 0.99; 95% CI, 0.87–1.11; P = 0.78) (Table 4).
Secondary Outcomes
There were no differences in the rates of total inappropriate antibiotic prescribing for ARIs between intervention and control sites (age-adjusted RR, 0.9; 95% CI, 0.7–1.2; P = 0.59; see Appendix E in the ESM). When stratified by pneumonia and pharyngitis, the RR of antibiotic ordering remained insignificant for both (age-adjusted RR, 0.1.0 and 0.6; 95% CI, 0.8–1.2, 0.3–1.1; P = 0.72, 0.08, respectively). The intervention group was not less likely to order rapid streptococcal tests in sore throat encounters or chest radiographs in cough encounters (Table 5).
DISCUSSION
Our study found no evidence for an effect of the CDS system along with academic detailing on antibiotic prescribing for visits that triggered the tool (sore throat, cough, URI) in diverse primary care settings. The results were robust to analyses stratified by practice, provider, and provider nested within practice. This lack of efficacy may be driven by the very low adoption and utilization rates of the tool at all clinical sites. While contrary to the results from our previous trial, our results are consistent with several CDS trials targeting antibiotic ordering in ARIs.6, 8
These results are surprising given the integrated use of multiple user-centered design methods grounded in relevant implementation frameworks. The study comprehensively assessed provider workflows and derived insights through iterative usability testing and adaption phase.12 Careful attention was paid to ensuring leadership buy-in from the participating practices. When lower than expected rates of use of the tool became apparent, we conducted repeated assessments of barriers at the sites and implemented several adjustments over time. Despite these efforts, the low adoption rates highlight several important barriers that we were unable to overcome, detailed in Mann et al.14
The first was the rise in usage of CDS and in particular of clinical alerts since the original iCPR study. At most institutions using a major EHR vendor system, there are now often hundreds of alerts targeting primary care providers.15 In contrast, during the original iCPR study at a large academic health center in 2009, there was only a handful of alerts (interruptive or non-interruptive) built to interact with primary care providers on a daily basis (not including drug interaction alerts). This change in the context of CDS and the well-documented rise in “alert fatigue” are likely contributors to the low adoption rate in this study.16, 17
In addition, while our implementation of user-centered design sought to iteratively tailor our CDS tool to the diverse primary care workflows we encountered, we hypothesize that this potentially caused some “over engineering” that may have contributed to low adoption rates. In order to foster provider and leadership support, we adapted our previously used tool to fit new workflows. For example, some providers used their medical assistants to “queue up” their documentation, which bypassed the typical trigger points for the iCPR2 tool. During usability testing, changes were made to the tool at sites using this workflow to insert the tool into a more amenable location, one that was not disruptive but may have been too passive and simply blended into the background—never having a chance to influence provider behavior.
Moreover, while our results likely represent a lack of engagement with CDS (suboptimal design, workflow barriers, etc.), they also suggest that, despite being supportive of the study and having tailored the tool to their preferences, providers may not have derived sufficient clinical relevance from the tool to warrant interaction. Low clinical relevance is a leading cause of CDS “ignoring” and fatigue.18 Providers may have felt they did not need “support” to appropriately prescribe antibiotics. This attitude is a well-known barrier to changing provider antibiotic-prescribing behavior and has been attributed to patient-, provider-, and system-level factors.5, 19 Issues such as patient expectations for antibiotics, time pressures, diagnostic uncertainty, provider overconfidence in the accuracy of their diagnosis, and skepticism towards the true risk of overprescribing continue to undermine efforts to reduce inappropriate prescribing.19, 20
The literature on CDS adoption rates is variable, reflecting the heterogeneity of designs, interventions, and the fast-changing environment within which CDS systems are deployed. Prior investigations of CDS achieved a wide range of adoption rates from less than 10% to more than 60% depending on the tool and context.4, 6, 8, 11, 21 Our results are similar to prior studies showing adoption rates of less than 10% with CDS tools targeting inappropriate antibiotic prescribing for ARIs.6, 8 For example, a similar study seeking to reduce antibiotic prescriptions for URIs was used in only 7% of eligible visits.6
Limitations
Our study had several limitations. Our use of an adaptive trial design allowed accommodation of diverse workflows, sizes, and other characteristics of the sites—but created challenges to directly comparing and understanding our results in relation to the original iCPR study. Secondary analyses identified small differences in antibiotic prescribing at the site level but additional analyses using a pre-post measure of prescribing determined that these were due to a random imbalance of baseline antibiotic prescribing. This, alongside the academic nature of the sites, limits the generalizability of our results; our observations show the capabilities of this tool, in this context, with our implementation approach. While overall utilization was low, there was slightly more utilization at the University of Wisconsin (UW). Interviews with participating providers at each institution revealed no difference in the reasons of use of CDS between sites. We also acknowledge that there may be significant provider-to-provider variation in the use of the CDS tools (though the observed overall usage rate was quite low). We are currently investigating these potential individual variations in tool adoption.
We also did not include pediatric sites, which may have altered our results for this age group. Although our background rate of antibiotic prescribing was 45% (cough/URI at 47% and sore throat at 39%), which is a little lower than the national CDC estimates of 50%1, these rates were similar to the iCPR1 study. Although we did measure and observe no changes in follow-up visits to primary care, the emergency room or the hospital after iCPR encounters, we did not perform a manual chart review of follow-up visits to corroborate visit diagnoses. However, our prior studies had established the safety of the iCPR CDS, and the implemented tools are considered best practices11.
CONCLUSION
In conclusion, we demonstrated that a CDS tool iteratively adapted to diverse primary care workflows did not significantly reduce antibiotic prescribing for ARIs. This contradicted our previous single site study. Failure to change provider behavior was likely driven by low usage rates of the CDS tool; a phenomenon that may have been affected by system factors like alert fatigue and complexity of EHR workflows, patient factors such as patient expectations for antibiotics, and multiple provider factors. The observed drop in engagement with the CDS tools between studies suggests that the mixed results of other studies in this field may be in part due to the changing healthcare delivery context. Future CDS tools targeting antibiotic prescribing for ARIs will need to overcome these barriers to change provider-prescribing behavior and reduce inappropriate antibiotic prescribing. Potential next steps may include pivoting the target of the CDS from providers to other members of the care team who are less burdened by the barriers we observed.
References
Fleming-Dutra KE, Hersh AL, Shapiro DJ, et al. Prevalence of Inappropriate Antibiotic Prescriptions Among US Ambulatory Care Visits, 2010-2011Inappropriate Antibiotic Prescriptions Among Ambulatory Care Visits, 2010-2011Inappropriate Antibiotic Prescriptions Among Ambulatory Care Visits, 2010-2011. JAMA. 2016;315(17):1864-1873.
Barnett ML, Linder JA. Antibiotic prescribing to adults with sore throat in the united states, 1997-2010. JAMA Intern Med 2014;174(1):138-140.
Barnett ML, Linder JA. Antibiotic prescribing for adults with acute bronchitis in the United States, 1996-2010. JAMA. 2014;311(19):2020-2022.
Mainous AG 3rd, Lambourne CA, Nietert PJ. Impact of a clinical decision support system on antibiotic prescribing for acute respiratory infections in primary care: quasi-experimental trial. J Am Med Inform Assoc. 2013;20(2):317-324.
Terry A. Do Clinical Decision Support Systems Reduce Inappropriate Antibiotic Prescribing For Acute Bronchitis? On-Line Journal of Nursing Informatics. 2017;21(1).
Gonzales R ATMCE, et al. A cluster randomized trial of decision support strategies for reducing antibiotic use in acute bronchitis. JAMA Intern Med 2013;173(4):267-273.
Linder JA, Schnipper JL, Tsurikova R, et al. Electronic health record feedback to improve antibiotic prescribing for acute respiratory infections. Am J Manag Care. 2010;16(12 Suppl HIT):e311-319.
Linder JA, Schnipper JL, Tsurikova R, et al. Documentation-based clinical decision support to improve antibiotic prescribing for acute respiratory infections in primary care: a cluster randomised controlled trial. Inform Prim Care 2009;17(4):231-240.
McCullough JM, Zimmerman FJ, Rodriguez HP. Impact of clinical decision support on receipt of antibiotic prescriptions for acute bronchitis and upper respiratory tract infection. J Am Med Inform Assoc : JAMIA 2014;21(6):1091-1097.
Meeker D, Linder JA, Fox CR, et al. Effect of behavioral interventions on inappropriate antibiotic prescribing among primary care practices: A randomized clinical trial. JAMA. 2016;315(6):562-570.
McGinn TG, McCullagh L, Kannry J, et al. Efficacy of an evidence-based clinical decision support in primary care practices: a randomized clinical trial. JAMA Intern Med 2013;173(17):1584-1591.
Feldstein DA, Hess R, McGinn T, et al. Design and implementation of electronic health record integrated clinical prediction rules (iCPR): a randomized trial in diverse primary care settings. Implement Sci 2017;12(1):37.
Richardson S, Rebecca Mishuris, Alexander O’Connell, David Feldstein, Rachel Hess, Paul Smith, Lauren McCullagh, Thomas McGinn, and Devin Mann. “Think aloud” and “Near live” usability testing of two complex clinical decision support tools. Int J Med Inform. 2017;106:1-8.
Mann D, Hess R, McGinn T, et al. Adaptive design of a clinical decision support tool: What the impact on utilization rates means for future CDS research. Digit Health. 2019;5:2055207619827716
Yoshida E, Fei S, Bavuso K, Lagor C, Maviglia S. The Value of Monitoring Clinical Decision Support Interventions. Appl Clin Inform 2018;9(1):163-173.
Ancker JS, Edwards A, Nosal S, et al. Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Med Inform Decis Making 2017;17(1):36-36.
Phansalkar S, van der Sijs H, Tucker AD, et al. Drug-drug interactions that should be non-interruptive in order to reduce alert fatigue in electronic health records. J Am Med Inform Assoc : JAMIA 2013;20(3):489-493.
Carli D, Fahrni G, Bonnabry P, Lovis C. Quality of Decision Support in Computerized Provider Order Entry: Systematic Literature Review. JMIR Med Inform 2018;6(1):e3-e3.
Ackerman SL, Gonzales R, Stahl MS, Metlay JP. One size does not fit all: evaluating an intervention to reduce antibiotic prescribing for acute bronchitis. BMC Health Serv Res. 2013;13:462.
Gibbs RS, Wieber C, Myers L, Jenkins T. A Continuing Medical Education Campaign to Improve Use of Antibiotics in Primary Care. J Biomed Educ. 2014;2014:6.
Litvin CB, Ornstein SM, Wessell AM, Nemeth LS, Nietert PJ. Use of an electronic health record clinical decision support tool to improve antibiotic prescribing for acute respiratory infections: the ABX-TRIP study. J Gen Intern Med. 2013;28(6):810-816.
McIsaac WJ, White D, Tannenbaum D, Low DE. A clinical score to reduce unnecessary antibiotic use in patients with sore throat. CMAJ: Journal de l'Association medicale canadienne. 1998;158(1):75-83.
Centor RM, Witherspoon JM, Dalton HP, Brody CE, Link K. The diagnosis of strep throat in adults in the emergency room. Med Decis Making. 1981;1(3):239-246.
Heckerling PS, Tape TG, Wigton RS, et al. Clinical prediction rule for pulmonary infiltrates. Ann Intern Med 1990;113(9):664-670.
Funding
NIH NIAID: R01AI108680
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interest
The authors declare that they do not have a conflict of interest.
Ethics Approval
The study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects and was reviewed by New York University School of Medicine’s Institutional Review Board. No animals were included in this study.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic Supplementary Material
ESM 1
(DOCX 237 kb)
Rights and permissions
About this article
Cite this article
Mann, D., Hess, R., McGinn, T. et al. Impact of Clinical Decision Support on Antibiotic Prescribing for Acute Respiratory Infections: a Cluster Randomized Implementation Trial. J GEN INTERN MED 35 (Suppl 2), 788–795 (2020). https://doi.org/10.1007/s11606-020-06096-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11606-020-06096-3