BACKGROUND

Direct-to-patient telemedicine is a rapidly growing venue for health care in the USA, and acute respiratory tract infections (ARTIs) are a common indication for these visits.1 National data from US ambulatory settings exclusive of direct-to-patient telemedicine practices showed almost one out of every three antibiotic prescriptions were inappropriate among adult patients,2, 3 with ARTI constituting the most common diagnoses associated with antibiotic prescriptions. Specifically, sinusitis and pharyngitis comprise two of the top three diagnoses associated with antibiotic prescriptions.2 Reducing unnecessary antibiotic prescriptions through antibiotic stewardship is a top public health priority, as antibiotic use is the most important factor driving the rise in antibiotic-resistant infections around the world.4, 5

Without intervention, telemedicine could greatly contribute to antibiotic-resistant infections as healthcare systems seek to expand access through different channels of care. Some studies have shown that inappropriate antibiotic prescribing may be more common in the telemedicine setting,6, 7 though this pattern has not been consistently observed.8 Most studies on clinician behavior change around antibiotic prescription have focused on traditional “brick-and-mortar” clinics, and no interventional studies have been published focusing on direct-to-patient telemedicine settings.9 Traditional ambulatory clinics have co-located clinicians who often share social, demographic, and clinic practice similarities.9 In contrast, virtual medicine practices have clinicians located across the country who may not have these shared similarities, and provide care without allied health professionals. Interventions which work for brick-and-mortar practices cannot be assumed to work for virtual ones.

The objective of this study was to examine the effect of two antibiotic stewardship interventions (education versus education plus individualized feedback reports) on antibiotic prescribing rates for upper respiratory infection (URI), bronchitis, sinusitis, and pharyngitis in a direct-to-patient telemedicine setting. We hypothesized that the education plus individualized feedback reports would be more effective in reducing antibiotic prescribing for these four conditions compared with education alone.

METHODS

Study Setting

We conducted a two-arm parallel randomized controlled trial of clinicians employed at a single telemedicine company, Doctor On Demand. Doctor On Demand is a national direct-to-patient video visit medical practice providing primary care and mental health services via an employed physician practice. Patients are connected to an on-shift physician licensed in the state in which the patient is physically located. Both “on-demand” and scheduled appointment visits are available. Clinicians document the encounter with a proprietary electronic health record system.

Participants and Eligibility

Clinicians who were employed at the practice on or after January 1, 2017, were included. All visits between January 1, 2018, and November 30, 2018, by study clinicians with at least one diagnosis of URI, bronchitis, sinusitis, or pharyngitis were included. Inclusion and exclusion criteria for visits were based on definitions used for Health Effectiveness Data and Information Set (HEDIS) measures.10 Any visits with a study diagnosis and another diagnosis that would or might warrant antibiotics (i.e., urinary tract infections) were excluded, based on a previously published tiered system of common diagnoses.2 If a visit had a diagnosis of upper respiratory infection or bronchitis (which do not warrant antibiotics) and a diagnosis of sinusitis or pharyngitis (which may warrant antibiotics), these visits were counted as sinusitis/pharyngitis and excluded from URI/bronchitis. For bronchitis, a more extensive list of the HEDIS exclusion criteria (e.g., diagnosis of immunodeficiency) was used to exclude visits from the bronchitis category only.10 Diagnostic codes used as inclusion and exclusion criteria are listed in Supplementary Tables 1 and 3. Visits with missing data on clinician location, patient age, or location were excluded.

The pre-intervention period was defined as January 1 to April 30, 2018, and post-intervention period defined as May 1, 2018, to November 30, 2018.

Design and Randomization

In April 2018, we randomized a group of 49 clinicians to either education alone (control) or education plus individualized feedback (intervention) (Fig. 1). A programming engineer not involved in the study randomized clinicians using a randomization sorting function.11 Study investigators were blinded to the randomization. De-identified visit data were collected from the electronic health record.

Figure 1
figure 1

Participant flow diagram.

Full size image

Interventions

We used Powell’s conceptual framework for understanding and reducing overuse by primary care clinicians12, 13 to design our interventions, targeting clinicians’ beliefs about expected outcomes of patients, of clinicians, and of the organization.

The control group received education through two components: a 1-h slides-based presentation by the Associate Medical Director (general medicine clinician) on national consensus treatment guidelines for ARTIs, and the Stanford University School of Medicine online continuing medical education course “To Prescribe or Not to Prescribe? Antibiotics and Outpatient Infections”.14,15,16,17,18 The intervention group received the same education components plus individualized feedback via an online dashboard which showed each clinician their personal rate of antibiotic prescription and practice-wide prescribing rates for URI, bronchitis, sinusitis, and pharyngitis (Fig. 2). Dashboards summarized antibiotic prescription rates for the previous month starting May 2018. Throughout the study period, general principles of antibiotic stewardship were reinforced during regular quarterly clinician meetings.

Figure 2
figure 2

Sample clinician dashboard for intervention group. Dashboards were personalized for each clinician in the intervention group. The entire practice’s antibiotic prescription rates for target conditions were included at the top under “Practice Summary.” Individual clinician prescription rates were included under “Your Results” in the middle, with a calculated “Your Difference from Practice” at the bottom.

Full size image

Core Components

We considered reviewing ARTI treatment guidelines (either through attendance at the presentation or self-review of slides or the online course) to be core components of the education strategy, and review of the online dashboard as the core component of the individualized feedback strategy. Intervention reach was determined as follows: (1) Clinician attendance was tracked during the treatment guideline presentations at the end of April 2018. (2) Clinicians unable to attend via video were emailed slides; those clinicians who downloaded the slides based on email analytic data were assumed to have reviewed them. (3) For the Stanford online course, clinicians were required to email in their certificate of completion to the Associate Medical Director.

Outcomes

The primary outcome was antibiotic prescription rates for each of the four diagnostic categories: URI, bronchitis, sinusitis, and pharyngitis. Given the difference in baseline prescription rates, we used an interaction term between group (intervention versus control) and time period (post versus pre) to assess if the change in antibiotic prescription in post- versus pre-periods in the intervention group was significantly different from the change in antibiotic prescription in the post- versus pre-periods in the control group. This interaction term is the ratio of two odds ratios (ORs): the OR of antibiotic prescription post versus pre in the intervention group divided by the OR of antibiotic prescription post versus pre in the control group.

As a secondary outcome to assess if any reduction in antibiotic prescription rate could be explained by diagnosis shifting from antibiotic inappropriate conditions (URI, bronchitis) to potentially antibiotic appropriate conditions (sinusitis, pharyngitis), we examined the proportion of total visits diagnosed as sinusitis or pharyngitis over time.

Statistical Analysis

For the primary outcome, given the different baseline antibiotic prescription rates between control and intervention groups, we used mixed effects generalized models with logistic link functions and clustered standard errors around clinicians to estimate antibiotic prescription for URI, bronchitis, sinusitis, and pharyngitis. The outcome was whether or not an antibiotic was prescribed. The independent variable of interest was the interaction term between arm (control versus intervention) and period (pre-intervention versus post-intervention). We used backwards selection to select covariates, which included clinician census region, patient census region, and patient age category.

For the secondary outcome of proportion of visits that were sinusitis or pharyngitis, we used mixed effects generalized models with logistic link functions and clustered standard errors. We included as covariates clinician census region, patient census region, and patient age category. All analyses were conducted in SAS version 9.4 (Cary, NC, USA). Figures were created in R version 3.6.0.19

Ethics and Consent

This study was considered exempt by the George Washington Institutional Review Board. Consent was obtained from all clinicians enrolled in the study and informed consent from patients was waived given the use of de-identified data.

RESULTS

Participant Flow

Out of 49 clinicians assessed for eligibility, 45 were ultimately randomized to either education (n = 23) or education plus feedback arm (n = 22) (Fig. 1). Among control clinicians, 10 (43.5%) attended the treatment guideline presentation via video, while 6 reviewed the slides separately (26.1%), 20 (87.0%) reviewed the Stanford online course, and 1 (4.3%) did not receive any education. Among intervention clinicians, 8 (36.4%) attended treatment guideline presentation, while 5 reviewed the slides separately (22.7%), 15 (68.2%) reviewed the Stanford online course, and 5 (22.7%) did not receive any education.

All clinicians were retained throughout the study period from January 1, 2018, to November 30, 2018. Out of 55,498 total visits with study diagnoses made by clinicians during the study period, 55,364 (99.8%) met inclusion criteria and are analyzed in Table 1. A summary of visits by diagnosis meeting inclusion criteria is included in Supplemental Fig. 1. We used intention-to-treat analysis.

Table 1 Characteristics of Patients and Visits Included in Intention-To-Treat Population
Full size table

Study Population

As shown in Table 1, clinicians in the control and intervention groups had similar mean years since residency (14.7 years vs 11.8 years, p = 0.09), proportion of female clinicians (78.3% vs 81.8%, p = 0.77), and geographic distribution with the majority of clinicians residing in the south (65.2% vs 59.1%, p = 0.86). The majority of clinicians in both groups were family medicine–trained (91.3% vs 86.4%, p = 0.08).

A total of 55,364 visits were made with study diagnoses, with 24,843 (44.9%) visits in the intervention group. The control and intervention groups had similar patient age distribution, with the largest proportion in the 18–44-year age range (57.7% vs 59.7%). Similar to clinician geography, the majority of patients also resided in the south (56.8% vs 53.2%).

The two groups had similar rates of diagnosis for the four target conditions URI, bronchitis, sinusitis, and pharyngitis.

Baseline Data

In the pre-intervention period, the control and intervention groups did have different baseline antibiotic prescription rates for URI (18.4% vs 15.0%), bronchitis (46.8% vs 64.0%), sinusitis (84.1% vs 87.2%), and pharyngitis (81.3% vs 74.9%) (Table 2).

Table 2 Effect of Intervention on Antibiotic Prescription Rates
Full size table

Outcomes and Estimation

The primary outcome was antibiotic prescription rates for the four target conditions URI, bronchitis, sinusitis, and pharyngitis. Monthly antibiotic prescription rate for each condition declined over the study period (Fig. 3). For bronchitis, antibiotic prescribing decreased in the intervention arm during the pre-period before dashboards were shared on May 1, 2018.

Figure 3
figure 3

Antibiotic prescription rate in intervention vs control group, by month. Control group is in blue; intervention group is in orange. The 95% confidence interval for antibiotic prescription is represented by the dotted ribbons. The start of education for control group, and education + feedback for intervention group, is indicated by the dotted black vertical line (May 1 2018).

Full size image

There was a decrease in crude antibiotic prescription rate across both arms, in the post- vs pre-period (Table 2). For URI, antibiotic prescription decreased from 15.0 to 7.8% in the intervention arm and from 18.4 to 12.8% in the control arm, while for bronchitis, it decreased from 64.0 to 32.1% in the intervention and from 46.8 to 35.3% in the control arm. For sinusitis, prescriptions decreased from 87.2 to 76.8% in the intervention and from 84.1 to 76.7% in the control arm, while for pharyngitis, it decreased from 74.9 to 65.5% in the intervention and 81.3 to 75.3% in the control arm (Table 2). After adjusting for covariates, the decrease in antibiotic prescriptions post versus pre in both intervention and control groups remained significant (URI intervention 95% CI [0.32, 0.73], URI control 95% CI [0.69, 0.95]; bronchitis intervention 95% CI [0.15, 0.40], bronchitis control 95% CI [0.49, 0.73]; sinusitis intervention 95% CI [0.51, 0.81], sinusitis control 95% CI [0.56, 0.67]; pharyngitis intervention 95% CI [0.48, 0.90], pharyngitis control 95% CI [0.64, 0.83]). For URI and bronchitis, there was a greater decrease in antibiotic prescription in the intervention group compared with the control group (URI interaction term ratio 0.60, 95% CI [0.47, 0.77], p < 0.001; bronchitis interaction term ratio 0.42, 95% CI [0.32, 0.55], p < 0.001). However, for sinusitis and pharyngitis, we did not find a significant difference between intervention and control groups for the reduction in antibiotic prescription from the post- to pre-periods.

For the secondary outcome to assess for diagnostic shifting (i.e., a shift in diagnostic codes from antibiotic inappropriate conditions to antibiotic appropriate ones), we examined the proportion of visits for potentially antibiotic appropriate conditions (sinusitis or pharyngitis) over time between the two groups (Supplementary Table 4). There was evidence of diagnosis shifting, as the OR of visits with sinusitis or pharyngitis increased in the post-period compared with that in the pre-period (aOR 1.36, 95% CI [1.29, 1.44], p < 0.001). There was less diagnostic shifting in the intervention group compared with the control group (aOR for interaction term 0.86, 95% CI [0.80, 0.93], p < 0.001).

DISCUSSION

Individualized prescribing feedback dashboards plus education to telemedicine clinicians was more effective than education alone in reducing antibiotic prescriptions for upper respiratory infections and bronchitis, but not for sinusitis or pharyngitis. Our results suggest education alone may be quite effective, given the reductions in antibiotic use seen in both arms over the study period. While there was evidence of diagnosis shifting over time, there was less shifting in the intervention vs control group, suggesting the reduction in antibiotic prescribing was not due solely to diagnosis shifting. Given the lack of a control arm without any interventions at all, we are unable to differentiate effects of either arm from secular time trends.

Our study is unique as a randomized controlled trial in a direct-to-patient telemedicine setting, and shows that implementation strategies like education and audit with feedback do work in the particular contexts of virtual medicine with a robust experimental design. Prior clinician behavior change studies focused largely on brick-and-mortar practices and used study designs (pre/post, PDSA cycles) more prone to omitted variable bias.

Our findings are similar to other quasi-experimental interventions. In one study at Kaiser, clinician education and clinical decision support targeting acute sinusitis reduced antibiotic prescription with an adjusted OR of 0.78 in the post- vs pre-period, although the absolute difference was small (2%).5 In a multicenter, quasi-experimental study in French pediatric emergency departments using interrupted time series, the intervention of local protocol implementation, education sessions, and center level feedback did change the slope for antibiotic prescription rate (− 0.4% per 15-day period), with a cumulative effect of − 30.9% in reducing antibiotic prescriptions.20 In a large pre-/post-study among 117 primary care clinics in the Midwest United States, a best practice alert to reduce antibiotic prescribing rates for acute sinusitis resulted in no change in oral antibiotics.21 Beyond reducing antibiotic prescriptions altogether, clinician-targeted interventions also reduce use of broad spectrum antibiotics for ARTIs in favor of narrower ones.20, 22

Antibiotics are not indicated for either upper respiratory tract infections or for bronchitis, so the target antibiotic prescribing rate for these conditions is zero. For sinusitis, antibiotics may be warranted when diagnostic criteria are met. Prior studies have used the lowest regional antibiotic prescription rate for sinusitis as the target (3–4% for adults).2 Appropriate rates for pharyngitis should be close to the prevalence of group A streptococcal pharyngitis of 18% for adults.2 In our study, the crude pre- and post-period antibiotic prescription rates were above appropriate levels across all conditions, similar to other nationwide studies.2 However, there was a decrease in antibiotic prescription rates towards target levels across conditions in both arms.

Previous studies have shown education has a large initial effect in reducing antibiotic prescription rates for ARTIs—however, this effect does not persist over time.5 A Cochrane synthesis of systematic reviews found that the quality of the evidence for clinician education interventions was very low given highly heterogeneous trial results, making it challenging to draw universal conclusions.9 The large temporal effect of education seen in our study (2–3 h total of online learning) may have been mediated by site-specific institutional culture. Key elements of success may have been organizational leadership support for antibiotic stewardship, and emphasis on practice improvement initiatives with bimonthly continuing education sessions on evidence-based guidelines in telemedicine. Individualized feedback may provide one way to maintain antibiotic stewardship in the forefront of a clinician’s consciousness, and correct their previously held beliefs about their own performance.12 Audit with feedback has been effective in behavior change for other conditions as well in primary care practices.23 Furthermore, the differential effect of the intervention on upper respiratory infections and bronchitis (where appropriate prescription levels are zero) compared with sinusitis and pharyngitis (where the appropriate prescription levels are not zero, but certainly lower than current practice) may point to a real difference in how clinicians respond to antibiotic stewardship programs based on the underlying conditions. When there is a potentially appropriate reason for antibiotics, each individual clinician may believe herself justified in prescribing, and be more open to external influences (concern over missed diagnosis of bacterial illness, time pressures), than when there is no appropriate reason for antibiotics at all. Future studies should explore tailoring the messaging and mechanisms of antibiotic stewardship interventions based on the underlying conditions, and clinician beliefs or assumptions about appropriate levels of prescription.

There were a number of limitations in this study. First, given data were collected continuously in real time rather than in discrete baseline and endline periods, there were changing time trends throughout the study which are not directly captured. Instead, we analyzed mean antibiotic prescription in the pre- and post-periods and attempted to address inter-period variability by examining the interaction term. Second, there may be limits in generalizability as the majority of clinicians are family medicine–trained, and there might be differences in intervention effectiveness by specialty that should be explored in future studies. Third, the fact that clinicians were aware of being observed, or the Hawthorne effect, may have played a role in the global reductions in antibiotic prescriptions seen during the study. Fourth, data were extracted from an electronic medical record without a separate database, which may have resulted in misclassification bias or incorrect data were entered, and missing data.

Based on our results, both education and education plus individualized feedback present effective ways to decrease antibiotic prescriptions for acute upper respiratory infections. These findings should be used to promote antibiotic stewardship across telemedicine and other ambulatory practices. In December 2018, Doctor On Demand expanded the intervention to provide feedback dashboards to all employed clinicians. Future studies should examine the long-term impact of education and feedback interventions, and maintenance of antibiotic prescription reductions.