Impacts on practice

  • In advanced clinical decision support systems, patient characteristics are taken into account to reduce the number of irrelevant signals.

  • We implemented five algorithms in the clinical decision support system that are shown directly to the prescriber during order entry.

  • For all five algorithms, the proportion of treatments in line with the advice significantly increased varying from 11 to 36% and three of the five algorithms had a compliance rate above 80%.

  • The acceptance rate of the advices varied from 14 to 90%.

  • For the algorithm to measure plasma levels for vancomycin and gentamicin, the proportion of patients receiving the correct dose after 48 h increased from 73 to 84%.

Introduction

Medication surveillance during physician order entry has an important role in improving medication safety and prevention of hospital admissions due to adverse drug reactions [1, 2]. Medication surveillance is a kind of decision support and consists of supporting the physician during the prescribing process with surveillance of drug-allergy, drug–drug interactions and duplicate therapy checking, among others. Since no patient characteristics are taken into account, basic clinical decision support is hampered by an overload of signals of which a substantial part is clinically irrelevant. A high number of irrelevant signals does result in so called alert fatigue, overriding of all generated medication safety alerts including those that are relevant [3]. A solution is the introduction of advanced clinical decision support systems (CDSS), using algorithms that take patient characteristics into account, such as age, co-medication, co-morbidities and laboratory values [4]. In our research group, we focus on the performance of medication surveillance and how advanced clinical decision support systems can improve it. Our focus is both on the reduction of irrelevant signals and the development of new algorithms that improve medication surveillance and medication safety. In the past few years, we implemented five algorithms in our hospital in the Netherlands and subsequently assessed the effect of the implementation on the number of alerts and acceptance rate of the recommendations. These results have been published before [5,6,7] and are merged in this publication to give an overview of how advanced CDSS can improve medication surveillance.

By individualizing the algorithms generating the alerts, the number of irrelevant signals can be reduced and alert fatigue prevented. Moreover, advanced CDSS can survey other issues, such as whether laboratory tests are ordered and the medication is adapted in line with the laboratory results. Thus far, the alerts of the advanced CDSS related to medication surveillance were shown to the pharmacist for judgment whether the alert was clinically relevant. If relevant, the pharmacist contacted the physician and recommendations in pharmacotherapy were discussed [8,9,10]. Managing the alerts this way is a rather time-consuming process for the pharmacist and the physician is interrupted during other proceedings.

In 2005, Kawamoto et al. identified in an analysis of 70 randomized controlled trials that CDSS possessing four system features were significantly more likely to improve clinical practice compared to CDSS lacking these features [11]. These features are decision support provided automatically as part of clinician workflow, decision support delivered at the time and location of decision making, providing actionable recommendations and computer based. If all these features were available, 94% of the CDSS improved clinical practice.

If the alerts of the CDSS are reviewed by the pharmacist and the physician is contacted in case of doubt about the correctness of pharmacotherapy, the CDSS does not meet the feature that the support is provided as part of clinician workflow. The physician is not contacted during the workflow of order entry, but later while working on other activities. Implementation of algorithms in the advanced CDSS that take patient characteristics into account and that are shown directly to the physician, would meet all the features mentioned by Kawamoto et al.

Aim of the study

In the Netherlands, algorithms for advanced clinical decision support related to medication surveillance have been developed by the Dutch Association of Hospital Pharmacists. We implemented five algorithms in our CDSS that are shown directly to the physician during order entry. We evaluated the effect of the implementation with a pre/post intervention analysis. These results have been published before [5,6,7]. The aim of our study was to evaluate the effect of the implementation of the algorithms on the number of alerts, the acceptance rate of the advice in the alert, and if possible the clinical effect.

Ethics approval

Implementation of the algorithms was part of regular care, and therefore no approval of an ethical committee was needed. The studies were approved by the institutional review board of the Spaarne Gasthuis (2015.0004, 2015.0065 and 2015.0110).

Methods

Setting

All studies were performed by the Pharmacy Foundation of Haarlem Hospitals (Haarlem, the Netherlands), the hospital pharmacy servicing the teaching hospital Spaarne Gasthuis (Haarlem/Hoofddorp, the Netherlands) [5,6,7]. The hospital information system Epic (Epic, Verona, WI) is used, containing a laboratory information system and integrated computerized physician order entry system.

Study design

We selected five algorithms, that were developed by the Dutch Association of Hospital Pharmacists and were suitable for alerting the physician during order entry. These algorithms generate an alert if:

  1. 1.

    Medication is ordered with an increased risk of gastrointestinal bleeding and, according to the Dutch guideline, gastrointestinal prophylaxis is indicated based on age and medication use, while not prescribed [12]. In the alert the recommendation is given to prescribe a proton-pump inhibitor;

  2. 2.

    Methotrexate is ordered for oral or subcutaneous administration, while no folic or folinic acid is prescribed;

  3. 3.

    A bisphosphonate is ordered, while no colecalciferol, alfacalcidol, calcitriol or dihydrotachysterol is prescribed;

  4. 4.

    One sodium lowering drug is ordered while the sodium level is below 130 mmol/l or two sodium lowering drugs are ordered concomitantly while the sodium level is below 135 mmol/l, with the advice to reconsider the sodium lowering drug(s). The selected sodium lowering drugs were diuretics, NSAIDs, SSRIs, venlafaxine, carbamazepine, oxcarbazepine, cisplatin and carboplatin.;

  5. 5.

    Intravenous vancomycin or gentamicin is ordered in a frequency other than once, and no recent plasma drug concentration was available nor a plasma drug concentration measurement was ordered. In the alert the advice was given to order a drug concentration measurement.

These algorithms generated an alert shown to the prescriber during order entry. In the pop-up, the advice was given how to manage the alert and if applicable the physician could order the recommended medication with two clicks (Fig. 1). For the algorithm of measuring plasma concentrations for vancomycin or gentamicin, a drug concentration measurement could be ordered with two clicks. The algorithm for gastrointestinal prophylaxis in the advanced CDSS replaced all drug–drug interaction alerts from the basic CDSS for combinations of drugs with an increased risk of gastrointestinal bleeding. Similarly, the algorithm for hyponatremia in the advanced CDSS replaced all drug–drug interaction alerts from the basic CDSS for combinations of sodium lowering drugs.

Fig. 1
figure 1

Alerts of the advanced clinical decision support system in Epic, shown to the prescribed (a) if a bisphosphonate is prescribed while no vitamin D or analogue is prescribed and (b) if two or more sodium lowering drugs are prescribed while the last sodium level measured is below 135 mmol/l. These alerts are translated into English

Full size image

If the alert was overridden without accepting the advice and the patient was hospitalized, the patient was shown on a list that was reviewed daily by the attending pharmacist. In this list, patients were also shown in whom a sodium level below the thresholds was measured after ordering the sodium lowering drug(s). All patients who were prescribed vancomycin or gentamicin were reviewed daily and after measuring drug concentrations, a dosing advice and an advice on follow-up plasma levels was given.

We performed a prospective intervention study, with a historical group for comparison to analyze the effect of the alert generated by the algorithms.

Participants

All inpatients and outpatients at the Spaarne Gasthuis, who were prescribed drugs with (1) an increased risk of gastrointestinal bleeding and an indication for gastrointestinal prophylaxis, (2) methotrexate per os or subcutaneous, (3) bisphosphonates, (4) sodium lowering drugs or (5) vancomycin or gentamicin for at least 48 h were included in the analysis. For drugs with an increased risk of gastrointestinal bleeding and for vancomycin and gentamicin, only patients treated in the Spaarne Hospital, or after the merger (22 March 2015) at the Hoofddorp site of the Spaarne Gasthuis (being formerly the Spaarne Hospital), were included in the study. The start of the historical comparison group was before the merger, and Epic was not introduced in the Kennemer Gasthuis at that time.

Study flow

We compared patients who started treatment before implementation of the algorithm with patients who started treatment after implementation. For the algorithm of gastrointestinal prophylaxis the time period 3 months before implementation (May 1, 2014–August 1, 2014) was compared with 3 months after implementation (December 1, 2014–March 1, 2015). The 4 months in between were used as an implementation period. For measuring blood concentrations in patients using vancomycin and gentamicin, we compared the time period 1 year before (May 1, 2014–May 1, 2015) and 1 year after implementation (June 1, 2015–June 1, 2016). For the other algorithms, we compared the 48 days before implementation (July 13, 2015–August 24, 2015) with the 43 days after implementation (August 25, 2015–October 11, 2015).

Outcome measures

For the algorithm gastrointestinal prophylaxis, we analyzed whether gastrointestinal prophylaxis was ordered within 1 h after the alert; for the algorithms methotrexate and bisphosphonate, we analyzed whether folic or folinic acid and vitamin D or vitamin D analogue was prescribed within 48 h after the alert; for the algorithm hyponatremia we analyzed whether one or more sodium lowering drugs were stopped within 1 h after the alert; and for the algorithm to measure blood concentrations in patients using vancomycin and gentamicin we analyzed whether a blood concentration was measured within 72 h after start of therapy as recommended by the guidelines [13, 14]. For the latter algorithm, we also analyzed whether the time the correct dose was prescribed differed before and after implementation and whether the correct dose was prescribed 48 h after start of vancomycin or gentamicin. For all algorithms, except gastrointestinal prophylaxis, we analyzed the frequency the pharmacist contacted the physician to make recommendations regarding the medication.

Methods for data acquisition

Data were extracted from the hospital information system Epic, using Crystal Reports (Walldorf, Germany).

Methods for data analysis

The unit of analysis was the number of orders that met the criteria of the algorithms. Changes in proportion before and after implementation of the algorithms were analyzed using the Pearson’s Chi square test. For the algorithm to measure blood concentrations in patients using vancomycin and gentamicin, the time from start of treatment until the correct dose was estimated by inputing doses, blood concentrations and patient data in MW Pharm (MW Pharm bv, Groningen, the Netherlands). MW Pharm is a computer program designed to support dose advices based on plasma drug concentrations and uses Bayesian computation to predict plasma concentrations. Differences in the time to the correct dose was prescribed were analyzed using the Log Rank test. Data were analyzed using SPSS (IBM Corp. Armonk, NY).

Results

For all algorithms, the proportion of treatments in line with the advice increased significantly after implementation (Table 1). The increase varied from 11 to 36% with three algorithms having a compliance rate above 80%.

Table 1 The effect of implementation of algorithms in advanced clinical decision support systems, shown directly to the prescribed during order entry
Full size table

For the algorithm gastrointestinal prophylaxis, 812 alerts were shown before implementation of the algorithm. Of these alerts, 91 (11.2%) alerts were correct and patients had an indication for gastrointestinal prophylaxis while it was not prescribed. For 244 orders, an alert should have been fired according to the guideline, but was not generated. After 4 of the 91 correct alerts (4.4%), gastrointestinal prophylaxis was prescribed within 1 h. After implementation of the algorithm, the number of alerts was reduced to 217. All these alerts were correct and we identified four orders for which erroneously no alert was fired. After 97 of the 217 alerts (44.7%), gastrointestinal prophylaxis was prescribed.

The algorithms for methotrexate and bisphosphonate were introduced without replacing basic medication surveillance alerts. Before implementation of the algorithm for methotrexate, thirteen alerts would have been generated. Although no alert was fired, the physician prescribed folic or folinic acid within 48 h after 7 of the 13 (54%) orders for methotrexate without folic or folinic acid at the moment of ordering. After implementation of the alert, folic or folinic acid was prescribed within 48 h after 19 of the 21 orders followed by an alert (90%). For the bisphosphonate algorithm, vitamin D was prescribed after 5 of the 47 orders (11%) for a bisphosphonate without vitamin D at the moment of ordering, before implementation. After implementation, 60 orders generated an alert and after 24 (40%) vitamin D or an analogue was prescribed within 48 h.

Before implementation of the algorithm hyponatremia, 27 drug–drug interaction alerts were shown when two or more sodium lowering drugs were combined simultaneously, irrespective of the actual plasma sodium level. In this period 119 alerts would have been fired and three times (3%) the sodium lowering drug was stopped within one hour. After implementation of the algorithm, 109 alerts were shown and 15 times (14%) the sodium lowering drug was stopped within 1 h.

For the algorithms methotrexate, bisphosphonates and hyponatremia, we analyzed the frequency that the pharmacist contacted the prescriber after reviewing the inpatients who were presented on the patient list. The pharmacists made eight phone calls, none for methotrexate, six for bisphosphonates and two for hyponatremia. After three of the eight calls (38%), the medication was changed in line with the recommendation.

After implementation of the algorithm vancomycin and gentamicin, the proportion of treatments where the plasma level was measured within 72 h increased from 47 to 80%. After 60 of the 138 alerts (43%), the physician ordered a vancomycin or gentamicin plasma concentrations within 10 min after the alert. During 38 treatments (28%), the pharmacist contacted the physician with the advice to order a first plasma level. The time to receive the correct dose was significantly shorter (p = 0.03) after implementation and the proportion of patients who received the correct dose 48 h after start of therapy increased from 73 to 84% (p = 0.03).

Discussion

The implementation of advanced CDSS algorithms that were presented to the physician during order entry, resulted in an increase in compliance with the guidelines. For all five algorithms that were implemented, the compliance significantly increased and in three of the five algorithms the compliance was above 80% after implementation. The acceptance rate of the advices given in the alert varied from 14 to 90%.

A prerequisite for implementation of CDSS shown during order entry is that the alerts have a high specificity. Alerting physicians too often with irrelevant alerts will result in alert fatigue. The advantage of the advanced algorithms that take patient characteristics into account, is that we could filter drug–drug interaction alerts for combinations of drugs that increase the risk of gastrointestinal bleeding. These alerts are not relevant if gastrointestinal prophylaxis is already prescribed. Similarly, we filtered alerts for combinations of sodium lowering drugs, that are not relevant if sodium levels are and remain within normal range.

The post implementation measurements were performed directly or shortly after implementation of the algorithm. However, after implementation of CDSS continuous maintenance is indicated to improve the performance. After this study, for example, we excluded furosemide as sodium lowering drug from the algorithm hyponatremia. Although included in the guideline, furosemide does not lower the sodium level and is often used in the treatment of hyponatremia [15]. Although we did not analyze the effect after this adjustment, it is likely that the acceptance rate has improved.

Although the goal is to have an acceptance rate as high as possible, it is unrealistic to expect that acceptance rates are 100%. In the algorithms gastrointestinal prophylaxis and vancomycin or gentamicin, the treatment can be started for a short time period, making the advice irrelevant. If drugs that increase the risk of a gastrointestinal bleeding are used shortly, the risk can be negligible and the necessity to prescribe gastrointestinal protection low. In the algorithm hyponatremia, there are numerous other causes for the hyponatremia than due to sodium lowering drugs. If the expected cause is not drug-induced, there is no necessity to stop these drugs. For the algorithm bisphosphonate, a recent plasma vitamin D level can be measured indicating that there is no vitamin D deficiency and no reason to start vitamin D treatment or a hypercalcaemia can be present. This item was nevertheless not included in the guidelines developed by the Dutch Association of Hospital Pharmacists and therefore not implemented in the algorithm.

For the algorithm gastrointestinal prophylaxis, we analyzed the acceptance rate of the basic drug–drug interaction alerts between drugs that increase the risk of gastrointestinal bleeding, in cases that no gastrointestinal prophylaxis was prescribed. In 4.4% of the basic drug–drug interaction alerts gastrointestinal prophylaxis was prescribed after the alert, versus 45% of the algorithms in the advanced CDSS, indicating that advanced CDSS are far more effective in stimulating that the recommendations are followed. For three algorithms, we analyzed the effect of the pharmacist interventions. Thirty-eight percent of the pharmacist interventions were accepted. Although these interventions were performed in most cases after ignoring the initial alert, this percentage indicates that also pharmacist interventions are frequently ignored.

In a previous study by Scheepers-Hoek et al., the effect of four different alert presentation methods were assessed on an intensive care unit (ICU) [16]. The compliance with the advices was highest if the alert was presented in a pop-up (41%), followed by pharmacy intervention (33%), in a patient list on the desktop of the physician (20%) and in specific section (CDSS tab) in the patients electronic health record (19%). However, in the study by Scheepers-Hoek et al. the pop-up was presented when the physician evaluated the electronic health record, and was not shown directly during order entry. Another major difference with the study by Scheepers-Hoek et al. is that their study was performed at an ICU, while our study was performed in the whole hospital including both inpatients and outpatients.

Our study has some potential strengths and limitations. A strength of our studies is that we used the same methods in the historical and intervention group to analyze whether guidelines were followed, to reduce potential information bias. A limitation is that we used a historical group as a reference. Although we are not aware of factors apart from the implementation that have affected the results, we cannot exclude it completely. However, it is unlikely that other factors have such an impact that they explain the change in effect found in our studies. The merger of the two hospitals could have influenced the effect of the vancomycin/gentamicin algorithm, although the algorithm was only analyzed in the Hoofddorp site of the Spaarne Gasthuis, and the policy of the merger was that all procedures at the Hoofddorp site were implemented at the other hospital and not vice versa. Another limitation is that we could not incorporate data from the medical history in the algorithms, because these are not unequivocally registered in Epic. In the algorithm for gastrointestinal prophylaxis, we could not use factors like having a peptic ulcer in the past although this is an item in the guideline. Another limitation is that we analyzed whether guidelines were followed, and did not analyze the effect on patient outcomes. To analyze these effects, we should have included more patients in our studies than was practical possible. However, the guidelines were composed based on a large study after medication related hospital admissions [12], making it plausible that an increase in guideline adherence will reduce the number of medication related hospital admissions. Further research should focus on the improvement of existing algorithms and the development of new algorithms to improve guideline adherence.

Conclusion

To conclude, we implemented five advanced algorithms with alerts shown directly to the physician during order entry. After implementation of these algorithms, the guideline recommendations were followed significantly more often. These studies have shown that an advanced CDSS that alerts directly to the physician during order entry is an effective instrument to improve guideline compliance.