FormalPara Key Points for Decision Makers

Only few studies investigating computer-based clinical decision support systems (CCDSS) measure patient-reported outcomes (PRO).

There are marginal positive effects of CCDSS on PRO.

Factors that could improve the use and effects of CCDSS on PRO are identified and discussed: systems that allow patients to fill in data, electronic-record integration and output at point of care.

1 Background

Electronic medical records have been introduced in most hospitals, outpatient clinics and in primary healthcare in Western Europe and the USA. Benefits in terms of data entry, access and readability compared with paper records are obvious. These records are primarily used in administration, but with improving processing power and mobility, the value of integrating computer technology in medical practice has escalated in recent years. This has led to the possibility of combining electronic medical records with current treatment guidelines in order to bring evidence-based medicine into clinical practice. However, implementation of guidelines is challenging. Complexity of the guidelines, awareness of the content, staff support and time are often encountered barriers to the implementation process [1].

The combination of individual patient data and guidelines is conceptualized as computer-based clinical decision support systems (CCDSS) [2]. In CCDSS, patient data are matched to a medical knowledge base while an algorithm generates a specific treatment recommendation for each patient.

Previous reviews have summarized the evidence that CCDSS can provide reminders regarding preventive examinations or vaccinations [3], can help with or control drug prescriptions [4], and can support the management of acute [5] and chronic diseases [6].

Two previous systematic reviews have specifically analyzed features associated with a positive effect of clinical decision support systems [7, 8]. Kawamoto et al. [7] identified four criteria for positive effects of CCDSS: (a) automatic provision of decision support in clinical workflow, (b) provision of a recommendation rather than just an assessment, (c) decision support by computer, and (d) time and location of decision making. However, the authors could not conduct a subset analysis for patient outcome measures because the number of studies was too small. Delpierre et al. [8] identified two similar criteria as influential for patient outcomes: justification of decision support by provision of research evidence and data standards in the system that support integration of the guidelines.

These previous systematic reviews have primarily demonstrated that CCDSS can improve practitioner performance and provide cost savings; however, the evidence of efficacy on patient outcomes in general is limited [3, 5].

One of the major tasks in the care of patients with chronic conditions in general, and in advanced cancer in particular, is symptom management. The treatment of these diseases and conditions is based on continuous assessment of patients’ symptoms and quality of life. These assessments are often summarized under the umbrella term patient-reported outcomes (PRO). A possible categorization of PRO encompasses generic PRO, such as overall quality of life, and specific PRO such as disease-specific symptoms. There is unique information in this original report from the patients, which cannot be obtained otherwise [9, 10]. In the past, most clinical trials have failed to include PRO as outcomes, but a change in paradigm is ongoing [11, 12].

The aim of this systematic review is, therefore, not only to focus on the effect of CCDSS on PRO, but also to investigate content and application of CCDSS and to analyze whether predefined quality criteria are present in the included studies.

2 Methods

2.1 Data Sources and Searches

A systematic review in MEDLINE and EMBASE according to the PRISMA statement (http://www.prisma-statement.org/) was conducted through OvidSP in May 2014. Searches were limited to the publication period 1996–May 2014 and the English language. The search terms covered “computerized clinical decision systems” and “patient-reported outcomes”.

The specific search strings are provided as supplemental material (see online resource 1, Appendix 1).

Both indexing terms and free text were applied in the query. Search terms representing study design were applied.

Full-text articles were retrieved for all potentially relevant articles. The references of selected articles were checked for further articles.

2.2 Definition of Computer-Based Clinical Decision Support Systems (CCDSS)

For the purpose of this review, a CCDSS was defined as a computer-based system in which individual patient data (input) are linked with treatment guidelines and a recommendation (output) for the specific patient is generated and delivered to the treating physician.

2.3 Inclusion Criteria

A “yes” on all of the following questions qualified a study for inclusion in the review:

  • Is this study on evaluating a CCDSS based on a medical treatment guideline?

  • Is the study a controlled trial where patient care with a CCDSS is compared with patient care without a CCDSS?

  • Is the CCDSS used by a healthcare professional in a clinical practice?

  • Does the CCDSS provide patient-specific information in the form of management options or probabilities and/or recommendations to the clinician?

  • Are PRO described as study outcomes, where patients are directly assessed?

2.4 Exclusion Criteria

Studies meeting one or more of the following criteria were not considered for inclusion in this review:

  • No decision support by treatment guideline applied.

  • Assessment or monitoring without recommendation.

  • Pilot study without comparison to a control group.

  • Decision support delivered to the patients alone; no treatment recommendation for the physician.

  • No PRO described as an outcome.

Retrieved titles and abstracts were screened independently by the two researchers (DB, SR). Inclusion by one of the researcher resulted in full-text assessment. Full-text articles were checked for inclusion and exclusion criteria by the two researchers. Final inclusion was reached by consensus.

2.5 Data Extraction

Data extraction was conducted by one of the researchers and controlled by the other one. Disagreements were discussed in the wider research team and solved by consensus.

The specific research questions were: In what context (disease group) is the CCDSS used? How is the software constructed with regard to input and output of data? Which guidelines did the CCDSS employ? When were the outcomes measured? What is the effect of CCDSS on PRO? Furthermore, the studies were specifically analyzed concerning research questions, results (PRO and other) and conclusion provided. The study quality and methodology were categorized according to study design, sample size calculation and intention-to-treat analysis (Cochrane Handbook for Systematic Reviews of Interventions 5.1.0).

From the two sets of key factors for the efficacy of decision support systems that were published before [7, 8], the following three key factors were applicable to our review and were actively examined in the included trials:

  • Patients fill in data (gold standard for subjective measurements).

  • Data presented to physician at point of care.

  • CCDSS integrates with electronic medical record.

The quality of the CCDSS was assessed against these three criteria.

3 Results

The database searches yielded 1,331 unique references, and 87 full-text articles were analyzed. Fifteen studies, representing 13,480 patients, qualified for inclusion according to the inclusion criteria: ten randomized controlled studies (RCTs), three controlled trials and two cohort studies. The process of selection is displayed in the flowchart in Fig. 1. The range of included patients per trial was 44–4,851.

Fig. 1
figure 1

PRISMA flowchart. CCDSS computer-based clinical decision support system(s), PRO patient-reported outcome(s)

Full size image

An overview of the included trials and their context (disease group) is shown in Table 1.

Table 1 Overview
Full size table

3.1 Patient-Reported Outcomes and Data Collection Time

In nine trials, both quality of life and symptoms as PRO were examined, and five trials examined symptoms only. One trial had only quality of life as a PRO. Three trials collected data immediately after intervention, two trials within weeks, two after 5 months and three after 6–12 months. In five trials, data were collected after 1 year or later, as shown in Table 1.

3.2 CCDSS Content

National or regional guidelines were applied in the CCDSS in 11 of the included trials. In one trial, the applied guideline was not specified; in another trial, the guidelines were developed by the study group on the basis of knowledge collected from a textbook; in two of the trials, institutional guidelines were applied, as displayed in Table 1.

3.3 CCDSS Application and Quality

Patients were actively involved in data entry in eight of the trials; in three of these trials, directly via a desktop computer, and, in two trials, by phone and a questionnaire, respectively. In the remaining three studies, the method of data entry was insufficiently described, as displayed in Table 2.

Table 2 CCDSS quality and data-flow
Full size table

Treatment recommendations were delivered to the physician at point of care in 11 trials. In seven of these trials, treatment recommendations were displayed on the physicians’ computer screen. In four of the trials, the treatment recommendation was sent to the physician by ordinary mail.

The quality according to the three criteria and specific data flow in the CCDSS is displayed in Table 2. Seven CCDSS fulfilled all three quality criteria (patients fill in data, electronic-record integration and output at point of care).

3.4 Effect of CCDSS

Three of the 15 trials demonstrated significant positive impact of CCDSS on PRO. Overall, no negative effects were reported. In the earliest trial, adult patients with a diagnosis of chronic obstructive lung disease or asthma were randomized in a 2 × 2 factorial design. Evidence-based guidelines were connected to the electronic medical journal on a CCDSS in order to provide decision support. The results demonstrated a lower proportion of patients suffering from acute exacerbations of asthma for physicians applying CCDSS compared with the control group [17 vs. 8 %, odds ratio (OR) 0.43, confidence interval (CI) 0.21–0.85]. Additionally, a lower proportion of patients were prescribed emergency nebulizations by physicians applying CCDSS (1 vs. 5 %, OR 0.13, CI 0.01–0.91) [13].

Two trials in schizophrenia treatment showed advantages of a CCDSS. In one trial, psychiatrists treating patients with schizophrenia were divided into four groups. Psychiatrists in the intervention group applied a CCDSS that was connected to an electronic medical journal and national guidelines. When a predefined constellation of symptoms occurred, treatment advice was displayed on the physician’s computer. The remaining three groups of psychiatrists were control groups applying electronic documentation without decision support, paper-and-pen documentation without decision support, and paper-and-pen method followed by group discussion on treatment without decision support, respectively. The trial demonstrated significant effects on positive symptoms in favor of CCDSS (p = 0.004). Further on, there were less re-hospitalizations when the CCDSS was applied (Chi-square 10.4, p = 0.016) [14].

The same group applied the identical CCDSS in a non-randomized study aiming to reduce re-hospitalization after hospital admission for schizophrenia. In addition to providing medical guidance, the CCDSS offered recommendations for complex psycho- and socio-therapeutic interventions based on patients’ socio-demographic profiles. A control group, matching the intervention group on defined criteria, was selected from the same institution. The re-hospitalization rate after 12 months was 41 % in the intervention group and 64 % in the control group (p = 0.018). Additionally, satisfaction with treatment was higher in the intervention group [15].

In a trial combining computerized detection of specific symptoms and decision support, patients in primary care facilities were screened for mood disorders [16]. Physicians received flags and advisory messages in the electronic medical record when a patient was diagnosed with depression. Mean depression score decreased over time, but there was no difference between treatment groups at follow-up.

In a similar approach, a computer-based case finding was combined with computer-generated treatment guidelines for common mental disorders in primary care [17]. There was an improvement in general health compared with usual care, with a small statistically significant, but clinically insignificant, difference at the first 6 weeks, which disappeared after 6 months.

Two trials at large academic primary care group practices failed to show positive effects. In one trial, angina and asthma guidelines were integrated into the computer system and tested in a 2 × 2 study design. 2,241 patients with angina and 1,760 patients suffering from asthma were included. Endpoints were a combination of generic and specific PRO measures. No difference between the four groups was shown, but employment of the software was low [18]. In the other trial, 480 patients with heart disease were included in a trial with a 2 × 2 study design. Guidelines were integrated with a CCDSS employed by the physicians in the intervention group. Physicians in the control group did not receive computerized decision support. There were no significant differences between the groups 1 year after inclusion [19].

In another study, patients with hypertension were randomized in a 2 × 2 factorial design, and guidelines were incorporated in computers employed by the physicians [20]. The primary outcome was quality of life. There were no clinically relevant or statistical significant differences between the four groups. A CCDSS with addition of symptoms was investigated in an RCT on heart failure patients [21]. No significant overall difference between groups was demonstrated.

The effect of a CCDSS in analgesic prescription in hospital inpatient care was tested in different inpatient units of a large US Hospital [22]. The CDDSS showed no improvement on pain. Similarly, the impact of a CCDSS in inpatients on pain control was investigated at a radiation oncology unit [23]. The CCDSS emailed treatment recommendations to the physician. Daily assessment revealed pain reduction in the intervention group, but there were no pain measurements in the control group.

Computer-assisted telephone interviews were carried out regarding asthma treatment of children, and subsequently computer-generated mails were provided to private practitioners [24]. An intention-to-treat analysis showed no difference in number of symptom days. Overall, the study failed to show any significant advantage of this CCDSS.

In a pragmatic trial in the community care setting, 501 patients with diabetes used a web-based color-coded diabetes tracker that was shared between patient and physician [25]. There were improvements in process of care detected, but no difference in quality of life was found between the study arms. A trial investigating the feasibility of a CCDSS in HIV patients revealed a trend toward including a greater number of symptoms in the intervention arm, but the study was not powered for number of symptoms [26].

3.5 Study Quality

Study quality is displayed in Table 3. Only two trials applied the quality indicators: randomization, sample size calculation and intention-to-treat analysis. All other trials lacked an intention-to-treat analysis, and five trials used a controlled design only.

Table 3 Study-quality
Full size table

The categorization according to these quality parameters is shown in Electronic Supplementary material 2, with the authors’ specific research questions and conclusions displayed.

3.6 Association of CCDSS Quality, Study Quality and Effect

All the three studies that yielded positive results provided decision support at point of care. An association between effect and study quality could not be seen, nor between effect and year of publication.

4 Discussion

We systematically reviewed the literature for studies investigating CCDSS and PRO. We found 15 studies applying a CCDSS and examining PRO, and only three of the included studies demonstrated a statistically significant effect employing a CCDSS. A relationship between study quality and effect could not be seen, nor between effect and year of publication. However, an influence of one of the proposed key factors for success can be seen: all the three positive studies provided decision support at point of care, which seems to be a requirement for the success of a CCDSS. Additionally, the CCDSS in the positive study on schizophrenia [14] provided research evidence to justify a particular recommendation [8].

Novel, partially contradictory, criteria for effectiveness were proposed in a recently published meta-regression analysis of RCTs [27] combining 162 trials: (a) system presents advice on interfaces other than electronic charting or order entry system, (b) practitioners have to provide reasons for not accommodating advice, (c) system offers advice concurrently to both practitioners and patients, and (d) CCDSS evaluated by the developers of the CCDSS.

If the former criteria facilitate the application (i.e., the system is “easier to use”), these novel criteria ensure the application (i.e., the system is “less easy to ignore”). In this sense, the delivery of treatment recommendations to both patients and physicians and a compulsory reason to override or ignore advice might be of additional value when developing a CCDSS.

In our results, two tendencies can be noticed: (1) specific PRO such as symptoms seem to be more responsive to change than general quality of life, and (2) studies in psychiatric settings may be more likely to be positive than those in somatic disease.

Possible reasons for the limited effect of CCDSS on PRO can be found on different levels. Implementation of guidelines is challenging, and it is difficult to prove that guidelines improve patient outcomes per se [28]. The CCDSS can only be as effective as the specific guidelines; thus, ineffective guidelines result in ineffective CCDSS. However, the studies in our review included the current state-of-the-art guidelines, and guidelines were adapted to local needs and practice. Furthermore, all authors in all the included studies revised the guidelines to make them applicable in the specific clinical setting.

Another limiting factor may be the employment of computer systems in the trials. Data entry requirements for physicians may be cumbersome and hamper the employment of CCDSS in a busy daily practice [19]. New elements in the workflow or working with the computer in specific situations in daily practice may be complicated and time consuming. Workflow may be disturbed, and integration in routine practice may be considered difficult [29].

The treatment recommendations from CCDSS were usually not mandatory and may not have been applied. Physicians in the control groups could have improved their adherence to guidelines, hence, diluting the effect size in randomized studies with a parallel group design.

Physicians may mistrust suggestions from the computer systems. Computers are intended to prevent medical errors of commission (doing the wrong thing) and omission (not doing the right thing), but computers can introduce new errors as well [30]. One study specifically addressed barriers for physicians in implementing computerized tools [20]. In this study, physicians trusted their own clinical skills and decisions more than instructions from a computer. They may believe that the art of medicine lays in tailoring individual treatment to their specific patient and that guidelines are often too general. In the opinion of physicians, guidelines are good educational tools but difficult to apply in daily work [20]. Alerts or advice from CCDSS are therefore often overridden, especially if they appear often or are irrelevant [31]. Strategies improving accommodation to treatment recommendations need to be carefully implemented and presented concurrently with evidence.

It is possible that the PRO are not responsive enough to change, even though validated and frequently used assessment tools have been applied in the included trials. This might be reflected by the fact that CCDSS had an effect on positive schizophrenia symptoms but not on negative, and the difference between psychiatric and somatic studies. In heart disease, specific symptoms (dyspnea) and overall quality of life measurements may be affected by many other influential factors, and might be more difficult to be influenced than dyspnea in an asthmatic exacerbation. Furthermore, the trials were often underpowered to detect differences in PRO [23].

As reflected in the main exclusion criterion “no PRO,” trials investigating CCDSS measure process outcomes far more often than PRO. These process outcomes are more often associated with positive effects, as shown in a recent synthesis of high-quality systematic reviews on computerized clinical decision support systems [32]. In this synthesis, 17 out of 35 retrieved systematic reviews were included, but impact on patient outcomes was found in only 25 of 91 original studies.

There are several limitations in the present systematic review. One limitation involves the indistinct definition of CCDSS in general. In the present study, we included only CCDSS incorporating a treatment guideline. We excluded simple reminders and required some data processing in the CCDDS and may have omitted valuable results. Another limitation is given by the distinction between clinical outcomes and PRO. The included trials are often powered for clinical outcomes rather than for PRO. Therefore, many studies in the review are lacking the power to detect differences in PRO. Because of the great variability of studies and outcomes, meta-analysis of the data was not possible.

One of the possible strengths of this systematic review is an evaluation on how CCDSS was included in the workflow, together with a focus on methods for data entry into the CCDSS and how information was presented for the physician, thus, providing a focus beyond efficacy alone. These aspects of CCDSS have, to our knowledge, not been evaluated systematically. This might help to better adapt CCDSS to specific situations in the future.

5 Conclusion

Despite the agreement in society about the benefit of modern information technology, there is only limited evidence that CCDSS improve PRO. Because computer systems are often introduced for economic reasons, more research on CCDSS and PRO and scientific evidence on how to improve and apply CCDSS are needed. The employment of accepted guidelines and relevant responsive PRO is central in trials. Besides the point-of-care requirement, the implementation of key factors (“easier to use” and “less easy to ignore”) and direct involvement of users, both patients and physicians (in order to improve acceptance and feasibility in clinical work-flow), in the development and employment of CCDSS may provide more favorable results in the future.