Introduction

Appropriate empirical antibiotic therapy decreases mortality with an odds ratio of about 1.6, relative to patients where appropriate therapy is initiated when a pathogen has been isolated and tested for its susceptibility to antibiotics [1]. Susceptibility results are typically available 48–72 h after blood is drawn for culture [2].

Rapid molecular diagnostics applied directly to primary blood samples have the potential to significantly shorten the time to pathogen identification, with turnaround times of 6–8 h [3, 4], thus shortening the time to appropriate therapy, which may lead to reduced mortality [1]. These rapid methods are, however, more expensive than the standard diagnostic procedure. A review by the National Institute for Health and Care Excellence (NICE) estimated that the incremental cost of rapid molecular diagnostics, such as those based on the polymerase chain reaction (PCR), range from 127€ to 373€ [5], whereas the price of a blood culture (BC) is approximately 33€ [6].

Selecting patients with higher pre-test probability of positive PCR result and higher mortality could contribute to improving the cost-effectiveness of rapid diagnostic methods. A method for selecting high-risk patients for rapid diagnostics could potentially be implemented as a clinical decision support system, integrated into the hospital’s electronic health record.

SepsisFinder (SF) is a stochastic model implemented as a causal probabilistic network, which predicts the probability of bacteremia based on laboratory variables and vital signs [7, 8]. Assuming that a high probability of bacteremia correlates with a high pre-test probability of a positive PCR, then SF may be able to predict a positive PCR based on a high probability of bacteremia. It has been shown that positive BC and positive PCR can be predicted using the same factors [9]. SF’s estimate of the probability of bacteremia could then be used to select patients for rapid diagnostic testing by stratifying them into low- and high-risk groups. For use in a laboratory setting, we used a modified version of SF: SepsisFinder-Lab (SFL).

The aim of this paper was to compare two strategies for selection of “high-risk” patients from among those who had blood cultures. The rates of bacteremia, DNAemia, and 30-day mortality were compared in groups of high-risk patients selected either by SFL or clinically selected based on Sepsis-3 definitions [10].

Methods

Patients and setting

Data were collected retrospectively for patients in the emergency department or infectious disease ward who had BCs drawn between May 1 and December 31, 2016 at five hospitals in the southern part of the Emilia-Romagna region of Italy. All patients were admitted through the emergency ward, but due to the local procedures at two of the hospitals, some patients had blood cultures drawn at the infectious disease ward. Inclusion criteria were suspicion of infection (drawing of BCs within 2 h of hospital admission, as part of standard clinical care) and age 18 years or older. Infection variables were collected retrospectively from the laboratory information system and included albumin, bilirubin, C-reactive protein (CRP), lactate, neutrophil fraction, and platelets. Values with timestamps closest to the time of BC draw, but within a window of 48 h before to 18 h after, were chosen.

Exclusion criteria

Patients were excluded:

  • If the associated data contained two or fewer infection variables

  • If the infection variables were not available within 1 h after BC draw

  • If the BC test result was not available

  • For patients with multiple visits, only the first encounter was included

Mortality data were collected from the local government population register.

The eligibility criteria for this study were designed such that the inclusion criteria were the same as, and the inclusion period spanned the period where a prospective trial of a rapid microbiology (RM) assay based on universal PCR amplification coupled with mass spectrometry (PCR/ESI-MS), referred to as the “RM study,” was conducted by AUSL Romagna [11].

Mathematical prediction model: SFL

We modified the original SF, a causal probabilistic network model that included temperature, heart rate, calculated mean arterial pressure, mental status, neutrophil fraction, platelets, CRP, lactate, creatinine, and albumin as input variables for prediction of the probability of bacteremia [7, 8].

The modified SF model was trained using the EM-learning procedure provided in Hugin (Hugin Expert, Aalborg, Denmark) following the same process used in construction of SF [7]. The training dataset was based on 4707 patients with a community acquired infection at Beilinson Hospital, Petah Tikva, Israel. For the training of the bacteremia prediction, the input variables were limited to those available in the data from the Emilia-Romagna hospitals (neutrophil fraction, platelets, bilirubin, CRP, lactate, creatinine, and albumin) (27th European Congress of Clinical Microbiology and Infectious Diseases, abstract P2055). Creatinine was subsequently eliminated from the model because it did not contribute to the predictive performance.

On the training dataset, the model had an area under the receiver operating characteristic curve (AUROC) = 0.717 (0.691–0.744) for prediction of bacteremia (Fig. E1a in Online Resource 1).

Risk assessment

The goal of the present study is to compare the ability of clinicians to select high-risk patients to that of SFL. Two subgroups were defined to make the comparison: Clinical high risk and SFL high risk; see flow diagram (Fig. 1).

Fig. 1
figure 1

Inclusion/selection flow diagram

Full size image

Clinical high risk

Physicians were requested to include high-risk patients in the RM study (defined as those with a SOFA score of 2 or more, thereby fulfilling the Sepsis-3 definitions [10]) by ordering concurrent blood samples for RM at the time of BC order. Where SOFA was not available, clinical judgment was used as to whether patients were included or not.

SFL high risk

An alternative selection was performed by SFL. We chose to include a number of SFL high-risk patients equal to the number of clinical high-risk patients by choosing the threshold value for high-risk post hoc in order to achieve the same number of patients in each group.

Microbiology

Positive BCs and positive PCR tests were considered clinically significant when contaminants were excluded. Such positive BCs and PCR tests, including those with fungal isolates, will be referred to as bacteremic (BC+) and DNAemic (DNA+), respectively. Bacillus spp. (except B. anthracis), coagulase-negative staphylococci, Corynebacterium spp., and Micrococcus spp. were considered contaminants, in accordance with local policy at AUSL Romagna. In the absence of clinical data indicating the site of infection, viridans streptococci were also considered contaminants, even though they may occasionally be pathogens.

Blood cultures were performed using the BacT/Alert® system (bioMérieux, Marcy l’Etoile, France), and RM with the Iridica-BAC BSI assay (Ibis Biosciences-Abbott Molecular, Des Plaines, IL). The techniques used are described fully in the RM study [11].

Outcomes and statistical analysis

The primary outcome was the rate of BC+. Secondary outcomes were all-cause 30-day mortality (M30) and the rate of DNA+. Outcomes were compared for the clinically and SFL-selected groups.

Dichotomous categorical variables (including the two outcomes) were compared using the binomial test, while continuous variables were compared using the Mann-Whitney U test or Wilcoxon signed-rank test. Difference in survival was computed using the Mantel-Cox log-rank test. Differences in rates across subgroups were tested using Pearson’s chi-squared test or Fisher’s exact test, as appropriate. SFL probability of BC+ was calculated using Hugin Researcher 7.6 (Hugin Expert A/S). Graphical presentations were constructed in Excel 2017 (Microsoft Corporation) and MATLAB R2016a (MathWorks Inc.) and statistical analyses were performed in SPSS 25 (IBM Corporation) and Excel 2017.

A preliminary version of the results in this manuscript was presented at the 27th European Congress of Clinical Microbiology and Infectious Diseases, Vienna, Austria, April 24, 2017 (OS0558B).

Results

The eligible study population included 1395 patients. Following exclusion, BCs were analyzed for a total of 1142 patients (Fig. 1). Of 1142 patients, 384 (33.6%) had a positive BC, and after excluding contaminants 296 (25.9%) were considered BC+. Of 296 BC+, 186 (62.8%) had a single Gram-negative isolate, 84 (28.4%) had a single Gram-positive isolate, and 2 (0.7%) had a fungal isolate, while 24 (8.1%) were polymicrobial (9 with multiple Gram-negatives, 1 with multiple Gram-positives and 14 mixed Gram-negative and Gram-positive). Table 1 presents a breakdown of the BC isolates.

Table 1 Isolates from the 384 positive cultures
Full size table

The infection variables CRP, neutrophil fraction, and platelets were available for > 98% of patients. Bilirubin was recorded for 53% of patients. Albumin and lactate were only available for 0.9% and 0.7% of patients, respectively. Descriptive statistics are presented in Table 2.

Table 2 Descriptive statistics
Full size table

Clinical high risk

Two hundred twenty patients were in the clinical high-risk group. According to the original RM study design, these patients should have had SOFA > = 2 on admission. However, calculating a SOFA score for these patients prior to inclusion was impractical. Retrospectively, the SOFA score was available for 180/220 patients, with 45 patients having SOFA < 2. Among the patients for whom SOFA was available, there was no difference in BC+ (33.3% vs. 31.1%, p = 0.78) or 30-day mortality (13.3% vs. 14.8%, p = 0.81) for patients with SOFA < 2 and with SOFA > = 2. There was a significant difference between the rate of BC+ in the clinical high-risk group compared to the remaining low-risk patients (32.3% vs. 24.4%, p = 0.02) (Fig. 2a). The sensitivity of the clinical high-risk for BC+ was 24.0% (71/296), while the specificity was 82.4% (697/846). The predictive power of inclusion in the clinical and SFL high-risk groups for BC+ prediction is detailed in Table 3.

Fig. 2
figure 2

Clinical and SFL high risk among the 1142 patients. a Comparison of clinical high risk vs. clinical low risk. b Comparison of SFL high risk vs. SFL low risk. c Comparison of clinical high risk vs. SFL high risk. BC+, bacteremia; DNA+, DNAemia; M30, 30-day mortality

Full size image
Table 3 Predictive power of inclusion in the high-risk groups for BC+ prediction
Full size table

SFL high risk

The SFL high-risk group had significantly higher BC+ rate than the SFL low-risk group (45.5% vs. 21.3%, p < 10−12, Fig. 2b). The sensitivity of SFL high risk for BC+ was 33.8% (100/296) and the specificity was 85.8% (726/846). The area under the ROC curve for prediction of BC+ was 0.70 for the study data (Fig. E3a in Online Resource 1). Calibration plots for training and study data can be found in Fig. E2 and E4 of Online Resource 1, respectively.

Figure 2c compares the clinical and SFL high-risk patients. Fifty-five patients were in both the clinical and the SFL high-risk groups (intersecting selections). The clinical and SFL high-risk groups are therefore not independent. To form independent groups, allowing the use of the binomial test for differences between clinical and SFL high-risk groups, we defined three groups which were then compared pairwise. The three groups defined by the overlapping sets are referred to as clinical only, both, and SFL only. The rate of BC+ was higher in the SFL-only group than in the clinical-only group (44.8% vs. 27.3%, p = 0.001), and patients had higher mortality than those in the clinical-only group (29.7% vs. 11.5%, p = 0.00004). There were no significant differences, neither in mortality nor in BC+ rate for SFL-only group compared to the both groups (p = 0.26, p = 0.75, respectively). However, in both groups, the rate of BC+ was significantly higher compared to the clinical-only selection (47.3% vs. 27.3%, p = 0.006), while the difference in 30-day mortality (21.8% vs. 11.5%, p = 0.06) did not reach significance. The DNA+ rate was also significantly higher (52.7% vs. 32.1%, p = 0.006) in both group compared to the clinical-only group.

Survival analysis

Figure 3 shows log-survival curves for patients included in the study. The upper panel (a) presents curves for SFL high-risk and low-risk patients, and the lower panel (b) presents curves for clinical high-risk and low-risk patients. SFL high-risk patients had lower survival than low-risk patients (p < 0.001). There was no significant difference for clinical high-risk vs. low-risk patients (p = 0.98). The patterns in survival remained when stratifying by age; the separation between survival curves for SFL high-risk and low-risk patients was still present (Fig. E5a in Online Resource 1), as was the lack of separation for the clinical high-risk and low-risk groups (Fig. E5b in Online Resource 1).

Fig. 3
figure 3

Survival curves. Upper panel (a) shows SFL high risk and SFL low risk. Lower panel (b) shows clinical high risk and clinical low risk

Full size image

Discussion

The primary finding was that risk assessment provided by SFL significantly outperformed the clinical risk stratification in terms of selecting patients with higher bacteremia rates. These patients may also have a higher DNAemia rate, if it is accepted that the DNAemia rate is linked to the bacteremia rate by a fixed odds ratio. In addition, the clinical selection appeared to fail: although there was a difference in BC+ rate between clinical high risk and clinical low risk, there was no difference between the Clinical only group and the Clinical low-risk. In addition, there was no difference in 30-day mortality for clinical high-risk and low-risk patients.

A strength of SFL is that the prediction of bacteremia seems robust with consistent areas under the ROC curve of 0.72 for the training set (Israel) and 0.70 for the validation set (Italy). SFL also provided a “fair” area under the ROC curve, even though the number of available variables was limited: none of the vital parameters and only three of the six lab values in the original version of SF [7, 8] were available. The absence of lactate in the present dataset was mainly logistical. Clinical practice at the time meant that lactate was measured by blood gas analysis in the hospital and available in the hospital’s patient record, but not in the laboratory information system. It is expected that the addition of both lactate and albumin measurements (particularly for ER patients) would improve predictive performance. Albumin is linked to mortality [12], while lactate is also used as a severity marker or a marker for septic shock [10, 13]. Bilirubin was also added in SFL due to data availability and lack of a measure of liver dysfunction in the model. In practice, the ability to stratify the patients based on lab values alone may be logistically important, since lab values in most hospitals/microbiology laboratories can be drawn automatically from the laboratory information system.

The clinical selection of high-risk patients performed poorly in identifying patients with a higher rate of bacteremia and higher mortality. We believe there are three possible explanations why clinicians were outperformed by SFL; the first explanation is that the mortality-based SOFA score, used according to the Sepsis-3 definitions [10], is unsuitable, or it was not used as intended. The unsuitability of SOFA is supported by the finding that there were no significant differences in mortality and bacteremia in patients with SOFA < 2 and SOFA ≥ 2. This is supported by other studies showing that while the Sepsis-3 criteria provide a good prognostic indication in the ICU [14], they are less effective in the emergency department [15]. However, it is a limitation of the study that clinical data, and thus also SOFA scores, were only collected for the clinically selected patients.

The second explanation is that the SOFA score has not been applied appropriately. It is possible that some patients who were critically ill were not clinically selected due to the severity of their illness, with the clinicians perhaps focused on fluid resuscitation and timely antimicrobial treatment [16] instead of enrolment in the RM study. This would create a selection bias disfavoring the clinical selection, but the study is limited in the sense that it does not provide the clinical data required to confirm or reject this explanation. It is also possible that clinicians may err on the side of caution, partly due to incomplete history and clinical data [17], resulting in an overestimation of the risk of bacteremia [18]. This is backed by the fact that 49 patients with SOFA < 2 were included in the clinical high-risk group.

The third explanation is that the comparison between SFL and the SOFA score is not fair: SFL selects for high bacteremia rates, while the SOFA score selects for high mortality. However, this explanation is incompatible with the finding that SFL selection outperforms the clinical selection with a significantly higher mortality.

It remains unexplained why the rate of positive BCs (33.9%) as well as of bacteremia (26.3%) was high compared to rates reported in the literature (5–15%) [19,20,21] for patients suspected of sepsis. However, the bacteremia rate in this study is consistent with historical averages for AUSL Romagna. Yearly bacteremia rates for cultures drawn in the emergency department ranged from 23 to 25% for 2013–2016 (Report di Sorveglianza Microbiologica AUSL della Romagna 2013–2016, available on request). One possible explanation of the high rate of bacteremia is that blood may not have been drawn for culture in all low-risk patients. However, this does not explain that the observed rate of bacteremia is twice as high as that predicted by SFL (see Fig. E4 in Online Resource 1). Other speculative reasons for the high rate of bacteremia include differences in workflow and/or culture-related processes such as temperature control of the sample or the rate of antibiotic therapy preceding blood sampling, but the exact reasons have not been determined. Improved culture technique would be consistent with the observation that the contamination rate (6.8%) is also about twice as high as those reported in the literature [21,22,23].

The lack of access to clinical data for the patients included in the study also made it difficult to provide a true determination of the contamination rate. It is possible, for example, that a number of the patients, whose blood cultures grew CoNS or viridans streptococci, had an infection where one of these organisms was the causative pathogen.

Since the completion of the study, the RM product has been withdrawn from the market. However, the concept of risk-based stratification remains valid as a selection method for improving the utilization of rapid microbiology. The methods presented here are test agnostic.

A recent review [24] of predictive models for bacteremia included two computerized models. One model was TREAT [20]. SF was originally a component of the decision support system TREAT [25,26,27], which advises on optimal antimicrobial therapy. TREAT requires more data including background information and the physical examination of the patient. The other predictive model was another Bayesian model [28] which focused on hospitalized patients, where many of the input variables would not typically be available in the emergency department. SFL had similar performance characteristics in terms of the AUROC reported for these other models, but with the practical advantage that it only uses data typically available in the emergency department, and that these data are often available in a structured electronic format.

SFL was able to select a group with a higher rate of bacteremia and higher mortality than in the clinically selected high-risk group. Presumably, early appropriate antibiotic treatment is more important in a severely ill group and thus the SFL high-risk group would benefit more from rapid diagnostics than the clinical high-risk patients. A practical implementation of SFL as part of a clinical decision support system for risk stratification may contribute towards improving the selection of emergency department patients with suspected sepsis for rapid diagnostics. Such an implementation should be assessed in a prospective study, where the true benefits can be assessed.