Introduction

In a recent editorial, it was stated that the Pitt bacteremia score (PBS) is the “crown jewel of all severity of illness scores” [1]. The PBS was developed in the 1990s for use in studies of bloodstream infection (BSI) with two objectives in mind: to control for variation in severity of illness (SOI) and to predict mortality [1]. However, the editorial provided no detailed evidence to support their claim that the PBS is the best model for measuring SOI and made no mention of other SOI models that have been utilized in studies of BSI. In fact, the frequency of utilization of specific SOI models in studies of BSI has not been investigated. There has also been discussion from the epidemiological perspective about how to appropriately utilize SOI when determining factors influencing the outcome of BSI but this has been largely “under the radar” [2, 3].

Therefore, the objectives of this review are (a) to identify the SOI models that have been utilized most often in studies of BSI in the past 2 decades, (b) to analyze the development of the commonly used models, and (c) to review utilizing a SOI measure as a potential confounding variable in the analysis of outcome of BSI.

Methods

To identify SOI models used in studies of bloodstream infection, 2 searches were performed using Google Scholar. The first search focused on studies of Gram-negative BSI published in English in adults (≥ 18 years old) between January 1, 2000, and March 23, 2021, using the following search terms in various combinations: bacteremia, bloodstream infection, E. coli, Klebsiella, Pseudomonas aeruginosa, Enterobacter, Proteus, Acinetobacter, community-acquired, hospital-acquired, nosocomial, healthcare-associated, epidemiology, and SOI. A second search focused on studies of S. aureus BSI published in English in adults (≥ 18 years old) between January 1, 2000, and March 23, 2021, using the following search terms in various combinations: Staphylococcus aureus, bacteremia, bloodstream infection, methicillin-resistant, methicillin-susceptible, community-acquired, hospital-acquired, nosocomial, healthcare-associated, epidemiology, and SOI. Studies selected for this review included cohort studies that evaluated the epidemiology of Gram-negative BSI or of Gram-negative organism-specific BSI, or S. aureus BSI and studies of treatment of Gram-negative BSI or organism-specific BSI or S. aureus BSI. Studies dealing with specific patient populations (oncology, hemodialysis, transplantation, human immunodeficiency virus, burns, pediatrics), outbreaks, abstracts, or letters to the editor were excluded. Also excluded were studies of S. aureus BSI or Gram-negative BSI that included other organisms (polymicrobial infection). The list of studies of Gram-negative and S. aureus BSI included in this review along with the results of the use of SOI models can be found in Supplemental Appendices 1 and 2, respectively.

Results

Utilization of Severity of Illness Models in Studies of Gram-Negative and S. aureus BSI, January 1, 2000–March 23, 2021

After literature review, 955 studies of Gram-negative and S. aureus BSI were identified that met inclusion criteria (Supplemental Appendices 1 and 2). The SOI models identified in these studies are listed in Table 1. Overall, 555 (58%) of the 955 studies utilized a SOI measurement. In the 555 that utilized a SOI measurement, the PBS was used alone or in combination with another SOI model (including the APACHE system [4, 5]) in 51% (N = 283), the APACHE score alone or in combination with another SOI model (excluding the PBS) in 40% (N = 223), and other SOI models alone in 9% (N = 49). Overall, the PBS or APACHE model was used in 91% of the 555 studies. Based on these findings, the remainder of this review will focus on the PBS and the APACHE model.

Table 1 Utilization of severity of illness models in studies of Gram-negative and S. aureus bloodstream infection, January 2000-March 2021
Full size table

Pitt Bacteremia Score

Evolution of the Pitt Bacteremia Score

The infectious diseases group at the University of Pittsburgh in collaboration with other investigators published 5 studies between 1991 and 1999 that describe the development of the PBS (Table 2) [6,7,8,9,10]. A prospective study of Pseudomonas aeruginosa bacteremia in liver transplant patients at the University of Pittsburgh was conducted to define risk factors for this infection and outcome [6]. The investigators developed a “degree of illness” model consisting of 5 components with a score assigned for each component (Table 2). Patients were considered critically ill if their score was ≥ 4 within 72 h before a positive blood culture. There was no explanation as to why these particular components were chosen but three had been utilized in an earlier study [11]. There was also no explanation for how the score for each of the components was determined, or how it was decided that a score of ≥ 4 defined critical illness. Because there were only 23 episodes of Pseudomonas aeruginosa bacteremia, it was not possible to make a valid assessment of this model for predicting mortality. In a multicenter, prospective evaluation of 129 episodes of Enterobacter bacteremia, the temperature component was modified and hypotension was undefined but there was no explanation for these changes (Table 2) [7]. Also, the timing of applying the model was changed without explanation. Severe illness was again defined as a score ≥ 4 points without explanation.

Table 2 Evolution of the Pitt Bacteremia Score in Studies published by the University of Pittsburgh investigators, 1989–1999
Full size table

In a multicenter, prospective study of 230 episodes of Klebsiella bacteremia, there was modification of the temperature and hypotension definitions of the model without explanation (Table 2) [8]. The timing of applying the model reverted back to within 72 h of the first positive blood culture. In multivariate analysis, after controlling for combination antibiotic therapy versus monotherapy and pneumonia as a cause of bacteremia, hospital-acquired infection and SOI score ≥ 4 were significant predictors of 14-day mortality. A multicenter, prospective study was conducted to evaluate the morbidity and mortality of 427 episodes of candidemia [9]. The criteria for temperature and hypotension were changed with no explanation (Table 2). The timing of applying the model was changed to the day of the first positive blood culture with no explanation. Among 369 patients treated for candidemia, the mortality was significantly higher with critical illness (35%; 75/214) compared to those without critical illness (15%; 24/155; p < 0.001 by chi square).

Lastly, in a literature review of combination antibiotic therapy versus monotherapy of Gram-negative bacteremia, the Pittsburgh investigators referred to their SOI model as the “Pitt bacteremia score” [10]. As shown in Table 2, the temperature component had more subcategories than previous versions of the PBS, and there were 3 definitions for hypotension but there was no explanation for these changes. The timing of applying the model was different from all previous models. The PBS model in this paper appears to be the last version to be published.

In summary, analysis of the development of the PBS found no explanation for how the components of the model were chosen or why changes in definitions of these components were made. The point system established for the components of the PBS was empirical and not based on statistical analysis. Because there was no statistical analysis, the cutoff for critical illness (point score ≥ 4) was also empirical and there was no way to determine which of the 5 components is the most important in terms of measuring SOI in these 5 studies [6,7,8,9,10]. In none of the 5 studies was there an explanation for how the cutoff for defining critical illness (score ≥ 4) was determined. Also, there was variation in the timing of when to measure the SOI among the 5 studies listed in Table 2.

Studies of the PBS by Independent Investigators

Three studies [12,13,14] of BSI provide informative examples of independent assessment of the PBS as a measure of SOI. A prospective study of 424 cases of S. aureus BSI to determine factors affecting incidence and mortality used the PBS and the Acute Physiology and Chronic Health Evaluation (APACHE) II system [4] to measure SOI [12]. The APACHE system had been previously utilized to measure SOI in patients with S. aureus BSI [15, 16]. However, the authors did not compare the discrimination of these two models for predicting 30-day mortality. Using the data provided in the paper and receiver operating characteristic (ROC) curve analysis [17], the area under the ROC curve for APACHE II was significantly higher (O.798; p = 0.045; 2-tailed) than for the PBS (0.704) indicating better discrimination in predicting death due to BSI with APACHE II (Supplemental Appendix 3). Importantly, the ROC curve analysis for the PBS indicated that a score ≥ 2 was the appropriate cutoff for defining critical illness in this study. The latter finding illustrates that an arbitrary cutoff (≥ 4 points) used in the PBS for defining severely ill patients may not be appropriate for all study populations.

In the editorial referred to in the introduction [1], an alternative model for predicting mortality related to BSI was mentioned, the Bloodstream Infection Mortality Risk Score or BSIMRS. Development of the BSIMRS was described in a retrospective study of 683 episodes of Gram-negative BSI [13]. The objective was to determine, after controlling for SOI using the PBS, if other clinical or laboratory factors were predictive of 28-day mortality. The results of this study are important because a statistical analysis was utilized to develop the scoring system for the BSIMRS. Using logistic regression analysis, independent predictors of 28-day mortality in the study population were defined (PBS plus 3 clinical factors: malignancy of any type, cirrhosis, and a non-urinary/non-intravenous catheter-related source of BSI) and the score of each component of the model was based on the regression coefficient including a score for each level of the PBS. The BSIMRS model had an area under the ROC curve of 0.84 indicating favorable discrimination between those who died and survived. However, the authors did not determine the area under the ROC curve for PBS alone to compare with the results for BSIMRS. Using data available in the paper, the area under the ROC curve of the PBS alone was calculated to be 0.72 (data not shown). Thus, the discriminatory power of the BSIMRS was substantially greater than that of the PBS alone for predicting mortality in the study population. The BSIMRS was validated in 4 studies of Gram-negative BSI [18,19,20,21], but at the time of writing of this paper, there has been minimal use of the BSIMRS in studies of BSI.

The most detailed evaluation of the PBS was performed to determine if it predicted mortality in patients with nonbacteremic carbapenem-resistant Enterobacteriaceae (CRE) infection compared to those with BSI due to CRE [14]. For both CRE BSI and nonbacteremic infection, mortality increased substantially when the PBS was ≥ 4 verifying that this was the appropriate cutoff for defining critical illness in the study population. For all infections combined, BSI alone, or nonbacteremic infection alone, a PBS ≥ 4 was significantly associated with 14-day mortality. Also, each of the 5 components of the PBS was significantly associated with 14-day mortality when analyzed separately. However, in a multivariate analysis with all components of the PBS included individually, the temperature component was not significantly associated with 14-day mortality. After the temperature component was dichotomized (< 36 °C or ≥ 36 °C), it was a significant predictor in a multivariable model. However, addition of the dichotomized temperature component did not significantly improve the predictability of the PBS by ROC curve analysis. This was the first time that the PBS had undergone a careful statistical analysis, and it was determined that the temperature component was not needed in the model.

APACHE Scoring System

Evolution of the APACHE Model

The APACHE model was initially described in 1981 with the objective of providing a method to accurately measure SOI in patients hospitalized in the intensive care unit (ICU) setting with a variety of diseases [22]. As originally developed APACHE consisted of two parts. The first part was the acute physiology score or APS which was the weighted sum of 33 measurements that indicated the level of physiologic derangement of seven major organ systems: neurologic, respiratory, cardiovascular, gastrointestinal, renal, hematologic, and metabolic. The second part was the chronic (pre-admission) health evaluation that consisted of 4 categories of health: good, mild to moderate limitations, severe limitations, and serious restriction of activity. This model was validated in 833 nonoperative patients admitted to one ICU [23]. In this study, there was a significant association between the APS of the APACHE and hospital survival. However, independent of the APS, age was also found to be a significant predictor of survival and this resulted in adding age as part of the chronic health evaluation in later APACHE models. Noteworthy is that the authors stated that even though their study focused on patients admitted to an ICU, the APACHE model could also be applied to non-ICU patients.

A revision of the initial APACHE model, APACHE II, was published in 1985 [4]. The original APACHE model contained 33 physiologic measurements, but after experience with using the model, some measurements were infrequently utilized (e.g., serum osmolarity, skin testing for anergy). In the revised model, the number of physiologic measurements was reduced to 12, points were specifically assigned to age categories, and the chronic health evaluation was revised. It is important to note that the scoring system established for both the original APACHE model and APACHE II was empiric, i.e., based on the consensus of critical care experts.

A second revision of the APACHE model, APACHE III, was published in 1991 that further refined the model [5]. For this revision, the weights for the physiologic measurements were estimated based on the results of multivariate analysis of 17,440 unselected medical/surgical ICU admission in 40 US hospitals. Age points were also revised based on a statistical analysis of the data collected, and the chronic health evaluation was also revised to include 7 specific diseases (AIDS, hepatic failure, lymphoma, metastatic cancer, leukemia/multiple myeloma, immunosuppression, cirrhosis). The latest version of the APACHE model, APACHE IV, was published in 2006 [24], and further refinements were made to the scoring system that improved the discrimination and accuracy of the model.

Is the APACHE Model Applicable to Non-ICU Patients With BSI?

It has been argued that the APACHE model is only applicable to the ICU setting and does not apply to non-ICU patients (2, 3). Of the 3 components of the APACHE model, APS, age, and chronic health evaluation, the most applicable for use in patients with BSI is the APS. In APACHE III, the APS has 15 vital sign and laboratory abnormalities and separate scoring for acid–base disturbances and neurologic abnormalities. Each of the vital sign and laboratory abnormalities are assigned weights for levels above and below a normal range (Supplemental Appendix 4). The importance of the APS is two-fold. First, most of the vital signs and laboratory abnormalities apply to all hospitalized patients regardless of location, ICU or non-ICU. Secondly, the weights assigned to each parameter of the APS in the APACHE III model have been determined based on a statistical analysis and, therefore, are cardinal numbers that permit analysis with descriptive statistics (mean, median, standard deviation). This is in contrast to the PBS in which the weights are ordinal and not appropriate for analysis with descriptive statistics.

Modification of the APACHE Model in Studies of BSI

Despite the potential usefulness of the APS of APACHE III in all hospitalized patients, there are limitations when it is applied to non-ICU patients. The main limitation is that some of the parameters of the APS are not frequently measured in non-ICU patients, e.g., A-aDO2, arterial pH, and neurological abnormalities (Supplemental Appendix 4]. This limitation was recognized and resulted in modification of the APS by eliminating the parameters that were not usually measured in non-ICU patients [25,26,27]. However, there needs to be standardization of the measurements that are eliminated.

In summary, the APACHE model has been used frequently to account for SOI in studies of BSI. The APS of the model may be most appropriate for measuring SOI rather than using the entire model. The APACHE II and III models include an age component and a separate chronic health evaluation but these factors should be analyzed separately when utilized in studies of BSI rather than aggregated with the APS because this could mask the importance of age and various underlying diseases as independent confounding variables. Because the scoring system for the APS of the APACHE III model is based on an extensive statistical analysis, it is the preferred version for use in studies of BSI.

Severity of Illness as a Confounding Variable in Studies of Bloodstream Infection

In the past 2 decades there has been discussion about whether or not one should consider SOI a confounding variable when determining factors influencing the outcome of BSI [2, 28]. This discussion has focused primarily on the APACHE model although it applies to the PBS or any other SOI model. The concern is that SOI at the time of diagnosis of BSI is in the causal pathway of infection, i.e., it is a result of the infection [28]. From an epidemiological perspective, because SOI at the time of a positive blood culture is in the causal pathway, it is considered an “intermediate variable” when measured at that time point and not a confounding variable when evaluating for factors predicting outcome of BSI [29].

Nevertheless, it is recognized that SOI is important to control for in studies of outcome of BSI [2, 28]. Because SOI is considered an intermediate variable, it has been recommended that it should be measured prior to the onset of BSI [2, 28]. For example, in a systematic review of methods used to evaluate the association between appropriate antibiotic therapy and mortality in patients with BSI, it was recommended to measure SOI 48 h before the initial positive blood culture [2]. However, the timing of measuring severity of illness in studies of BSI has not been evaluated to determine how often this recommendation has been followed. Therefore, references in Supplemental appendices 1 and 2 that used the PBS or APACHE model were reviewed to determine the timing of measuring SOI (Supplemental Appendix 5). The PBS was measured on the day of the first positive blood culture in 25% of 283 episodes, before the day of the first positive culture in 7%, and timing was not stated in 58% of the studies. The APACHE model was measured on the day of the first positive blood culture in 37% of 267 episodes, before the day of the first positive culture in 21%, and timing was not stated in 42% of the studies. Thus, in only a small percentage of studies of BSI was SOI measured before the day of the first positive blood culture. However, there was a high percentage of studies in which it was not stated when the SOI measurement was performed relative to the day of the first positive blood culture.

Several studies were identified that compared the analysis of outcome predictors when the APACHE measurement was done at a designated time before or at the onset of BSI to determine if there was any variation in the findings [3, 15, 30,31,32,33,34]. Review of these 8 studies found that they varied in the size of the study population, in the organisms causing BSI, and the APACHE model utilized. Thus, given the variation in study design of these studies, the optimum time to determine SOI prior to BSI documentation remains unclear.

Discussion

This review documented that the PBS and the APACHE model were the most common SOI models used in 555 studies of Gram-negative and S. aureus BSI that measured SOI. Based on these findings, this review analyzed the development of these two models and evaluated their usefulness in measuring SOI in studies of BSI.

In an analysis of the development of the PBS and of studies using the PBS by independent investigators, it was found to be an empiric model with no statistical basis and had several other important limitations as discussed in this paper. In reviewing the development of the APACHE system, one finds that the initial version [22] and APACHE II [4] were empiric models developed for use in patients admitted to the ICU setting. A subsequent revision, APACHE III, provided a sound basis for the scoring system based on a statistical analysis of > 17,000 ICU patients [5]. Of the three components of the APACHE model, the APS may be the most useful for measuring SOI in patients with BSI and is applicable for use in non-ICU patients. However, a limitation of the APS is that some of the vital sign and laboratory findings may not be available for non-ICU patients. Additional studies modified the APS by eliminating infrequently used components but there needs to be standardization of the vital sign and laboratory tests used in a modified version. Based on the findings of this review, it is not possible to determine if the PBS or APACHE model is the preferred method for measuring SOI. However, the weaknesses of the PBS argue against it being considered the best model for measuring SOI in studies of BSI.

Regardless of which model is utilized, the issue of timing of measuring SOI in studies of BSI is important and needs to be standardized. From an epidemiological perspective, measuring SOI at the time of the first positive culture represents the effect of infection and is considered an intermediate variable and not a confounding variable [29]. However, as shown in this paper, in a substantial percentage of BSI studies, SOI was measured on the day of the first positive culture, and in some of these studies, this measurement was used as a confounding variable in analyses of outcome. Adjusting for an intermediate variable (SOI measured at the time of a first positive blood culture) may bias the relationship between an exposure (BSI) and outcome (mortality) [28].

To deal with the problem of timing of measuring SOI, it has been recommended that it should be done prior to identification of BSI [2]. Several studies of BSI were identified that evaluated various times for measuring the APACHE score (on admission to the hospital, 24–72 h before a positive culture, and the change in APACHE score over time). However, the findings of these studies were inconclusive for determining the best time to measure SOI. Choosing the time point before a positive blood culture when a patient is at “baseline” SOI is, at best, an educated guess. Patients may have infection present from a few hours to several days before a blood culture is done. Another approach that has had limited study but should be investigated further is using the change in APACHE score between two time points, e.g., the score on the day of first positive culture minus the score one day before a positive culture [15]. For patients with community-acquired BSI, measuring SOI prior to identification of a positive blood culture is not possible. In this situation, there are two potential ways to approach analyzing the effect of SOI on outcome. In studies that include both hospital- and community-acquired BSI, it has been suggested that these groups be analyzed separately in terms of outcome [2]. In studies focusing only on community-acquired BSI, it has been suggested that outcome analyses be compared with and without the SOI measure as a confounding variable to determine the predictors of mortality accurately [28].

Conclusions

At the present time, based on the analysis in this review, there is no one best model for measuring SOI in studies of BSI. The PBS, the modified APS of APACHE III, or other models such as the BSIMRS can be utilized for measuring SOI. Clearly, SOI is important to control for in studies of BSI but how this is accomplished should be consistent with accepted epidemiological principles. Investigators need to be aware that the timing of measuring SOI in studies of BSI determines whether this parameter functions as an intermediate or confounding variable. Thus, timing of the measurement of SOI may be as important or more important as the SOI model utilized. Investigators need to be aware of this issue when designing studies of BSI in which outcome is being evaluated as do peer reviewers and journal editors who are evaluating studies for publication. However, the optimum time for measuring SOI prior to identifying BSI has not been determined. Future studies of BSI should utilize SOI models that have been developed based on sound statistical analysis and identify the optimum time to apply a model in order to provide valid information.