Introduction

Acute kidney injury (AKI) occurs in more than 30% of ICU admissions [1, 2], and increases the risk of persistent renal dysfunction and need for long-term renal replacement therapy (RRT) after discharge [3, 4]. Despite the well-recognized need to prevent AKI, the disease itself has been variably defined by the Risk Injury, Failure, Loss of kidney function, Endstage disease (RIFLE) criteria [5], Acute Kidney Injury Network (AKIN) criteria [6], and Kidney Disease: Improving Global Outcomes (KDIGO) criteria [7]. Variability in outcome definitions have made endpoint selection for clinical trials challenging. A recent shift toward targeting patient-centered renal outcomes in clinical research has led to the development and use of the Major Adverse Kidney Events within 30 days (MAKE30) endpoint [8]. Analogous to “major adverse cardiac events” [9], MAKE30 is a composite endpoint of mortality from any cause, need for renal replacement therapy, and persistent renal dysfunction [8, 10].

Accurately predicting renal outcomes may facilitate both patient care and study of interventions. Recognition of patients at high risk for renal outcomes early in a hospitalization could allow providers to avoid risk factors (e.g., nephrotoxic drugs [11] and radiocontrast dye [12]). Additionally, identifying patients at risk for an outcome may be a useful tool to enrich clinical trial enrollment with high-risk patients to reduce sample size [8, 13] or assess for heterogeneity of treatment effect [14].

We sought to develop a predictive model using variables present at ICU admission and available in the electronic health record to accurately predict the clinical outcome MAKE30.

Methods

Patients

This retrospective cohort study was approved by the Vanderbilt Human Research Protection Program with waiver of informed consent (IRB #1413349). The cohort consisted of all adults ≥18 years admitted to the medical, cardiovascular, neurological, trauma, or surgical intensive care units at Vanderbilt University Medical Center between January 1, 2014 and December 31, 2014. Patients discharged from the hospital and subsequently readmitted to an eligible ICU during the study period were included as a separate admission; ICU readmissions without an intervening discharge from the hospital were excluded.

Endpoints

The primary outcome was Major Adverse Kidney Events to 30 days (MAKE30), defined as in-hospital mortality, new RRT, or persistent renal dysfunction [8, 15]. In-hospital mortality was defined as death from any cause prior to hospital discharge, censored at 30 days. New renal replacement therapy was defined as receipt of RRT at any point between ICU admission and hospital discharge, censored at 30 days. Persistent renal dysfunction was defined as a creatinine at discharge or day 30 ≥ 200% of the baseline serum creatinine value. Patients who had received RRT prior to ICU admission could meet the MAKE30 endpoint via mortality but not the RRT or creatinine criteria. Identifying the MAKE30 outcome from EHR data has been previously validated at the study institution with a kappa of 0.95 compared with manual chart review [15].

Data collection and definitions

All data were electronically abstracted from the Vanderbilt enterprise data warehouse (EDW), which contains data exported daily from the electronic health record paired with billing ICD-9 and ICD-10 codes, patient registration system data, and laboratory clinical information system data. As the goal of the model was prediction of renal outcomes of critical illness, we limited the potential variables to include only those present at or before ICU admission (or within 24 h of admission in the case of receipt of vasopressors or mechanical ventilation). Demographic variables including age, gender, race, and body mass index (BMI) were obtained from the admitting system data. The Vizient University Health System Consortium (UHC) provides an estimated mortality for inpatient encounters derived from simple demographics and principle diagnosis based on the Medicare Severity-Diagnosis Related Groupings (www.vizientinc.com). These mortality estimates were extracted from our EDW. ICD-9 and ICD-10 codes, restricted to those signifying comorbidities present on admission, were used to generate the Elixhauser comorbidity index [16, 17].

In addition to the Elixhauser comorbidities, one of which is renal failure, we separately defined baseline creatinine, chronic kidney disease, and acute kidney injury as previously described [18]. Baseline serum creatinine was defined hierarchically as (1) the lowest value measured between 12 months and 24 h prior to hospital admission whenever available, (2) the lowest value between 24 h prior to hospital admission and the time of ICU admission when no pre-hospital value was present, and (3) an estimation from a previously described formula [19] [creatinine = 0.74–0.2 (if female) + 0.08 (if African American) + 0.003 × age (in years)] when no measured values were available. Acute kidney injury stage II or higher was defined by the Kidney Disease Improving Global Outcomes (KDIGO) creatinine criteria [7] as a serum creatinine at least 200% of the baseline value OR (a) greater than 4.0 mg/dL and (b) increased at least 0.3 mg/dL from the baseline value. Chronic kidney disease stage III or greater was defined as a glomerular filtration rate less than 60 ml/min per 1.73 m2 using the Chronic Kidney Disease Epidemiology (CKD-EPI) Collaboration eq. [20] and the patient’s baseline creatinine value.

Receipt of RRT was determined electronically as previously reported [15] by Current Procedural Terminology and International Classification of Disease codes (Appendix A). When present prior to the ICU admission, the code flagged the patient as previous RRT receipt and made them eligible for the mortality endpoint only. When the code occurred between ICU admission and hospital discharge (censored at 30 days), and the patient was not known to have previously received RRT, this constituted meeting the new RRT component of the MAKE30 endpoint.

Statistical analyses

We fit a logistic model to predict MAKE30 using a set of risk factors selected a priori. Risk factors included age, UHC expected mortality, baseline creatinine, quantity of isotonic crystalloid fluid received in the prior 24 h, race, gender, admission service, ICU, source of admission, non-invasive or invasive mechanical ventilation within 24 h of ICU admission, receipt of vasopressors within 24 h of ICU admission, RRT prior to ICU, AKI at ICU admission, CKD as defined by baseline creatinine value, and renal failure as defined by Elixhauser index. We allowed age, UHC expected mortality, and baseline creatinine to have a nonlinear relationship with the outcome using restricted cubic spines with 3 knots. Complete case analysis was performed. The model was calibrated by internal bootstrap with 300 repetitions. A post hoc model was developed excluding those factors which are not available in real time (UHC expected mortality and Elixhauser index renal failure) as a sensitivity measure. All statistics were done with R version 3.3.0 (Foundation for Statistical Computing, Vienna, Austria).

Results

The full cohort consisted of 11,572 patients, of which 524 and 70 were excluded for missing UHC mortality data and source of admission, respectively. The analysis cohort consisted of 10,983 patients. Median age was 58 years. Fifty-nine percent were male, and 82% were white. A total of 46.9% were admitted to the ICU from the emergency department, 23.9% from the operating room, 13.6% from another hospital, and 9.1% from the hospital ward. Twenty-eight percent of patients were mechanically ventilated, and 22% were receiving vasopressor medications. AKI was present on ICU admission for 10% of patients, and 19% had CKD. Additionally, 3.7% had received RRT prior to ICU admission. Baseline serum creatinine was available from more than 24 h prior to hospital admission in 52% of patients, 24 h before hospitalization to time of ICU admission in 37%, and calculated for 11%. Additional demographic information is summarized in Table 1 and Appendix B.

Table 1 Baseline characteristics
Full size table

The MAKE30 endpoint was met by 1489 patients (13.6%): 967 for in-hospital mortality, 296 for new RRT, and 662 for persistent renal dysfunction, with some qualifying by more than one criterion (Appendix C). The logistic model had a bootstrap-corrected area-under-the-curve (AUC) of 0.903 (Fig. 1). After bootstrap calibration, mean absolute error was 0.017, meaning the average difference between predicted probability and actual probability of MAKE30 across the range of probabilities was 1.7%, indicating high precision. The calibration curve is depicted in Fig. 2. The odds ratio (OR) for risk factors in the model are summarized in Table 2 and plotted in Appendix D. For the full equation of the model, see Appendix E.

Fig. 1
figure 1

Receiver-Operator Curve of the MAKE30 Prediction Model. After bootstrap correction for overfitting, AUC was 0.903

Full size image
Fig. 2
figure 2

Bootstrap-Validated Calibration Curve of MAKE30 Prediction Model. Bootstrap overfitting-corrected calibration curve estimate for the MAKE30 logistic model

Full size image
Table 2 Summary of multivariable model for MAKE30
Full size table

The most predictive variables in the model, as assessed by chi-squared minus degrees of freedom, were UHC expected mortality, AKI at baseline, RRT prior to ICU admission, and Elixhauser renal failure. Presence of AKI at baseline increased the odds of MAKE30 by 4.98 (95% CI 4.12 to 6.03), and Elixhauser-defined renal failure increased the odds by 3.22 (95% CI 2.64 to 3.94). RRT prior to ICU admission was significantly predictive of not experiencing MAKE30, with an OR of 0.07 (95% CI 0.05 to 0.1). Baseline creatinine was also independently predictive, with higher baselines predicting fewer outcomes, OR of 75th:25th percentile of 0.68 (95% CI 0.6 to 0.76).

The post hoc model excluding factors not available for real time prediction (UHC expected mortality and Elixhauser-defined renal failure) retained significant discrimination, with an AUC of 0.832.

Discussion

Our study demonstrated that a logistic regression model using data present at ICU admission has good predictive capacity for the MAKE30 outcome. Our model showed very strong discrimination between patients who will and will not experience MAKE30, with bootstrap-corrected AUC of 0.903. Moreover, the validation curve demonstrates strong precision with bootstrap-corrected mean absolute difference in predicted versus actual probably difference of 1.7%. The findings have important implications for targeted prevention of AKI in the ICU, risk adjustment, and clinical trial conduct.

The risk factors found to be most strongly predictive of the MAKE30 outcome in our model are largely intuitive. UHC predicted mortality is a well-validated predictor of mortality among inpatients, and since one component of the MAKE30 outcome is in-hospital mortality, it is logical that UHC predicted mortality performs well at identifying patients likely to experience the MAKE30 endpoint. Similarly, those patients with AKI or renal dysfunction at ICU admission have been shown in prior studies to have increased risk for progression to RRT [21, 22], another component of the MAKE30 outcome. CKD defined by baseline creatinine was not independently predictive of MAKE30 in our study, but the presence of the Elixhauser renal failure variable and other covariates may have reduced CKD’s independent impact in the model. Because patients who were receiving RRT prior to ICU admission were ineligible for the new RRT or persistent renal dysfunction components of the MAKE30 composite outcome and could only qualify by experiencing in-hospital mortality, RRT prior to ICU admission was associated with decreased odds of the overall MAKE30 outcome. Similar to the findings of two recent investigations, race was not independently predictive [21, 23] for adverse kidney events in our model. Female sex demonstrated a lower odds ratio for MAKE30 in our model (OR 0.84), consistent with older studies [24, 25], though some recent studies did not report gender to be independently predictive of acute kidney injury [21, 23].

Our investigation is strengthened by the large sample size, exceeding 10,000 patients, which increases the reliability of the model predictions and improves their stability across populations. Moreover, we selected factors to include in the model a priori. Stepwise or machine-learning methods can increase the apparent model performance characteristics, but may lead to unstable models that are less likely to validate on new data sets [26]. To confirm the stability of our model, we performed bootstrap validations which demonstrated very robust discrimination (AUC of .903) and precision (mean absolute error 1.7%).

Our model also has limitations. By including all the variables that were chosen a priori, there are some parameters in the model that have limited added benefit (isotonic fluids administered in the preceding 24 h, race, and CKD). While a simplified model could be developed with their elimination, we retain them here recognizing that the full a priori defined model is more likely to validate to a new population. Despite development in a large cohort, all patients are from a single institution. Wider use would require external validation. The model adheres to the statistical principle of only including data in the prediction that are present at admission; however, because the model incorporates data that are derived from billing codes, the model is only able to calculate a risk prediction post-hoc when the billing codes are incorporated into the data. The pre-specified model, as derived, cannot be available for clinical use, but the predictions remain useful to risk-adjust admissions for quality metrics or clinical trials analysis. The post-hoc model excluding variables not available in real time, however, retained significant discriminative capacity, suggesting a reduced model without UHC expected mortality and Elixhauser renal failure could be employed for clinical prediction.

Conclusion

The composite MAKE30 outcome can be reliably predicted with good discrimination and precision by data present at the time of ICU admission. The derived model will be useful to evaluate the effect of interventions on the MAKE30 outcome.