Introduction

Postoperative pulmonary complications (PPCs) occur in up to 30% of patients undergoing non-cardiac surgery and are a significant contributor to the overall risk of surgery [1,2,3]. PPCs substantially increase the risk of postoperative mortality [4] and contribute to increased hospital costs [5]. Patients undergoing major emergency abdominal surgery have a 20% risk of suffering a PPC in the postoperative period [6], which is why identifying patients at high risk of PPCs and focusing on optimising care pathways for these patients is essential [7].

The Assess Respiratory Risk in Surgical Patients in Catalonia (ARISCAT) score was developed in 2010 based on prospectively collected data from 59 Spanish hospitals. The score predicts the in-hospital risk of PPCs based on seven pre- and intraoperative variables [8]. The ARISCAT model was developed for patients undergoing scheduled or emergency surgery in general or regional anaesthesia, excluding patients undergoing obstetric procedures or organ transplantation. A preoperative risk score model to predict PPCs in a high-risk surgical population undergoing major emergency abdominal surgery could support clinical decision making and assist clinicians in identifying patients in need of perioperative optimisation and specific postoperative initiatives to improve outcomes.

This study aimed to externally validate ARISCAT in a cohort of patients undergoing major emergency abdominal surgery at a Danish University Hospital.

Methods

The study was approved by the Danish Data Protection agency (no: REG-042-2017). The study did not qualify for ethics approval by Danish law as no intervention was carried out. The reporting of this study adheres to the TRIPOD (transparent reporting of a multivariable prediction model for individual prognosis or diagnosis) statement [9].

Data source and study population

All patients who underwent major abdominal surgery (emergency laparotomy or laparoscopy, Supplementary Table 1) at the Department of Surgery at Zealand University Hospital, between 1st March 2017 and 31st January 2019 were included. The Department of Surgery at Zealand University Hospital is a referral University Hospital with regional specialised functions in emergency surgery and colorectal cancer surgery. All patients were treated with a standardised pre-, intra- and postoperative bundle of care focusing on reducing time-to-surgery, simplifying logistical barriers and implementing best practice of care [10]. A postoperative physiotherapeutic assessment and standardised nutritional initiatives were part of the standardised bundle. All data were collected prospectively.

Table 1 ARISCAT risk score equation
Full size table

Outcome and predictors

The variables included in the ARISCAT score are age, preoperative SpO2, respiratory infection within the last month, preoperative anaemia, surgical incision, duration of surgery and type of surgery (emergency procedure or elective procedure). The outcome measure of ARISCAT was postoperative pulmonary complications (PPCs). According to the original ARISCAT publication [8] a PPC was defined and grouped into respiratory infection, respiratory failure, pleural effusion, atelectasis, pneumothorax, bronchospasm and aspiration pneumonitis. The authors of the ARISCAT development article [8] were contacted for the ARISCAT risk score equation. This equation was then used to calculate an exact percentage of risk for each patient. The linear predictor for an individual, reflecting the seven presurgical predictors, was calculated according to Table 1.

Statistics

The distribution of continuous data were assessed by visual inspection of histograms. Categorical data were presented as the number of cases and percentages. The outcome of interest for the statistical analyses was postoperative pulmonary complications. Three statistical methods were applied to compare observed and estimated PPC: calibration, discrimination and accuracy. Calibration was assessed with Hosmer–Lemeshow goodness-of-fit, which investigates whether observed event rates are equal to expected rates. Patients were divided into deciles based on ascending order of estimated risk of PPCs. A p value > 0.05 indicated no significant difference in observed and estimated PPCs and thereby a good fit of the model. Furthermore, we assessed calibration by calculating slope and intercept of a calibration plot, where the diagonal line indicates perfect calibration.

Discrimination was assessed by the area under (AUC) a receiver operating characteristic curve (ROC). An AUC between 0.7 and 0.8 was considered fair discrimination, 0.8–0.9 as good discrimination and 0.9–1.0 was considered excellent discrimination. Discrimination describes the models ability to discriminate between those with and those without a PPC. The accuracy of the prediction model was assessed with a Brier score where 0 indicates a perfect prediction and 1 indicates inferior prediction.

IBM SPSS Statistic 27 for Windows was used to compute the statistics.

Results

The study included a total of 585 patients with a median age of 70 years. The majority of patients underwent laparotomy without bowel resection. Patient characteristics are presented in Table 2. A PPC occurred in 211 (36.1%) patients. The most common PPC was respiratory infection, which occurred in 120 (20.5%) patients, followed by respiratory failure (53 patients, 9.1%), pleural effusion (32 patients, 5.5%), atelectasis (3 patients, 0.5%), pneumothorax (2 patients, 0.3%) and bronchospasm (2 patients, 0.3%). Respiratory comorbidities were present in 15.2% of the patients and 22.7% were active smokers. The mean time from surgery to first PPC was three days (IQR 1 days–6 days). The in-hospital mortality was significantly higher for patients with PPCs when compared with patients without PPCs (30.8% (n = 65) vs. 7.8% (n = 29), p < 0.001).

Table 2 Demographics and clinical characteristics
Full size table

Calibration, discrimination and accuracy

PPCs occurred in 36.1% of the patients, while the ARISCAT score predicted a PPC frequency of 24.9%. The overall observed over estimated PPC frequency rate was 1.5. The calibration plot is presented in Fig. 1. The slope of the calibration plot was 0.9546 with a y axis interception at 0.1269 and the plot was well fitted to a linear slope. The Hosmer–Lemeshow goodness-of-fit analysis showed a good calibration with p > 0.25. ARISCAT showed good discrimination with AUC 0.83 (95% CI 0.79–0.86) (Fig. 2) and good accuracy with a Brier score of 0.19.

Fig. 1
figure 1

Calibration plot in an external validation of ARISCAT. Every dot represents a decile of risk. Slope = 0.9546 and interception = 0.1269

Full size image
Fig. 2
figure 2

Receiver-operating characteristic curve. AUC 0.83 (95% CI 0.79–0.86)

Full size image

Discussion

This study found that ARISCAT is a promising model to predict PPCs in a high-risk surgical population undergoing major emergency abdominal surgery. The model had a satisfactory ability to discriminate between patients with and without PPCs. Furthermore, the models performed well in predicting the correct risk of getting a PPC within all risk groups.

When ARISCAT was developed, it showed good discrimination in the original validation cohort with AUC 0.88 [8]. Our study demonstrated an AUC of 0.80, which is impressive given the selected high-risk patients in our cohort. Calibration was not evaluated in the original validation cohort. To our knowledge ARISCAT has only been externally validated previously once in the prospective PERISCOPE cohort (5099 patients), which consisted of a surgically broad international patient population [11]. ARISCAT showed good discriminative abilities with AUC 0.80, however, a poor calibration and clinical usefulness was found. In the PERISCOPE cohort, the ARISCAT score underestimated the risk of PPC. This was attributed to a difference in case mix between the development cohort and the PERISCOPE validation cohort. The PERISCOPE cohort is a broad mix of surgical procedures were only 26.7% of the included patients underwent abdominal surgery. ARISCAT has not previously been validated in a cohort of patients only undergoing major emergency abdominal surgery. In our cohort of patients all undergoing major emergency abdominal surgery, we found that the risk of PPCs was well estimated in all patient groups. The PERISCOPE study suggested that ARISCAT is better applicable in a western European cohort when compared with an eastern European and Spanish cohort [11].

To our knowledge, no other studies investigate the validity of ARISCAT except the above-mentioned study, no other studies calculate validity measures such as discrimination, calibration, accuracy, positive predictive values or negative predictive values. However, ARISCAT seems to be clinical useful and several studies have already implemented ARISCAT in the stratification of surgical patients [2, 12]. Furthermore, ARISCAT have been found to be an independent risk factor for postoperative pulmonary complications [13, 14].

A previous study have found the occurrence of a 30-day postoperative complication to be a critical factor in determining the survival after major surgery [4] which further emphasises the importance of identifying patients at risk of complications in regard to preventing postoperative mortality. Risk prediction models can potentially contribute to optimising the peri- and postoperative course when integrated in a clinical perioperative bundle. Studies report reduced mortality by 6–8% when optimising and organising emergency surgical patients’ peri- and postoperative care [12, 15]. ARISCAT could potentially help identify high-risk patients who could benefit from an intensified peri- and postoperative course. A meta-analysis found that postoperative continued positive airway pressure (CPAP) can reduce the risk of pulmonary complications after general surgery [16] and such modifications in the postoperative course could potentially reduce morbidity and mortality for patients at high risk of postoperative pulmonary complications in the future. The overall advantage of ARISCAT is the use of easily collected variables.

There are some limitations to this study. This is a single-centre study and the data may therefore not be completely generalisable to other study populations. The group of patients with a low risk of postoperative pulmonary complications was small, which might reduce the statistical power for this group. The strength of the study is that data were collected prospectively and few patients had missing data. A recalibration of the ARISCAT model was not necessary as the slope and intercept of the calibration plot are almost perfect and as the measure points are distributed almost evenly on both sides of the fitted line.

In conclusion, ARISCAT was found to have good calibration, good discrimination and good accuracy in predicting postoperative pulmonary complications in a high-risk surgical population undergoing major emergency abdominal surgery. The score may guide clinicians to identify patients that potentially could benefit from early postoperative interventions to prevent pulmonary complications.