Introduction

In the primary treatment for rectal cancer, surgical resection is the mainstay of therapy. Since its beginnings, it has been performed in an open surgical technique. Over the last decades, this has been challenged by the laparoscopic approach suggesting improved short-term results due to reduced surgical trauma and less immunological impairment [1, 2].

Aside from these short-term effects, question arose whether the laparoscopic approach is suitable to reach comparable oncologic accuracy especially in completeness of total mesorectal excision (TME). Attention was set to the surgical accuracy of the circumferential (CRM) and distal resection margin (DRM) being closely related to local recurrence and long-term survival [3, 4].

Moreover, the localization of the tumor in the upper, mid, or lower rectum seemed to influence the oncological results and therefore was debated to determine the optimal surgical approach.

Until now, several randomized controlled trials aimed to investigate these issues leading to inconsistent results. The COLOR II and the COREAN trial, investigating the recurrence rate and the disease-free survival, respectively, indicated an oncological equivalence of both approaches [5, 6]. Aside from these results, neither the ALACART nor the ACOSOG trial, both performed by specialist groups, could prove non-inferiority of laparoscopic compared with open surgery investigating pathological quality indicators as CRM and DRM as well as the completeness of TME [7, 8].

Against the background of numerous single and multicenter studies revealing equivalence or even superiority of the laparoscopic approach, these results and especially its transferability to general practice are currently debated [9, 10], especially under reflectance on the individual surgeon [11,12,13].

This study aimed to investigate two patient collectives of specialized centers for either the laparoscopic or the open resection of rectal cancer, regarding short-term quality indicators as well as long-term survival and disease progression of rectal cancer patients. This study avoided bias resulting from learning curve or low volume performance, to closely focus on the influence of the distinct techniques. The selected centers particularly qualify for this analysis, because Lübeck (HL) is a long-standing center in the field of laparoscopic colorectal surgery, whereas Erlangen (ER) is a long-standing center for open colorectal surgery with high patient volume.

Patients and methods

The study was designed as a retrospective analysis of two prospectively maintained institutional colorectal cancer surgery databases of the University Surgical Centers of Lübeck and Erlangen in Germany. To enhance comparability, study cohorts were propensity score–matched.

The analysis of the databases comprised the time span of 10 years from January 2006 to March 2016. This period was chosen because from 2006 more laparoscopic than open rectal resections were performed in Lübeck, and therefore, an advanced learning curve could be assumed.

In both centers, patients were followed up for at least 5 years with physical examination, estimation of carcinoembryonic (CEA) levels, abdominoperineal ultrasonography, chest X-ray, computed tomography (CT) of the pelvis, colonoscopy, and rectoscopy. Thereafter, a minimum vital status was checked annually. This was performed by annual correspondence (progression and survival) with the patients’ primary care physicians as well as the citizen registration offices. Staging in Lübeck and Erlangen included rectoscopy, rectal endosonography, and MRI scan. For patients cN + or ≥ cT3 and tumorlocalization in the mid or lower rectum, a neoadjuvant treatment was indicated. In Lübeck as well as in Erlangen, neoadjuvant treatment was performed as a conventional fractionated radiotherapy up to 50.4 Gy supported by a 5-FU chemotherapy supplemented by oxaliplatin in selected cases. Surgery was performed after 6–8 weeks.

Perioperative management in both centers was quite similar. A preoperative mechanical bowel preparation was performed in both centers; PDA was used if applicable. Drains were used in both centers. In Erlangen, drains were cleared early whereas in Lübeck, drains generally stayed for 5 days. Postoperative feeding was performed in a step-up approach in both centers using liquid diet for the first postoperative days which was then completed by solid food depending on bowel movement. No fast track protocol was applied in both centers.

We included patients with surgery for rectal cancer with histologically confirmed invasive adenocarcinoma. Patients from Lübeck had completed laparoscopic resection whereas patients from Erlangen had open resection, both with total mesorectal excision (TME) [14]. In the upper rectal third carcinomas, partial mesorectal excision (PME) was performed.

Exclusion criteria were carcinoma at multiple primary locations, emergency surgery, conversion from laparoscopic to open procedure (n = 5), multivisceral resections, accompanying inflammatory bowel disease, Union for International Cancer Control (UICC) stage IV, and incomplete documentation or follow-up (Erlangen n = 1, Lübeck n = 2).

This selection led to two groups of 507 patients in Erlangen and 248 patients in Lübeck. To adjust risk profiles, a 1:1 propensity score matching was performed according to the following criteria: age, sex, neoadjuvant treatment, UICC stage, American Society of Anesthesiologists (ASA) score, tumor site, and adjuvant chemotherapy. The evaluation comprised clinical data of all patients, indicators of process quality, postoperative complications, loco-regional as well as distant recurrence, and survival analysis. For analyses of prognosis, only patients with a curative (R0) resection treated before January 01, 2012, with a possible follow-up of at least 5 years were included. This resulted in 328 patients who were followed up for a median of 81 month (range 1–134), in Erlangen (n = 160; med 82; 1–134) as well as in Lübeck (n = 168; med 77; 1–133). The tumor localization within the rectum was divided according to the proposal of the UICC: lower rectum 0–< 6 cm, middle rectum 6–< 12 cm, and upper rectum 12–16 cm from anal verge when measured with a rigid rectoscope [15]. For analysis of anastomotic leaks, only patients with grades B and C leaks according to Rahbari et al. were considered [16]. Loco-regional recurrence was defined as recurrence of tumor in the pelvis. Urinary dysfunction implied the need to use suprapubic bladder drainage, urinary leg bags, or absorbent pads at the time of discharge.

Potential confounders of this study were the retrospective approach, the single-center analysis, potential changes in technique over observation period, and the fact that propensity score matching, despite its powerful matching capability, only accounts for observed (and observable) covariates. The study size was defined by the number of eligible patients over the defined study period.

Statistics

Statistical analysis was performed with IBM SPSS Statistics v24.0 including R Essentials for SPSS (IBM Corporation, Chicago, USA) and R v2.15.3 software (www.r-project.org) including the MatchIt package. Patients were matched 1:1 by propensity score (nearest neighbor matching with logistic regression, caliper 0.2 without replacement) using the covariates age, sex, neoadjuvant treatment, UICC stage, ASA score, tumor site, and as well as adjuvant chemotherapy.

Categorical data were analyzed using chi-squared and the Mann–Whitney U test was utilized for comparison of continuous data. Loco-regional and distant recurrence was calculated as a 5-year rate using the Kaplan–Meier estimation and a log-rank test for comparison. Also, the survival analysis was performed using the Kaplan–Meier estimation with the date of tumor-related therapy (resection or neoadjuvant therapy) as starting point. Overall and disease-free survival curves for both groups were compared using the log-rank test. Univariate and multivariate survival analyses were performed by the Cox proportional hazard model. All variables with p < 0.1 in univariate analysis were included in the multivariable model, which was adjusted for age in the analysis of disease-free and overall survival. The significance level was set to p = 0.05. Missing data were not substituted.

Results

The analysis of both groups revealed 507 patients for Erlangen and 248 patients for Lübeck fulfilling the inclusion criteria. The Erlangen cohort had 149 (29.4%) female patients and 358 (70.6%) male patients with a median age of 64 (18–93) whereas the Lübeck cohort had 84 (33.9%) female and 164 (66.1%) male patients with a median age of 68 (38–91) differing significantly (p = 0.001). Also, for ASA classification, a significant difference was seen, resulting from a higher percentage of ASA 1–2 patients in the Erlangen cohort (78.1% vs. 54.8%; p < 0.001). Moreover, the distribution of the tumor site showed significant differences resulting from a higher percentage of patients from Erlangen in the middle third (6– < 12 cm) (47.3% vs. 37.9%; p = 0.013) and a lower percentage of patients from the Erlangen cohort in the upper third of the rectum (12–16 cm) (18.3% vs. 26.2%; p = 0.013).

Concerning neoadjuvant and adjuvant treatment, cohorts differed significantly resulting from higher percentages in the Erlangen cohort (neoadjuvant 57.0% vs. 48.0%, p = 0.020; adjuvant 67.5% vs. 44.0%, p = 0.002). Also, histopathological UICC stages differed significantly resulting from the higher neoadjuvant treatment rate in Erlangen (p = 0.039). Length of follow-up of both cohorts showed no significant differences (median 63 vs. 58 months; p = 0.399) (Table 1).

Table 1 Clinical data of Erlangen and Lübeck patients
Full size table

To adjust these differences, a 1:1 propensity score matching was performed as described in the methods section. This lead to two groups of 248 patients, showing no significant differences in age (p = 0.769), sex (p = 0.849), ASA (p = 0.526), tumor site (p = 0.889), neoadjuvant treatment (p = 0.857), adjuvant treatment (p = 0.171), and UICC stage (p = 0.985).

Evaluating the process quality (Table 2), more sphincter-saving procedures were performed in the open (Erlangen) cohort (84.3% vs. 73.8%; p = 0.004). R classification showed no significant differences (p = 0.253). In Lübeck, there were five R1 resections, whereas in Erlangen, there was one R2 resection and one Rx resection. In addition, the quality of TME/PME did not differ among groups (p = 0.198). However, significant differences were seen in the negative circumferential resection margin (> 1 mm) with 98.3% in the open cohort and 92.0% in the laparoscopic (Lübeck) group (p = 0.003). Also, the distal resection margin showed significant differences between groups with 99.6% negative in the open and 97.1% in the laparoscopic group (p = 0.047). Moreover, the number of examined lymph nodes was significantly higher in the open group (median 22 vs. 14; p < 0.001), which was confirmed when only pN0 patients were considered (median 26 vs. 15; p < 0.001).

Table 2 Indicators of process quality
Full size table

Assessing the postoperative complications, several significant differences were seen among groups (Table 3). The overall postoperative morbidity was lower in the open group without reaching significance (21.8% vs 29.0%; p = 0.063). The anastomotic leak rates (grades B and C) were significantly different with excellent 3.4% in the open group and 12.6% in the laparoscopic (p = 0.001). Also, postoperative bleeding (0.4% vs 3.2%; p = 0.019), intra-abdominal abscess (0% vs 3.2%; p = 0.004), postoperative peritonitis (0.4% vs 3.2%; p = 0.019), postoperative ileus (0.8% vs 4.4%; p = 0.011), stoma complications (0.4% vs 2.8%; p = 0.032), re-operations (6.5% vs 16.5%; p = < 0.001), and superficial wound infections (2.4% vs 7.7%; p = 0.008) differed significantly favoring an open approach. Postoperative urinary dysfunction was lower in the laparoscopic group (1.6% vs. 4.4%; p = 0.066) without reaching significance. Cardiovascular and pulmonary complications showed no differences. Also, in-hospital mortality as well as 30-day and 90-day mortality was similar among both groups.

Table 3 Postoperative complications and international comparison
Full size table

Moreover, results were compared with the short-term outcomes of the COLOR II and the COREAN trial. The data demonstrate the comparability with international results. We would like to clarify at this point that the quality of the laparoscopic group as far as the postoperative complications is within the range of published results of current randomized controlled trials. The results of the open surgical group in contrast reflect surgical quality above the current published results from RCTs, reflecting single-center excellent therapy.

In this context, an analysis of the loco-regional and distant recurrence data (Table 4), with a stratification by tumor height (lower, middle, upper third), revealed no significant differences for loco-reginal progress overall and in the lower and middle third, but a significantly lower 5-year recurrence rate for the laparoscopic group in the upper third (0 vs. 3.2%; p = 0.042). For distant metastasis, no significant differences could be shown overall and stratified by tumor heights.

Table 4 Loco-regional and distant recurrence
Full size table

The Kaplan–Meier analysis investigating the disease-free and overall survival stratified by thirds of the rectum revealed no significant differences among groups (Table 5).

Table 5 Disease-free and overall survival; the Kaplan–Meier analyses
Full size table

A Cox regression analysis for influencers of the disease-free survival adjusted for age including all patients with treatment until December 31, 2011, revealed ASA 3–4 (p < 0.001 HR 2.7), UICC stage y0 (p = 0.045; HR 0.2), and abdominoperineal excision (p = 0.019; HR 1.7) as significantly influencing factors in the univariate model. ASA 3–4 (p = 0.002; HR 2.1) and UICC stage yIII (p = 0.026; HR 2.0) proved to be independent prognostic factors in the multivariable model (Table 6).

Table 6 Disease-free survival; the Cox regression analysis, adjusted for age
Full size table

Correspondingly, a Cox regression analysis of influencing factors of the overall survival showed ASA 3–4 (p < 0.001 HR 3.3) as well as UICC stage y0 (p = 0.039; HR 0.1) and abdominoperineal excision (p = 0.005; HR 1.9) as significantly influencing factors in the univariate model. ASA 3–4 (p < 0.001; HR 2.5) and UICC stage yIII (p = 0.037; HR 2.0) proved to be independent prognostic factors in the multivariable model (Table 7).

Table 7 Overall survival; the Cox regression analysis, adjusted for age
Full size table

All patients with a R1 resection (n = 5) were separately analyzed concerning the following oncologic workup. As demonstrated, most patients could be re-resected with subsequent tumor-free specimen (Table 8).

Table 8 Treatment of R1 patients in Lübeck
Full size table

Discussion

In this retrospective study, based on two prospectively maintained institutional colorectal cancer databases, we compared patients with rectal cancer who had laparoscopic or open resection in a propensity score–matched analysis. Both contributing hospitals were high-volume centers. The comparison focused on four major topics which were process quality, complications, loco-regional as well as distant recurrence, and long-term survival according to tumor height.

The analysis of the process quality revealed significant differences in sphincter preservation, resection margins, and lymph node retrieval, all in favor of the open approach. Overall, postoperative morbidity in summary did not differ significantly, but several complications such as anastomotic leakage, postoperative ileus, wound infection, and re-operation rate showed benefits for the open approach. In-hospital, 30-day and 90-day mortality did not differ significantly. Concerning loco-regional recurrence, the laparoscopic group showed a significantly lower rate for the upper third of the rectum, although numbers (n = 0 HL; n = 2 ER) were quite small in this category. This is remarkable, as anastomotic leakage, contrary to our results showing no difference in oncologic outcome [17], has been reported to be associated with a significantly increased risk of loco-regional recurrence [18, 19]. For distant metastases or for recurrence in the mid or lower rectum, no significant differences were shown. Analyzing the overall and disease-free survival, no significant differences were seen. In summary, this study shows a short-term benefit for the open approach, especially based on perioperative complications. It is also remarkable that the higher rate of LN retrieval and lower rate of CRM and DRM margins in the open group did again not translate to a lower recurrence rate or long-term survival. As mentioned before on the contrary, a lower rate of local recurrence was found in the laparoscopic group for the upper rectum.

The selection of patients enabled comparability and transferability of results. Moreover, propensity score matching algorithm led to a high accordance among groups. Patient databases of both centers had a stringent and high follow-up quality with at least annual data updates. A caseload of 496 matched patients allowed sustainable statistical calculations.

Nevertheless, this study has some restrictions and bias. The study design was retrospective implying unknown selection bias and mis-classification or information bias. Moreover, as mentioned, propensity score matching, despite its powerful matching capability, only accounts for observed covariates. Factors that affect recurrence or survival but that cannot be observed cannot be accounted for in the matching procedure [20]. Finally, today, the modern treatment of colorectal cancer is performed in centers with multiple disciplines involved. Quality indicators are used to assess not only the surgery but the overall concept, including the results of partners in radiology, radiotherapy, oncology, pathology, and others. Particularly, the judgment principles of the pathologists concerning TME, CRM, and DRM quality and the number of lymph nodes over a time span of 10 years cannot be stated thoroughly, particularly since the concept of interdisciplinary tumor conferences has just been established during the observational period. Both university pathological departments followed the current guidelines for the judgment of specimen quality.

Reviewing the current literature and connecting our findings to the findings of other groups, study results appeared to be heterogeneous. A majority of publications revealed equivalence of the laparoscopic approach, in retrospective studies (Table 9) [10, 21,22,23,24].

Table 9 Retrospective studies comparing laparoscopic and open rectal surgery
Full size table

Apart from these observational studies, randomized controlled trials (RCT) such as the COLOR II study and the COREAN trial revealed comparable rates of loco-regional recurrence and disease-free as well as overall survival for laparoscopic and open surgery [5, 6]. Our findings contribute to this important finding as in the current collective loco-regional recurrence did not differ significantly even though the rate of detected anastomotic leakage was almost three times as high in the laparoscopic group. The group of Ng on the contrary found improved postoperative recovery, reduced short-term and long-term morbidity rates, and equivalent long-term survival for laparoscopic resection of mid and low rectal cancer [9].

These results were supported by numerous meta-analyses. Chen et al. included seven RCTs and non RCTs with 4353 patients, revealing that there are no significant differences between laparoscopic and open surgery in terms of survival and pathological outcomes with the exception of negative DRM favoring the open approach. This study also suggested that laparoscopy elicits faster recovery [25]. Creavin et al. evaluated four RCTs including 2319 patients leading to the conclusion that mesorectal excision showed a small difference in achieving an intact mesorectum in favor of open surgery. However, this did not affect long-term oncological outcomes [26].

Aside from the reported advantages of the laparoscopic approach, recent RCTs focused especially on pathological outcomes. After analyzing results of 475 patients, the authors of the ACOSOG trial came to the conclusion that among patients with stage II or III rectal cancer, the use of laparoscopic resection failed to meet the criterion for non-inferiority (boundary − 6%) for pathologic outcomes. These assumptions were derived from a composite endpoint, including CRM (> 1 mm), DRM negative, and complete or nearly complete TME [8]. Likewise, a similar study also comparing the laparoscopic and the open approach in rectal cancer patients with focus on histopathological quality, the ALaCaRT trial, could not prove non-inferiority (boundary − 8%) of laparoscopic surgery compared with open surgery among patients with T1–T3 tumors. In the ALaCaRT trial, a composite endpoint consisting of complete total mesorectal excision, a clear CRM (≥ 1 mm), and a clear DRM (≥ 1 mm) was applied [7]. Weakness of both studies is that the composite endpoints have not been validated for long-term oncologic outcomes, and therefore, long-term results of these studies have to be awaited. Finally, a meta-analysis investigating the rate of positive CRM and the quality of mesorectal excision, analyzing 14 RCTs including 4034 patients, found that the risk for achieving a non-complete (nearly complete or incomplete) mesorectal excision is significantly higher in patients undergoing laparoscopy compared with open resection (13.2% vs. 10.4%) [27].

Comparing the abovementioned results to our study cohorts, it is noticeable that the broadly reported short-term benefits of the laparoscopic approach compared with open surgery could not be shown. This may be due to reporting bias, but is comparable with recent RCTs and might also result from excellent comparative data in the open group from Erlangen. Key items, as postoperative ileus rates of 0.8%, wound infection rates of 2.4%, and pulmonary complication rates of 3.6%, show that open surgery in specialized centers can be performed with very low postoperative morbidity. Another crucial point, the differences in anastomotic leak rate, may be a result of varying anastomotic techniques. Whereas in Lübeck, a double-stapling technique with a linear stapler to dissect the rectum and a circular stapler for transanal anastomosis was performed, in Erlangen, a single-stapling technique using a purse-string suture for the rectal stump and a circular stapler for transanal anastomosis was done. Mobilization of the left flexure and protective ileostomy for low anastomosis was performed in both groups. Nevertheless, all morbidity data of the laparoscopic group was in line with the results of the COLOR II study (Table 3) and therefore compares with the review board-approved study centers.

Moreover, laparoscopic group shows a significantly higher amount of abdominoperineal excisions. This is in line with other studies showing a similar extirpation rate [28]. When analyzing Table 1, slightly more patients in the laparoscopic group had a low tumor localization, which raises the probability for extirpations. Unfortunately, the database allows no differentiation according to exact tumor height, nor do we have data in functional outcomes after sphincter preservation. Generally, the indication for extirpation was made according to tumor localization and sphincter infiltration and functionality but not depending on the technique.

Evaluating the pathological quality, data from this study demonstrate a significant benefit for the open approach concerning negative CRM (> 1 mm) and DRM (> 1 mm) and therefore are in line with the recent RCTs like ALaCaRT and ACOSOG. Looking at the TME quality, no significant differences were seen [7, 8]. This demonstrates a comparable oncological quality of the specimen achieved by laparoscopic surgery. However, the retrieval of lymph nodes was significantly higher in the open group revealing a value of median 22 lymph nodes being also an exceptional result compared with current literature with a median of 12–19 nodes, confirming not only high surgical quality but also the quality of the pathologists [27]. Even though these differences were seen in the observed long-term analysis, this did not translate to the finally strongest and most important parameters local recurrence and survival.

When comparing the overall recurrence and survival data with literature, the results were in line with RCTs such as COLOR II or the COREAN trial, showing no significant differences among groups [5, 6]. The analysis of loco-regional recurrence according to tumor height showed a significant benefit for the laparoscopic group for the upper third of the rectum, although numbers in this category were quite small (n = 0 HL, n = 2 ER). Here, differences were seen in comparison with COLOR II showing significant higher rates of loco-regional recurrence at the open approach in the lower third of the rectum [5].

Evaluation of distant metastasis as well as overall and disease-free survival according to tumor height demonstrated no significant differences in correlation to the results of COLOR II and COREAN trials [5, 6].

Whereas the COLOR II trial demonstrated a benefit for the laparoscopic approach in the lower rectum resulting from lower loco-regional recurrence and higher negative CRM rates, our long-term oncological findings cannot support a recommendation favoring a surgical approach according to tumor height [5].

Analyzing the data of nearly 500 patients in this study in univariate and multivariate Cox regression models, it revealed that the surgical approach had no impact on overall or disease-free survival. High ASA grading as well as advanced UICC staging proved to have a significant impact on overall and disease-free survival, additionally supporting the consistency of the dataset.

In summary, it could be demonstrated that excellent open surgical treatment of rectal cancer can result in outstanding short-term outcomes that do surprisingly not translate in better loco-regional control. The results of open surgery were in some parameters superior compared with the laparoscopic approach. Adequate histopathological results could be achieved with both techniques. Consecutive, oncologic long-term results did not differ significantly and especially did not reveal relevant differences according to tumor height. In expert hands, for both approaches, open and laparoscopic, a beneficial technique for defined tumor levels as far as oncological outcome cannot be concluded.