Introduction

Systematic reviews can be defined as points of view focusing on primary studies that are performed according to established and explicit methods, enabling thorough appraisal of the process and results of the review, as well as, when appropriate, repetition of the reviewing effort [1]. Meta-analysis is a statistical technique enabling statistical pooling of data from separate primary studies. Indeed, most high-quality meta-analyses are performed within the context of a systematic review, but they can occasionally be based on selected datasets. Instead, overviews of reviews (i.e., umbrella reviews) are observations on a given subject that deliberately use a systematic approach to search, select, abstract, appraise, and pool secondary research studies (i.e., reviews or meta-analyses). Systematic reviews have being criticized since their introduction in the 1970s [1–3], yet they dominate the evidence base hierarchy [4, 5], and this study design receives the highest number of scholarly citations [6]. Finally, they are probably the most cost-effective type of research endeavor [7–9].

Given their increased statistical power and heightened external validity, systematic reviews are ideally equipped to explore and test treatments capable of providing prognostic (i.e., survival) or symptomatic benefits in patients undergoing surgery. However, there is to date no systematic critique of systematic reviews of nonsurgical treatments aimed at decreasing mortality or major cardiac complications in patients undergoing surgery. Moreover, although scholarly citation is often considered a posteriori a quality benchmark for scientific endeavors in general and in cardiac anesthesia in particular, it is unclear whether such citations are associated with internal validity, reporting quality, or findings of meta-analyses in this specific clinical setting.

We hypothesized that review findings and validity are positively associated with scholarly citations. We thus aimed to undertake a comprehensive appraisal of meta-analyses on nonsurgical treatments aimed at decreasing mortality or major cardiac complications in patients undergoing surgery to appraise key methodological features as well as patterns and predictors of scholarly citations.

Methods

Study search

Systematic reviews focusing on nonsurgical treatments aimed at decreasing mortality or major cardiac complications in patients undergoing surgery were searched without language restrictions in MEDLINE/PubMed (updated on 31 December 2008) with the following strategy: systematic[sb] AND (surgery[tiab] OR surgic*[tiab] OR operation*[tiab]) AND [(myocardial AND infarction) OR (death* OR survival OR mortality OR prognosis)] AND (prevent* OR reducti* OR reduci*). References of short-listed studies were also checked for additional reviews.

Study selection

Inclusion criteria were (a) study design explicitly stated as systematic review; (b) focusing on nonsurgical treatments aimed at decreasing mortality or major cardiac complications in patients undergoing surgery encompassing either general or locoregional anesthesia; (c) with meta-analytical pooling; and (d) endpoints including in-hospital death or myocardial infarction. Exclusion criteria were (a) lack of explicit and systematic methods for reviewing effort; (b) focus on nonsurgical patients or on patients appraised well before or well after (±7 days) surgical intervention encompassing either general or locoregional anesthesia; (c) comparison of specific surgical techniques or devices; and (d) lack of meta-analytic pooling of study data.

Data abstraction

The following data were systematically captured in a dedicated electronic case report form piloted over the first ten reviews: authors, length (pages), origin, publishing journal, year of publication, treatment and comparator under study, type of surgery, target population, studies included, patients included, effect size for death, myocardial infarction, duration of stay in the intensive care unit (days), and postoperative length of stay. The 2007 impact factor (number of citations in 2008 and 2009 divided by number of articles published in 2007) and citations (normalized according to years since publication) in Institute for Scientific Information Web of Science (ISIWOS), Google Scholar (GS), and Scopus by 30 September 2009 were also extracted to compare predicted versus observed citations.

Validity appraisal

The appraisal of internal validity was based on the following items: study search explicit and extensive (yes/no), explicit methods for study selection, abstraction, and pooling (yes/no), inclusion of randomized clinical trials only (yes/no), inclusion of double-blind randomized clinical trials only (yes/no), underlying statistical heterogeneity/inconsistency (yes/no/not reported), competing conflicts of interest (yes/no/not reported), and funding (yes/no/not reported). Thus, the overall credibility and robustness of the systematic review were judged (yes/no), based on a comprehensive assessment of the following items: (a) study search explicit and extensive, (b) explicit methods for study selection, abstraction, and pooling, (c) only double-blind randomized controlled trials (RCT) included, (d) statistical homogeneity/consistency, and (e) concordance between review findings and authors’ interpretation. Reviews meeting all these five criteria were judged at low risk of bias, those meeting two to four of these criteria were judged at moderate risk of bias, and the others at high risk of bias or at unclear risk of bias, depending on thoroughness of reporting.

Analysis

Continuous variables are reported as mean (standard deviation) or median (1st–3rd quartile). Unpaired t, Mann–Whitney U, analysis of variance (ANOVA), and Kruskal–Wallis tests were used to compare continuous variables between two or more groups. Correlation was appraised by means of Pearson correlation methods. Bland–Altman plots were performed to compare bibliometric findings. Categorical variables are reported as n (%) and were compared with chi-square test, Fisher exact test, or logistic regression analysis. Predictors of citations in bibliometric indexes were explored, after normalization for time from publication, by means of multivariable linear regression with backward stepwise selection [10–12]. Statistical significance was set at the two-tailed 0.05 level. Given the reliance on multivariable models, such analyses (and corresponding p values) are reported without adjustment for multiplicity. Conversely, other analyses (including Pearson correlation tests) should be viewed as exploratory only given the risk of alpha error inflation. All computations were performed with SPSS 18 (IBM, Armonk, NY, USA).

Results

Searches produced 2,239 citations, with 2,063 of them excluded at the title or abstract level and 92 at the full text level (Fig. 1). Finally, 84 systematic reviews and meta-analyses, with 7 of them reporting on multiple comparisons, for a total of 97 analyzable subsets, were identified (Tables 1, 2, 3; Online only tables 1, 2; 3, 4, 5, 6).

Fig. 1
figure 1

Review profile

Full size image
Table 1 Key features of included reviews (references of included studies are available from the corresponding author upon request)
Full size table
Table 2 Clinical features of included reviews
Full size table
Table 3 Quality features of included reviews
Full size table

The number of reviews published each year increased steadily, demonstrating the ongoing success with authors, editors, and readers of this type of study (Fig. 2). Most reviews were authored by North American (42.9 %) or European (42.9 %) corresponding authors, with a median of 4 (range, 3–6) co-authors and 10 (8–13) pages per article (Table 1). Reviews included a median of 11 (range, 7–20) studies, with 1,340 (863–2,670) patients (Table 2). Included patients underwent a variety of surgical procedures, most commonly cardiovascular surgery (40.2 %) and abdominal surgery (10.3 %). Several different interventions were appraised by the included meta-analyses, including cardiovascular drugs (25.8 %), hemostatic drugs or techniques (22.7 %), and anesthesia drugs and techniques (12.4 %). Most commonly, a placebo acted as control (38.1 %), with one or more active controls explicitly employed in only a minority (12.4 %).

Fig. 2
figure 2

Steady increase over time of systematic reviews and meta-analyses published yearly up to 2009 on ancillary treatments aimed at improving surgical outcomes and indexed in PubMed

Full size image

The internal validity of the reviews appeared largely robust, as most reviews were at low (50.5 %) or only moderate risk of bias (23.7 %), as testified by satisfactory and explicit methods for reviewing in most of the meta-analyses (Table 3). Review findings suggested a significant benefit on survival and on postoperative length of stay with several interventions, with less common beneficial impact on perioperative myocardial infarction and intensive care unit stay (Online only Table 2).

Citations were, on average, 24.5 in GS, 17.0 in ISIWOS, and 18.0 in Scopus (Table 1). Mean normalized yearly citations for the included reviews were 5.6 in GS, 4.3 in ISIWOS, and 4.9 in Scopus, and appeared highly correlated, despite evident differences, especially for extremely quoted reviews (Online only figure). In particular, normalized yearly citations in ISIWOS were significantly associated with those in GS (R = 0.858, p < 0.001) and in Scopus (R = 0.975, p < 0.001), with those in GS also significantly associated with those in Scopus (R = 0.856, p < 0.001).

In keeping with the primary objective of the study, multivariable-adjusted appraisal of the predictors of citations showed that normalized yearly citations in GS were significantly and independently associated (adjusted R 2 = 57.6 %) to the following factors: number of authors (B = 2.100, p = 0.002), number of patients included (B = 0.003, p < 0.001), noncardiothoracic surgery (B = 12.709, p = 0.001), acknowledgement of funding (B = 13.732, p = 0.006), and lack of conflicts of interest (B = 17.823, p = 0.002) (Online only table 4). Normalized yearly citations in ISIWOS were significantly and independently associated (adjusted R 2 = 29.8 %) with the following factors: number of studies included (B = 0.201, p < 0.001), and noncardiothoracic surgery (B = 5.278, p = 0.001). Normalized yearly citations in Scopus were significantly and independently associated (adjusted R 2 = 38.6 %) with the following factors: number of studies included (B = 0.298, p < 0.001), North American location of corresponding author (B = 4.146, p = 0.043), and noncardiothoracic surgery (B = 7.362, p < 0.001).

Intriguingly, systematic reviews significantly and consistently outperformed the corresponding journal impact factor (Fig. 3), confirming previous data showing that they are the study type at highest likelihood of scholarly citation [6]. We also appraised whether the number of meta-analyses in each subject area influenced citations, but did not find evidence of such an association at correlation and regression analysis (all p > 0.05). We explored the correlation between year of publication and several covariates, finding that more recent reviews included fewer studies (R = −0.271, p = 0.013) and patients (R = −0.235, p = 0.034). In addition, analyzing factors associated with statistical heterogeneity/inconsistency, we found that reviews pooling heterogenous/inconsistent data were less likely to be quoted in ISIWOS (p = 0.006) and GS (p = 0.021), but surprisingly more often focused only on double-blind RCTs (p = 0.027). Finally, reviews reporting statistically significant findings included more patients (p = 0.041) and were more likely to pool both randomized and observational studies (p = 0.020) rather than only randomized trials.

Fig. 3
figure 3

Association between journal 2007 impact factor (number of citations in 2008 and 2009 divided by number of articles published in 2007) and yearly citations for included systematic reviews (normalized for time since publication), demonstrating that these articles consistently outperformed the corresponding journal average citations (which should equal half the impact factor). ISI Institute for Scientific Information

Full size image

Discussion

This work, providing for the first time a comprehensive systematic review of currently available meta-analyses focusing on nonsurgical treatments aimed at decreasing mortality or major cardiac complications in patients undergoing surgery, has the following implications: (a) systematic reviews currently represent a key element in defining state-of-the-art management of these subjects; (b) most recent reviews included in this study appeared of high internal validity and at low risk of bias; (c) the citation success of available systematic reviews is, however, not associated with prognostically relevant findings or quality features, but mostly with other factors such as number of authors, studies, patients, location of corresponding author, and scope; (d) thus, further efforts should be implemented to foster diffusion and dissemination of results of systematic reviews showing prognostically relevant benefits of ancillary treatments for patients undergoing surgery.

Current research context

Clinical practice has evolved in the past few decades, and now evidence-based medicine pervades practices worldwide [4]. However, the ever-increasing production of clinical evidence items translates into ongoing difficulties for clinicians, healthcare decision makers, and patients as synthesis becomes more and more difficult. Systematic reviews and meta-analyses have indeed been hailed as one of the most important aspects of the current success of evidence-based medicine, as they provide a reliable and systematic tool to summarize key clinical issues and provide quantitative assessments guiding practical decision making [1, 4, 13]. However, the quality of systematic reviews varies largely, and there remains uncertainty on the impact of any given published systematic review, its thoroughness and quality, and its eventual role in shaping clinical decisions. This concern holds even truer in the setting of patients undergoing surgery, who still face significant risks of perioperative morbidity and mortality, despite major advancements in surgical techniques and perioperative medical management.

Despite the increase in systematic reviews focusing on ancillary treatments aimed at improving surgical outcomes and published in the last decade, very few investigations have been performed with the goal of appraising the interplay among their findings, quality, likelihood of citation, and impact [3]. Thus, our work is aimed at disentangling such complex issues, whose pivotal importance is testified by the fact that, even in the setting of medical management of surgical patients, systematic reviews and meta-analyses represent the first and foremost source of clinical evidence sought by decision makers.

Contributions of the present study

This study is the first to systematically search, appraise, and extract key items from systematic reviews focusing on ancillary treatments aimed at improving surgical outcomes and testing the impact of these features on subsequent scholarly citations. Indeed, our sensitive search strategy identified more than 2,000 potentially pertinent citations, which were carefully screened, first at the title or abstract level, and then, if necessary, as full text. This effort finally lead to the inclusion of 84 systematic reviews, with 7 (8.3 %) providing more than a single comparison. Thus, a total of 97 comparisons between different treatment strategies were appraised.

The most common type of procedure was cardiovascular surgery, which is indeed a frequent and a prognostically relevant intervention. Accordingly, most reviews focused on cardiovascular drugs. Placebo was chosen as the control arm in more than one third of treatment comparisons, testifying to the effort to minimize bias (by exploiting the double-blind placebo-controlled design). Notably, a survival benefit was demonstrated in several reviews, with corresponding benefits on other important clinical endpoints. Thus, several reviews were of great clinical relevance and impact, and, if incorporated into clinical practice, likely to substantially benefit patients. This observation holds even more true as the quality and internal validity of these works appeared adequate in most. Despite the case of two conflicting reviews on aprotinin [14, 15], other reviews consistently identified cheap and effective perioperative interventions such as statins, volatile anesthetic agents, early enteral nutrition, neuraxial anesthesia, and α-2-adrenergic agonists.

Finally, we found that most reviews were frequently cited despite their relatively recent publication, with likelihood of scholarly quotation being mostly associated with the number of authors, North American location of the corresponding author, the number of studies included, the number of patients included in the review, focus on surgical procedures other than cardiothoracic surgery, explicit statements concerning funding, and lack of conflicts of interest. Although most of these associations are expected, as, for instance, North American authors are well known to be more commonly quoted than those from other countries [16], it is surprising that reviews reporting survival benefits or other clinically relevant benefits were not significantly more likely to be quoted. Indeed, associations between citations and selected predictors may appear spurious. However, Biondi-Zoccai et al. [8] have already demonstrated the association between apparently minor features of systematic reviews (e.g., number of pages) and review quality, and several explanations can be provided for the association between most of the identified predictors and citations.

Limitations of the present work

This overview has several limitations, including those typical of epidemiologic inquiries [1]. The main drawback of our work is likely the emphasis on ancillary treatments aimed at improving surgical outcomes, which excludes several other important healthcare interventions. The selective inclusion of meta-analyses reporting on death or myocardial infarction [17–19] may be prone to reporting bias. Finally, we emphasize that meta-analyses should not be pooled, as they may be duplicates at risk of redundancy [8], or because this may mean mixing “apples with oranges.” Indeed, we aimed not to pool disparate reviews but rather to provide a comprehensive appraisal from an epidemiologic perspective of currently available meta-analyses. Specifically, we focused on key baseline and validity items, and appraised the association between such features and scholarly citations. Thus, we do not propose to exploit the present work to guide clinical practice, but rather to inform clinicians and researchers to better read and make use of systematic reviews and meta-analysis.

Conclusions

Systematic reviews currently represent a key element in defining state-of-the-art ancillary treatments of patients undergoing surgery. However, the citation success of available meta-analyses is not significantly associated with prognostically relevant findings or quality features.