Keywords

1 Introduction

1.1 General Assumptions

The volume of knowledge currently generated in medical journals worldwide is phenomenal [1, 2]. At the same time the science of evidence synthesis has also developed rapidly [3–5]. Furthermore the importance of high-quality bodies of evidence together with the evolution of the evidence based from primary clinical trials to high-quality systematic reviews has significantly strengthened the quality of evidence currently available within most scientific and medical fields [6]. This overwhelming amount of knowledge and information both in quantitative and in qualitative areas has resulted in the need for a novel method of research – a systematic and umbrella review of systematic reviews.

This handbook elaborates on the methodology, developments, and challenges, as well as the dilemmas regarding the conduct and reporting of methodological issues concerning umbrella reviews. The aim of this chapter is to present a case study of why and how an umbrella review was conducted and reported within the context of a specific clinical research question. In this chapter we present a case study on the conservative management of adolescents with idiopathic scoliosis (AIS). We demonstrate how a careful methodological analysis of current existing systematic reviews can provide health-care professionals as well as service users engaged in the care of patients with AIS with new evidence and information regarding the credibility of published secondary research papers. We present and discuss the findings of an umbrella review of systematic reviews regarding both the screening for AIS and the effectiveness of nonsurgical interventions in adolescents with AIS.

1.2 Why Was this Umbrella Review Needed?

Screening programs are legitimate, provided that early treatment interventions for patients diagnosed with AIS as a result of screening are effective [7] – on the other hand, patients with false-positive diagnoses may undergo unnecessary and potentially harmful treatments. In the case of AIS, screening and nonsurgical treatment methods are both the subject of controversial opinions and published papers with discordant results from systematic reviews. These have resulted in contradictory recommendations and statements that have frequently lead to opinion-based practice guidelines being developed (Tables 18.1 and 18.2). The selection of this case was also prompted by the description of the specific role of the umbrella review in informing evidence-based decision-making, highlighted by Cornell and Laine in their editorial [8]:

(…) practice guidelines often require answers to not only one but several linked questions. A common example concerns screening interventions for which direct clinical trials that randomly assign participants to a screening or control group are lacking. In such cases, guideline developers typically seek answers to a sequence of interrelated questions: How prevalent is the condition? What is its natural history? Is a good screening test available? What are the benefits (and harms) of early detection and treatment? (…)

We performed an umbrella review to find answers to research questions regarding the reliability (for quantitative questions) and credibility (for qualitative questions) of all reviews addressing the screening as well as the nonsurgical management of a single condition – AIS, to inform policy makers and all engaged in the diagnosis, prevention, therapy, and education of people with this condition.

Systematic reviews, including those in the orthopedics [9], surgery [10], and physiotherapy [11] fields, vary in terms of their quality. Thus, another reason for this umbrella review was to assess the quality of the evidence from secondary studies, regarding the conservative management of AIS.

We aimed at evaluating the validity as well as the methodological quality of all existing systematic reviews on this topic. Our goal was also to examine which papers labeled as “systematic reviews” were in fact opinion pieces or narrative reviews.

1.3 Terminology and Methodological Issues: The Report as a Case Study within this Methods Handbook

Before we discuss the case study, we would like to address a number of methodological issues as well as issues concerning terminology characteristic for the comparatively new, developing, and evolving type of research synthesis methods: the umbrella review, which we believe are important in the context of the case report that we present.

Currently different terminologies are used to describe systematically developed reviews of systematic reviews (and, in some instances, also of primary studies). These terms include complex systematic reviews [12], overview of reviews [13, 14], systematic umbrella review [15], overview of systematic reviews [16, 17], umbrella systematic review [18, 19], analysis of systematic reviews [20], metaepidemiologic study [21], systematic review of systematic reviews [22], systematic review of systematic reviews and meta-analyses [23] or systematic review of meta-analyses [24], systematic map of systematic reviews [25], and even the term “survey” of reviews [26] that has been used. We will be using the term “umbrella review” throughout this chapter, even when referring to publications whose authors have used different terms for studies of this type.

The term “umbrella review” also has a number of different usages and understandings [13, 27–30]. We conducted an “umbrella systematic review of systematic reviews.” We did not include primary studies for analysis. We conducted an umbrella review, as elaborated within the Cochrane Handbook [13] and the Joanna Briggs Institute’s (JBI) [29] guidelines. We also followed the guidelines described by Ioannidis [27] and Grant and Booth [30].

Umbrella reviews of systematic reviews are needed not only to summarize large bodies of evidence [27, 28]. Different types of research questions are addressed within umbrella reviews. These have included the quality of reporting methods [31], financial conflicts of interest of review authors [20], reporting the quality of search methods in systematic reviews [26], reporting publication bias [21], as well as the handling of missing outcome data [23]. To our best knowledge, we conducted the first comprehensive, explicit, and systematic overview of systematic reviews addressing screening and nonsurgical interventions for adolescents with idiopathic scoliosis.

The rigor for the selection of systematic reviews for inclusion within umbrella reviews in terms of the design and type of systematic reviews is increasing. Umbrella review of meta-analyses [32], umbrella review of systematic reviews and meta-analyses of observational studies and randomized trials [33], and umbrella review of meta-analyses of observational studies [34] have all been published. Although rare and unusual, quantitative data syntheses (meta-analysis) are also possible within umbrella reviews [35, 36]. In our case study, we present the more typical umbrella review, with a narrative describing the evidence from the systematic reviews of any types of primary studies that were included.

2 Methodology and Reporting

In contrast to systematic reviews of primary studies, umbrella reviews lack dedicated, published reporting guidelines. We followed both the PROSPERO protocol, which is universal for systematic reviews of primary studies and overviews of systematic reviews, and the PRISMA reporting guidelines [37], which though originally developed for systematic reviews of primary studies is also applicable to umbrella reviews [38].

2.1 The Case Study on Adolescent Idiopathic Scoliosis:Report of the Umbrella Review

To address the whole subject matter regarding the two interrelated issues – the screening programs and the treatment methods – we divided our umbrella review process into two parts. The first part discusses the systematic reviews on school screening for AIS. The second part presents the systematic reviews on nonsurgical interventions for AIS. Such approaches have recently been used in both a published combined report of an “umbrella review of systematic reviews” and “a systematic review of primary studies regarding conservative interventions for tendinopathy” [16]. Similar methodologies have also been reported in an umbrella review of systematic reviews and guideline documents addressing diet and physical activity interventions and policies [39].

Protocol registration and resource publications. This chapter is based on our study “Effects of nonsurgical management for patients with idiopathic scoliosis. An overview of systematic reviews” that has been registered at PROSPERO (CRD42013003538) and has previously been presented in two reports, regarding screening [40] and nonsurgical interventions [41].

2.2 Background

2.2.1 Description of the Health Problem

The prevalence of AIS is estimated at 2–4 % of children aged 10–16. The ratio of girls to boys ranges from 1:1for spinal lateral deviations (curve angles) below 10° to over 10:1 for curves exceeding 30°. The risk of progression of untreated scoliosis depends both on the skeletal maturity and the size of the curve. The tendency for progression is also more frequent among girls [42–45]. The deformity may have lasting consequences and can be accompanied with pulmonary complications (life endangering in patients with very severe curves), pain symptoms, difficulties in participation, and psychological disorders [42–46].

2.2.2 Screening and Nonsurgical Management

As a prevalent condition, with individual variations, affecting both the physical and psychosocial functioning in the long term, AIS is considered by numerous clinicians, researchers, and authors to be an “important healthcare problem meeting the requirements for a screening program” [47, 48].

Nonsurgical interventions for the treatment of adolescents with AIS typically constitute a variety of physical modalities: braces of various types and modes of application, scoliosis-specific exercises, as well as a number of diverse physical therapy modalities such as electrical stimulation, manual therapy, and different types of osteopathic and chiropractic interventions [49–51].

2.2.3 National and International Guidelines and Recommendations

School screening The mandate for school-based screening programs for adolescents with scoliosis is a highly controversial, strongly debated issue [52–54]. As a result of controversial and discordant recommendations, in the USA, screening has remained mandatory in some US States (Arkansas, Alabama, California, Florida, Pennsylvania, Texas, Utah) is recommended in Minnesota, is not required in Montana and Oregon, and was repealed in Indiana and Maryland. Table 18.1 summarizes the recommendations and position statements of different institutions. The table also classifies the available documents as either opinion-based statements or systematically developed practice guidelines.

Table 18.1 Summary of current recommendations regarding school screening for scoliosis
Full size table

Conservative treatment Nonsurgical interventions for adolescents with AIS are subject to similar controversies. To illustrate the polarity, examples of opinions from experts can be seen in Table 18.2.

Table 18.2 Opinions regarding nonsurgical interventions for adolescents with idiopathic scoliosis
Full size table

2.3 Methods

2.3.1 Inclusion and Exclusion Criteria

To deliver the most comprehensive evidence synthesis possible, we summarized, appraised, analyzed, and synthetized all available studies meeting the minimal criteria for a systematic review. Papers were considered as systematically developed reviews if they reported on methods to search, identify and select studies, and critically appraised relevant evidence [55]. We considered all systematic reviews with meta-analyses as well as qualitative systematic reviews with descriptive syntheses of findings from individual studies. To address research questions other than effectiveness – risk of adverse effects, prevalence, and test accuracy – and to allow for the analysis of the methodological rigor of the systematic reviews that were included, we did not limit the inclusion criteria to systematic reviews of randomized controlled trials, but considered systematic reviews of any types of primary studies, including those of different types of non-randomized studies.

2.3.2 Search Strategy

We prioritized databases and other resources and began searching the potentially more productive databases (this approach termed or known as “purposive searching” [5] is described as being more efficient and less time-consuming than the typical comprehensive search strategies applied within systematic reviews of primary studies) [56]. These databases included the following types: databases of systematic reviews, databases with separate indexing of systematic reviews, guideline registries, general bibliographic databases, and then websites of relevant institutions. Search strategies, keywords, and limits are detailed in the resource publications [40, 41], and their detailed reporting exceeds the volume of this chapter. To document the process of searching, Table 18.3 has been included and shows databases and other resources that were searched in the “screening” umbrella review [40] together with the order of the search. In the “interventions” review [41], we performed a similar search, but with details specific to the different research question.

Table 18.3 Databases searched in the screening umbrella review together with the “purposive” [5] order of searching beginning with the potentially more productive databases [40]
Full size table

2.3.3 Study Selection

We independently conducted the searches as well as the initial selection of studies by their title and/or abstract. Full papers were then examined for eligibility. Disagreements were resolved by discussion. The combined PRISMA search flow for the selection of included studies for the two parts of the umbrella review is shown in Fig. 18.1.

Fig. 18.1
figure 1

Compiled PRISMA flow diagrams for the selection of included systematic reviews for screening (left) and treatment methods (right)

Full size image

2.3.4 Scope of the Systematic Reviews

Inclusion and exclusion criteria, formulated to the two umbrella reviews according to the PEO (problem/population–exposure/issue–outcome) and PICO (problem/population–intervention–comparator/control–outcome) principles, respectively, are summarized in Table 18.4.

Table 18.4 Criteria for inclusion and exclusion in the screening and intervention reviews
Full size table

2.3.5 Types of Outcomes

We analyzed and summarized both patient relevant and surrogate (or intermediate) outcomes of both the screening of patients with AIS as well as the effectiveness of nonsurgical interventions, taking into account qualitative and other not “numeric” issues. For us this is especially relevant as regards AIS – surrogate outcomes (e.g., curve angle, angle of trunk rotation, curve progression) are frequent in the available systematic reviews whereas patient-oriented ones (e.g., quality of life, body image) are less popular and frequent.

2.3.6 Data Extraction, Methodological Quality, and Level of Evidence Assessment

2.3.6.1 Data Extraction

We independently extracted the data, using predefined data extraction forms. Discrepancies were resolved through discussion.

2.3.6.2 Methodological Quality

We used the “Assessment of Multiple Systematic Reviews” (AMSTAR) risk of bias tool [57] to assess the methodological quality of included reviews. The AMSTAR comprises 11 items addressing criteria relating to the assessment of methodological rigor (Table 18.1). The items are scored “yes,” “no,” “cannot answer,” or “not applicable.” The maximum score is 11. Scores 0–4, 5–8, and 9–11 indicate low-, moderate-, and high-quality reviews, respectively [58]. We conducted the appraisal independently. Exceptions were the Cochrane reviews [59, 60] that were included and coauthored by JB-S, when MP and a collaborator performed the independent appraisals. Assessments were conducted using guidelines for scoring AMSTAR questions [57, 58]. Disagreements were resolved by discussion.

2.3.6.3 Levels of Evidence

We assessed the level of evidence from each included review, considering the types of studies included, using the new Oxford Centre for Evidence Based Medicine (OCEBM) [61, 62], the JBI [63, 64] classifications, and, for the screening reviews, additionally, the improved National Health and Medical Research Council (NHMRC) hierarchy [65]. We decided to apply more than one classification because different classifications vary as regards their content and characteristics, and this allowed us to assess the included papers more comprehensively. The NHMRC document proposed the only hierarchy corresponding to the characteristics of screening reviews, while the OCEBM and JBI classifications are more suitable for intervention reviews.

2.4 Results

2.4.1 Description of Included Reviews/Quantity of Research Available

For the screening review, six articles met the criteria for inclusion within the analysis from a total of 224 papers (see Fig. 18.1 and Table 18.1): two quantitative systematic reviews, one of which included a meta-analysis and four systematic analyses of evidence which were part of or supplementing recommendation documents. For the nonsurgical interventions, from a total of 469 titles or titles and abstracts of papers, 21 papers met the criteria for inclusion: 18 systematic reviews addressed the effectiveness of different interventions, 1 review evaluated usual physical activity, and 2 systematic reviews addressed the side effects in braced patients. Overall the reviews addressed numerous, patient-centered, and surrogate short- and long-term outcomes.

2.4.2 Methodological Quality of Included Reviews/Quality Assessment

Overall the quality of systematic reviews regarding screening ranged from the comparatively recent (2009) moderate-quality (AMSTAR score 6) (Fong et al. [66] and Sabirin et al. [67]) through to the outdated (2002) moderate-quality NHMRC review [68] to the poor-quality recent (2011) UK NSC [46] and the outdated 2004 USPSTF review [69], that is nonetheless still used for current and recent recommendations [52, 70]. The quality of the intervention reviews ranged from low methodological quality to high quality. Only two of the included reviews were of high quality [61, 62], while three were of moderate quality [72–74], and the remaining 16 reviews were found to be of low methodological quality [75–90] (Table 18.5).

Table 18.5 AMSTAR ratings for reviews included in the quality analysis
Full size table

2.4.3 Levels of Evidence of the Included Reviews

The six reviews relating to screening that we analyzed matched neither the improved NHMRC nor the new OCEBM levels of evidence hierarchy, with the exception of Fong et al. [66], which can be classified as a level 3 evidence in the OCEBM classification (Table 18.2). The levels of evidence from the reviews on interventions ranged from 1+ to 4, with some reviews not matching the OCEBM and the JBI hierarchies. The classification of the levels of evidence depended mainly on the type of included individual studies and also on the specific level of evidence hierarchy applied (Table 18.6).

Table 18.6 Evidence from higher quality, more recent systematic reviews on screening and nonsurgical interventions in AIS
Full size table

2.5 Discussion

Detailed narrative characteristics of the content of the reviews that were included in this umbrella reviews are out with both the volume and the scope of this chapter. Therefore our report was limited to a short description of the quantity and quality of the evidence from the systematic reviews that were included and the characteristics of the evidence available from the most recent and methodologically sound reviews. All relevant detailed information, comprising all the included papers as well as all the excluded papers, can be found in the original publications [40, 41].

2.5.1 Brief Summary of Evidence from Included Reviews

The evidence from included higher quality reviews is summarized in Table 18.2 according to the type of management (screening and nonsurgical interventions) and in the order of descending levels of evidence. The table does not include any studies on “usual physical activity” and adverse events [88–90] nor some highly cited older systematic reviews [81, 86] as all those reviews were of low or very low quality.

2.5.2 Screening

The screening reviews were heterogeneous, both with regard to the research questions asked and the methodology used for their development. The reviews which supported the recommendation for school screening as well as those recommending against screening are based on different papers selected for inclusion and thus on different evidences or assumptions of the absence of evidence (Fig. 18.2). Conclusions were based on different criteria as follows: the set of criteria for appraising – feasibility, effectiveness, and appropriateness of screening programs, accuracy of screening tests, treatment effectiveness as a criterion justifying the need for screening, and cost-effectiveness. Three of the systematic reviews were found to be of moderate quality; Fong et al. ’s systematic review of retrospective cohort studies with a meta-analysis [66] as well as MaHTAS systematic review by Sabirin et al. (2010) [67] supported screening under certain conditions. The NHMRC document [68], which included a recommendation against screening, was also found to be of moderate quality.

Fig. 18.2
figure 2

Example Venn diagram showing overlaps across included systematic reviews: overlaps of papers included in four screening systematic reviews. Numbers and percentages inside the ellipses show the overlap of studies included in the four reviews. Percentages outside the ellipses illustrate the percentage of all 109 studies included in each of the four reviews. n number of papers included in the reviews, USPSTF the US Preventive Services Task Force, MaHTAS Health Technology Assessment Section, Ministry of Health Malaysia, UK NSC the UK National Screening Committee; this review also included the systematic review by Fong et al.

Full size image

2.5.3 Scoliosis-Specific Exercises (SSE)

The most recent of the available reviews was the rigorous Cochrane review [60]. This provided no convincing evidence from RCTs for or against these interventions in terms of curve progression as a primary outcome and no evidence of risks or side effects from performing scoliosis-specific exercises. A moderate-quality review by Negrini et al. [74] recommended the use of SSE exercises based on primary studies classified by the authors as level 1b evidence.

2.5.4 Manual Therapies

A recent good-quality systematic review of controlled studies [72] found one high-quality RCT showing no evidence to support osteopathic manual therapy as an effective treatment for mild AIS.

2.5.5 Bracing

A rigorous Cochrane review [59] found very low-quality evidence supporting the effectiveness of bracing in reducing curve progression and low-quality evidence favoring hard braces as compared to soft braces. In an earlier, moderate-quality systematic review of prospective controlled trials, Lennsinck et al. [73] concluded that due to the low-power, weak methodological quality, and clinical heterogeneity of the included studies, drawing firm conclusions was impossible. However the effectiveness of bracing and SSE treatments in reducing curve progression appeared to the authors to be promising.

2.5.6 Quality Analyses

The methodological quality of the majority (19 out of 27; see Table 18.5) of the systematic reviews that were retrieved was disappointingly low (Table 18.5), regardless of the limitations of the primary studies included in the reviews. Common errors included the following: no second independent reviewer and blind study selection and/or data extraction, no lists of included and excluded studies, no comprehensive search for evidence, and, perhaps most importantly, no quality assessment of included studies conducted. This is crucial as if the studies in question were of poor quality. then we should not be basing clinical practice or recommendations on the results of these reviews.

The screening reviews also differed significantly with regard to the databases selected and other resources searched. Moreover, significant heterogeneity was found within the reviews as follows: different research designs were considered (prospective trials and retrospective observational studies, systematic reviews, editorials), which were, except in the Fong et al. meta-analysis [66], analyzed separately.

In some of the reviews, the level of evidence hierarchy classification (categories of studies) was reported as a quality assessment. Further, a number of excluded reviews (listed in detail, with reasons for exclusion, as supplementary files to the resource publications) were called “systematic” but actually comprised only a structured and systematic literature search and then presented as a narrative discussion of a few papers of diverse designs. The only intervention systematic review with a meta-analysis by Rowe et al. [86] was seriously flawed methodologically (Table 18.6). Further and crucially the patient group was not homogenous and did not differentiate between juvenile and adolescent IS.

2.5.7 Limitations of the Study

Firm conclusions cannot be drawn from this umbrella review as it cannot clearly be established from the individual systematic reviews included that the interventions tested may differ significantly from each other within similar papers even if they have the same label (e.g., bracing – there are considerable differences in the construction, biomechanical principles of action, as well as the length of time worn [49]). The same applies to scoliosis-specific exercise treatment with at least six different schools of thought and approaches available [50].

Secondly, we were not able to firmly distinguish the methodological quality of the process for conducting systematic reviews from the quality of reporting of the reviews that we included and analyzed. The AMSTAR tool does not clearly distinguish between the two, and we did not utilize any measure of the reporting quality of the included reviews (such as PRISMA). It is also important to consider that this overview will need an update every few years to reflect recently published work. This could be undertaken by using another (or modified) appraisal tool rather than the AMSTAR tool (as the appraisal tool itself may influence the findings from an overview of systematic reviews [91]). The reliability and validity of the findings of umbrella reviews conducted with the use of different versions of this tool requires further studies [92].

2.6 Conclusions: What this Study Adds

2.6.1 Methodological Considerations

When many systematic reviews exist about a given topic, it is critical that the analysis of the methodological quality (i.e., the rigor for the development) of the systematic reviews and – consequently – their credibility are fully assessed as well. Systematic reviews should not only be considered as the base for an umbrella review that summarizes and synthetizes the findings from the currently available systematic reviews [29]. In this scenario an umbrella review’s role is not only to summarize or synthetize but also to critically appraise, analyze, and assess the limitations of the systematic reviews that are currently available [13]. The role of an umbrella review is very important as poorly conducted systematic reviews may mislead and provide erroneous guidance for stakeholders – patients, therapists, insurance providers, policy makers, and researchers.

2.6.2 Conclusions of Case Study Issue

In this chapter we have explained why an umbrella review undertaking a best evidence synthesis approach was urgently needed for the school screening and nonsurgical treatment of adolescents with idiopathic scoliosis. We have described how we performed the two reviews and illustrated how umbrella reviews may be useful in informing end users in avoiding the misinterpretation of the available evidence from reviews of various quality and credibility.

2.6.3 Implications for Practice

A recent Cochrane Collaboration’s analysis of the process of guideline development has revealed the underutilization of systematic reviews and meta-analyses in developing practice guidelines [93]. The results of our study can aid policy makers and guideline developers in producing better evidence-informed, up-to-date guidelines, both for the screening of schoolchildren in the risk groups of the development of AIS and for the nonsurgical treatment of adolescents with idiopathic scoliosis.

2.6.4 Conclusions and Implications for Research

In conclusion, good-quality primary studies of higher level designs are urgently needed in the areas of screening and conservative methods for the treatment of adolescents with idiopathic scoliosis. Further developments in the conduct of systematic reviews, especially – in the case of the subject of this chapter – using multiple types of studies in systematic reviews [94], will hopefully facilitate finding the right answers to the complex and diverse research questions currently found in research, practice, and policy, through conducting more valid and reliable umbrella reviews based on the greater trustworthiness currently found within systematic reviews.

3 Additional Details

We would like to thank Dr. Igor Cieśliński for his contribution to the AMSTAR assessment of the included Cochrane reviews. MP registered the protocol, conceived, and designed the experiments and prepared data extraction tables. MP and JB-S performed the experiments, analyzed the data, and contributed to the writing of the manuscript. The review followed the protocol with the exception of some details of database searching (additional specialty websites and guideline registries, instead of AMED, CINAHL, and EMBASE databases searches were conducted), as the “productivity scheme” of searching was preferred to the comprehensive search strategy [5, 57].