Skip to main content

Advertisement

SpringerLink
Log in

A guide to appropriately planning and conducting meta-analyses—Part 1: indications, assumptions and understanding risk of bias

  • Review
  • Published:
Knee Surgery, Sports Traumatology, Arthroscopy Aims and scope

Abstract

A meta-analysis is the quantitative synthesis of data from two or more individual studies and are as a rule an important method of obtaining a more accurate estimate of the direction and magnitude of a treatment effect. However, it is imperative that the meta-analysis be performed with proper, rigorous methodology to ensure validity of the results and their interpretation. In this article the authors will review the most important questions researchers should consider when planning a meta-analysis to ensure proper indications and methodologies, minimize the risk of bias, and avoid misleading conclusions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Data availability

Not applicable.

References

  1. Arthur W, Zaaza Z, Checketts JX, Johnson AL, Middlemist K, Basener C, Jellison S, Wayant C, Vassar M (2020) Analyzing spin in abstracts of orthopaedic randomized controlled trials with statistically insignificant primary endpoints. Arthroscopy 36:1443-1450.e1. https://doi.org/10.1016/j.arthro.2019.12.025

    Article  Google Scholar 

  2. Begg CB, Mazumdar M (1994) Operating characteristics of a rank correlation test for publication bias. Biometrics 50:1088–1101

    Article  CAS  PubMed  Google Scholar 

  3. Bhandari M, Devereaux PJ, Montori V, Cinà C, Tandan V, Guyatt GH, Evidence-Based Surgery Working Group (2004) Users’ guide to the surgical literature: how to use a systematic literature review and meta-analysis. Can J Surg 47:60–67

    PubMed  PubMed Central  Google Scholar 

  4. Bland M (2015) Estimating mean and standard deviation from the sample size, three quartiles, minimum, and maximum. Int J Stat Med Res 4:57–64

    Article  Google Scholar 

  5. Cohn LD, Becker BJ (2003) How meta-analysis increases statistical power. Psychol Methods 8:243–253

    Article  PubMed  Google Scholar 

  6. Cook DJ (1997) Systematic reviews: synthesis of best evidence for clinical decisions. Ann Intern Med 126:376

    Article  CAS  PubMed  Google Scholar 

  7. Dickersin K (1990) The existence of publication bias and risk factors for its occurrence. JAMA 263:1385–1389

    Article  CAS  PubMed  Google Scholar 

  8. Dijkman BG, Abouali JA, Kooistra BW, Conter HJ, Poolman RW, Kulkarni AV, Tornetta P, Bhandari M (2010) Twenty years of meta-analyses in orthopaedic surgery: has quality kept up with quantity? J Bone Joint Surg 92:48–57

    Article  PubMed  Google Scholar 

  9. Duval S, Tweedie R (2000) Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics 56:455–463

    Article  CAS  PubMed  Google Scholar 

  10. Egger M, Davey Smith G, Schneider M, Minder C, Mulrow C, Egger M et al (1997) Bias in meta-analysis detected by a simple, graphical test. Br Med J 315:629–634

    Article  CAS  Google Scholar 

  11. Farrokhyar F, Karanicolas PJ, Thoma A, Simunovic M, Bhandari M, Devereaux PJ, Anvari M, Adili A, Guyatt G (2010) Randomized controlled trials of surgical interventions. Ann Surg 251(3):409–416

    Article  PubMed  Google Scholar 

  12. Furukawa TA, Barbui C, Cipriani A, Brambilla P, Watanabe N (2006) Imputing missing standard deviations in meta-analyses can provide accurate results. J Clin Epidemiol 59:7–10

    Article  PubMed  Google Scholar 

  13. Gerbarg ZB, Horwitz RI (1988) Resolving conflicting clinical trials: Guidelines for meta-analysis. J Clin Epidemiol 41:503–509

    Article  CAS  PubMed  Google Scholar 

  14. Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, Norris S, Falck-Ytter Y, Glasziou P, DeBeer H (2011) GRADE guidelines: 1. Introduction—GRADE evidence profiles and summary of findings tables. J Clin Epidemiol 64:383–394

    Article  PubMed  Google Scholar 

  15. Guyatt GH, Oxman AD, Kunz R, Vist GE, Falck-Ytter Y, Schünemann HJ (2008) What is “quality of evidence” and why is it important to clinicians? BMJ 336:995–998

    Article  PubMed  PubMed Central  Google Scholar 

  16. Higgins JPT, Thompson SG, Deeks JJ, Altman DG (2003) Measuring inconsistency in meta-analyses. BMJ 327:557–560

    Article  PubMed  PubMed Central  Google Scholar 

  17. Higgins JPT, Thompson SG, Spiegelhalter DJ (2009) A re-evaluation of random-effects meta-analysis. J R Stat Soc Ser A Stat Soc 172:137–159

    Article  PubMed  PubMed Central  Google Scholar 

  18. Hooijmans CR, Rovers MM, de Vries RBM, Leenaars M, Ritskes-Hoitinga M, Langendam MW (2014) SYRCLE’s risk of bias tool for animal studies. BMC Med Res Methodol 14:43

    Article  PubMed  PubMed Central  Google Scholar 

  19. Hozo SP, Djulbegovic B, Hozo I (2005) Estimating the mean and variance from the median, range, and the size of a sample. BMC Med Res Methodol 5:13

    Article  PubMed  PubMed Central  Google Scholar 

  20. Israel H, Richter RR (2011) A guide to understanding meta-analysis. J Orthop Sport Phys Ther 41:496–504

    Article  Google Scholar 

  21. Jones C, Rulon Z, Arthur W, Ottwell R, Checketts J, Detweiler B, Calder M, Adil A, Hartwell M, Wright DN, Vassar M (2021) Evaluation of spin in the abstracts of systematic reviews and meta-analyses related to the treatment of proximal humeral fractures. J Shoulder Elbow Surg 30:2197–2205

    Article  Google Scholar 

  22. Kassirer JP (1992) Clinical trials and meta-analysis. N Engl J Med 327:273–274

    Article  CAS  PubMed  Google Scholar 

  23. Lewis S (2001) Forest plots: trying to see the wood and the trees. BMJ 322:1479–1480

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  24. Macleod MR, O’Collins T, Howells DW, Donnan GA (2004) Pooling of animal experimental data reveals influence of study design and publication bias. Stroke 35:1203–1208

    Article  PubMed  Google Scholar 

  25. Manta A, Opingari E, Saleh AH, Simunovic N, Duong A, Sprague S, Peterson D, Bhandari M (2018) A systematic review of meta-analyses in orthopaedic surgery between 2000 and 2016. Bone Joint J 100:1270–1274

    Article  PubMed  Google Scholar 

  26. McAuley L, Pham B, Tugwell P, Moher D (2000) Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses? Lancet 356:1228–1231

    Article  CAS  PubMed  Google Scholar 

  27. Olivo SA, Macedo LG, Gadotti IC, Fuentes J, Stanton T, Magee DJ (2008) Scales to assess the quality of randomized controlled trials: a systematic review. Phys Ther 88:156–175

    Article  PubMed  Google Scholar 

  28. Petrisor B, Bhandari M (2007) The hierarchy of evidence: Levels and grades of recommendation. Indian J Orthop 41:11

    Article  PubMed  PubMed Central  Google Scholar 

  29. Reed DA, Cook DA, Beckman TJ, Levine RB, Kern DE, Wright SM (2007) Association between funding and quality of published medical education research. JAMA 298:1002

    Article  CAS  PubMed  Google Scholar 

  30. Sacks HS, Berrier J, Reitman D, Ancona-Berk VA, Chalmers TC (1987) Meta-analyses of randomized controlled trials. N Engl J Med 316:450–455

    Article  CAS  PubMed  Google Scholar 

  31. Schiavo JH (2019) PROSPERO: an international register of systematic review protocols. Med Ref Serv Q 38:171–180

    Article  PubMed  Google Scholar 

  32. Slim K, Nini E, Forestier D, Kwiatkowski F, Panis Y, Chipponi J (2003) Methodological index for non-randomized studies (Minors): development and validation of a new instrument. ANZ J Surg 73:712–716

    Article  PubMed  Google Scholar 

  33. Stern JM, Simes RJ (1997) Publication bias: evidence of delayed publication in a cohort study of clinical research projects. BMJ 315:640–645

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  34. Sterne JA, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M et al (2016) ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ 355:i4919

    Article  PubMed  PubMed Central  Google Scholar 

  35. Sterne JAC, Savović J, Page MJ, Elbers RG, Blencowe NS, Boutron I et al (2019) RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ 366:l4898

    Article  PubMed  Google Scholar 

  36. Thoma A (2005) Challenges in creating a good randomized controlled trial in hand surgery. Clin Plast Surg 32(4):563–573

    Article  PubMed  Google Scholar 

  37. Thorlund K, Imberger G, Walsh M, Chu R, Gluud C, Wetterslev J, Guyatt G, Devereaux PJ, Thabane L (2011) The number of patients and events required to limit the risk of overestimation of intervention effects in meta-analysis—a simulation study. Biondi–Zoccai G (ed) PLoS One 6:e25491. https://doi.org/10.1371/journal.pone.0025491

    Article  CAS  Google Scholar 

  38. Wan X, Wang W, Liu J, Tong T (2014) Estimating the sample mean and standard deviation from the sample size, median, range and/or interquartile range. BMC Med Res Methodol 14:135

    Article  PubMed  PubMed Central  Google Scholar 

  39. Wetterslev J, Thorlund K, Brok J, Gluud C (2008) Trial sequential analysis may establish when firm evidence is reached in cumulative meta-analysis. J Clin Epidemiol 61:64–75

    Article  PubMed  Google Scholar 

  40. Wetterslev J, Thorlund K, Brok J, Gluud C (2009) Estimating required information size by quantifying diversity in random-effects model meta-analyses. BMC Med Res Methodol 9:86

    Article  PubMed  PubMed Central  Google Scholar 

  41. Wilke J, Krause F, Niederer D, Engeroff T, Nürnberger F, Vogt L, Banzer W (2015) Appraising the methodological quality of cadaveric studies: validation of the QUACS scale. J Anat Wiley-Blackwell 226:440–446

    CAS  Google Scholar 

  42. Wolff RF, Moons KGM, Riley RD, Whiting PF, Westwood M, Collins GS, Reitsma JB, Kleijnen J, Mallett S (2019) PROBAST: a tool to assess the risk of bias and applicability of prediction model studies. Ann Intern Med 170:51–58

    Article  PubMed  Google Scholar 

  43. Yavchitz A, Ravaud P, Altman DG, Moher D, Hrobjartsson A, Lasserson T, Boutron I (2016) A new classification of spin in systematic reviews and meta-analyses was developed and ranked according to the severity. J Clin Epidemiol 75:56–65

    Article  PubMed  Google Scholar 

Download references

Funding

No funding was received for the implementation of this study.

Author information

Authors and Affiliations

  1. Division of Orthopaedic Surgery, McMaster University, Hamilton, ON, Canada

    Jeffrey Kay & Darren de SA

  2. Department of Orthopedic Sports Medicine, Boston Children’s Hospital, 300 Longwood Avenue, Boston, MA, 02446, USA

    Jeffrey Kay

  3. Department of Orthopedic Surgery and Sports Medicine, Sports Medicine and Shoulder Service, Hospital for Special Surgery, New York, NY, USA

    Kyle N. Kunze & Ayoosh Pareek

  4. Department of Orthopaedic Surgery, Amsterdam Movement Sciences, Amsterdam UMC, Location AMC, University of Amsterdam, Meibergdreef 9, 1105 AZ, Amsterdam, The Netherlands

    Jari Dahmen

  5. Department of Orthopaedics, Sahlgrenska University Hospital, Sahlgrenska Academy, Gothenburg University, Gothenburg, Sweden

    Jon Karlsson

Authors
  1. Jeffrey Kay
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kyle N. Kunze
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ayoosh Pareek
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jari Dahmen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Darren de SA
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jon Karlsson
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed substantially to conception and design, or acquisition of data, or analysis and interpretation of data; drafted the article or revised it critically for important intellectual content; provided the final approval of the version to be published; and agreed to act as guarantor of the work (ensuring that questions related to any part of the work are appropriately investigated and resolved).

Corresponding author

Correspondence to Jeffrey Kay.

Ethics declarations

Ethical approval

This review did not involve primary data collection from patients.

Conflict of interest

The authors declare no conflict of interest with regards to this publication.

Informed consent

None.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kay, J., Kunze, K.N., Pareek, A. et al. A guide to appropriately planning and conducting meta-analyses—Part 1: indications, assumptions and understanding risk of bias. Knee Surg Sports Traumatol Arthrosc 31, 725–732 (2023). https://doi.org/10.1007/s00167-022-07304-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00167-022-07304-9

Keywords

Search

Navigation