Introduction

Over the last several decades, systematic reviews have become an integral part of the medical and biomedical literature landscape. Systematic review methodology was pioneered in the 1980s and 1990s [1] and formalized over the ensuing years, culminating with the publication of the Cochrane Handbook for Systematic Reviews of Interventions [3] and the PRISMA Statement [16]. In the past two decades, there has been a significant increase in the number of published systematic reviews, and with over 30,000 review protocols registered in PROSPERO [19], the number of published reviews is expected to continue to grow. Ioannidis [9] found that in the period from January 1, 1986, to December 4, 2015, PubMed tags 266,782 items as “systematic reviews” and 58,611 as “meta-analyses.” Annual publications between 1991 and 2014 increased 2728% for systematic reviews and 2635% for meta-analyses versus only 153% for all PubMed-indexed items.

The overall quality of systematic reviews is also a topic of discussion in the literature. What was once considered a unique and highly sought-after publication used to inform clinical practices and health policy guidelines has become one of the most commonly published types of articles in the medical arena. However, growth does not necessarily indicates quality and objectivity. Bias in systematic reviews is a subject long discussed in various studies. Among the most common issues are: (1) Exclusion of non-English clinical trials reports [10, 18] (2) Mostly reporting on positive results [11] (3) The inclusion of unpublished trial outcomes [7, 8] (4) Corporate influence and its involvement in sponsoring systematic reviews [4].

The high volume of systematic reviews publications has raised several questions regarding their quality, necessity and applicability. Several studies indicated that an increasing number of such publications are focusing on unimportant questions, that many are redundant and unnecessary and that a significant number of systematic reviews publications are inherently flawed [2], while only 3% of systematic reviews are both well done and clinically useful [17]. Ioannidis [9] concluded that instead of promoting evidence-based medicine, the mass production of systematic reviews serves mostly as a marketing tool and are mostly misleading, and using fragmented published information.

Systematic reviews have also been examined from a bibliometric perspective. Several bibliometric studies focusing on systematic reviews analysis from various perspectives were found in the literature. Most of these studies examined the geographic origins of the publications, the journals in which they were published, number of authors and institution and so on. Most studies found that most systematic reviews originated in the United States, the United Kingdom, Australia and Canada, followed by China in recent years [6, 13, 14]. There is a relatively small group of journals that publish the majority of systematic reviews, led by PLOS Medicine and The Lancet [5, 24]. Finally, studies found that the overall impact of systematic reviews from a citations perspective is moderate, with very few that can qualify as highly impactful [6, 24].

The current study aims to deepen our understanding of systematic reviews as a publishing phenomenon on the one hand and analyze their impact on the other. Therefore, the purpose of this study is two-fold. First, it seeks to outline some of the temporal characteristics of systematic reviews, such as their growth rate, the top institutions that publish them, the most common topics covered in them and how these topics have varied through the years. Second, it seeks to analyze systematic reviews from scientific and social impact perspectives in order to assess their value. Indicators include qualifiers such as journals’ impact, citations, readership and various media mentions.

The paper therefore addresses a series of questions that were organized based on the two main themes listed above as follows:

  1. (1)

    Overall characteristics—This section covers the temporal characteristics of systematic reviews including number of published reviews by year, leading institutions, main topics and their evolution through the years.

  2. (2)

    Scientific impact—This section covers the impact of systematic reviews from a scientific perspective, as revealed by the following bibliometric indicators: citations rates, journals’ impact and usage.

Methods

Data Collection

Since this study focuses on temporal characteristics of systematic reviews and their impact over time, it was important to include as many publications as possible from the earliest year available until today. To achieve this, Clarivate Analytics Web of Science database was queried for the term “Systematic Reviews” in the title of the publication. The rationale behind using this simple query is its straightforwardness, as all published systematic review articles include this phrase in the title of the article. This query was not limited by year and retrieved records from as early as 1886. The query, which was conducted in August 2019, resulted in 134,109 records from 1934 to 2019. The full data was used to analyze the temporal characteristics of the dataset, including publications by year, growth rates, and number of publications by affiliation and topical analysis.

However, to track the scientific and societal impact of a publication, a smaller set of records with unique identifiers was used. Unfortunately, due to missing identifiers, such as a digital object identifier (DOI) or a PubMed ID, not all records could be analyzed using our analytical platform. These identifiers are crucial in order to track citations and other impact indicators. Therefore, the dataset used for tracking impact indicators comprised 106,038 publications that had a unique identifier.

Data Analytics Platform

The 106,038 records that contained identifiers (DOI or PubMed ID) were uploaded to PlumX for analysis. PlumX is a platform built to track a variety of impact metrics associated with ‘artifacts’ including peer-reviewed papers, conference proceedings, books and more. The platform is licensed by the authors and is able to not only ingest a variety of identifiers, including DOIs, PMIDs, ISBN numbers, and more, but also track, in addition to citations, a wide variety of altmetric indicators, such as downloads, views, clicks and social and news media mentions. These metrics are collected from dozens of outlets and are aggregated per each record. Finally, all metrics are updated continuously and are available for download for further analysis.

Findings

Temporal Characteristics

Overall Growth by Year

Systematic reviews were identified as early as 1866, with two identifiable publications including Scudder’s systematic review on fossil insects [21] and Wilhelm’s systematic review on ‘class of birds’ [12].

Until the mid-1990s, systematic reviews were published quite sporadically, with no evidence of steady or significant growth. While certain years had seen more systematic reviews publications, they never exceeded 12–13 in peak years. From 1994 on, the systematic reviews arena began to expand dramatically, demonstrating exponential growth rate since 1990 (see Fig. 1).

Fig. 1
figure 1

Systematic reviews growth by year

Full size image

This could probably be attributed to the establishment of the Cochrane Database of Systematic Reviews, whose first issue was published in 1994. According to Turner et al. [22], the publication of the database had an undeniable effect on the vast growth of systematic reviews publications.

Top Institutions

Our analysis identified 2133 publishing institutions. Table 1 depicts the top 10 publishing institutions and their respective countries. Unlike other studies, our data show that institutions in the United Kingdom, the Netherlands, Canada and Australia are among the top producing institutions for systematic reviews, led by the University of London, University of Amsterdam, University of Toronto and University of Sydney, respectively. In the United States, the Mayo Clinic and University of California system lead this trend, but with much fewer systematic reviews publications (see Table 1).

Table 1 Leading institutions and countries in systematic reviews publications
Full size table

Recurring Topics for Top Three Producing Institutions

We analyzed the recurring topics published for the top three institutions (University of London, University of Amsterdam and University of Toronto). In order to map these topics, we collected the titles of all articles per institution, removed stop words (i.e. of, in) and then uploaded the text to https://wordcounter.io/ [23]. This free web-tool identifies one-word frequency as well as two- or three-word phrases based on the number of times that they appear in the title. For this analysis, we used the three-word phrase recurrences since they reveal topics in a clearer manner. Table 2 depicts the top three recurring topics per institution.

Table 2 Top recurring topics of systematic reviews per institution
Full size table

Although there is not much in common between the areas of research or institutions, PTSD and type 2 diabetes were the two topics, on which the University of London and the University of Toronto published the most systematic reviews.

Topic Shifts by Decade

In order to gain some insight into the history and evolution of systematic reviews from a research perspective, we mapped the topics of systematic reviews publications by decades. In order to do so, we sorted the publications by year, collected all the titles per decade, removed stop words and then uploaded to https://wordcounter.io/ [23]. We used the three-word phrases identified by the web-tool in order to map these topics. As can be seen in Table 3, the first four decades show topics focusing on animal species and bio-ecologies such as animal forest, beetles and rats as well as specific bio-ecologies around the world. The purpose of these types of systematic reviews was to recommend best procedures for identifying specific species or identifying species in specific areas of the world based on a collection of specimens and fossils. Table 3 also demonstrates that the shift from topics related to species identification and bio-ecologies to the human medical and bi-medical arenas began relatively late, in 2000. Since 2000, topics such type 2 diabetes, irritable bowel syndrome, lung cancer and others are seen in the titles of published systematic reviews. The growth in publications within this period could be related to this shift in research topics. As seen in Fig. 1, the significant growth in publications occurred around the same time, therefore drawing attention to the potential link between medical-related systematic reviews and the overall growth of this type of publication.

Table 3 Top recurring topics by decade
Full size table

Scientific Impact

In this section, we examine the scientific impact of systematic reviews through the number of citations and the impact factor of the journals in which they are published.

Average Citations Per Year

In order to analyze the citation impact of systematic reviews, we collected the total number of publications per year and total number of citations per paper. However, it is known that publications gain more citations over time. In order to account for this natural skewness, we calculated the average citations per publication. As can be seen in Fig. 2, the average number of citations per systematic review paper decreases the more such reviews are published, diminishing the scientific significance of these papers. Special notice should be paid to the drop in average citations per paper from 2015 to 2018. These years have seen an astonishing growth in the number of systematic reviews while displaying the lowest number of citations per paper.

Fig. 2
figure 2

Average citations per paper by year

Full size image

The main purpose of systematic reviews in the medical arena is to provide medical ‘best practices’ by curating all available data from published research. Therefore, one of the most significant indicators of impact, especially when it relates to systematic reviews, is for a paper to be cited in an official policy document. In order to identify such citations, we analyzed the type of citation using PlumX [20]. PlumX not only provides the total number of citations but also distinguishes citations in policy documents. Out of the 105,000 + systematic reviews documents that we uploaded to PlumX, only 38 reviews were cited in policy-related documents. Table 5 lists the number of citations in policy documents compared to the overall number of systematic reviews publications per year. It should be noted that none of these articles was cited more than one time. In addition, the first year that policy citation appears in our analysis is 2005, five years after the above-seen growth in publications, and the last year that we see any citation of a systematic review in a policy document is 2016. Finally, each year from 2005 to 2010, we see only one annual citation of a systematic review article in a policy document, whereas the total number of systematic reviews publications reached over 1,000 from 2006 (see Table 4). This could mean that the influence of these systematic reviews on actual health-related policies was grossly insignificant (Table 5).

Table 4 Number of citations in policy documents compared to the overall number of systematic reviews publications per year
Full size table
Table 5 Some examples of systematic reviews that were cited in policy documents
Full size table

Journal Impact

One of the indicators of the quality of scientific papers is the impact factor of the journal in which they are published. In this section, we examined the top 10 journals that were shown to publish the most systematic reviews and compared their impact factor scores (IF scores). Table 6 demonstrates that there is no correlation between the number of systematic reviews and the IF score, and in fact, the IF score seems to be higher for journals that publish fewer systematic reviews. As can be seen in Table 6, PLOS One publishes the most systematic reviews, with almost double the publications compared to BMJ Open. Interestingly, these journals both publish a significant number of systematic reviews and demonstrate similar IF scores, hovering in the 2.3–2.7 range. This could be another proof that systematic reviews do not generate high numbers of citations, which in turn translates into low IF scores.

Table 6 Top publishing journals and their IF score
Full size table

Usage

Usage metrics are an indication of utility and are considered ‘alternative metrics’ or altmetrics. PlumX tracks statistics for views and downloads from several publishers’ platforms as well as repositories. Views include articles that were viewed on a publishing platform, while downloads include articles that these systems tracked as downloaded via the various platforms. The importance of these metrics lies in the fact that they are indicators of interactions of readers with the publications. Whether articles are viewed online or downloaded to a local computer, these are indicators of readership. Figure 3 depicts the actual and average views and downloads of systematic reviews per year and per paper. It is clear that the number of views and downloads steadily declines from 2010. This means that in the past several years, systematic reviews have been viewed and downloaded less and less. This could be a direct result of the sheer number of these publications, which might be overwhelming to the point where they are no longer viewed as useful. This phenomenon can also be tied to the overall decline in quality and significance of these publications.

Fig. 3
figure 3

Average views and downloads (usage) per paper and per year

Full size image

Discussion

As can be seen in Fig. 4, the number of systematic reviews increased dramatically and almost perfectly exponentially during 1998–2018. The drop seen in 2019 is due the fact that not all articles published in 2019 were indexed at the time in 2020 when the data was analyzed. As can be seen, each of the data types demonstrates different orders of magnitude through the years. Generally, publication count ranges between 1 and 10,000, the average usage between 10 and 1000 and average citations between 1 and 10.

Fig. 4
figure 4

Number of systematic reviews published per year, and their cumulative average citations, usage and social media mentions until 2019

Full size image

The average number of citations per review shows a steady annual decline, which deteriorates further over the years. This is mainly because the citation counts are cumulative which means that while citations to reviews published in, for instance, 2010 can be accumulated over a time period of 10 years, those published in 2019 can generate citations only during one single year, namely the first year of their existence. In addition, the fact that citation distribution is known to follow a longitudinal pattern makes decline plotted in Fig. 4 for citations deeper.

The usage counts show a pattern that is similar to that of citations. Since usage counts tend to mature faster than citation counts, the curve for usage counts shows maximum usage around the year 2012, a phenomenon that is not visible for average citation counts [15].

Conclusions

Our data included 106,038 systematic reviews retrieved from Web of Science. In this study, we analyzed both the qualitative and qualitative aspects of this body of work. Our analysis shows that the growth of systematic reviews publications has been striking, soaring in triple digits from 1994. This could be related to the Cochrane Database of Systematic Reviews, which published its first issue at the same time.

Our scientific impact analysis included citations, citations in policy documents, journals’ impact factor and views and downloads. As can be seen, citations of systematic reviews are declining, and so are the numbers of views and downloads per paper. In addition, we found that very few systematic reviews are being used to inform policy around medical conditions. A mere 38 were cited in policy documents compared to the hundreds of thousands of articles that we analyzed. Finally, the IF score of the journals in which they are published averages 2.3–2.7, which is a low impact factor range, especially in the medical and biomedical arenas. In comparison, The New England Journal of Medicine has an IF score of 70.6; The Lancet, 59.1; and the Journal of the American Medical Association (JAMA), 51.273.

The sheer number of systematic reviews publications is not an indication of quality or of impact. In fact, our data show that these are on the decline. There seems to be saturation in this area, which results in less interest in and utility of systematic reviews. The main concern here is that systematic reviews are supposed to be used as a source for evidence-based medicine and for medical policy. The fact that their utility is declining is concerning, as it might be another indication of low quality and overall importance. Since systematic reviews require an enormous amount of time and work to compile, analyze and write, potential authors should consider these findings before embarking on such a project.

We identified some leading institutions that are seen to publish significant numbers of systematic reviews, which, in some cases, are twice as many as for others. While most institutions published a few hundred systematic reviews through the years, some have published thousands. These include the Universities of London, Amsterdam, Toronto, Sydney, and McMaster University. It is unclear why these universities publish more than others do. This could be related to their faculty size or the size of their residency programs or to internal, unknown requirements.

When looking at the most recurring topics of systematic reviews within the top three publishing institutions, there is not much in common except diabetes and PTSD, which are seen to be the topics appearing at two universities at the same time.

Interestingly, our topic mapping of systematic reviews through the decades shows that the switch to medical topics occurred in or around the year 2000. Beforehand, most of the topics that we identified were related to biology and bio-ecologies.