Introduction

When there is a considerable volume of literature dedicated to the study of a specific problem the results reported by different studies can seem contradictory, thereby making it difficult to integrate them and draw clear conclusions. As long ago as the 1970s, Gene V. Glass (1976) (Smith and Glass 1977) noted that there were differences in the conclusions reached by studies on the effectiveness of psychotherapy. It is to Glass that we owe not only the term meta-analysis, which he defined as ‘the statistical analysis of a large collection of analysis results from individual studies for purposes of integrating the findings (Glass 1976, p. 3)’, but also the idea that there were ways of combining and integrating results in an attempt to resolve or explain the differences and inconsistencies found in original articles, i.e., primary research. Glass’ paper was soon followed by the publication in the social sciences field of three other studies that used statistical procedures to combine the results of independent studies, specifically in relation to the effects of interpersonal expectations on behaviour (Rosenthal and Rubin 1978), the validity of employment tests for black and white workers (Hunter et al. 1979), and the relationship between class size and achievement (Glass and Smith 1979).

Although, therefore, meta-analysis has its origins in the field of psychology it has now, in a period where the amount of scientific work is almost unmanageable, become an indispensable tool in many other areas of knowledge (Schmidt 2008). Indeed, the last 30 years has seen a growing interest in the development and improvement of meta-analytic methods and their application to the study of specific issues.

A key point in the history of meta-analytic methods was the publication in the 1980s of the first substantive texts that sought to combine -comprehensively in a single document- the measures of effect size most commonly used in quantitative synthesis and the different methods for integrating results and analysing moderator variables (e.g., Cooper 1984; Glass et al. 1981; Hedges and Olkin 1985; Rosenthal 1984). These landmark publications have been followed by other reference manuals that also focus on meta-analytical methodology. Although this methodology essentially remains the same, new approaches have appeared and existing methods have been studied in greater depth. Furthermore, new methods for assessing publication bias and combining results have also been proposed. Noteworthy in this regard are the new editions of previously published books (e.g., Borenstein et al. 2009; Cooper 1998; Rosenthal 1991) and new contributions such as the books by Cooper and Hedges (1994); Hartung et al. (2008); Hunter and Schmidt (1990, 2004); Lipsey and Wilson (1996, 2001); and Rothstein et al. (2005), all of which provide a clear explanation of meta-analysis and its applications.

More recently, the meta-analytic approach has been used to study a wide range of topics including the effects of second-generation antipsychotics in the treatment of schizophrenia (Leucht et al. 2009), the relationship between body mass index and the incidence of cancer (Renehan et al. 2008), the effects of mobile phones on driver performance (Caird et al. 2008), the relationship between academic achievement and job performance (Kuncel et al. 2004), and the reliability coefficients of personality tests (Viswesvaran and Ones 2000).

Although meta-analysis has been supported by much of the scientific community some researchers (Bailar 1997; Feinstein 1995; Shapiro 1994; Sharpe 1997) have argued that it has a number of limitations, such as publication bias, reducing the effect to a single number, and the problems known as mixing apples and oranges, or garbage in, garbage out. Borenstein et al. (2009) provide arguments to refute these criticisms and conclude that these issues are also typical of narrative reviews. However, they also stress that the strength of meta-analytical reviews rests fundamentally on the steps of the meta-analytic procedure being clearly described, such that the selection and integration of results is completely transparent. As a result, meta-analysis is considered to be one of the most useful ways of reviewing a pool of scientific literature on a given topic.

One way of studying the body of meta-analytic literature would be to apply bibliometric procedures. Bibliometrics is increasingly used to systematically measure scientific output in order to understand the genesis of a discipline or area of knowledge and to map its publication pattern. Similarly to meta-analytic studies, bibliometrics needs hindsight to assess the trends observed in a particular subject area. In this context, the main aim of the present study was to analyse the evolution of meta-analysis in the field of psychology, from the first published paper to the present day, as well as to study the impact of these publications on science. To this end, standard bibliometric indicators (number of papers, number of authors and productivity by country, among others) and the standard bibliometric laws (Price’s, Lotka’s and Bradford’s laws) were applied in order to study the evolution of meta-analytic publications. To our knowledge this is the first study to address the meta-analytic literature using bibliometric techniques.

Methods

Data collection

The documents included in the present study were identified using the Thomson-ISI Web of Science (WoS), which includes the Science Citation Index Expanded (SCI-EXPANDED) and the Social Sciences Citation Index (SSCI) databases. In order to retrieve the available scientific literature on the subject the search was performed in the topic field (which runs the search in titles, keywords and abstract) and using the truncated form of the word meta-analysis (metaanaly* and meta-analy*). Documents included in the study were restricted to articles and reviews published up until December 2010 in journals classified in any of the eleven areas of psychology proposed by the WoS database.

A total of 8,861 articles were initially retrieved. However, the first inspection of results revealed that those articles in which the keyword meta-analysis only appeared in the keyword plus field (n = 5,289) were not related to the subject. To corroborate this, 50 of these articles were randomly selected and two independent reviewers confirmed that none of them was related directly to meta-analysis. All of these articles were therefore discarded. Titles and abstracts from the remaining pool of articles (n = 3,572) were then manually and independently checked by two reviewers to ensure that the paper actually was interested in or implemented a meta-analysis. Articles were coded by both reviewers, who achieved almost perfect agreement (Cohen’s kappa coefficient of 0.975). The final sample comprised 2,874 articles.

The number of citations for each article since its year of publication until 2010 was also obtained from the WoS database, while the impact factor of the most productive journals was acquired through the ISI Journal Citation Report database (JCR 2010 or, failing that, the most recent available). The format used for author names was standardised in order to avoid duplicates. When necessary the institution and co-authors were also taken into account. Each article was also classified according to whether its contribution was mainly empirical or methodological.

Data analysis

Descriptive statistics were used to study the main variables. In addition, other bibliometric indicators were calculated to study scientific growth over time, the dispersion of scientific output across journals and their impact, and the authors’ productivity. The first of these, Price’s law (Price 1963), proposes that the growth of scientific production over time follows an exponential function. In order to test whether our data fitted Price’s law, different regression models were performed including linear, exponential and logistic curves, the latter being applied to assess the hypothesis of literature growth saturation.

The second bibliometric tool, Bradford’s law (Bradford 1934, 1948), describes how the articles in a specific area are scattered across journals, postulating a model of concentric productivity zones with a decreasing information density. Following the proposal of Egghe (1986, 1990) the Bradford multiplier was obtained by \( k = \left( {1.781 \cdot y_{m} } \right)^{1/P} \), where y m is the number of articles published by the most productive journal and P is the number of zones including the nucleus. The predicted frequencies were also fitted according to Leimkuhler’s formulation (Leimkuhler 1967), obtaining the constants as \( A = y_{0} /\log_{e} k \) and \( B = \left( {k - 1} \right)/r_{0} \), where y 0 is the constant number of articles in each group (\( y_{0} = a/P \), where a is the total number of articles and P is as defined above), k is as defined above and r 0 is the expected number of journals in the core (\( r_{0} = \frac{T(k - 1)}{{k^{P} - 1}} \), where T is the total number of journals, and k and P are as defined above). Finally, the estimated cumulative number of articles produced by the journals of rank 1, 2,…, r was obtained by: \( R(r) = A \cdot \log_{e} \left( {1 + B \cdot r} \right) \).

The scientific impact of the articles from the most productive journals was then compared by means of the a index (Abt 2011), which is independent of the age of publication. Although this index was initially proposed to compare the scientific output of individual researchers working in a particular field and with different career duration, it was used here to compare journals with a different life span. The a index was defined by the quotient between the cumulative citations of the journal in meta-analytic articles L sum(t) and the square of years since the journal published its first meta-analytical paper t 2:

$$ a = \frac{{L_{\text{sum}} (t)}}{{t^{2} }}. $$

Finally, Lotka’s law was applied using the method proposed by Pao (1985), including in the analysis both the first author and co-authors. According to Lotka’s law the number of authors (y x ) with x number of articles is inversely proportional to x. This relationship is expressed by the formula:

$$ x^{n} \cdot y_{x} = C, $$

where y x is the number of authors producing x number of articles in a given research field, and C and n are constants that can be estimated from the observed data set. Although many authors take a value of two as the value of the exponent, as did Lotka in his original paper (Lotka 1926), it is known that the n exponent can vary depending on factors such as the inclusion of co-authors in the analysis or the number of pairs included in the calculation of the exponent (Pao 1985). For the present study, the least-squares method was used to calculate the n exponent, expressed by the formula:

$$ n = \frac{{N\sum {XY - \sum {X\sum Y } } }}{{N\sum {X^{2} - \left( {\sum X } \right)^{2} } }} $$

where N is the number of pairs considered, X is the logarithm of x and Y is the logarithm of y x .The constant C, the proportion of authors who contributed with one article to the subject area, is calculated using the formula:

$$ C = \frac{1}{{\sum {(1/x^{n} )} }} $$

Finally, in order to verify that the observed data fitted the estimated distribution, the non-parametric Kolmogorov–Smirnov goodness-of-fit test was applied.

Data analysis was performed using R version 2.12.1 and PASW Statistics 17.

Results

Of the 2,874 articles retrieved the vast majority were empirical studies about meta-analysis (n = 2,605, 90.6 %), with only 9.4 % (n = 269) being classified as methodological studies. The word meta-analysis appeared in the title of 77.6 % of the articles, while in the remainder it featured in either the abstract (21.6 %) or the key word field (0.8 %). As regards the number of citations received there was a significant difference [F(1, 2869) = 10.967, p = 0.01] between articles with and without the word meta-analysis in the title, this being an aspect which covariated with the impact factor of the journal (\( \bar{x} = { 52}. 5 2 \), SD = 94.36 and \( \bar{x} = { 39}. 2 4 \), SD = 72.47, respectively).

Temporal evolution

As shown in Fig. 1 there has been a clear upward trend in the percentage of publications since the mid-1970s. Specifically, 8.07 % of the articles were published during the 1970s and 1980s, 25.61 % were published during the 1990s, and 66.32 % during the first decade of the twenty-first century. Note that the peak of productivity was found in 2010, a year in which 10.09 % of the articles were published, and also that 50.52 % of the publications appeared between 2004 and 2010.

Fig. 1
figure 1

Temporal evolution of percentage of publications

Full size image

An interesting pattern was observed when considering the type of study over time. During the 1980s and early 1990s the annual percentage of methodological studies on meta-analysis was between 20 and 40 %, such that between 60 and 80 % of publications concerned applied meta-analysis. However, from the mid-1990s onwards the percentage of methodological studies falls to between 0 and 20 %, with applied studies accounting for between 80 and 100 %.

The linear, exponential and logistic regression models were fitted in order to test whether the data followed Price’s law. All three models were statistically significant and explained a similar proportion of the variance: linear (R 2 = 0.869) logistic (R 2 = 0.868) and exponential (R 2 = 0.862).

Authors

The total number of authors who contributed to the output set was 5,445, with the mean number of authors per article being 2.75 (SD = 1.53, Md = 2). The data showed that 17.12 % of the articles had a single author, 34.86 % had two, 25.12 % had three, and only 22.89 % had more than three authors. It can be seen in Fig. 2 that over time there is an upward trend in the number of authors per article.

Fig. 2
figure 2

Temporal evolution in the number of authors by article

Full size image

With regard to the productivity of authors the data showed that 79.5 % of them contributed just one item to the subject. Lotka’s law, which describes the productivity distribution among scientists, states that a handful of researchers are responsible for most of the literature, whereas the large majority contribute a very small number of publications. Here this law was evaluated using the method proposed by Pao (1985), and all the authors of the publications (first authors and collaborators) were considered for the analysis. To determine whether the data fitted Lotka’s law the n value was calculated using the least-squares method (n = 2,787), obtaining a C value of 0.80. The critical value obtained by the non-parametric Kolmogorov–Smirnov goodness-of-fit test was 0.022. As the maximum difference between the observed and the estimated accumulated frequencies was 0.005, which is below the critical value, we can conclude that the data fitted Lotka’s law (Fig. 3).

Fig. 3
figure 3

Data fit to Lotka’s law

Full size image

Countries

A total of 44 countries over four continents have participated in publishing meta-analytic studies, although it should be noted that the provenance of authors was not available for 193 articles (6.71 %). The top ten countries in terms of contributions are, in decreasing order: USA (57.52 %), Canada (7.41 %), England (7.41 %), The Netherlands (6.09 %), Germany (5.29 %), Australia (2.71 %), Spain (2.16 %), China (1.11 %), Italy (1.11 %) and Belgium (1.04 %). The other countries had participated in fewer than 30 publications each.

Collaborations

The mean number of participating countries was 1.16 per article, which indicates that a large proportion of publications are produced by authors from the same institution or through collaboration at national level. Among those articles where some kind of collaboration was observed (i.e., excluding single-author articles) and the name of participating countries was reported (n = 2,189, 76.17 %) the collaboration involved authors from the same country in 84.60 % of cases. Figure 4 shows how the type of collaboration (national/international) evolved over the time period analysed. It can be seen that national collaborations predominated during the first part of this period, but since the second half of the 1990s there has been an increasing number of international collaborations.

Fig. 4
figure 4

Temporal evolution of type of collaboration

Full size image

Journals

The articles included in the present study were published in a total of 394 journals, although some of these are noted for publishing a considerable number of meta-analytic studies. Differentiating their contribution in terms of type of study (methodological or empirical), Table 1 shows the ten most important journals in this regard, which together published just over 25 % of all the documents. Note that most of the scientific journals are interested in meta-analytical studies applied to different substantive areas, the exceptions being the journals Educational and Psychological Measurement and Psychological Methods, which mostly publish papers on advances in meta-analytic methodology; this is particularly the case of the latter journal, which publishes the highest number of methodological papers in absolute terms. The most productive journal, Psychological Bulletin, was also the journal with the highest a index (i.e., 38.15). A significant correlation was observed between the a index and the impact factor of the journal (r = 0.813, p = 0.004, R 2 = 0.661), indicating that the citation of meta-analytic papers makes a high contribution to the impact factor of the journal.

Table 1 Top ten productive journals
Full size table

Application of Bradford’s law revealed that the articles were distributed in nine zones.Footnote 1 Table 2 shows the expected number of journals in each zone (r 0 k, r 0 k 2,…, r 0 k P−1) given the Bradford multiplier (k = 1.95), the actual number of journals in each zone, the number of articles included in each zone, the cumulative number of articles, the estimated values of k (ratio of the number of journals in any zone to the number of journals in any immediately preceding zone) and, finally, the predicted cumulative number of articles R(r). To test if the data fitted the Bradford’s law (1:n:n 2 :…:n P−1), the percentage error when comparing the actual and predicted cumulative number of journals was obtained. Taking the mean value of estimated k (1.93), the percentage error reached a value of 1.09 %, suggesting the goodness-of-fit of Bradford’s law. Note that the ten most productive journals are included in the first four Bradford zones (nucleus and zones 1–3). The estimated parameters for Leimkuhler’s formulation were A = 479.18 and B = 1.02, hence the predicted cumulative number of articles published by journal rank was given by \( R(r) = 478.17 \cdot \log \left( {1 + 1.02 \cdot r} \right) \) and are also shown in Table 2.

Table 2 Data fit to Bradford’s law
Full size table

Journal areas

According to the classification of journal areas used in the WoS the area with the highest frequency of articles about meta-analysis was clinical psychology, followed by psychology, multidisciplinary psychology and applied psychology. In order to examine the growing trend in each area and the data fit of Price’s law, linear, exponential and logistic regression models were applied. Table 3 shows the frequency and percentage of each area and the proportion of variance explained by each regression model. All the areas showed an upward trend, although given the low frequency of articles in the areas biological and psychoanalysis, regression models were not performed in these cases. Applied psychology showed a better fit to the linear model, while the remaining areas followed Price’s law, showing a good fit to the exponential model.

Table 3 Regression fit of psychological areas according to the WoS classification
Full size table

Discussion

The present study has analysed the main bibliometric indicators in relation to the psychological literature involving meta-analysis. The results show a significant presence of meta-analytic studies in the field, as well as a growing trend towards the use of this approach. This interest extends to all areas of psychology, as evidenced by the fact that most of the psychological areas fit an exponential model.

However, this upward trend in productivity does not apply equally to methodological and applied studies. Applied studies are far more common than are methodological contributions (90.7 vs. 9.3 % of the articles), and the analysis of temporal evolution showed that methodological studies were present mainly from the 1980s to the early 1990s. Interestingly, the growth in applied studies during the 1990s coincides with the increase in both international collaborations and the number of authors per article.

Given that meta-analysis is essentially a methodological approach one would expect to find that a considerable number of initial publications would be devoted to explaining, comparing and discussing the advantages and disadvantages of meta-analytic procedures, and that subsequently these methodological contributions would be used in meta-analyses of specific topics. The results of the present study are entirely consistent with this pattern. Specifically, we observed that while methodological studies predominated in the early years (the 1970s and 1980s), these initial efforts by various authors to provide, discuss and improve the different methodological contributions have led over the last two decades to an increased number of empirical studies applying these new procedures.

The temporal analysis also showed an increase in the number of authors per article. Whereas in the late 1970s and 1980s the majority of articles were the work of one or two authors, nowadays more than a third of publications have three or more authors. This increase in the number of authors has also been accompanied by greater international collaboration, which now accounts for around 25 % of articles.

It is also worth noting that most of the publications use the word meta-analysis or a synonym in the title of the article. This not only indicates the relevance of the technique itself, but also serves as a mechanism for authors to highlight the nature of their work and ensure it is easily identified by the scientific community. This latter aspect is corroborated by the fact that those articles which have the word meta-analysis in the title receive more citations than do those which do not use this word or a synonym, this being the case even when the number of citations was controlled for by the impact factor of the journal.

In view of the present findings, and despite some of the criticism levelled against the quantitative integration of results, it appears that meta-analysis is widely accepted among the scientific community as a technique for integrating quantitative data. In fact, while criticism of meta-analysis soon emerged, it was mostly confined to the 1990s. Since the first meta-analytic study in the field of psychology the number of publications has steadily grown, especially in recent years, such that consideration of the study period in terms of 5-year intervals would show successive growth in the number of publications. This finding is not surprising if one considers the high scientific value that has been ascribed to review studies, as well as the fact that the objective of meta-analysis is precisely to collate the results of previous studies and, therefore, to produce a considerable volume of literature over time.

In this context it is worth noting the emphasis that the scientific community has placed on the integration of research results through reviews. If an author is looking for an up-to-date view on a specific topic it is highly likely that he or she will consult a narrative review and/or meta-analytic study, if they exist. Of course, a meta-analytic study will allow the author to draw conclusions that cannot be reached by narrative reviews. Therefore, reviews tend, on average, to receive more citations than do other types of papers (Amin and Mabe 2000; Dong et al. 2005), and there has been much discussion about whether or not the citations received by reviews should be included in the calculation of a journal’s impact factor (Moed and Van Leeuwen 1995). The results of the present analysis suggest that the psychology journals with the highest impact factors are precisely those that have published a greater number of meta-analytic papers. It would therefore be interesting to explore whether this also occurs with methodological papers on meta-analysis, and to study each particular journal across its history.

In summary, the present study provides an overview of the evolution of scientific output as regards the use of meta-analysis in psychology. The results show that since the first such publication until the present day there has been a clear upward trend in the number of articles, the number of authors by article and in international collaborations. These findings suggests that meta-analytic activity is far from reaching any kind of peak and one would expect to see attention turning not only to all fields of knowledge but also to the updating of existing reviews when the publication of new original findings makes this necessary.