Introduction

Scientists have access to the mass media like everybody else, which suggests that the scientific impact of a research will be boosted by coverage in newspapers, magazines, television, radio etc.… (Bucchi and Mazzolini 2003; Kiernan 2003; Martin-Sempere et al. 2008; Tsfati et al. 2011). The importance of this effect needs to be carefully assessed, not only to improve the communication strategies of researchers, but also because a “publicity effect” could represent a distorting factor in the scientific process. Science and the media have fundamentally different agendas (Burnham 1987; Stryker 2002; Vantrigt et al. 1995): the former is (or should be) determined exclusively by empirical and theoretical significance, whilst the latter tend to promote research that is unusual, controversial, fashionable, related to sex and health, simple to understand, with immediate applications, etc.… (Bartlett et al. 2002; Clark and Illman 2006; Elmer et al. 2008; Entwistle 1995; Stryker 2002, see also Song et al. 2010). A feedback of the media on research, therefore, might alter traditional scientific priorities and values.

To the best of the author’s knowledge, only two studies have assessed directly the effects of media reporting on scientific impact, measured by citation counts. In what are almost unrepeatable circumstances, Phillips et al. (1991) counted the number of citations received by papers in the New England Journal of Medicine that were featured in the New York Times, and compared them to papers that had been “earmarked” by NYT editors but never reported due to a strike in 1978. The former received over 70 % more citations than the latter during the following 10 years, which strongly suggested that appearing in the NYT increases the scientific impact of a study (defined as the “publicity” hypothesis) (Phillips et al. 1991). A later study, which covered other scientific journals and different kinds of media in the US, also found a citation advantage for stories that had made the news, although it could not exclude that these stories were intrinsically more important than the others, and only for this reason were cited more often (defined as the “earmark” hypothesis) (Kiernan 2003). Of course, the two effects are not mutually exclusive.

This study measured the relative importance of the publicity effect, by comparing the frequency of citations of studies featured in prominent British and Italian newspapers (The Sun, Daily Mail, The Mirror, The Times, The Guardian, La Repubblica, Il Corriere della Sera and La Stampa), with a ‘control’ sample of studies that were not featured in any of these. We predict that, if the publicity effect is true and prevailing over the earmark effect, coverage by British newspapers should be linked to a greater increase in citations compared to the Italian ones, and that these latter should increase citations from Italian authors. This follows form a few simple considerations. Most newspaper content is now available online—which is where the news articles included in this study were sampled from. Most active scientists make routine use of the Internet, which gives them potential access to the media of any country and language. English is the language of scientific communication, spoken by the vast majority of living scientists, whilst Italian is used by a relative minority, mostly residing in Italy. Therefore, the potential scientific audience for news reporting is much greater for English-speaking media.

The probability of a paper to be cited is known to vary by discipline and with journal impact factor (Perneger 2010). To control for these confounding effects, the analyses focused on studies published in one high-impact multidisciplinary journal: Proceedings of the National Academy of Science. The study first assessed whether newsworthiness criteria differ between the two countries, then compared citation patterns.

Methods

Sample collection

The terms “PNAS” and “Proceedings of the National Academy of Sciences” were used to search the online archives of Corriere della Sera, La Repubblica, La Stampa, and the Lexis-Nexis database for articles from The Sun, Daily Mail, The Mirror, The Times, and The Guardian. The search was limited to articles published in 2008 and 2009.

These newspapers were selected because they are generalist, published daily, and are amongst the most widely read in their respective countries (based on circulation figures from Italy’s Accertamento Diffusione Stampa and UK’s Audit Bureau of Circulations). The number of British newspapers included in the sample is higher because very few relevant articles could be retrieved from The Sun, and The Mirror. Moreover, these latter (and the Daily Mail) represent “tabloid” newspapers, while The Guardian and The Times represent “quality” journalism. The “tabloid” format does not exist among Italian daily newspapers.

All articles retrieved from Italian newspapers were included in the analysis. All articles from The Sun and The Mirror were included, while 20 articles were sampled at random from the Daily Mail, Guardian and Times. The analysis focused on news reports and excluded feature-style (i.e. longer) articles. Inclusion of these latter, however, did not alter the results in any substantial way.

The “control” papers were obtained by sampling at random N = 150 scientific articles from the 2008 and 2009 issues of PNAS, and then excluding those that had been reported in any of the newspapers considered.

Papers were sampled using a pseudo-random number generator, assuming a uniform distribution.

Data collection

For each newspaper article included in the analysis, we retrieved the scientific study it referred to and recorded its discipline. PNAS classifies its papers in a multitude of disciplines and sub-disciplines. To simplify the analyses, disciplines were grouped in the three traditional domains of physical, biological and social sciences, and were divided between pure and applied, following criteria of previous studies (Fanelli 2010). Regression analyses that used a higher number of disciplines yielded substantially similar results, and are not reported for brevity. PNAS classifies psychology as both a social and a biological science. To make results comparable to previous studies, psychology was classified as a social science.

To verify whether scientific papers had been accompanied by a press release, the online archive of Eurekalert (http://www.eurekalert.org) was searched using key words and the names of authors of each study.

All papers citing the studies included in the sample were retrieved on 22 April 2012, and classified as self or non-self citations following a maximally conservative procedure. If the surname and initials of at least one of the authors corresponded to one of the authors in the cited study, the paper was considered a self-citation and excluded. For brevity, the text will use the term “citations” to indicate non-self citations. Self-citations are excluded to improve the accuracy of the study, although including them did not change the results in any substantial way.

A similarly conservative logic was used in attributing papers and citations to countries: if any of the co-authors was based in Italy or the United Kingdom (i.e. England, Scotland, Wales and Northern Ireland), the paper was classified as coming from that country.

Statistical analyses

Multivariate analyses were conducted with a generalized linear model (henceforth GLM). Data comparing newsworthiness between Italy and the UK used a non-weighted GLM with binomial distribution. Analyses on citation frequencies, instead, assumed a Poisson distribution of errors, whilst those on the proportion of citations from Italy and UK assumed a binomial distribution and weighted the proportions by the total number of citations. Over-dispersion was noted in all analyses, so quasi-likelihood was specified in all models. Influential points were tested with Cook’s distances. Papers featured in more than one newspaper were counted only once.

To check how much the news from the two countries were likely to be similar, each non-significant Chi-square test had its post hoc statistical power calculated. In each case, in the text we give the estimated probability to reject the null hypothesis for a small (w = 0.1), medium (w = 0.3) and large (w = 0.5) effect at the 0.05 level of significance. For the GLMs, where non-normality of data and over-dispersion would make power calculations non-trivial, statistical power was not calculated. This is not a limitation since, unlike the case of country comparison for news characteristics, in the GLMs we want to compare the strength of an effect within the same regression model, so statistical significance is only of secondary interest.

To the extent possible, data was collected using purposely written software (Java code), which minimized the risk of error and subjectivity. Analyses and graphs were produced using the open source software R 2.12 (R Core Team 2012) and G*Power 3.1 (Faul et al. 2009).

Results

Newsworthiness factors

Biomedical research represented the vast majority of papers published by PNAS (84.8 %, against 13 % for the physical sciences and 2.2 % for the social sciences), but studies pertaining to the social sciences were proportionally more likely to be featured in newspapers of any country (12.6 %, χ2 = 11.103, df = 2, P = 0.004, Fig. 1). Press releases were issued at similar frequencies in all three scientific domains (χ2 = 2.627, df = 2, P = 0.269, statistical power to detect a small, medium and large effect = 0.26, 0.99 and >0.99), and the probabilities of being reported in at least one journal were significantly higher for studies accompanied by a press release (χ2 = 35.478, df = 1, P < 0.001, Fig. 2).

Fig. 1
figure 1

Proportion of PNAS papers covered by newspapers in Italy, United Kingdom, both, or neither, classified by domain. Bar width is proportional to sample size

Full size image
Fig. 2
figure 2

Proportion of PNAS papers covered by newspapers in Italy, United Kingdom, both, or neither that were advertised by a press release. Bar width is proportional to sample size

Full size image

Italian newspapers were non-significantly different from the British in their coverage of different scientific domains (χ2 = 0.512, df = 2, P = 0.774, statistical power: 0.13, 0.73, 0.99), or of studies with/without press release (χ2 = 1.524, df = 1, P = 0.217, statistical power: 0.16, 0.82, >0.99), but were significantly different in the nationality of research represented in each: Italian newspapers reported more research with at least one author based in Italy, and British newspapers vice versa (χ2 = 24.721, df = 3, P < 0.001, Fig. 3).

Fig. 3
figure 3

Proportion of PNAS papers covered by newspapers in Italy, United Kingdom, both, or none that had at least one co-author based in Italy, UK, both, or neither. Bar width is proportional to sample size. Numbers for the overall sample were: 21 with at least one Italian author, 22 with at least one British author, 4 with authors from both countries, and 188 having no author from either country

Full size image

Citation patterns

Compared to papers not reported in any of the newspapers considered, those making the news were cited nearly 40 % more (b = 0.38 ± 0.12, t = 3.24, P < 0.001, controlling for number of authors—which is a known predictor of citations (Glanzel and Thijs 2004)—year of publication, domain, pure versus applied discipline, and whether the article was accompanied by a press release). The effect of newspapers varied significantly between countries: being featured in Italian newspapers alone had a much smaller, not statistically significant effect on citations (Table 1; Fig. 4). Having a press release was not linked to significantly higher citations (b = 0.15 ± 0.12, t = 1.27, P = 0.207, controlling for all factors above).

Table 1 Generalized linear model parameter estimate, standard error and statistical significance predicting the number of non-self citations received by scientific papers depending on the following study characteristics: newspaper coverage in Italy, United Kingdom or both (compared to papers that were not covered in any of the newspapers considered), scientific domain of the research (social and physical sciences compared to the reference category of biological sciences), whether the discipline was pure or applied, year of publication (2009 compared to 2008), and number of authors (log-transformed)
Full size table
Fig. 4
figure 4

Number of non-self citations (counted on 22 April 2012) to studies published in PNAS during 2008 and 2009, which had been featured in newspapers in Italy, United Kingdom, both countries, or neither (labelled “controls”). Sample sizes are indicated in parentheses. Box plots show median, interquartile range, and outliers. Mean uncorrected citation frequencies are, respectively: 20.2, 23.35, 38.0, and 35.4

Full size image

The proportion of citations from Italian authors was significantly higher for studies featured in newspapers, particularly Italian ones (Table 2; Fig. 5). The effect on the proportion of UK citations was smaller and only marginally significant (Table 3; Fig. 6). Being announced by a press release did not increase citations from the UK, and was linked to significantly fewer citations from Italy (Tables 2, 3). This latter effect, however, disappeared if the model controlled for discipline instead of domain (b = −0.22 ± 0.17, t = −1.287, P = 0.199).

Table 2 Generalized linear model parameter estimate, standard error and statistical significance predicting the proportion of non-self citations from papers with at least one author based in Italy, depending on the following study characteristics: newspaper coverage in Italy, United Kingdom or both (compared to papers that were not covered in any of the newspapers considered), scientific domain of the research (social and physical sciences compared to the reference category of biological sciences), whether the discipline was pure or applied, year of publication (2009 compared to 2008), and number of authors (log-transformed)
Full size table
Fig. 5
figure 5

Proportion of non-self citations by papers with at least one author based in Italy (out of the total non-self citations counted on 22 April 2012) received by studies published in PNAS during 2008 and 2009, which had been featured in newspapers in Italy, United Kingdom, both countries, or neither (labelled “controls”)

Full size image
Table 3 Generalized linear model parameter estimate, standard error and statistical significance predicting the proportion of non-self citations from papers with at least one author based in the United Kingdom, depending on the following study characteristics: newspaper coverage in Italy, United Kingdom or both (compared to papers that were not covered in any of the newspapers considered), scientific domain of the research (social and physical sciences compared to the reference category of biological sciences), whether the discipline was pure or applied, year of publication (2009 compared to 2008), and number of authors (log-transformed)
Full size table
Fig. 6
figure 6

Proportion of non-self citations by papers with at least one author based in the UK (out of the total non-self citations counted on 22 April 2012) received by studies published in PNAS during 2008 and 2009, which had been featured in newspapers in Italy, United Kingdom, both countries, or neither (labelled “controls”)

Full size image

Discussion

Papers published in PNAS were cited more often if they had been featured in newspapers, and this effect varied significantly between countries. Although Italian and British newspapers selected their news with very similar criteria—they preferred studies in the social sciences, those accompanied by a press release, and those with local authors, all of which confirms previous observations in these and others countries (van Rooyen 2003; Elmer et al. 2008; Bauer et al. 2006)—British newspapers exerted around four times the effect of Italian newspapers. These latter did not differ statistically from controls. Italian newspapers, however, had a significant local effect, by increasing the proportion of Italian citations.

These patterns are most parsimoniously explained in terms of a “publicity effect”, assuming that English-speaking newspapers have a wider, more international readership, whilst non-English speaking media are limited to scientists from one country. A complementary “earmark” effect cannot be ruled out, but only if we assume that newspapers in the UK select studies of higher intrinsic quality. Our findings do not support this latter assumption. Newspapers in the two countries appear remarkably similar in their criteria of newsworthiness, at least as far as choice of discipline goes. Moreover, studies that were advertised by a press release (which according to the earmark hypothesis would signal their intrinsic importance) had no significant citation advantage, once the media coverage was controlled for. In any case, the link between newspaper reporting and increased proportion of local citations—observed in both countries albeit at different frequencies—is unlikely to be explainable other than in terms of a local publicity effect.

The main limitation of this study is that it could not control all the media within Italy and the UK—let alone other countries. Since it would be virtually impossible to do so, this is an unavoidable limitation of a study of this kind. If radio, television and other newspapers in Italy and the UK had reported on the same PNAS-related news included in the study, then our estimate of the newspaper-related publicity effect would be inflated; if instead they featured some of our control studies, then the effect would be underestimated. We cannot know if either of these was the case, and it is plausible that these effects might to some extent cancel each other out, because some non-included media might have reported on the same stories and others had actively sought different ones.

The role of the Internet could also, ideally, be controlled independently of the other media. However, apart from the technical difficulties in doing so, this omission is probably a lesser limitation than the ones above. At least in recent years, all “traditional” news sources are present online, so the Internet is likely to act as an amplifier, with an impact proportional to that of the original outlet it represents. Indeed, the fact that scientists have online access to all news media was at the core of the prediction we tested. Finding that that news in different languages do indeed have different effects seems to confirm that the web exerts a neutral amplification role.

Ultimately, the critical issue is whether the central finding of this study—differences in citation patterns between Italy and UK—could be an artefact created by not controlling for all kinds of media. There is no reason to believe so. It seem very unlikely, in particular, that differences in the effects of newspapers versus radio, TV and other media would vary significantly in the two countries.

These results, therefore, add further strength to the hypothesis that research covered in the media enjoys a significant increase in citations, at least in part because of the publicity it receives. In line with previous studies, moreover, we found evidence that the agenda of included newspapers (as manifested in the choice of stories to report) differs from the scientific agenda. Three relevant messages follow from this. On a more superficial and practical level, researchers and institutions willing to maximize their citation impact should keep trying to make the news in as many mass-media as possible (Dunwoody et al. 2009), and particularly in English-speaking ones. On a slightly deeper level of analysis, these results may add elements of doubt over the reliability of citation data as a measure of scientific excellence—already a matter of great controversy (e.g. Panaretos and Malesios 2009). In principle, a researcher may boost its citation impact entirely thanks to a clever “spin” of her work, independent of any scientific and technical considerations. The extent to which this is true for any field of research and for papers not appearing in prominent journals like PNAS remains to be established. Thirdly, and perhaps most importantly, evidence of a strong publicity effect suggests that biases typical of the media might feedback into science. The long list of distortions known to affect the scientific literature currently includes only a “media attention bias”, which affects public perception (Song et al. 2010). A “publicity effect”, in which biases intrinsic to the media and to the general public affect a paper’s scientific impact, should be added to the list.

The general validity and impact of the scenarios described above remains to be established. Future research, in particular, could replicate and extend these results in the following directions:

  1. 1.

    Comparing English-speaking media with languages other than Italian, which would confirm this study’s conclusions. The publicity hypothesis would predict a stronger local effect for countries that are linguistically more isolated, such as China or Japan.

  2. 2.

    Assessing the publicity effect in other journals, which would help to understand how general and important this effect really is. In particular, future research could aim to assess how the media influence the impact of papers published in lower-ranking journals. We predict the publicity effect to be even stronger in these latter, and evidence of the contrary would lower the concerns expressed above about publicity-driven biases.

  3. 3.

    Comparing the publicity effect of different media. This study’s predictions were based on the assumptions that scientists read news in print and online, and that the confounding effect of other media could be ignored. Sufficiently large studies could quantify the separate effects that TV, radio, and even different kinds of magazines or websites have on citations. Analyses of citing authors, moreover, could identify how the various media influence different strata of the scientific population.

  4. 4.

    Measuring historical trends of the publicity effect. Before the advent of the Internet, scientists presumably could access only their local news sources, which leads to the prediction that, prior to one or two decades ago, the publicity effect had a stronger local component. Independent of this, it would also be interesting to assess whether the rapid changes occurring in how science is communicated have increased or decreased the publicity effect.

Given the compelling evidence for a publicity effect in science, and the growing importance it might assume in the electronic age, future research should carefully test these predictions.