Introduction

In journal peer review, manuscripts are usually reviewed by several peers before the journal staff editor makes a decision to accept or reject for publication. Some journals “rely on reviewer judgements to a great extent (e.g. always accepting submissions if two, or a majority of, reviewers recommend this)…However, in many journals, although editors base their decision on the reviewers’ comments, they do not necessarily follow the reviewers’ recommendations about acceptance or rejection” (Wager et al. 2006).

In this study we investigated the extent to which staff editors are also able to validly assess the quality of submissions without the support of reviewers. To answer this question, we first examined a comprehensive dataset on the manuscript reviewing process at the journal Angewandte Chemie International Edition (AC-IE) to see whether the staff editors made valid initial evaluations of a manuscript—after it had been submitted but before external peer review. To examine validity we used the citation counts for the manuscripts later published. In a second step, we examined the number of manuscripts for which there was a change in the editors’ evaluations between the initial internal evaluation and their final decision (after peer review).

Citation rates have been a controversial measure of quality (Bornmann and Daniel 2008d). Nevertheless, Lokker et al. (2008) succeeded in demonstrating for clinical articles that publications regarded as important by experts in the appropriate research field shortly after their appearance were cited much more frequently in subsequent years than publications that were less highly regarded. Furthermore, the results of a comprehensive citation content analysis conducted by Bornmann and Daniel (2008b) show that “an article with high citation counts had greater relevance for the citing author than an article with low citation counts” (p. 35).

Methods

Manuscript reviewing at AC-IE

AC-IE is one of the prime chemistry journals in the world, with a higher annual Journal Impact Factor (JIF, provided by Thomson Reuters, Philadelphia, PA) than the JIFs of comparable journals (at 10.879 in the 2008 Journal Citation Reports, Science Edition). AC-IE is a journal of the German Chemical Society (Gesellschaft Deutscher Chemiker (GDCh), Frankfurt am Main, Germany) and is published by Wiley-VCH (Weinheim, Germany).

A manuscript submitted to AC-IE usually undergoes internal and external review. First, staff editors at the journal evaluate the quality of the manuscript and determine whether the manuscript contributes to the development of an important area of research (internal review). The editors are fulltime members of the editorial staff. According to Gölitz, the Editor-in-Chief of AC-IE, “the fate of manuscripts and the selection of referees is determined by me, the two Deputy Editors, Neville Compton (Inorganic Chemistry) and Haymo Ross (Organic Chemistry), and four Senior Associate Editors, Frank Maass (Inorganic Chemistry), José Oliveira (Organic Chemistry), Guy Richardson (Inorganic Chemistry), and Diane Smith (Supramolecular Chemistry). All these editors have Doctorates in their specialist areas and in most cases postdoctoral research experience as well as many years experience as full-time editors” (Gölitz 2005, p. 5540). The editors rate the manuscript initially using the following 5-point scale from very positive to very negative: +, ++/−, +/−, +/−−, and −. After this initial evaluation by the editor, the submitted manuscript is sent to several independent reviewers (external review), who review it using an evaluation form and a comment sheet. The journal staff editors then make the decision to accept or reject a manuscript for publication on the basis of these reviews and their own evaluations. In the manuscript reviewing process at AC-IE, the staff editor thus evaluates one and the same manuscript twice—once before and once after peer review.

AC-IE introduced peer review in 1982, primarily in conjunction with one of the document types published in the journal, “Communications,” which are short reports on work in progress or recently concluded experimental or theoretical investigations. What the staff editors of AC-IE look for most of all is excellence in chemical research. Submissions that reviewers deem to be of high quality are selected for publication: For most submissions a manuscript is published only if two external reviewers rate the results of the study reported in the manuscript as (very) important and also recommend publication in the journal (Bornmann and Daniel 2009, 2010). The AC-IE staff editors are thus guided in their publication decisions to a great extent by the reviewers’ recommendations.

Database for the present study and conducting of citation analysis

For the investigation of manuscript selection at AC-IE we used information on all 1,899 manuscripts that were reviewed in the year 2000. Of the 1,899 manuscripts, 46% (n = 878) were accepted for publication in AC-IE, and 54% (n = 1,021) were rejected. A search in the literature databases Science Citation Index (SCI) (Thomson Reuters) and Chemical Abstracts (CA) (Chemical Abstracts Services, CAS, Columbus, OH) revealed that of the 1,021 rejected manuscripts, 959 (94%) were later published in 136 other (different) journals.

For accepted and rejected (but published elsewhere) manuscripts, we determined the number of citations for a fixed time window of 3 years after the publication year. “Fixed citation windows are a standard method in bibliometric analysis, in order to give equal time spans for citation to articles published in different years, or at different times in the same year” (Craig et al. 2007, p. 243). The citation analyses for the present study were conducted in the year 2007 based on CA. CA is a comprehensive database of publicly disclosed research in chemistry and related sciences (see http://www.cas.org/).

Of all 1,837 manuscripts published in the AC-IE (accepted manuscripts) or another journal (rejected manuscripts, see above), 1,817 could be included in the citation analysis. For 20 rejected manuscripts that were published elsewhere between 2004 and 2006, a citation window of 3 years (1 year after publication up to the end of 2006) was not available.

The statistical analyses of the citations in Bornmann and Daniel (2008a, c) show that the manuscripts accepted by AC-IE were on average cited statistically significant more frequently than manuscripts rejected by the journal and published elsewhere. This finding provides evidence for the predictive validity of the editorial publication decisions at AC-IE.

Statistical analysis

The assumption of a negative binominal distribution of citation counts (Allison 1980) makes it necessary to calculate a negative binomial regression model to test for differences between citation counts for manuscripts that are assessed differently by the AC-IE staff editor (before peer review) (Bornmann et al. 2008; StataCorp 2007). A posteriori contrasts with Bonferroni correction were tested in order to determine what pair-wise differences in mean citations counts are statistically significant.

In order to test whether there was a change in the evaluation category (from accept to reject) for a statistically significant number of the manuscripts between the initial internal evaluation by the AC-IE editor (before peer review) and the editorial decision (after peer review), the McNemar (1947) chi-square test was calculated. The McNemar test is used when two dependent samples (here, manuscripts assessed before and after peer review) are compared regarding a dichotomous variable (here, accept and reject).

Results

Table 1 shows the distribution of the initial internal evaluations by the AC-IE staff editors. The editors rated 13% of the manuscripts positively, choosing one of the two positive categories +(1%) or ++/− (12%); they assigned 8% of the manuscripts to the negative category +/−−. Only 8 manuscripts were assigned to the very negative category (−). Approximately two-thirds of the manuscripts (67%) were not rated either positively or negatively by the staff editor before peer review; these manuscripts were assigned the rating +/−. The high number of manuscripts in +/− category indicates that in many cases, the editors are uncertain as to the publication worthiness of a manuscript and make the decision to accept or reject for publication based on the external reviewers’ recommendations (see Methods section).

Table 1 Initial internal evaluations by the AC-IE staff editors
Full size table

As Fig. 1 shows, the initial internal evaluations by the AC-IE staff editors correspond with the later citation counts for all submissions accepted by AC-IE or rejected by AC-IE but published elsewhere (n = 1,800): Manuscripts that the editors rated the highest before peer review (+, n = 25), were cited 22 times on average (median). Manuscripts that the staff editors rated ++/− (n = 222) were cited 15 times on average (median). The other average citation rates (median) for the differently rated manuscripts are: +/− (n = 1,201) = 11, +/−− (n = 133) = 7, and − (n = 7) = 3. The result of the negative binomial regression models shows that whereas the mean citation rates for the manuscripts rated + and ++/− do not differ statistically significantly (P = .22), the differences between the mean citation rates of all of the other manuscript groups are statistically significant (P < .05). These findings indicate that the initial evaluations of the manuscripts by the editors can be called valid. The analyses concerning the validity of the editors’ final decisions (after peer review) showed similar results: accepted manuscripts were cited more frequently on average (median = 14) than rejected (but published elsewhere) manuscripts (median = 9) (Bornmann and Daniel 2008a, c; Bornmann et al., submitted).

Fig. 1
figure 1

Box plots for citations of manuscripts that received different initial internal evaluations by an AC-IE staff editor (n = 1,800: n = 25 (+), n = 222 (++/−), n = 1,201 (+/−), n = 133 (+/−−), n = 7 (−), and n = 212 with no assessment). The horizontal line in the middle of each box indicates the median, and the top and bottom borders of the box mark the 75th and 25th percentiles, respectively. The whiskers above and below the box mark the upper and lower adjacent values. The points above the whiskers are defined as outliers

Full size image

Table 2 shows the connection between initial internal evaluation and editor’s final decision. Of the manuscripts that the editor evaluated positively (+ or ++/−, n = 253) before peer review, 199 (79%) were later accepted for publication, and 54 (21%) were rejected. Of the manuscripts that the staff editor evaluated negatively (+/−− or −, n = 155) before peer review, 134 (86%) were later rejected, and 21 (14%) were accepted. Thus, for the majority of manuscripts that the editor rated positively or negatively upon the initial internal evaluation, the initial internal evaluations and final decisions correspond. (Only those manuscripts were included in the analysis that the staff editor had evaluated positively or negatively before peer review—that is, manuscripts rated in the category +/− (n = 1,256) were not included.)

Table 2 AC-IE staff editor’s final decision (after peer review) by the editor’s initial internal evaluation (before peer review)
Full size table

To check whether a significant number of manuscripts were assigned a different rating between the initial internal evaluation and the final decision, the McNemar chi-square test was computed. The data in Table 3 are based on the McNemar test. So that uniform categories at both time points would be available for the analysis, the evaluation categories + and ++/− were combined in a category called ‘accept,’ and the evaluation categories +/−− and − were combined in a category called ‘reject’. As the result of the McNemar chi-square test in Table 3 shows, that the marginal proportions are statistically significant from each other. Thus, there is a statistically significant proportional difference in “Rejection” between the initial internal evaluation and the final decision.

Table 3 AC-IE staff editor’s final decision (after peer review) by the editor’s initial internal evaluation (before peer review)
Full size table

As a final step, for the manuscripts that were assigned a different evaluation by the staff editor after peer review, we examined which of the two evaluations (initial evaluation of the manuscript before peer review and final decision after peer review) of the individual submission was the more valid evaluation, using ex-post determined citation counts for the manuscripts as a quality indicator. Following van Raan (2004), a positive evaluation by the AC-IE editor was called valid if the manuscript was cited a far above-average number of times after publication (and vice versa). (As a yardstick we did not choose above average but instead far above average, because the AC-IE seeks to publish excellence in chemical research.) For this we compared the citation count of an individual manuscript with the average citation rate for papers published in the same chemical sub-field (CA section) (Bornmann and Daniel 2008c). If the citation count was higher than 1.5 times of the mean value, we called the impact of the manuscript far above average.

Of the 54 manuscripts that the staff editor evaluated positively for the initial internal evaluation but rejected in the final decision (see Table 3), 48% had far above-average citation counts (eight manuscripts could not be included in the analysis, as no citations were available; see Methods section). Of the 21 manuscripts (see Table 3) that the editor initially evaluated negatively but were later accepted for publication, 75% had far above-average citation counts (in this analysis, one manuscript could not be included due to missing citations). Thus, for the majority of these manuscripts, the staff editors’ assessments are more valid after peer review (that is, in the final decision).

Discussion

The results of this study indicate that the initial internal evaluations are valid. Further, it appears that external review is indispensable for the decision on the publication worthiness of manuscripts: (1) For the majority of submitted manuscripts, staff editors are uncertain about publication worthiness; (2) there is a statistically significant proportional difference in “Rejection” between the editors' initial evaluation and the final editorial decision (after peer review); (3) three-quarters of the manuscripts that were rated negatively at the initial internal evaluation but accepted for publication after the peer review had far above-average citation counts.

All in all, peer review increases the validity of the publication decisions by the journal staff editors.