Introduction

The JIF is computed by Thomson-Reuters for each year (Y) according to the following equation (Glänzel and Moed 2002):

$${\text{JIF}}(Y) = \frac{{{\text{Citations}}\_{\text{in}}\_Y\_{\text{to}}\_\operatorname{documents} \_{\text{published}}\_{\text{in}}\_Y1\_{\text{and}}\_Y2}}{{{\text{Citable}}\_{\text{items}}\_{\text{published}}\_{\text{in}}\_Y1\_{\text{and}}\_Y2}}$$

In the above equation Y1 and Y2 are the 2 years before Y. To calculate the numerator of the JIF, Thomson-Reuters counts citations to all types of documents, whereas the denominator includes only citable documents (Glänzel and Moed 2002; Golubic et al. 2008). The JIF is one of the most frequently used scientometric indicators and it is often used as a basis for decision-making for research grants, hiring, and even salaries. Thus analyses of this strategic research tool can yield much useful information about science dynamics.

However, scientists have mixed feelings on the use of JIF. For example, according the results of a questionnaire completed by about 1,700 researchers from 86 different countries, “the opinion is slightly above the median which could be understood as neither positive nor negative” (Buela-Casal et al. 2012, p. 281). Many scientists and journal editors have published letters or editorial comments with positive and negative opinions on JIFs and their use.

In 2012, the journal Scientometrics devoted an issue to problems with and alternatives to this important indicator (Scientometrics, vol. 92, issue 2, August: Special Discussion Issue on Journal Impact Factors). The lead paper in this issue addressed some interesting points (Vanclay 2012). In a comment on this paper, one of us suggested that an interesting research project would be to evaluate the effect of adding or removing citations in the equation used to compute the JIF (Campanario 2012).

A single citation is the minimum “amount” of impact that we can use to compute the JIF (Campanario, in revision). There is a similar situation in Physics. The minimum amount of electric charge that can be added to an atom is the charge associated to one free electron; there is no possible smaller amount of electric charge. In fact, electron charge is used as a unit in the Atomic Units system instead of the Coulomb, the unit used in the International System. If we were constructing atoms from electrons, neutrons and protons, we would be able to add only one electron (or neutron or proton) at a time, so the amount of electric charge added each time would be the charge of the electron (the charge of protons is equal but positive). Similarly, in scientometric studies the minimum amount of “impact” we can study is the impact associated with a single citation. Thus, while the number of citations is a discrete magnitude, the JIF is not.

Using a similar conceptual approach, Vinkler recently studied the possibility of increasing the Hirsch-index, π-index or CDS-index by increasing the number of publications or citations only by unity (Vinkler 2013). In their theoretical study the results obtained with the “successively built-up indicator” model showed that as the number of citations increases, the indices may increase substantially.

Althouse et al. (2009) studied the evolution of JIFs across fields and over time. They find that the average number of citations in reference lists had increased gradually. Thus, there is a kind of inflation of JIFs over time. They discovered also that field-specific variation in the fraction of citations to literature indexed by Thomson Scientific’s Journal Citation Reports is the single greatest contributor to differences in JIFs. Togia and Tsigilis (2006) studied the stability of two JCR groups: “Education and Educational Research” and “Education, Special” from 2000 to 2005. They discovered that journals belonging to the “Education and Educational Research” category had relative stable JIFs. However this was not true for the other category. Tsigilis et al. (2010) examined the stability of JIF of journals in the group “Sport sciences”. They discovered a increase in the median impact factor from 2000 to 2006 following a linear trend or even a quadratic trend.

Changes from the usual JIF to JIFs computed by adding 1, 2, or more citations could result in changes in the internal ranking of journals within their JCR groups. For example, the 2008 JIF of the journal Adv Insect Physiol changed from 5.000 to 5.250 (5 %) when only one more citation was added to the number of citations used by Thomson-Reuters. This sharp change in JIF did not result in any change in rank because the JIF of the journal that precedes it in the JCR group is 11.918 (Annu Rev Entomol). Consider now the journal Order (JCR group Mathematics). The JIF of this journal would change from 0.333 to 0.356 by adding only one more citation to the number used by Thomson-Reuters to compute the JIF. This change would move the journal from position 173 in the JCR group ranking to position 164 (9 places). Had this journal obtained 5 additional citations that contribute to JIF, its rank position would have changed from 173 to 143 (30 places).

This point is relevant when studying the rank of journals in different JCR groups. Thomson-Reuters classifies journals in JCR groups according to their scope; consequently many journals are listed in two or more JCR groups. Journals are ranked within their JCR group according their JIF. In some countries, journals rank (rather than absolute JIF) is used to evaluate the scientific merit of papers published in a given journal. Thus, there is fierce competition among journals to occupy the leading positions within their JCR groups. It is accordingly not unusual for journals to advertise their JIF and JCR rank on their website, nor is it surprising that researchers prefer to submit their papers to journals that occupy the top positions in their JCR group ranking. We should keep in mind, however, that the typical distribution of JIFs within a given JCR group is highly skewed. As a consequence, small increases in this indicator for a given journal can result in significant changes in a given journal’s rank position. This may happen particularly in journals that occupy positions in the middle or near the bottom of the ranking.

In light of these considerations, we suggest here a new approach to study the ranking of journals in JCR categories. The objectives of this study are:

  1. 1.

    to empirically evaluate the effect of additional citations on the computation of the JIF (i.e., changes in the JIF) in a large set of journals, and

  2. 2.

    to ascertain the influence of increases in citations on the rank order of journals according their JIFs within JCR groups.

Method

All journals included in the Science edition of the 2008 JCR with a JIF greater than zero were used in this study (n = 6567). The journals were distributed among 173 JCR groups. Note that many journals were included in two or more JCR groups. The following variables were used:

  1. (a)

    Number of citations and articles used by Thomson-Reuters to compute the JIF

  2. (b)

    JIF0: JIF as computed by Thomson-Reuters

  3. (c)

    JIFn: JIF computed adding n citations to the number of citations used by Thomson-Reuters to compute the JIF. These new JIFs are designated JIFn, where n = 1, 2, 3, etc. This yields variables JIF1, JIF2, JIF3, and so on. All JIFn were rounded to 3 decimal places.

  4. (d)

    CJIFn: Change in the JIF. This variable is computed by subtracting JIF0 from JIFn (for example CJIF3 = JIF3 − JIF0).

  5. (e)

    Rn: Rank of a given journal in its JCR group (according to the classification published by Thomson-Reuters) when JIFn is used (keeping all other values the same)

  6. (f)

    CRn: Change in the rank position that a given journal would experience when its JIF is replaced with JIFn, keeping everything else equal. Obviously, regardless of whether JIFn and JIF are computed to the same number of decimal places, CRn is always zero or positive.

In this study, rank is defined by the position of a given journal in its JCR group when all journals are ranked from highest to lowest JIF. Thomson-Reuters ranks journals with the same JIF at the same position. However, the rank of some journals depends of the number of other journals with the same JIF. For example, consider the journals BMC Struct Biol; Clin Biomech; Comment Inorg Chem; Gold Bull; Int J Stroke; J Database Manage; J Microbiol Meth; J Muscle Res Cell M; Manual Ther; Oligonucleotides and Rev Chem Eng. All these journals share the same 2008 JIF of 2.000. Their rank in the whole JCR database is 2,177 in all cases. The journal with the next highest rank is Plant Soil (2008 JIF = 1.998) and its JCR rank is 2188. Let us imagine that the JIF of Plant Soil increases from 1.998 to 2.001. Its rank position would then change from 2,188 to 2,177 (11 places). However, this journal would actually rise in rank order by only one position. To avoid this spurious increase, we used a more natural scheme in which rank order is always successive. In our example, Plant Soil rank associated with a JIF = 1.998 would thus be 2,178 instead of 2,188.

Box 1 illustrates the approach used to compute the effect on rank order of adding citations in the JIF formula. This box shows a set of journals in a given group ranked by JIF0. We computed the variable JIF1 by adding 1 citation to each journal, and compared the resulting rank order of each journal against the initial rank order (keeping everything else equal). In our example, all journals would maintain their original rank order except IEEE T Aero Elec. This journal’s JIF would change from JIF0 = 1.024 to JIF1 = 1.028. Comparing JIF1 (1.028) to the old set of JIF0 numbers shows that this journal gained one position, because its JIF1 is higher than the JIF0 for the journal AIAA J, which was ranked third according to JIF0. In summary, if IEEE T Aero Elec received only 1 more citation that contributed to JIF0, it would move up in the rank order from fourth to third position (keeping everything else equal, i.e. keeping the JIF0s of other journals unchanged). The rank increase for IEEE T Aero Elec can be computed as Rank0 − Rank1 = 1 (last column in Box 1).

Box 1 Example of computation of changes in rank within a given JCR group when a single citation was added to the number of citations that contributed to the JIF. See text for a detailed explanation
Full size table

It is important to note that we compared the new JIF1 (or JIF2, JIF3 or JIFn) for a given journal with the old JIF0 of all journals in the JCR group. We were interested in what would happen when a given journal received one or more additional citations while all other inputs remained unchanged. We did not compare all the new JIF1s (or JIF2s, JIF3s, etc.) for all journals. Note that when the new JIFn for a given journal was equal to the old JIF0 of another journal, the rank corresponding to the JIFn was equal to the rank corresponding to the old JIF0 for the other journals. This decision minimized the possibility of spurious changes in rank order.

As noted above, many journals were included in two or more JCR groups. When this was the case, we computed the JCR group statistics for all JCR groups.

Simple descriptive statistics were used to report the distribution of changes in JIF for the whole set of journals and changes in rank positions within JCR groups.

Results and Discussion

Figure 1 shows the distribution of about 98 % of JCR journals according CJIF1, CJIF5 and CJIF10 respectively, this is, the changes in JIF from JIF0 to JIF1, JIF5 and JIF10. As can be seen, the distributions are very similar but the scales are different. The distributions are highly skewed. As expected, journals benefit when citations are added, but this gains in JIFn are distributed in a similar way when n changes. Thus, it appears that no particular groups of journals experienced a notable increase in JIF when more citations were added.

Fig. 1 
figure 1

Distribution of journals according to CJIF1, CJIF5 and CJIF10. To avoid the effect of extreme values, the figures include 98.7 % of journals (CJIF1), 98.8 % of journals (CJIF2) and 97.8 % (CJIF10)

Full size image

As noted above, a given journal can be listed in more than one JCR group. When the effect of additional citations on rank order are calculated, we can expect different effects for the same journal in different groups. We accordingly studied the effect of adding n citations on the rank order of journals in different JCR groups by computing statistics for JCR groups separately. Note that in the tables, the total number of journals appears greater because many journals are listed in more than one JCR group.

Table 1 shows the percentage of journals in each JCR group whose rank order changed with JIFn (n = 1, 2, … 10). In most JCR groups the percentage of journals that changed their rank order when we computed different JIFn did not change in a linear fashion with n (the number of citations we added). In 31 JCR groups (about 18 % of the whole set), the addition of just 1 more citation to the number of citations used to compute the JIF changed the rank order for more than 25 % of journals in these JCR groups. The most extreme cases were the JCR groups History & Philosophy of Science (57.5 %), Mathematics, Applied (62.9 %) and Mathematics (74.3 %). There are 24 JCR groups (about 14 % in the whole set) in which no journal changed its rank order when we computed JIF1. These JCR groups contain lower numbers of journals (typically from 20 to 30). However, other JCR groups with similar numbers of journals experienced changes affecting more than a 10 % of journals. Thus, the size of the JCR group is not a determining factor for the absence of changes in rankings.

Table 1 Percentage of journals that changed their rank in each JCR group when 1, 2, 3 to 10 citations were added to the number of citations that contribute to the JIF
Full size table

Figure 2 shows the association between the number of journals in each JCR group and the percentage of journals that changed their rank order when 1, 5 or 10 citations were added to compute the JIF. There was an obvious effect of the number of citations: more citations led to more changes.

Fig. 2
figure 2

Relationship between the number of journals in each JCR group and the percentage of journals that changed their rank when 1, 5 or 10 citations were added to compute the JIF

Full size image

A more important outcome than the percentage of journals that change their rank order in each JCR group is the magnitude of the changes. This result is important because the relative position within a JCR group is the main indicator usually used to compare journal ranks. As explained above, the variable CRn measured the change in rank for each journal with different values of n. We have studied the percentage of journals whose CR changed when JIF1, JIF5 and JIF10 were computed (Tables a–c, supplementary material). For example, in the case of JIF1 (i.e., when one citation was added; Table a-supplementary material), 86.7 % of journals in the Agriculture, Dairy & Animal Science group showed no change in rank position (CR0), whereas 11.1 % improved their rank by one position (CR1) and 2.2 % moved up in the rank by two positions (CR2).

In the case of n = 1 (i.e. when we added 1 citation to compute the JIF; Table a-supplementary material), there were only 18 JCR groups (about 10 %) in which 10 % or more of journals increased their rank by two or more positions. The largest changes involved the journals Group Geom Dynam and Math Intell (JCR group Mathematics). When just 1 citation was added, the former journal increased its JIF from 0.519 (position 119) to 0.556 (position 106). The JIF of the latter journal increased from 0.344 (position 169) to 0.375 (position 156). In the case of n = 5 (Table b-supplementary material), there were only 15 JCR groups (about 8.7 %) in which more than 50 % of journals increased their rank by two or more positions. In the case of n = 10 (Table c-supplementary material), there were 45 JCR groups (about 26 %) in which more of 50 % of journals increased their rank by two or more positions.

Table 2 shows the absolute number and percentage of journals according to the value of CRn (0, 1, 2, … 15 or more changes in rank position) when n = 1, 2 … 10. As expected, the percentage of journals that did not experience changes in rank order decreased as n increased (Fig. 3). The percentage of journals that moved up by one position in rank order increased initially and then decreased slowly. The percentage of journals that moved up two or more positions increased as n increased. However, the increase in the percentage of journals tended asymptotically to stable values and the rate of increase decelerated (see Fig. 4). Thus, the decrease with n in the percentage of journals with CRn = 0 as n increased resulted in an increase in all percentages for CRn greater than CR1.

Table 2 Absolute number and percentage of journals according the value of CRn (0, 1, 2, to 15 or more changes in rank) when n = 1, 2 to 10
Full size table
Fig. 3
figure 3

% Journals that did not change their rank (CR0) when n = 1, 2 to 10 citations were added to the number used to compute the JIF

Full size image
Fig. 4
figure 4

% Journals that changed their rank by 1, 2, or more positions (CR1, CR2, …) when n = 1, 2, to 10 citations were added to the number used to compute the JIF

Full size image

A different approach to study the effect on rank order of adding citations to calculate JIFn is to compute the minimum number of citations that a given journal needs to change its position. For example, consider the journal J Comput Acoust in the Acoustics group (JIF0 = 0.585). Two citations need to be added to obtain a JIF2 = 0.623, which is higher than the JIF0 of Acoust Phys + (JIF0 = 0.622). If we had added only 1 more citation, the JIF1 for J Comput Acoust would have been JIF1 = 0.604 (note that everything else equal remained equal, i.e., we compared the new JIFn for a given journal with the old JIF0 for all other journals). Thus, the minimum number of citations that J Comput Acoust would need to improve its rank position is 2. Similar calculations can be done for each journal in every JCR group. In 53 JCR groups (about 31 %), 50 % or more of the journals changed their rank position when 1–5 citations were added to the number used to compute the JIF. In contrast, in 63 JCR groups (about 36 %), 50 % or more of the journals did not change their rank position even after 10 citations were added (Table d-supplementary material). Here we include all journals that occupied the top position in each JCR group. These journals, by definition, cannot change their rank position.

One of the referees suggested that, given that the JIF distribution is very skewed, the low and middle ranges are much more sensible to changes then the top ones to the additional citations. To address this comment, we have examined the effect of adding different number of citations (1, 5, and 10) when considering different quartiles (Q1, Q2, Q3 and Q4). To avoid spurious results, we only considered groups having 50 or more journals (n = 84). We divided these journals in quartiles and we computed the mean of CR1, CR5 and CR10 for each group and quartile. The results are sometimes mixed, but, in general, journals in Q2, Q3 and Q4 showed higher values of CR, as could be expected (Table e-supplementary material).

Conclusions

In a previous paper one of us noted that an interesting research project would be to evaluate the effect of adding or removing citations and/or articles in the equation used to compute the JIF (Campanario 2012). The results of our research suggest that an increase of only 1 citation leads to small changes in the JIF in most journals, and therefore the effect on journal rankings within JCR groups is, in general, not very large. When we studied the changes in journal rankings within JCR groups, it was clear that the effect of these changes was not very large.

When we added more citations, we found that the distribution of journals according to changes in their JIF were similar, but with different scales. Thus, there appeared to be no particular type of journals whose JIF increased when citations were added. Moreover, the percentage of journals whose rank position changed when we computed JIFs with additional citations did not change linearly with n, the number of additional citations added.

The effect of additional citations on the internal structure of rank orders within JCR groups obviously increased with n, the number of citations added. However, in general, the rank orders tended to be relatively stable after small increases in citations. In about one third of the JCR groups, about half of the journals changed their rank when to them 1–5 citations were added.

Journals often compete for citations in order to increase their JIF. One possible mechanism of manipulation is related with journal self-citations (Monastersky 2005; Yu et al. 2010). The results of our research suggest that the number of citations necessary to alter the rank order of journals within JCR groups is small for many journals, and that for most of them, more than 5 additional citations would be needed for the journal’s rank order to change. When considering quartiles, more changes are observed, in general, in lower quartiles, given that JIF are very skewed. The journals in lower quartiles have more similar JIFs and small changes in citations can promote them to higher ranks that similar changes in citations in journals in the first quartile.

It should be keep in mind, however, that we studied changes in only one journal while keeping all other factors equal. When even small numbers of citations are added, the resulting changes in rank may interact to produce very different rank orders.