Introduction

On June 18th, 2016, Thomson Reuters issued the latest JCR report 2015 in which two novel bibliometric indicators were launched. One was “average journal impact factor (JIF) percentile”, the other one was “normalized eigenfactor”. Average JIF percentile converted JIF ranking into percentile to provide more intuitive transverse comparison between journals (Thomson Reuters 2016). Analysis of average JIF percentile would help understand characteristics of the indicator and provide guidance to journal evaluation, which was of significant value.

It had been a long tradition since percentile was introduced into bibliometric indicators. Early in 1930s, Bradford (1985) introduced the well-known Bradford’s law, in which scientific journals were sorted by number of publications in descending order, and then divided into core section, relevant section and irrelevant section, with each section having the same number of publications, interestingly the ratio between number of journals in each section was found to be 1:n:n2. Many other scholars continued to conduct various studies on the distribution law of journal level bibliometric indicators. Hintze and Nelson (1998) proposed that using percentile to normalize number of citations can better evaluate the comprehensive impact of journal from different disciplines, document type and publication time. Bornmann and Mutz (2011) divided percentile into 6 grades: top 1 %, 99–95 %, 95–90 %, 90–75 %, 75–50 %, 50–0 %, which was called percentage rank 6 (PR6), and intended to use it as indicator of evaluating journal. Later, he developed the idea and, together with Leydesdorff, proposed integrated impact indicators (I3) indicator (Leydesdorff and Bornmann 2011) which used percentile function beyond six grades. Leydesdorff (2012) then investigated the relationship between “integrated impact indicators” (I3) and the first 10 and 25 % of citation frequency to see whether it can surrogate impact factor. Campanario (2014) analyzed publications of Spanish university and introduced h core indicators based on citation frequency of articles, and proposed evaluation methods using 10, 20 or 30 % percentile.

The basic reason of adapting percentile rank of a certain indicator to evaluate journals was due to the data distribution features of bibliometric indicators as well as the aim of better comparison between journals. Vinkler (2008) found that citation distribution was right skewed and not normal distribution. Adler et al. (2009) thought, according to power law, citation distribution was usually right skewed, and as a result arithmetic mean only represented the citation value of highly cited articles. Percentiles have several advantages in general (Bornmann et al. 2013), they are not strongly influenced by extreme values, do not rely on the choice of a specific probability density function, can be calculated even if the data are skewed and once calculated, they also permit the classification of publications into meaningful citation impact classes (Rousseau 2012; Bornmann 2013). But percentiles have deficiencies such as overvaluing or devaluing an object because of percentile ranking schemes, ignoring precise citation variation among those ranked next to each other, and inconsistency caused by additional papers or citations (Zhou and Zhong 2012) The average JIF percentile is an indicator that intends to combine the advantages of percentiles in many respects and has kept the precision of traditional JIF.

Because average JIF percentile was a completely novel indicator, what are the characteristics of the indicator like? Is it different from impact factor? And what’s the relationship between the indicator and other established bibliometric indicators? These questions remained to be further explored. The paper started by analyzing the meaning of average JIF percentile and adopted statistical methods to compare its difference with impact factor. Factor analysis was used to further analyze the relationship between average JIF percentile and other bibliometric indicators.

Data and processing

The research used data from JCR2015. In order to improve the robustness, economics discipline with comparatively large number of journals was used to do the study. Mainly 12 indicators were published in JCR2015: average journal impact factor percentile, total cites, journal impact factor, impact factor without journal self cites, 5-year impact factor, immediacy index, eigenfactor score, article influence score, normalized eigenfactor, cited half-life, citing half-life and citable items. Use the uppercasing in the name of the indicators as they appear in the JCR.

There were 333 economics journals in JCR2015. 51 journals were eliminated due to missing data so 282 journals remained. Table 1 shows descriptive statistics of each indicator.

Table 1 Descriptive statistics of indicators
Full size table

Comparative study of average journal impact factor percentile and impact factor

According to Thomson Reuters, the definition of average JIF percentile (JIFP) is as follows.

$${\text{JIFP}} = \frac{(N - R + 0.5)}{N}$$
(1)

In formula (1), JIFP was average JIF percentile, N was the number of journals in the discipline, R was the rank position of impact factor sorted in descending order. 0.5 was added to get rid of value 0 when calculating average JIF percentile. Obviously, transformation from impact factor to average JIF percentile was non-parametric transformation. Average JIF percentile can reflect the ranking of journals, but cannot be used to reflect the relative gap between journals. For example, journal A has impact factor 5.0 and average JIF percentile 80; journal B has impact factor 4.0 and average JIF percentile 40. It can be claimed that impact of journal B is 0.8(4.0/5.0) of that of journal A, but it’s absolutely wrong to claim according to the average JIF percentile that impact of journal B is 0.5(40/80) of that of journal A. Average JIF percentile originated from impact factor. They have exactly the same meaning but of different representation form. Moreover, journal ranking based on these two indicators were totally the same. Scatter plot of them were shown in Fig. 1.

Fig. 1
figure 1

Scatter plot

Full size image

Because most journals have low impact factor value, the ranking transformation rendered journals with relatively low impact factor value obtain higher average JIF percentile. In general, average JIF percentile had higher value than impact factor.

After normalization of average JIF percentile and impact factor, namely divided respectively by the maximum value and multiplied by 100, the result is shown in Fig. 2. When impact factor was of high value, it had good degree of differentiation, while average JIF percentile had lower distinctive capability but was generally in uniform distribution.

Fig. 2
figure 2

Comparison of data distribution after normalization

Full size image

To further compare the statistical characteristics of impact factor and average JIF percentile, we used minimum to maximum ratio, median to maximum ratio, coefficient of variation, skewness, kurtosis, and Jargue–Bera test. Results are shown in Table 2.

Table 2 Statistical characteristics of impact factor and average JIF percentile
Full size table

Observing the statistical characteristics, average JIF percentile had higher minimum to maximum ratio, median to maximum ratio than impact factor while had lower coefficient of variation, which indicated that average JIF percentile, after data transformation, had both improved relative value of impact factor and decreased data fluctuation. Observing the data distribution, although both indicators didn’t obey normal distribution, average JIF percentile was closer to normal distribution and can serve better to compare journals.

According to Table 2, average JIF percentile, compared with IF, had higher minimum maximum ratio and median maximum ratio, but lower coefficient of variation, which indicated that after data transformation, the average JIF percentile had improved relative value of impact factor and decreased data fluctuation. Although neither indicators have normal distribution, average JIF percentile can serve better in comparing journals because it’s more like normal distribution.

Relationship between average JIF percentile and other indicators

Multivariate regression analysis was commonly used to analyze the relationship between average JIF percentile and other bibliometric indicators. Because bibliometric indicators had high mutual correlation, traditional regression method inevitably had multicollinearity problem that would make the variation of regression coefficient increase and t test value decrease etc. To solve this problem, the paper first used factor analysis to extract key factors from bibliometric indicators other than average JIF percentile. Since key factors had no correlation between each other, the multicollinearity problem was completely solved.

Prior to factor analysis, KMO and Bartlett test must be conducted to decide whether factor analysis was applicable. Results showed that KMO test value was 0.811, which was higher than empirical value 0.5. Bartlett test value was 7769.165 and concomitant probability was 0.0000. The hypothesis of not using factor analysis was rejected, indicating that factor analysis was applicable.

Factor analysis showed that 3 factors had eigenvalue over 1. Variance contribution of the first factor was 53.94 %, variance contribution of the second factor was 15.68 % and variance contribution of the third factor was 10.89 %. The accumulated variance contribution was 83.59 %, which meant these three factors explained 80.52 % of the total information and meanwhile mutually were not correlated.

Rotated matrix is illustrated in Table 3. The first factor (F1) reflected total cites, impact factor, impact factor without journal self cites, 5-year impact factor, immediacy index, eigenfactor value, paper influence score, normalized eigenfactor, which mainly explained the journal impact; The second factor (F2) mainly reflected citable items; The third factor (F3) mainly reflected cited half-life and citing half-life. The regression results were as follows:

Table 3 Rotated component matrix
Full size table
$$\log ({\text{JIFP}}) = \mathop {4.333}\limits_{{(11.192^{***} )}} + \mathop {1.086\log (F1)}\limits_{{(15.378^{***} ){\kern 1pt} }} + \mathop {0.381\log (F2)}\limits_{{(3.711^{***} )}} - \mathop {0.708\log (F3)}\limits_{( - 4.226)} \quad R^{2} = 0.476$$
(2)

Generally speaking, as transformation of impact factor, average JIF percentile had no high goodness of fit with other bibliometric indicators. R 2 value was 0.476. Impact got the highest regression coefficient. Elastic coefficient was 1.086 which meant that with each 1 % increase of journal impact, the average JIF percentile will increase 1.086 %; it’s followed by regression coefficient of half-life which was −0.708, indicating that with every 1 % decrease of journal half-life, the average JIF percentile would increase 0.708 %. It’s normal that regression coefficient was negative because the third factor was immediacy indicator. Longer half-life meant the journal less timely and impact decreasing. Citable items had the lowest regression coefficient with the elastic coefficient 0.381, which meant that on every 1 % increase of citable items, the average JIF percentile would increase by 0.381 %.

Because the average JIF percentile emphasized more on ranking position, in order to analyze the relationship between average JIF percentile with other bibliometric indicators at different quantile value, quantile regression was used to do the study. In total there were 9 quantiles as shown in Table 4.

Table 4 Quantile regression results
Full size table

To more clearly analyze the relationship between the average JIF percentiles at different quantile value with other bibliometric indicators, the results were mapped in Fig. 3. It was found that regression coefficient of impact was gradually decreasing, i.e. when average JIF percentile was low, regression coefficient of impact was even higher.

Fig. 3
figure 3

Quantile regression results

Full size image

The relationship between regression coefficient of citable items and average JIF percentile was in horizontal curve. But when average JIF percentile was very high or very low, citable items had nothing to do with average JIF percentile. This was in accordance with reality, because for journals with very high average JIF percentile, they relied on quality of articles to accumulate impact.

When average JIF percentile was at the lowest 10 quantiles, it had the highest regression coefficient with journal half-life. In other situations, it was relatively stable.

Conclusions and discussions

Firstly, average JIF percentile had changed the statistical characteristics of impact factor. Compared with impact factor, average JIF percentile improved the relative value of impact factor, had smaller coefficient of variation and the data distribution was closer to normal distribution, which was good for horizontal comparison among journals. But because average JIF percentile was kind of transformation based on impact factor ranking, it’s in fact non-parametric transformation, the average JIF percentile can only reflect ranking order while being unable to measure the relative difference of impact between journals.

Secondly, average JIF percentile had medium correlation with other bibliometric indicators. Average JIF percentile had the highest regression coefficient with journal impact indicators that included total cites, impact factor, impact factor without journal self cites, 5-year impact factor, immediacy index, eigenfactor value, article influence score, normalized eigenfactor. It’s followed by journal timeliness indicator that included cited half-life and citing half-life. The lowest regression coefficient was with citable items.

Thirdly, quantile regression analysis on journal impact, timeliness, citable items and average JIF percentile showed that the lower average JIF percentile, the higher that elastic coefficient of journal impact; When the average JIF percentile was very high or very low, citable items had nothing to do with average JIF percentile, because for journals with very high average JIF percentile, they relied mainly on quality of articles to achieve impact; When average JIF percentile was very low, the elastic coefficient of journal timeliness was even higher.

Fourthly, average JIF percentile was not a proper indicator for journal multi-attribute evaluation. Because average JIF percentile was in fact a ranking indicator of non-parametric nature, it’s improper for evaluating indicator system but could be used for direct comparison among the standing of journals being more intuitive.

Fifthly, average JIF percentile had the same advantages and disadvantages of impact factor. Thus, it had the same limitation in application fields as the impact factor.