Introduction

Several metrics based on citation counts have been developed to evaluate the impact of scholarly journals (van Raan Anton 2004), one of which, the impact factor published by Thomson Scientific (also called ISI impact factor) (Garfield 2006), has been the dominant measure for ranking a journal’s impact, which is used by research institutions, policy makers, and journal editors alike.

Even though most evaluators stick to some form of the traditional impact factor, one of the earliest proposals was a weighted measure for journals developed by Pinski and Narin (1976). There are other exceptions like analyses carried out by Liebowitz Stanley and Palmer (1984), Palacios-Huerta and Volij (2004), Kalaitzidakis et al. (2003), and Kodrzycki and Yu (2006), who rank economic journals using an iterative procedure.

A recent trend is aimed to develop metrics which represent scientific impact as a function not of just the quantity of citations received but of a combination of the quantity and the quality (Palacios-Huerta and Volij 2004; Bollen et al. 2006; Ma et al. 2006; Bergstrom 2007). In particular (Rousseau et al. 2009) applies an alternative approach to the measurement of scholarly quality which summarizes the incidence, intensity, and inequality of these journals’ highly cited articles.

The SCImago Journal Rank (SJR) (González-Pereira et al. 2010), presents an indicator of can be called “journal prestige" (Bollen et al. 2006), that belongs to a new family of indicators based on eigenvector centrality. The SJR indicator is a size-independent metric aimed at measuring the current “average prestige per paper" of journals for use in research evaluation processes. It has already been studied as a tool for evaluating the journals in the Scopus database, compared with the Thomson Scientific Impact Factor and shown to constitute a good alternative for journal evaluation (Leydesdorff et al. 2010).

The increasing pressure for publication among academics has given rise to a debate whether the gap between the more important scholarly journals and the less important ones is widening. For example, a point of discussion may be whether or not it is true that, during the last decade, famous journals are getting more important and less prestigious journals are getting less influential.

In fact, impact factors of review journals are much higher than those of “normal” journals, while the impact factors of translations are considerably lower (Rousseau and Van Hooydonk1996). This is also true in the scientific community, with more visibility leading to higher impact factors. Already famous journals may receive more credit than they truly deserve, while recognition of less prestigious journals is often withheld (Rousseau et al. 2009). This is called the Matthew effect (Merton 1968), in science.

Given this debate, appropriate summary measures, which provide additional information beyond analyzing the inequality of the whole ranking-score distribution for academic journals in a given subject area, are of key importance for an empirical assessment of the development of the prestige gap of journals with ranking score below a given threshold as well as the overall prestige of journals above a threshold.

The prestige gap of journals with ranking score below a threshold has already been in the spotlight of academic research. Thus Garcia et al. (2011) introduced two different measures of the prestige gap which are free from various arbitrarinesses. They are essentially ordinal and satisfy reasonable sets of axioms. In particular they are sensitive to the distribution of numerical score (using a given journal ranking model) among the less important academic journals (e.g., below the top ten). Also Garcia et al. (2011) showed a longitudinal analysis of the prestige gap of journals with ranking score below the top ten during the period between 1999 and 2009, on the subject areas of Scopus (The Scopus website). It may be interesting to uncover whether, for any reasons, the overall prestige gap of journals below the top ten has been either understated or overstated in certain subject areas; and whether a reduction of such a prestige gap has been much slower in some areas than in others. In addition to this, given that the Matthew effect influences a journal’s impact factor (Rousseau et al. 2009; Podsakoff et al. 2008; Lee et al. 2005; Kelchtermans and Veugelers 2011; Kapeller 2010), it is indisputable that we should study the prestige gap of journals below a given threshold, for example, following the approach used in Garcia et al. (2011).

In this paper we propose that the more important academic journals (e.g., first quartile journals) should also become a particular focus of attention, especially in the context of any possible change in citation and publication patterns. Thus, the primary aim of this paper is to introduce a measure of the overall prestige of journals with ranking score above a given threshold (e.g., in the first quartile Q 1), based on numerical scores returned by a journal ranking model. An axiomatic approach is used to derive the summary measure of overall prestige. This axiomatic measure will be sensitive to the distribution of numerical scores given by the journal ranking model.

The journals and subject areas (or fields) may be different concepts at different levels. And, it is problematic to analyse the relative performance of subject areas via journal ranking scores. But it may be possible to accomplish the analysis of subject areas by means of appropriate summary measures of journal ranking scores, which provide additional information beyond analysing the inequality of the whole ranking-score distribution for academic journals in each subject area. In this paper we propose that the relative performance of each subject area can be evaluated by using the overall prestige of journals with ranking score above a given threshold (e.g., in the first quartile). Here we follow an axiomatic approach in order to derive a significant summary measure for each subject area. And precisely that is what makes a difference in evaluating subject areas: The axiomatic summary measure. In fact, indicator design and academic ranking may introduce some problems in the analysis (Opthof and Leydesdorff 2010; van Raan et al. 2010; Moed 2010; Spaan 2010; Bornmann 2010). Thus we have proposed one axiomatic summary measure following a well-known mathematical approach in order to avoid them.

But, what are the limitations of the axiomatic approach? It is not rare that one would like to impose more axioms that are jointly compatible. It may also happen that the axiomatic summary measure resulting from the original list of axioms is found to react very bad to some significant set of journals. One must then formalize the characteristics of the particular area and state an additional axiom that specifies how the summary measure should behave in this situation, and finally determine the greatest subset of axioms from the original list that are compatible with the new axiom. Of course, compatibility may hold for several distinct such subsets.

The secondary aim of this paper is the longitudinal analysis of the overall prestige of first quartile journals during the period between 1999 and 2009, on the subject areas of Scopus (The Scopus website). Our analysis is based on the SJR indicator (González-Pereira et al. 2010), that was here selected as representing the journal ranking score. All the data were retrieved from the website SCImago Journal and Country Rank portal (SCImago Research Group). The data were downloaded in February 2011, and there are 26 subject areas, plus a general subject area containing multidisciplinary journals.

The Scopus database, which is larger than the Web of Science (Leydesdorff et al. 2010), was selected as representing the composition of world science on a large scale. Scopus, now officially named SciVerse Scopus, is a bibliographic database containing abstracts and citations for scholarly journal articles. It is owned by Elsevier and is provided on the Web for subscribers. SciVerse Scopus is the worlds largest abstract and citation database of peer-reviewed literature and quality web sources (The Scopus website): Nearly 18,000 titles from 5,000 publishers worldwide including coverage of 16,500 peer-reviewed journals in the scientific, technical, medical and social sciences (including arts and humanities) fields; includes over 3 million conference papers; contains 41 million records, 70% with abstracts; and offers sophisticated tools to track, analyse and visualize research.

The setup of the paper is organized as follows: Sect. 2 introduces a summary measure of the overall prestige of journals above a threshold. Section 3 reports the results of our longitudinal analysis. Section 4 concludes.

The overall prestige of academic journals with ranking score above a given threshold

In journal ranking models, the ranking score is a numerical value assigned to a journal representing an indicator of its scientific prestige and influence.

Let \(U =\{ j_1,j_2,\ldots,j_n\}\) be the set of scholarly journals for a subject area or category. Also, let x i be the ranking score of a journal j i in the set U following a given journal ranking model, where a higher value implies that it is a more important journal.

Then, the journals of the set U can be numbered in a nondecreasing order of ranking-score, i.e.,

$$ x_1 \leq x_2 \leq \cdots \leq x_m \leq x_{m+1} \leq x_{m+2} \leq \cdots \leq x_n. $$
(1)

The set of academic journals in U with ranking-score higher than z is called S(z). For example, let z be such that S(z) is the quartile including the top ranked journals which is called first quartile and denoted Q 1.

A very simple measure of the overall prestige of journals above a given threshold may be the percentage of journals with ranking-score higher than thresholdz

$$ H= m/n $$
(2)

where \(m= \#\{i|x_i >z;i=1,2,\ldots,n\}.\)

If we want to analyse different citation and publication patterns this is not a proper definition of the overall prestige, since if no journal above a given threshold (e.g., in Q 1) changes its status, neither a change in a journal’s ranking-score nor a transfer (of ranking-score) between journals will change this index.

Let the prestige gap g i of any individual journal be the difference between threshold z and his own ranking-score x i following a journal ranking method:

$$ g_i= x_i - z. $$
(3)

Then we may define another measure of overall prestige above a given threshold z based on the aggregate prestige gaps for the journals:

$$ G =\frac{1}{n} \sum_{i = 1}^n \max \{ g_i,0\} $$
(4)

with g i being as given in Eq. 3.

The advantage of this definition compared to the measure H given in Eq. 2 is that this aggregate prestige gap is increasing in ranking-scores. However, it is not bounded by the unit interval and is an absolute measure of the overall prestige. Also it follows that a transfer (of score) between two Q 1 journals will not change this definition of overall prestige. Further on, this absolute measure is not scale invariant, since multiplying all ranking-scores with a scalar increases G by this factor.

To overcome these drawbacks, in the following we propose a number of constraints which an axiomatic measure of the overall prestige of journals above a threshold must satisfy, for example, it should be bounded by the unit interval. And following the approach given in Garcia et al. (2011), here we also propose that in order to measure the overall prestige of journals, e.g., in Q 1, we must take into account the number of journals in Q 1 and also the distribution of ranking-score for the journals in Q 1. Thus, we can define a summary measure R of the overall prestige as the normalized weighted sum of the contribution of each one journal to the overall prestige as follows:

Definition 1

A measure of the overall prestige R of journals with ranking score above a threshold z, for a ranking-score configuration x(z), is given by a normalized weighted sum of journal contributions to the overall prestige using weighting function f, as follows:

$$ R =\frac{1}{n} \sum_{i = 1}^n f \left( \frac{ x_i }{z}\right), $$
(5)

where the mathematical form of f depends on a set of axioms to be proposed.

We now present a set of axioms in order to define the exact form of a summary measure as that given in Definition 1 which shall have some desirable properties. To this aim we reformulate to the study of the overall prestige of journals above a threshold a number of constraints which were first used in an axiomatic approach to economic poverty measurement (Sen 1976; Takayama 1979; Peichl et al. 2008).

Thus, a first axiom states that a journal with a ranking-score not higher than z should not influence the overall prestige of journals with ranking score above threshold z.

Axiom 1

Given two ranking-score configurationsx(z) andx′(z) of the same size where the scores of journals above thresholdz (e.g., inQ1) are the same in both cases, the index of the overall prestige of journals above thresholdzmeasured on either configuration should give the same value.

Now, a second axiom can be justified on the idea that small changes in the ranking-score configuration shall not lead to discontinuously large changes in the index of overall prestige.

Axiom 2

The index of the overall prestige of journals above a given thresholdzshould be a continuous function of ranking-scores in the configurationx(z).

In the following, a third axiom states than an increment in the ranking score of a scholarly journal (e.g., in Q 1) shall increase the summary measure.

Axiom 3

An index of the overall prestige of journals above a given thresholdzshould increase whenever the ranking-score of a journal above thresholdzrises.

Next an axiom states a property of subgroup decomposability. That is, the index has to be additively decomposable, i.e., the index of overall prestige is a weighted sum over several subgroups of journals in which the complete set U can be partitioned.

Axiom 4

The degree of overall prestige of journals above a given threshold may be decomposed into the weighted sum of subgroup-prestige indices.

And the following axiom requires that the index of overall prestige shall increase after a progressive transfer of ranking-score (from a more important journal to a less important one) between journals above the threshold (e.g., in Q 1).

Axiom 5

An overall prestige index should increase when a rank-preserving progressive transfer between two journals above the threshold takes place.

Next, following Peichl et al. (2008), a theorem states that these five axioms determine an axiomatic index of overall prestige for a given ranking-score configuration.

Theorem 1

A summary measure of the overall prestige of journals with ranking score above a thresholdz, given by a normalized weighted sum of ranking-scores in the configurationx(z), using a weighting functionfas follows:

$$ \frac{1}{n} \sum_{i = 1}^n f \left( \frac{ x_i }{z}\right), $$
(6)

and such that satisfies Axioms 1 through 5, it can be defined as:

$$ R_{\beta} = \frac{1}{n} \sum_{i=1}^{n} \left( 1 - \left( \frac{z}{x_i} \right)^{\beta} \right)_{+} $$
(7)

with β > 0; and where\(\left( y \right)_{+}=\max (y, 0). \)

Proof

Given a ranking-score configuration x(z), let R be a normalized weighted sum of the individual scores of the set of journals using weighting function f

$$ R=\frac{1}{n} \sum_{i = 1}^n f \left( \frac{ x_i }{z}\right) $$
(8)

where we have that f should be a continuous function in order to satisfy Axiom 2, i.e., to verify that small changes in the configuration x(z) shall not lead to discontinuously large changes in the index R.

But also it follows that weighting function f should be a strictly increasing function, since Axiom 3 states that an increment in the ranking score of a scholarly journal above threshold z should increase the overall prestige index R.

From Axiom 1, a journal with a ranking-score not higher than z should not influence the index R, i.e., R is independent of the ranking-scores not higher than z. Hence to fulfill Axiom 1 we have that

$$ f \left( \frac{ x_i }{z}\right) = 0, $$
(9)

for x i  ≤ z.

Now, from Axiom 4, the index R may be decomposed into the weighted sum of subgroup prestige indices. Thus it follows that the index R has to be additively decomposable.

Finally, following Axiom 5, the index R should increase after a progressive transfer of ranking-score (from a more important journal to a less important one) between journals with ranking score above the threshold. Hence we have that weighting function f has to be concave, and thus, the relative ranking-scores \(\frac{x_i }{z}\) then have to be transformed by a function that is concave on \((1,\infty).\)

For example,

$$ f\left( \frac{x_i}{z} \right)=\left( 1 -\left(\frac{ z }{x_i} \right)^\beta \right) $$

is concave for x i  > z and β > 0.

The result is that we shall transform the relative ranking-scores \(\frac{ x_i }{z}\) to the unit interval by a strictly increasing transformation function f, with \(\lim_{y\rightarrow\infty}f(y)=1\).

Since ranking-scores of journals above threshold z only have a lower bound (i.e., z), we have that alternative weighting functions that were either linear or convex, do not allow for a standardization and therefore the resulting index of overall prestige will be unbounded in those cases.

To sum up, following Axiom 1 through Axiom 5, the overall prestige index R

$$ R = \frac{1}{n} \sum_{i = 1}^n f \left( \frac{ x_i }{z}\right) $$
(10)

shall satisfy that \(f:R_+ \rightarrow[0,1]\) is a strictly increasing and concave function on \((1,\infty).\)

Following Chakravarty (1983), if we define weighting function f as:

$$ f(y)= \left( 1 - \frac{ 1 }{y^\beta}\right)_+ $$
(11)

where \(\left( v \right)_{+} = \max (v, 0),\) we obtain an index of overall prestige, that resembles Eq. 7 satisfying Axiom 1 through Axiom 5:

$$ R_{\beta} = \frac{1}{n} \sum_{i = 1}^n f \left( \frac{ x_i }{z}\right) $$
(12)
$$ = \frac{1}{n} \sum_{i=1}^{n} \left( 1 - \left( \frac{z}{x_i} \right)^{\beta} \right)_{+} $$
(13)

with β > 0, since f being defined as given in Eq. 11 it is a strictly increasing and concave function \(f : R_+ \rightarrow [0,1]\) on \((1, \infty).\) □

The index R has other interesting properties as follows.

For example, given that weighting function f in Eq. 11 is a concave function to satisfy Axiom 5, it follows that the corresponding index of overall prestige increases when we have a more equal ranking-score distribution of journals above threshold z. Therefore, the index of overall prestige increases when homogeneity (among journals above the threshold) rises.

In addition to this property, we have also that when all ranking-scores and threshold z are scaled by the same factor, the overall prestige index R remains unchanged. That is, R is scale invariant.

Development of the overall prestige of Q 1 journals

Here we show a longitudinal analysis of the overall prestige of journals above a given threshold z in 26 subject areas, plus a general subject area containing multidisciplinary journals, since the 1999.

Of course a measure of overall prestige decisively depends on threshold z. Here, we are computing the overall prestige of the set of academic journals in the first quartile Q 1.

To this aim we use index R from Theorem 1, with β = 3. Here this value of β = 3 was selected taking into account that R β resembles the measure H in Eq. 2 for \(\beta \rightarrow \infty. \) It is illustrated by Fig. 1. This figure shows a 2D-plot of index R β on the Computer Science journals in Q 1, which was computed based on SJR ranking scores, for different values of β = 1, 3, 5, 7, 10, 20, 30, 40.

Fig. 1
figure 1

Overall prestige of first quartile journals on the Computer Science area

Full size image

Our analysis is based on the SJR indicator (González-Pereira et al. 2010), that was here selected to obtain journal ranking scores. The Scopus database was selected as representing the composition of world science on a large scale. All the data were retrieved from the website SCImago portal (SCImago Research Group). The data were downloaded in February 2011.

Figures 2, 3, 4, 5, and 6 illustrate the overall prestige of first quartile journals for the 26 subject areas, plus a general subject area containing multidisciplinary journals, since the 1999. To this aim, these figures show 2D-plots of axiomatic index R computed from 1999 to 2009. Table 1 summarizes the results of the per-subject graphs. Also, Fig. 7 illustrates the comparative performance of subject areas.

Fig. 2
figure 2

Overall prestige of first quartile journals

Full size image
Fig. 3
figure 3

Overall prestige of first quartile journals

Full size image
Fig. 4
figure 4

Overall prestige of first quartile journals

Full size image
Fig. 5
figure 5

Overall prestige of first quartile journals

Full size image
Fig. 6
figure 6

Overall prestige of first quartile journals

Full size image
Table 1 Development of the overall prestige of first quartile journals in each subject area of Scopus, since the 1999
Full size table
Fig. 7
figure 7

Overall prestige of first quartile journals

Full size image

Following these results, we have that index R indicates that overall prestige of Q 1 journals decreased in absolute terms on the following areas: Biochemistry, Genetics and Molecular Biology; Dentistry; Earth and Planetary Sciences; Immunology and Microbiology; Neuroscience; Pharmacology, Toxicology and Pharmaceutics.

By the contrary, R indicates an increase in the overall prestige of the Q 1-journals in absolute terms on the following areas: Agricultural and Biological Sciences; Arts and Humanities; Business, Management and Accounting; Chemical Engineering; Chemistry; Computer Science; Decision Sciences; Economics, Econometrics and Finance; Energy; Engineering; Environmental Science; Health Professions; Materials Science; Mathematics; Medicine; Nursing; Psychology; Social Sciences; and Veterinary.

The index R also indicated the presence of strong oscillations in the overall prestige on the areas: Physics and Astronomy; and Multidisciplinary.

There was a number of areas with a (relatively) high overall prestige, since the 1999: Agricultural and Biological Sciences; Biochemistry, Genetics and Molecular Biology; Chemistry; Earth and Planetary Sciences; Environmental Science; Immunology and Microbiology; Medicine; Neuroscience; Pharmacology, Toxicology and Pharmaceutics; Physics and Astronomy; and Multidisciplinary.

There was a number of areas with a (relatively) low overall prestige since the 1999: Arts and Humanities; Business, Management and Accounting; Decision Sciences; and Social Sciences.

Between 1999 and 2009, several areas simultaneously showed:

  • High and increasing overall prestige: Agricultural and Biological Sciences; Chemistry; Environmental Science; and Medicine.

  • High and decreasing overall prestige: Biochemistry, Genetics and Molecular Biology; Earth and Planetary Sciences; Immunology and Microbiology; Neuroscience; Pharmacology, Toxicology and Pharmaceutics.

  • High and oscillating overall prestige: Physics and Astronomy; and Multidisciplinary.

  • Low and increasing overall prestige: Arts and Humanities; Business, Management and Accounting; Decision Sciences: and Social Sciences.

Conclusions

The basic assumption was that it should be possible to study different subject areas by means of appropriate summary measures of the journal ranking scores, which provide additional information beyond analysing the inequality of the whole ranking-score distribution. To this aim we proposed an axiomatic index R of the overall prestige of journals with ranking score above a given threshold.

From a longitudinal analysis of the overall prestige of first quartile journals, between 1999 and 2009, quite striking differences between subject areas have been shown. For instance, index R indicates a relatively low overall prestige since the 1999 for Arts and Humanities; Business, Management and Accounting; Decision Sciences; and Social Sciences. On the contrary, R showed a (relatively) high overall prestige for Agricultural and Biological Sciences; Biochemistry, Genetics and Molecular Biology; Chemistry; Earth and Planetary Sciences; Environmental Science; Immunology and Microbiology; Medicine; Neuroscience; Pharmacology, Toxicology and Pharmaceutics; Physics and Astronomy; and Multidisciplinary.

But even though we have that 2D-plots of index R showed distinct levels of overall prestige for first quartile journals of different subject areas from 1999 to 2003, it follows that differences in overall prestige between subject areas decreased since the 2004.

From the results showed in this paper, the overall winner regarding absolute overall prestige of first quartile journals was the multidisciplinary subject area, even though it had strong oscillations since the 1999.

The lowest overall prestige in absolute terms was given by the Arts and Humanities. But the value of R for this area varied significantly over time, and thus, index R had a high increase from 1999 to 2009.

In a follow-up study it will be performed a longitudinal analysis of the ranking of the 26 subject areas of Scopus, plus a general subject area containing multidisciplinary journals, since the 1999. Several measures should be used for a distinct analysis of structural changes at the score distribution of journals in each subject area. Thus the subject area ranking will be based on three summary measures: The prestige gap for journals below the top ten; the overall prestige of first quartile journals; and the overall prestige to prestige gap ratio. Taking into account at different aspects of the subject area should lead to a more reliable picture than focussing on a single summary measure.