Introduction

Traditional citation-based metrics (e.g. citation counts, average citations per paper) do not quantify an individual’s scientific research output well, and then the h-index was proposed to measure the broad impact of an individual’s work by taking both the quantity and impact of the scholar’s publications into account (Hirsch 2005). The h-index is a more objective and correct metric for assessing an individual’s work by papers and citations (Van Noorden 2010). However, it is necessary and important to observe that the h-index is not flawless (Bornmann and Daniel 2007). For this reason, scholars have done much work (Glänzel 2006, 2010; Schubert 2007) and proposed many derivative indices such as the g-index (Egghe 2006), h (2)-index (Kosmulski 2006), w-index (Wu 2010), a-index (Jin et al. 2007), m-index (Bornmann et al. 2008), e-index (Zhang 2009), and q 2-index (Cabrerizo et al. 2010).

To discuss the characteristics of various indices in theory and practice, axiomatic and mathematical interpretations of the h-type indices have been widely addressed. Woeginger (2008a, b) provides an axiomatic characterization of the h-index featuring three natural axioms and then studies the g-index in a similar way, except that he also accounts for new papers without citations. Rousseau (2008) checks some of Woeginger’s axioms or domination relations on the g-index, h (2)-index, and R 2-index. Quesada (2010) proposes three characterizations without using the monotonicity axiom. Bouyssou and Marchant (2014) analyze several bibliometric indices using an axiomatic approach. Marchant (2009) advances a formal framework to axiomatically characterize bibliometric rankings and argues that the objectively true ranking does not exist. Although a few axioms or domination relations for scientific impact measures are introduced, there is no specific way to judge the validity of a measure. This study proposes a criterion, called basic domination relation, which helps to judge whether a given indicator is valid.

The basic domination relation

A scholar with n ≥ 0 publications is represented by a vector \(x = \left( {x_{1} ,x_{2} , \ldots ,x_{i} , \ldots ,x_{n} } \right)\), where \(x_{i}\) denotes the number of citations of the ith publication. Publications are ordered according to the number of citations received so that \(x_{i} \ge x_{i + 1}\). Let X stands for the set of all such vectors (Woeginger 2008a; Rousseau 2008).

Definition of the basic domination relation:

A vector \(x = \left( {x_{1} ,x_{2} , \ldots ,x_{i} , \ldots ,x_{n} } \right) \in X\) is dominated by a vector \(y = \left( {y_{1} ,y_{2} , \ldots ,y_{i} , \ldots ,y_{m} } \right) \in X\), if n = m holds and if \(x_{i} \le y_{i }\) for \(1 \le i \le n\).

Assume that scholars X and Y have the same number of papers, with citations counts represented in vectors X and Y as specified above, each sorted by the number of citations. The basic domination relation states that if Y dominates X then scholar Y is not inferior to scholar X. In other words, if each paper in scholar Y’s list is cited at least as much as the corresponding paper in scholar X’s list, then scholar Y cannot be considered inferior to scholar X.

We propose that if an index violates the basic domination relation, then the index is defective. To all appearances, a large number of indices, such as the h-index, g-index, h (2)-index, and w-index, do not breach the basic domination relation. However, not all indices obey this relation.

Indices that violate the basic domination relation

Indices investigated

The papers of a scholar’s publication output that contribute to the calculation of the h-index are usually referred to as the h-core (Rousseau 2006). The total number of citations of the h-core papers is called the h-core citations, denoted as \(C_{h}\). The concept of the Hirsch core is essential for the description of the indices listed in Table 1. In the present study, we look at four variants of the h-index that violate the basic domination relation, namely the a-index, m-index, e-index, and q 2-index.

Table 1 Definitions of the h-index and some of its variants
Full size table

Examples

Suppose that scholar X has produced five publications, one with ten citations, one with four citations, one with two citations, and two with zero citations, and scholar Y has also published five papers: one cited eleven times, one cited four times, one cited three times, and two cited zero times. The number of citations for the two scholars is listed in Table 2. The difference between them is that the first and the third papers written by scholar Y each have received one more citation. Therefore, scholar Y cannot be considered inferior to scholar X.

Table 2 Number of citations for two scholars (X, Y)
Full size table

Calculating indices gives us the results in Table 3. Commonly, the publication number, citation counts, and average citations per paper are regarded as the three most important traditional measures for research evaluation. Using the traditional evaluation method to evaluate scholars X and Y, scholar Y is not inferior to the scholar X. However, the a-index, m-index, e-index, and q 2-index behave in a way that we consider counterintuitive, ranking scholar X above scholar Y. Our conclusion, therefore, is that those four indices have provided undesirable rankings of scholars X and Y. To analyze the undesirable property of the a-index, Egghe (2010) has given a similar example, and argued that the a-index has the undesirable property that one more citations might lead to its drop.

Table 3 The results of evaluation of indicators
Full size table

Discussion and conclusion

Any index based on citation data should be used prudently when applied to decisions such as promotion, allocation of research funds, and other factors which are vital interests of every scholar (Abbott et al. 2010). When applying an index to an actual evaluation, it is necessary to check whether the index is valid, particularly in cases where the results of the index are counterintuitive. However, “which criteria can tell us that a given indicator is valid?” (Gingras 2016, p. 71).

The basic domination relation proposed in this article can play a part in judging whether the index is valid or not, and can be used as a principle factor. The basic domination relation is based on the citation data and focuses on the most simple and useful criteria. If scholar Y’s papers are cited at least as much scholar X’s, comparing them one-against-one in descending order of citation count, then scholar Y must not be considered inferior to scholar X. Violation of the basic domination relation indicates that an index is defective. This study demonstrates that the a-index, m-index, e-index and q 2-index do violate the basic domination relation given the same number of papers, implying these latter four indices should be used with caution. Evaluators of scholars should note that these four indicators are not unbiased and should not be used independently.

Woeginger (2008b) assumes that publications without citations cannot influence the impact of a researcher and gives an axiomatic characterization of the g-index by adding uncited publications. If a publication without citations is deemed to have no impact, the basic domination relation can be applied to scholars with different numbers of papers if it happens that their number of papers with citations is the same. Ignoring the uncited papers lets us apply the basic domination relation in this special case. But is a publication without citations really worthless? That assumption is useful in comparing scholars with different numbers of papers, but subtracting zero-cited papers may skew the results of other indicators. Consider three traditional evaluation methods as examples. For the scholar who has more uncited papers, removing those papers makes the publication number become smaller and the average citations per paper become larger than the scholar who has fewer uncited papers, and the citation counts remain unchanged. Therefore, the zero-cited papers are valuable when the publication number is used as the evaluation method, play a negative role when the average citations are used as the evaluation method, and are inconsequential when the citation counts are used as the evaluation method.

Whether to extend the basic domination relation using Woeginger’s principle is debatable, and scholars with different numbers of papers cannot be compared according to the basic domination relation. Therefore, future research may be done to see whether using a padded vector (by padding the shorter vector using fictitious papers having zero citations) for the basic domination relation test is appropriate. In particular, does padding to apply the basic domination relation imply that the padded vector should be used for all indices when comparing two scholars?

Using a common example, we find that the a-index, m-index, e-index, and q 2-index do not obey the basic domination relation. The given case is realistic and can also be modified into a large number of other examples, but the case method is only a relatively simple way to prove a statement; it gives little other information. This is a limitation of this study. A better and more complicated approach would be to generalize the case, and perhaps even determine theoretically all conditions under which the a-index, m-index, e-index, and q 2-index will violate the basic domination relation. This seems to be a worthy research direction.