Introduction

The h index was proposed by Jorge E. Hirsch in 2005 and is by now a well-established informetric means of measuring scholarly impact via a combination of publication and citation counts. It is defined as follows: “A scientist has index h if h of his or her Np papers have at least h citations each and the other (Np–h) papers have ≤ h citations each” (Hirsch 2005, p. 16,569).

But the internet and social networking services have changed the way people communicate. Research has not been unaffected, as scientific output is not excluded from the internet’s everlasting discussion. Classic indicators like the h index do not do justice to the newest ways of expressing our interest anymore. As a result, altmetrics emerged. Priem et al. (2012) define altmetrics as “the study and use of scholarly impact measures based on activity in online tools and environments” (para. 8) which are necessary since “[i]n growing numbers, scholars are moving their everyday work to the web” (Priem et al. 2010, para. 3). Priem et al. (2010) describe altmetrics thus as a means to measure impact in a “diverse scholarly system” (para. 5).

It still is open to debate what exactly altmetrics measure (Meschede and Siebenlist 2018). Haustein et al. (2014b) find that traditional citations and Twitter citations are likely to measure different types of impact. This impression was approved by Costas et al. (2015) who suggest the same for citations and altmetrics in general, and furthermore account for weak but positive correlations between them. Rather than for altmetrics replacing traditional metrics, they suggest exploring “potential complementarity of altmetrics as a source of evidence of other types of impact not captured by citations” (Costas et al. 2015, p. 2015). A great advantage of altmetrics is that they are much faster than citation counts which can take a few years after publication in order to be ready for assessment (Thelwall et al. 2013). This paper tries to formulate an indicator which could be used for such additional assessment.

Yet it should be noted that altmetrics are very heterogeneous, as Haustein (2016) states: “Altmetrics compromise many different types of metrics, which has made it difficult to establish a clear-cut definition of what they represent” (p. 415). It is only logical that for example a mention on a social networking service like Twitter does not mean the same as a read on Mendeley, a social reference management system which was launched in 2008 and taken over by Scopus’ operator Elsevier in 2013. Thus, the indicator proposed here will take into account only one altmetric instead of putting them all together, as attempted in Altmetric.com’s Altmetric Attention Score.

Aduku et al. (2017) state: “Currently, Mendeley readership statistics seem to be the most closely related to citation counts, in comparison to other altmetrics” (p. 575). They furthermore name several studies examining Mendeley readers and citation counts and summarize that there is evidence for Mendeley readers being of use for scholarly impact assessment (Mohammadi and Thelwall 2014) and that either positive or weak correlations between the two have been noted (e.g. Li et al. 2011; Li and Thelwall 2011; Haustein et al. 2014a). And as many classic informetric methods base on citation counts, this paper proposes an indicator based on Mendeley readership counts as an additional indicator of author-based academic evaluations as the citation-based h indices alone do not cover the whole story of scientific activity and impact (Teixeira da Silva and Dobránszki 2018a, b).

Since positive correlations between the two have been noted in previous research, the treatment of Mendeley readers like citations is proposed here in order to make the huge reader numbers comparable. Taking the h index’s approach and molding it to fit the case of Mendeley, a new indicator which shall be called hmen index, as in h index for Mendeley, emerges:

An author has an index hmen if hmen of its Np publications have at least hmen Mendeley readers each and the other (Np–hmen) publications have ≤ hmen Mendeley readers each.

This paper shall investigate the comparison for h and hmen values for 29 information scientists in order to find out about the reliability of the hmen index as an indicator to assess scholarly impact through data provided by the altmetric data source Mendeley. Due to the positive correlations that could be attested to between citation counts and Mendeley readers, a positive correlation between h and hmen index rankings is to be expected. The h and hmen values shall also be explored in relation to an author’s scientific age by making use of the time-oriented h index, also called m index.

Methods

The authors assessed in this paper were selected according to two criteria. Since the extent of Mendeley readership counts varies among different disciplines (Mohammadi and Thelwall 2014), authors had to publish in the same discipline. Here, only information scientists have been regarded. Second, different levels of coverage have been reported for older and newer documents on Mendeley. With Mendeley having been launched in only 2008, older documents are significantly less covered according to Haustein et al. (2014a). Due to this age bias, authors were categorized into two groups with regard to the degree of establishment of an author in the discipline in order to be able to check for correlations between h index and hmen index within different age groups of authors.

The first is a group of well-established authors consisting of the nine top-ranking authors from Cronin and Meho’s (2006) similar study on correlations between the h index and citation counts, and of nine members of the International Society for Scientometrics and Informetrics (ISSI) Scientific Committee, following the selection by Dorsch et al. (2018). The second group of eleven young authors consists of academic staff from the Department of Information Science at Heinrich Heine University, Germany, since only young researchers work there. Professors were excluded because they are expected to be established authors already.

As the documents covered and the number of citations change depending on the information service considered, Web of Science (WoS) and Scopus have both been used to generate two different h values for the authors. Both databases were consulted on March 17, 2018. On Scopus, the author’s publication lists were generated via the author IDs. It was checked whether there were several IDs referring to the same author. If yes, the lists were combined. On WoS, lists were generated via the author search. Author name variants due to umlauts like Schlögl, C*, Schloegl, C* and Schlogl, C* were considered. Each generated list was manually checked for erroneously included documents. For both databases, the h value was manually derived.

Mendeley readers were obtained via Webometric Analyst 2.0 (Thelwall 2009) on March 20, 2018. Author name variants were considered similarly to WoS. From the generated lists, only documents with a matching probability between author name query and document of 1.0 were considered. Each list was manually checked for erroneously included documents and duplicates were merged via a title check. The hmen value was manually derived.

The scientific age of an author for the purpose of calculating the time-oriented h index was determined by checking for the oldest publication on an author’s personal publication list as taken from his or her institutional or personal website. If no such list was available, the date of the oldest publication available on WoS was taken into account instead.

Pearson correlations were calculated via the statistical software R.

Results

Table 1 shows the obtained h and hmen values for all authors, as well as their scientific ages. As can be seen, the young author group has significantly smaller values for each index, which is to be expected due to their young scientific age. In the established author group, there is a greater variance in values due to varying levels of scientific age. The highest values for all three indicators were achieved by Loet Leydesdorff with h values of 59 on Scopus and 54 on WoS and an hmen value of 71. Each author has a higher h value or at least the same on Scopus in comparison to WoS. In the young author group, all authors have higher hmen than h values. In the established author group, this is the case as well for most authors, though some have an hmen value as high as their Scopus h value and four of the authors have a lower hmen than h value. In these cases, the effect of the age bias on Mendeley as described by Haustein et al. (2014a) might be showing.

Table 1 h values on Scopus and WoS, hmen value and scientific age for each author
Full size table

Correlation analysis

The correlations calculated between the three index values for both groups of authors taken together are shown in Table 2. Tables 3 and 4 show the results for the young and established authors respectively when taking into consideration only one author group at a time.

Table 2 h and hmen value correlations for all authors
Full size table
Table 3 h and hm value correlations for young authors
Full size table
Table 4 h and hmen value correlations for established authors
Full size table

The correlation values for all authors taken together are very strong, positive and statistically highly significant. The strongest correlation of 0.989 is and was to be expected between the Scopus and WoS h values since both are based on citation counts. Both Scopus and WoS show similarly significant positive correlations with the hmen index, with Scopus having the slightly stronger correlation of 0.954 and WoS the weaker one with 0.948.

For the young authors, the highest correlation was found between the Scopus h index and the hmen index, 0.869, significant at the 0.001 level, which is not only a stronger but also a more significant correlation than the one between Scopus and WoS h indices. The weakest correlation and at the same time the one with the lowest significance could be established between WoS h index and hmen index with only 0.686.

The established author group shows only correlations significant at the 0.001 level. They behave similarly to the correlations for all authors taken together with the strongest being between Scopus and WoS h indices, followed by Scopus h and hmen index with 0.869 and the slightly weaker correlation between WoS h and hmen index with 0.862. In all three cases, the correlations are stronger and of higher or the same significance as the results for the young author group.

Time-oriented h and h men indices

The h index has been criticized for making comparisons between authors of different scientific ages impossible. In accordance with Stock and Stock (2013, p. 383) following Hirsch (2005), each author’s index values were divided by his or her scientific age, leading to the results shown in Table 5.

Table 5 Time-oriented h values on Scopus and WoS and hmen value for each author
Full size table

The smallest value of 0 can be found for young authors only, since six out of eleven have not been cited in at least one of the databases yet. On Mendeley, however, six out of eleven have values greater than 1 which is quite strong. The highest-ranking young author on Scopus and Mendeley is Franziska Zimmer who has a very low scientific age of 1 with values of 1.0 and 2.0 respectively while having an WoS value of 0. On Scopus, she is followed by Kaja J. Fietkiewicz and Agnes Mainka whose values also range among the higher ones on WoS and Mendeley. The two of them are part of the older half of young authors regarding scientific age. While the WoS values for young authors are still quite weak, the values for Mendeley can compete with those of the established authors.

The highest-ranking author for all three indices is Lutz Bornmann with 2.5 on Scopus, 2.4 on WoS and 3.4 on Mendeley. In the established author group, values greater than 2.0 have only been achieved by the youngest three authors: Lutz Bornmann with an age of 15 and Stefanie Haustein and Cassidy R. Sugimoto with an age of 11.

Discussion

This paper suggested the hmen index as a variant of the h index taking into account readership counts on the social reference manager Mendeley and checked for correlations between the two different h values on Scopus and WoS and the hmen values for 29 information scientists from two author groups. It also compared the time-oriented values for the three services, taking into account the scientific age of the author.

When all authors are taken together, correlations in each case were found to be very strong, positive and highly significant. Furthermore, the Scopus h index always showed a higher correlation with the hmen index than the WoS h index. The coverage of documents on WoS is much smaller than on Scopus (Dorsch et al. 2018) and each document on Scopus is featured on Mendeley, if only with a reader count of zero. Yet in the Mendeley lists, only few documents per author had zero readers, speaking for a high coverage of Scopus documents through Mendeley readers and there might be additional documents not covered in Scopus included. So since the coverage on those two services is higher, it makes sense for the correlations to be stronger as the assessment is more wholesome.

The correlations with the hmen index for all authors were 0.954 for the Scopus h index and 0.948 for the WoS h index, significant at the 0.001 level respectively. Cronin and Meho (2006) reported a correlation of 0.9, significant at only the 0.01 level, for citation counts and h index and deemed the h index “a reliable indicator of scholarly impact and influence” (p. 1278). Following this line of interpretation, this would mean that the hmen index is reliable for scholarly impact assessment as well.

Less clear and less significant are the results for the young author group. A factor influencing the analysis might be the small size of the data set and the problem of very small values with little variety. Especially for the young authors on WoS, six out of eleven authors had an h index of 0, four of 1 and only one author of 2. In a range of only three values for eleven authors, the correlation analysis is not as impactful.

Yet the results hint at the age bias on Mendeley not necessarily impacting the assessment through the hmen index, as the established author group, which would be expected to have the lower and more insignificant correlations with the hmen index, actually shows stronger and more significant ones for both Scopus and WoS.

The values for the time-oriented h and hmen indices show in general no big differences between indices, but they reveal exceptional authors who were found to rather have a young scientific age. Since the young authors are able to compete with the established authors on Mendeley, the hmen index shows different proportions than the h indices. Aylin Ilhan and Christine Meschede, both with an age of 3, have very low values for Scopus and WoS, while having values of 1.7 and 1.3 respectively for Mendeley. Authors who have just started to publish, like Franziska Zimmer with only few publications at a scientific age of 1 and an hmen value of 2.0, are to be regarded as exceptions. Maybe a threshold value for the scientific age should be suggested to make a comparison reasonable. In general, the results hint towards a faster reception of scientific work via Mendeley readers than via citations. The hmen index may thus provide clues to the further career development of these young authors.

As already stated, for the young authors group, the size of the sample probably renders the results unreliable to a certain degree, which is a clear limitation of this study.

Furthermore, Mendeley readership lists might be inaccurate. An example of this would be the case of Haustein and Sugimoto who have the same co-authored document on their respective list’s top spot, yet Haustein has 618 readers while Sugimoto only has 610 due to incomplete information on the authors on some records. In general, all lists were in severe need of correction. Thus, the problem of data quality which is inherent to altmetrics (Haustein 2016) is a problem with Mendeley readership as much as with other altmetric data sources.

An index similar to the hmen index, called alt-index, was already proposed by Hassan and Gillani (2016), but based on weighted social citations in general. They reported a Pearson’s correlation value of 0.247. Thus the hmen index is believed to be much more reliable due to data homogeneity by taking into account only one data source, since heterogeneity is a severe challenge in altmetrics.

Shrivastava and Mahajan (2016) argue that citation counts and Mendeley readers are different indicators and not similar in nature, for example because they show impact with regard to different user groups—only scholars for citation counts, a broader audience for Mendeley readers—but that should not be seen as an argument to discard the hmen index.

It might have limitations, but it also promises new insights. The hmen index seems to be similar to the h index and it can thus be stated that it does indeed a good job of measuring impact of scholarly documents. And if it is indeed different in nature, yet shows similarly reliable results as the h index, the hmen index should be seen as a means to capture an even broader impact of research than the h index, while at the same time staying close to traditional means of impact assessment in comparison to most other altmetrics. The rise of altmetrics happened due to the fact that scholarly impact does not show the complete truth anymore and the hmen index might be a step into the desired direction of measuring scholarly impact and more.

Possibly, the hmen index might also give additional insights, like the prediction of future citation values since altmetrics, including Mendeley readers, are much faster than citations. It could also be possible that a young researcher’s hmen value might hint towards their career development in the future.

For this to be known, future studies testing the hmen index would be necessary. It would be interesting to look deeper into age groups, but also large scale studies about randomly chosen authors would provide important hints to comprehensively understanding the nature of the hmen index. Comparisons between disciplines might be interesting as well due to differing citation behaviors.

In conclusion, the hmen index seems to be a promising indicator which comes close to the traditional scholarly impact measure of the h index, but diverges from it to a small degree. The correlation analysis shows strong, positive and highly significant results, especially when taken together with the Scopus h index. The small degree of divergence should be regarded as a chance for possibly measuring scholarly impact and more, a form of impact on a broader audience, and thus coming one step closer to impact measurement as intended by the emergence of altmetrics.