A valuable initiative: rebuttals and rejoinders

The initiative that was taken by Scientometrics to offer the opportunity of three groups involved in scientometric and/or bibliometric analyses, or with familiarity with the Hirsch (h)-index, to offer critique and comments of our original paper (Teixeira da Silva and Dobránszki 2018) is commendable. This is because open academic discussions usually have limited place for expansion after initial publication, as post-publication peer review or commentary, often limited to blogs or social media. Thus, at the outset, we believe that this journal-based and -concentrated form of publication will become increasingly important as the publishing industry evolves towards a state of open access and thus open science, assuming greater open debate. We wish to keep our rejoinder as short and succinct as possible, to offer some feedback to the ideas in three letters that will be co-published alongside our opinion piece, by Judit Bar-Ilan, Rodrigo Costas and Thomas Franssen, and Lutz Bornmann and Loet Leydesdorff. Our original analyses were based on data until early October 2017, and since the rebuttals to our original paper are based on those numbers, the original paper has not been updated. However, in this rejoinder, we offer an update in Tables 1 and 2. Also, most of the background references are not repeated for simplicity, so readers are referred to the original manuscript and to the three accompanying letters for more background literature.

Table 1 Summary of h-indexes for both authors of this paper based on four most popular academic databases: Google Scholar (GS), Scopus, Web of Science (WoS), and ResearchGate (RG)
Full size table
Table 2 Summary of h-indexes for both authors of this paper based on four most popular academic databases: Google Scholar (GS), Scopus, Web of Science (WoS), and ResearchGate (RG)
Full size table

Comments to Judit Bar-Ilan

Bar-Ilan (2018) states correctly that “the accuracy of the h-indices of Google Scholar (GS) and ResearchGate has not been established.” As we explained clearly in our original paper, this was the spring-board for analyzing these databases, precisely because the first author was requested to provide his Scopus and GS-based h-index for official purposes, even though the accuracy of these indexes is unknown. Bar-Ilan correctly points out a major weakness of our analysis, which we ourselves had pointed out, namely that it was based on only two researchers, and both within the plant sciences. However, our paper was neither a review of the h-index, nor was it meant to be a large quantitative analysis with a large dataset. It was more of a reflective perspective showing how such metrics and citation databases can in reality distort the representation of the publishing record of some academics. We do agree that much has been published on the h-index and other author-based metrics, but in a world where such metrics are constantly being used, or abused, for formal purposes, and in some cases commercialized (Teixeira da Silva 2017), new case studies that exemplify the issues, especially practical ramifications, are always welcome, in any medium. We fully agree that the number we present and those presented by Bar-Ilan differ for Web of Science (WoS), but these considerable discrepancies only fortify our message, namely that, at least for the first author of this paper, the indexing agencies that have operated for or on behalf of the publishers involved have done a poor job of accuracy.

In other words, when the first author publishes as [TEIXEIRA DA SILVA, JA], it is expected that every paper he ever published that is indexed by WoS should be listed as such. The responsibility of accurate indexing lies with the publisher as well as with Clarivate Analytics, and formerly Thomson Reuters. To show these discrepancies in WoS, we repeated the search both for [TEIXEIRA DA SILVA, JA] and [DA SILVA, J]. When the full compound surname of the first author [TEIXEIRA DA SILVA, JA] was used, WoS was unable to identify the vast majority of papers of the first author. As a result, in this case, less than 10% of first author’s papers factually included in WoS collection were involved in the analysis of the h-index. The same problem was not encountered for the second author with a single family name. We present the details of these enormous existing differences and discrepancies in Supplementary Figures 1 and 2. This issue with the first author’s compound family name is resolved, i.e., it does not exist, in Scopus (Supplementary Figures 3 and 4). The fact that only a tiny fraction of those publications are accurately indexed in WoS by what some consider to be the world’s leading indexing agencies and/or databases, is of concern. In practical terms, any other academic who might wish to identify publications by Teixeira da Silva would have to spend additional energy trying to sift through a whole series of possible permutations, or variations in (sometimes nonsense) name-abbreviations to find the right author group, as pointed out by Bar-Ilan. Supposedly, the issue does not exist with simple family names, as for the second author of this paper, but this lack of accuracy may be affecting the citation accuracy of countless number of academics with compound family names (Ruiz-Pérez et al. 2002; Black 2003). Incorrect indexing of family names can negatively impact authorship and thus recognition of their achievement (Aleixandre-Benavent et al. 2008), and this is pronounced for Spanish and Portuguese academics (Avila-Poveda 2014). This phenomenon, together with the erroneous representation of authors’ names in the literature, thereby adversely affecting indexing and thus downstream citation, is termed “snub publishing” (Teixeira da Silva 2013, 2014).

In order to clearly demonstrate this skewed inaccuracy of WoS, at least regarding the first author, we have added an updated Table 1, and a revised Table 2 that also reflects publications in these databases that are in fact the first authors’ papers, but which are inaccurately representing his name. This suggests that indexing is very culturally dependent. This then begs an important question: which individuals, of what nationality, might be responsible for indexing the names of international academics, i.e., is there a cultural bias in the current indexing industry that could account for this culturally insensitive and/or erroneous indexing? We thus agree with Bar-Ilan that a wide-scale meta-data analysis for a larger pool of scientists of an equally wide range of nationalities is absolutely essential. More details regarding the data used in Tables 1 and 2, and a more fine-scale assessment of Bar-Ilan’s critique can be found in the online Supplementary file.

This “inaccuracy” can also be equated with reduced visibility. Dorsch (2017) also noted that as much as 50% of papers by two information scientists, Blaise Cronin and Wolfgang G. Stock, were not visible in databases including WoS, but still better than the < 10% visibility of papers for the first author of this paper (see Supplementary Figure 1 versus Figure 2). Using Scopus AuthorID, which we did not assess, Kawashima and Tomizawa (2015) found that this database showed 98% recall success and 99% precision for Japanese researchers’ names in the KAKEN database, but this is likely because Japanese have mostly single family names.

Comments to Rodrigo Costas and Thomas Franssen

The opinions by Costas and Franssen (2018) tend to support our general opinion that the h-index derived from different citation metrics can vary widely, even more so when the cultural aspect is taken into consideration, such as compound family names, and this sometimes very wide variability, as exists for the first author of this paper, can affect both the user’s confidence and trust in accuracy of these multiple h-indexes. The Costas and Franssen letter adds great value by pointing out a “profusion of platforms providing individual-level indicators” that provide “h-indexes and collecting bibliographic and citation data at the individual level”, but that lack transparency and, even more importantly, none of which have been validated, i.e., current usage of such performance indices is being based on blind trust. Costas and Franssen also offer a unique and thought-worthy assessment of the challenges of using various forms of the h-index, and how these are embedded in a vanity-based publishing culture rich with distortions. Our conclusions, fortified by the Costas and Franssen opinions, remain unchanged, and have been strengthened by reanalyses in Tables 1 and 2. A fine-scale assessment of the Costas and Franssen critique can be found in the online Supplementary file.

Comments to Lutz Bornmann and Loet Leydesdorff

The opinions of Bornmann and Leydesdorff (2018) accentuate the dispute over the use of the h-index in research evaluation, fortifying our warning that there are unsolved problems with using any h-index, mainly if it is used as a single author-based metric. Their letter highlights two additional disadvantages of using the h-index to evaluate researchers. We fully agree with Bornmann and Leydesdorff that the h-index is database-dependent and that differential coverage of databases is among the main factors affecting the calculated index. However, fundamental technical problems, such as how databases can handle compound family names, can lead to misleading conclusions, as they pointed out in their letter considering the result of WoS for the first author. Examining the problem in detail, however, we realized that when a search was performed for a compound name (in our case, for the correct professional publishing name of the first author of our paper, i.e., TEIXEIRA DA SILVA, JA), that only less than 10% of the data covered by WoS for this author was available (see Supplementary file). WoS has also shown other weaknesses relative to other databases (Falagas et al. 2008).

Bornmann and Leydesdorff argue that “The volume represented in WoS and Scopus does not differ significantly from “reality”—as the authors claim—but these databases are based on professional criteria,…”. We agree with Bornmann and Leydesdorff that one of the main aims of WoS and Scopus includes a selected collection of journals and a limited coverage for books, book chapters or conference papers.Footnote 1 However, there is no direct relationship between the quality of an author’s publication and the databases’ selective coverage mainly for journal papers. If selection is not based strictly on the quality but, as an example in this case, on a preferred type of publication (i.e., journals papers), then those databases may differ considerably from the academics’ publishing reality.

Bornmann and Leydesdorff argue for abstaining from using the h-index in its bare form and propose its use as a normalized indicator. We agree that normalization can improve the usefulness of indicators by eliminating some distorting factors, as they stated, but no form of normalization can fully eliminate limitations originating from database coverage or weaknesses of data handling, as in the case of compound names in WoS. A more fine-scale assessment of Bornmann and Leydesdorff’s critique can be found in the online Supplementary file.

Concluding statements

In our original paper, we presented an additional practical problem that highlights the risks that academics face when using third-party services for their academic profiles. We showed one example of a science watchdog’s GS profile, of Leonid Schneider, that gave a highly skewed perception of his h-index as a direct result of the inclusion of false positives. The first author of this paper has, ironically, found the same errors in his GS profile, and in a bid to clean up false positives that had been added to his profile, most likely by Google, without his implicit permission, three revisions were made of the profile, revealing a considerable number of false positives and also a change in the h-index as a result (Fig. 1). This clean-up of errors that are not introduced by academics is, without a doubt, an infringement of their rights and can result in a waste of time and cause frustration. Bar-Ilan, Costas and Franssen, and Bornmann and Leydesdorff support our notion that any single h-index, when used as an author-based metric, poses risks, and should be used with caution. Collectively, we hope that these letters may spur more detailed research and clarity regarding the calculation, use and thus usefulness of this metric.

Fig. 1
figure 1

Source: https://scholar.google.com/citations?user=dWCOqCkAAAAJ&hl=en

Academics need to take care of “false positives” that are added to their Google Scholar profile automatically, but which are in fact not theirs. As one example, the first author of this paper laboriously verified his Google Scholar profile to find that 29 papers with citations and 57 papers without any citations had been added. Collectively, these 86 “false positives” inflated the citation count by 132 citations. Screenshots of Google Scholar account A with “false positives” and B without “false positives” following a clean-up. C An example of an erroneous entry. All screenshots taken on December 8, 2018. Externally introduced “false positives” should not be academics’ responsibilities, but if left uncorrected, may cause reputational damage. The current Google Scholar profile has once again changed between December 8, 2018 and January 25, 2018, and may once again include “false positives”.

Full size image