Introduction

With the recent increase in scholars’ participation in the social web, usage data have been widely available heralding the deployment of their application from information organization into the evaluation of scientific impacts on the Web. The indicators emerged from this new evaluation environment called “social references”, “altmetrics” or “alternative metrics”, reflect a variety of social impacts such as downloads, bookmarks, likes, recommendations, mentions, and posts in weblogs, social scientific networks (SSN) or social bookmarking sites (SBS). Altmetrics have some priorities over the traditional citation-based metrics including a wider scope, document variety, and reflection of researchers’ formal and informal scholarly communication) Wang et al. 2012; Li et al. 2012; Li and Thelwall 2012; Bar-Ilan et al. 2012; Thelwall 2012). It is, therefore, hoped that they can depict the impact of all kinds of information sources (not only those included in citation databases) on all kinds of users (not only authors) (Mohammadi et al. in press; Li et al. 2012; Bar-Ilan et al. 2012). In addition, they can be used for studying reading habits, information behavior and information usage not reflected in citations (Borrego and Fry 2012).

The newly emerging indicators, however, may have some unknown deficiencies challenging their validity as a scientific measure. The challenges can be detected over time in the light of research and exploration. That’s why, despite many research efforts in the field, there is emphasis on the need for further research on new indicators, their accuracy and probable advantages over the pre-existing ones (Butler 2008; Harnad 2008; Zitt and Bassecoulard 2008) especially in large scales (Costas et al. 2014).

The correlation between altmetrics and traditional scientometric indicators is a part of the studies attempting to put to test their validity at least as regards the correspondence between the results obtained by the two kinds of indicators. Some of the studies showed strong correlations (Haustein et al. 2010), while some others got to moderate (Mohammadi and Thelwall 2014; Bar-Ilan et al. 2012; Li and Thelwall 2012; Bar-Ilan 2012; Zahedi et al. 2014; Schlögl et al. 2013; Haustein et al. 2014) or weak correlations (Li et al. 2012; Bar-Ilan et al. 2012). Bornmann (2015) revealed that the pooled correlation obtained by the meta-analysis of the altmetrics studies is negligible for micro-blogging counts, small for blog counts and medium to large for bookmark counts from CiteULike and Mendeley.

In library and information science (LIS), weak-to-moderate correlations between articles’ citations and Mendeley bookmarks was reported for few professional journals (Bar-Ilan et al. 2012; Maflahi and Thelwall, forthcoming; Schlögl et al. 2013). Sotudeh et al. (2015) revealed that there was a significantly weak positive correlation between citation counts of LIS articles and their CiteULike bookmarks (r = 0.23). Their findings also showed that research articles, reviews, and proceeding papers were more frequently bookmarked compared to other document types. The bookmarked articles outperform the non-bookmarked ones in terms of the amount of citations received.

As far as our literature review goes, the correlation studies are commonly focused on a limited collection of articles or journals, the large-scaled research of Sotudeh et al. (2015) and Costas et al. (2014) being among the few exceptions. To ensure the accuracy of the results, it is necessary to replicate the correlation studies in a broader scope. Furthermore, aside from Haustein et al.’s (2010) research on 45 physical journals, there exist no investigations conducted at author or journal levels. To judge the applicability of the alternative measures in evaluating scientific impacts, it is necessary to investigate the social-web performance of all scientific entities including journals and authors. Focusing on LIS scientific productions indexed in Social Science Citation Index (SSCI) during 2004–2012 and bookmarked in CiteULike, the present communication studies the correlation between citation-based and bookmark-based indicators at author and journal levels. The impact indicators include journal impact factor (JIF) and author mean citation per paper (AMC) as citation-based indicators, as well as journal mean bookmarks per paper (JMB), the percentage of bookmarked articles in the journal (JB %) and author mean bookmarks per paper (AMB) as bookmark-based indicators.

Literature review

Bookmarking scientific articles by users in scientific bookmarking sites and online reference managers, such as CiteULike, BibSonomy and Mendeley have attracted the attention of many scientometricians in recent years and created a new research trend alongside the traditional citation analysis. “Users counts” is of the indicators derived from scientific bookmarking sites (SBS) showing the frequency of users storing and managing documents in their personal profiles. The indicator shows readership size of a document (Li et al. 2012) and is believed to be able to measure its popularity among user communities in a valid and reliable manner (Taraborelli 2008). The existence of a statistically significant correlation between the indicator and citation counts reveals that SBS can provide useful sources for measuring research impacts from users’ and readers’ perceptions (Bar-Ilan et al. 2012; Li et al. 2012). “Bookmark counts” is introduced as another social impact indicator. Considering that every user can bookmark a work only once in her profile, the measure is equal to “users counts” within a single SBS, while probably varying across different SBSs.

According to literature findings, there is a significant correlation between bookmark and citation counts across articles in Genetics, Social Sciences and Humanities in Mendeley (Mohammadi and Thelwall 2014; Li and Thelwall 2012). Furthermore, there is a significant relationship between WoS- and Google-Scholar- citations received by Nature and Science journals and their bookmark counts in CiteULike and Mendeley (Li et al. 2012). Observing a significant correlation between citation counts received by JASIST articles in WoS, Scopus, Google Scholar and their bookmarks in Mendeley, Bar-Ilan (2012) found that bookmark analysis can be an appropriate complement to the traditional citation analysis. Moreover, the two indicators are found to be significantly correlated across articles published in some other LIS journals including IPM, LISR, Documentation, JASIST, and the Journal of Strategic Information Systems (Schlögl et al. 2013; Maflahi and Thelwall, forthcoming). Focusing on the articles indexed in PubMed, Haustein et al. (2014) found a significant correlation between citation and bookmark counts in Mendeley, with Engineering & Technology and Humanities showing the strongest and weakest correlations, respectively. In a previous study, they had analyzed the relation between the JIFs of physics journals and their bookmarks and found that bookmarking sites can be used as appropriate sources for journal evaluation (Haustein et al. 2010).

Zahedi et al. (2014) studied and compared the quality and accuracy of the indicators derived from Mendeley, Plus One and Altmetric.com. Confirming a weak positive correlation between citations and altmetrics extracted from Altmetrics.com in a large scaled research, Costas et al. (2014) emphasized the merit of altmetrics as complements to citation-based indicators, as well as the necessity for further research in order to clarify their implications for research evaluation. Studying altmetrics extracted from impactstory.org, they also introduced Mendeley as an appropriate source for altmetrics studies. According to their findings, research articles and reviews are more frequently bookmarked and shared, compared to other document types (Zahedi et al. 2014).

Confirming the correlation between bookmarks and citations for articles published in five subject fields in WoS, Mohammadi et al. (Forthcoming) conceived Mendeley as a source of articles hidden impacts on non-authors. Mohammadi, Thelwall and Kousha (Forthcoming) found different motivating factors for bookmarking in Mendeley for various subject fields. Their findings showed that about half of the studied bookmarkers did -or were going to- read at least half of the articles bookmarked in their profiles. A majority of the users bookmarked the articles in order to cite them in their future authorship.

The coefficients yielded by the studies vary from study to study and from one social network to another, signifying different size effects of the correlations. In his meta-analysis on the altmetrics studies, Bornmann (2015) uncovered the overall effect size of the correlations. According to his finding, citation counts are moderately to largely correlated with bookmark counts from CiteULike and Mendeley (pooled r = 0.23, pooled r = 0.51, respectively), while they are inadequately to poorly associated to blog counts and micro-blogging counts (pooled r = 0.12, pooled r = 0.003, respectively). He concluded that the higher the correlation between the corresponding altmetric and traditional citations, the more research-intensive the social media, while lower association denotes a societal impact.

As the related literature shows, the altmetrics studies have been mostly conducted during the recent 2 years, implying the emergence of a new research stream. They are, however, largely concentrated on a limited number of articles, journals and subject fields. Although the correlation between altmetrics and traditional measures is frequently confirmed by the studies, it is required to conduct additional research on different scientific entities, in various subject fields and in a large scale in order to further validate the claims on the validity and appropriateness of altmetrics as scientific impact measures.

Research aims

To test the feasibility of using bookmark analysis in evaluating journals and authors, the present study aims to explore relationship between citation- and bookmark-based indicators across LIS authors and journals indexed in SSCI during 2004–2012 and bookmarked in CiteULike. To do so, it tries to verify:

  1. 1.

    The correlation between JIF and JMB across the journals;

  2. 2.

    The correlation between JIF and JB % across the journals; and

  3. 3.

    The correlation between AMC and AMB across the authors.

Research methodology

Method

Using a citation analysis method, the present communication concentrates on the articles published in LIS highly prestigious journals. The sample was purposefully selected from LIS journals covered in the Journal Citation Report (JCR) 2011 and bookmarked in CiteULike. A 9-year span from 2004 to 2012 was selected as the publication window.

Data collection

Data was gathered in three phases during May and June 2013. At first, LIS journals included in JCR 2011 was identified from “Information Science & Library Science Category”. These amounted to 83 titles encompassing 82,999 articles. We, then, downloaded the articles data from SSCI in WoS. To do so, an advanced search strategy composed of “IS=”, limited to “2004–2012” and “English” was used to seek out the journals by their ISSNs, publication years and language. At last, the articles bookmarking data was collected from CiteULike. Given the high amount of the retrieved articles, CiteULike was searched at journal level by using the command “journal: && [2004 TO 2012]” in its advanced search mode. All articles published in a given journal in the time span and bookmarked by users were retrieved and the frequency of bookmarks for each article was counted. The bookmark number of each article equals the number of users having bookmarked the article in CiteULike and is observable in the “posted by” section, under each bookmarked article. The data were then aggregated under their related authors and journals.

Data analysis

In order to verify the correlation between the alternative and conventional indicators, Pearson and (in the case of non-normality of the data distribution) Spearman correlation were carried out. Cohen’s (1988) guidelines were applied to judge the strength of the correlations. Accordingly, the effect size is weak for the coefficients ranging from 0.10 to 0.29, medium for coefficients from 0.30 to 0.49, and strong for those ranging from 0.50 to 1.

To determine the percentage of variance the two variables share, the coefficient of determination is calculated, by squaring the r values and multiplying it by 100 (Pallant 2013).

Findings

Descriptive findings

Of the 83 LIS journals indexed in JCR, 66 titles (78.5 %) had at least one article bookmarked in CiteULike. Five journals, including Journal of Informetrics, Journal of the American Medical Informatics Association, Annual Review of Information Science and Technology, International Journal of Computer-Supported Collaborative Learning, and Journal of Computer-Mediated Communication are of the top LIS journals as to their JIF, JB %, or JMB (Appendix).

The 82,999 articles received a total number of 154,028 citations and 13,471 bookmarks, which are far lower in comparison. 18,407 (accounting for 22.17 %) of the articles received at least one citation and the rest (87.9 %) remained uncited. 5165 articles were bookmarked by CiteULike users. The percentage of the bookmarked articles, which can be identified as “bookmarkedness” reached to 6.22 % of the total articles (Table 1).

Table 1 The brief status of the recognition of the LIS articles
Full size table

The MC value of the bookmarked articles (12.11) is much higher than that of the total (1.85) signifying their average citation advantage. The highly cited article with 870 citations belongs to the group (Table 1).

The correlation between citation- and bookmark-based indicators

In order to determine the association between the journals’ JIF and JMB values, Pearson correlation test was used across the LIS journals. Given the 9-year time window of the present study, both 2 and 5-year JIFs were tested to achieve an accurate analysis.

According to the results of the correlation analyses depicted in Table 2, there is a statistically significant, positive correlation between the journals’ 2 and 5-year JIFs and their JMBs (r = 0.430 and r = 0.305, respectively). The correlations are of medium strength, based on Cohen’s (1988) suggestions.

Table 2 The correlation between citation- and bookmark-based indicators across journals and authors
Full size table

The JIFs are also revealed to be positively and moderately correlated to the JB % values (r = 0.517 and r = 0.435, respectively). According to Cohen’s suggestions, the coefficients are of medium to large strength.

To get an idea how much variance the bookmarks and citations share across journals, let have a look at their determination coefficients (R 2). As seen in Table 2, 2-year JIF shares 18 % of variance with JMB and 27 % of variance with JB %.

To examine the correlation between AMC and AMB values, Kolmogorov–Smirnov test was used in order to explore the normality of the data distribution across authors. Given the non-normality of the distribution, Spearman correlation test was used (Table 2). The results showed a statistically significant positive, though, weak correlation between the two indictors (r = 0.172, p = 0.01, N = 9283). Based on the R 2 value, the two variables share just 3 % of variance.

Discussion

According to the findings, the articles published in the LIS journals are largely remained uncited. Nevertheless, the uncitedness (87.9 %) seems almost lower in comparison to the unbookmarkedness, which amounts to 93.78 % of the total articles. The difference can be observed in the amounts of citations and bookmarks, as well. The citations are about 11 times larger than bookmarks in number. The relatively lower level of bookmarks and bookmarkedness can result from the limited community of the SBS users in comparison with the potential author community. Besides, a user may bookmark a paper just once in her profile, while she can cite it several times in her papers depending on her scientific productivity. Furthermore, the low amount of the bookmarks could be resulted from the CiteULike database which is limited in its coverage of documents, authors and institutions compared to other reference managers like Mendeley (Li et al. 2012; Bar-Ilan et al. 2012). It is, therefore, necessary to replicate the research in Mendeley in order to control for the limitation in the SSN coverage.

There is a moderate correlation between journals’ 2 and 5-year JIFs on the one hand and their JMBs on the other hand. The impact factor represents the citations per paper and the mean bookmark shows bookmarks per paper for each of the journals. Based on the results, the more cited a journal is, the more visible it is likely to be in CiteULike. As far as our search has reached, there were no investigations at LIS journal level to explore the association between citation- and bookmark-based indicators. However, the finding is in line with Haustein et al. (2010) who discovered the association between the JIFs and bookmarks from Bibsonomy and CiteULike in Physics, however, the correlations are different in their strengths. It is, also, in accordance with Bornmann’s met-analysis (2015) confirming medium to large pooled correlations for citation and bookmark counts from CiteULike and Mendeley.

The correlations found in the present and previous studies are of different strength implying that citation- and altmetrics-based indicators measure different aspects of research impact. As Bornmann (2015) put it, a lower correlation reflects a rather societal impact of a social media, while a higher one implies its research-intensive nature. It is still open to question how big a correlation should be in order that the two related constructs reflect an acceptable statistical similarity and hence a satisfactory reliability. Nonetheless, based on the R2 values yielded in the present studies, the bookmarks share utmost a quarter of the variance of the citations received by the authors and the journals. Therefore, they are preferably a potential complement, but not an alternative, to citations.

The non-strong correlation may have roots in the different environments the indicators emerge from. To explain, the social sources, including CiteULike, are in their infancy and have a long way to attract all potential citing authors worldwide. Consequently, they are considerably limited in their user numbers. Besides, each author can cite a work several times in his authorship, while she can only bookmark it once in her account (in an SBS or SSN). As a result, the indicators signify different levels of usage: consider in-text citations measure a first usage level that reflects the repeated usage of a cited document by authors. It, also, implies the number of citing authors and their paper numbers. Citations—as appeared in references of the citing documents- reduce the impact to a second usage level, by ignoring the usage times. They just imply the number of citing authors and their paper numbers. In the same way, bookmarks lower the impact to a third level of usage, i.e. number of users. It is insensitive to usage times and paper numbers.

However, the indicators differ in that bookmarks broaden the impact scope from authors to non-authors generally missed by the citation measures. In fact, citation count is limited to measuring the cited work’s impact on an author (Garfield 1986; Mohammadi and Thelwall 2014; Li et al. 2012; Li and Thelwall 2012), while bookmarking an article in SBSs reflects its influence on bookmarkers not only as authors but also as readers, fans, searchers, human problem solvers, educators, etc., the roles widely ignored by citation analysis. In spite of the fact, the social sites have not yet achieved a critical mass to realize their actual users. As assessing bookmarking situation of documents together with their citation performance may cover other aspects of their impacts on laymen, non-specialists and scholars communities, it is required to carry out further research to test the association between users’ and citing authors’ counts in order to measure the strength of the correlation more accurately. In summary, citing and bookmarking represent two different activities in two different spaces: while citations measure a scientific work’s direct (formal) impact on research stream, bookmarks mirror its indirect (informal) influence on all kinds of users with explicit or implicit ultimate goals.

Similarly, a journal’s bookmarking situation can reflect expanded aspects of its influence as perceived by users. The bookmarking data is not affected by some of the limitations of JIF and appears to be a useful complement in journals evaluation. For instance, JIF is calculated for the journals indexed in Thomson Reuters’ citation databases and ignores other non-indexed journals. Besides, it excludes some materials published in the journals considered as “uncitable”. It is, also, affected by such deficiencies in the citation databases as the limited coverage of journals, Anglo-American biases, and unbalanced subject coverage. In bookmarking sites, on the contrary, there are no “unbookmarkable” materials. In fact, users can bookmark all kinds of materials including journal articles (regardless of their indexing services), books, book chapters and sites in all languages and subjects.

It is worth mentioning that JIF is insensitive to journals (un)citedness. This is despite the fact that few articles are responsible for a large percentage of citations a journal receives. For instance, 89 % of Nature’s JIF was generated by 25 % of its articles and the rest 75 % received few or no citations (Ogden and Bartley 2008). Given the importance of journals (un)citedness in determining its impact, we also studied the association between journals JIFs and their “(un)bookmarkedness”, i.e. the percentage of their bookmarked articles. Based on the findings, there is, also, a positive, moderate correlation between journals’ JIF (either with a 2 or 5-year window) and their “bookmarkedness”. The finding is in line with Haustein et al.’s (2010) confirming a powerful correlation between 5-year JIF across 45 Physics journals and their bookmarks in CiteULike and BibSonomy. The difference in the strength of the correlation may stem from the inherent differences between Physics and LIS regarding their disciplinary natures, the size of their scientific communities (including authors, journals, audience, departments, etc.), information behaviors and citation and publication practices (Sotudeh 2010), all resulting in different visibility and impact patterns in the fields. In addition, scientific communities and user groups in different disciplines may vary in their approaches towards scientific bookmarking sites leading to differences in the bookmarking patterns. Further deep explorations are, hence, needed to accurately analyze any probable disciplinary differences in bookmarking patterns and their association with traditional recognition indicators.

At author level, a weak positive correlation between AMC and AMB is observed, as well. As a scientific entity different from journals and articles, authors need their own specific indicators to evaluate their scientific performance. There was no previous study investigating the relationship between traditional impact indicators and bookmarking data at author level, to contrast the findings against.

The noteworthy point is that author-specific indicators such as publication number, citation number, MC, h index and g index mainly evaluate authors’ scientific performances as to their formal scholarly and research output. They, however, cannot evaluate all aspects of their scientific influence and research creativity. Authors’ web presence and visibility are of the most important aspects of their scientific behaviors (Bar-Illan 2012), that cannot be reckoned by the conventional scientometric indicators. By acting in SBSs, SSNs, and weblogs, users as researchers and authors extend their scientific contribution to the social web by sharing their outputs, questioning and answering, and communicating. In this way all users, whether authors or non-authors, can easily and directly communicate and get informed of new outcomes in their fields of interests in a short time. Furthermore, they contribute to science as evaluators and gate-keepers by mentioning, liking, downloading, discussing and recommending others’ works. As a result, the data derived from the social web can help measuring these aspects of authors’ scientific influence on public users in the Web. In other words, the scientific contribution of an author as well as her visibility and impact can be traced by monitoring her profile and her fans’ (or followers’) in one or more social sites.

Nevertheless, as any other quantitative measures, altmetrics may have deficiencies in measuring quality related aspects of science. The field is not yet matured enough due to its lack of theory, low data quality and biases (Bornman 2014). Moreover, like citation analysis, the altmetrics analysis can be affected by variations in users’ motivations and interactions in the social web. The altmetrics may signify general impact rather than scientific quality: non-specialist users, text simplicity and intelligibility and hot or controversial topics are among the main reasons to inspire people to bookmark a document. Besides, due to the lack of quality control in the social web, the metrics would be subject to deliberate manipulation or gaming, especially when they would begin to be officially adopted and enforced. Further research are required to understand how users are influenced by a scientific entity and led to bookmarking it. Alongside the user studies carried out based on questionnaires (Mohammadi et al. Forthcoming) and user profile analyses (Mohammadi et al. Forthcoming; Zahedi and Van Eck 2014), mixed methods are required to dig into the users’ behaviors and incentives (Costas et al. 2014).

Concluding remarks

The existing knowledge reveals the association between traditional and altmetrics indicators at article level. The present study contributes to the knowledge in highlighting a moderate to large correlation at journal level and a weak correlation at author aggregation level. As a result, bookmarks can be used as a complement to citation metrics in evaluating the entities’ scientific performance in terms of their broad impacts. However, we should be cautious in applying altmetrics in research evaluation, even as a complement to citation analysis. This is due to the immaturity of the field caused by its not-yet achieved critical mass, lack of theory, lack of quality control mechanisms, inconsistencies and multiplicity of social web sources, data, tools and methods.