Introduction

Research influence has been evaluated using the impact factor or h-index based on the number of citations through internationally recognized index systems such as WoS and Scopus. However, in addition to the problem of citation accumulation time delay (Priem & Hermminger, 2010; Thelwall et al., 2013), these indicators cannot measure the influence on the practice, learning, and education of non-academic readers such as clinicians, educators, and students (Cho 2017). Therefore, altmetrics, which have been proposed as an alternative indicator of research influence, are drawing attention. Altmetrics is a method for tracking the social influence of research based on interactions and traces formed by researchers conducting academic communication on the web. When indexing the social impact of research, data are usually captured through Twitter, Facebook posts, Mendeley bookmarks, and other social media tools. To date, studies on altmetrics have mainly focused on whether it can replace or supplement the impact of a traditional evaluation system by examining the correlation between the number of citations and the altmetrics (Luo et. al. 2018; Ouchi et. al. 2019; Zahedi et al., 2014). However, studies have recently been conducted to analyze the lifecycle of the altmetrics source and compare it with the citation system (Fang & Costas, 2020; Ortega, 2018a). As the altmetrics research front is widening, it is forming a new research domain in the field of information science.

Meanwhile, it is known that various factors affect the sensitivity of altmetrics. It has been proven that open-accessed papers are more sensitive to altmetrics because such papers can be more freely accessed by the public than papers published in subscription-based journals. Although the advantages of open access on altmetrics still need to be verified, it has been announced that open-access papers have effect on social media mentions (Cintra et. al. 2018; Wang et. al. 2015). In addition, it is known that the sensitivity of altmetrics appears differently depending on the academic field and sub-research topic. Classics and Literature & Literary Theory are less exposed to altmetrics (Thelwall, 2018), whereas physical science and engineering sciences etc. have quickly attracted social attention (Fang & Costas, 2020). Even in the same research field, social influence appears differently depending on the sub-research topics. Most frequent words in tweet were disorder and depression in the field of psychiatry and topics about care and HIV showed high attention in the field of health policy (Tint & Na, 2017). In addition, even within the same discipline, there are differences in the sensitivity of the altmetrics depending on the journal. In the LIS field, it has been announced that there are journals that respond faster to altmetrics (Ifeanyi and Cyprian 2017).

This study attempted to find out how many highly cited papers in the field of library and information science have been exposed to altmetrics and which research sub-topics are more sensitive. In addition, we examine the influence of OA on the altmetrics. To do so, we set up the following research problems:

  • (RQ 1) What are the altmetrics of LIS research papers? Does open access affect the attention given to altmetrics?

  • (RQ 2) What is the source of altmetrics that correlates with the number of citations in the LIS field?

  • (RQ 3) How is the sensitivity of altmetrics different for LIS research sub-topics?

Literature review

Altmetrics as an alternative indicator for a citation-based impact assessment system refers to a method of measuring the degree to which academic research papers or research data respond to social media on the Web, and is an activity that measures the impact of research using these methods. Because the possibility of implementing the altmetrics philosophy has been noted, web-based applications such as PlumX (plumanalytics.com) and Altmetric (www.altmetric.com) have been developed and used to measure the social impact of research. Altmetrics visualizes how much attention has been paid to articles and research data on social media, reference management tools, and in other areas by calculating them as scores and displaying them on the Web. It has also been used as a complementary impact assessment tool on Scopus, PLOS, and BioMed Central, among other sites.

Studies shedding light on the relationship between altmetrics and bibliometrics have also been conducted (Luo et. al. 2018; Ouchi et. al. 2019; Thelwall et al., 2013; Zahedi et al., 2014), and the limitations and advantages of altmetrics are being actively discussed (Aharony et al., 2019; Gumpenberger et al., 2016; Shekhawat & Chauhan, 2018). Research that has recently attracted attention includes a study on the lifecycle of altmetrics indicators. Fang and Costas (2020) argued that one of the most important characteristics of altmetrics is the reaction speed, which can compensate for the limitations of the citation-based metrics caused by a time delay. This is because academic interactions with social media platforms can take place within an extremely short time. In this study, the authors measured the cumulative speed of 12 altmetrics sources. As a result, Twitter, news, and Facebook accumulate information rapidly, and Wikipedia accumulates information slowly. Similarly, Ortega (2018a) observed altmetrics longitudinally using PlumX. Tweets and blogs respond quickly and have a short lifecycle, whereas Mendeley readers have the highest numbers and the highest growth. By contrast, the number of citations was the slowest and the growth was the lowest.

Let us consider a previous study analyzing the relationship between OA and altmetrics, one of the research problems addressed in this study. Alhoori et al., (2015) defined the open access altmetric advantage (OAAA) as a new term describing the influence of open access on altmetrics. In addition, they investigated whether OA papers from 14 online data sources were mentioned more than non-OA papers. As a result, eight of the data sources reported that OA articles received higher mentions than non-OA articles. Holmberg et al., (2020) investigated whether OA journals from Finnish universities were receiving more altmetrics from the research results. It was reported that the OA journal received more altmetrics mentions in the fields of veterinary sciences, social and economic geography, and psychology, although in the medical field the result was the reverse. In addition, Teplitskiy et. al (2017) examined whether a reference appears in Wikipedia, targeting 250 journals with high usage rate in SCOPUS. As a result, they reported that open-access journals were 47% more likely to be referenced in Wikipedia, and that open access journals can spread science to a wide audience through intermediaries such as Wikipedia.

However, let us summarize an article analyzing the difference in sensitivity of altmetrics based on the academic field, which is another research problem in this study. Ortega (2018b) studied the differences in altmetrics between disciplines with 3793 research papers published in 2013. As a result, it was reported that the usage of the social science field was higher than other field. Similarly, Vaughan, et al., (2017) verified the differences in altmetrics in the fields of natural sciences, engineering, medicine, social sciences, and humanities. As a result, it was stated that social sciences and the arts and humanities fields reflect a broader impact much more than the impact in citing authors. Thelwall (2018) explained that, as a result of analyzing the altmetrics in the humanities and social sciences, the fields of social, political, and gender studies were often exposed to news sources, and the literature field was the least exposed. Htoo and Na (2017) suggested that altmetrics can be a leading indicator of citations in the fields of psychiatry, clinical psychology, and political science. Research topics with high sensitivity to altmetrics were identified in each field. Sexual and therapy in the field of psychology, care and HIV in the field of health science, and innovation topics in the field of management all showed a high tweet frequency. In addition, related studies are being conducted in the fields of chemistry (Chi et. al, 2019), dermatology (Nip & Feng, 2020), and dentistry (Kolahi et. al. 2020).

Finally, we look at a study that analyzed altmetrics in the field of library and information science. Saberi and Ekhtiyari (2019) studied altmetrics in the LIS literature, which is highly cited in Google Scholar. As a result of analyzing the correlation between usage, captures, mentions, and social media of open-accessed papers, it was found that capture and citation showed a significant relationship. Ezema and Ugwu (2019) compared the relationship between the impact and altmetrics of 85 LIS journals in WOS, SCOPUS, and GS. As a result, 18 journals including the Journal of the American Medical Informatics Association, Journal of Computer-Mediated Communication, and Scientometrics, showed relatively high altmetrics attention. In addition, it was stated that Google Scholar showed a higher correlation with altmetrics than the other two DB. Erfanmanesh (2017) analyzed the altmetrics of 563 Iranian LIS papers and found that 12% of social media mentions exist. In addition, Mendeley and CiteUlike verified that there is a correlation with the number of citations.

Similar to the previous studies, this study explores the altmetrics of LIS papers, and attempts to verify some facts that have yet to be revealed. This study will explore LIS sub-research topics that are sensitive to altmetrics and verify whether open access affects the level of attention given to altmetrics.

Method on data collection and analysis

This study analyzed 1000 highly cited papers in 10 sub-topics of LIS. The 10 topics were determined as shown in Table 1 based on the high frequency of peer review papers registered in LISA (Cho 2017). After searching the WOS with 10 topic keywords, the DOI, open access status, and citation count information of the top 100 cited articles were extracted. WOS provides information on OA status through Unpaywall. The information on open access is divided into five statuses. First, bronze indicates no CCBY license, although articles on the publisher’s site can be viewed free of charge for a limited time. Second, DOAJ gold refers to a state published in a gold journal registered with the DOAJ. Third, other gold refers to a state in which a user has a CC license but is not present in DOAJ. Fourth, green means that the final version has been uploaded to the open-accessed repositories and is open. Finally, the status of being included in a subscription-based journal and distributed for a fee is classified as closed.

Table 1 Sub-topics, OA status, altmetrics source classification
Full size table

And the altmetrics attention scores of 1000 papers extracted were surveyed in early 2019 through PlumX. PlumX Metrics are comprehensive article-level metrics which is incorporated into Elsevier's products. These are data captured using a snapshot method based on a specific point in time, and thus the cumulative number may change over time. The altmetrics index is divided into usage, capture, mention, and social media, and finally, the number of WOS citations was collected. For the metrics source for each indicator, as shown in Table 1, “usage” means the EBSCO view, “capture” is the number of Mendeley bookmark readers, “mention” is the number of blog and Wiki reference, and finally, “social media” is the number of tweets.

An analysis of the collected data was carried out as follows.

First, the status of the altmetrics and open-access aspect of the LIS research papers were identified, and whether open access affects the altmetrics was analyzed through the following procedure. The number of citations, views, bookmark readers, blog mentions, Wiki references, and tweets of the collected papers were analyzed through a descriptive statistics analysis, and the OA status was also analyzed. Then, whether the OA affects the usage, capture, mention, and social media attention of the altmetrics was identified through a Mann–Whitney analysis, which is a non-parametric analysis, and is presented visually through a correspondence analysis. A correspondence analysis requires the creation of a cross table for the relationship between the two variables. To this end, a total of 120 papers showing the greatest attention for each source (citation, view, reader, blog, wiki reference, and tweet) were extracted. The 120 extracted papers were selected from 6 sources, 20 each. We then checked the OA method of the papers and created an altmetrics-OA method cross table for visually expressing it on a two-dimensional map.

Second, to determine the type of almetrics sources that correlate with the number of citations, a correlation analysis was conducted between the number of citations and the attention count of each metric source. In addition, to visually understand the relationship between the number of citations and altmetrics, sources with similar properties were identified through a factor analysis and classified into two components, and a scatter plot was created using the factor score. As a result of reducing all variables to two components with similar properties through a factor analysis, the reduced components are “Bookmark or citation”, which include the number of citations and number of Mendeley bookmark readers. In addition, another component is “Social media mention,” which includes the remaining altmetric sources except view (blogs, Wikis, reference tweets). By creating a scatter plot using the factor scores assigned to the two components, it was visually checked whether the article cited in an academic paper or bookmarked for reading showed a sensitive reaction to social media as well. The factor extraction model used the varimax rotation method through a principal component analysis.

Third, to understand the differences in altmetrics sensitivity of 10 LIS sub-subject topics, 120 papers showing a high sensitivity for each of the 6 sources were selected, and a correspondence analysis was conducted on the relationship with the subject topics. In the same way as previously suggested, a two-dimensional map was created based on the cross-tabulation of the altmetrics-subject topics to visually grasp the relationship between the two variables. Through this, it was understood whether there was a difference in topics frequently cited in academic papers, topics frequently read by the public outside of academia, and research topics frequently mentioned through social media. SPSS 25 was used for a descriptive statistics analysis, Mann–Whitney analysis, factor analysis, and correspondence analysis.

Analysis

Status and correlation between altmetrics and open access

First, as a result of examining the altmetrics attention statistics of 1000 LIS papers, it was confirmed that 63% of the data had a reader and 36% of the data were viewed, as shown in the Table 2 below. Articles mentioned in blogs and Wiki references were 3% and 4%, respectively, whereas 17% of papers were mentioned in tweets. Looking at the mean, the number of views was the highest at 210, followed by readers (69) and tweets (1.5). It was found that the number of citations showing an average of 38 cases was half the number of readers.

Table 2 Altmetrics attention score by source
Full size table

Next, the results of examining the open-access status of the papers to be analyzed are as follows (Table 3). Most LIS articles up to 80%, were found to be closed. Among the open papers, DOAJ gold (9.8%) was the most common, and bronze and green each made up approximately 4%.

Table 3 OA status
Full size table

In addition, whether open access has an effect on the number of citations was examined as follows (Table 4). It was found that the number of citations was higher in closed papers, whereas the results of the Mann–Whitney analysis conducted to analyze the differences between groups were interpreted as not having a significant difference (p = 0.09). As a result of examining open access by the source of the altmetrics, open-accessed articles showed higher numbers in views, blogs, Wiki references, and tweets. However, the number of readers was higher for closed papers, similar to citations. However, as a result of verifying the significance of differences between groups, only views, blogs, and tweets (p < 0.00) showed meaningful differences. Therefore, it can be summarized that open-accessed papers only from these altmetrics sources show a sensitivity.

Table 4 Differences in attention to altmetrics according to open access
Full size table

If so, let us visualize and understand the relationship between the altmetrics source and the five OA methods. To this end, 120 papers, each showing a high ranking in the altmetrics source, were extracted. For the extracted papers, a cross table for the altmetrics–OA relationship was created, and a correspondence analysis was conducted based on this. A correspondence analysis is a multivariate data analysis method that visually draws row and column information in a two-dimensional space to explore their correspondence relationship. The relative position between variables is expressed through a dimensional reduction technique. As a result, a map (Fig. 1) showing 86% of the two-dimensional cumulative explanation rate was calculated as follows: Looking at the nodes located on the map, one can see that there is a closed node in the middle, and nodes of citations, readers, and references are located nearby. These positions on the map indicate that closed papers may have a high number of citations, numerous readers, or relatively high number of Wiki reference mentions. By contrast, bronze appears at the bottom of the map. Bronze means that the CCBY license does not exist, but the paper is open. Around bronze, the view is close and a citation is far. Therefore, although the articles accessed using the bronze method were not cited much, they may have been read numerous times outside the academic world. However, there is a gold method including othergold and DOAJgold on the right side of the map, and tweets are closely located. This means that there may be a relatively large number of papers that are exposed in tweets using the gold open-access method. In summary, the visualized result is the same as the previous descriptive statistics analysis result. Papers with a high number of citations and readers are likely to be non-OA. In addition, there is a probability that the papers that have been viewed or mentioned numerous times on social media have been open accessed.

Fig. 1
figure 1

Correspondence chart of the relationship between altmetrics scores and OA method

Full size image

Relationship between citations and altmetrics

Looking at the citation and altmetrics source correlation through Table 5 the number of citations and readers showed a weak correlation of r = 0.29 at p = 0.01. However, because there is no correlation with the other metric sources, only the number of bookmark readers among the almetrics sources is analyzed to show a correlation with the number of citations.

Table 5 Results of Pearson’s correlation analysis between number of citations and altmetrics attention
Full size table

Next, let us visually examine what similarities are shown between altmetrics sources, how many components can be made that similar sources reduced to, and what types of relationships exist between components. To do so, we conducted a factor analysis to see how the six altmetrics sources, including citations, can be reduced. As a result of a KMO and Bartlett’s test to examine the correlation between the data, KMO was 0.49, and the Bartlett test significance probability was 0.000, as shown in Table 6. The component matrix rotated by varimax was reduced to components 1 and 2. Component 1 consists of a blog, a Wiki reference, and a tweet. Each factor in loading represents 0.831, 0.564, and 0.675, respectively. Component 2 consists of a reader and citation showing factor loadings of 0.787 and 0.770, respectively. Because component 1 is concentrated as a value that indicates the social influence, it is referred to herein as the unified term “Social media mention,” and component 2, which consists of citations and a Mendeley bookmark reader, is called a “Bookmark or citation.” Component 2 occurs when a researcher cites a paper during the writing process, or bookmarks a reference management tool to read for the purpose of research, learning, or practice. This can be explained as an index accumulated during the researcher's academic and professional activities. However, it can be stated that component 2, which represents cases mentioned in blogs and Wiki references, or exposed to social media, is a trace of various activities outside of academia, unlike component 1. To visually confirm the relationship between components 1 and 2, a scatter plot, as shown in Fig. 2, was created using the factor scores assigned to each analysis paper. As a result, it was confirmed that the scatterplot showed a variance close to the inverse relation. In other words, it was interpreted that simply because the number of “Bookmarks or citations” increases, “Social media mentions” cannot be said to increase as well. Articles that were mainly read or cited for the writing of academic papers were not mentioned much on social media. Conversely, it can be summarized that papers with a high social influence are not necessarily cited in academic papers.

Table 6 Results of a factor analysis of citations by number and attention to metrics
Full size table
Fig. 2
figure 2

Scatter plot of Component 1 (Social media mentions) and Component 2 (Bookmarks or citations)

Full size image

Differences in altmetrics sensitivity across LIS 10 subject areas

To examine the difference in attention given to altmetrics in the sub-subject topics of LIS, a total of 120 articles were selected by extracting 20 articles each with the most presence for the altmetrics sources. In addition, after mapping the selected 120 papers to the subject topics, a cross table was created using the altmetrics source and 10 subject topics as rows and columns. Based on this, a correspondence analysis was conducted. As a result, the following map Fig. 3 was calculated, and the two-dimensional cumulative explanatory power between altmetrics and subject topics was high at 77%. As shown in Fig. 3, information technology and knowledge management are located in the bottom-right part of the map where the citation and reader are located. To interpret this, there are relatively many cited articles for writing academic papers or bookmarked articles for reading information technology and knowledge management topics among the 10 sub-subjects of LIS. The highest number of citations in information technology is the paper published in 2012 in MIS Quarterly. This paper, cited the most at 1168 times, is a closed paper. Although this paper shows a high citation rate, there are no data in the indicators that explain its social influence such as usage, readers, mentions, and social media attention. Meanwhile, the paper published in International Journal of Information Management, 2011, which is classified as a knowledge management, with a high number of readers at 780, is also a closed paper. It was cited 120 times while showing a high number of readers, but it was never mentioned on social media. In this way, the subject area located in the lower-right corner of the map can be summarized as a topic showing a higher influence among academic researchers, rather than social influence. Meanwhile, let us look at the upper-right view (usage). The subject located around this area appeared in public libraries and websites. In other words, the subjects related to public libraries and websites were not cited relatively frequently, but they were interpreted as being widely used by readers inside and outside the academic world, such as in learning and practice. A paper with a high number of views (6754) in the public library area is an article about rural public libraries digital challenges published in Information Technology and Libraries in 2014. This paper, open accessed as a DOAJ gold paper, was cited only 18 times, but it showed numerous views because it was an OA paper. In addition, for website topic article about credibility of online information, published in the Journal of Computer-Mediated Communication in 2013, which showed 682 views. It is also a bronze OA paper. It was cited approximately 20 times, but was tweeted 13 times. Finally, let us look at the space around the tweet, which is located on the left and is the farthest from the citation. The subject areas are also arranged here, and you can see that scholarly communication and librarian topics are the closest. A study on academic communication and librarianship topics was frequently mentioned in the tweets. An LIS article that was tweeted 141 times was discovered, which is the article published in Insights-The UKSG Journal, DOAJ Gold Journal. This paper contains content that provides guidelines for researchers to prepare for damage to predatory journals. Although this paper on the topic of ​​academic communication has only been cited four times, is noted as a paper that has been tweeted and retweeted several times by researchers and librarians. In addition, the paper tweeted 89 times was classified into the topic of librarians as a paper related to OA policy and license development in the UK. This paper, published in Insights-The UKSG Journal in 2016, has a small number of citations like the paper above, but has been tweeted by many researchers and the library community, showing a high social influence. Unlike the subject areas located at the bottom-right of the map, these papers can be said to be representative papers that have shown a significant social influence outside of the academic world. In summarizing the relationship between altmetrics and research topics, the fields of information technology and knowledge management have attracted more attention in the academic world because of their high number of citations and large number of readers. By contrast, public libraries and websites can be said to be areas that have been read a lot outside of academia for practice and learning. In addition, in the recent environment where major changes occur in the academic ecosystem, it is inferred that academic communication is a topic being mentioned to the public while responding sensitively to tweets. Although the analysis was not conducted on a large number of data, it was confirmed through the above analysis that topics that are frequently cited or bookmarked by researchers in the LIS field, and topics that are attracting attention outside the academic world, can be identified.

Fig. 3
figure 3

Correspondence chart for altmetrics scource and topic relationships

Full size image

Conclusion

The findings of this study, which analyzed altmetrics on 1000 highly cited papers in LIS, are as follows. First, in 1000 LIS papers, readers were in 63% of the analysis data, and views were in 36%. Articles mentioned in blogs and references were scarce at 3% and 4%, respectively, whereas 17% of the articles were mentioned in tweets. Meanwhile, it was confirmed that most of the LIS papers, up to 80%, are distributed in non-OA. Second, views, blogs, and tweets showed greater sensitivity in open-accessed papers. The number of readers and the number of citations were larger in non-OA papers, but did not show a significant difference from OA papers. Third, it was found that there was a correlation between the number of citations and the number of bookmark readers at r = 0.29, but not with other metric sources. Through a factor analysis, citations and readers were grouped into one component (“bookmark or citation”), and blogs, references, and tweets were also grouped into another component (“social media mention”). In addition, the higher the number of “bookmark or citation” instances the fewer the number of “social media mention” instances, and thus it was understood that papers that were frequently read and cited by researchers within academia and those with social influence outside the academia were distinguished. Fourth, the topics of information technology and knowledge management showed high citations and a large number of readers, and therefore it was analyzed that they have been read and cited numerous times by researchers. By contrast, the fields of public libraries and websites were widely used for learning and practice, and it was interpreted that they showed numerous usages outside of academia. Finally, it is estimated that the field of academic communication is highly sensitive to tweets, and is relatively frequently mentioned by policy developers and industries related to library and academic information distribution. Citations only explain the influence they had on the authors of the paper in academia, whereas altmetrics can capture influence from various angles. The number of Mendeley readers also shows a strong correlation with the number of citations, and thus it is possible to trace the hidden influence of research achievements read by various members of the public, although not academic authors. Social media did not correlate with citations, but it can capture public discussions and interests.

Because this study only considered papers with a high number of citations for LIS 10 topics, it was revealed that other research results may appear if the number of analyzed papers that can represent the subject is increased. In addition, because this study utilized the sources provided by PlumX, it should be added that if other altmetrics tools are used, slightly different results may occur.

However, the facts that open-accessed papers have a higher impact on altmetircs (Cintra et al., 2018; Wang et al., 2015), Mendeley reader and citation rates have a correlation (Zahedi, 2014; Erfanmanesh, 2017) were found as common results with precedent studies. In addition, this study also suggested a new result that there may be differences in the sensitivity of altmetrics according to the sub-topics of LIS. There is a difference in topics frequently cited in academic papers, topics frequently used by the public outside of academia. If the citation index is only used in the evaluating research outcomes, such topics that are more prominent outside the academia inevitably would be devalued. In a field with a strong interdisciplinary nature, such as LIS, journals that have a relatively high citation rate due to their academic interest, and journals focusing on library practice coexist. Journals that share their experiences in library practice have inevitably a small academic community and a relatively low citation rate. Therefore, by using the altmetrics as well as the citation, it should properly evaluate the value of papers that are widely used outside the academia.