Introduction

The scientific journal is an important part of scholarly communication and for a long time it has functioned both as the sole dissemination channel of research findings and as an important indicator of research quality. The reputation of the publication venue is often determined by the Journal Impact Factor (JIF), which is calculated by dividing the number of citations articles in a journal has received in a given year, by the number of articles it has published in the two preceding years. The JIF thus aggregates article-level metrics to a journal-level metric. Although the Journal Impact Factor has been highly criticized and shown to be flawed in certain situations (e.g., Vanclay 2012) it is still frequently used for different types of research evaluations. A recent report by the EU Working Group on Rewards under Open Science (2017) reviewed evaluation criteria currently used for different types of research assessments and stated that “scholarly publications and their number is the most widespread indicator of performance”, followed by citations and other citation-based indicators. This finding is in clear contradiction with the recommendations of other recent reports and declarations (e.g., San Francisco Declaration on Research Assessment (http://am.ascb.org/dora/), the Metric Tide report (Wilsdon et al. 2015), the Leiden Manifesto (Hicks et al. 2015) and the EU Expert Report on Alternative Research Metrics (Wilsdon et al. 2017) that all recommend the use of multiple indicators, including both quantitative metrics and qualitative methods to assess research. Leonelli (2017) writes in a recent report on incentives and rewards to engage in Open Science activities that for the recommendations to become everyday practice requires “a shift in citation cultures, and a move away from prestige-led assessment grounded on the reputation of the publication venue in which articles appear or the research location where the work is being conducted”.

Different altmetrics are being investigated for their applicability for research evaluation and it has been suggested that altmetrics could expand our view of what impact looks like and what is making that impact (Priem et al. 2010), thus having potential to push for a change in how research is being evaluated. To many researchers and other people reading scientific articles altmetrics appear probably most often in the form of the Altmetric Attention Score provided by Altmetric.com. The Altmetric Attention Score (AAS) is displayed by many publishers next to the online article and it is an indication of the cumulative attention an article has received on different online platforms. The score is calculated by weighing the attention or number of mentions from different online sources (e.g., news, blogs, Twitter, Facebook, Wikipedia, LinkedIn), so that for instance a news article is more valuable than a blog entry and a blog entry is more valuable than a tweet (for the default weightings see: https://help.altmetric.com/support/solutions/articles/6000060969-how-is-the-altmetric-attention-score-calculated). It is worth emphasizing that the AAS is not intended as a reflection of the quality of the article itself. It is an article-level metric and a way to rank research outputs based on the attention they have received on selected online sources. Although the AAS is perhaps the most visible part of altmetrics, earlier altmetrics research has mainly focused on investigating specific data sources of altmetrics instead of aggregations of altmetrics. Recently the focus of the investigations is moving away from finding possible connections between different altmetrics and citations to specific research outputs and towards identifying possible factors that are driving altmetrics, i.e. factors of the papers, authors or journals that are potentially associated with higher altmetrics counts (e.g., Haustein et al. 2015; Didegah et al. 2018). Although the JIF is still often considered as an indicator of research quality similar journal-level indicators for altmetrics have, to the best of our knowledge, not been extensively investigated before. This leaves a gap in current research, one that this research aims to partly fill by investigating aggregation of altmetrics at journal-level and the online presence of selected journals.

Background

Altmetrics are being investigated for their applicability for research assessment. The potential of and hope attached to altmetrics is that they could function as complements to more traditional metrics and possibly reflect some aspects of new forms of online scholarly communication. Much of the earlier research has focused on investigating possible connections between altmetrics and citations. Some earlier studies have shown that in specific cases, such as for articles with highest AASs in dental journals (Delli et al. 2017) and in urology journals (O’Connor et al. 2017), no correlation could be found between the level of online attention and citations. This led the authors to suggest that the online audience place a greater value on different attributes of research and research articles than the scientific community does. On the other hand, journal’s own activity in disseminating information about articles they publish may also lead to higher online attention, as was demonstrated by Wang et al. (2017). Wang et al. (2017) discovered that the AAS was significantly higher for neurosurgery journals that had social media presence than for journals that did not have social media accounts. The study did, however, not investigate how active the journals were in fact in using their social media presence to disseminate information about published articles. In this regard, we pay attention to a recent editorial of British Journal of Surgery (BJS) strongly urging its authors to actively communicate their research outputs with public through social media (Mayol and Dziakova 2017). Similarly many universities and research institutes are increasingly pushing their researchers to more actively engage with the public through social media. Further, a publisher (c.f. Frontier) recently started pushing notifications that guide readers to tweet about new research publications. As more and more researchers and publishers start to disseminate information about their own recently published articles the altmetrics for those publications may get exaggerated, resulting in an advantage over other publications, at least when altmetrics are concerned.

Other factors such as journal impact factors and international collaboration have also been discovered to significantly associate with altmetric scores (Didegah et al. 2018). The authors also discovered that the factors driving Mendeley readership were similar to the factors driving citations, suggesting at the similarities between the two measures. The results from a study specifically investigating altmetrics of South Korea-based research articles have come to similar conclusions, with Mendeley readership counts possibly reflecting a more traditional aspect of scholarly communication while other social media sources only showed a weak connection to citation counts (Cho 2016). Thelwall (2017) goes further and suggests that Mendeley readership counts could be used as early indicators of future citation impact. Similarly to earlier findings there were also disciplinary differences between the data sources used in the study by Cho (2016). Another aspect that drives citations is the open availability of articles: there is plenty of evidence that Open Access articles attract more citations than articles in subscription-based scientific journals (Harnad and Brody 2004; Houghton and Sheehan 2009; Kousha and Abdoli 2010; Wang et al. 2015). A similar advantage has been discovered for altmetrics too, as Open Access articles tend to receive more online attention (e.g., Alhoori et al. 2015) and more downloads over a longer time (Wang et al. 2015). The Open Access Altmetrics Advantage may also be an incentivizing factor for researchers to publish more Open Access articles, thus pushing the Open Science movement forward.

Journals are typically ranked by their JIF, but some national systems where journals are ranked by expert panels exist also. In an effort to produce an altmetrics-based ranking of journals Loach and Evans (2015) tested several methods to produce such rankings. Their results suggested that “there is a consistency between social media based approaches and traditional citation based methods”, but also that the found differences between the used data sources show how different data sources are capturing different features of journal impact or different types of impact. The results suggest that altmetrics may be able to shed some light on the online presence or the online attention that scientific journals receive.

South Korea (Korea hereafter) has imposed a strict evaluation system on domestic journals that can be selected in the Korean Citation Index (KCI). Similarly to many international scholarly abstracting and indexing services, the KCI adopted a ranking system based on expert panel evaluation, publication activity, and citation performance. For research internationalization the Korean government succeeded in incorporating several KCI journals into Web of Science (WoS)’s main interface (c.f. https://clarivate.libguides.com/webofscienceplatform/kjd) and implemented the Scopus recommendation program as well. Further, government has provided scholarly associations with publication subsidies with the objective of wider acknowledgement of Korea-based journals in order to increase their scientific presence around the world (Park et al. 2016). Although Korea has implemented a variety of policy measures, the citation performance of Korea journals has not lived up to government’s expectation relative to scientific productivity that has rapidly increased over past two decades (Yang and Jung 2016. Because government has heavily put a priority on a performance-based incentive program (Jeon and Kim 2018), the broad examination of Korea journals has been largely neglected in term of multifaceted evaluation that would include webometric and altmetric methods. Given that online visibility can increase dissemination and awareness of national research output, government needs to set appropriate policy directions for increasing both societal and scientific impact of Korea journals.

While factors driving citations and factors driving altmetrics at article level have been researched earlier, aspects that may influence altmetrics at a national journal level have, to best of our knowledge, not been investigated before. Similarly, the use of altmetrics to investigate the online presence of scientific journals has not been studied. This research will fill this gap by asking:

  • In the case of Korea-based scientific journals, could altmetrics aggregated to a journal-level be a useful indicator of the performance of the journals?

  • What aspects may have contributed to higher levels of online attention for Korea-based scientific journals?

By answering this question this research contributes to an increased understanding of the potential value (or lack thereof) of aggregating altmetrics to a journal-level and of the aspects that are driving for higher altmetrics in general.

Data and methodology

This study examines the online attention that Korea-based international journals receive and the possible reasons for the attention. The data collection procedure of the Korea journals started by searching for Korea journals indexed in the Web of Science in the Journal Citation Reports of the Science Citation Index 2016 using its location information (Thomson Reuters 2016). This resulted in identification of 114 Korea journals. The journal’s websites were visited to determine if the journal was an Open Access journal (gold or hybrid). The citation performance of these journals was retrieved in terms of the journal impact factors of the journals and the number of citations the journals had received. These were thought to provide an indication of the academic reputation of the Korea journals within the global academic community. Thirdly, internationally-coauthored articles in the Korea journals in the years 2013 and 2014 were respectively obtained from http://www.scimagojr.com. This data was used to determine the level of internationalization of the journals. The 2-year time window used because the number of recorded altmetric events has been steadily increasing since Altmetric.com started to collect the data, therefore using a longer time window would possibly skew the results and give bias to more recent years. Fourthly, we retrieved the altmetrics data from Altmetric.com using its public explorer option (https://www.altmetric.com/explorer) on 17 March 2017 to download all the altmetrics mentions during the past year, for the peer-reviewed research outputs published in the Korean journal under investigation published between 2013-01-01 and 2014-12-31.

Out of the 114 journals only 55 journals (242 articles) had received some form of online attention that Altmetric.com had captured. As earlier research has shown (e.g., Thelwall et al. 2013; Sugimoto et al. 2017) the coverage of altmetrics vary greatly between the sources, thus it was decided to focus the analysis on the data sources with highest number of altmetric events, namely news stories, blog posts, tweets on Twitter, Facebook posts, Wikipedia pages and Google + posts. Most of the online attention received by the journals were on Twitter (1359 tweets in total), while the other sources only received a fraction of the total attention (301 Facebook posts; 116 news stories; 59 blog posts; 51 Wikipedia pages; 25 Google + posts). The other altmetrics data sources monitored by Altmetric.com were excluded due to even lower levels of accumulated online attention by the journals or a complete lack thereof. The altmetric data was overall highly skewed, with for instance only a few journals receiving many tweets (two journals received a total of 792 tweets, while the remaining 53 journals all received 567 tweets in total. If combining the mentions on the different online sources together it is clear that two journals, Journal of Medicinal Food and Journal of Korean Medical Science, attracted the majority of the online mentions. These two journals had attracted a total of 722 and 364 mentions respectively. The remaining 53 journals shared a total of 825 mentions among them.

Earlier research has suggested that Mendeley could be a promising data source for early indicators of citation impact (Thelwall 2017). For the 114 journals included in this research Mendeley readership counts could be found in the data provided by Altmetric.com for 56 journals, with the combined readership count totaling at 4810 (almost 3.5 times the number of tweets). A high Spearman Rank correlation of 0.873 between Mendeley readership counts and the number of tweets by journals reflected how the same journals that are frequently mentioned in tweets also attract more readers on Mendeley. Using the Mendeley data retrieved by Altmetric.com is, however, problematic as Altmetric.com retrieves Mendeley readership data only for outputs that have already registered altmetric events on some other platform (Sugimoto et al. 2017), thus potentially missing plenty of events that appear only on Mendeley. Because of this potential issue the Mendeley data provided by Altmetric.com was not used in this research.

In four analyses the data was grouped in various ways to investigate disciplinary differences between the journals, the influence of international collaboration on the attention received (both citations and selected altmetrics), influence of scientific reputation (journal impact factor) on the attention received, and influence of open access. The data was analysed by both comparing mean counts between different groupings of the data and between different measures of attention, and by investigating correlations between different measures.

Results

Disciplinary differences

The WoS’s categories of the journals were converted and grouped into the five OECD categories; agricultural sciences (4 journals), engineering and technology (11 journals), medical and health sciences (27 journals), natural sciences (11 journals), and social sciences (2 journals). The mean values for international collaboration, journal impact factor, citations, and selected altmetrics for the journals were calculated and compared for each category. Of these Engineering and technology attracted significantly more online attention on the observed platforms compared to the other categories (Table 1). This is similar to Cho’s (2016) findings; in her study, Scopus-indexed publications in engineering and technology sector showed the highest correlation between citation performance and altmetric mentions. Medical and Health Sciences also received somewhat more attention on Twitter and Facebook than the remaining categories, but clearly less than Engineering. It may be that topics such as technological breakthroughs and medical discoveries are topics that attract wider attention and are thus covered on Twitter and Facebook, as these are topics that may have an immediate influence on many people’s lives.

Table 1 Average values of the journals grouped according to their OECD categories
Full size table

International collaboration

The level of international collaboration of the journals was retrieved from SCImago. The journals were listed and sorted in descending order from low levels of collaboration to high levels of collaboration. The journals were grouped into two groups, with the first group consisting of journals with lower than average levels of international collaboration (29 journals) and the second group consisting of journals with higher than average levels of collaboration (26 journals). The level of international collaboration did not appear to be a significant factor in the level of attention the journals had attracted. The group of journals with low international collaboration had attracted 685 tweets in total, while the group with high collaboration had received 674 tweets, leaving only a marginal difference between the two groups (on average, about 24 tweets per journal in the low collaboration group compared to 26 tweets in the group of high collaboration). In news stories, blog posts, and Facebook posts the journals with high collaboration also attracted only marginally more attention (Table 2).

Table 2 Number of citations, news stories, blog posts, tweets and Facebook posts by journals grouped into journals with low and with high international collaboration (mean counts in parentheses)
Full size table

Journal impact factor

The journals were also ranked by their Journal Impact Factors and grouped into two groups; first group with JIFs lower than average (32 journals) and the second group with JIFs higher than average (23 journals). For each of the investigated sources the journals with higher than average JIF received more accumulated online attention (Table 3). For instance, on average the journals with high JIF received about twice as many tweets compared to the journals with low JIF (34.3 tweets vs. 17.8 tweets respectively) and almost fivefold the attention on Facebook (2.2 posts vs. 10.0 posts).

Table 3 Number of citations, news stories, blog posts, tweets and Facebook posts by journals grouped into journals with low and with high Journal Impact Factors (mean counts in parentheses)
Full size table

The data was also analysed to find further evidence of a possible connection (or lack thereof) between the altmetrics measures and the measures of the level of collaboration and scientific attention towards the journals (Table 4). The JIF of the journals appears to have a connection with the level of online attention the journals attract, demonstrated by positive correlations between all the values. The Spearman rank correlation between for instance the number of tweets the journals receive and their JIF shows a rather strong connection (0.526), this does, however, not suggest any causation between the two. On the other hand, the results further confirm that international collaboration do not associate with altmetrics measures.

Table 4 Spearman rank correlation between the observed data sources
Full size table

Open access

Another aspect that has been proven to be a driver for higher altmetric counts has been the openness of the scientific articles, with Open Access journals receiving higher altmetric counts (as well as more citations). This aspect was next investigated and as the journals had attracted most attention on Twitter the possible Open Access advantage on the number of tweets was tested. The websites of the journals were visited in order to determine whether they were Open Access journals, hybrid journals, or subscription-based journals. Of the 55 investigated journals only 8 were completely subscription-based, 15 were hybrid journals with some open access articles, and the overwhelming majority of the journals, a total of 32 journals were (gold) open access. Of the hybrid journals only journals from Springer showed the number of Open Access articles that were published in the journals. The journal with most Open Access articles was Asia Pacific Education Review (https://springerlink.bibliotecabuap.elogim.com/journal/12564), which had published 28 Open Access articles out of a total of 759 articles (3.7% OA). The journal with second most OA articles was Archives of Pharmacal Research (https://springerlink.bibliotecabuap.elogim.com/journal/12272) with 19 OA articles out of a total of 5022 articles (0.4%). The remaining hybrid journals all had published between 1 and 12 OA articles. The hybrid journals could thus be considered as almost solely subscription-based. This was also reflected in the results, as the subscription-based journals received on average 3.8 tweets per journal and the hybrid journals received on average 4.8 tweets per journal. The open access journals, however, received on average 39.3 tweets per journal (Fig. 1), suggesting at a clear open access altmetrics advantage.

Fig. 1
figure 1

Number of tweets on average to subscription-based journals, hybrid journals, and Open Access journals

Full size image

Also, interestingly, 4 out of the 8 subscription-based journals did not receive any attention on Twitter, while 3 out of the 15 (20%) hybrid journals had received no tweets and only 2 out of 32 (6%) open access journals had not received a single tweet (Fig. 1). Openness seems thus to be a strong factor in driving online attention.

Discussion

In this research we set out to investigate journal-level altmetrics and whether they could be a useful indicator of the performance of the journals and also which aspects may have contributed to higher altmetrics counts. While the scientific impact of the journals seemed to have a connection to the amount of tweets and the other altmetrics events the journals attracted, the causation for the connection is unclear. Many journals and publishers are actively promoting articles they publish on social media and thus it was hypothesized that perhaps the journals receiving more online attention are also themselves contributing to that online attention, and thus pushing the altmetric data. As the number of tweets showed the strongest connection between JIF we set to search Twitter for official accounts of all the journals investigated. We could not find any accounts that would be directly linked to the investigated journals. On the other hand, the open access status of the journal clearly contributed to increased online attention.

In an effort to explain causation for higher altmetric counts the articles published in the two journals that received most online attention were studied more closely. In Journal of Medicinal Food (JMF) 39 articles and 16 articles in Journal of Korean Medical Science (JKMS) had received some online attention. The 16 articles in JKMS were tweeted a total of 333 times, while the remaining 31 altmetric events were identified on the other investigated altmetric data sources. One popular article in JKMS, Differentiated Thyroid Carcinoma of Children and Adolescents: 27-Year Experience in the Yonsei University Health System (https://doi.org/10.3346/jkms.2013.28.5.693) had been tweeted 252 times (76% of all tweets). In JMF four articles had received a total of 232 tweets (51% of all tweets) out of a total of 458 tweets to all articles published in the journal. This shows that journal-level altmetrics are heavily influenced by the popularity of specific articles. Altmetrics aggregated to a journal-level would thus not appear to be a useful or reliable reflection of the performance of the journals, but rather a result of availability and the presence of a few highly popular articles. Journal-level altmetrics would thus seem to have inherited some of the issues also present with Journal Impact Factors.

The present research is not without limitations. Focusing the research on only Korean journals indexed in the Web of Science limits the number of potential journals studied and may have an impact on the generalizability of the results. Exclusion of Mendeley readership data may also be seen as a limitation, especially as Mendeley have been suggested to be a prominent data source of scholarly activities and Mendeley readership counts have been shown to correlate with citations. If, however, the goal of altmetrics research is to investigate data sources that might go beyond citation-based indicators, then focusing on those data sources that are least like citations may broaden our understanding of the influence science has on society. Future research could address these issues by expanding the scope of analyzed journals and include Mendeley readership counts (retrieved through the Mendeley API) in the analysis.

Based on our findings we can conclude that Open Access availability is a clear driver for higher online attention for Korean scientific journals, however, this attention is also heavily influenced by the popularity of a few articles that have attracted significant online attention. Because of the influence some popular articles can have on the aggregated journal-level attention, journal-level altmetrics do not appear to be reliable nor useful indicators of the performance of scientific journals.