Introduction

As the publishing sector continues to grow, the need to measure the quality of the science produced grows in parallel. Scientometrics, the field of study that emerged in response to this need, focuses on measuring and analyzing scientific literature, measuring the impact of scientific articles and journals, and using such measurements in purposeful contexts. Individuals and institutions use scientometric indexes to research or to describe the scientific impact of researchers, articles, institutes or universities, and even countries. Academic title upgrades, libraries’ selection of journals to which journals will be subscribed to, calculations of the scientific impact of academic articles, and planning academic financial support allocations are the primary uses of scientometric indexes

Although scientometric indexes are based on calculations of citations and articles, the count and type of articles and citations included in the calculation differ. In the Science Citation Index Expanded (SCIE) database, the most widely used database to globally access scientific data, the journal impact factor (JIF) is the main highlighted index, though several options exist. It is the primary scientometric data in a highly reliable citation index and has widespread use; however, there are essential criticisms against JIF [2]. Like JIF, the Scopus database incorporates CiteScore (CS), another journal-based measure of the average number of citations per paper [3]. Alternatively, the Scopus database also includes Hirsch (h)-index based on their citation data [3, 4]. Although h-index provides a different view on the overall impact of a journal by both the number of citations and published papers, it is affected by academic age since both accumulate with age, therefore having a positive bias toward seniority [4, 5]. Newer alternatives use computational models that take into account citation potentials and average prestige calculations. Source Normalized Impact per Paper (SNIP), another scientometric index included in the Scopus database, accounts for field-specific differences in citation practices and measures contextual citation impact [4, 5]. Several computational models measure average prestige per article, considering the source of citations and the number of citations. This model is included in the SCIE database as the Eigenfactor score (ES) and in the Scopus database as the Scientific Journal Ranking (SJR) [6].

For a scientific article to have an impact, in addition to its scientific value, it must reach the relevant readership. Medical journal authors publish using the open access (OA) publishing model to reach wider audiences, and this publication model is becoming widespread [7]. OA journals and periodical journals are peer-reviewed, and while facilitating access to scientific content, they accumulate more citations compared to subscription-based journals [8]; however, in addition to the pre-existing submission fees and page charges for medical journals, costs, such as article processing and publishing fees (Article Processing/Publishing Charge, (APC)), of the OA model can affect scientists’ choice of medical journals. There are two main OA models, namely journals that publish all articles OA (full OA) for free or with an APC and journals publishing OA on demand (hybrid OA).

The average number of articles and cumulative citations differ based on the medical disciplines. In addition, the average citation time for a published article varies. Therefore, interdisciplinary evaluations of scientometric indexes are not appropriate; specific studies are needed for each discipline. Although various studies that have compared the scientometric data of journals according to publication policies are available in other fields of medicine [9,10,11], the evaluation of rheumatology journals on this issue is not available in the literature. Moreover, there is no study in the literature analyzing the scientometric data of full OA and hybrid OA journals in rheumatology. Based on this need, we aim to (i) present scientometric data of journals published in the field of rheumatology and (ii) analyze the scientometric data of rheumatology journals according to the full OA and hybrid OA publication policy.

Methods

Journals published in the rheumatology category in the Science Citation Index Expanded (SCIE) database were evaluated. Among the 32 journals included in the database, those that have published less than 30 articles in the last 3 years or less than 10 in the last year (n = 2), that only accept articles by invitation (n = 4), that do not publish in English (n = 2), and that have missing data (n = 2) were excluded from the study. Twenty-two journals were included in the final analysis.

Scientometric indexes and bibliometric data were obtained from publicly accessible data on the Clarivate Analytics InCites, Scopus, and Scimago Journal & Country Rank websites in January 2021. The current data available were for the year 2019. The count of citable articles printed by the journals in the last 5 years, the JIF, the ES, the count of articles published as OA, the count of articles published for the subscription-only model, and the count of citations to OA articles and subscription articles were obtained from the Clarivate Analytics InCites database [12]. Based on these data, two calculations were made: (i) OA article publication rate in all journals (count of OA articles divided by the total count of articles) and (ii) C/C (citation/citable articles) ratio: ratio of citations to OA articles to the ratio of citable OA articles in journals with a hybrid OA publication policy (calculated as (the count of citations to OA articles divided by the count of all citations) divided by (the count of citable OA articles divided by the count of all articles)). CS and SNIP data were accessed from the Scopus database [3]. The SJR score and the h-index (HI) were publicly available in the Scimago Journal & Country Rank database [13]. Quartiles were not directly included from the indexes to prevent selection bias, but they were recalculated according to the six scientometric indexes (JIF, CS, HI, SNIP, ES, and SJR) included in the study.

The data on the journals’ publications were accessed from the journals’ and publishers’ websites. The types of manuscripts published by the journals (review, original article, case report, etc.) were categorized into three main categories: (i) journals that publish all articles, (ii) review-only journals, and (iii) journals that publish all articles except case reports. The OA publication policies of the journals were categorized as (i) full OA with APC, (ii) full OA without charge, and (iii) hybrid OA. The fees requested from the authors, the timing of the fee (before or after submission), the amount, and the purpose of the fee requested (for manuscript submission, page charges, or OA APC) were obtained from journal and publisher websites. Color printing fees are not included in the analysis. The US dollar (USD) was used as the requested fee unit.

The data were analyzed using SPSS for Windows version 23.0. Continuous variables are expressed as the median (minimum–maximum), and categorical data are expressed as values and percentages. Chi-square tests for categorical data, Spearman’s rho correlation for categorical correlations, and the Mann–Whitney U test for continuous variables were used. Spearman’s rho correlation was chosen because it is a non-parametric comparison used for variables with non-normal distribution. Each correlation coefficient was interpreted as very strong (at least of 0.8), moderately strong (0.6 up to 0.8), fair (0.3 up to 0.6), and poor (less than 0.3) [14]. For all statistical tests, p < 0.05 was accepted as the statistical significance limit.

Results

The bibliometric data, scientometric data, and publication policies of 22 rheumatology journals in the SCIE database are summarized in Table 1. Two-thirds of the journals included in the analysis publish all types of articles. Only one journal has a review-type article publishing restriction. Six journals have a full OA publication policy, and the median OA article publication rate is 10%. The median C/C ratio in hybrid OA journals was 0.99. While nine journals demand a mandatory fee from the authors, only four of these journals charge a fee for the OA publication policy after accepting the manuscript. Only one hybrid OA journal charged a 40 USD submission fee, and four hybrid OA journals charged page charges after publication acceptance (60, 70, 75, and 120 USD per page). The median OA APC fee was 3300 USD (Table 1).

Table 1 Bibliographic and scientometric data and publication policies of rheumatology journals
Full size table

In this study, we evaluated six different scientometric indexes. We observed statistically significant positive correlations between all pairs of scientometric indexes. Notably, strong positive correlations exist between JIF and CS, JIF and SNIP, JIF and SJR, CS and SNIP, CS and SJR, HI and ES, and SNIP and SJR. Moderate positive correlations were observed between JIF and HI, CS and HI, CS and ES, HI and SNIP, HI and SJR, SNIP and ES, and ES and SJR. A fair positive correlation exists between JIF and ES (Table 2). The quartile distribution of the journals of the six different scientometric indexes is visually expressed in Table 3. In the evaluation made with reference to JIF, it was observed that the top three journals in the first quartile were found in all indexes, albeit with different rankings. A journal in the first quartile of JIF was observed in the second quartile of CS, SNIP, and SJR, and the last quartile of HI and ES. Among the OA journals, only one journal is in the first quartile for JIF, CS and ES. In the rankings of HI, SNIP, and SJR, all first-quartile journals are hybrid OA journals. Two journals that publish a full OA free of charge rank consistently in the last two for all six scientometric indexes (Table 3).

Table 2 Spearman’s rho correlation coefficients of scientometric indexes of rheumatology journals
Full size table
Table 3 Distribution of rheumatology journals into quartiles according to six different scientometric indexes
Full size table

When the bibliometric and scientometric data of rheumatology journals that publish full OA and hybrid OA were compared, we observed that the bibliometric data were similar. The median OA article rate of 16 journals with a hybrid OA publication policy was 8.68%. While JIF, ES, and SJR were similar for full OA and hybrid OA journals, HI was higher for hybrid OA journals (p = 0.03, Mann–Whitney U test). When OA APC fees were compared, we observed that full OA journal fees were less expensive by a median of 935 USD (p = 0.007, Mann–Whitney U test) (Table 4).

Table 4 Comparison of bibliographic and scientometric data of rheumatology journals with all OA with APC and hybrid OA publishing policies
Full size table

Discussion

Scientometric indexes are widely used in medical publishing; however, it should be noted that the literature is controversial regarding non-ideality. Most of the existing indexes are under heavy criticism for different reasons, and the main criticisms are directed toward JIF. The biggest criticisms regarding JIF are its specificity to the study field and that journals that publish reviews have higher JIFs [15, 16]. This situation may cause some journals to tend to publish reviews with higher citation potential than original studies, a manipulation resulting in achieving a relative increase in JIF. Some of the few articles with very high citation counts in high-prestige journals have the principal share in the JIF of the journal, which may create unrealistic values for other articles with low citation counts. Supporting these criticisms, it was observed that three of the four articles previously published in high-prestige medical journals had lower citation counts than the JIF of the journal [2]. CS, a similar journal-based scientometric index to JIF, differs from JIF in several ways: (i) CS is based on Scopus rather than on Web of Science; (ii) CS can be more widely used since Scopus indexes a larger number of journals; (iii) CS is available to anyone, JIF is only available by subscription; and (iv) a now-revised CS takes into account the last 4-year performance rather than a 2-year window [3]. The pitfalls can be considered similar to JIF. In addition, it should be emphasized that the San Francisco Declaration on Research Assessment (DORA) recommends avoiding the use of journal-based scientometric indexes and states that JIF should not be used as an evaluation tool [17]. The HI, which measures both the number of articles produced and the citation counts of the articles, has been criticized for creating a positive bias toward more senior researchers and journals that publish for longer periods. Moreover, the HI does not take the study field into account, so it does not consider publication and citation profiles of different fields. It can also be manipulated in various ways, such as self-citations. Alternative indexes, such as SNIP, ES, and SJR, consider citation count, citation potential and citation prestige; however, these indexes are much newer than JIF and have not yet been evaluated compared to other indexes in all disciplines [3,4,5]. One of the aims of this study was to demonstrate and analyze scientometric data of rheumatology journals.

This study is the first study in the literature to analyze the scientometric data of rheumatology journals and to show the relationships between them. In our study, strong positive correlations were observed between SJR and JIF and ES and HI pairs. Previous studies comparing hematology and clinical allergy and immunology journals observed similar correlations between scientometric indexes [10, 11]. In another study comparing the scientometric indexes of journals published in radiology, nuclear medicine, and medical imaging, correlations between scientometric indexes were higher than in our studies [18]. In addition, in a study examining journals in sleep science, the authors observed that the journal with the highest JIF was a review-only journal, with a JIF almost twice that of its closest competitor; however, when evaluated according to the HI, both journals shared the first place [4]. In a study conducted on anatomy and morphology journals, contrary to the literature, it was shown that the correlations between JIF, ES, and SJR were weak: the first journal according to JIF ranked third according to SJR and 20th according to ES [6]. Similar to our current study, the correlation coefficients of different scientometric indexes were strong in studies conducted for hematology and clinical allergy and immunology journals, although the ranking of journals in different scientometric indexes varied [10, 11]. Along with the available data, these results show the specificity of the scientometric indexes to the field. Our findings show that SJR and JIF and ES and HI have strong correlations that can be used interchangeably to evaluate rheumatology journals. As a preliminary criterion, Roldan-Valadez et al. previously proposed selecting journals in a population of JIF ≥ 1 journals using the SJR and ES scientometric indexes together and using all three indexes [1]. Therefore, it is evident that one should use more than one scientometric index. In light of these findings, when choosing a rheumatology journal, the JIF and HI pair or the ES paired with CS or SNIP can be used to express different rankings since they are based on different databases and use different calculation methods. It should not be forgotten that these results are specific to the discipline of rheumatology, and studies should be carried out for other fields of science.

Disseminating scientific research to a broader audience as quickly as possible has become a fundamental goal for scientific impacts in the digitalizing world. The OA publication model in medical journals is one of the consequences of this need. This emerging paradigm shift will cause changes in researcher behavior, from how authors evaluate journals to how funds required for scientific studies are allocated. Scientists have already begun to consider the OA publication model an important factor in their journal choices [19]. The support and financial contributions to the OA publication model have increased significantly over time [7]. Although some data support the claim that the OA publication model receives more citations than the subscription publication model, there is controversy in the literature on this issue. An article-based study showed that articles published in OA journals were cited at a rate of 1.3 times higher [20]. Similarly, OA publishing is one of the main factors increasing citations in psychiatry journals [8]. Contrary to these results, the OA publication policy does not influence citations in the field of ophthalmology [21]. In a study demonstrating correlations between scientometric indexes and the OA APC fee of journals published in the field of surgery, very weak correlations were observed between the scientometric indexes and the OA APC fee [22]. We have shown that most journals in the field of rheumatology use the hybrid OA publication model; however, the median ratio of OA articles in these journals is low (10.6%) but higher than both hematology (5.94%) and clinical allergy and immunology journals (3.26%) [10, 11]. Only six journals had the full OA publication policy, and four of these journals had a mandatory OA APC fee policy. Similar to the results of our previous studies on hematology and clinical allergy and immunology journals, no significant benefit was observed in the scientometric data in favor of the full OA publication model [10, 11]. Specifically, two journals implementing the free full OA model consistently ranked in the last two in all six scientometric indexes. This situation is open to scrutiny. These results contradict the literature reporting that the OA publication model causes an increase in the number of citations; however, this result may be discipline-specific.

When publishing in rheumatology journals, OA publishing is a crucial point to consider in the economic evaluation of journal selection because the median cost of OA publishing is 3300 USD. This cost can be prominent, especially for underdeveloped and developing countries with low per capita income, when the exchange rate difference between currencies is considered. Full OA journals have a median APC of 935 USD cheaper than hybrid OA journals. In hematology journals, this difference was similar, with a median of 900 USD [10]. This difference was much higher in clinical allergy and immunology journals (2895 USD) [11]; however, it is vital to make this decision by considering the scientometric indexes of journals that have a full OA publishing policy with hybrid OA. OA APC fees may also be subject to waivers or discounts, especially for manuscripts from underdeveloped or developing countries, and journals may choose to publish some articles free of charge at their discretion, which was not considered in our study. Among the scientometric indexes, only HI showed a difference in favor of hybrid OA journals, and others were similar.

The most important limitation of our study is that the available data are based on journals, not articles. In article-based studies on OA, the results may contradict this study, and future studies should focus on this. Another significant limitation is that bibliographic and scientometric data and publishing fees were obtained from databases. These results may not always correlate with real-life data.

Conclusion

Researchers should pay attention to how the index is calculated, what it represents, and whether the measure is discipline-specific when evaluating scientometric indexes. It is necessary to determine the advantages and disadvantages of data calculations and the extent to which they relate. According to our results, we recommend that the JIF and HI pair or the ES paired with CS or SNIP to be used together to evaluate rheumatology journals. We could not find any evidence that the OA model has a positive effect on the scientometric indexes of journals in rheumatology; our results contradict the literature reporting that the OA publication model causes an increase in the count of citations. These findings should be evaluated specifically for the rheumatology discipline, and further studies should focus on making these comparisons based on articles.