Introduction

Citation-based metrics are commonly used as a proxy measure of the quality, prestige, and impact of scholarly journals, the articles published in them, and the researchers who author those articles. Publication outputs and their citations weigh heavily in international and national university rankings, which influence the public image of a higher education institution (HEI), and in turn its ability to attract students and staff (Robinson-Garcia et al., 2019). Where a HEI receives government funding, the amount received may be dependent on the ‘performance’ of its faculty, which again often takes into account publication- and citation-based metrics (Jonkers & Zacharewicz, 2016). As a result, some HEIs offer monetary rewards for publication in ‘top’ journals—those that garner more citations (Abritis & McCook, 2017). On an individual level, the number of articles a researcher publishes, the number of citations those publications attract, and the perception of the quality of that journal (generally determined by the number of citations its articles attract) all play a role in determining who gets hired, who gets promoted, and who receives tenure (McKiernan, 2019). This is despite an array of arguments against the unquestioned application of such metrics to measure research/er quality (e.g. Fire & Guestrin, 2019; Jain et al., 2021; Mañana-Rodríguez, 2015; McKiernan et al., 2019; Nicholas et al., 2017; Niles et al., 2020; Rice et al., 2020; Tian et al., 2016).

In a recent study, my co-authors and I compared the internationality of journals in the field of higher education that were in the highest quartile of citations (Q1)—those thought to be of the highest quality and afforded the highest level of prestige—with those in the lowest quartile of citations (Q4), according to the commonly used SCImago Journal & Country Rank (SJR) (Mason et al., 2021), which uses data from the Scopus database. The study concluded that Q1 journals were more likely to include studies from and about a small number of core anglophone countries. On the other hand, Q4 journals were “more likely to include research and researchers from outside of the core anglophone countries, making an important contribution to the diversity of scholarship beyond the dominating western and English-language discourse” (p. 1).

Notwithstanding the significance of that finding, an interesting observation was made during the data collection phase that was mentioned briefly in the paper. The observation was that one of the journals included in the corpus was ranked simultaneously as both a Q1 journal and a Q4 journal. The journal referred to is Christian Higher Education (CHE), published by Taylor & Francis. The journal falls under two subject categories: Education (under the broader subject area of Social Sciences), and Religious Studies (under the broader subject area of Arts and Humanities). In 2018, CHE was ranked Q1 in Religious Studies, but among journals in Education, it was ranked Q4. This makes the journal one that is likely considered to be ‘high prestige’ ‘high quality’ and ‘high impact’, and at the same time may be perceived to be ‘low prestige’, ‘low quality’ and ‘low impact’. Considering that the content of the journal is the same regardless of what subject category it is placed into, conflating a SJR indicator with quality (or impact or prestige) in the case of CHE appears nonsensical. Is it a journal of high quality, or limited quality? Would a paper in this journal be valued by HEIs, or not? Would it be looked at favorably by a tenure review board or a funding agency, or not? The answer to these questions cannot logically be both.

We conducted a thorough review of the literature to try to determine how common it was for journals to belong to different quartiles under the various disciplines in which they are categorized. We identified several studies that make reference to journals being attributed to more than one quartile, in relation to either the SJR, or the Journal Citation Report (JCR) indicator, a separate but also widely-used citation-based journal metric that uses data from the Web of Science database (Liu et al., 2016; Miranda & Garcia-Carpintero, 2019; Pajić, 2015; Vȋiu & Păunescu, 2021). In each of these papers attention is given to the methodological implications of this phenomenon, but detailed quantification of the extent or nature of this inconsistency across the database was not within their scope. This led us to conduct this small study in the spirit of curiosity, to determine if this was merely an anomaly, or if it was more common than we might expect. This led us to develop the following research questions (RQ), within the context of all journals in both the SJR and the JCR databases:

  • RQ1 How common is it for journals to be placed into multiple subject categories?

  • RQ2 How common is it for journals to fall into different quartiles?

  • RQ3 How common is it for journals to fall into non-adjacent quartiles?

  • RQ4 How common is it for journals to fall into quartiles that cross the ‘top’ and ‘bottom’ half of journals?

We believe the answers to these questions may be consequential, and it is surprising that they have yet to be answered in a systematic way, despite requiring relatively uncomplicated data collection, data cleaning, and descriptive analyses. Perhaps this is the very reason, as while it is of interest to us as curious researchers, it might not meet the narrow expectations of a full-length scholarly (publishable) paper. So, we welcome the opportunity to publish our findings as a ‘short communication’, a somewhat undervalued genre in academic journals, but an important channel to publish small but otherwise robust studies (Joaquin & Tan, 2021).

Journal classification and ranking

Scientific literature databases use various classification systems to categorize their entries in order to organize large volumes of publications, and to facilitate their search and retrieval. Some systems, such as the Chinese Library Classification, classify at the publication-level, although most classification systems are established at the journal level (Shu et al., 2019). The largest and most well-known international databases are Scopus and Web of Science, which have some important similarities. Both use a journal-level classification system, and are both multidisciplinary in nature, with a broad coverage across many research fields. Journals are classified into different subject categories based on their title, scope, or citation patterns (Gómez-Núñez et al., 2011). Though the processes for determining categorization are generally not made available in any detail, it may include a mix of manual and algorithmic approaches. While the actual categorizations used by SJR and JCR are different, they both adopt a hierarchical structure with a large number of lower-level subject areas organized under a smaller number of broader disciplinary themes, and both allow their various journals to potentially be given multiple classifications (Wang & Waltman, 2016). The journals within each lower-level category are divided into quartiles according to the number and weight of citations they have received in the previous two (JCR) or three-year period (SJR). Those in the top quartile are given a Q1 rating, those in the second highest quartile are given a Q2 ranking, and so on.

While in reality a journal quartile ranking indicates no more than relative citation attraction within a specific subject category to allow for the different definitions and conceptualizations of ‘quality’ in different disciplines and traditions, it is regularly conflated with broader journal quality, article quality, and researcher quality. In reference to JCR indicators, Vȋiu and Păunescu (2021) note that they have “become popular not only as a retrospective but also as an a priori impact assessment tool. Problematically, it is used to assess the distinct articles published in a journal and the merit of individual scholars” (p. 1496). While the evidence of this is pervasive, it is often subtle. For example, a quick online search found multiple cases of academic libraries listing bibliometric databases under ‘journal quality indicators’ and similar headings. Researchers are encouraged to publish in journals of a certain ‘standard’ often with reference to citation-based metrics. For example in Australia, institutional guidelines for researchers at Murdoch University (2021) state, “ideally, you want your articles to be published in journals that are ranked in either Q1 or Q2”. As a result, the pressure to publish in highly-cited journals is felt by researchers around the world and at all career stages (e.g. Fernández-Navas et al., 2020).

SCImago Journal & Country Rank (SJR)

In order to answer the research questions in relation to SJR, we began by downloading data for all publications within the freely available platform in mid 2021, with data current as at 2020. After excluding conference proceedings, book series, trade journals, and outlets without a calculated quartile ranking, a total of 24,978 journals were identified for analysis. As shown in Table 1, half of the journals identified were published in Northern Europe and Northern America. While one quarter of journals were published by either Elsevier, Springer, Wiley, or Taylor & Francis, the majority were published by one of thousands of smaller publishers.

Table 1 Characteristics of SJR journals, n = 24,978
Full size table

The almost 25 thousand journals included in the analysis were from across 308 of the possible 311 subject categories used by SJR at the time of data collection (this has increased to 313 at the time of writing). Missing only were journals from Dental Assisting, Respiratory Care, and Reviews and References (Medical). One-third of all journals (n = 7561) in the sample were designated to a single subject category, meaning that the majority were designated to more than one (Table 2). This ranged from two up to 13 different subject categories, with an average of 2.4 per journal. These subject categories are further organized under 27 broader subject areas, all of which were represented in the sample. Half of all journals crossed more than one broader subject area. Thus, in response to RQ1, it is common for journals to be categorized into more than one subject area, which includes crossing broad disciplinary boundaries.

Table 2 SJR journals according to number of subject categories and subject areas, n = 24,978
Full size table

Table 3 shows the breakdown of journals against the number of quartile rankings attributed to them. The majority of journals (n= 17,779, 71%) were ranked within a single quartile, of which just over half were assigned to one subject category (n = 10,218, 57%), and as such could only be assigned to one quartile. Around one quarter of all journals were ranked in two different quartiles (n = 6592, 26%), with the majority of those (n = 6264, 95%) being placed in adjacent quartiles. Just over 2% of all journals were ranked in three or more quartiles, including 17 journals that were ranked within all four, meaning that they were simultaneously ranked as Q1, Q2, Q3, and Q4 journals. In response to RQ2, around one-third of all journals (n = 7192, 29%) were assigned to more than one quartile, and therefore this is far from a rare occurrence.

Table 3 SJR Journals according to number of quartile rank indicators, n = 24,978
Full size table

Next we disaggregated the journal data according to the makeup of their quartile rank indicators, providing answers to RQ3 and RQ4. Looking at Table 4, we can see that a total of 935 journals (4%) were placed in non-adjacent quartiles. Further, over 3000 journals (n = 3214, 13%) crossed the central line between the ‘top’ and ‘bottom’ half of journals by citation, including 62 journals that are assigned at the same time within the highest quartile of citations (Q1), and the lowest quartile of citations (Q4).

Table 4 Breakdown of SJR journals, by quartile(s) attributed, n = 24,978
Full size table

Journal Citation Reports (JCR)

To answer the same questions in regards to JCR, the subscription-based platform was accessed in early 2022 and all journal data from 2020 were downloaded. Journals from the Social Science Citation Index (SSCI) and Science Citation Index Expanded (SCIE) were included, which both incur an impact factor and corresponding quartile ranking. 57 journals were removed that did not provide details of a quartile ranking. The platform provides much less detailed journal-level data than SJR, and as such we are unable to similarly report publisher details in this section.

The more than 12 thousand journals included in the analysis were from across all of the possible 254 categories used by JCR. Around half of all journals (n = 6709) in the sample were designated to a single subject category, with one third designated to two categories, and others designated from three up to six categories (Table 5). These categories are further organized under 21 broader disciplinary categories, that JCR refers to as ‘groups’ (referred to as ‘subject areas’ by SJR). The lower-level categories may fall under more than one of these broader groups. In this sample, a slight majority of all journals were placed into more than one broader group. As with SJR, it is common for journals to be categorized in the JCR into more than one category at both upper and lower levels.

Table 5 JCR journals according to number of categories and groups, n = 12,323
Full size table

The majority of JCR journals (n = 9867, 80%) were ranked within a single quartile, of which two thirds (n = 6709, 68%) were assigned to one subject category (Table 6). One-fifth of journals were placed in different quartiles across the different categories they were assigned, and Table 7 shows that 352 journals (3%) were placed simultaneously within non-adjacent quartiles, and 1131 (9%) crossed the line between the top and bottom half of cited journals.

Table 6 JCR Journals according to number of quartile rank indicators, n = 12,323
Full size table
Table 7 Breakdown of JCR journals, by quartile(s) attributed, n = 12,323
Full size table

Implications

Journals are often multidisciplinary in nature, and thus being assigned to more than one category is a necessity. Our investigation, which was born out of curiosity regarding what seemed to be a peculiarity in our data, found that it is not uncommon for journals to be placed into different quartiles across the different categories to which they may be assigned. This in itself is understandable, as there are many factors that influence how and why articles are cited (both valid and spurious). However, if citations are a valid measure of broader notions of research/er quality, then we would expect that journals would be ranked similarly across their different categories because the content remains the same. This is clearly not the case, with many journals in both SJR and JCR assigned to more than one quartile ranking. This includes journals assigned to divergent quartiles. We define such journals as those that cross the central line where journal quality (or lack thereof) is arbitrarily but often determined, as well as those that are (also) placed in non-adjacent quartiles, making the “rather arguable assumption that a couple of citations more or less is making a big difference” to research quality even more contentious (Pajić, 2015, p. 990). Such divergent cases involve 3214 (around 13%) of SJR journals, and 1131 (around 9%) of JCR journals, and the many more researchers who contribute to their articles.

For researchers still developing a career and identity, “publications in journals with a high impact factor and number of citations are seen as important signs of recognition”, and thus play an important role in accumulation of credibility (Hessels, et al., 2019, p. 135). A failure to publish in such journals may be manifested as misplaced feelings of inadequacy, but may also have tangible impacts on career progression. It may also impact the ways in which researchers engage with their work, leaning more toward scholarly publishing than other forms of research dissemination that may not be held in the same regard and for which there is less support (Merga & Mason, 2021). In a recent study, Niles et al. (2020) found that while researchers most value journal readership in deciding where to publish, they believe that their peers most value metrics. Broader discourse in scholarly publishing that centers citations as the main indicator of quality may result in engagement in practices that may not align with scholars’ own values, particularly for those in precarious employment. It may empower researchers to know that journals may be perceived as ‘top’ journals and ‘bottom’ journals with the same content, a clear challenge to the notion that citation indicators can accurately measure research/er quality.

There are some efforts in place at different levels to limit the role that citation metrics play in research/er evaluation. The San Francisco Declaration on Research Assessment (DORA) (Raff, 2013), and the Leiden Manifesto (Hicks et al., 2015), are both international initiatives aimed at challenging the inappropriate application of quantitative metrics, both providing a range of actions and recommendations to facilitate change. These international movements have led to the development of various national and institutional policies. For example, China has recently banned institutions from offering financial incentives for publication, a practice which has resulted in an increase in inappropriate publication and citation practices (Mallapaty, 2020). In an Australian example, an alliance between schools of computer science in four universities has published a position statement that begins:

This alliance advocates practical and robust approaches for evaluating research, aligned to those of DORA. Venue [e.g. journal] impact factors and rankings are not measures of the scientific quality nor impact of an article’s research. We strongly discourage inclusion of such rankings in job applications, promotion applications, and other career (-progression) and evaluation processes (Australian Computing Research Alliance, 2022, para. 3).

While the phenomenon of journals being ranked differently across subject categories is common, we believe that it may not be commonly known, or at least the implications have not been engaged with to any serious degree. Although the point of reference for citation-based metrics is, in principle, other journals within the same category, the reality in modern academia is that citations are regularly perceived to be a proxy measure of research quality and researcher success, and there are various systems and practices that enforce this. One way this is perpetuated is through the common practice of journals promoting their ‘best quartile’ on their homepages. Indeed, the SJR platform provides an automated function for journals to share their best quartile. We have yet to find a case where a journal promotes their indicators across the multiple subject categories that may make up the multidisciplinarity of their journal.

Studies in various disciplines have shown that there are many more factors at play than article quality when articles are cited (or not), including the race (e.g. Chakravartty et al., 2018), gender (e.g. Dion et al., 2018), and national affiliation of authors (e.g. Nielsen & Anderson, 2018). Studies published in English (e.g. Liang et al., 2013) and those that report significant results (e.g. Jannot et al., 2013) are more likely to be cited than otherwise, and certain types of studies are privileged over others (e.g. Mendoza, 2012; Slyder et al., 2011; Vanclay, 2013). Further, not all citations equate to a positive evaluation of the original source. Garfield (1964) identified 15 different reasons why authors might cite other works, which in addition to positive applications, also included negative applications such as to criticize previous work, to disclaim previous work, or to dispute specific claims.

It is important that citation metrics are shown and seen in ways that reflect their true function. Our study has shown that a journal may attract different rates of citation by researchers in different fields, resulting in cases where journals may be placed in different quartiles across their different categorizations. This is understandable and even to be expected if quartile rankings are used in a valid way, that is, to measure a journal’s ability to attract citations within a specific category. When defined in this way, it invites reflection on why some journals and articles attract more citations, and to “interrogate how just these [citation] practices are” (Citational Justice Collective et al., 2021, p. 360). However, and despite some counter efforts, the conflation of citation metrics with broader concepts of research/er quality continue. That a journal, with the same content, can be perceived to be both of high quality and low quality at the same time shows the illogicality of such conflation. This adds to the many existing arguments against the continued and ubiquitous application of citation-based metrics as a proxy measure of journal, article, and research quality, and their influence on individual researchers’ prospects for employment, promotion, funding and tenure.