Introduction

Research evaluation is multi-dimensional. The impact of research is traditionally measured by the number of citations received by research or the impact factor of journals. Obviously, indicators such as research citations and the impact factor of journals can only evaluate the impact on the academic community and do not embrace the broader impact of research (Huang et al., 2022). In this regard, Nightingale & Scott (2007) state: “research that is highly cited or published in top journals may be good for the academic discipline but not for society.” (p. 547). On the other hand, governments are interested in recognizing the importance of research funded by public sector (1) for the private and public sectors (e.g., health care), (2) to address social challenges (such as climate change), and (3) for the educational and training purposes (Bornmann et al., 2016). Such standpoints have led to the emergence of a new research area in research impact assessment, which is referred to as societal impact of research. Societal impact is a term used to describe the effects of research beyond what is measured in the traditional research impact assessment system (Bornmann, 2013). Societal impact is used to demonstrate and measure the economic, social, environmental, political, cultural, organizational, and health effects of research (Joly et al., 2015). Nowadays, scientific research is often valued not only based on its academic impact (usually based on citation metrics) but also based on its societal impact. In the UK’s Research Excellence Framework (REF), which was developed to evaluate the quality of research in UK higher education institutions, research impact beyond academia is defined as the effect, change, or benefit to the economy, society, culture, policy or public services, health, environment, or quality of life, and it accounts for 25% of the overall assessment (Yu et al., 2023). There is an increasing interest in quantifying the societal impact of research, and in this regard, altmetric data has grabbed the attention of researchers (Bornmann, 2014; Haunschild & Bornmann, 2017; Thelwall et al., 2016). Altmetric is used to measure the broader impact of research based on its mention in online platforms and tools such as blogs and social networks, Wikipedia, policy documents, patents, etc. (Shrivastava & Mahajan, 2023).

Policy documents are one of the important data sources for examining the societal impact of research. The mention of research in policy documents, which implicitly involves the interaction of science and policy, is considered an important indicator of societal impact and the importance of research (Drongstrup et al., 2020; Zong et al., 2023). When policymakers mention research in policy documents, it also means evidence-based policy making (Huang et al., 2022).

Until recently, it was only possible to find out the citations of a policy document to research by manual examination. Since 2014, Altmetric.com has added a feature to its tool that allows tracking policy citations to research outputs (Lui, 2014). In addition, in 2019, Overton developed a large database of policy documents and their citations to research literature (Szomszor & Adie, 2022). With the advent of these databases, numerous studies have been conducted on the extent of citation to research in policy documents and the factors affecting it. In the same vein, the present study aims to investigate the policy citations of scientometric articles.

Scientometrics is one of the fields that contribution to policy has been frequently emphasized in its definition and scope. According to Tague-Sutcliffe (1992), scientometrics is a part of the Sociology of Science and has implications in science policy. Van Raan (1997) introduces scientometrics as a field that is focused on quantitative investigations of science and technology. Its objective is to enhance understanding of the progress of science and technology, particularly in connection to societal and policy inquiries. Hood & Wilson (2001) also states that scientometrics encompasses all aspects of quantitative science, scientific communication, and science policy. Mingers and Leydesdorff (2015) while mentioning the main topics of scientometrics, notice to the use of its indicators for research management and policy. Hicks and Isett (2020) state that many scientometric studies aim to be relevant to research policy, but these studies rarely have a considerable impact on research policy. However, four quantitative science studies that they selected for a case study had a significant impact on research policy. Contribution to science policy can be considered as one of the goals of scientometrics and its social function. Therefore, the presence of scientometric works in policy documents become important. As mentioned, altmetrics provides the possibility of examining the presence of scientific works in online platforms, including policy documents. Using this facility, the extent and quality of the impact of scientometrics on policy-making can be understood. To best of our knowledge, no research has been conducted on this topic previously. Accordingly, the aim of this study is to investigate the extent of the presence of scientometric articles in policy documents. Thus, it is possible to find out to what extent scientometric research has taken steps towards the goal of contributing to science policy. We are particularly interested in finding answers for following questions:

  • How much scientometric articles are cited in policy documents?

  • How is the subject distribution of the articles cited in policy documents?

  • What are the characteristics of the articles being cited in policy documents?

Literature review

After the emergence of Altmetric.com and Overton database, which provide the policy citations of academic articles, many studies have been conducted to examine the presence of articles in policy documents with the aim of measuring the broader impact of research. These studies are presented in three categories:

The impact of research on policy documents

Based on Altmetric.com data, Bornmann et al., (2016) found that only 1.2% (2341 out of 191,276) of climate change articles were cited in policy documents. In another study by Bornmann et al., (2022), based on 10,846 climate change policy documents in the Overton database, it was revealed that the cited articles in policy documents were published in high-impact journals, and climate change articles cited in policy documents received significantly more citations than the same articles not cited in these documents. In addition, climate change policy documents cited recent articles more than other policy documents, so that the average age of cited articles in climate change policy documents was 5.8, While the average age of cited articles in all policy documents was 6.7. Another finding of the study was that 4.98% of articles indexed in the Scopus were cited at least once in policy documents. Haunschild & Bornmann (2017) conducted a study on 11,254,636 articles indexed in different subject categories of Web of Science. Using Altmeric.com for collecting policy citations data, they showed that less than 0.5% of articles were cited at least once in policy documents. In a similar study, this time using Overton database, Fang et al., (2020) examined the presence of more than 18 million documents indexed in Web of Science between 2010 and 2019 in policy documents. They found that 3.9% of publications were cited at least once in policy documents, as older documents received more citations from policy documents and the social sciences and humanities were cited more than other subject areas in policy documents. Tattersall & Carroll (2018) analyzed research outputs from the University of Sheffield indexed in Altmetric.com. They found that out of 96,550 research outputs, 1463 were cited at least once by a policy document, meaning that 1.41% of the University of Sheffield’s research outputs in all fields were cited at least once by policy document. Yin et al., (2021) investigated the relationship between policy and science in the field of COVID-19. They used Overton database to collect policy citations data and found that the policy documents in the field of COVID-19 cited more recent, peer-reviewed, and high-impact articles. Lemke et al., (2022) using Altmetric.com did an altmetric analysis of medical research in Germany and found that out of 334,940 articles, 7838 (4.2%) articles received at least one citation from policy documents. Policy documents also cited high-impact journals more often in cancer-related journals.

The relationship between open access and policy citations

Taylor (2020) examined the difference in the coverage of open access and non-open access books in policy documents. He obtained policy citations data from Altmetric.com. The results showed that citations to open access books and book chapters in policy documents were significantly higher than citations to non-open access ones. Adie (2020) selected Nature Communications articles published between 2014 and 2015 as a sample and compared the citations of open access and non-open access articles in policy documents based on data obtained from Overton database. The results showed that the proportion of open access articles cited in policy documents (3.4%) was significantly higher than the proportion of non-open access articles (2.3%). Recently, Zong et al., (2023) investigated the impact of open access on citation in policy documents in Library & Information Science research outputs. In this research policy citations data were collected from Overton database. The results showed that open access had a positive and significant impact on citation in policy documents.

The relationship between collaboration and policy citations

Huang et al., (2022) studied the relationship between scientific collaborations in Library & Information Science research and policy impact. The research articles published between 2000 and 2018 in 86 journals of the category of Library & information Science were included in this study and policy citations data were obtained from Overton database. This study showed that the number of collaborating countries had a significant and positive relationship with policy citations, but the number of authors and the number of organizational affiliations of articles had no relationship with policy citations. In another study Xu & Zong (2023) using Overton database for collecting policy citations investigated the impact of international research collaboration on policy citations based on the Lancet articles published between 2000 and 2019. They found that international research collaboration increases policy citations of articles. They argue that when multiple countries are involved in a collaboration, researchers from different nations connect their work to diverse research communities, thereby expanding the community network and increasing research visibility. As a result, research articles with international collaboration are more likely to be noticed by policy-makers and cited in policy documents. Furthermore, international research collaboration, especially in subject areas like medicine, allows recruitment of global participants, ensuring a large and diverse population. This diversity enhances the generalizability and reliability of research findings. The unbiased and generalized nature of research can influence its adoption in policy-making.

Methods

The dataset used in this study includes 5525 scientometric articles indexed in Web of Science between 2013 and 2022. This time span was chosen for two reasons: first, while embracing considerable amount of scientometric articles, it helps to measure the impact of scientometric researches on policy based on recent publications. Second, the process of matching articles and policy documents is highly dependent on DOI. Since most of the recently published articles have DOI, the selection of this period almost eliminates the bias arising from the exclusion of articles without DOI. Data were collected in March 2024. To collect the scientometric articles, the method used by Khasseh et al., (2017) was used, which took place in several steps: first, all published articles between 2013 and 2022 in Scientometrics Journal and Journal of Informetrics were included in the study. In addition, articles from six journals including Journal of the Association for Information Science and TechnologyFootnote 1 (JASIST), Information Processing & Management, Journal of Documentation, Journal of Information Science, Research Evaluation, and Research Policy, which cited one of the articles of the Scientometrics Journal and Journal of Informetrics, were included in the dataset. For this purpose, the Cited References option of the Web of Science was used. In the Cited Title field, the name of the journal (once for Scientometrics Journal and once for Journal of Informatics) and in the Cited Year(s) field, the years 2013–2022 were entered. The results were limited to Article document type and were filtered using Publication Titles to include articles of 6 abovementioned journals. There were cases where some scientometric articles published in these journals had no citation to Scientometrics Journal or Journal of Informetrics. To retrieve such articles, the following search strategy was developed based on frequently used keywords extracted from previous works and entered in Web of Science advanced search box:

TI= (“informetric∗” OR “bibliometric∗” OR “scientometric∗” OR “webometric∗” OR “citation∗” OR “cite” OR “∗citation” OR “indicator∗” OR “productivity” OR “mapping" OR “h-index” OR “h index” OR “Hirsch index” OR “∗index” OR “coautho∗” OR “impact factor∗” OR “link analys∗” OR “link structure” OR “patent analys∗” OR “Zipf∗” OR “Bradford∗” OR “Lotka∗” OR “collaboration network∗” OR “scientific collaborat∗”) AND SO=(“Information Processing and Management” OR “Journal of Documentation” OR “Journal of Information Science” OR “Research Evaluation” OR “Research Policy” OR “Journal of the American Society for Information Science and Technology” OR “Journal of the Association for Information Science and Technology”) AND PY=(2013-2022).

The results of the above query also were limited to Article document type. Finally, after aggregating the retrieved articles in the above steps and removing duplicates, 5525 scientometric articles were identified and analyzed. As shown in Table 1, most of these articles were published in Scientometrics Journal, Journal of Informetrics, and JASIST. To collect policy citation data, the Overton database was used. Overton is an impressive resource that provides access to a vast collection of policy-related documents, parliamentary transcripts, government guidance, and think tank research. According to Maleki & Holmberg (2022), Overton database has significantly better coverage of policy documents than Altmetric.com.

Table 1 Frequency of scientometric articles in studied journals
Full size table

The bibliographic information for each article in the sample along with its policy citations was entered into Excel sheets. To measure policy citations, the three indices proposed by Haustein et al., (2015) were calculated: coverage, density, and intensity. Coverage refers to the percentage of publications with at least one citation from policy document. Density is the average number of citations received from policy documents by all publications (documents with policy citations and documents without policy citations). Intensity is the average number of citations received from policy documents by publications that have been cited at least once in policy documents. To assign the subject areas of the articles cited in policy documents, the classification proposed by Mansourian (2010) was used with some modifications. This classification divides scientometric studies into four main categories, which are: (1) the formulation of research policies and guidelines, (2) the study of scholarly communication and citation analysis, (3) quantitative and qualitative evaluation of scientific publications, and (4) measurement of research output, productivity and impact. Subsequently, under each category its important research areas are listed which leads to a total of 50 research areas of scientometrics. To determine the category of each article, its title, abstract, and if necessary, the full text of the article was studied in detail by all researchers and final decision on article category was made after researchers’ consensus. Finally, Mann–Whitney test was used to examine the difference between policy citations of open access and non-open access articles as well as single-authored and multi-authored articles.

Findings

Policy citations of scientometric articles

The policy citations of sample articles were presented in Table 2. As shown in Table 2, out of 5525 articles, 4604 articles had no citation in policy documents and 921 articles had at least one citation in policy documents. By dividing 921 by the total number of sample articles, the coverage rate was obtained, which was 16.67%. The total number of policy citations received by these 921 articles was 2739, which by dividing it by the total number of articles, the density rate of 0.5 was obtained. In other words, the articles in the dataset were cited on average 0.5 times in policy documents. Finally, by dividing the total number of citation (2739) by the number of articles that had at least one policy citation (921), the intensity rate of 2.97 was obtained. This means that articles that had one policy citation received an average of 2.97 citations from policy documents. The data in Table 2 also show that 2 articles received more than 40 policy citations. One of them entitled “R&D and non-linear productivity growth” is on the Research and Development subject area, authored by d’Artis Kancs and Boriss Siliverstovs, and published in Journal of Research Policy in 2016. Another article, entitled “What is the social impact of science and how can it be measured?” is Lutz Bornmann’s article on the Societal Impact of research, published in JASIST in 2013.

Table 2 Policy citations of scientometric articles
Full size table

Policy citations of articles were analyzed in terms of publication year. As shown in Fig. 1, policy citations of older articles are more than recent ones. In addition, higher proportion of articles published in earlier years were cited in policy documents. This seems to be reasonable when considering time lag for being cited in policy documents. Scientific articles need time to be cited in scientific documents as well as in policy documents. The citation pattern in scientific articles typically exhibits a noticeable peak approximately 2–4 years after the publication of the referenced paper (Bornmann et al., 2016). In a similar way, approximately half of the papers referenced in policy received their initial citations three years after their publication (Pinheiro et al., 2021).

Fig. 1
figure 1

Policy citations of scientometric articles in terms of publication year

Full size image

Analyzing the policy citations of articles in terms of publishing journals revealed that Scientometrics Journal had the most policy citations among the studied journals, as 524 articles of this journal received 1224 policy citations. This is partly due to the design of the study as all papers from Scientometrics Journal in the studied period were used. The total number of policy citations is mainly driven by the number of papers included in the study. However, Research Policy ranked first in terms of coverage (64.64%), density (3.65), and intensity (5.65). Research Evaluation ranked second in terms of coverage (33.33%), density (1.54), and intensity (4.61) (Table 3).

Table 3 Policy citations of articles in terms of publishing journal
Full size table

Scientometric articles being cited in policy documents were from different subject categories of the field. The results showed that the scientometric articles addressed topics such as national/international scholar collaborations, scholar productivity/scholar performance, funding and sponsorship, science & technology Assessment, and Science, Technology and Innovation (STI) Indicators received more citations from policy documents (Table 4).

Table 4 Frequency of subjects of articles being cited in policy documents (Top 20 subjects)
Full size table

Access profile of articles cited in policy documents

In this section, the policy citations of open access articles were compared with non-open access ones. As Table 5 shows, 23.99% of open access articles were cited in policy documents, while this figure for non-open access articles was 11.85%, indicating that open access articles had a higher coverage than non-open access articles in policy documents. The average citation in policy documents (density) for open access and non-open access articles was 0.77 and 0.32, respectively. Also, the average citation in policy documents for open access and non-open access articles that had at least one citation in policy documents (intensity) was 3.21 and 2.66, respectively. The Mann–Whitney test showed that the policy citations of open access and non-open access articles was significantly different. Therefore, open access articles were cited more frequently in policy documents than non-open access articles.

Table 5 Policy citations of open access and non-open access articles
Full size table

The policy citations of scientometric articles in terms of the number of authors

Only 151 (16.40%) out of 921 cited articles in policy documents were single-authored, while the rest of the articles (83.60%) had at least two authors. The coverage rate for single-authored and multi-authored articles were 15.01 and 17.04, respectively. Single-authored articles were cited on average 0.41 times, while multi-authored articles were cited on average 0.52 times in policy documents. The highest policy citations were for articles with three authors (Table 6). The policy citations of single-authored articles were compared with multi-authored articles using Mann–Whitney test. Given that the P-value of test was less than 0.17, it can be concluded that there was not a significant difference between policy citations of single-authored and multi-authored articles, that is multi-authored articles received policy citations as much as single-authored articles.

Table 6 Policy citations of scientometirc articles in terms of the number of authors
Full size table

Discussion

The results showed that 16.67% of studied scientometric articles had at least one citation in policy documents. This number was 1.2% for climate change articles (Bornmann et al., 2016), 4.98% for articles indexed in Scopus (Bornmann et al., 2022), 3.9% for more than 18 million documents indexed in Web of Science between 2010 and 2019 (Fang et al., 2020), 1.4% for research outputs of the University of Sheffield (Tattersall & Carroll, 2018), and 2.4% for medical research in Germany (Lemke et al., 2022). The coverage of scientometric articles in policy documents is fair compared to other fields. This is expected from a field whose main goal is to contribute science policy. However, the policy impact of scientometric articles could be greater, but it seems that there are obstacles on its way. Some of these obstacles are as follows: (1) the probable existing gap between researchers and science policymakers, that is a disconnect or disparity between these two groups. It highlights a lack of effective communication or collaboration in their efforts, (2) scientometric studies may be relevant to researchers but not to science policymakers, (3) Policymakers may not have seen the relevant articles or may not have given them the necessary references if they used them.

Like citation pattern in scientific articles, policy citations also have a bias towards older works. Policy documents cited older articles more than recent ones. This finding was consistent with Fang et al., (2020). Therefore, just as a time window is considered to receive academic citations, an appropriate time window should also be considered to measure the impact of research outputs on science policy.

According to the results, Scientometrics Journal ranked first in terms of the number of articles being cited in policy documents as well as the number of policy citations. However, Research Policy and Research Evaluation journals ranked first and second, respectively, in terms of coverage, density, and intensity indices. It should be noted that only scientometric articles from Research Policy and Research Evaluation were included in the present analysis, and the results may have been different if all their articles had been included in the research sample. These journals are among the high-impact journals and are usually ranked in first or second quartile of their respective subject category in Journal Citation Report. Previous studies by Yin et al., (2021), Bornmann et al., (2022), and Lemke et al., (2022) also found that articles cited in policy documents were mostly published in high-impact journals. The subject analysis of scientometric articles being cited in policy documents showed that articles on scientific collaborations, research productivity at different levels, funding and sponsorship, science & technology assessment, and Science, Technology and Innovation (STI) indicators received more attention from policy documents. On the other hand, Khasseh et al., (2017) found that topics such as citation analysis and theoretical foundations, scientific collaboration in universities, industry-university-government collaboration, and information retrieval and visualization were at the center of Information Science debates. The comparison between the topics of interest to policymakers and researchers shows some overlaps, but for greater impact on policymaking, it is necessary that the topics of interest to policymakers and researchers to be closer to each other.

The results showed that open access articles were cited more frequently in policy documents than non-open access articles. Furthermore, the policy citations of open access articles were significantly higher than non-open access articles. This finding is consistent with the results of Taylor (2020), Adie (2020), and Zong et al., (2023). They also found that open access articles were cited in policy documents more than non-open access articles. This can be explained by the fact that open access increases the visibility of articles and helps them receive more citations from other research (Ferreras-Fernández et al., 2015; Aguillo, 2020). Similarly, open access makes articles more visible to policymakers and increases the likelihood of being cited in policy documents. Finally, the results showed that most articles (83.60%) being cited in policy documents had at least two authors. Based on previous studies, multi-authored articles have a high research impact (Musa et al., 2021; Mrygold et al., 2021; Pohl, 2021; Campbell & Simberloff, 2022). According to the fact that policymakers tend to use the best scientific evidence in the policymaking process (Kay, 2011), and the best scientific evidence is usually those with high research impact (Regan & Henchion, 2019), it was expected that multi-authored articles have a chance to be cited more frequently in policy documents. However, the policy citations of multi-authored articles were not significantly higher than single-authored articles. No relationship has been reported between the number of authors and the policy citations in previous studies, but a significant relationship has been observed between the number of collaborating countries and policy citations (Huang et al., 2022) as well as between international research collaboration and policy citations (Xu & Zong, 2023).

Conclusion

The coverage of scientometric articles in policy documents was 16.67%. This number is considerably higher than the average coverage of Web of Science and Scopus documents in policy documents. Given that one of the main objectives of scientometrics is to contribute to science policy, this finding is not surprising. For a greater impact of scientometric articles on science policy, more interaction between scientometric researchers and policymakers is required. This article has practical implications for improving the impact of scientometric articles on policymaking. Researchers can consider publishing research in open access journals and pay more attention to topics of interest to policymakers that were revealed in this study to increase their impact on policy. Journal publishers can also develop green open access policies to increase the visibility and, consequently, the research and policy impact of articles. Research institutions and funding agencies can develop policies that encourage the publication of articles in open access journals. There were limitations to our study, some of which were specific to this research and some of which were common to all policy impact studies. This study focused on scientometric articles, and the results may not be applicable to other fields. Also, since no previous study had examined the citatons of scientometric articles in policy documents, comparing the research findings with previous studies was difficult for us. On the other hand, some policy documents that cited articles may not have been indexed by the tool used in this study. Also, policymakers may have used articles in policymaking but did not cite them. Finally, the process of matching articles and policy documents is highly dependent on DOI, which excludes articles without a DOI from research. This study began with scientometric articles and tracked them in policy documents using the Overton database. Future research may start with scientometric policy documents in the Overton database and analyze their content and citations. The result of such a study can help expand the findings of this study and increase our knowledge on the impact of scientometric research on policymaking.