1 Introduction

Originating from the concept of quantitative analysis of publications proposed by Otlet (1934), bibliometrics has experienced rapid development in the past half-century, particularly after the 1960s. Through the efforts of pioneers such as Eugene Garfield and the establishment of information systems and databases, bibliometrics has become an established academic practice. Furthermore, the advancement of computer visualization technology has enriched its form and essence as a research method, offering a more human-centric mode of presenting quantitative research data (McCormick et al., 1987). This has led to the emergence of an essential branch of bibliometric research known as bibliometric mapping (Van Eck & Waltman, 2010), which combines the bibliometric research method with visualization techniques.

Garfield (2006) highlighted that bibliometric mapping allows for the examination of a field’s history and structure, information flow within the field, the influence of journals, and the citation status of publications over an extended period. When combined with bibliometric mapping, it becomes possible to visualize the most productive authors, institutions, and countries in a specific discipline for further analysis, thereby discerning the trend of literature production over time. This plays a crucial role in outlining and providing an overview of fields. As an evolving discipline, educational technology necessitates introspection regarding its development process, an evaluation of its current state, and predictions for future trends. Consequently, bibliometric analysis, as a unique analytical method, has gained significant attention and has been frequently applied in the field of educational technology in recent years, becoming a prominent area of focus for many researchers (De Bellis, 2009; Hwang & Tu, 2021).

However, the application of emerging technologies in new disciplines often encounters challenges due to subjective biases and other factors. For instance, there may be an overemphasis on the application of certain software, leading to a failure to correctly interpret bibliometric indicators in studies and to conduct objective and comprehensive analyses of the research field. Therefore, it is necessary to critically evaluate and assess existing research on the application of bibliometric mapping technology in educational technology. By understanding the current state of research, we can assess the methodological system’s level of maturity, the organizational structure of the content, and the credibility of research conclusions in the field. Considering the limitations observed in prior studies involving the utilization of bibliometric mapping techniques and the absence of comprehensive reviews concerning these applied methodologies in the field of Educational Technology, this systematic review employs a methodology akin to that of previous meta-reviews (referred to as reviews of reviews) (i.e. Polanin et al., 2016; Tamim et al., 2011). Its primary objective is to function as a meta-review of existing bibliometric studies. The intention is to uncover deficiencies, discrepancies, disagreements, and incongruities within bibliometric studies, pinpoint fundamental elements that should be integrated into standard bibliometric investigations, and provide valuable insights to steer future development (Grant & Booth, 2009). Furthermore, it seeks to clarify the portion of the educational technology discipline that has already been analyzed using bibliometric analysis and the portion that remains unexplored. Based on this foundation, the present study primarily focuses on addressing the following research questions:

  • RQ1: What is the overall trend in the development of bibliometric research in the field of educational technology in terms of scientific productivity and performance analysis?

  • RQ2: Which topics are primarily subjected to quantitative analysis in bibliometric research in the field of educational technology?

  • RQ3: What are the normative and scientific challenges associated with conducting bibliometric research in the field of educational technology?

2 Contemporary bibliometric review practices in the field of educational technology

By examining the historical utilization of bibliometric analysis in the field of educational technology, one can comprehend the notable contribution this technique has made to the advancement of the field of educational technology. With the emergence of the bibliometric analysis technique, it has evolved into a conventional method for conducting a quantitative evaluation across various research fields. In 2014, Cheng et al. (2014) took the initiative by publishing bibliometric research in Educational Research Review. By examining 324 publications in the field of workplace e-learning spanning from 2000 to 2012, the study identified six primary research themes and four distinct dimensions within this research field. Following this, many scholars in the field of educational technology have increasingly emphasized bibliometric research methods (Valtonen et al., 2022), leading to a proliferation of research utilizing this approach for the analysis of educational technology. For example, Zawacki-Richter and Naidu (2016) employed this method to analyze a specific journal, Distance Education. They reviewed 515 publications within this journal from 1980 to 2014, revealing the patterns of thematic evolution and development trajectories of Distance Education over the past 35 years. This endeavor promoted the emergence of a new methodological perspective for reviewing the evolution of journals in the field of educational technology (Vošner et al., 2016; Lopez-Belmonte et al., 2021; Ozyurt & Ayaz, 2022). The analysis of technologies and emerging trends in the field of educational technology is a prominent focal area within bibliometrics. As an illustration, the visual analysis conducted by Hwang G.J.’s team on the utilization of AI technology across different fields offers a comprehensive overview of the amalgamation AI technology in education. This sheds light on the trajectory for subsequent developments (Hwang & Tu, 2021; Hwang et al., 2022). Some scholars have endeavored to perform a bibliometric analysis encompassing the entire realm of educational technology research (Valtonen et al., 2022), along with investigating popular subjects within educational technology (Kushairi & Ahmi, 2021; Liu et al., 2021), such as MOOCs and flipped classrooms. These endeavors yield unique insights into specific research topics.

From the above review, it becomes evident that bibliometric mapping has evolved into an indispensable approach for analyzing pertinent technologies and essential fields within the contemporary landscape of educational technology (Arici et al., 2019; Jing et al., 2023; López-Belmonte et al., 2020; Pei et al., 2021; Wang et al., 2022). “Understanding the historical and extant research progress, the development of technologies applied, and the drivers of fresh ideas” shows remarkable effectiveness (Chen et al., 2022a, p.29) due to the outstanding advantages of bibliometric mapping techniques. Compared with traditional review methods (such as scoping review and narrative review), bibliometric mapping has unique features. Firstly, it is suitable for dealing with large-scale data sets. At the same time, the method using manual coding technology is usually limited by the continuous growth of literature data (Lesnikowski et al., 2019). Amidst the rapid growth of educational technology and the increasing pace of literature publication, the systematic and comprehensive compilation of literature necessitates the management of substantial volumes of data. In addition, the results obtained through bibliometric analysis can be characterized by enhanced objectivity and reliability. Compared with the traditional content analysis method, the research strategy is typically pre-established, and the conceptual categories are predetermined (Lesnikowski et al., 2019). Conventional narrative reviews also rely on scholars’ subjective understanding of a research field. Given that educational technology is a rapidly evolving interdisciplinary files rooted is technical foundations, there is often a need for a more objective and rational approach to conducting summary analysis.

The application of bibliometrics within educational technology filed seeks to improve individual’s understanding of the progress in educational technology and capture the development frontier of a discipline that is constantly changing due to technological advancements (Chen et al., 2022a). At the same time, applied research further broadens the scope of bibliometric mapping and bibliometrics, fulfilling the above two purposes. This study attempts to systematically review relevant literature to present and analyze the overview and key points of research concerning bibliometrics and educational technology. Based on this, the paper offers profound insights that contribute to the advancement of this topic.

3 Research materials

This study utilizes the Systematic Literature Review (SLR) as the research methodology (Xiao & Watson, 2019). This choice is based on the acknowledgment by previous scholars of the value of SLR in analyzing research papers on specific topics (Kumar et al., 2022). The data retrieval and screening processes are conducted following the PRISMA Framework (Fig. 1) (Moher et al., 2009). The following sub-sections (i.e. 2.1—2.3) offer a comprehensive description of the literature identification, screening, eligibility, inclusion and coding processes.

Fig. 1
figure 1

PRISMA flowchart of the study

Full size image

3.1 Initial literature search (identification step)

To acquire the necessary analysis data for this study and ensure the inclusion of high-quality analytical studies, this study refers to the methods employed by Bozkurt (2019) and Chang et al. (2022). The study selects representative journals in the field that demonstrate high quality. The primary database utilized in this research is the Web of Science, which serves as a significant scientific literature indexing tool (Cobo et al., 2015). A total of 24 journals were reviewed, and their placement within the quartiles (quartile 1/quartile 2 slices) in JCR can be found in Table A1 in the appendix. All of these journals belong to the category of “Education and Educational Research,” which is the most commonly studied category in the field of education and frequently used by scholars for academic analysis (Orbay et al., 2021).

The search period for this study spans from January 2000 to December 2022. To ensure comprehensive and meticulous literature retrieval, the following strategies are employed: (1) The Field Tags “SO” are used to limit the search to the 24 selected journals, linked together with “OR”; (2) The topic search type is defined as TS = (“bibliometric*” OR “scientometric*” OR “scientific mapping” OR “science mapping” OR “visual analysis” OR “visualization” OR “knowledge map” OR “knowledge graph” OR “bibliographic” OR “authorship analysis”), focusing on literature containing relevant keywords; (3) The retrieval expressions from the aforementioned two parts are combined using “AND,” with language restricted to English and literature types limited to articles and reviews. After the initial search, a total of 784 articles were retrieved from the 24 selected journals.

3.2 Manual screening (screening and eligibility steps)

The retrieved literature is often a mixture of seemingly related but unrelated articles. Hence, to ensure the inclusion of literature closely aligned with the research topic, a manual screening process is necessary. To conduct this screening and maintain the quality of the included literature, the research team has established the screening criteria (Table 1).

Table 1 Inclusion criteria
Full size table

The manual screening process consists of two steps, i.e. Screening and Eligibility. In the Screening step, based on titles and abstracts, articles that are completely unrelated are excluded. This process involved the participation of three members of the research team and lasted approximately one week, resulting in the retention of 306 articles.

In the Eligibility step, the full texts of the retained literature were carefully examined to accurately identify the literature required for this study. Four members of the research team participated in this step, which spanned approximately three weeks. Ultimately, 71 articles were retained, and the list of them can be found in the appendix (List A1). These articles were sourced from 16 journals.

3.3 Analytical coding (included and coding step)

After completing the manual screening, two authors thoroughly examined the full text of the 71 articles that were retained and collected detailed information in five categories: literature metadata, literature methodology information, literature screening process information, literature performance analysis information, and other research details of the articles. The complete coding framework is detailed in Appendix (Table A2).

Given that the objective of this study is not to test research hypotheses but rather to enhance our understanding of the field by exploring various important aspects, key concepts were not predefined. Consequently, data was collected using an open coding approach, which is consistent with the methodology employed in previous studies (Shadiev & Li, 2022).

Following the design of the coding system, two authors independently conducted coding to ensure research reliability. Subsequently, two Excel files were created based on their individual coding. The consistency between the two codings was assessed using rater reliability, a commonly used measure in educational statistics and measurement. The analysis revealed that 66 documents had identical coding results (M is 66). By applying the average mutual agreement and reliability formula:

$$\begin{array}{l}\mathrm{Average\;mutual\;agreement}:\mathrm{ K}=\mathrm{M}/{\mathrm{N}}^{*}100\mathrm{\%}\\ \mathrm{Reliability}:\mathrm{ R}={\mathrm{n}}^{*}\mathrm{K}/(1+{\left(\mathrm{n}-1\right)}^{*}\mathrm{K})\end{array}$$

Using this formula, the K value was determined to be 92.96%, and the reliability was calculated to be 0.964, which is greater than 0.9. Therefore, the coding framework demonstrates high reliability (Gaur & Kumar, 2018).

4 Result and discussion

4.1 Performances analysis

Descriptive statistics of the fundamental information of existing research can provide insights into the overall landscape of bibliometric mapping in the field of educational technology. This study will analyze four aspects: periodical and time distribution, country distribution, main author groups, and other quantitative performance information.

4.1.1 Periodical distribution and time distribution of research on this topic

To gain an understanding of the general landscape of journals published articles on bibliometric research in the field of educational technology, the study organized journals related to the field. Table 2 presents the Journal Citation Indicator (JCI), which measures the impact of journals based on the average category standardized information of citation publications over the past three years. A JCI value of 1 indicates the average value for the field. All the journals included in the analysis have JCI values higher than the intermediate level of disciplines, indicating a stronger disciplinary influence. Hence, the inclusion of these studies ensures the quality of the research in this study.

Table 2 Journal-related information (ranked by JCI)
Full size table

By examining the publication trends of bibliometric research in the field of educational technology, we can gain insights into the macro-level trajectory of this topic (see Fig. 2.). Among the 71 articles included in the analysis, the earliest study dates back to 2014, focusing on the analysis of e-learning in the workplace (i.e. Cheng et al., 2014). In 2016, there was a modest increase in research activity as six related studies emerged. However, it was after 2019 that the topic experienced a significant surge. Scholars started to focus on this research topic, leading to a continuous growth of research articles.

Fig. 2
figure 2

Time trend of the publications

Full size image

The majority of publications on this topic were published between January 2021 and December 2022. Within the past two years, the published articles accounted for 63.4% of the total. This significant increase can be attributed to the gradual acceptance and recognition of bibliometric research methods by scholars in the field of educational technology (Hwang & Tu, 2021). Additionally, the newly developed journal, EAIT, has emerged as the primary publishing platform for this topic. It has played a crucial role in disseminating bibliometric research in the field of educational technology, thereby fostering the development and growth of this field to a considerable extent.

4.1.2 Country distribution of research on the topic

The country distribution of articles published on the topic is illustrated in Fig. 3. The analysis shows that 71 articles originate from 21 countries. Among them, China contributed the highest number of publications (n = 26), followed by Spain (n = 9), and Turkey (n = 8). These three countries account for a total of 43 articles, forming the core of research on this topic. Furthermore, the year-by-year variations depicted in Fig. 3. indicate an increasing number of nations engaged in this research area, reflecting a diverse and dynamic growth akin to the blossoming of a hundred flowers and the contest of a hundred schools of thought.

Fig. 3
figure 3

Distribution of articles across different countries over the years

Full size image

4.1.3 Leading author groups in the field

Through an analysis of the authors of the 71 articles, it has been observed that the bibliometric research in the field of educational technology is led by two core research groups and two sub-core research groups. Among them, the core research team, led by Hwang G.J., has published seven articles focusing on various topics such as the application of AI technology in education, Concept Map-Supported Education, and Peer Assessments. Additionally, the team has conducted quantitative research on concept mapping in computer-supported learning environments over a span of thirty years. They have also studied the evolution of Virtual Patients in Interactive Learning Environments using social network analysis, identifying core scientific productivity and leading journals in the research field.

Another core research team, led by Chen X.L., has published nine articles covering research areas such as innovative learning, learning analytics, technology application in classroom dialogue, and personalized language learning. The team commonly utilizes Structural Topic Modeling (STM) as a research tool for bibliometric analysis. STM, which is now an R language Package (Roberts et al., 2019), offers an unsupervised topic modeling approach. Each document is treated as a term, each topic as a temporal distribution, and each document as a collection of topics, allowing for a clearer visualization of the discipline structure within the research field.

In addition to the aforementioned two core research teams, two additional sub-core teams have made significant contributions to advance the research field. Among these, the team led by Chang C. Y. focuses on various learning environments and analyzes the development of Concept Mapping (CM) in computer-supported learning environments over the past thirty years (Chang & Yang, 2022; Chang et al., 2022). Furthermore, they employ social network analysis to examine the evolution trends of Virtual Patients (VP) in interactive learning environments over the past three decades. This analysis pinpoints key scientific contributions and leading journals within the VP research field.

The team led by Bozkurt A., on the other hand, primarily focuses on the research field of Distance Education (DE). They employ bibliometric mapping techniques to conduct detailed analyses of the trends and patterns of DE, finding that the theoretical basis of DE lies in social learning theories and that the research paradigm in the DE field underwent a transformation around the year 2000 due to the development of online network technology (Bozkurt, 2019; Bozkurt & Zawacki-Richter, 2021).

4.1.4 Additional quantitative performance information

In addition to the aforementioned information, this study also collects the following data from the 71 articles: the number of authors, open-source status of the literature, literature types, research period, number of samples included in the analysis, and number of cited articles. The analysis reveals that the average number of authors per document included in the study is 3.113. Out of the 71 articles, 23 are open-access, accounting for 32.4% of all the works included in the analysis. In terms of literature types, 52 works are categorized as articles, while the remaining 19 are reviews. This suggests that most scholars perceive bibliometric research as original thesis research, providing new knowledge and insights for specific research fields.

The research periods vary significantly among different studies. Guo et al. (2016) analyzed international cooperation in educational technology over a span of 50 years, from 1961 to 2012, while Chen et al. (2020) reviewed the journal BJET for 50 years, representing the two studies with the longest research periods among the 71 articles. In sharp contrast, Zhang et al. (2022) analyzed research conducted between January 2020 and August 2021, resulting in a research period of less than two years. However, on the whole, most studies fall within the range of 10 to 30 years.

As for the number of documents included, Valtonen et al. (2022) conducted the analysis with the highest number, studying a total of 30,632 publications. The present study comprehensively examines the developmental trends and evolution of educational technology over half a century, focusing on the most significant 66 publications in the field. Additionally, two studies incorporate over 10,000 articles (Guo et al., 2016; Schöbel et al., 2021). The study with the lowest number of included analyses is a quantitative analysis of Horizon Reports, which includes seven reports (Dubé & Wen, 2022). Overall, the average sample size of the studies included in the analysis is 1,847. However, due to the presence of a small number of studies with a substantial number of documents included in the research, the median holds more significance than the average. The median number of papers included in the calculation among the 71 articles is 555. The study also presents the number of documents and references included in the analysis (see Fig. A1 in the appendix). The analysis indicates that the average number of references cited in these documents is 79. The number of citations varies significantly across different journals. For instance, four articles published by ERR have an average of 186.5 cited articles, while four articles published by ET&S have an average citation of only 47 articles. In terms of distribution, the majority of cited articles in the 71 studies fall within the range of 40 to 120, accounting for 78.87% of the total research.

4.2 Content analysis

The content analysis of the 71 articles primarily focuses on four areas, ranging from macro to micro: research topics, research tools and database information, data screening process information, and research content statistics information.

4.2.1 Research topics

The bibliometric research on education encompasses core research topics that can be categorized into five main categories, with a few studies that do not fit into any specific category (Fig. 4).

Fig. 4
figure 4

Overview of core research topics

Full size image

The first category focuses on bibliometric analysis of specific journals in educational technology. There are nine studies in this category, including CHB, BJET (2 articles), C&E, AJET, JCAL, EAIT, DE, JRTE. The details can be found in Appendix Table A3. Among these nine articles, the study analyzing C&E journal has the largest research sample size (Sample size: 3,963). Furthermore, the analysis of BJET journal has the longest research period, spanning 50 years (Bond et al., 2019; Chen et al., 2020). BJET, as a prominent journal in educational technology, holds a rich history and is highly regarded.

The second category encompasses the bibliometric analysis of emerging technologies in education. These studies cover a wide range of technologies, including gamification-related technologies (Liu et al., 2020), the application of AR, VR, and other Extended Reality technologies (e.g., López-Belmonte et al., 2020), and the application and development of AI technology (e.g., Hwang et al., 2022). Some scholars have conducted in-depth research on technology-enhanced learning in higher education, identifying and analyzing five development trends and hot topics in each direction (Shen & Ho, 2020). Additionally, comprehensive technologies such as Multimodal Learning Analytics have been extensively analyzed (Pei et al., 2021). These studies provide valuable insights into emerging technologies in various educational domains, offering a clear framework for future research and guiding subsequent scholars.

The third category focuses on the quantitative analysis of learning environments in educational technology. This study encompasses a wide range of teaching and learning settings, including virtual laboratories (Raman et al., 2022), venue learning (Tang et al., 2022), Digital Learning Environments (Schöbel et al., 2021), Computer-supported Learning Environments (Chang & Yang, 2022), Interactive Learning Environments (Mostafa, 2022), and other virtual learning environments. As technology continues to empower education, changes in learning environments have become a general trend. The learning environment, as a fundamental aspect of teaching and learning, plays a crucial role in examining and promoting educational reform in the digital transformation era. The aforementioned research provides insights into different learning environments from the perspectives of research strength, topics, and evolutionary trends, facilitating the design and construction of learning environments at both theoretical and practical levels.

The fourth category pertains to literature on online and distance learning. With the proliferation of personal computers and the popularity of mobile devices, research in this area has expanded. It includes the analysis of online open-access resources (Zancanaro et al., 2015), the application of E-books in English learning (Chen et al., 2021a), e-learning (Tibaná-Herrera et al., 2018), social media used in education (Barrot, 2021), mobile learning (Koon, 2022), and Distance Education (Bozkurt & Zawacki-Richter, 2021), among others. The recent impact of the pandemic has further emphasized the significance of online learning, making it a crucial direction for educational technology research. For instance, Zhang et al. (2022) conducted a bibliometric mapping analysis of online learning in global higher education, highlighting the importance of employing innovative teaching methods in online teaching and learning.

The fifth category focuses on the quantitative analysis of subject concepts in educational technology. As an independent discipline, educational technology has introduced numerous unique concepts, such as MOOCs (Wahid et al., 2020), GBL (Game-Based Learning) (Liu et al., 2020), Flipped Classroom (Al Mamun et al., 2022), Computational Thinking (CT) (Ozyurt & Ozyurt, 2022), Technological Pedagogical And Content Knowledge (TPACK) Framework (Zou et al., 2022), and Information and Communications Technology (ICT) (Batanero et al., 2019). Analyzing these subject concepts quantitatively helps reveal the causal relationship between technology and changes in educational models. It also sheds light on the new requirements for learners and teachers when technology is integrated into education, such as the need to cultivate computational thinking and information literacy among learners and the need for teachers to possess TPACK-related knowledge and abilities to adapt to the evolving educational models driven by rapid technological advancements.

In addition to the above five research topics, there are a few studies that do not fit into these categories. These include the analysis of Horizon Reports (Dubé & Wen, 2022), the bibliometric analysis of Education 4.0 (Dao et al., 2022), and the exploration of mainstream conferences on educational technology (Chen et al., 2022b). Moreover, some macro-level studies on educational technology exist. For instance, Valtonen et al. (2022) summarized the nature and building blocks of educational technology, while Guo et al. (2016) analyzed the structure and evolution model of the international cooperation network in educational technology.

4.2.2 Analysis of highly cited articles

Shih et al. (2008) emphasized that highly cited literature generally signifies the works most acknowledged by scholars within a research field. Such literature often offer profound perspectives on key issues within the field, providing readers and researchers with valuable viewpoints, or attempting to anticipate the forthcoming evolution development of critical research questions. To identify highly cited literature, we conducted a citation analysis of 71 selected articles using the Web of Science Core Collection (WoSCC) database with citation counts as of August 6, 2023. WoSCC has been selected due to its reputable encompassments (Ding & Yang, 2022), enhancing the analytical significance of its information. We selected the top 5 articles based on their citation counts. Of these, 3 were published in 2016, while the remaining 2 were published in 2019 and 2020, respectively. Notably, the most cited article is the study by Heradio et al. (2016), which comprehensively analyzed the development and research status of virtual and remote laboratories from the perspectives of science mapping and performance analysis. Table 3 presents the relevant information and citation counts of the 5 highly cited studies.

Table 3 Highly cited articles
Full size table

However, among the 71 selected articles, a considerable proportion emerged in 2021 and 2022 (as indicated in Fig. 2), with 18 articles published in 2021 and a notable increase to 27 publications in 2022. While these studies hold valuable insights, their relatively recent publication dates have limited their citation counts. Nevertheless, these studies function as indicators of the prevailing research focal points within the field of educational technology. To capture the notable contributions from these recent publications, we examined and ranked the top 5 most cited articles from these two years, comprising a collective total of 45 publications. The results are presented in Table 4.

Table 4 Highly cited studies published in 2021 and 2022
Full size table

Analyzing the highly cited literature published from 2021 and 2022 reveals a notable focus on emerging disciplinary concepts within the educational technology field. These encompass areas such as digital competence, smart learning. Notably, a literature review on online learning during the pandemic garnered significant citations. This underscores scholars’ heightened attention to the unpredictable impact of pandemic factors on education and its transformative potential. Additionally, the bibliometric study conducted by Chen et al. (2022c) concerning the utilization of artificial intelligence in education has attracted considerable academic attention. In recent years, the advancing technological capabilities of AI have facilitated its practical implementation across various domains, and the field of educational research has undoubtedly reaped significant rewards from this evolution (Yan, 2023). The extensive citation of Chen et al.’s (2022c) study further indicates the scholarly need to focus on the possibilities and future trajectories of AI technology within educational context.

4.2.3 Research tools and research databases

The advancement of bibliometrics is closely tied to the progress of quantitative analysis technology, visualization technology, and database technology. Consequently, it imposes specific requirements on scholars to keep abreast of cutting-edge analysis technologies. In order to comprehend the technical development trajectory of bibliometric research, this study compiled data on research and analysis tools, as well as databases, employed in the field of educational technology. Figures 5 and 6 depict the utilization of bibliometric mapping tools and databases, respectively. It is important to note that, during the coding process of the articles, nine studies did not utilize any bibliometric mapping tools for their bibliometric analysis. These instances were excluded from Fig. 5. Furthermore, some studies utilized multiple bibliometric mapping tools (e.g., Tang et al., 2022), thus, a single article might be counted multiple times in different tool categories during the data analysis. The same principle applies to the statistics concerning databases, as several studies made use of multiple databases.

Fig. 5
figure 5

Evolution of bibliometric research tools

Full size image
Fig. 6
figure 6

Evolution of bibliometric research databases

Full size image

As depicted in Fig. 5, the utilization of analytical tools has demonstrated a noticeable trend of development. One prominent example is VOSviewer, an integrated tool for bibliometric analysis and visualization, which has garnered significant attention from researchers since its publication by Van Eck and Waltman (2010). Over the past decade, VOSviewer has undergone several editions and has emerged as the mainstream tool for bibliometric research (Pan et al., 2018). Additionally, the Bibliometrix package in the R language has gained popularity in recent years due to its flexibility and ease of adjustment. Researchers favor this rising star, and the open-source nature of R further facilitates its dissemination and adoption. It holds great promise for bibliometric statistical and visual analysis (Valtonen et al., 2022). In contrast, other analysis tools have struggled to keep pace with the advancements in research fields or have diversified characteristics. For instance, both Pajek and UCINET offer the advantage of easy operation, but their update speed lags behind that of the Bibliometrix package in Gephi and R languages (Donthu et al., 2021). While VOSviewer is user-friendly for creating visual maps, its limited adjustment space prevents the dragging and dropping of nodes within the map. On the other hand, CiteSpace and certain programming tools present greater challenges in operation, but they allow for the creation of more aesthetically appealing and personalized maps. Researchers must consider the strengths and weaknesses of each bibliometric visualization tool to select the most suitable analysis tool for their research objectives.

During the statistical analysis of research tools, the research team observed that CiteSpace, a visual econometric analysis software, was utilized by only a few scholars. As an open-source software, it offers abundant functionalities and high integration. It provides unique temporal and timeline diagrams (Pan et al., 2018) to illustrate the developmental trajectory and evolution of a discipline. Moreover, it offers diverse geospatial analyses (Pessin et al., 2022), making it an intriguing option for scholars in the field of educational technology.

Like the bibliometric atlas drawing tool, databases also exhibit their distinctive characteristics. Figure 6 illustrates that in recent years, two mainstream databases, namely Web of Science (WoS) and Scopus, have emerged as the leading choices, particularly the WoS database. A total of 51 studies have utilized this database, followed by Scopus, which has been employed in 30 studies. Compared to these two databases, others are generally considered only when the research subject is noteworthy. For instance, the bibliometric research of the engineering technology discipline may consider IEEE Xplore, while ERIC is relevant for bibliometric research in the education discipline.

Both WoS and Scopus possess their respective advantages. WoS is renowned for its quality and has long been regarded as the most authoritative indexing tool for scientific literature (Chen et al., 2018). It holds a crucial position as a bibliometric database (Cobo et al., 2015). However, in terms of comprehensiveness, Scopus surpasses WoS and boasts the largest collection of peer-reviewed journals (Norris & Oppenheim, 2007). Hallinger (2020) also emphasized that Scopus offers greater comprehensiveness in terms of the number of publications and the breadth of journals within the social sciences. Additionally, as this study focuses on bibliometric research in educational technology, the ERIC database, closely related to the field of academic research, is frequently included. Figure 6 demonstrates that scholars have continued to utilize this database in recent years for educational technology research.

4.2.4 Data screening process specification

In bibliometric research, ensuring the scientific rigor of the data acquisition process is essential as it directly influences the reliability of research conclusions (Cobo et al., 2011). Moreover, assessing the quality of bibliometric research necessitates transparency in the data screening process. This process involves four primary elements: the document retrieval style, retrieval framework, inclusion and exclusion criteria for documents, and the manual screening process for data cleaning. In the present study, a statistical analysis was performed on these four elements across 71 articles.

The analysis of Fig. 7 reveals that most of the articles have provided information regarding the retrieval methods employed to acquire the analysis materials. Out of the analyzed articles, 48 explicitly mentioned the inclusion and exclusion criteria for document screening. However, less than half of the literature sources provided a description of the manual screening process. Moreover, a mere 26 articles utilized the widely recognized PRISMA framework for literature retrieval and screening, indicating the importance of improving adherence to standardized practices in this research field.

Fig. 7
figure 7

Statistics on the normative elements of the data screening process

Full size image

The implementation of a scientific literature retrieval and screening framework is crucial for ensuring rigorous and methodical bibliometric research. Currently, since there is no universally accepted retrieval framework in bibliometric research, scholars often adopt a modified version of the PRISMA framework proposed by Moher et al. (2009). Additionally, bibliometric research can benefit from the SPAR-4-SLR protocol introduced by Paul et al. (2021) and other methodologies used in systematic literature reviews. Notably, the data screening framework plays a significant role in standardizing research data, as observed in the studies conducted by Hwang and Tu (2021) and Liu et al. (2020). Therefore, it is imperative for bibliometric research in the field of educational technology to actively incorporate and apply retrieval and screening frameworks from established scientific systems, ultimately enhancing repeatability and standardization. Moreover, the development of a specialized data retrieval and screening framework within bibliometrics is an important area of focus for future research.

Out of the 71 articles analyzed, all 14 studies included in the evaluation encompassed the four data screening elements mentioned earlier. Our research team conducted a comprehensive examination of the data retrieval and screening processes in these 14 studies, with a particular focus on assessing the reproducibility of the provided information. As a result of our evaluation, we identified six articles that hold significant reference value in this regard (see Table A4 in the appendix).

The study also examines the literature categories of 71 articles. Except for the research conducted by Dubé and Wen (2022), which utilized Horizon Reports, the remaining studies obtained their analysis materials from scientific databases. It is worth noting that 24 articles did not explicitly specify the specific categories of the analytical materials. Among the remaining 46 articles, the majority of the analyzed materials consisted of journal papers (e.g., Chen et al., 2021b), while a few studies included conference papers (e.g., Huang et al., 2023). The choice of literature for analysis in bibliometric research does not have a definitive answer and varies among scholars. Some argue that selecting peer-reviewed journal papers ensures the inclusion of high-quality research, although it may sacrifice comprehensiveness. However, the inclusion of literature types need to be determined based on the specific research discipline. For example, certain disciplines may require the inclusion of conference papers, books, and other literary forms to account for their relatively short development time and ensure an adequate sample size. Additionally, certain computer research fields often include conference papers due to their equivalent authority to journal publications. Ultimately, the decision regarding the types of documents to include involves a trade-off between the quality and comprehensiveness of the analysis.

4.2.5 Statistical information of research content

In bibliometric research, as defined by Garfield (2006), a systematic analysis typically involves presenting and analyzing the historical development, structural characteristics, and core productivity of the field. Additionally, the influence of journals and the long-term citation impact of publications are also examined. In the present study, a statistical analysis was performed on the performance and other content-related aspects of 71 articles, focusing on eight key analysis points.

As depicted in Fig. 8, most studies in performance analysis focus on quantitative descriptions and analyses of authors, journals, and countries. However, there is a noticeable dearth of research on the quantitative analysis of institutions. Performance analysis serves as a vital tool for quantitatively depicting the development of a discipline or research field, providing insights into the distribution of core scientific productivity within a specific domain. Therefore, it is imperative for comprehensive bibliometric research to include a statement and a concise analysis of performance analysis. This component is an essential piece of the puzzle in presenting a comprehensive overview of a specific research field.

Fig. 8
figure 8

Statistics on the main points of performance analysis

Full size image

Among the various analysis points, keyword co-occurrence analysis comprises the largest proportion, with 59.2% of the studies conducting this type of analysis. On the other hand, evolutionary analysis constitutes a relatively smaller portion, with only 39.4% of the studies focusing on this aspect. Both keyword co-occurrence analysis and evolutionary analysis are integral components of bibliometric analysis. Keyword co-occurrence analysis aims to outline the existing structure of disciplines, while evolutionary analysis seeks to depict the logical development and changes within research domains. Together, they provide a comprehensive framework for understanding the structure and development of a discipline. In addition to these two points, cluster analysis and co-citation analysis are also crucial analysis elements in bibliometric research. Cluster analysis often reveals the primary research topics within a discipline, while co-citation analysis sheds light on the discipline’s structure and the path of knowledge development (Small, 1999). Among 71 articles, nearly half of them conducted cluster analysis and co-citation analysis on topics or keywords (Fig. 9).

Fig. 9
figure 9

Statistics of other analysis elements

Full size image

It is important to note that not all aspects of these four research points related to the internal details of the discipline need to be covered in every study. Researchers need to select the aspects they wish to analyze based on their research questions and combine them with performance analysis points to convey the essence of their research.

5 Conclusions, implications and prospects

5.1 Conclusions

This study is a systematic review of bibliometric research in the field of educational technology to describe its core scientific productivity and overall trend (RQ1). With increased popularity of bibliometric research in recent years, our review offers a unique perspective for observing and evaluating the evolution of bibliometric research in the field of educational technology. The analysis of core productivity reveals that China, Spain, and Turkey are the leading publishing countries, contributing over 60% of the research in this field. Collaborative efforts among core author groups, such as those leading by Hwang G.J., Chen X.L., and other scholars, have resulted in the acceptance and publication of numerous high-quality studies in reputable journals, notably EAIT and ILE. Moreover, the contributions of IRRODL and C&E have been instrumental in advancing the field towards standardization, scientific rigor, and institutionalization.

Furthermore, this study addresses five major research topics (RQ2) in bibliometric research, including the analysis of specific journals, emerging technologies, learning environments, online learning, and distance learning, as well as subject concepts in the field of educational technology. These studies provide comprehensive insights from a macro perspective. They contribute to reviewing the historical development, promoting the integration of technology in education, deepening our understanding of the subject, and constructing teaching models in the context of technological interventions. Overall, these studies shed light on the rich and extensive history of technology-enhanced teaching and learning, providing a clear picture and framework for the development of the discipline. Additionally, they present cutting-edge perspectives on the topics under investigation and offer valuable recommendations for future research (e.g., Arici et al., 2019). By tracing the development logic from the past to the present and projecting into the future, they establish a coherent narrative centered around technology-supported teaching and learning.

The study also examines the standardization and scientific rigor of the tools, databases, and research methods employed in the field (RQ3). VOSviewer and Bibliometrix emerge as the most frequently used bibliometric visualization tools, while Web of Science and Scopus are widely recognized comprehensive databases. However, the selection of visualization tools and databases is a complex task that depends on the specific research topic. To assess the standardization of research in this field, particular attention is given to the data screening process. By analyzing the data screening procedures of 71 articles, this study offers enlightening suggestions for screening frameworks, including literature types and other dimensions. Additionally, it identifies six highly standardized studies in the data screening process. It is crucial to emphasize that academic research places great emphasis on the reproducibility of analyses. Therefore, the systematicity and standardization of a study can be determined by the provision of clear and replicable explanations in the methodology and data retrieval processes, which aligns with the core principles of the PRISMA framework.

5.2 Implications

This study provides valuable insights into three research inquiries related to the focal research topic. Firstly, an analysis of scientific productivity offers a clear overview of the historical and contemporary landscape of the topic, which holds significant importance for scholars, especially those exploring interdisciplinary domains (RQ1). Secondly, an examination of research themes within the field of educational technology sheds light on areas that have yet to undergo comprehensive bibliometric scrutiny. This analysis informs future researchers about unexplored research content in educational technology, highlighting gaps in the complete landscape of the discipline that await investigation using bibliometric techniques (RQ2). Furthermore, the study conducts a statistical assessment of the technologies and databases utilized in current research on the focal topic. Through a critical appraisal of methodological rigor, the study identifies exemplary articles that demonstrate noteworthy structural attributes (refer to Appendix Table A4), providing standards and exemplars for subsequent researchers in the field of educational technology who wish to employ bibliometric techniques in their own investigations (RQ3).

5.3 Limitations of current reviews

This study not only tackles the aforementioned research inquiries but also presents forward-looking questions and critical viewpoints regarding the current development of the topic.

Firstly, it is paramount for bibliometric research to possess a profound understanding of the discipline. While it adopts a quantitative research model, it still should capture the language system of the discipline and delve into its development, providing insights into its future trajectory. Presently, some studies lack a comprehensive conceptual framework (e.g. the study conducted by Veletsianos and Shepherdson (2015) doesn’t provide any rigorous and scientifically grounded literature screening guidelines) but rely solely on the accumulation of quantitative information and software applications to outline the development trend and structure of the research field (e.g. the study conducted by Mostafa (2022)). Such an approach falls short of fully harnessing the potential of bibliometric research in the field of educational technology.

Secondly, there are notable gaps in the understanding of the forefront of bibliometrics in the field of educational technology. For instance, there is limited utilization of scientific indicators (e.g., h-index, g-index) that are seldom mentioned in our statistical analysis to evaluate scientific productivity within the subject area. Similarly, when analyzing the subject structure, few scholars explore item analysis as an alternative to keyword analysis, which could provide a more comprehensive understanding of the subject.

Additionally, very few scholars consider the value of the quantitative laws of bibliometrics, such as Bradford’s Law, Lotka’s Law, and Zipf’s Law when examining the contents of selected literature. Consequently, researchers need to actively grasp the development trends and cultivate a correct understanding of bibliometrics to enhance their research endeavors.

5.4 Recommendations for best practices

In order to enhance the rigor and transparency of future bibliometric mapping studies, we strongly advocate adhering to the PRISMA and other well-known reporting guidelines (example.g., SPAR-4-SLR protocol proposed in the study conducted by Paul et al. (2021)). By adhering to these established guidelines, researchers can guarantee a systematic and comprehensive approach to conducting their reviews, leading to more accurate and reliable results. It is important to note that throughout our review of existing studies, we identified instances where the “Search” and “Study Selection” sections of the PRISMA guidelines (Moher et al., 2009) were not adhered to, alongside the lack of a standard four-phase flow diagram. This might have affected the quality and credibility of the findings presented in those studies. Therefore, we recommend researchers in the field of bibliometric mapping to carefully adhere to the PRISMA guidelines in their future work in order to maintain consistency and standardization in the research process. By doing so, researchers can contribute to the advancement of our knowledge in this domain and promote the credibility of bibliometric mapping studies as a whole.

To better serve the field, we encourage researchers to employ bibliometric mapping techniques to explore new technologies (e.g., AI-generated content exemplified by ChatGPT) and emerging concepts (e.g. the Metaverse) within the educational technology filed. Furthermore, conducting comparative analyses of bibliometric studies across diverse research fields will yield valuable insights and promote interdisciplinary knowledge exchange. For instance, juxtaposing bibliometric research in the field of educational technology with studies in scientometrics could offer insights into proper calculation and meaningful utilization of metrics. Similarly, comparing bibliometric research in educational technology with that in medical education may provide us with insights on the scientific application of the PRISMA and other established guidelines for literature screening.

By implementing these best practices (refer to Appendix Table A4), future studies can contribute to a more robust and reliable body of knowledge in educational technology research. We believe that these recommendations will enable researchers to avoid common pitfalls and advance the understanding and impact of bibliometric mapping techniques in this field.

5.5 Limitations and future research

The present study, like any other, is not without its limitations. One limitation is the narrow scope of the review, which primarily focuses on a selection of prominent journals, thereby potentially overlooking significant contributions from other sources. For instance, this study regrettably omits an analysis of bibliometric reviews pertaining to emerging and cutting-edge concepts such as the Metaverse in the field of education, despite the existence of relevant research published in certain journals (Tlili et al., 2022). Moreover, this study concentrates solely on bibliometric research within the educational technology field, neglecting bibliometric research exploration in other disciplines. Consequently, the merits of 71 articles under investigation cannot be fully assessed through comparative analysis. Furthermore, the category of “Education and Educational Research” may prove inadequate for article selection, necessitating the inclusion of supplementary categories such as “psychology,” “educational sciences,” “education in scientific disciplines,” “specialized education,” “computer science interdisciplinary applications,” and more). In future studies, enhancing the scope for selecting articles becomes crucial to ensure comprehensive coverage and relevance.

Besides, establishing more robust inclusion and exclusion criteria, as well as criteria for evaluating the quality of literature, will help mitigate potential declines in literature quality resulting from the expanded analysis scope. Additionally, future studies need to endeavor to compare the application of bibliometric techniques across various research fields. By engaging in comparative analysis, more valuable insights can be derived, facilitating the rational and orderly advancement of bibliometric research in the field of educational technology. Recognizing these limitations and outlining the steps for future research will enhance the overall credibility and applicability of bibliometric studies in the field of educational technology and contribute to the continuous refinement and improvement of research methodologies.