Keywords

1 Introduction

Information literacy (IL) has been one of the most extensively researched concepts within Library and Information Science (LIS), understood and interrogated as a skill, competence, social practice, and phenomenon. IL has been investigated through a range of research methods; nevertheless, these have not themselves figured prominently as a research focus of IL studies. On the other hand, the numerous bibliometric studies that have been conducted to map the field indicate the overwhelming use of quantitative research methods [1, 2]. However, these mapping studies, especially until more recently, have not focused on the content of IL research but on publication or citation characteristics and patterns, and have restricted analysis to a brief time span and a narrow range of LIS databases [3].

A relatively popular research method, introduced in IL research by Christina Doyle [4] to develop a comprehensive IL definition and competence outcomes, is the Delphi method. The Delphi method was developed in the United States in the early 1950s by Norman Dalkey, Olaf Helmer, Ted Gordon and associates in “Project Delphi,” an Air Force-sponsored RAND Corporation study to elicit expert opinions to predict the effect of technology on warfare and prepare for national security threats in the context of the Cold War. Since this was classified defence research, it was released only a decade later when Gordon and Helmer introduced Delphi to the research community as a new method [5]. Since then, the Delphi method has evolved and been used in a variety of fields and disciplines, and today a number of its types and variants exist [6].

The Delphi method has grown in popularity especially since the publication of a seminal methodological work by Linstone and Turoff in 1975 [7]. They defined it as “a method for structuring a group communication process, so that the process is effective in allowing a group of individuals, as a whole, to deal with a complex problem.” [5, p. 3] Applied in quantitative or a mixed-method approach, but predominantly in a qualitative fashion, Delphi is inductive and data-driven, often used in exploratory studies on specific research topics or questions for which limited or no empirical evidence exists [8]. It is especially useful when the analysed problem can benefit from subjective judgments on a collective basis [5], to aggregate varied individual opinions [9], and to reach consensus among experts on a topic where information sought is subjective [10].

A key factor in the success of the Delphi method is the selection of experts for the Delphi panel(s). Delphi requires qualified specialists, who either have deep understanding of the problem at hand [8], or who represent the key aspects of the relevant issue [11], and who are not necessarily ‘experts’ but who have an insider’s perspective and the most intimate knowledge of and experience with the issue [12]. Although Linstone and Turoff recommended a minimum of 10 and a maximum of 50 experts [5, p. 86], there is no strict rule on this; indeed, Delphi can be successfully executed with higher and lower numbers of experts provided they are carefully selected.

Delphi typically works through a series of ‘rounds’ or ‘iterations’ of mostly asynchronous, anonymous surveys (questionnaires) with controlled feedback which allow forecasting, issue identification, prioritization, problem solving, and decision making. The number of rounds also varies, with two considered the minimum and three the most effective, although neither is there a strict rule on this aspect [5, 13].

This flexibility and versatility make Delphi “particularly well suited to new research areas and exploratory studies” [8, p. 27] and, if carefully designed and properly and rigorously executed, requiring also a justification of the responses provided by the experts, Delphi can contribute to both theory and practice [8]. Therefore, it is well suited to IL research, including potentially to information (literacy) experience, especially if complemented with other methods, since it allows a qualitative exploration of subjective judgments and individual experiences (cf. [14, 15]). This is in line with its underlying philosophy that “truth may be experiential and not just based upon prior assumption.” [16, p. 69].

Recently, several studies have explored the use of Delphi in LIS [17,18,19] and the key features of the method as applied to LIS research [15]; however, they have not focused on its application to IL research. Therefore, this research aims to develop a critical understanding of how IL research is shaped, operationalized and executed by means of the Delphi method and its current state of usage in IL research. It is hoped the research may serve as a useful base for IL theory and practice, providing guidance for future IL research, both content and methodology-wise, but also for encouraging expansion in the use of the Delphi method.

2 Research

2.1 Research Questions

This research is exploratory in nature and seeks to provide a descriptive insight into the key features of IL research which has utilized the Delphi method as well as the key features of the method itself as used in IL research. Specifically, it is guided by the following research questions:

  • What IL issues and which research contexts are studied using the Delphi method?

  • Which types of the Delphi method have been utilized in IL research?

  • What is the number of rounds and of experts in each round of IL Delphi studies?

  • What is the main population of IL Delphi studies and how have experts been selected?

  • What are the most common types of findings of IL Delphi studies?

These research questions, treated as analytical categories, have been formulated mainly on the basis of methodological literature on the Delphi method and a previous study of a similar kind [15], but extending the scope to more than one database and focusing on the usage of Delphi in IL research.

2.2 Method

A systematic literature review with elements of a critical literature review [20] was the principal research method. A review of IL research studies utilizing Delphi was undertaken in April and December 2022, using studies retrieved from five databases: Web of Science; Scopus; Library and Information Science Source; ProQuest Library and Information Science Collection; and Library, Information Science and Technology Abstracts. The databases were queried for: “information literacy” AND research AND (Delphi OR “Delphi method” OR “Delphi study” OR “Delphi technique”).

A literature search performed on the basis of pre-defined inclusion criteria (peer-reviewed academic journal articles written in English, indexed in the above databases in the field of LIS, and the search terms appearing in the title, abstract and/or keywords), with a non-defined temporal span, retrieved a total of 799 articles. The final number of articles for analysis was determined with reference to clear exclusion criteria; that is, after the elimination of duplicates, articles whose full text could not be accessed, or those wrongly categorized or indexed. Articles were also vetted for relevance: articles were excluded which only mention Delphi but do not actually use it as a research method or those which report the usage of Delphi but not in the field of LIS and not in IL research. This resulted in a total of 38 individual articles identified as relevant for analysis and this research. A critical literature review with elements of qualitative content analysis and descriptive statistical analysis was used in relation to the research questions (analytical categories).

3 Findings and Discussion

The literature review of IL research utilizing the Delphi method indicated that Delphi was not a common research method for IL studies: only about two studies per year have been published on average since the first use of the method in IL research, with a few peaks of increased use since 2010, as shown in Fig. 1.

Fig. 1.
figure 1

Trend in the publication of IL research articles utilizing the Delphi method

Full size image

This is consistent with the prior research of LIS Delphi studies [15] which showed a significant increase of studies since 2000 (when electronic and online variants of Delphi emerged and eased the application of the method), also averaging two studies per year since the inception of the method.

3.1 IL Issues and Research Contexts Studied Using the Delphi Method

While it was found that Delphi was not a common method for IL research, it was used to study various research issues and in a variety of research contexts (Table 1).

IL research which used Delphi most commonly investigated the issues of digital and computer literacy and digital competences (N = 9; 23.7%). This includes, for instance, research by Louise Hamilton et al. [21] who examined the role of digital and information literacy in the context of information management knowledge transfer activities in the occupational therapy profession. The other most frequent research question was health (information) literacy and the related issues of competences and practices (N = 8; 21%), for example Lukenbill et al.’s [22] research on the role of school and public librarians in improving health IL in their communities and increasing the dissemination of health information. Other frequently studied issues were IL (competence) standards, models and threshold concepts (N = 5; 13%); this was the case in the research carried out by Pinto et al. [23] who explored the application of IL standards and models in an Ibero-American context. Delphi was also used to study the issue of information behaviour (N = 4; 10.5%), including Poirier and Robinson’s [18] research into the application of “Slow principles” in information behaviour research and practices which also introduced the unique “Slow Delphi” variant.

Table 1. Main research issues in IL Delphi studies
Full size table

Table 2 shows that research was predominantly conducted in the context of education (N = 13; 34.2%); an example is a study by Secker and Coonan [24] to design a new curriculum for teaching IL in higher education. An equal number of studies was conducted in the context of higher education (N = 6; 15.8%) and primary or secondary education (N = 6; 15.8%), while one study pertained to the context of vocational education (2.6%). The second most investigated context was that of health care (N = 11; 28.9%); for instance, the above-mentioned study by Louise Hamilton et al. [21]. Librarianship represents the third-largest context/cluster of studies (N = 9; 23.7%). Here, research was conducted mostly in academic libraries (N = 5; 13%), but also school and public (N = 3; 7.9%) and special (medical) libraries (N = 1; 2.6%). For example, Saunders [25] conducted a study forecasting a possible evolution of IL in academic libraries.

Table 2. Main research contexts in IL Delphi studies
Full size table

3.2 The Types of the Delphi Method Utilized in IL Research

Although not all the examined articles explicitly declared the type of Delphi utilized in the reported IL studies, and many did not provide sufficient quantity and quality of evidence on the research design, a variety of types can be identified, as shown in Table 3. The most common type of Delphi used in IL research was modified Delphi, a variant which employs two rounds and/or a unique form of analysis or distribution of surveys/questionnaires (N = 14; 36.8%). One example is the study by Connolly et al. [26] on a new approach to IL development in Ireland focusing on community of practice and enhanced advocacy. Modified Delphi was followed by classical Delphi, a type employing three full rounds and usually used for forecasting or opinion-gathering (N = 7; 18.4%), including the above-mentioned studies by Pinto et al. [23] and Saunders [25]; online Delphi (N = 5; 13%) (e.g., [21]) and modified e-Delphi (N = 5; 13%), for instance, Frank and Pharo’s [27] research on the perceptions of data IL and attitudes towards IL instruction for meteorology graduate students. The other types represented were classical e-Delphi, including Townsend et al.’s [28] study identifying threshold concepts for IL, policy e-Delphi [29], critical Delphi [11] and grounded Delphi [30].

Table 3. Types of Delphi method, number of Delphi rounds and of experts used per rounds
Full size table

When compared to similar prior research of Delphi types in LIS, a slightly different order of popularity was observed: classical Delphi was the most common type, followed by modified Delphi [15]. On the other hand, Ju and Jin’s [17] review did not report the Delphi types used in the 87 studies they examined, while Poirier and Robinson, as part of a larger Delphi study, provided only a short review of the different variants of Delphi [18] without examining their popularity and frequency of use.

3.3 The Number of Rounds and of Experts in Each Round in IL Delphi Studies

As shown in Table 3, most of the IL Delphi studies used only two rounds of iteration (N = 17; 44.7%), for instance [27, 31], followed by studies with three rounds (N = 13; 34.2%) found in, for example [22, 30]. The least frequent (N = 4; 10.5%) were studies with four rounds (e.g., [28]). The methodology in four studies (10.5%) did not provide descriptions of the study design in terms of the number of rounds used [24, 26, 32, 33].

These findings are consistent with the Delphi methodological literature which indicates that no rigid rules dictate the number of rounds that should be undertaken but that most Delphi studies run over two to three rounds, with two considered to be the minimum and three the most effective number [5, 13]. The findings are also consistent with prior research on the Delphi method in LIS which also found that the majority of studies used two or three rounds of iteration, and only rarely four, to obtain information [17]. Although the most comprehensive to date, Lund’s review of Delphi in LIS [15] did not examine this aspect of the method, despite this being considered its critical aspect and its most distinguished feature [17]; instead, it analysed the amount of attrition (panellist drop-out) over the course of the Delphi studies.

In the present study (similar to the findings of the review of LIS Delphi studies [15]), virtually no attrition was observed (Table 3), although that was considered one of the weaknesses of the method. IL Delphi studies used a minimum of 7 and a maximum of 79 experts, averaging 17. This is in contrast with the recommended, but not prescribed, minimum of 10 and a maximum of 50 [5], which is also the most common range found in research [17] as well as the average number in LIS Delphi studies [15]. Indeed, like other qualitative approaches, Delphi does not depend on or seek to ensure a representative statistical sample [8, 34]; rather it is the profile of experts (their expertise), and not their number, that is the key factor in the success of the method. Therefore, as stated in the Introduction, Delphi can be successfully executed with both a higher and a lower number of experts provided they are carefully selected.

3.4 The Main Population for IL Delphi Studies and the Expert Selection Method

The main population for IL Delphi studies were librarians (in N = 15 studies), as shown in Table 4. This broad category comprised librarians from a variety of library types, mostly academic, but also general, school, special (medical) and public libraries. This population was often selected when researchers were examining IL competences and outcomes and the current or future roles and skills required by librarians for IL instruction, for instance [25, 27], or to meet changing users’ needs in the modern library landscape [31]. Behind librarians were information professionals (N = 9), who contributed in similar research into the required skills and knowledge of information professionals, such as Howard et al.’s study in the converged gallery, library, archive and museum (GLAM) sector in Australia [30]. The third main population were information science researchers (N = 8), whose opinion was elicited in, for instance, research exploring information behaviour [18], and medical professionals (N = 8), including in research into the health literacy curriculum [35].

In line with the Delphi method and its benefit in allowing the efficient bringing together of diverse groups of experts with different areas and levels of expertise, these populations were often empanelled within the same Delphi study, or together with other groups of experts, such as schoolteachers or academic researchers (therefore, the frequency/number of studies in which they participated is shown in Table 4, not the percentage/share). The same populations – librarians (general and academic) and information science researchers – were found to be the most common also in broader LIS studies [14].

Table 4. Main populations for IL Delphi studies and main expert selection methods
Full size table

In terms of geographical coverage and the representation of experts from different continents and regions, most studies used experts from (or were conducted in) Asia (N = 8; 21%), followed by those conducted at global level (or with international experts) (N = 6; 15.8%) and in the Americas (N = 6; 15.8%), of which four were conducted in the United States. Europe accounted for four studies (10.5%) and Oceania three (7.9%), all in Australia, while only one study was conducted in Africa (2.6%). As many as ten studies (26.3%) did not state where the experts came from or where the study was conducted.

As regards the selection and identification of experts for IL Delphi studies (Table 4), the most frequent method was through employment (N = 17) in a specific occupation and/or in an institution (for instance, an academic librarian or a clinician in a university library or hospital) (e.g., [35]). The second most frequent signifier of expertise was scholarly publication (N = 13), for instance in [18], while the third was position in/membership of a professional organization (N = 8) such as the American Library Association and the Association of College and Research Libraries, as in Saunders’ research [25]. The same top three participant selection methods were found in LIS Delphi studies [15].

Other less frequent methods included education, researchers’ professional network (and snowballing), posting on ListServs/mailing lists, and selection and identification based on proximity. Multifaceted sampling strategies were often used, meaning that the methods of selection and identification of experts were used jointly, especially when diverse groups of experts were sought, or when a sufficient number could not be recruited by only one method; in such cases, the initial more scrutinous methods were usually complemented with snowballing and/or posting on ListServs/mailing lists.

It should also be noted that a number of studies (N = 8; 21%) did not specify the selection and identification method. This is problematic since one of the objections and core limitations of the Delphi method (or, more precisely, of the studies utilizing it), apart from the lack of an elementary statistical analysis of the data, is the vagueness of the concept of “expert”/“expertise” and the lack of a sampling procedure and objective criteria to select experts and assess their expertise (for instance, a procedure detailed in [8, 36]). This has an impact on the validity of a given study [17, 37,38,39].

3.5 The Most Common Types of Findings of IL Studies Using the Delphi Method

IL Delphi studies retrieved a variety of findings and were used for various purposes, producing a number of different outcomes, often more than one in a study. By far the most common type of finding retrieved was a framework, frequently a skills and competence framework (N = 17), for example in studies that aimed to develop a framework to guide health curriculum design [29], or to inform the development of an integrated IL framework for paramedic science students [40]. This type of finding was also the most common in LIS Delphi studies [15].

This result is somewhat surprising given that IL is today increasingly understood and investigated, by researchers and practitioners alike, as a socially enacted practice rather than a skill and/or competence framework even though much IL research in the educational setting (the main research setting in the reviewed studies here, see Sect. 3.1) has focused on developing skills and competences. On the other hand, this finding speaks to the changing role of librarians – the main expert population in IL Delphi studies (see Sect. 3.4) – from service providers to active educators working with researchers, teachers and other educators to integrate IL into the curriculum.

The second most frequent type of finding in IL Delphi studies was opinion on a topic or an issue (N = 9), for example to gather views on “IL life cycle” [23], followed by a tool for development/validation/refinement (N = 7), for instance, a questionnaire about digital competences [41]. Other findings retrieved included a curriculum (N = 5) [24], forecast (N = 5) [25], and the identification of themes (N = 4), concerns (N = 2), a taxonomy (N = 2), or the development of a policy (N = 1). These were also the most common findings/purposes of studies that emerged in Ju and Jin’s review [17].

Finally, it should be noted that, in IL Delphi studies, the Delphi method was often used in conjunction with other methods (usually literature reviews, interviews, focus groups, or surveys) as part of larger, multi-phase mixed-method studies, for instance, to inform the next phases (including to select the topic and define the research questions), or to refine and validate the results of the preceding ones (cf. [6]).

4 Conclusion

This research aimed to develop an understanding of how IL research was operationalized by means of the Delphi method as well as an understanding of the key features of the method itself used in IL research. A systematic review of IL Delphi studies retrieved from five databases indicated that Delphi was not a common research method for IL studies, averaging only about two studies per year. Nevertheless, it was used to study various research issues, including digital and health (information) literacy, and in various research contexts, mainly those of education, health care and librarianship, and leading to a variety of findings, most often in connection with IL competence or skill framework. Delphi was used highly flexibly, its design being situational, accommodating to and guided by different research needs and problems, and utilized diverse groups of experts, mostly librarians and information professionals but also experts from other disciplines and fields with various signifiers of expertise.

The review provided here is undoubtedly not complete, and not without limitations. It should be replicated with a team of coders to carry out an intercoder reliability test, not performed in this study since it was conducted by one researcher. Directions for future work may also include extension to other analytical categories (for instance, theoretical frameworks, disciplinary contributions, bases for the number of rounds, consistency of views, and configuration of instruments). The research may be extended by employing additional sources, such as Google Scholar, and combining article search with snowballing and secondary literature searches to encompass other scholarly publications such as proceedings, books and research reports, or doctoral dissertations where Delphi is fairly common, and to cover literature in languages beyond English.

Notwithstanding, it is hoped this review may provide guidance for future research, both content- and methodology-wise, as well as to encourage an expansion in the use of the Delphi method. As the review indicated, the flexibility and versatility of the design and application of the method, and the use of experts from various fields and disciplines, may be beneficial to IL research and development. It has the potential to bridge the theory-practice gap and overcome the current silos and narrow locus in IL research, especially if complemented with other methods, as it can help widen research outside LIS and provide insight into how the concept of IL is understood and operationalized in other disciplines or fields. However, more attention should be devoted to establishing the methodological rigour of the Delphi method to address the flaws in its application (and reporting) and thus enhance its development and utilization in IL research in future.