Keywords

1 Introduction

The rapidly growing relevance of AI-based digital learning platforms (AILPs) for the development of knowledge societies and Industry 4.0 is a challenge in defining new educational, research, social, and economic policy goals from regional to European levels. AI-based digital ecosystems comprising learning communities around platforms and social media, the high-tech sector providing software and services, the related research and its governance, are crucial in ensuring a positive impact and wide acceptance of learning-related AI strategies. It is also important to select priorities for EU-financed educational projects in Horizon Europe and in other forthcoming research programmes. When supported by public funds, AILP development is deeply rooted in the PEST (Political, Economic, Social and Technological) environment, and it requires a thorough strategic plan of the social and economic impacts over a mid- to long-term perspective. Strategic planning should be aligned with technological progress, specifically in the emerging areas of Artificial Intelligence, Big Data, and Global Expert Systems [10], with an emphasis put on recommendation and decision support [12].

Despite the relevance of AILPs, very few publicly accessible AILP strategies or descriptions of strategic technological planning approaches exist. Those available refer mostly to e-learning course repositories [2, 5, 6]. When defining the policy framework for an AI-based knowledge repository developed within a Horizon 2020 EU research project [www.moving-project.eu], the above situation created a need to develop methodological foundations for AILP-oriented strategic planning in the context of future learning technology needs. An outline of this methodology is presented in the next section. Section 3 presents a novel expert knowledge elicitation and processing tool that utilizes innovative ‘extrapolation Delphi’ surveys [www.forgnosis.eu] to construct an AILP technological strategy and estimate its social and economic impact. In Sect. 4, we present an application of this tool to define the social, economic, market, and business-oriented research environments of learning platforms. These were applied in the technological roadmapping of the generic learning platform that stores online courses, manuals and scholarly papers, video files, economic and other information useful for learners. We will refer to this AILP as the Platform. Its operation is supported by a number of AI-based tools, from content-based automatic video annotation and retrieval, to intelligent educational recommenders and creativity stimulation tools [12]. Its primary application is to support learning and provide open guided knowledge to public administrators, students and young researchers [13].

In this presentation of the AILP strategy building process and its educational implications, we will focus on the methodology of generating future visions of the Platform, functioning with a flexible Delphi survey support system based on a novel forward extrapolation methodology. The survey offers a variety of question and/or statement types, sophisticated statistical analysis and other methods to handle uncertainties, as well as a user-friendly interface. It can be run in various modes that suit the survey goals and gather expert knowledge in multiple rounds, as a real-time Delphi or as a hybrid of both. The cloud-based Delphi application was offered to the project team in SaaS mode [11]. It can also be used as a basis for designing further customised expert information retrieval and fusion exercises for a broad spectrum of learning and research needs, as well as it can serve as an AI-based learning tool itself.

2 Methodological Approaches to AILP Impact Assessment

Impact modelling and strategy building for the Platform was designed as a generic process to serve a large class of AILPs. It was split into the following two stages:

  1. a.

    Establishing the boundary conditions for AILP activity, exploitation and learning community building (Stage 1).

  2. b.

    Delivering the final exploitation strategy aimed at ensuring digital sustainability, financial viability and social acceptance, taking into account plausible scenarios of the PEST environment resulting from an expert Delphi survey (Stage 2).

Forecasts were obtained from experts at both stages as outcomes of the Delphi survey and used to build an anticipatory network (AN) impact model cf. e.g. [9, 14]. The AN-based methodology has already proved useful in multicriteria strategic planning [14]. The ANs provide constructive algorithms for computing nondominated strategic plans that comply with a given anticipatory preference structure. The preferences of stakeholders, policy makers and other decision makers (i.e. those responsible for shaping the Platform’s future and beneficiaries) can be taken into account. We will provide indications on how to apply anticipatory decision-making principles in constructing and filtering scenarios corresponding to rational and sustainable future AILP visions. AN-based assessment processes allow the analyst to select a subset of normative scenarios corresponding to the most preferred states of the future and subsequently run an AN-based backcasting [14]. By definition, the best normative scenario describes the most desired future elicited from AILP stakeholders, starting from the current best-compromise decision of the AILP management team and passing through the intermediate states that correspond to the interim goals of the AILP development project. The strategic goals were derived from:

  • Expert information concerning future trends in education technology.

  • A study of the PEST environment followed by a SWOTC (Strength, Weaknesses, Opportunities, Threats, and Challenges) analysis.

  • An AN-based impact model within an analytic strategic planning process that follows a technological roadmapping scheme.

This analytical and collaborative process ensures the selection of the best-compromise decision sequence, or another scenario that comes close, in the sense of reaching the best expected values of the prescribed Platform performance criteria, under different forecasts of external circumstances. Specifically, given the forecasts or scenarios, the strategic planning algorithm computes decisions that correspond to the optimal social and economic impacts of the Platform operation. This is the core procedure of the backward planning process. As the Platform’s external circumstances are policy- and technology-dependent, the above procedure allows us to determine the conditions that can make the above optimum provisions and favourite circumstances real. Assuming that AILP usage principles by an average individual user do not differ considerably worldwide, we can derive general indications on the functioning of intelligent digital repositories, learning, and other knowledge platforms.

2.1 A Delphi Support System to Elicit Forward-Oriented Expert Knowledge

Future visions of the Platform’s function, its PEST environment and learning technology progress are fundamental to the strategy building process. These have been obtained from experts with a flexible Delphi survey support system (DeSS, [11]). Unknown future parameters to be inserted into the social and economic impact models resulted from a novel forward extrapolation approach used in the survey, which can be summarised as follows.

First, experts provide quantitative estimations of the relevant variables for a few predefined forecasting horizons. These estimations, together with a current state assessment, are then used to fit a regression curve that can yield significant extrapolations even beyond the farthest predefined forecasting horizon. The survey offers a variety of question and/or statement types, sophisticated statistical analysis and other uncertainty handling methods, as well as a user-friendly interface. It can be run in policy or in decision Delphi modes [11, 13] as well as gather expert knowledge over multiple rounds, as a real-time Delphi, or as a hybrid of both. The survey software used (ForgnosisTM, cf. [11] for details) provides a number of features. Among other things, it can detect duplicate replies, correlated replies, as well as verify the content of each individual reply (e.g. detect random entries, check the justification consistency).

The objective nature of the information obtained from experts is ensured by the manner in which the survey is organised [4]. A preceding “Round 0” is established to allow the Platform stakeholders to determine the scope of the survey. The first stage is aimed at reaching an internal consensus among the decision makers and experts, while the main goal of the second stage is to expand the future vision and seek new opportunities and challenges, cf. [11, 15]. Both stages consist of two rounds. Rounds 1 and 2 were performed in multi-round mode [11] while Rounds 3 and 4 were organised as a real-time Delphi [3] – an adaptive advisory activity. Participation in the survey was open to the Platform owners and developers staff as well as external experts who were not directly involved in the project’s execution, such as higher university management staff, members of the scientific and supervisory councils, etc.

The results of the survey are intended to evaluate the internal conditions and external circumstances under which the Platform will be exploited. This information is necessary to build its sustainability strategy. Thus, the survey touched upon development trends of AI technologies enabling the future evolution of the Platform. These included autonomic web search technologies, Internet evolution prospects, creativity support systems, data mining techniques, content-based information retrieval, and multimedia searches. Furthermore, when responding to the survey, experts provided relevant information on the anticipated economic, social, and political environments, Platform’s learning services and functionalities that may be offered in the future, and feasibility of strategic goals and future business models of the Platform.

The first stage of the survey was aimed at eliciting project staff opinion on the exploitation of the Platform during the project’s durability period (2019-2024) and beyond (until 2030). This stage corresponded to the ‘decision Delphi’ [13] type of exercise, popular in corporate foresight activities. Its characteristic feature involves decision makers that may have some influence on the future visions provided in the survey. According to the ‘decision Delphi’ principles, Rounds 1 and 2 were focused on internal consensus building regarding all aspects of the Platform’s sustainability, in particular digital sustainability, cf. [1]. Participation in this stage of the survey allowed the project staff to better understand diverse aspects of the Platform’s technological viability and the relation to its future development.

The second stage consisted of Rounds 3 and 4. They included questions that did not yield consensus during the first stage of the survey and focused on research and macroeconomic aspects of AILP development. Besides reaching a consensus on general issues, the aim of Stage 2 was also to detect any change in the internal or external circumstances that might affect the Platform’s development and performance.

The cloud-based Delphi application (DeSS) was used in Software-as–a-Service (SaaS) mode, with some Platform-as–a-Service (PaaS) features. This application turns out suitable to design further customised expert information retrieval and fusion exercises for various educational and research needs. The survey has been available at the dedicated web page www.moving-survey.ipbf.eu where experts can register to participate in the ongoing activities. More details on the underlying Delphi methodology are provided in [11].

2.2 Expert Information Elicited and Analysis of Survey Results

The Delphi survey variant with confidence management was selected as best suited for this type of survey, where the participants were experts in specialised fields [10] such as learning technology, pedagogy, or sociology. According to its principles, the survey questions are presented to all participants but not all of them had to reply all questions. Experience resulting from earlier surveys within thematic areas related to e-learning and learning support systems has been taken into account (cf. [5, 7, 8]).

The replies obtained from different respondents to the same question have been fused together using weighted averaging approaches and triangular fuzzy values to account for confidence coefficients. An initial value for “degree of confidence” (usually based on a 5-value Likert scale) could be self-assessed by the respondents for each question or group of questions. This made it possible to consider opinions of those respondents who are not experts in the specific field covered by the survey but are nonetheless capable of contributing valuable ideas, yet with a lower weight. Uncertainty handling with random-fuzzy distributions turned out to be a suitable technique for this kind of data. The fuzzy factors described the uncertainty related to diversified competences, while the stochastic properties of the reply dataset were used to fuse individual replies. The overall methodology is explained in detail in a context-based online help manual available to users after logging into the survey system.

3 Analysis of Survey Results and Intra-round Convergence

A DeSS questionnaire with three subordinated questions concerning the relevance of future risk factors that may affect the Platform demonstrates the survey in action (cf. www.moving-platform.ipbf.eu: Subsect. 3, question 6). An analysis of the replies shown in Fig. 1 (next page) looks at the relevance of the six risk factors related to the implementation and operation of learning platforms (listed in the Fig. 1 legend). The risk factors with the highest potential impact on the Platform were pre-selected during the Delphi “Round-0” from about 20 candidate factors. The impact assessments (horizontal axis) are confronted with the uncertainty (vertical axis) of respondents assessing them. The latter is the standard deviation of replies. Both factors are expressed in 5-point Likert scale points, where the numerical values 1 to 5 correspond to naturally ordered scale values: “irrelevant” (1), “low-relevance” (2), to “very relevant” (5). One can observe that data protection and data privacy have been identified as the most relevant risk factors, with slightly decreasing relevance until 2030. The uncertainty related to risk factor assessment increases with time for the three factors (a), (b) and (e), which may be explained as a natural consequence of confidence intervals growing with more distant forecasting horizons. However, an uncertainty increase accompanied by the growing relevance of the remaining risk factors (c), (d), and (f) indicates a growing consensus regarding the expected threat growth. These results have been applied in the risk analysis component of the Platform’s sustainability strategy.

Fig. 1.
figure 1

Relevance/uncertainty analysis of risk until 2030. Values on both axes are expressed in 5-value Likert scale points factors with the highest potential impact on the Platform. The arrow directions coincide with time flow, ending at the state expected

Full size image

A similar analysis of Delphi results was performed for the group relevance assessment of learning technologies, such as user creativity measurement and stimulation tools, augmented and virtual reality, or serious educational games.

Presented below is another sample set of results for the following Delphi question:

figure a

Table 1 contains a sample set of statistical characteristics of replies and a basic analysis. Unless indicated otherwise, table entries are given in percent or in percent points. It also provides the values of classical consensus measures yi, i = 1, 2, 3, for question 1.6, where y1 is the standard deviation σ, y2 – the sum of standard semi-deviations, y2 = σ_ + σ+, and y3 is the interquartile range (IQR), by definition IQR: = 3rd quartile – 1st quartile. The above measures have been used to measure consensus achievement for all questions. The convergence between rounds has been measured as

Table 1. Statistical analysis of replies to Question 1.6 of the Delphi survey (www.moving-survey.ipbf.eu); statistical characteristics calculated with respondents’ competence coefficients
Full size table
$$ \xi \left( y \right) = \left( {1 - {{y\left( {\text{round}2} \right)} \mathord{\left/ {\vphantom {{y\left( {\text{round}2} \right)} {y\left( {\text{round}1} \right)}}} \right. \kern-0pt} {y\left( {\text{round}1} \right)}}} \right) * 100\% ,\text{where}\,y = y_{1} \,{\text{or}}\,y = y_{2} \,{\text{or}}\,y = y_{3} $$
(1)

or as the Kullback-Leibler divergence of the empirical reply distributions in the subsequent (n-th and (n + 1)st) rounds.

The quartiles provided in Table 1 are complemented by the quintiles shown in Fig. 2 (right). The left part of Fig. 2 shows the histograms of replies to question 1.6 for the forecasting horizons 2020, 2025, and 2030, showing the number of clusters of replies.

Fig. 2.
figure 2

Histograms (upper diagram) and the quintiles with the median (lower diagram) of replies to Question 1.6

Full size image

The emergence of statistically significant clusters indicates a lack of consensus but provides indications concerning the potential existence of several exploratory scenarios of the investigated factor or variable [15]. It is worth noting that in the above question, there was only one significant cluster of replies and a consensus was reached in round 2 for all forecasting horizons. It should be noted that a consensus, in terms of IQR/2 < 20%, was reached for 48 out of 96 questions in Round 2 for the main forecasting horizon of 2025, selected as the end of the minimum durability period of the Platform. The number of questions with consensus decreased to 41 out of 96 for the year 2030. However, all but one of the Delphi statements for 2025 exhibited a convergence between subsequent Rounds 1 and 2 with respect to all the indicators (1) with y = yi, i = 1, 2, 3, and the Kullback-Leibler divergence used as convergence measures.

4 The Strategy Building for an AILP

The information received from survey respondents was fused and analysed. The outcomes of the survey together with bibliometric, patentometric and demographics data were then used as the input needed to build an anticipatory network (AN) that models all relevant future impacts relative to the AILP analysed. The network includes all relations between actors, factors, trends, and plausible random future events. ANs are a relatively new tool in decision theory, rooted in anticipatory system theory [14]. This theory formalises and fuses multi-stage multicriteria forward planning, and multicriteria backcasting. ANs generalise earlier anticipatory models of decision impact in multicriteria problem solving and constitute an alternative decision model to utility or value function estimations and to diverse heuristics [14]. An AN is a directed multigraph with no loops, nodes modelling the decision problems, and edges modelling the relations between them. Every AN must have at least one starting node (with no predecessors) which models a present-time multicriteria strategic decision problem. The other nodes model decisions made for future multicriteria problems that will be solved by the same or other decision makers. The edges of the first kind model the causal dependence of decision problems on solutions to previous problems. There may be several causal relations and corresponding edge classes in one network. Subsequent decisions that are made along a chain of causal dependences in an anticipatory network model the consequences of decisions made at earlier nodes in the chain.

The decision is made after a constructive analysis of causal relations that link the outcomes of the current problem with their future consequences [14]. In case of the learning platform, the starting problem corresponds to the decisions of the coordinating bodies of the consortium jointly developing this AILP, while the ‘next generations’ correspond to the future management of this AILP. The other decision makers are responsible for user community building, technology acquisition, or development. The overall strategic technological planning process comprised of the following:

  • A forward-looking activity aimed at identifying the internal and environmental factors influencing the future performance and impact of the Platform. The activity combines a four-round/real-time novel policy and decision Delphi survey with an AN impact model established with the parameters delivered as survey outcomes.

  • A dynamic (2-stage, real-time) SWOTC (SWOT with Challenges) exercise, which provides additional inputs to the Risk Matrix analysis and the final roadmapping.

  • Technological and anticipatory planning that delivers the final strategy with three exploitation scenarios resulting from the Delphi survey and an AN-based action plan ensuring AILP digital sustainability, economic viability and social acceptance.

  • Roadmapping-based technological planning of the learning platform operation.

  • Social impact analysis with ANs, cellular automata, and Bayesian network models.

A scheme of the overall generic strategy building process, which was applied for the Platform as a representative case of AILP, is presented in Fig. 3, cf. also [13].

Fig. 3.
figure 3

A technological planning scheme for a generic AI-based digital learning platform

Full size image

5 Conclusions

As a general conclusion, it should be noted that the expert knowledge acquisition and strategic decision making methods presented in this paper define a complete computing background for technological planning of an AI-based learning platform. It consists of an online Delphi survey support system (DeSS), an AN building tool, and generic roadmapping support system that form together a strategic Decision Support System (DSS) allowing an analyst to gather and efficiently deploy technological foresight results. Nevertheless, the above-presented DeSS can also be used as a stand-alone DSS. When used jointly with analytic impact modelling tools, future circumstances can be analysed quantitatively and algorithmic action plans can be defined for complex AI-based learning management systems such as AILPs. Thanks to a growing number of publicly available foresight results and an accessibility of open source web information repositories, the hitherto barely affordable strategy building processes can be performed satisfactorily as a combination of an online Delphi survey, other collaborative activities such as SWOTC, roadmapping, and interactive multicriteria decision making with anticipatory preference information and trade-offs between criteria.

The novel forward extrapolation methodology used in the above-presented DeSS offers a variety of question and/or statement types, a sophisticated statistical analysis and other uncertainty handling methods as well as a user-friendly interface. It can be run in policy as well as decision Delphi modes, gather and fuse expert knowledge in multiple rounds, as a real-time Delphi, or as a hybrid of both. The final results of outcomes was presented to the decision makers as technology or functionality rankings.

We have shown in previous sections that the overall approach proved useful in building a multiple-context learning platform strategy, exemplifying a larger class of similar applications. The Delphi survey assured a persistent deployment of contributions from experts knowledgeable in the field of technologies and AILP markets.

Finally, let us note that the models and applications presented in this paper can benefit from a synergy with other foresight and forecasting methods and IT tools such as foresight and roadmapping support systems.