Keywords

1 Introduction

The main aim of the paper is to present a methodology to evaluate suitability, acceptance and use of personalised learning units/scenarios.

Methodology is refereed here as a model and method to evaluate learning units (or Units of Learning, UoLs). UoLs are referred here as methodological sequences of learning components (learning objects (LOs), learning activities (LAs), and learning environments (LEs) that are often referred to as virtual learning environments). High-quality UoLs should consist of the learning components optimised to particular students according to their personal needs, e.g. learning styles.

In the paper, personalised UoLs are referred to as UoLs composed of the learning components having the highest probabilistic suitability indexes [21] to particular students according to Felder-Silverman Learning Styles Model [6].

Probabilistic suitability index is the main value used to establish the preference list of learning components according to their suitability level to students’ learning styles. It is based on probabilistic model of students’ learning styles and ratings (values) of learning components’ suitability to particular students according to their learning styles [21].

Finally, the methodology analysed in the paper is based on acceptance and use evaluation criteria proposed by Educational Technology Acceptance & Satisfaction Model (ETAS-M) which in its turn is based on well-known Unified Theory on Acceptance and Use of Technology (UTAUT) model.

The rest of the paper is organised as follows: related research is presented in the following Section, proposed methodology to evaluate personalised learning units is presented in Sect. 3, and Sect. 4 concludes the paper.

2 Related Research

2.1 Personalisation of Learning Units

Learning personalisation became very popular research object in scientific literature during the last years [1, 5, 10, 17, 22, 23, 28]. Research topic on creating full learning units [13] and smaller learning components (LOs [11, 14], LAs [9] and LEs [16, 20]) that should be optimal (i.e. the most suitable) to particular students based on expert evaluation methods and techniques has also become highly demanded, and there are some relevant methods and techniques proposed in the area [12, 15, 19].

According to [12], suitability of learning unit/scenario to particular learner should be evaluated according to the following framework:

According to [18], future education means personalisation plus intelligence. Learning personalisation means creating and implementing personalised UoLs based on recommender system suitable for particular learners according to their personal needs. Educational intelligence means application of intelligent (smart) technologies and methods enabling personalised learning to improve learning quality and efficiency.

In personalised learning, first of all, integrated learner profiles (models) should be implemented. It should be based on e.g. Felder and Silverman Learning Styles Model (FSLSM) [6]. Dedicated psychological questionnaires like Soloman and Felder Index of Learning Styles questionnaire [27] should be applied here. After that, we should integrate the rest features in the learner profile (knowledge, interests, goals, cognitive traits, learning behavioural type etc.).

FSLSM [6] classifies students according to where they fit on a number of scales pertaining to the ways they receive and process information:

  1. (a)

    By information type: (1) Sensory (SEN) – concrete, practical, oriented towards facts and procedures vs (2) Intuitive (INT) – conceptual, innovative, oriented towards facts and meaning;

  2. (b)

    By sensory channel: (3) Visual (VIS) – prefer visual representations of presented material e.g. pictures, diagrams, flow charts vs (4) Verbal (VER) – prefer written and spoken explanations;

  3. (c)

    By information processing: (5) Active (ACT) – learn by trying things out, working with others vs (6) Reflective (REF) – learn by thinking things through, working alone; and

  4. (d)

    By understanding: (7) Sequential (SEQ) – linear, orderly, learn in small incremental steps vs (8) Global (GLO) – holistic, systems thinkers, learn in large leaps.

According to [21], after filling in Soloman and Felder’s Index of Learning Styles questionnaire [27], one could obtain e.g. the following learning style initially stored in his/her student profile/model (Table 1):

Table 1. Example of learning style stored in the student profile (according to [21])
Full size table

After that, methodology on creating optimal UoLs for particular learners based on expert evaluation and intelligent technologies should be applied as follows:

According to [18], in personalised learning, first of all, integrated learner profiles should be implemented, and ontologies-based recommender systems should be created to suggest learning components (LOs, LAs and LEs) suitable to particular learners according to their FSLSM-based profiles. Thus, the whole personalised UoLs could be created for particular learners for each topic according to study programmes at Universities or curriculum at schools.

According to [18], a number of intelligent technologies should be applied to implement this approach, e.g. ontologies, recommender systems, intelligent software agents, multiple criteria decision making models, methods and tools to evaluate quality and suitability of the learning components etc.

Ontologies and recommender systems should be based on established interlinks between students’ profiles and learning components. While establishing those interlinks, high-quality learning styles models and vocabularies of learning components should be used, on the one hand, and experienced experts should participate in this work generating collective intelligence, on the other.

Since the aim of the paper is to present UoLs suitability, acceptance and use evaluation methodology, first of all, one should identify a system of decision (evaluation) criteria (i.e. model).

According to [13], decision criteria are rules, measures and standards that guide decision-making. Quality criterion is a tool allowing comparison of alternatives according to a particular point of view. When building a criterion, the analyst should keep in mind that it is necessary that all the actors of the decision process adhere to the comparisons that will be deduced from that model. Criteria (relatively precise, but usually conflicting) are measures, rules and standards that guide decision-making, which also incorporates a model of preferences between the elements of a set of real or fictitious actions.

In identifying criteria for the decision analysis, the following considerations (i.e. principles) are relevant to all the multiple criteria decision analysis (MCDA) approaches [13]:

(1) Value relevance; (2) Understandability; (3) Measurability; (4) Non-redundancy; (5) Judgmental independence; (6) Balancing completeness and conciseness; (7) Operationality; and (8) Simplicity vs complexity.

Learning scenario/unit quality evaluation model based on these MCDA criteria identification principles is presented in Fig. 2.

According to [13], UoL is technology consisting of LOs, LAs and LEs. Therefore, UoL quality criteria should consist of the quality criteria identified for all its components:

2.2 Application of UTAUT Model in Education

The components’-based UoL evaluation model presented in Fig. 1 has its shortages, e.g. there are different criteria to evaluate different learning units’ components. This approach is quite time-consuming and requires different and high level expertise from evaluators. According to Sect. 2.1, personalised UoLs are as high-quality as they fit students’ personal needs based on FSLSM. Therefore, we could apply the same criteria-based evaluation of all components by the users.

Fig. 1.
figure 1

(according to [12])

Framework for evaluating suitability of UoL to learner

Full size image
Fig. 2.
figure 2

(according to [13])

Learning unit’s quality model

Full size image

This kind of evaluation is based on Unified Theory on Acceptance and Use of Technology (UTAUT) model. In the paper, UTAUT is examined while being applied in education in terms of acceptance and use of information and communication technologies for personalised learning purposes.

In this section, the original UTAUT model proposed by Venkatesh et al. [31] is analysed supplemented by 10 carefully selected studies on UTAUT application in education.

According to [31], information technology acceptance research has yielded many competing models, each with different sets of acceptance determinants. The eight models reviewed in [31] are the theory of reasoned action, the technology acceptance model, the motivational model, the theory of planned behaviour, a model combining the technology acceptance model and the theory of planned behaviour, the model of PC utilisation, the innovation diffusion theory, and the social cognitive theory. In [31], seven constructs appeared to be significant direct determinants of intention or usage in one or more of the individual models. Of these, the authors theorise that four constructs will play a significant role as direct determinants of user acceptance and usage behaviour: (a) performance expectancy (PE), (b) effort expectancy (EE), (c) social influence (SI), and (d) facilitating conditions (FC) as presented in Fig. 3:

Fig. 3.
figure 3

(according to [31])

UTAUT model

Full size image

Study [7] aimed to investigate students’ acceptance and use of Moodle employing the model of UTAUT and further understand the four constructs of the model. Data collected revealed that PE, EE, and SI were the major three keys of the UTAUT model to assess the acceptance of Moodle. Behavioural intention acted as a mediator to urge students to involve in the use of Moodle.

According to [30], acceptance of e-learning by employees is critical to the successful implementation of e-learning in the workplace. To explain why employees might accept the e-learning technology, motivational factors must be considered. According to [30], the effects of intrinsic motivators mediated the effect of extrinsic motivators.

According to [29], the UTAUT proposes that PE, EE, and SI predict behavioural intention towards the acceptance of information technology. The theory further proposes that FC and behavioural intention predicts use behaviour in the acceptance of information technology. Ever since its inception, the theory has been assessed using different applications. In [29], based on 37 selected empirical studies, a meta-analysis was conducted in order to harmonise the empirical evidence. The outcome of the study suggests that only the relationship between PE and behavioural intention is strong, while the relationships between EE, SI and behavioural intention are weak. Similarly, the relationship between FC, behavioural intention and use behaviour is also weak. Furthermore, the significance of the relationship between FC and use behaviour does not pass the fail-safe test while the significance of the relationship between behavioural intention and use behaviour does not pass the fail-safe test satisfactorily.

The main focus of paper [2] is to contrast and combine results from 20 different studies using the UTAUT and its extensions, in the hope of identifying patterns among studied results, sources of discrepancy among those results, or other existing relationships that may come to light in the context of these studies.

The review [3] evidently shows that variables that need to be applied to determine users’ acceptance or adoption of technology vary. The effect of exogenous variables EE, PE, SI on endogenous variable ‘behavioural intention’ is not consistent across countries, within country, and unit of studies. According to the results of [3], EE (0.4, p < .05) significantly predicted Behavioural Intention to use technology, SI and PE were statistically insignificant, as was Behavioural Intention on Use Behaviour. However, FC (β = .26, p < .01) significantly influenced Use Behaviour.

According to [8], technology acceptance studies are a common medium of determining approval and predicting future use of technologies in the field of Information Systems. Numerous technology acceptance studies have been done in the area of education however there still remain hindrances in the use of computers in education. The aim of the study [8] is to analyse published research materials in the area of technology acceptance in education and identify the current research patterns. Upon identifying these patterns, a future research path is presented. For this purpose, initially the popular technology acceptance theories are studied so as to build a firm base for examining the technology acceptance works in education domain. The technology acceptance research works were thoroughly scrutinised to identify important aspects like acceptance theory used, constructs used, causal relationships and user types. Based on all these aspects a future research pathway is suggested. In [8], the acceptance of the two technology enabled phases of education i.e. e-learning and e-assessment was discussed. In the starting, e-learning along with its different types, advantages and disadvantages was explained. The difference between e-assessment and e-learning was explained because they are often thought as the same. Then the different types of e-assessments were discussed so as to show their variety. Next, it was found that the majority of the acceptance studies in education area have been on e-learning barring a few on e-assessment.

According to [26], among the fourteen theories reviewed in the paper, UTAUT seems to be an improved theory that could provide a useful tool to assess the likelihood of success for technology acceptance studies.

According to [4], in understanding how active and blended learning approaches with learning technologies engagement in undergraduate education, current research models tend to undermine the effect of learners’ variations, particularly regarding their styles and approaches to learning, on intention and use of learning technologies.

Study [24] seeks to explore the factors that influence students’ usage behaviour of e-learning systems. Based on the strong theoretical foundation of the UTAUT and using structural equation modelling, this research paper examines the impact of PE, EE, hedonic motivation, habit, SI, and trust on student’s behavioural intention, which is later examined along with FCs on student’s usage behaviour of e-learning systems. The results revealed direct positive effect of PE, hedonic motivation, habit, and trust on student’s behavioural intention to use e-learning explaining around 71% of overall behavioural intention. Meanwhile, behavioural intention and FC accounted for 40% with strong positive effects on student’s usage behaviour of e-learning systems. However, both EE and SI influence did not impact student’s behavioural intention.

This review shows that UTAUT was never applied earlier to evaluate technology like learning unit/scenario.

The only study [25] was found in scientific literature which proposed UTAUT-based model that could be applied to evaluate personalised UoLs.

Paper [25] examines various extensions of UTAUT and related frameworks from a theoretical and empirical point of view. The theoretical contribution of the paper consists of substantial extensions/improvements of the UTAUT which are embedded within the theoretical paradigm of social constructivism. It is argued that the usability aspects of e-learning systems cannot be treated independently from their impact on learning behaviour and the pedagogical setting in which they are implemented.

Based on new empirical data from an experimental, undergraduate statistics course the authors provide strong support for a newly proposed Educational Technology Acceptance & Satisfaction Model (ETAS-M) (Fig. 4):

Fig. 4.
figure 4

(according to [25])

ETAS-M

Full size image

In this paper, a novel ETAS-M based methodology is proposed to evaluate personalised learning units.

3 Personalised Learning Units Evaluation Methodology

Based on related research analysis, we propose UoL evaluation model based on MCDA criteria identification principles (Fig. 2), Educational Technology Acceptance & Satisfaction Model (ETAS-M) (Fig. 4), and probabilistic suitability indexes (SI) to identify learning components’ suitability to particular students’ needs according to their learning styles [21].

Proposed model is components’ based, on the one hand, and ETAS-M-based, on the other. Evaluation criteria are performance expectancy (PE), effort expectancy (EE), facilitating conditions (FC), and influence of pedagogical paradigm (IPP) instead of social influence (SI) in UTAUT.

It’s more convenient in comparison with purely components-based model presented in Fig. 2 because it is based only on acceptance and use evaluation made by the users, and fully reflects their needs and points of view.

Additionally, this kind of model does not require specific high-level technological expertise from experts-evaluators to evaluate UoL alternatives by learning components’ internal quality criteria.

Proposed personalised UoL evaluation model is presented in Fig. 5.

Fig. 5.
figure 5

Proposed personalised UoL evaluation model

Full size image

After creating personalised UoL evaluation model, we should apply some evaluation method in order to evaluate particular UoL.

Proposed UoL evaluation method is based on Fig. 5. It could be expressed by formula (1):

$$ f\left( x \right) = \left( {\frac{{\mathop \sum \nolimits_{i = 1}^{n} SI_{i} }}{n}} \right)\left( {\mathop \sum \nolimits_{j = 1}^{m} \alpha_{j} f_{j} \left( x \right)} \right) $$
(1)

where i is learning component (LO, LA or LE), n = 3, SI i is probabilistic suitability index of corresponding learning component i to particular student, a j is a weight of criterion j, and f j (x) is a value of criterion j, m = 4 (PE, FE, FC and IPP). In this paper, the weights of criteria are referred as equal.

Thus, in order to identify numerical value of UoL evaluation function, one should (1) multiply the values of all ETAS-M-based evaluation criteria by their weights for all learning components, (2) add these numbers together and identify the sum, (3) multiply all these sums by average probabilistic suitability indexes of corresponding learning components, and (4) identify the total sum. The higher the numerical value of f(x) the better is the UoL for particular learner.

Practical value of the proposed methodology to evaluate suitability, acceptance and use of UoLs to particular students is as follows:

At any University or school, teachers have to create some kind of UoLs (modules, lessons etc.) for their students composed by learning content (LOs), learning activities, and learning environment. First of all, recommender system should recommend the most suitable learning components to particular students according to appropriate probabilistic suitability indexes applying UoLs personalisation methodology [18] described in Sect. 2.1. Additionally, there are a number of tools created to automatically compose UoLs from the most suitable learning components that teachers could use in their pedagogical practice.

The main problem here is how to create the most suitable UoLs for particular students that should have the highest level of acceptability and use by these students. In Fig. 5 and Formula (1), we present the model and method to evaluate suitability, acceptance and use of particular UoLs to particular students.

Thus, teachers should create personalised UoLs that should be (1) the most suitable for particular students in terms of the highest values of average probabilistic suitability indexes, and (2) the most acceptable and usable by students in terms of UoLs performance expectancy, effort expectancy, facilitating conditions, and influence of pedagogical paradigm used by teachers.

4 Conclusion

In the paper, the authors propose personalised learning units/scenarios suitability, acceptance and use evaluation model based on MCDA criteria identification principles, learning components’-based evaluation model, and Educational Technology Acceptance & Satisfaction Model (ETAS-M) based on UTAUT model. Every UoL’s component (LO, LA and LE) should be evaluated according to ETAS-M. Personalisation of UoL components according to FSLSM should be guaranteed by identifying corresponding average probabilistic suitability indexes.

The proposed model is components’ based, on the one hand, and ETAS-M-based, on the other. It’s more convenient in comparison with purely components-based model presented in Fig. 2 because it is based only on suitability, acceptance and use evaluation made by the users, and fully reflects their needs and points of view. Additionally, this kind of model does not require specific high-level technological expertise from experts-evaluators. On the other hand, proposed model is better than pure ETAS-M/UTAUT-based model because it’s more flexible since it takes into consideration all different components of UoL separately as well as corresponding average probabilistic suitability indexes.

Finally, in the paper, personalised UoLs evaluation method was proposed by formula (1).

Proposed methodology is feasible to be applied in real-life pedagogical situations in educational institutions. In order to create and evaluate personalised UoLs, educational institutions should establish FSLSM-based students’ profiles, use high quality vocabularies of learning components, and have enough expertise to identify corresponding suitability indexes.