Keywords

1 Introduction

Currently, there is concern related to the low and decreasing engagement with science, technology, engineering and mathematics (STEM) education across many countries and education levels ranging from secondary to third level [31, 39, 47]. This issue is drawing increasing attention from bodies such as OECD and EU given that STEM graduates play a crucial role in today’s knowledge economies through technological innovation [20, 46]. Often, students start to become disengaged with STEM at an early age due to factors such as perceived difficulty of STEM modules [47, 51], negative perceptions of the STEM field and negative beliefs in their abilities for STEM [1, 58]. Various innovative technologies and pedagogical approaches have been proposed in order to increase student’s engagement with STEM subjects, including: adaptive and personalised learning [44], inquiry-based learning [31], and remote fabrication labs and virtual labs [48].

Virtual labs help to overcome the costs associated with maintaining physical labs, as well as a solution to make practical science education available to online learners [35, 36]. In a virtual lab, students can practice at their own pace in a safe simulated environment, being able to define their own experiments and to repeat them as many times as they want. While many virtual labs were developed over the years for different subjects and education levels, most of these are not personalised to learner’s needs and profile. Furthermore, few research studies have evaluated the benefits of virtual labs in terms of their impact on learner motivation aspects such as interest, confidence, engagement and enjoyment.

This chapter presents the results of a small-scale pilot conducted in two secondary schools from Ireland to evaluate the Atomic Structure interactive personalised virtual lab. The lab enables students to learn about atoms, isotopes and molecules in an active learning way by integrating various technologies and pedagogies, including: inquiry-based learning, multimedia, interactive builders, quizzes, and gamification. The pilot used a multidimensional methodology that applied knowledge tests and surveys before and after the learning session in order to assess the impact of the Atomic Structure virtual lab on learners’ knowledge, motivation and usability.

1.1 The NEWTON Project

This research work is part of the NEWTON Project (http://newtonproject.eu), a large-scale EU H2020 innovation action that involves 14 academic and industry partners. The project focuses on employing innovative technologies to support and improve STEM education.

NEWTON innovative technologies include solutions for user modelling, adaptation and personalisation in order to increase learner quality of experience, improve learning process, and potentially increase learning outcome [17]. Interactive educational computer-based video games and gamification are used to stimulate and motivate students [18], augmented reality allows learners to access computer generated models of scientific content, while interactive avatars guide students with special learning needs in a manner which suits them the best [10]. Virtual teaching and learning laboratories [9] and remote fabrication labs [55] allow students to experiment in simulated environments and eventually transform their solutions into real life products. The project also employs adaptive multimedia to overcome network and device limitations and improve the quality of experience [25, 40, 42], as well as multi-sensorial media (or “mulsemedia”) that helps engage three or more human senses in the learning process, including smell and touch [7]. Different innovative pedagogical approaches are also deployed as part of the STEM teaching and learning process such as flipped classroom, game-based and problem-based learning [12, 45, 61].

All these technologies are implemented and deployed within an educational platform called NEWTELP (http://newtelp.eu). The platform enables educational content to be stored and delivered to learners as part of real-life pilots to see whether and how they help students to engage more with STEM subjects. Over thirty NEWTON small- and large-scale pilots on various technologies, were conducted in different primary and secondary schools, university and vocational institutions from across Europe. This chapter is an extended version of the paper presented at CSEDU 2019 conference [24]. The chapter includes additional literature review on self-directed learning and personalisation as well as more detailed results from the small-scale pilot that evaluated the Atomic Structure interactive personalised virtual lab.

1.2 Chapter Structure

The rest of this chapter is organized as follows. Section 2 discusses recent related works on self-directed learning, personalisation and virtual labs. Section 3 presents the Atomic Structure interactive personalised virtual lab. Section 4 presents the research methodology for the evaluation study. Section 5 presents the results analysis in terms of learning, motivation and usability. Section 6 discusses the main findings and limitations of the study and concludes the paper.

2 Related Work

2.1 Self-directed Learning

One of the innovative aspects of the Atomic Structure interactive virtual lab is that it integrates personalisation based on student’s level of self-directed learning. Self-directed learning (SDL) is not a new idea, with the Greek biographer and essayist Plutarch stating in the first century AD that: “the mind is not a vessel to be filled, but a fire to be kindled”. While there are various definitions, the most cited and well-known is the one proposed by Malcolm Knowles in his book “Self-directed learning: a guide for learners and teachers” [34]:

“In its broadest meaning, self-directed learning describes a process in which individuals take the initiative, with or without the help of others, in diagnosing their learning needs, formulating learning goals, identifying human and material resources for learning, choosing and implementing appropriate learning strategies, and evaluating learning outcomes.” [34]

As summarised in [56], self-directed learning involves several different activities:

  • Setting own learning goals;

  • Identifying appropriate learning resources;

  • Selecting appropriate learning strategies;

  • Selecting important from unimportant;

  • Integrating material from different sources;

  • Time management;

  • Monitoring achievement of learning outcomes;

  • Monitoring effectiveness of own study habits.

While self-directed learning was commonly associated with adult learning and third level education [19], this is also increasingly applied to other levels of education, such as secondary education [2], vocational education [11], and primary education [15]. Self-directed learning is considered especially important for online learning environments where there is a separation between teacher and student. Online environments provide learners with an increased level of control, and also influence learner’s perception of their self-direction [52].

Various frameworks for self-directed learning were proposed in the literature. According to the framework proposed by Tan and Koh [54], self-directed learning can be applied both in schools and out-of-school contexts. Moreover, the learning experience in each context can be either structured or unstructured. Song and Hill [52] have proposed a conceptual model for understanding self-directed learning in online environments that incorporates SDL as a combination of personal attributes and a learning processes.

2.2 Personalisation

Personalisation is a key factor in modern education, as the differences between learners are now widely recognised by both researchers and educators alike. Personalisation is also one of the biggest current trends in the e-learning industry [16]. There is an increasing need for personalised and intelligent learning environments as learners differ in levels of knowledge, motivation, and have a variety of learning styles and preferences [26]. Personalised learning can support individual learning and further engage learners in their studies. Gaps between slow and fast learners are consistently emerging, and teaching to cater for these differences between students is being noted as one the most challenging aspects by science educators [32, 35].

There is a call for personalisation to be implemented into modern pedagogies, in order to meet the needs and interests of different types of learners [4]. In Technology Enhanced Learning (TEL), personalisation is one of the key features, and can assist in bringing the focus of the learning experience to the student instead of the teacher [57].

Personalisation can be implemented at learning system or course-level, or specifically at a content-level. At course level, several intelligent learning systems have implemented localisation and internationalisation [28, 33, 50]. Other systems enable educators and course organisers to add, develop and choose their own specific modules and courses [14, 33, 50]. Learning paths-based personalisation enables provision of content in a manner most suitable for each student. User modelling is also commonly used as a personalisation attribute in a learning system, as gathering data about leaner enables the system to adapt to the learner. Innovative pedagogies are also implemented as part of personalisation, and include e.g. self-directed learning (SDL), game-based learning and inquiry-based learning.

Content level personalisation is more common and utilised in many systems at different levels. The different types of content-level personalisation include gamification, feedback-based personalisation, variation in levels of difficulty, and innovative pedagogies (learning loops, special education, learning paths-based personalisation, user-model based personalisation).

2.3 Virtual Labs

While many virtual labs were developed over the years, most of them targeted third-level education rather than secondary school education, although universities typically have more resources and better physical laboratories and equipment. Moreover, this is despite the fact that learners’ disengagement from the STEM area starts during secondary level education in many countries when students start choosing which subjects they wish to pursue [1, 8].

Table 1 presents a summary of some existing virtual labs and platforms. Several European projects have focused on virtual labs. The Go-Lab project [33] has created a platform that enables educators to host and share with other users virtual labs, apps and inquiry learning spaces. The VccSSe project [27] created a virtual community collaborating space for science education that provided virtual labs and training materials in physical laws including simulation-based exercises. The GridLabUPM [22] platform hosts a number of virtual laboratories that offers students practical experiences in the fields of electronics, chemistry, physics and topography. The BioInteractive [30] platform provides science education resources including activities, videos and interactive media (i.e., virtual labs, click & learn, interactive videos, 3D models, short courses). Other virtual labs/platforms include the Gizmos mathematics and science simulations [21], Chemistry Lab and Wind Energy Lab [38], ChemCollective [60], Open Source Physics [13], and Labster [53].

Table 1. Summary of existing virtual labs and platforms [24].
Full size table

Most of these virtual labs offer simulation-based exercises, interactive activities and online tutorials to assist the student in their learning journey. The online tutorials and the multimedia educational resources are suitable to present the theoretical aspects, while the interactive activities and simulation-based exercises are important in achieving the practical skills and in understanding the phenomena/concepts. While virtual labs offer students a chance to practice their all-important practical skills in a safe environment, most virtual labs lack personalization and adaptation features, and neglect inclusive education. Many virtual labs have also been criticised for over simplification of experiments, with the result that students do not learn all the necessary skills associated with specific exercises.

A number of research studies have conducted evaluations of virtual labs. Aljuhani et al. [3] evaluated a chemistry virtual lab in terms of usability and knowledge improvement. The virtual lab was found to be an exciting, useful, and enjoyable learning environment during user trials. The main drawbacks of their study were the low number of participants and the lack of control and experimental group. Migkotzidis et al. [38] evaluated the Chemistry and the Wind Energy Lab in terms of usability, adoption, and engagement with the virtual labs. The participants expressed a positive opinion regarding the virtual lab interface and high engagement rates. Bogusevschi et al. [10] evaluated a virtual lab with 52 secondary school students in terms of learning effectiveness. The results had shown a statistically significant improvement in the experimental group using the virtual lab as compared to the control group learning using classic teacher-based approach. Bellou et al. [6] did a systematic review of empirical research on digital learning technologies and secondary Chemistry education. The results of the review of 43 studies had shown that the researchers were mainly interested in the chemistry topics and to use digital learning technologies for visualisation and simulations but not in personalising the learning journey.

Despite much research and development in the area, there still is a lack of personalised virtual labs and a need for more comprehensive evaluation studies that look at the impact of virtual labs from multiple dimensions such as learner knowledge, motivation and usability. This study contributes to the area of research through a comprehensive multidimensional evaluation study of the Atomic Structure interactive personalised virtual lab with secondary school students.

3 Overview of the Atomic Structure Virtual Lab

Atomic Structure is an interactive personalised virtual lab for secondary levels students, that teaches abstract scientific concepts such as the structure of atoms, bonding of molecules, gaining and losing electrons, that can be hard for students to grasp, and difficult for teachers to present with traditional teaching materials [23, 37]. The Atomic Structure virtual lab places the student in the centre of the learning experience by implementing personalisation at various layers.

The pedagogical foundations of this virtual lab are self-directed learning, learning in flow, and inquiry-based learning. These innovative pedagogies are beneficial for enabling learners to carry out their own experiments, analyse and question, and take responsibility for their own learning [59], while personalisation makes the learning experience an individual one and keeps the learner engaged.

Figure 1 shows the models built into the Atomic Structure virtual lab to enable personalisation and adaptation. The virtual lab covers concepts such as: atoms, isotopes and molecules. The learning path is guided by the Curriculum Model structure and organisation. For example, a student can only start the isotopes part of the virtual lab when they meet the prerequisite of completing the atoms.

Fig. 1.
figure 1

Adaptation and Personalisation input models: Pedagogical Model, Curriculum Model, Content Model and Learner Model [24].

Full size image

The Content Model contains various learning materials and contents available in the virtual labs: instructional content with videos, e-assessment, interactivity where students can create and perform their own experiments through inquiry-based learning. The Learner Model is updated during the entire learner journey and includes information about the learner knowledge, level of self-directness, motivation (confidence), and special education needs.

Personalisation in the Atomic Structure virtual lab is implemented at different levels throughout the entire learning journey. The levels of personalisation include:

  • learning loop-based personalisation;

  • feedback-based personalisation;

  • innovative pedagogies-based personalisation (inquiry-based learning, learning in flow, and self - directed learning);

  • gamification-based personalisation;

  • special education needs-based personalisation (e.g., sign language translation for hearing impaired students as shown in Fig. 2).

    Fig. 2.
    figure 2

    Atomic Structure video with embedded avatar for sign language translation.

    Full size image

Student’s levels of motivation and self-directness are determined at the beginning of the lab by asking them to answer few questions displayed on the screen. These are used to personalise the difficulty level of questions they receive in the quizzes, what types of atoms, isotopes and molecules they are given to build, as well as what type of feedback they will receive. For example, low and medium motivated students are restricted to atoms, isotopes and molecules which have been deemed suitable to each of those levels, and highly motivated students have access to more complex atoms, isotopes and molecules.

Figure 3 illustrates the process of building an atom of boron with the Atomic Structure virtual lab. The inquiry-based learning phase is offered at the end of each of the three stages in the form of interactive atom, isotope and molecule builders.

Fig. 3.
figure 3

Building an atom of Calcium in the interactive atom builder.

Full size image

Once the students master building the suggested objects, they can freely choose their own objects, and experiment further within the atom, isotope and molecule builders. The Atomic Structure virtual lab also includes gamification elements such as award badges for completing different stages (see Fig. 4).

Fig. 4.
figure 4

Gamification badge awarded for completing the Isotope stage.

Full size image

4 Pilot Study Methodology

This section details the research methodology for the case study conducted with the aim to evaluate the Atomic Structure virtual lab in secondary schools.

4.1 Student Profile

A total of 78 secondary level students from two schools in Ireland have participated into the study. The students were divided in a control group and an experimental group. The wide majority of students (i.e., 69 students) were in the 13–15 age group, 6 students were in the 16–18 age group, and 3 participants did not indicate their age group. The control group had 36 students (23 boys, 11 girls, 2 did not respond) and the experimental group had 42 students (26 boys, 15 girls, 1 did not respond). Students from the control group attended a traditional teacher-led classroom while the students from the experimental group studied by using the Atomic Structure virtual lab on computers in the classroom. The control group was also exposed to the Atomic Structure virtual lab after the evaluation study.

4.2 Study Workflow and Procedure

The evaluation of the Atomic Structure virtual lab was done following the multi-dimensional methodology for pedagogical assessment in STEM technology enhanced learning [43]. The dimensions assessed were: learning outcome, motivation and learner satisfaction (usability-based). The assessment procedure used for the study is illustrated in Table 2.

Table 2. Assessment procedure [24].
Full size table

A description of the research study was given to participants, and consent and assent forms were collected before the actual study. Pre-learning experience surveys were given before and after the learning experience. The pre-surveys included: demographics questionnaire, knowledge pre-test and learner motivation pre-survey for both the control and experimental group. The learning experience of the experimental group was a personalised learning journey through Atomic Structure virtual lab, while the learning experience of the experimental group was traditional teacher led-class session. Knowledge post-tests and Learner motivation post-survey were given to students from both experimental and control group. Furthermore, the experimental group filled in a usability survey.

The knowledge tests contain both multiple choice and input answer questions. Learner motivation was assessed through dimensions such as interest, self-efficacy, engagement, positive attitude and enjoyment. Interest was assessed through Linkert scale interest question [41, 49], self-efficacy (confidence) was assessed following Bandura’s guidelines [5], while engagement, positive attitude and enjoyment was assessed using a 5 point Likert scale [29]. The usability survey contained questions related to four dimensions (usefulness, ease of use, ease of learning and satisfaction), as well as questions where students were asked to rate tow much they liked different features on the Atomic Structure virtual lab on a 5-point Likert scale, as well as open answer questions to indicate the top three things they liked, top 3 things they didn’t like, and if they have any comments or suggestions.

5 Results

5.1 Use of Technology

In the demographic questionnaire, students were asked to report their frequency of use of technology such as smartphones, computers and video games. The results presented in Fig. 5 show that most students are frequent users of technology. The majority of students have access to a PC or laptop at home (81% for experimental group and 83% for control group). The percentage of students that use a smartphone every day is even higher, with 98% for the experimental group and 100% for the control group. A higher variety in student’s answers can be observed when looking to see if they play computer games on a gaming console.

Fig. 5.
figure 5

Student’s use of technology.

Full size image

5.2 Learning Assessment

An analysis of the pre-test and post-tests knowledge was conducted to investigate the impact of the Atomic Structure virtual lab on students’ learning outcome. This analysis excluded the participants that did not answer any question of the pre-test and/or post-test. This approach was treating the participants as absent rather than awarding them a score of 0, which would not be a correct representation of their knowledge level. Participants with true 0 for pre and/or post-test (i.e., answered all questions wrong), were not excluded from the analysis. 11 participants were excluded from the control group and 2 participants were excluded from the experimental group. As such, the pre and post-test scores of 25 participants from the control group and 40 participants from the experimental group were considered for the learning outcomes analysis.

Figure 6 presents the average correct response rates for the control and experimental groups on the pre and post knowledge tests.

Fig. 6.
figure 6

Learning results in terms of mean correct response rates for the two groups [24].

Full size image
Table 3. T-test results between pre and post-tests.
Full size table

The experimental group had a mean correct response rate of 53.5% for pre-test and 75% for post-test, which results in a 21.5% increase. The results of a paired t-test for dependant groups showed that the post-test results were statistically significant higher than the pre-test results for the experimental group at α = 0.05 significance level (see Table 3). The control group had a mean correct response rate of 48% for pre-test and 60% for post-test, which results in a 12% increase. The results of a paired t-test showed that the post-test results were statistically significant higher than the pre-test results for the control group at α = 0.05.

The experimental group had 5.5% higher correct response than the control group for pre-test, and 15% higher for post-test. The results of a t-test for independent groups showed that the experimental and control groups had statistically equivalent pre-test score at α = 0.05 (t(51) = 0.938, p = 0.353). However, the post-test results for the experimental group were statistically significant higher than for the control group at α = 0.05 (t(38) = 2.051, p = 0.047).

5.3 Motivation Assessment

An analysis of the learner motivation and affective state questionnaires filled by the students before and after the session was conducted to investigate the impact of the Atomic Structure virtual lab on students’ motivation. This analysis excluded the participants that did not answer all the questions (i.e., 4 participants from the control group and 3 participants from the experimental group). The data from 31 participants from the control group and 39 participants from the experimental group were considered for the learner motivation analysis.

Figure 7 presents the motivation analysis results. The percentage of students answering that they are very or extremely interested in science classes has increased between the pre and post-session questionnaires with 18% for the experimental group and with 9.6% for the control group.

The percentage of students answering that they are very or extremely confident in being able to solve science problems and challenges has increased with 28.2% for the experimental group and with 6.5% for the control group.

The percentage of students answering that they are very or extremely engaged in science lessons has increased with 30.8% for the experimental group and with 9.7% for the control group.

Fig. 7.
figure 7

Increase in percentage of learners with high ratings for different motivation dimensions between the post and pre-session questionnaires [24].

Full size image

The percentage of students that agreed or strongly agreed that they felt positive during science classes has increased with 20.5% for the experimental group and with 3.3% for the control group.

The percentage of students that agreed or strongly agreed that science classes are interesting has increased with 20.5% for the experimental group and with 6.5% for the control group.

The percentage of students that agreed or strongly agreed that they enjoy science classes has increased with 10.3% for the experimental group but did not change for the control group.

5.4 Usability

An analysis of the learner usability questionnaire completed by the experimental group after interacting with the Atomic Structure virtual lab was also conducted. 5 participants were excluded from this analysis as they did not answer all the questions, thus the data from 37 participants from the experimental group were used.

The results analysis showed the following main findings:

  • 68.5% of students provided agree or strongly agree ratings and 11.7% of students provided disagree or strongly disagree ratings on usefulness dimension;

  • 71.2% of students provided agree or strongly agree ratings and 18% of students provided disagree or strongly disagree ratings on ease of use dimension;

  • 81.1% of students provided agree or strongly agree ratings and 6.8% of students provided disagree or strongly disagree ratings on ease of learning dimension;

  • 60.4% of students provided agree or strongly agree ratings and 13.5% of students provided disagree or strongly disagree ratings on satisfaction dimension.

Figure 8 shows the percentage of users that indicated that they liked or loved the different features/technology of the virtual lab as follows: 86.5% for videos, 83.8% for quiz and reminder of correct answer after the quiz, 73% for feedback after the quiz, 64.9% for atom builder, isotope builder and receiving badges, and 75.7% for reading facts about atoms and isotopes.

Students also provided subjective feedback. As part of the negative aspects, they mentioned the fact that the atom and isotope builders “took a while” to load and were “sometimes slow”, or “it was slow loading the build atom game”. One student reported that had to “load the page as it didn’t work”. Students were using school’s computers and internet connection. Another area for improvement suggested by students was to add more “examples or instructions to do the exercises”.

As part of the positive aspects, they mentioned “it is easy to use”, “it is fun”, “it was interesting”, “gets to the point”, “you can do it yourself”. They reported on their perceived learning as well: “I have a better understanding of it now”, “my knowledge of the topic has improved”, “the videos helped me to learn by hearing”, “I liked the quiz as I could see for myself what I had learned”, “it helps you understand easier”, “I liked how easy it was to understand.”

Fig. 8.
figure 8

Percentage of learners that liked/loved the different features of the Atomic Structure virtual lab [24].

Full size image

6 Conclusion

The use of technology in education is increasing at a fast pace, with many innovative technology-enhanced learning solutions being implemented both in online and face-to-face learning. The NEWTON EU H2020 project focused on supporting and improving STEM education by developing a large-scale platform that integrates many innovative technologies such as virtual labs, remote fabrication labs, augmented and virtual reality, gamification and game-based learning, adaptive multimedia and mulsemedia, platform and content level personalisation, and innovative pedagogical approaches such as flipped classroom, game-based and problem-based learning. In particular, virtual labs are an effective solution to provide experiential learning not only to online students but also a cost saving alternative to physical labs.

This paper presented results from a pilot study that was conducted as part of the NEWTON project to evaluate the Atomic Structure virtual lab with secondary school students. Most existing virtual labs lack personalisation and adaptation. As compared, the Atomic Structure is an interactive personalised lab that integrates multiple layers of personalisation based on the learning loop, feedback, innovative pedagogies, gamification and special education needs. The evaluation followed a multidimensional approach assessing the virtual lab’s impact on knowledge achievement, learner motivation, and usability dimensions.

The results analysis showed that experimental group students that learned with the Atomic Structure virtual lab, achieved statistically significant higher knowledge than the control group students that attended the traditional teacher-led session. The Atomic Structure virtual lab had a higher impact than traditional learning on increasing learner’s motivation across different dimensions such interest, confidence, engagement and enjoyment. Moreover, most students have provided high ratings for the different usability dimensions (usefulness, ease of use, ease of learning and satisfaction), and liked/loved the features/technology for the virtual lab.