Keywords

1 Introduction

Providing feedback is an important component of just about all learning contexts. This is even more apparent for online learning systems, where students interact remotely with their course and participate in synchronous and asynchronous modes of learning. In particular, feedback has been strongly related to student achievement and improve the self-awareness, enthusiasm and motivation of students for learning [1, 2]. It is also reported that students have a higher change to become engaged after receiving an intervention [3]. In general, there is a great variety of different types of feedback from formative to summative, immediate to delayed, and which can have both positive and negative effects on learning [4]. As a result, it is essential for online learning systems to provide appropriate feedback mechanisms and strategies that monitor the behaviour of students and automatically provide interventions to support their learning and engagement with the course. In addition, personalised feedback aims to provide students with information and instructions that are most suitable to their learning needs and to the problems they have with their course.

In this paper we present AMASE [5] – a framework that monitors the behaviour of students in an online course and which accordingly triggers automated interventions. In this case, a learning course is perceived as a learning activity that combines in a unified manner learning content (that students have to study) and tasks (that students have to perform). The learning activity is generated and personalised by the framework according to the user’s prior knowledge, preferences and needs. Advanced monitoring mechanisms are also used to capture and analyse the learner’s behaviour and interactions with their course. As a result, we can determine the level of engagement and progress of students, if they struggle with a specific task as well as to identify potential problems with their course. Accordingly, AMASE will trigger suitable interventions that are personalised to student’s needs, and which will encourage students to follow the provided recommendations. These motivations take the form of interactive feedback (requiring student input), advices (providing informative instructions), reminders (about events or deadlines) and emails. We believe that such forms of personalised interventions can guide and stimulate students in their online course.

The remainder of the paper is structured as follows: Sect. 2 describes related approaches of how personalised interventions have been used in the field of technology enhanced learning (TEL) and how they may have affected the behaviour and performance of students with their online course. Section 3 presents the AMASE approach and framework to provide automated and personalised interventions to students. Section 4 describes an authentic learning environment where students study an SQL database course and receive interventions. Section 5 investigates how the provided interventions have leveraged the learning behaviour of students. Finally, Sect. 6 summarises the main contributions of our approach.

2 Related Work

Learning Management Systems (LMS) have had a large impact on learning and teaching trends in higher education [6]. Specifically, Personalised Learning Environments (PLE) provide more effective delivery of courses as they can enhance the learning experience of students through tailored content [5]. Widely used LMS are based on asymmetric interactions. In this case, the learner is not prompted to login or engage with their course activities. Instead a level of self-discipline is required and this may become a problem for students that need direction [7]. LMS such as Moodle, Blackboard, Edmodo and others enable instructors to provide learners with feedback and interventions. However, the form of feedback is quite limited. One type of feedback consists of a text area where instructors can provide feedback based on some activity that the learner has completed. Blackboard for example allows instructors to provide learners with feedback for assessments. Another category of feedback supported by popular LMS includes pre-defined feedback which students can see immediately [8]. These assessments are usually in the form of quizzes with answers provided by the instructor and the LMS provides feedback based on the student’s answer. Edmodo, for example, allows instructors to create assessment consisting of multiple-choice questions, fill-in the blanks, essay based answers and true/false questions. With the exception of essay based questions (which requires the instructor to correct), the remaining questions can have pre-configured answers provided by the instructor. This allows Edmodo to instantly provide users with feedback as they answer quiz questions. Although this type of feedback is immediate, it is not adapted to learners nor does it support continuous improvement during the learning process [9].

Vasilyeva et al. [10, 11] discuss the usefulness of presenting learners with elaborated feedback showing correct answers and providing additional information such as corresponding learning materials and explanations. Their study provides adapted feedback using the learner’s answers and certitude (answer certainty provided by learner). The evaluation of their study showed that students provided more positive than negative responses about the feedback that was directly shown to them or recommended to them. Lubega et al. [9] discuss the importance of tailoring feedback to the individual learners by monitoring their learning process and assessment results. OFES [8] is a web-based tool that allows instructors to construct student feedback for specific assignments through a feedback template form. The template allows instructors to enter comments and performance related to assessment criteria for individual students. The personalised feedback is provided to learners through a personal feedback space, and graphics showing emotions are included in the feedback to attempt to motivate students. The evaluation of OFES showed that the general consensus amongst students was that the feedback was motivational.

From the related work, it can be seen that feedback and interventions are important aspects of the students’ learning experience and most LMS and PLE integrate some form of feedback. However, the feedback is at the discretion of the instructors, where the instructor must provide either through quizzes with pre-configured answers or after an assignment or assessment is completed by the student. In the case of the former, the feedback can be limited; however, it is immediate, automated and can allow some form of personalisation. In the case of the latter, the feedback can be personalised and made motivational for the learner, however, its timeliness is dependent on the instructor and it is not automated. Four feedback factors have been discussed [12] which focus on promoting learning and engagement. These include timeliness (immediate feedback is most effective), motivational (constructive), personalised (in line with students’ goals) and manageable (easy to interpret).

AMASE integrates all four factors into its design. In summary, the AMASE framework has been specifically designed to monitor and analyse complex student behaviour and trigger different types of motivations dynamically. The motivations can be triggered instantly, on specific time or interval, or upon the instructor’s request. The motivations are elaborate and can take the form of interactive and constructive feedback, informative advices and reminders. In addition, based on student’s responses they can be escalated to complex dialogs. They are also personalised to the learning needs, preferences and context of students.

3 AMASE: A Framework for Monitoring and Providing Automated Interventions to Students

AMASE provides a highly dynamic and adaptive framework for the automatic composition, assignment and enactment of personalised learning activities to students [5]. In this context, a learning activity is considered to be an educationally-driven sequence of learning content that the students have to study and user-centric tasks that the students have to perform. In addition, AMASE provides advanced monitoring mechanisms to capture and analyse the learner’s behaviour and interactions with their learning environment and course. As a result, we can determine the level of engagement and progress of students, what resources they are using, how much time they spend with specific activities, if they struggle with a specific task as well as to identify their potential problems with the course. Accordingly, the engine will trigger suitable personalised interventions to guide, assist and motivate the students. These motivations take the form of interactive feedback (requiring student input), advices (providing informative text and instructions), reminders (about events or deadlines) and emails (see Fig. 1). We believe that such forms of personalised interventions can provide effective student guidance and feedback as well as stimulate and sustain their engagement in the online course. Finally, the interventions are automatically triggered by the system and sent to students at specific periods (see Fig. 4) and are personalised according to their current progress and engagement with the course.

Fig. 1.
figure 1

Personalised interventions are automatically sent by the system to provide feedback to students as well as to promote their motivation and engagement.

Full size image

4 An Authentic Learning Environment

In order to apply and evaluate our research approach and framework, we created an authentic learning environment, where undergraduate students study a personalised SQL database course. On the background, the learning environment utilises the AMASE framework to compose for each student a personalised version of the course, monitor and analyse their behaviour and interactions with the course as well as to automatically trigger appropriate interventions when it is considered necessary. In this case, the students interacted with their course for a period of 3 months (that is a semester) via the online learning environment (see Fig. 2). This is a type of a blended course, where students also participate in lectures and use the online course in their own time for additional content study, practical activities and course support.

Fig. 2.
figure 2

Learning environment with personalised SQL database course.

Full size image

As seen the SQL database course is perceived as a learning activity that is assigned and customised to students. As an example, the learning activity could be described by the following sequence of steps, see Fig. 3. Initially, in the online course we assess the prior knowledge of students on database concepts and accordingly the students are assigned specific content to study based on the evaluation of their learning needs. In addition, as the students’ progress with their course, they receive a number of tasks that they have to perform in order, either individually or as part of a team and in parallel to their content study. More specifically, first the students are provided with an online SQL environment in which they have to practice and test their SQL skills. In the next phase they will be automatically allocated to groups by the framework and assigned a group activity in which the students will have to find appropriate resources for a given topic and create an audio presentation that will be uploaded to a YouTube channel. Next, the students will be assigned a course project with specific design and implementation instructions and related supporting tools. Finally, the framework will automatically assign three projects to students that were submitted on the previous activity by their peer-students for review.

Fig. 3.
figure 3

A sample learning activity (path) for the SQL database course.

Full size image

As a result, the SQL course is split into two (2) main and parallel parts, the “Study Content” in which the students study their assigned content and another in which they have to perform their practical activities (see Fig. 4). The practical activities are performed in order and corresponding to the following 5 subsequent course phases; (1) the “Practice”, (2) the “Web Quest”, (3) the “Presentation”, (4) the “Project” and (6) the “Peer Review” phases. In each period, the students receive personalised interventions (notifications) regarding their level of engagement and progress with the system.

Fig. 4.
figure 4

SQL course phases and interventions.

Full size image

5 Evaluating the Role of Interventions on Student’s Behaviour

In this section we investigate how the automated interventions sent by the system may leverage the learning behaviour of students. The main aim of automated interventions is to guide students with their learning as well as to promote the motivation and engagement of students with the course.

5.1 Student Usage of the Interventions

In the course, the personalised interventions are sent to all students both by email and a message appearing on their learning environment. In general, the students responded well to their interventions by taking on board the advices and working through their assigned tasks. For example, upon the interventions that were related to the practice phase, 72% of the students took the advices (and responded back). On the remaining 28% of the students, 82% of the students read the advices but didn’t respond back, where the remaining didn’t read or notice their advices.

5.2 Analysing the User Behaviour upon the Received Interventions

This section analyses in more detail how the students reacted once they received the automated interventions from the system. In this case, the interventions sent were assessing the current level of engagement of students. As a result, Fig. 5 depicts the student interactions over the entire course period and in particular over the different course phases and interventions sent to students. In the graph, it is even visually apparent that almost all big spikes were caused by interventions sent during the related course period. For example, once the interventions for the practice phase were sent on day 12, the overall student interactions escalated rapidly and increased from the previous peak (3663) to 11982 (that is 3.27 times higher). Similarly, although not so rapid, was the increase of user interactions for the interventions sent on the web-quest, presentation and review periods. Finally, less but quite noticeable was the increase of interactions for the study content and the project periods.

Fig. 5.
figure 5

Student interactions upon the received interventions.

Full size image

Similarly, Fig. 6 depicts the student duration (study/work time) over the entire course period and in particular over the different course phases and interventions sent to students. As before, it is apparent that once the interventions for the practice phase were sent on day 12, the overall student duration escalated rapidly and went from the previous peak (166244 sec) to 528249 sec (that is 3.18 times higher). Similarly, although not so big was the increase on student’s study time once we sent the interventions for the web-quest, presentation and review periods. Also, the increase for the study content and the project phases was more noticeable this time. This is because, in our case the duration is regarded as a more reliable metric of the student behaviour than the interaction with a course material.

Fig. 6.
figure 6

Student duration upon the received interventions.

Full size image

Next, in Fig. 7 we depict how the overall level of engagement of students developed over the entire course period and in particular in relation to the interventions sent to students. As before, the overall engagement of students was increased in all cases, however in some course periods the interventions had more impact (see practice and review period). As a result, it appears that the level of increase depends on specific design requirements of the course, for example the nature, volume and weighting of learning content and tasks assigned to students at each course phase.

Fig. 7.
figure 7

Student overall engagement increase upon the received interventions.

Full size image

Finally, in Fig. 8. we depict how the overall level of engagement of students developed over time for the “Practice phase” and compare against a previous year where interventions were not used. In this case, the first intervention took place quite early in order to prepare and alert the students, whereas the second took place later in order to inform the students regarding their current level of engagement. Before the first intervention, the overall level of engagement of students was quite low at 1.62%. Almost at the same time, the year with no interventions the level of engagement of students was 10.42%. Once the first advice was sent, the overall level of engagement increased from 1.62% to 9.73%. At the same time, the level of engagement of students the year with no interventions was approximately 13.39%. As a result, it appears that during this period their difference in engagement had already dropped from 8.8% to 3.66%. Similarly, once the second advice was sent, the overall level of engagement increased rapidly within the following few days, from 9.73% to 52.07%. On the other side, in the corresponding period the year with no interventions, the level of engagement was only 23.74%. If we compare the two periods it appears to have a considerable increase of 28.33% the year where the interventions were used. In particular, the year with no interventions it took students 8 more days to reach a similar level of engagement (54.86%). Subsequently, we believe that personalised interventions can provide the right stimulation to increase the level of engagement of students within a short period of time.

Fig. 8.
figure 8

Comparing student’s engagement with and without interventions during practice phase.

Full size image

5.3 Student’s Perception on Interventions and Feedback

Finally, in Fig. 9 we present the results of a questionnaire of how students perceived and experienced the interventions provided to them. The results are based on a feedback that we received from 65 student replies. The 4 questions that fell into that category are the following:

Fig. 9.
figure 9

Student’s perception on received interventions.

Full size image

  • Q1: “The system provided me with interventions throughout the online course and during different stages”. In this case the majority of the students (82.82%, that is 60.94 + 21.88) responded positively to that statement.

  • Q2: “The system provided me with interventions about specific course content and tasks at the right times”. In this case 56.25% of the students found the timing of the interventions to be correct and appropriate, However, another considerable proportion (26.56%) disagreed (feel they wanted more control).

  • Q3: “The interventions send by the system helped me to focus and meet the course deadlines”. In this question, the majority of students (53.97%) found the motivations send by the framework help them to meet their course deadlines. However, another considerable population (33.33%) had a different opinion, suggesting that they desired more notice or no notice at all (feel they didn’t need to).

  • Q4: “I found the interventions send by the system distracting and inappropriate”. In this case most of the students (49.23%) found the interventions not distracting and inappropriate. However, another considerable population (32.31%) disagreed.

6 Conclusions

In this paper we presented the AMASE approach and framework which was used to monitor and leverage the online learning behaviour of students while studying a database course with personalised interventions. The aim of interventions was to provide students with automated feedback and guidance as well as to motivate their engagement with the course. In overall, our findings shown the students reacted very positively to the received interventions and different aspects of their online learning behaviour had been considerably increased (leveraged) such as the number of times students interacted with learning resources, study time (duration) and their level of engagement with the course. Finally, the students perceived the received interventions and feedback as very useful to focus on different parts of the course. Even so, there was some students denoted that they would preferred to feel more autonomous and less interrupted to their course. Preferably, for these students it would be better to have the ability to control when the interventions should be triggered.