Keywords

1 Introduction

In online learning environments, students’ active involvement in the learning process is lower than traditional classroom settings, and hence they tend to get demotivated, leading to high dropout rates [1]. Such learning environments generate a large amount of logged data through student interactions with learning resources, discussion forums and course related quizzes. This data often contains valuable information that can highlight resource usage trends, reveal learner engagement levels and present peer comparisons. One of the aims of Learning Analytics is to present this data to learners in a consumable format to help motivate students to enhance their engagement with course content [2]. In addition, it is important that the logged data be displayed in a form that guides learners to understand and explore it, and thus support self-reflection [6]. One way of addressing this issue is to provide learners with visual stories that can be navigated and explored to help make sense of this data.

Information visualization provides a simple means to understand digitized data as it maps data attributes to visual properties [3]. Mapping appropriate visualizations to datasets or segments of data is common in visual analysis and visual exploration tools [4, 5] where a chosen dataset or a selected segment of data is automatically visualized. Storytelling in information visualization or visual narrative can be defined as an ordered sequence of steps consisting of visualizations, which are linked or connected to make the communicated message more memorable [7]. Visual narratives also provide effective ways of highlighting facts, making points and passing on information [8]. In addition, visual data exploration supports end users in gaining a better understanding when little is known about the data and exploration goals are vague [9]. Visual drill downs allow end users to explore details behind the presented visualizations [10] as they render a more detailed view of a specific element within the visualization that is under examination.

This work aims to evaluate the effectiveness of explorable visual narratives in supporting learners to engage with online course content. In this work, supporting learners to engage refers to encouraging students to study course content, work through activities and submit deliverables within the time allocated. Studies were conducted using undergraduate students in Trinity College Dublin enrolled in the Information Management course. These students were presented with personalized explorable visual narratives. The course was delivered through AMAS [11], a Personalized Learning Environment (PLE) providing educational content and services to learners.

AMAS generates and sends students personalized notifications specifying the individual’s level of engagement with the course and provides advice. Each learner has access to his/her own explorable visual narrative (hereafter, referred to as visual narrative), which uses the individuals logs, gathered by the AMAS PLE, to communicate a personalized message. The message provides the student with explorable visualizations presenting the level of engagement with the course, the time spent on activities and facilitates peer comparisons. The visual narratives are constructed using VisEN (Visual Exploration with Narrative), a generic visualization framework for generating explorable visual narratives. For AMAS, VisEN used student traces logged by the PLE to dynamically generate the visual narratives. In addition to providing personalized visual narratives, VisEN enables learners to drill down and explore data or explore related data to that shown in the view, hence making it an explorable visual narrative. The visual narratives also support anonymized peer comparisons, enabling learners to explore engagement; time spent on, and completed activities by peers. This paper analyzes the effect visual narratives had on the engagement of students using the AMAS PLE. It primarily focuses on students who improved their engagement levels. It was found that the visual narratives gave the majority of students that were engaging poorly with their course a strong motivational pull to engage with the PLE and motivated them to complete assigned activities and submit deliverables on time.

The remainder of this paper is structured as follows: Section 2 discusses related work. Section 3 describes an overview of the VisEN design. Section 4 introduces a use case of a typical student using the AMAS PLE. Section 5 evaluates the effectiveness of explorable visual narratives in enhancing student engagement over two academic years and Sect. 6 presents conclusions.

2 Related Work

Research on the use of Information Visualization in the Technology Enhanced Learning (TEL) domain has largely focused on three areas: Learning Analytics (LA) [2], Open Learner Modelling (OLM) [27] and Educational Data Mining (EDM) [23]. Across these three areas, visualizations have been used as a means of supporting reflection to promote learner motivation and engagement. In LA, visualizations have been used to present student activity data to either instructors to support the monitoring of students, or to students to promote a change in behavior or to both instructors and students [16, 17, 25]. In OLM, visualizations show student competencies and level of understanding using inferences from learner interactions to support reflection and assessment [21, 27]. EDM applies data mining algorithms to logged data to offer learners feedback and promote reflection [23]. The approach adopted by VisEN and discussed in this paper uses student activity data to generate explorable visual narratives to promote motivation and engagement amongst learners. Hence the remainder of this section analyzes systems that fall into the LA domain and specifically those that present logged data to learners in a format that enables self-reflection and behavioral change.

A number of systems [16, 17, 22] that present student traces facilitate exploration in the form of selecting elements within the visualizations and drilling down to view details. For example, SAM [16] provides visualizations of student activities to learners who can monitor what they have been working on and the time they spent on activities. The visualizations are interactive and facilitate drill down by clicking on visual elements, which reveal statistical details regarding resource usage and activity times. The CAM Dashboard [17] consists of goal-oriented visualizations for students to reflect on time spent on assigned activities and progress made towards goals. The explorations supported by the dashboard come in the form of drill down, where clicking a bar showing weekday activity numbers updates two visualizations to present actions and activities for the selected day. The StepUp dashboard [22] provides visualizations of student traces to promote reflection amongst learners. The dashboard contains sparklines showing overall activity per learner. Students can interact with the dashboard and explore a breakdown of their activities by clicking on the sparklines to load bar charts showing the time spent on activities and the distribution of forum posts.

From the above, it can be seen that there is a focus on providing visualizations that learners can interact with and explore to uncover details to support self-reflection. However, the exploration is predominantly selecting elements within the visualization and drilling down to explore details. VisEN supports visual exploration in two forms: the first is through drill down, where the student clicks on an element within the visualization, which loads a visualization showing details directly related to the element selected. The second form of exploration is manifested through links at the bottom of the visualization which, when clicked load data related to the data shown in the visualization. For example, if a visualization presents interaction timings showing individual student timings, then a related exploration could show the timings of the top engaging learners. In addition, the visualizations are offered through a narrative to enhance guidance and help make the communicated message more memorable [7]. Previous work has shown that visual narratives support students engage with study content in online learning environments [24].

While there are some indications [12] showing that students can at times be averse to peer comparisons, several systems [16, 17, 25] show that they can support learner motivations and engagement by allowing students to compare their own progress with peers. For example, Moodog [25] is a Moodle plug-in that uses the Course Management Systems logged data to visualize students’ learning activities. It presents comparisons between students such as the number of times course material was viewed, the time spent on the course and forum activity. The eMUSE platform [26] provides learners with an integrated learning space to access a number of web tools, some of which include Twitter, Blogger and YouTube. It provides students with visualizations to compare their involvement and interactions with the tools against peers. TrAVis [13] provides learners with visualizations showing student tracking data on Computer Mediated Communications such as discussion forums, chat and blogs. Students can view peer comparisons at a number of communication levels through spiral graphs showing learner and peer activities. The systems facilitating visual exploration, discussed above, also support peer comparisons. For example, SAM provides a number of visualizations showing the number of documents used, average document usage time and average time spent on activities by peers. The CAM dashboard enables students to view comparisons with fellow learners regarding progress made towards goals. The StepUp dashboard presents the number of posts, time spent on course activities and tweets made by peers.

From the related work, it can be seen that there is also a strong emphasis on enabling peer comparisons amongst learners to support motivation and engagement. VisEN supports peer comparisons by enabling learners to select peer nodes through the visualization and explore detailed comparisons of engagement per activity and time spent on activities. In addition, it supports related data comparisons, which presents student comparisons related to the data shown in the visualization and these are offered through a visual narrative.

3 VisEN Design Overview

Users or narrative constructors use VisEN to build narratives that are automatically complemented with appropriate visualizations. In addition, VisEN enhances the visual narrative through explorations, called derivatives, which present data that is related to each section of the narrative. It supports end users (who are students in this deployment) in analyzing and exploring visual narratives and enables them to gain deeper insights into the visual narrative. This section describes an overview of the VisEN design by describing its architectural components and sub-components, shown in Fig. 1, used to generate explorative visual narratives. The design uses some of the principles discussed in sequencing in visual narratives [18], Template Editor and Shelf Configuration visual interface design approaches [19] and the visualization pipeline [3].

Fig. 1.
figure 1

VisEN architecture

Full size image

3.1 Narrative Builder

The objective of the Narrative Builder is to enable narrative constructors (instructors, in this case) to connect to a data source, analyze the data and author narratives through a web-based interface.

  • The Data Connection component is used to connect to data sources. Narrative constructors specify database connection parameters or select preconfigured data source parameters. For the AMAS deployment, a visual narrative template was created prior to the start of the course using test data, as logged data via student interactions was not available until the course started. Once the course commenced, the visual narratives used logged data generated by the PLE.

  • The Data Analysis Interface displays the source tables and fields using the jQuery Accordion widget. The interface provides a Narrative Canvas with a number of panels to drop data fields and to specify filters. The jQuery Draggable widget is used to enable narrative constructors to drag and drop data fields onto relevant panels to build the narrative. When a field is dropped onto a filter panel, queries are generated to fetch data values and narrative constructors select the values to use in the filter. As data fields are dragged and dropped onto panels, the Visualization Engine is invoked to generate appropriate visualizations and render these to the narrative constructor. Narrative constructors enter captions to complement and explain the visualization, which can then be saved. This typically forms one part of the narrative and is called a narrative slice and several slices can be linked to form a visual narrative.

  • The Data Derivative Builder extracts the data from a constructed narrative slice and generates a number of queries to fetch derivatives of the data used in the slice. It generates mappings using the data source and these are used to create derivative queries. The generated mappings are used to create new queries to produce appropriate visualizations for the explorations. These queries are created by replacing fields and/or filters in the narrative slice queries with those in the mappings. For example, in the case of AMAS, one of the mappings pairs the individual student with the group that she belongs to. In this scenario, whenever a narrative slice presents data about the student, e.g. engagement breakdown by activity, she will have an option to explore the same breakdown for peers in her group.

3.2 Visualization Engine

The objective of the Visualization Engine is to generate a set of appropriate visualizations for the data in the narrative slices and in the derivatives.

  • The Query Builder uses the data fields dropped onto the Narrative Canvas and filter data, if specified, to create and run SQL queries against the connected data source. The query results are formatted to specify the structure of the data, for example, univariate or multidimensional data, data types, etc. This formatting supports the Rules Engine to identify an appropriate set of visualizations for the narrative slice.

  • VisEN uses visualization libraries to source visualization techniques. Research evaluating affordances and characteristics of visualization techniques [14, 15, 20] has shown the suitability of various visualizations against datasets. The Rules Engine consists of XML files (created by VisEN developers), which encode the characteristics and affordances of visualization techniques, which are used to map narratives slices to appropriate visualization techniques. In some cases, a few visualizations are appropriate for a narrative slice, in which case, all the matching techniques are made available to the narrative constructor to choose which one to use.

  • The Visualization Builder generates the visualizations from the matching visualization technique(s) by fetching the templates from the appropriate visualization libraries and populating it using the formatted query results as JSON objects. The populated visualization techniques are then made available to the Narrative Constructor.

3.3 Visual Narrative Explorer

The Visual Narrative Explorer presents the visual narratives to end users (students in this deployment) through a web-based interface. This facilitates visual narrative consumption and tailored exploration.

  • The visual narratives are presented to end users using JQuery tabs, where each tab represents a narrative slice, containing a title, visualization and a caption describing the data. End users are presented with the first narrative slice and they can view and analyze the entire narrative by navigating across the tabs, reading the descriptions and interacting with the visualizations.

  • Derivative explorations are made available to end users to explore data related to the narrative slice. These are accessed by clicking on either elements within the visualization or links below it. When an end user wishes to explore a derivative, she clicks the element or link which fires an AJAX request loading a new visualization with the derivative data in a popup window. The exploration links at the bottom of the narrative slice are tailored to individual preferences. The Exploration Tailoring Engine analyzes how end users interact with the derivative links and infers preferences per individual. Exploration links are ordered by the level of exploration by the user as the engine computes the number of times the data presented in the exploration was viewed and time spent viewing it to determine the order.

4 Use Case

This section describes a use case of a student with typical learning and engagement behavior, and visual narrative usage shown by many learners using the AMAS PLE.

Joe is an Engineering student using the AMAS PLE to study Information Management. During the first month course, Joe has rarely used the PLE. During week 4 of the semester, Joe receives a notification by email informing him of his poor engagement with the course content and advising him of the upcoming deadline.

Joe is determined to improve his engagement and he immediately views his personal visual narrative through the PLE. He navigates through his visual narrative analyzing how he has engaged with the overall course and per activity. He explores the times spent by peers by clicking on activities and compares engagement levels. He also explores the time taken by peers to complete the next three activities that he is assigned. Having analyzed his visual narrative, Joe is motivated to start working through his activities. As he completes his tasks, Joe continuously revisits his visual narratives where he analyzes his engagement score and can see how it is increasing.

5 Evaluation

This section evaluates the impact that the explorative visual narratives had on learners’ course engagement through three studies. Two of these studies use the logged data from students studying the Information Management course at Trinity College Dublin using the AMAS PLE over two academic years (2013–2014 and 2014–2015) and the third study analyzes student feedback. The system, including the visual narratives remained the same over the two years. The course was run for a period of three months (October - December) in each year with a total of over 120,000 student interactions logged over the two academic years. During the three months course, AMAS emailed personal engagement notifications at regular intervals (one every two to three weeks) to students informing them of their level of engagement with the course. These notifications described the learner’s current engagement with the course (bad, poor, good, or excellent) and provided related advice. Course interactions (page clicks), study time (reading and reviewing durations), submissions and questionnaire scores were used by AMAS to calculate engagement. Course activity and grades from the preceding years’ were used to determine thresholds for activities when calculating the level of engagement. The engagement notifications did not explicitly make any references to the visual narratives and interacting or viewing these narratives did not impact engagement scores. The home page of the PLE presented the current engagement score and students had the option to click ‘view my progress’ to load their visual narratives, which amongst other things includes how the engagement score was calculated. This evaluation primarily focusses on students who improved their engagement levels. Specifically those that received a bad or poor engagement notification, and following an improvement, received a good or excellent notification. It analyzes the impact the visual narratives had on encouraging these students to engage and subsequently improve their engagement. Figure 2 shows some of the visualizations from a learner’s visual narrative.

Fig. 2.
figure 2

Selected visualizations from a student’s visual narrative

Full size image

In total 233 students participated in the course over the two academic years. From these, a total of 125 students improved their engagement score from bad or poor to good or excellent. While the remaining students did not improve their engagement to the same degree, a relatively small number showed no improvements in engagement at all.

The first study analyzed the logged data to determine the correlation between students’ course engagement and the number of times the learners revisited their visual narrative before and after receiving a bad or poor engagement notification. This study examined the extent to which the 125 students used their visual narratives to help them achieve a good or excellent engagement score. This study also analyzed the number of unique visual narrative visits by these students before and after receiving a bad or poor engagement notification. The second study used the logged data to assess if the visual narratives were immediately used by any of the 125 learners to understand a bad or poor engagement notification. It also examined the percentage of interactions these students had with their visual narrative versus the total course content interactions following such a notification. The third study assessed the responses to the AMAS course questionnaire completed by students once the course was over. The study examined the responses related to motivational factors and usefulness of the visual narratives.

5.1 Study 1: Visual Narrative Views Before and After Bad or Poor Engagement Notification

In the 2013–2014 academic year, 61 students improved their engagement to good or excellent following a bad or poor engagement notification and in 2014–2015, 64 students showed a similar improvement. Study 1 examined the number of times these students viewed their visual narratives before and after the notification by calculating the spread of the visual narrative revisits during both engagement periods. It also determined the correlation between course engagement and visual narratives visits using the Pearson correlation coefficient for both academic years during the study period leading to an improvement in engagement.

Increase in Visual Narrative Visits.

The boxplot shown in Fig. 3 presents the spread in visual narrative revisits between a period of poor and bad engagement with the course and the subsequent period of good or excellent engagement for the same students. In 2013–2014, A and B in the figure show the increase in visual narrative visits, A showing the number of visits during low engagement and B showing the number of visits during improved engagement. The same holds true for 2014–2015, shown through C and D. The shift from A to B and from C to D clearly shows how these students increased their usage of their visual narratives as their course engagement improved. Study 1 also found that in the 2013–2014 academic year, 78 % of students who improved their engagement to good or excellent showed a minimum of a fourfold increase in their visual narrative views. In the 2014–2015 academic year, this figure was at 72 %. This finding showed that as the students worked through course content and improved engagement, they continuously returned to their visual narratives.

Fig. 3.
figure 3

Visual narrative revisits during low and high engagement

Full size image

Strong Correlation Between Visual Narrative Views and Engagement.

The Pearson correlation coefficient was used to determine if an improvement in course engagement correlated to increased visual narrative visits. In the 2013–2014 academic year, during the period of improved course engagement, the Pearson correlation coefficient was 0.74 and the 2014–2015 academic year, it was 0.76. This showed that an increase in visual narrative views was strongly correlated to an increase in learner engagement.

Finally, students who received a bad or poor engagement notification but did not improve to good or excellent engagement score (total of 86 students) across both academic years were analyzed. Of these students, 44 showed slight improvement in engagement and 37 of these had increased their visual narratives with the improvement in engagement.

5.2 Study 2: Learners’ Immediate Visual Narrative Interactions Following a Bad or Poor Engagement Notification

The second study analyzed how learners, who improved their course engagement from bad or poor to good or excellent, interacted with their visual narratives immediately after receiving a bad or poor engagement notification. It also examined the percentage of interactions these students had with their visual narrative against overall course interactions during the period when the engagement improved. The aim of this study was to determine if learners were drawn to their visual narratives to understand a poor engagement notification and weather these students continually visited their narratives as they worked through course content. It is important to remember that the notifications did not reference the visual narratives. Moreover, interacting with the visual narrative did not affect learner engagement scores, as these were calculated using course content interactions, study times and submitted deliverables.

Visual Narratives as a Means to Understand and Promote Engagement.

This study found that in the 2014–2015 academic year 74 % of learners who improved their engagement to good or excellent executed at least half of their total visual narrative interactions on the first day after reading the bad or poor engagement notification. In the 2013–2014 academic year, this figure was 70 %. In addition, the study analyzed the percentage of these learners that repeatedly visited their visual narratives to understand their course engagement. It was found that at least one fifth of the total interactions with the system were with the visual narratives for 71 % of these students from 2014–2015. The same analysis found that this figure was 65 % for the 2013–2014 students. The study found a similar trend but to a much lesser degree for the students that received a bad or poor engagement notification and only slightly improved their engagement and still remained in the poor or bad engagement category.

From this study, it was concluded that the majority of students who improved their course engagement used their visual narrative as the first place to go to understand the reasons behind a bad or poor engagement notification. It also showed that a minimum of 20 % of overall interactions during the period when their engagement improved were dedicated to their visual narratives. This suggests that the visual narratives had an important role in encouraging and supporting engagement amongst these learners.

5.3 Study 3: Assessing Usefulness of Explorable Visual Narratives Through Feedback

The third study analyzed all the learner responses (including those that did and did not improve their engagement) to some of the questions related to the visual narrative from the AMAS questionnaire completed by students at the end of the course. This study analyzed the responses from all the students highlighting the responses from those who improved their engagement from bad or poor to good or excellent as they interacted with their visual narratives the most. In total 147 students completed the questionnaire, 79 in 2013–2014 and 68 in 2014–2015. Of these students, 85 were from the category of students that improved their engagement as mentioned above. The primary aim was to determine if the visual narratives had a role in motivating learners to engage with course content after interacting with their narratives. Hence this study analyzed student feedback from the following three statements.

Statement 1: The Engagement and Task Duration Visualizations Motivated me to Engage with the Course.

Statement 1 incorporates all the visualization from the visual narrative presented to individual students. The figures in bold are from the students who improved engagement from bad or poor to good or excellent, with 80 % and 77 % strongly agreeing or agreeing from the 2013–2014 and 2014–2015 students respectively. This response confirms the findings from studies 1 and 2, that the visual narratives had an important role is motivating and supporting engagement of students who improved their engagement. The response from the entire class also suggests that the visual narratives motivated learner engagement, despite the fact over 40 % of these students used their visual narrative to a lesser degree (Table 1).

Table 1. Responses of the class and those whose engagement improved to good or higher (bold)
Full size table

Statement 2: It was Useful to Explore Related Data Through Other Visualization.

A core and novel component of the VisEN architecture are visual explorations of data related to what is shown in the visualizations of the narratives to enable users to gain a deeper insight into the data presented. The response (in bold) from statement 2 showed that 78 % (2013–2014) and 72 % (2014–2015) of the students who improved engagement from bad or poor to good or excellent found this feature beneficial for their study. The response from the entire class was also quite positive, which highlights the usefulness of related data exploration in the TEL domain (Table 2).

Table 2. Responses of the class and those whose engagement improved to good or higher (bold)
Full size table

Statement 3: I Found it Beneficial to be able to Track my Progress and Compare it Against Other Class Students Through Personalized Visualization.

Research [16, 17] has shown that peer comparisons can promote self-reflection and it has also shown that students can be averse to such comparisons [12]. The responses to this statement were quite spread, with 15 % and 9 % of the students (in bold), who improved their engagement from bad or poor to good or excellent and regularly used their visual narratives, undecided regarding the benefit of peer comparisons. Although the students agreeing with the statement were in majority, the responses suggest that some students were unsure about the benefit and others averse to such comparisons (Table 3).

Table 3. Responses of the class and those whose engagement improved to good or higher (bold)
Full size table

6 Conclusions

This paper introduced the VisEN framework and described how explorative visual narratives can be constructed and used by end users to analyze and explore. The related work section showed how VisEN progresses the state of the art by introducing related data explorations within visual narratives. The paper focused on evaluating the effectiveness of students’ individual explorative visual narratives, which were generated by VisEN, in supporting learners to engage with their course content using the AMAS PLE. Three studies were carried out using logged data and questionnaire responses from a total of 233 students over two academic years. The studies found that:

  1. 1.

    The visual narratives motivated learners to engage with course content.

  2. 2.

    The visual narratives were regularly used to understand course engagement and analyze progress, amongst learners who were improving their engagement.

  3. 3.

    An increase in engagement strongly correlated with an increase in visual narrative usage amongst students who were improving their engagement levels.

  4. 4.

    Students found it useful to explore their personal visual narratives through data related to what was shown in the visualizations.

These findings show that the visual narratives gave students a strong motivational pull to engage with course content, however, further work is required to accurately elicit the feelings of the students after viewing a narrative and connecting it to self-assessment.