Keywords

1 Introduction

The application of Information and Communication Technologies (ICTs) to learning processes offers a new way to deliver instruction in face-to-face and distance learning. The best example of this is the use of learning platforms, such as Learning Management Systems (LMS). LMS give support to online and blended learning. In online and blended learning, due to the lack of face to face interaction, instructors and course coordinators need tools to track students. LMS register large amounts of information about student interactions, but this information is usually stored in the LMS databases as raw data, and thereby the extraction of meaningful information usually requires further processing [1]. The records in LMS logs store abundant information about student and teacher interactions –as well as access to resources and system functions. This information may give an idea of how and when students perform their assignments and tasks, course engagement, etc. However, extraction of meaningful data and transformation of this information into actionable knowledge is a difficult task. New educational disciplines, such as educational data mining, academic analytics or learning analytics offer different but convergent perspectives, methodologies, techniques and tools aiming to facilitate this transformation process.

Educational data mining includes a series of techniques oriented to extraction of educational data through statistical machine learning and data-mining algorithms, for analysis and solution of educational research issues [2]. Academic analytics takes a different approach, focusing on the analysis of institutional data about students; therefore, it has a stronger focus on institutional policy decision making [3, 4]. Finally, the main goal of learning analytics is “the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs” [5].

From the above, it is patent that, although there are some differences between the three disciplines, they have as a common objective the understanding of teaching and learning in order to make informed instructional decisions oriented toward the improvement of learning processes [6].

There is currently a wide choice of tools that facilitate educational data extraction and analysis for learning analytics purposes. A first broad categorization of these tools would include [7]:

  • Cross-platform and platform-specific general purpose dashboards. Dashboards provide information about platform activity of the different learning agents –mainly, students and teachers–, generally in a visual and condensed form.

  • Ad hoc tools. The design and implementation of ad hoc tools seeks to perform tracking and analysis of very specific types of information adapted to very specific contexts.

  • Learning analytics tools for analysis of specific issues. These tools aim to provide information, and usually have very specific types of representation. It is also very common that they offer cross-platform capabilities.

  • Learning analytics frameworks and tools. The design of learning analytics frameworks is directed toward standardization of learning ontologies and their implementation in different systems. They also pursue the exploration of student behaviors in different educational contexts and offer the user customizable visual representations of the information.

Taking into account the great variety of applications for learning analytics, this research study aims to describe some of them, and apply and compare their results using a common dataset from courses taught in a Moodle LMS. The results of this comparison will highlight the usefulness, advantages and disadvantages of the different approaches and perspectives, and how they can complement each other. This study has therefore two differentiated parts: first, it introduces the different learning analytics tools that will be analyzed, and then the empirical work will cover the results, including a comparison of the tools after analysis of datasets from existing courses. Finally, this study will wrap up the conclusions about the results from applying the different tools.

2 Analysis of Tools for Learning Analytics

This research covers both cross-platform and LMS-specific tools. Different versions of Moodle are required for testing of the different tools, as not all analytic tools are available for all the versions of Moodle. Most of these tools analyze user interaction from LMS log data. That means that most of the tools extract and transform data from the mdl_log database table. Until version 2.7 each developer could potentially add their logs to this table from an application, leading to log formats that could be “not standard”. This problem is solved by the definition of a new log system in Moodle version 2.7. The new log system gathers more detailed information about user interaction that the previous system and, more important, it provides a standard API to write and read logs and increase system performance. Both log systems may coexist in Moodle installations with version numbers 2.7 and higher. Nevertheless, taking advantage of the new log system capabilities requires an adaptation of the different tools, and some of them have not updated to compatible versions yet. Therefore, the comparison presented in this study entails the use of different Moodle versions –note that the main objective of this research is to compare tools for learning analytics, not to address the problems related to how logs are stored in Moodle. The following subsections describe and analyze the different tools, according to the categorization showed in Sect. 1.

2.1 General Purpose Dashboards

Dashboards provide information about students or teachers activity in the platform, and present it in an aggregated and visually rich form –mainly tables and graphs with varying degrees of interactivity. Dashboards can be applied to different platforms [8, 9], or to a specific one [10]. These tools are primarily focused on the description of the activity carried out in LMS using very specific metrics, showing some relevant indicators at a glance, but they do not generally offer further information about how those metrics relate to each other. The main dashboard application for Moodle is Moodle Dashboard. There are other dashboards for Moodle, such as LearnGLASS or GoogleAnalytics, but these require adaptation and mapping of users’ accounts to external systems and/or hardcoding Moodle source code.

Moodle Dashboard is provided as a block, and it allows users to graphically or literally display the result of any query made in Moodle. When used in standard course formats, the block gives access to an extra page that displays the data rendered for the specified query. There are different options to visualize the information returned from the queries: tables (linear tables, tabular tables and tree views), plots (line graphs, bar graphs, pie graphs, and “doughnut” graphs), geospatial and map graphs, and timelines. Moodle Dashboard may display the rendered data directly, but it may also combine with other blocks to form a complex, highly customizable dashboard. It has powerful data filtering capabilities, as well as a functionality to automatically generate data exports [11]. Moodle Dashboard is supported up to Moodle version 2.5.

Apart from Moodle Dashboard, the default Moodle reporting tool might also be considered a dashboard. Moodle reporting tool facilitates analysis of information about users’ interactions in the platform, in different contexts. The different contexts available are site, course or activity, and the reports show information about user comments, course activity (most active, courses with most enrolled users, highest participation), LMS events logs (information of user’s interaction in the LMS) and live logs (interactions occurring at a specific moment), and graphs and statistics about users’ activity and view/post actions. Further filtering of this information is possible. On a course and activity level, it is also possible to gather information about course and activity completion, time spent to complete an activity, and grading information.

2.2 Ad Hoc Tools for Learning Analytics

Ad hoc tools are designed to track or analyze very specific bits of information, and to address a specific need in a very concrete context, with a set of defined constraints and conditions. The main problem of these solutions is that they are generally neither flexible nor scalable. This section describes two of these tools: (1) Interactions, a Moodle plug-in that groups types of interactions for later analysis, and (2) a web service that facilitates individual assessment of students in teamwork contexts.

Interactions is a plug-in that runs in Moodle versions 1.9 and 2.0 to 2.3. The plug-in is installed as a reporting block that adds functionality to the default reporting tool, with independent access permissions. Basically, Interactions adds a library that expands that functionality –including filtering capabilities– by creating a MS Excel spreadsheet with two different worksheets. The first one is an exact replica of the MS Excel file from the log reporting tool. The second worksheet processes each record and assigns it to a category within three different classifications (by agent, by frequency of use, and by participation mode) [6]. Experts on Moodle and eLearning participated in establishing the correspondence between actions and categories. The final output shows the total number of interactions of each category for each user in the platform. Because the results are already in Excel format, graphs can easily be derived from the output. Furthermore, the format allows easy integration with statistical analysis tools such as SPSS. It is noteworthy that the assignment of each record to any given category (a record can fall into one and only one category for each classification group, but it may appear in all the groups) is hardcoded in the processing library, and therefore any change to those assignments requires modification of the plug-in code.

The other tool is an ad hoc web service to assess student’s performance in teamwork contexts. Building from Fidalgo-Blanco et al.’s [12] work, the web service proposes an approach to validate data about interactions as predictors of individual performance in teamwork contexts based on the Comprehensive Training Model of the Teamwork Competence (CTMTC) framework [13]. CTMTC indicates how to collect evidences from three sources: forums, cloud-based file storage services, and wikis. The system extracts students’ interactions, enabling assessment of individual students and detection of conflicts. The tool uses the Moodle Web service layer [14] and extracts data from Moodle logs, focusing on forum posts and threads. It works in Moodle versions from 2.1 to 2.6 (its use in Moodle 2.7 or higher would require adaptation to the new log system). The tool allows choosing a forum within the course and then display the data for the student interactions with their peers, and has three different view modes: forum-based, team-based and thread-based. The tools gives information about the number of total messages in the forum/team/thread, as well as the number of people registered (the total number of team members), average participation of each student, the list of teams and the total list of students with their respective of the number of messages, creation dates of the first and last thread, list of threads (with the date of creation), and team members and degree of participation. In addition, rules of action can be defined based on thresholds set upon the number of messages [15].

2.3 Learning Analytics Tools for Analysis of Specific Issues

This set comprises tools of application focused on specific data and offering a very specific type of representation. These applications have very particular functionalities, and therefore they may or not fit institutional and personal needs. Some examples of cross-platform tools in this category are LEMO, SNAPP, StepUp!, while LMS-tied tools include Moodle Engagement Analytics, Moodle Learning Analytics Enriched Rubric or GraphFES.

Our analysis will focus on two tools for social network analysis –SNAPP (cross-platform) [16] and GraphFES (Moodle exclusive)–, which facilitates detection of disconnected students and gives information about the social interactions in the class, and Moodle Engagement Analytics [17], a Moodle block that provides information about at-risk students.

GraphFES (Graph Forum Extraction Service) is a web service that connects to both types of Moodle logs (legacy log and the new standard log) and extracts information from all the message boards in a given course. All the information that GraphFES collects is then processed to create three different graphs: (1) a graph including all the messages added by all users and how they relate to each other (i.e., a map of all the posts and how they are connected and organized in threads); (2) a graph connecting all the users in the course based on who has read contents posted by others, and how many times; (3) a similar graph to the previous one, but in which relations between course users are based on who replies to whom. GraphFES builds the social network graph and returns it as a .gefx file that can be opened in Gephi. The main idea behind GraphFES is that social network analysis is best done outside of the learning platform, using SNA specialized tools such as Gephi. Some advantages and applications of Gephi to the analysis of higher education courses from a social learning analytics approach can be found in [18, 19].

SNAPP (Social Networks Adapting Pedagogical Practice) [16] works as a bookmarklet that extracts information from message boards in Sakai, Blackboard, Moodle and Desire2Learn, and then builds up the resulting social network in a Java applet. There are two versions of SNAPP (v.1.5 and v.2.1), and their functionalities are similar. SNAPP is structured in tabs, the first three of which are interactive. The first tab shows the graph of the social network from the interactions and allows the user to manipulate the graph by filtering, applying different layouts to the social graph and selecting individual nodes –nodes in SNAPP represent participants in the message board. SNAPP v.2.1 also displays a timeline of the messages posted in the forum. A second tab displays the values of the number of posts per user in v.1.5 and the main social network parameters (degree, in- and out- degree, betweenness and eigenvector centrality, and network density) in v.2.1. Finally, the third tab allows exporting the graph in GraphML and VNA formats in v.1.5, or writing annotations in v.2.1. (export capabilities are included in the first tab in v.2.1., adding the ability to export to.gefx format).

Engagement Analytics is a Moodle plug-in provided as a block that gathers and shows information, in the form of indicators, about student progress. As the name suggests, the block gives users insight on the level of engagement of a student, where engagement refers to activities which have an impact on student success in an online course. The block provides real-time (“live”) information about students’ interaction with resources, and incorporates a set of indicators and a risk alerting algorithm. This information may be useful for teachers to detect at-risk students and make decisions about when to intervene in order to avoid student failure. The indicators included in Engagement Analytics relate to student assessment, students’ participation in forums and number student login frequency, and it is possible to assign each indicator different weights to describe and model students’ risk of failure in a more customized way. Indicators comprise different items, and item weights can also be modified. This plugin is available from Moodle 2.2 up to Moodle 2.7, and the plug-in allows extension of the predefined indicators.

2.4 Learning Analytics Frameworks and Tools

The fourth category of tools includes applications and frameworks that can be applied to several platforms or contexts to explore different aspects of learning using different visual representations. Some examples of this kind of tools are SAM, VeLa or GISMO (this analysis covers the latter two).

VeLA (Visual eLearning Analytics) [20] is a framework that uses web services to extract information from LMS logs. VeLA provides different representations of the information and displays it in an interactive way. For example, users can filter, search, or change dynamically the representation of the information. VeLA offers four different functionalities: (1) a semantic spiral timeline that facilitates tracking users’ platform activity during specific periods of time; (2) an interactive semantic tag cloud that allows users to analyze the most relevant terms and concepts used in a course; (3) a social graph that shows users’ interactions; and (4) a tool to compare and establish relationships among the data stored in the LMS and users’ activity. VeLA is supported by visual analytics techniques.

GISMO is a graphical interactive monitoring tool that provides visualization of students’ activities in online courses. GISMO is a plug-in available for Moodle versions 1.9.X and 2.X. that enables teachers to examine diverse information about students, such as the course attendance, reading of materials, or assignment submission. GISMO provides comprehensive visualizations that give an overview of the whole class, not only a specific student or a particular resource. GISMO provides seven different visualizations: access overview, access to the course, access to resources, assignment overview, quizzes overview, access to resources overview, timeline of access to resources by student, and access to resources by student [21].

3 Method

In order to assess how the different tools work, this study uses data of 119 students from a Programming course at the Universidad Politécnica de Madrid. The course methodology aims to promote teamwork between students. The use of forums, wikis, web resources, videos, quizzes and assignments in the course is intensive, and therefore it is an adequate test bed for all the tools detailed in Sect. 2. The main results from the application of these tools are detailed next:

  • Moodle Dashboard. The last version of the tool works properly for Moodle 2.5. We have tested it in a Moodle 2.6, and no results are returned from a simple query. With debugging mode activated, it is also possible to see an error, but no information is shown. The tool may be not adapted to versions of Moodle higher than 2.5.

  • Moodle default reporting tool displays more than 122640 log entries (111644 are view actions, 9398 are actions to add resources and 821 are update actions). Detailed but raw information about each action is displayed in a table, and it is possible to export the results to a spreadsheet.

  • Interactions. The plug-in does not work properly in Moodle versions 2.3 and higher. However, because it only processes data from the Moodle log table, it was possible to import the data directly via MySQL import and process data in Moodle 2.1. The result is a spreadsheet, where it is up to the teacher or course administrator to create graphs from the data to display information (see Fig. 1 for one example) and detect abnormal levels of different types of activity. The data may also be analyzed with statistical packages such as SPSS.

    Fig. 1.
    figure 1figure 1

    Graphs created in MS Excel using data from interactions

    Full size image

  • Teamwork assessment tool requires activation of Moodle web services. The tool provides a list of links of course forums. After selection, it is possible to see the participation in each forum, in a group, and individual participation. From this information it is possible, for instance, to know that those groups working in the mornings (there is an specific forum for them) have published 4974 posts with an average of 81,54 per user, and also who is the person with more posts (192 messages in this case). By inspecting a single group (group M9 in this case), the tool reports 990 posts, 6 users, 183 short messages (less than 140 characters) and 807 long messages, 141,43 posts per user, and how participation is distributed between students (in this case, between 13 % and 19 %) (Fig. 2). Further filtering on a per thread basis is also possible.

    Fig. 2.
    figure 2figure 2

    Teamwork assessment tool showing information about participation of students in a forum.

    Full size image

  • GraphFES offers a front end that requires login credentials, the platform’s URL and course ID. After activation of the web service and granting user permissions in Moodle, GraphFES extracts all the information about forum activity from the Moodle data log table via the web service, and creates two lists of nodes (messages and users) and the relationships between them. The output consists of three.gefx files that can be opened and analyzed in Gephi (we refer to [18] for more information on how to use Gephi to perform social learning analytics). Figure 3 shows the three social graphs, including all users and posts (i.e., not filtered and not using node attributes), of the course.

    Fig. 3.
    figure 3figure 3

    Social graphs of messages posted (9241), and read messages and replies among users (124, including teachers).

    Full size image

  • Version 2.1 of the SNAPP applet does not work properly in any version of Moodle, and requires to configure security exceptions in Java Runtime Environment because it connects to an external source to perform the analysis. In order to build the social graph, the applet loads all threads in a forum and processes the HTML content. The problem seems to lie in the building of the social graph, and no participants are identified. Version 1.5 does not render any result for a message board either, but it allows analyzing individual threads. Unfortunately, it only works in earlier versions of Moodle (2.1), where it was not possible to restore the course data.

  • Moodle Engagement Analytics aims to detect at-risk students. The setup for this study assigned equal relevance (weight) to logins, forum participation and assignments. Figure 4a shows the results of the analysis (at-risk students are displayed in red, on the left side). The tool detected 18 students at-risk (failure probability above 65 %). Clicking on the name of the student shows a report explains why this person is considered at risk (Fig. 4a, right side).

    Fig. 4.
    figure 4figure 4

    Engagement block showing at-risk students, and the report of a specific student (left, 4a) and report of actions in a specific forum in GISMO (right, 4b).

    Full size image

  • VeLA uses Moodle log data, and it revealed that the message board threads were used mainly to solve doubts, carry out the teamwork tasks, and publish news. This tool also represents users’ interactions with peers and resources using force graphs. VeLA is an integrated framework, which means that filters and selections can be applied to all views at once [20].

  • GISMO provides different visual representation of users’ interactions. The example below presents the number of global actions in forums. Students’ read and write actions are clearly distinguishable, and it is very easy to compare who has the most read actions (979) or who has published most posts (259). GISMO can also show students’ actions in a specific forum (Fig. 4b) and other activities and resources.

4 Discussion

From the analysis of the tools, we can observe their strengths and shortcomings. Nevertheless, it must also be noted that the choice of tool will highly depend on the users’ needs. For example, Moodle’s default reporting tool offers a vast amount of information and filtering capabilities, but the information it provides consists of raw data, and therefore it offers very detailed information but it is not capable of providing meaningful aggregated information about the courses. As an example, the tool cannot answer a simple question like “How many students have not started a course yet?”, or more complex questions regarding students’ progress in a course.

Theoretically, Moodle Dashboard could give an answer to these questions, including visualizations of data (despite its lack of interactivity). However, this study could not test Moodle Dashboard due to its extremely difficult configuration process and its restricted compatibility with latest Moodle versions. Furthermore, Moodle Dashboard lacks flexibility for customized queries and reports, which makes it necessary to use ad hoc tools for particular purposes.

This study has explored two of these ad hoc tools. Interactions represents numerically users’ interactions in a spreadsheet, allowing customization of graphs and facilitating statistical analysis, and the Teamwork assessment tool has a web interface and focuses on analysis and assessment of students’ participation in message boards. Both tools solve very specific problems; however, their specificity makes it difficult to apply them in other contexts or platforms.

The study also described tools designed to address specific issues: two tools for social learning analytics (three, if we consider the social graph included in VeLA), and a tool for students’ progress tracking and at-risk student detection. The main difference between the first two tools is that SNAPP includes a basic social network analysis module within the platform –note that SNAPP could not be tested with the study data due to malfunctioning– while GraphFES allows performing a complete and more detailed analysis using an external program. Regarding student tracking and at-risk student detection, Moodle Engagement Analytics relies on predefined indicators, and facilitates live monitoring of a course, which in turn allows teachers to take action when the system detects at-risk students; a major drawback is that, despite allowing customization of indicators’ weights, the indicators are not intuitive and addition of new indicators requires additional coding.

Learning analytics frameworks aim to overcome the limitations of the above mentioned types of tools, and they integrate data, different functionalities and visualizations, and interactive data manipulation in one system. Obviously, learning analytics frameworks do not adapt so well to specific tasks because of their general purpose design. In a way, these frameworks could be considered some kind of advanced dashboard that integrates information but that can also provide very detailed information about courses and students.

As a conclusion, a qualitative analysis of the different tools included in the study shows that it is necessary to add some learning analytics capabilities to LMS such as Moodle within the same platform. For the sake of simplicity and compatibility, some basic dashboard and alert system would fit this task without need for further user training. Nevertheless, we show in this study how the different tools complement each other adding new functionalities, and that a more insightful analysis of educational data requires integration, complex visualizations and interactivity, for which learning analytics frameworks are suitable tools. A focus on the development, flexibility and stability of the LMS web service layer would be critical in order to facilitate implementation of these frameworks. Furthermore, a by-product of a consistent web service layer is the ability to use multiple existing external specific-purpose programs for analysis (as illustrated by the case of GraphFES and Gephi) that can provide a deeper level of analysis than some basic LMS plug-in.

Finally, we believe that the use of complex learning analytics frameworks is not oriented toward students or teachers (whose needs should be covered by basic dashboards). To reach their full potential, frameworks should also integrate institutional and academic data, and be managed and operated by experts with a role of learning platform analyst. Analysts would act then as “learning consultants” of the different agents in the learning process (course coordinators, teachers, students).