Keywords

1 Introduction

Recent research shows the importance of self-regulated learning (SRL) in traditional and online learning contexts [1]. Self-Regulated Learning refers to how students become masters of their own learning processes [2]. However, this definition can vary depending on the theoretical model used as a reference as well as the research context or focus of analysis (motivation, cognition, meta-cognition, feelings) [3]. In online contexts, the learners are required to have greater autonomy than in face-to-face classes and they are expected to be able to deploy SRL strategies in order to achieve their objectives. That is, learners who are able to self-regulate their learning are more likely to succeed in completing courses [4, 5]. Self-regulation skills are even more relevant in a MOOC, which is characterized by the massiveness and heterogeneity of the participants; the lack of guidance from a tutor during the course; and the flexibility of schedules over time [6].

Recent research indicates that some SRL strategies are associated with the learners’ performance and achievement of their goals. For example, strategies such as goal setting and strategic planning, as well as time management have been demonstrated to have an influence in performance and fulfillment of the learners’ goals [6,7,8]. Likewise, [7, 8] showed that learners use strategies such as organization, help seeking and effort regulation to when working in a MOOC. However, current MOOC platforms do not offer adequate technological support for the deployment of learners’ SRL strategies [9, 10]. For example, the Coursera platform offers the option of consulting the time spent on video lessons. In addition, it has a submission timetable that, together with email notifications, help learners to keep engaged with the course. Despite of this, researchers agreed that these mechanisms are not enough and it is necessary to develop new tools to support SRL in online platforms [11, 12]. Although tools have been developed to support learners’ SRL in the context of traditional online learning [13,14,15,16], as well as in the MOOC context [17,18,19,20], there is a strong mismatch between the goal of the tool and its evaluation [21]. Furthermore, in the case of the MOOC context, the development of this type of tools is new, few tools are implemented, and more evaluations are required in these massive contexts to understand the impact on the learners’ self-regulation [22]. The research points out a severe weakness regarding the evaluation of existing tools [22,23,24,25], as they focus their evaluation on usability and usefulness [23]; leaving a gap in the measurement of the tool’s impact of the SRL strategies that they support.

In this light, the development of new tools aimed at supporting self-regulation in MOOC environments is a challenge that remains open. The lack of evaluations to measure the impact on SRL does not allow us to understand what characteristics should be considered in the design of new tools or how the self-regulation strategies that the learners use with the interactions they perform with the tool are related. In addition, there is no guide for the design, implementation and evaluation of this type of tools.

In this paper, and in order to understand the current state of the art in the development of tools designed to support learners’ self-regulatory processes online, we present a systematic literature review that extends a previous work [22], but focusing on: (1) analyzing the relations between learning activities and self-regulation strategies defined in the design of the tools; (2) analyzing the characteristics and indicators used in the tools; and (3) presenting the lessons learned in each of the papers to understand what these tools should be design in a MOOC context.

2 Prior Work

In this section we analyze the results of the two literature reviews [21, 26] we found in the area of supporting learners SRL strategies online and summarize the results of our previous study of the literature [22].

Jivet et al. [21, 26] conducted two literature reviews on 26 tools to support learning processes in online environments. Of the 26 tools analyzed, 13 of these were designed for supporting self-regulation in online environments. The results show that SRL is supported through tools that provide learners’ awareness and trigger reflection about their learning process. In addition, the authors point out that there is a separation between the purpose of the tool and its evaluation. Although these reviews shed some light on how SRL is addressed, they do not analyze in detail the characteristics of these tools in terms of design, nor the self-regulation strategies that they aim at supporting.

In the a previous literature review [22], we analyzed 21 tools aimed at supporting learners’ self-regulation. In this review we analyzed their characteristics in terms of design, the SRL strategies supported, the methodology for their evaluation, and their impact of learners’ self-regulation. The main findings are the following: (1) there is a lack of tools to support SRL in MOOC environments; (2) the evaluation of the existing tools is not aligned with the objectives of the research; (3) current research present proposals of tools but very few reach the implementation stage; and (4) current existing tools tend to support many SRL strategies at the same time.

The main gap identified in this prior work is the lack of alignment between the purpose of the tools in supporting self-regulation and the evaluations performed to assess their effectiveness. In this study, we propose to expand the previous literature review with the purpose of providing more insights about the relationship between the design of the tools, and how their functionalities relate with learners’ self-regulated strategies in the course. Specifically, we defined 5 research questions to guide the literature review: RQ1. What is the context in which each tool has been applied, including the educational level and learning environment?, RQ2. What characteristics have been considered for the design of the tools to support the learners’ SRL strategies?, RQ3. What SRL strategies are supported by these tools?, RQ4. How does the design of the tools relate with the learners’ self-regulated learning activities? RQ5. How was the impact of the tool on learners’ self-regulation measured?

3 Methodology

For the systematic literature review, we followed the phases proposed by Kitchenham [27]: planning, execution and reporting. However, for this review a process we did not carried out an analysis to determine the quality of the papers, given that the interest of the study is to include as many publications as possible. The search process was conducted in 5 databases were most of the papers in Technology Enhanced Learning can be found: Scopus, ACM Digital Library, IEEE Explorer, SpringerLink and Science Direct. The following keywords were used to formulate the search queries: Self-Regulated Learning, Self-Directed Learning, Tools, System, Dashboard, Online, MOOCs. This query is expressed symbolically as: (Self-Regulated Learning, Self-Directed Learning) AND (Tools, System, Dashboard) AND (Online OR MOOCs). The first part of the query focuses detecting articles related to self-regulation; the second part identifies tools proposed or implemented; and the third part identifies the context at which the research has been conducted. The review was conducted by 3 researchers. Two investigators reviewed and selected the articles and the third investigator intervened in case the two investigators had doubts about the inclusion of an article.

1.829 articles were retrieved according to this search criteria. From these, we conducted a selection probes based on articles’ the titles/abstracts and keywords. From this first pool of articles, we excluded those that did not match the following criteria: articles that do not describe a tool, articles that support self-regulation, but not through a tool; tools that support self-regulation, but not in an online environment; articles that addressed the use of tools such as social networks and e-portfolios to support self-regulation, but no development is proposed; and tools that support self-regulation, but are not designed for learners. At the end of this process, we ended up with 42 articles. Then, we eliminated duplicates (11) and conducted the analysis of the whole article. In order to broaden the range of tools analyzed, we also included in the analysis those references that were identified from the references of the articles analyzed (7).

A total of 38 articles was considered for this review. This selection considered articles that describe tools designed for supporting learners’ self-regulation in both traditional online learning environments and MOOCs. The articles related to the same tool were counted, but for the analysis they were considered as a single tool. The analysis was performed on 22 tools described in the selected articles. Figure 1 depicts the process selection criteria conducted in this review. Although an important number of data sources were considered for the systematic search, there is a possibility that some publications that propose or implement tools have been left out of the study, wich we assume as a limitation.

Fig. 1.
figure 1

Papers selections process

Full size image

4 Results

The results are presented to answer each of the research questions posed. A total of 22 tools were analyzed in the literature review (see Table 1). From this pool, 19 tools are implemented and 3 propose only the design of a tool [20, 28, 29].

Table 1. Description of tools designed to support learners‘ SRL in online environments.
Full size table

4.1 RQ1. What Is the Context in Which Each Tool Has Been Applied, Including the Educational Level and Learning Environment?

9 of the tools were designed for supporting self-regulation in higher education; 2 for high school; 2 to professional training; 5 for general education (tools that do not focus on a specific level of education). 4 of the tools do not specify the educational level. 14 tools were designed for supporting SRL in traditional online learning environments and 8 in MOOCs [4, 10, 19, 20, 28, 30,31,32]. Two of the tools designed for MOOCs are only design proposals, but have not been implemented [20, 28]. 19 of the tools were designed only for the web, 3 for mobile technologies [12, 20, 33], and only 1 of supports both web and mobile devices [30].

4.2 RQ2. What Characteristics Have Been Considered for the Design of the Tools to Support the Learners’ SRL Strategies?

For analyzing the characteristics of the tools, we took as a references the categories defined by Bodily and Verbert [24]. These categories include: (1) visualization, if tool use any type of visualization to display data; (2) class comparison, if tool included a system that allowed learners to compare their data with other learners’ data; (3) recommendation, if tool included a system that provided a recommendation to a learner; (4) feedback, if the tool offers feedback through text; and (5) interactivity, if it offers the possibility of clicking and exploring its data. In addition, two categories were included, (6) collaboration, if tool included a system that learners shared materials or knowledge (7) input forms, if the tool has forms for data entry. In Table 2, shows a summary of the categories identified in the analysis.

Table 2. Functionality and types of indicators identified in the tools (Link to the complete list of indicators identified in the tools https://drive.google.com/open?id=1-U2xEnelilQPKZjL-0nZ7rHyA71bKxK-5W8XaLsGnkM).
Full size table

Visualization: 13 tools use some type of visualization to support self-regulation strategies. The progress or interaction of the learner with the activities is displayed through using graphs, tables, networks, calendars or progress bars [10, 12,13,14, 16, 19, 28, 31,32,33, 36, 38, 41]. Visualizations such as conceptual maps are used to present the objectives produced by the learners [14, 31].

Class comparison: 5 of the tools report the use of social comparison components to support self-regulation. The tools offer mechanisms for the learners to compare their performance with the performance of their classmates [16, 33], or with the learners from previous editions [10, 19].

Recommendation: 9 of the tools use recommendation mechanisms. They recommend learning objectives or activities [12, 13, 28, 36,37,38], learning routes [36], strategies or tips for SRL [12, 20, 41], and the use of tools (widgets) [30].

Feedback: 4 tools offer textual feedback to the learners through motivational messages for performing an activity [36], presenting the correct answers to an exercise [16], time invested [35], or sending notifications [11].

Interactivity: 4 tools allow some kind of interactivity with the information presented to the learners. Learners can interact with the information and select the activity to analyze [13, 16, 19, 36], and activate or disable the social comparison [16, 19].

Colaboration: 11 tools integrate collaboration mechanisms that support learners’ help seeking. Among these mechanisms are: the use of social networks, wikis or blogs [12, 13], discussion forums [13, 32], shared learning spaces [13, 14], and sharing of learning resources for getting feedback [12, 14, 31, 34, 36].

Input forms: 10 tools use some mechanism for allowing data entry by the learner. Learners can define and plan their goals [12, 13, 19, 33, 35,36,37,38, 40], record the time of an interruption in the study and the reason for the interruption [12], record the beginning and the end of an activity [33], and record the level of completeness of the activities [40]. In addition, 5 tools propose the use of widgets or plugins to support learners’ SRL [10, 13, 19, 28, 30].

To analyze the type of indicators proposed to support SRL, we categorized them according to the 6 groups proposed by Schwendimann et al. [25]: (1) action-related; (2) content-related; (3) results-related; (4) social-related; (5) context-related; and (6) learner-related. A total of 78 indicators were identified. Most of the indicators fall into two categories: action-related (30 indicators) and content-related (34 indicators). 13 tools use the action-related category and the same number of tools use the content-related category (Table 2). 10 of the tools used results-related indicators.

4.3 RQ3. What SRL Strategies Are Supported by These Tools?

For tools dedicated to traditional learning environments we identified 10 SRL strategies that are generally supported:

  • Goal setting: present in 14 tools [4, 10, 30, 33, 40], those that implement mechanisms so that the learners can set their learning goals such as the selection of skills to develop [36] or the definition of activities to be developed on certain dates [13, 19, 20, 28, 34].

  • Self-evaluation: present in 12 tools. The self-evaluation strategy is interpreted from two perspectives in the tools. First, to provide feedback when the learners complete the evaluation activities suggested in the course [4, 16, 31, 32, 34, 35, 41], and second, to provide learners’ with information to evaluate their progress in their activities [12, 13, 19, 20, 33, 35,36,37,38, 40].

  • Help seeking and organization: they are supported in 9 tools [13,14,15, 29, 31, 34, 41]. Help seeking is generally supported by enabling shared spaces, forums, chats or by integrating social networks. Organization is supported through the use of notebooks or supporting the generation of concept maps for content organization.

  • Self-efficacy is supported in one tools [13] and self-motivation is supported in 2 tools [37, 39].

For tools dedicated specifically for supporting SRL in MOOCs, we identified 7 strategies as the most supported: (1) goal setting [19, 20, 28, 30, 32], which remains the most supported, (2) time management [10, 19, 20, 28], (3) help seeking [31] being the least supported strategy. The support of SRL strategies in MOOC is consistent with what the literature points out, as goal setting, strategic planning and time management are strategies shown as effective to achieve learners’ objectives [6,7,8]. Time management is generally supported by displaying the time invested by the learners in the activities in study sessions [10, 19, 35], and procrastination [10, 19]. Time management is also supported through the scheduling and organization of activities [20, 35].

4.4 RQ4. How Does the Design of the Tools Relate with the Learners’ Self-Regulated Learning Activities?

In 19 of the tools analyzed, it is not described how the design of the tool establishes a relationship between the activities of the learners and the SRL strategies that it tries to support. Only 3 of the tools describe some type of relationship between the activities and SRL strategies. For example, in [12] there is a diagram with 7 different transition states that the learners can perform in the tool. In Fig. 2, an example of two states of the diagram are shown. The states are associated with the SRL phases of the Zimmerman model [3]. The transitions indicate specific activities that the learners perform interacting with the tool functionalities. In this way the transitions between one state and another allow to relate the activity with a self-regulation phase. However, the information about user transitions is not used for evaluating the effectiveness of the tool but for representing the learners’ interaction with it.

Fig. 2.
figure 2

State transition diagram to SRL, extracted from [12]

Full size image

In another case such as [13] SRL activities are defined by learners. They define 7 groups or categories connected with the tool functionalities: (1) Search & Get Recommendation, (2) Plan & Organize, (3) Communicate & Collaborate, (4) Create & Modify, (5) Train & Test, (6) Explore & View Content, and (7) Reflect & Evaluate. Each group of features is associated with one of the phases of the SRL model that the tool is based on. The learners have the option of classifying the activity performed with a widget within one of these functional groups, thus trying to relate the activities performed by the learners to one of the phases of SRL.

In [36] an approximation is made relating the strategies of SRL with the tools’ functionalities, in order to evaluate the usefulness perceived by the learners in the execution of self-regulation. The goal setting strategy is associated with the recommendation feature and the delivery of the information useful for the learner. The monitoring strategy is associated with the delivery of the information useful for the learner.

4.5 RQ4. How Was the Impact of the Tool on Learner’ Self-Regulation Measured?

The evaluations of the tools implemented focused on measuring aspects such as: usability (6), usefulness (4), satisfaction (4), and learning outcomes (4). However, this section presents the evaluations that proposed measures for analyzing the impact of the intervention with the tools on learners’ behavior or performance. Three of the tools designed for MOOC assess the impact on the learners’ behavior and completeness rate. In [10], the impact of the tool on the learners’ behavior is measured with respect to evaluations. The results show a positive effect in the assignments delivery times, with the learners sending evaluations in advance. However, the authors point out that no evidence of changes in the learners’ behavior was found. In [4, 10], the impact of the tool is measured by the learners’ completion rate. In both cases the results show an increase in the completion rate.

In [33] the authors measure the impact of the tool in learners’ Time Management strategies using the Online Self-Regulated Learning Questionnaire (OSLQ). The results show a positive effect on the learners’ time management ability with the use of the tool and the social comparison component. In [37] the monitoring of the schedule defined by the learners was analyzed as a measure of time management. As a result, it was observed that the learners who closely follow their schedule and prioritize their studies against other activities, usually work harder. In [39] the activities performed by the learners to manage their time and monitor their learning were analyzed as a measure of the impact on learners’ performance. The results show that the execution of these activities minimizes the opportunities for interruption and loss of discipline at the time of studying.

In [40], a pre and post self-report test about self-regulation is used to measure the effect of the goal setting functionality included in the tool. In [16], the effect of the social comparison on the learners’ engagement, performance, navigation and motivational profile is evaluated. The results indicate a positive effect of the social comparison component on engagement, efficiency, effectiveness and motivation.

In [13], authors analyze the interaction of the learners with the widgets (15 base widgets) developed to support SRL. The results show that few learners use SRL widgets. In the spaces where the learners add at least one SRL widget, the classification of Plan & Organize and Reflection & Evaluation is used, while in the other spaces, the Collaborate & Communicate classification is more frequent. Finally, the authors concluded that SRL is a new concept for the learners and the evaluation of the impact of the SRL on the learners requires long-term studies. In [35], the navigation of the learners was evaluated, and it was observed that the group that performed a non-linear navigation had a higher learning output. In addition, the time invested by the learners in the use of each strategy was evaluated and it was found that the learners usually spend more time on ineffective learning strategies used to select, organize and integrate multiple representations of the topics. Finally, in [34], the scores of the learners’ evaluations were analyzed. The results show that the graphic and interactive visualization of the concepts of study contribute to improving the programming ability of the learners. In addition, a pre and post test was used to evaluate the impact on cognitive and meta-cognitive self-regulation strategies. The results show that learners improved their cognitive and meta-cognitive strategies.

5 Lessons Learned

In this study we have performed an analysis of tools that support learners’ SRL in online contexts in order to understand how to develop tools that support these strategies in MOOCs. As a result of this analysis, we highlight three of the lessons learned that could help inform the development of future tools to support self-regulation strategies in MOOC-type of learning environments.

5.1 Visual Mechanisms, Interactive, and Social Comparison

The tools use different mechanisms to support self-regulation of learners: visualizations, social comparison, recommendation, collaboration, and interfaces for data entry. The results show that tools that use visualization and allow some type of interactivity have a positive effect on learners’ motivation. In the learning environment of MOOCs this can be an important mechanism to maintain learners’ motivation. The social comparison component also has a positive effect on both the MOOCs environment and the traditional online environment. The effect is reflected in the time management and the commitment of the learners. This is a mechanism that must be explored in greater detail to measure its impact on learners’ performance and behavior. In addition, in the context of MOOCs, it is necessary to analyze which comparison parameters have the greatest effect on learners, for example, comparing their performance with the learners from the previous editions or the same edition.

5.2 Design of the Tool Related to Self-Regulation Strategies

The purpose of supporting the learners’ SRL strategies is clear in all the tools analyzed. However, the design of the tools does not seem to have a clear connection to this purpose. The description of the tools focuses on explaining the features or mechanisms included in the tool, without offering enough detail about how the activities performed by the learners with these mechanisms support specific SRL strategies. The design stage of the tool should be more relevant than the implementation itself. In this stage it is necessary to establish clear relations between the activities performed by the learners, a specific SRL strategy and how the tool enhances support these activities. It is necessary designing the tool according to a theoretical-based model so as to define and integrate functionalities towards the strategies defined I the model. There is a lack of evaluations that relate learners’ activities with the tool functionalities and SRL. For example, a tool aimed at supporting Time management evaluates its impact through the learners’ self-report, without analyzing the planning and behavior changes of the learners regarding time spent on activities.

The report of the tools should detail the indicators used to measure the self-regulation activities of the learners. Characteristics are presented, but the indicators and how they relate to self-regulation strategies are not specified. The results show that the tools collect a lot of indicators about the learners’ events on the platform and about the content. However, few of these indicators are used to evaluate the tool. Future work should consider the evaluation methods in advance and define the indicators carefully. The indicators must be defined during the design process of the tool and associated with learners’ self-regulation strategies defined in the theoretical model taken as a reference.

5.3 Evaluations Aligned with the Purpose of the Tool

Most of tools are evaluated in terms of usability and usefulness. However, there is little research on the impact of tools on learners’ self-regulation behavior. In addition, few mechanisms that measure this impact are present in current studies. The self-report questionnaires are the instruments more frequently used to evaluate the impact of the tool on the learners’ SRL. However, new evaluation proposals are required to understand how the tool contributes to supporting self-regulation and learners’ performance. For example, and since goal setting is one of the most common strategies supported in the tools analyzed, the evaluations could focus on analyzing the behavioral patterns from learners’ traces, with respect to their goal setting, the fulfillment of the goals, the gap between the goals established and reached, or the percentage of the goals achieved. The learners’ interaction with the SRL mechanisms implemented in the tools should be monitored in order to find correlations with performance. In addition, researchers should consider from the beginning what is the association between the activities performed by the learners with the tool, and the strategies of SRL so as to facilitate evaluation processes. Only few works propose this relationship, and most of the tools evaluations are poor. Finally, tools should be evaluated in actual learning environments, with actual users. Studies with controlled and small groups should be limited to test the tools, but not to evaluate its impact. This scenario is even more important on the tools that support self-regulation in MOOCs courses, given that the characteristics of the learners are more particularly heterogeneous.

6 Conclusion

In this literature review, we analyze the relation defined between the activities performed by the learners and the SRL strategies that the tools support. The results indicate that only few researchers define this relationship and, consequently, it difficult to evaluate what is the impact of the tool in learners’ SRL strategies. Further, evaluating the impact of the tool should be based in both self-reported questionnaires and actual interaction patterns of learners’ activity with the online environment, the specific tool and their learning outcomes or performance.

In the MOOC context, there are already some tools designed to support SRL. However, most of these tools have not been evaluated in terms of impact on learners’ strategies. The design of the future tools should be based on a clear relationship between learners’ activities and SRL strategies to facilitate measuring their impact. The great challenge in the MOOC context will be how to measure the impact in the short and medium term, since most of the courses are only from 5 to 10 weeks. As future work, the features identified in the different tools were analyzed could serve as a guideline to evaluate tools for supporting SRL in MOOCs or online learning environments.