Introduction

Supporting self-regulated learning (SRL) in online learning environments has become an important topic of research in the last years (Tsai et al. 2013; Wong, Baars, et al., 2019; Wong, Khalil, et al., 2019). Since several learner SRL strategies (such as time management, metacognition, effort regulation, and critical thinking strategies) correlate positively with academic outcomes (Azevedo & Aleven, 2013; Broadbent & Poon, 2015; Carson, 2011; ChanLin, 2012; Klingsieck et al. 2012), several approaches were proposed for supporting SRL strategies in online learning settings. Indeed, Zheng (2016) reports several types of support during learning, namely use of self-monitoring tools (Chang & Lin, 2014; Kim & Pedersen, 2011), tutor feedback (Biesinger & Crippen, 2010; Wäschle et al. 2014), or guidance through an intelligent tutoring system (Duffy & Azevedo, 2015). Other approaches use prompts to induce SRL strategies and enhance learning outcomes (Bannert & Reimann, 2012; Kauffman et al. 2011; Kizilcec et al. 2017).

Among behavioral regulatory strategies, effort management, time management, and help-seeking proved to be positively correlated with academic outcomes (Richardson et al. 2012). Effort management strategies help students focus their attention on the task they are performing and use their effort for achieving it effectively. During this process, students acquire competences that enable them to deal with failure, persist and overcome difficulties. To this end, these strategies foster motivation and commitment to completing one’s goals even when there are problems or distractions. Time management strategies allow students to acquire skills related to setting goals and priorities, planning, self-monitoring, conflict resolution, negotiating, classifying and assigning tasks, negotiating as well as problem solving. Students achieve successful time management if they can maximize their use of time to facilitate academic performance, balance, and satisfaction. Help-seeking strategies involve processes of seeking assistance from others (e.g. teacher, peers) or other sources that facilitate accomplishing desired goals in a learning environment. These strategies are associated with student motivation and can assist students not only to address their immediate learning needs but also to improve their performance, by gaining knowledge and skills and alleviating difficulties, which ultimately enhances understanding, achievement and subsequent independence.

Automated assessment tools are systems that show students’ performance while carrying out a task. They can be useful for both students and instructors, providing ranking and immediate feedback which can have a strongly positive effect on student learning. Within the recent few years a significant number of automated assessment tools have been implemented to assess programming assignments (Pettit et al., 2015). They utilize mechanisms for assessing programming assignments, examining the type and nature of student errors and providing meaningful feedback. They may be also capable of: employing grading methods, allowing students to correct their errors in real time, and making them aware of their misconceptions, knowledge gaps and their own learning more accurately and in a timely manner. Distributed systems, also known as distributed computing, are systems with multiple components located on different machines that communicate and coordinate actions in order to appear as a single coherent system to the end-user and achieve a common goal. Distributed systems focus on advanced programming techniques and skills in order to build software systems that can better meet the demands of distributed platforms. To that end, they need to be supported by adequate methods and tools (Daradoumis et al. 2019). In this context, supporting students’ self-regulation of behavior can be crucial. However, there is hardly any study that has investigated how an automated assessment tool for distributed programming could strengthen students’ self-regulation of behavior. Some efforts performed in computer science showed that when students employed SRL strategies achieved good performance in introductory programming (Alhazbi & Hassan, 2013; Bergin et al., 2005). Another study showed that students face difficulties in computer science education when they are not aware of SRL strategy use, so fostering them is important (Alharbi et al. 2012).

Therefore, the objective of this study is to examine how the use of an online Distributed Systems Laboratory (DSLab) can improve students’ self-regulation of behavior when they deploy, execute and assess their distributed programming assignments.

We organize the rest of the paper as follows: In Section Literature review on students’ self-regulation of behavior, we review the literature regarding SRL supports in online learning settings in general and in computer programming environments in particular; then we set the purpose and research questions of our study. In Section Research aims, we describe the methodology used in this study. Section Method presents the results whereas Section Results discusses and analyzes these results with respect to the research questions. Finally, we present the conclusions and future work.

Literature review on students’ self-regulation of behavior

Supporting students’ self-regulation of behavior in the context of online learning environments

Self-regulation of behavior constitutes an important aspect of SRL and includes time and effort management as well as help-seeking items (Pintrich, 2004). Our study focuses on specific strategies that learners can use to manage their effort and time, and seek help in order to learn effectively. In their survey, Wong, Baars, et al. (2019), Wong, Khalil, et al. (2019)) found that several SRL-supports were implemented in traditional online learning environments, such as prompts, feedback, and integrated support system. Stahl and Bromme (2009) used prompting together with graphic organizers on help-seeking processes. Sitzmann and Ely (2010) reported that continuous prompting could affect effort management, whereas Yeh et al. (2010) found that self-explanation prompts urged students to spend more time learning.

Different types of prompting were examined. Cognitive prompts were employed at different phases of the learning process to support searching for information, planning, orientation, goal specification, monitoring, and evaluation of learning (Bannert & Reimann, 2012). Metacognitive prompts were provided to guide or make learners reflect during a problem solving process (Kauffman et al. 2008), or to support conceptual understanding during learning (Moos & Azevedo, 2008); besides, Bannert et al. (2015) suggest that these prompts support SRL strategies in the short-term, though they achieve to improve learning performance in the long-term.

Unlike prompting, feedback was much less used as an SRL support. In fact, it was not used as a method to support self-regulation of behavior but rather to promote reflective activities (Biesinger & Crippen, 2010; Wäschle et al. 2014). Other approaches showed that the combination of reflection prompts with tutor or peer feedback supported more some motivational factors (Duffy & Azevedo, 2015; Van den Boom et al. 2007), making students dedicate more time and effort to learn and achieve their goals.

Moreover, Kramarski and Gutman (2006) implemented a tool that used metacognitive feedback as a method to support students’ self-monitoring. Besides, El Saadawi et al. (2010) found that when an intelligent tutoring system (ITS) provided students with immediate feedback, combined with other metacognitive scaffolds, had an impact on learning. Moreover, integrated support systems that used SRL-enabling tools or prompts focused more on cognitive or metacognitive scaffolds to provide instructions on regulation or elicit metacognitive activities in a collaborative learning environment (Molenaar et al. 2011, 2012) or help goal setting (Manlove et al. 2007). Yet, training proved to be an effective way to improve students’ SRL knowledge, metacognitive strategies, cognitive strategies (elaboration, organization and critical thinking), and resources management strategies, such as time-management and effort regulation (Broadbent et al. 2020).

Students’ self-regulation of behavior in online computer science education

In online computer science education, Pettit et al. (2015) found out that automated assessment tools (AAT) can be useful in reinforcing the accuracy of assessment, assisting teachers and enhancing student learning. Despite the proliferation of ATTs in the last decade, only few studies attempted to examine ways for supporting students’ self-regulation of behavior. Such studies used the Motivated Strategies for Learning Questionnaire (MSLQ) (Pintrich, 2004) to assess college students’ self-regulated learning in introductory computer programming courses (Alhazbi & Hassan, 2013; Bergin et al., 2005; Chen, 2002). The MSLQ instrument was used to measure students’ awareness of SRL skills and the positive effect of effort regulation on learning computer concepts (Chen, 2002). In their survey, Pettit et al. emphasized the need to explore students’ opinions more extensively in order to improve AATs’ design and use with new features.

Standard compiler/interpreter feedback and tracking of students’ actions have been considered as important features of assessment tools which have been used to analyze and assess students’ programming assignments (Grivokostopoulou et al. 2017; Rivers & Koedinger, 2017; Robinson & Carroll, 2017; Sukhoroslov, 2018). However, the effect of AAT feedback on students’ self-regulation of behavior has not been investigated.

Alharbi et al. (2014) designed an online learning object system to support the self-regulated learning of programming languages concepts. They emphasized the importance of improving students’ metacognitive skills by incorporating more features into the learning material, such as self-assessments with instant feedback, and self-reflection support. Pedrosa et al., (2019) used the SimProgramming pedagogical approach to help students improve their SRL strategies, namely: information search, work reviewing, time management, social seeking assistance, resolution of difficulties, and environmental structuring.

The relationship between CMC strategy use and self-regulation behavior

When learners are able to adopt various self-regulated learning strategies, such as behavioral and motivational, among others, they become active participants in their learning and they can manage their learning process better as well as improve their academic achievement (Panadero & Järvelä, 2015; Zimmerman, 2013). There are several SRL strategies which students may employ to help them study, apply knowledge and skills and ultimately learn. These strategies fall into at least four categories: (a) cognition (strategies to remember or elaborate information); (b) metacognition (planning, setting goals, monitoring, and evaluating); (c) motivation (enhance self-efficacy, intrinsic task interest); and, (d) behavior (help-seeking, time management, and creating a positive learning environment for learning task) (Andrade & Evans, 2013; Manso et al. 2016). Due to the broadness of these categories and the novelty of our research, we focused our study on the behavior regulation area.

Yet, there is strong evidence that the relationship between students’ Cognitive, Meta-cognitive and Critical thinking strategy use (CMC) and their self-regulation of behavior is also very important. On the one hand, self-regulation of behavior is not an innate personal characteristic; rather, students will be able to improve their self-regulation abilities especially when they are trained with CMC strategies (Wang et al., 2009). Students tend to self-regulate their behavior when they experience self-efficacy and a sense of achievement and mastery through successful application of CMC strategies (Cleary & Chen, 2009; Greene et al. 2010). On the other hand, in online learning environments, when students achieve adequate self-regulation of behavior, they can use CMC strategies more often, spend more time, and also produce better-structured assignments (Delen et al., 2014). CMC strategies can be considered specific skills that can be exercised and reinforced when students are able to self-regulate their behavior; doing so, they can apply them in real contexts more effectively (Zimmerman, 2013). As a consequence, the need to explore the relationship between students’ CMC strategy use and self-regulation behavior in the context of an online environment for programming learning is a challenging task.

Finally, the limited number of studies that used feedback as a means to support students’ self-regulation of behavior does not allow drawing any reliable conclusion on this matter. As a consequence, more systematic research is needed to thoroughly explore the effect of feedback on self-regulation of behavior in online learning environments. In the context of online distributed programming, this need is even more evident since there is no such an integrated support system developed so far. Our DSLab tool pretends to be such an integrated support system endowed with feedback and other features, aiming at improving students’ self-regulation of behavior when they deploy, execute and assess their distributed programming assignments.

Research aims

Context

Regarding the area of regulation of behavior, we consider the items of effort planning or management, time management, as well as help-seeking, which are the kinds of activities that are part of behavioral control (Pintrich, 2004).

Effort management involves attempts to control effort in order to perform well in a task. In the MSLQ instrument, effort management strategies were adapted from Zimmerman and Martinez-Pons (1986) and include scales that reflect students' attitude in working diligently as well as students’ persistence when working on tedious tasks (e.g. how students try to regulate their behavior in terms of their effort in the face of difficult, boring, or uninteresting tasks). These strategies are essential when students aim at achieving academic performance on different types of learning tasks (Corno, 1986; Zimmerman & Martinez-Pons, 1986, 1988).

Time management refers to “scheduling, planning, and managing one’s study time” (Pintrich et al. 1991, p. 25). Time management is a vital factor influencing online learning (Broadbent & Poon, 2015; Kizilcec et al. 2017). Time management is also related to work load, since students can take the initiative and decide how they will allocate their effort and the intensity of their work (Zimmerman & Martinez-Pons, 1986). The MSLQ has a scale focused on time management, called time/study environment. In this work we examine the items of time saving and workload reduction.

Finally, help-seeking is another behavioral regulatory strategy that can be very helpful for learning. It is important that not only self-regulated learners know when, why, and from whom to seek help (Babin et al. 2009; Karabenick & Sharma, 1994) but also the environment should provide adequate and timely help whenever needed (Stahl & Bromme, 2009).

The current study presents findings based on the analysis of students’ perceptions as to what extent our automated assessment environment (DSLab) facilitates a range of self-regulated learning strategies that students may employ during the realization of their programming assignments with the tool. In particular, it examines the following research hypothesis: “The DSLab environment helps students accomplish their assignment successfully by supporting them to self-regulate their behavior (SRB)”.

To that end we defined the following variables: effort management (EM), time management (TM) and help-seeking (HS) strategies. At the same time, we explored the relationships between students’ Cognitive, Meta-cognitive and Critical thinking strategy use (CMC) and their self-regulation of behavior. As such, we defined the following variables as well as: rehearsal (REH), elaboration (ELA), organizational (ORG) cognitive strategies, metacognitive (MET) and critical thinking (CT) strategies.

Research questions

We break down our research hypothesis into more specific research questions:

RQ1

To what extent has the DSLab tool managed to enhance students’ self-regulation of behavior?

RQ2

In the context of DSLab, has CMC strategy use helped students engage in more self-regulatory behavior in terms of effort and time management as well as helpseeking?

Method

Materials

The realization of our case study involved undergraduate students in a Distributed Systems online course who had to carry out a programming assignment that comprised four consecutive implementation phases. To that end, students could freely use the DSlab platform to upload each phase of their assignment and execute it, as many times as they wish, in order to assess the correctness of its code. As soon as students achieved a correct code, they could pass to implement the next phase until they complete the assignment. Students are able to use DSLab feedback from a previous execution to explore its logs, review their code for possible errors and thus improve their program. Immediate feedback constitutes an important feature of DSLab. Figure 1 (result of all submissions) and Fig. 2 (execution logs) illustrate how feedback functions in DSLab and is presented to a student. More information about DSLab features and functioning can be found in Marquès et al. (2020).

Fig. 1
figure 1

A completed assessment of an assignment in DSLab

Full size image
Fig. 2
figure 2

An extract of the execution logs of level INFO and TRACE generated during the execution of an assignment

Full size image

In general, students can look up the logs in the web interface (as shown in the snapshot) or download all execution logs in a zip file that contains a file for each instance. The file of each instance contains the logs generated by that instance.

Participants and procedure

Participants were a sample of 132 online university students who performed the same assignment during seven weeks either individually or in pairs. Students’ perceptions were collected and analyzed though an online questionnaire at the end of the learning experience and then went through an in-depth quantitative analysis. The data collection was on a voluntary basis. We designed a customized questionnaire with the aim to specifically respond to our RQs, which allowed us to collect quantitative data for our analysis (Table 1). The questionnaire was based on and adapted from the Motivated Strategies for Learning Questionnaire (MSLQ) of Pintrich (2004), focusing primarily on the behavior regulation area and secondarily on the cognition regulation area. MSLQ is a widely used instrument to assess college students’ self-regulated learning (Chen, 2002) and its validity has been supported through extensive literature on college student learning and teaching (Gable, 1998; Taylor, 2012). The adaptation of the original MSLQ questions was guided by the particular context of our study, its application to the programming field and the features of our DSLab environment (e.g. the tool feedback).

Table 1 Questionnaire of twenty-one (21) question items related to students’ perceptions of cognitive, metacognitive and critical thinking strategy use and self-regulation of behavior
Full size table

Data collection

The questionnaire aimed to examine students’ perceptions of self-regulation of behavior (RQ1) as well as the correlation between students’ cognitive, metacognitive and critical thinking strategy use and their self-regulation of behavior (RQ2). As such, it included 21 items divided into two sections (one for each RQ), as shown in Table 1.

As regards RQ1, we associated a set of questions to the three variables (EM, TM and HS) we defined for measuring students’ perceptions of self-regulation of behavior.

Effort management (EM) strategies reflect students' attitude in working diligently (question 13) or students’ persistence when working on difficult, uninteresting, or boring tasks (questions 14, 15 and 17 respectively). Time management (TM) is related with workload reduction (question 19) and time saving (question 20). Help-seeking (HS) strategies are related to students’ motivation for making full use of and deriving the best benefit from the tool feedback (questions 16 and 17) as well as to students’ perceptions of receiving timely feedback whenever needed (question 21).

As regards RQ2, we associated a set of questions to the five variables (REH, ELA, ORG, MET and CT) we defined for measuring students’ cognitive, metacognitive and critical thinking strategy use.

The use of specific metacognitive strategies (MET) has been captured by MSLQ question items which were adapted to examine whether students used ways for achieving comprehension monitoring (e.g., self-questioning-question 5), regulation or modification of learning (e.g. re-reading-question 7) and planning of learning (question 10-think of alternative solutions, i.e., programming code variations, and test whether the tool provides flexibility by accepting them as correct). Similarly, the use of specific critical thinking strategies (CT) has been captured by MSLQ question items which were adapted to examine whether students used ways for applying prior knowledge to new situations (apply prior knowledge, obtained when students used the tool to execute previous assignment phases, to do a new phase-question 6), and analyzing and evaluating information in a thoughtful manner (e.g. use prior experience to analyze information provided by tool feedback more thoughtfully – question 9).

The questionnaire was finally answered by 111 students, that is, more than 84% of the participants.

A five-point Likert-type scale was used on a continuum from 1 (Strongly disagree) to 5 (Strongly agree); therefore, quantitative data is obtained.

In addition to the quantitative data collection, we also saw the need to provide some qualitative data so that to supplement the overall description of the results and to help clarify key points being made, as we show in next Section The results with regard to the RQ1. To collect the necessary qualitative data, we performed short interviews with focused questions on those questionnaire items that required clarifications. Interviews were conducted with student volunteers. Ten participants were randomly selected out of 35 students who voluntarily expressed their willingness to participate.

Data analysis

Data were analyzed using descriptive statistics techniques, based on the Kolmogorov–Smirnov test for a sample and a frequency table. In addition, we calculated Pearson correlations between certain variables of our study with the aim to provide a more comprehensive answer of our research questions.

The Cronbach’s alpha coefficient was applied to the student data to ensure the reliability of data collection. In the analysis of questionnaire data, we obtained a Cronbach’s alpha (0.78), which being higher than 0.70, strengthens the reliability of our items.

In addition, we examined the coefficients of multivariate skewness and kurtosis for assessing multivariate normality. Critical values of all test statistics were calculated. The results showed that data were normally distributed as absolute values of skewness and kurtosis did not exceed the allowed maximum (2.0 for univariate skewness and 7.0 for univariate kurtosis). The statistical results are presented in detail in the following section.

Results

The results with regard to the RQ1

Here, we analyzed students’ perceptions of self-regulation of their behavior in relation to three parameters: (1) EM: Students’ use of effort management strategies; (2) TM: Students’ use of time management strategies, and (3) HS: Students’ use of help-seeking strategies.

The corresponding descriptive statistic measures of the above items are presented in Tables 2 and 3. The calculation of the significant values in Table 2 is based on the Kolmogorov–Smirnov test for a sample.

Table 2 Kolmogorov–Smirnov test for a sample
Full size table
Table 3 Frequency data concerning students’ perceptions of self-regulation of their behavior with regard to the three parameters: AM, TM and HS
Full size table

Tables 2 and 3 indicate that all EM variables (except EM2) obtained significant values. The same holds for HS variables (except HS1 that obtained rather neutral values). Instead, TM variables obtained non significant values.

Effort management (EM) involves attempts to control effort in order to do well in the course. As such, it includes students' persistence (showing determination, perseverance, dedication, commitment, or assiduousness) at difficult or boring tasks as well as working diligently (being hard-working, assiduous, rigorous, thorough, attentive, dedicated, or committed). In our study, we tried to reflect whether students showed diligent work habits (EM1), or they tried to regulate their effort in the face of difficult (EM2), uninteresting (EM3), or boring (EM4) tasks.

Our analysis shows that, in the context of DSLab, students generally employ effort management strategies in order to carry out the assignment effectively. In particular, a large number of students (72%) reported showing commitment to understanding what the tool feedback was saying even if it didn’t make sense to them (EM1). In addition, 65% of students reported perseverance and kept on working willingly until they finished the assignment, though they did not find it so interesting (EM3). Moreover, 67% of students opined that they worked hard to get a good grade even if they did not like the assignment (EM4). Yet, when students realized that the assignment was difficult for them, their opinions were divided regarding the effort they should dedicate for it; in fact, 44% of students thought of giving up or doing only the easy parts against 42% of them who were determined to do their best in completing this assignment. In general, students have been trying to regulate their behavior in terms of their effort in a positive way.

Time management (TM) in the DSLab environment involves finding and allocating time slots to cope with the different phases of the programming assignment as much effectively as possible. The use of the tool creates expectations for time saving in many students. In addition, students’ interest focuses on how the tool can help them alleviate their effort and the intensity of their work (work load reduction). Our analysis showed that time management was not sufficiently supported by DSLab tool. Just 28% of students perceived that the use of the tool had reduced their workload (TM1). A few more (37%) agreed that the tool saved them time (TM2). More than a third of the students expressed their disagreement that using the tool could facilitate workload reduction or time saving. This might be connected with the fact that some students had to perform multiple submissions of the same assignment so that they achieve a correct solution or with their effort to read and analyze the tool feedback. To gain a better insight into the factor that most negatively affected time management, students’ qualitative answers were obtained. Their answers revealed their preference for more expressive and explanatory feedback, related to the detected errors and providing a clearer guidance to error correction.

As regards help-seeking (HS), students could seek help either from their tutor or from DSLab tool so that to tackle problems when executing their assignments in this environment. Here, help-seeking is closely related to the assistance that DSLab tool provides the students. Students expect adequate and timely help whenever needed. Our analysis of indicators HS2 and HS3 reveals students’ clear tendency for seeking help from DSLab tool. In particular, a large number of students (73%) reported showing a clear preference for using the help of DSLab tool in order to obtain a more robust solution, than doing the assignments without it (HS2). Moreover, the great majority of students (80%) opined that the tool facility to test different solutions was very helpful, whereas they appreciated the quick feedback offered by DSLab as a valuable help, since it allowed them to check the correctness of their assignments rapidly and reliably (HS3). Yet, they often found that, when the tool was giving them feedback, they did not feel too motivated to take full advantage of what the feedback really offered them (HS1). Just 33% of students felt very motivated to be fully involved and get the most out of the feedback help. Most students reported being neutral (45%) or negative (22%) with regard to this aspect. In general, students were selective with the tool feedback, perhaps because it was too technical or they wanted to get straight to the point.

In conclusion, the DSLab tool provided an environment which managed to enhance students’ self-regulation of behavior to a large extent. In particular, it promoted effort management considerably, since students broadly employed effort management strategies, showing persistence and working diligently, thus being able to regulate their effort in case the assignment seemed boring or uninteresting to them as well as for understanding the tool feedback. When the assignment seemed difficult, students’ opinions were divided. In addition, the great majority of students found in DSLab an appropriate environment for responsible and solid help-seeking to handle their assignments. Feedback could certainly be a better and more effective means of reducing students’ workload and saving them time. The latter constitute two crucial issues for online students. For this reason, our research is now focusing on increasing DSLab feedback flexibility with an educational chatbot for answering students’ queries, which could add more expressiveness and more personalized guidance to the students needs, thus improving time management.

The results with regard to the RQ2

We conclude the present study by exploring the relationships between students’ perceptions of cognitive, metacognitive and critical thinking strategy use (CMC) and their self-regulation of behavior (SRB). To that end, we provide the Pearson correlations between the variables that comprise CMC and the variables that constitute SRB, as shown in Table 4. We analyze these results in the Discussion section.

Table 4 Correlation between cognitive, metacognitive and critical thinking (CMC) and self-regulation of behavior (SRB)
Full size table

Discussion

In the previous section, based on descriptive statistics analysis, we found that the fact of working with DSLab tool fostered students’ self-regulation of behavior to a large extent (RQ1), namely effort management and help-seeking strategies, except time management. Moreover, our analysis gave us the opportunity to reveal some important shortcomings with the feedback tool functionality which, if improved, could further increase students’ self-regulation of behavior. On the basis of these results, in this section we seek to delve more deeply at the research question RQ1 and seek an answer for RQ2, comparing our results with relevant findings of the existing literature.

Enhancing students’ self-regulation of behavior (RQ1)

Here we want to provide a more comprehensive response as to what extent DSLab has managed to enhance students’ self-regulation of behavior (RQ1) within the context of the existing literature, taking into account that no previous studies have attempted to answer this question in the area of programming in a systematic manner.

As concerns effort management (EM) strategies, the results of the study revealed that DSLab provided an environment where most students worked diligently to understand the tool feedback (EM1) and they really tried to regulate their effort in the face of uninteresting (EM3) or boring (EM4) tasks. When students found that the assignment was difficult, their opinions were divided regarding the effort they should dedicate for it. The importance of effort management has been highlighted in various studies, since it proved to be a strong predictor of academic achievement (Altun & Erden, 2013; Kim et al. 2015; Tanriseven, 2014; Wolters et al. 2005). This makes an important contribution to the field of programming learning assisted by automated assessment tools.

Help-seeking has been one of the various SRL strategies students were encouraged to use by the SimProgramming approach in a computer programming course (Pedrosa et al., 2019). The online learning object system designed by Alharbi et al. (2014) to support the self-regulated learning of programming languages concepts also considered help-seeking as one of the six SRL strategies indirectly supported by the system. In studying the most relevant factors in predicting computer science students’ online help-seeking behaviors, Hao et al. (2016) found that problem difficulty is the most important predictor. Unlike these studies, help-seeking has been directly and immediately supported by our DSLab tool feedback, since a large number of students have explicitly used it to correct or improve their code, though one third of the students exploited the full potential of feedback.

Finally, time management is an aspect that DSLab definitely has to improve, since a few students have perceived that the use of the tool had reduced their workload and saved them time. Since learning programming is a complex task requiring thinking and creative skills in problem solving, time management is crucial for students’ success. However, no systems have been developed so far to provide support to time management strategies in computer programming courses. In SimProgramming approach, Pedrosa et al. (2019) supported time management by encouraging students to develop awareness of doing their work regularly and adopting study routines. To that end, tutor feedback was used to accompany the tasks that students performed, offering at the same time monitoring support for self-reflection and self-regulation. In their study of the relationship between learning styles and self-regulation in programming, Çakiroglu et al. (2018) found that time management was identified as a leading self-regulation strategy among learning styles. As a consequence, more research is necessary to better understand and facilitate students’ time management strategies in online environments for programming learning.

The relationship between students’ perceptions of CMC strategy use and their self-regulation of behavior (RQ2)

An important contribution of this study is to explore the correlation between students’ perceptions of cognitive, metacognitive and critical thinking strategy use (CMC) and their self-regulation of behavior (SRB); namely, whether CMC strategy use helped students engage in more self-regulatory behavior in terms of effort and time management as well as help-seeking (RQ2). As far as we know, no studies have systematically explored this relationship. In this respect, we go back to Table 4 (in the Results section) which presents the Pearson correlations between CMC and SRB variables.

We see that there is a positive correlation between rehearsal strategies and effort management strategies only. In particular, the more students chose to perform multiple submissions of their assignment (REH), the more effort they dedicated to understand the tool feedback in order to guarantee the assignment correctness (EM1). Moreover, rehearsal strategies helped increase students' persistence (effort) at uninteresting (EM3) or boring tasks (EM4), making them to continue working until they finished and get a good grade. In addition, rehearsal strategies do not show any relationship with issues such as giving up the assignment or just do the easy parts in case students considered the assignment difficult (EM2). In other words, rehearsal strategies have a tendency to increase students’ regulation of their effort in the face of difficult, boring, or uninteresting tasks. However, rehearsal strategies did not show any significant relationship with time management and help-seeking. Students perceived rehearsal strategies as time-consuming, whereas it seems that they do not consider them as an important reason or means for seeking help from DSLab tool.

Elaboration strategy use in analyzing the tool feedback may serve to reinforce students’ regulation of their effort, time and help-seeking behavior in several situations during the realization of their assignments. Indeed, elaboration strategies correlated positively with effort management strategies in the sense that students’ strategies (ELA1, ELA2) used the tool feedback in order to correct their mistakes or to find clues that could incorporate in their code. This signified that students tried to regulate their effort in situations where they were striving for understanding the tool feedback (EM1) or when the assignment was not so interesting to them (EM3), avoiding thinking of giving up or doing only the easy parts (EM2). Moreover, elaboration strategies, such as transforming hints found in the tool feedback into important ideas in the code (ELA2) had very positive, significant correlations with time management. In addition, elaboration strategies correlated positively and significantly with help-seeking. Indeed, the students’ analysis of the tool feedback in order to correct their mistakes or to find clues that could incorporate in their code (ELA1, ELA2) signified that students opted to use DSLab so that they achieve a more robust solution (HS2) and, most importantly, to get quick feedback when testing different solutions (HS3). Yet, both elaboration strategies had a rather neutral relationship with students’ motivation to take full advantage of the tool feedback (HS1).

Organizational strategies did not show a significant relationship with effort management; instead, they proved to be crucial for time management whereas they had a tendency to prompting students to seek help from the tool feedback. As regards time management, all three organizational strategies (organizing different pieces of information–ORG1, discerning useful information from the tool feedback–ORG2, or outlining the best feedback to use it in the next phases more effectively–ORG3) had very positive, significant correlations with time management. As regards help-seeking, when students tried to organize different pieces of information, including information from the tool feedback (ORG1), this had a tendency to increase help-seeking behavior from the tool quick feedback when testing different solutions (HS3), while it increased students’ motivation to take full advantage of the feedback information (HS1). When students were able to discern useful information from the tool feedback (ORG2), they felt more capable of using DSLab both to obtain a more robust solution (HS2) and use the tool for testing different solutions based on its quick feedback (HS3). The latter (HS3) was also supported when the students were able to outline the best feedback in order to use it in the next phases more effectively (ORG3). This organizational strategy also correlated positively, though moderately, with students’ motivation to take full advantage of the tool feedback (HS1) and students’ use of DSLab to obtain a more robust solution (HS2).

Metacognitive strategies (MET) seem to improve all effort management strategies, may contribute to time saving to some degree but not to workload reduction. They also showed a positive involvement with help-seeking only in some aspects. More specifically, MET strategies associated with monitoring learning (through monitoring feedback comprehension–MET1) or regulating/changing learning (through rereading feedback–MET2) correlated significantly with all effort management strategies. Using these strategies, students showed persistence and worked diligently with tool feedback to achieve quality performance both at product (EM1, EM3) and grade level (EM4). MET strategies also helped students overcome tendencies of giving up or doing only the easy parts when encountering difficulties with the current assignment (EM2). As regards time management, there was only a significant but moderate correlation between monitoring feedback comprehension (MET1) and time saving (TM2). Regulate or change learning through rereading feedback (MET2) had no influence on time management. Moreover, the students’ perception that the use of DSLab made them the notion of assignment correctness too strict (MET3) has not helped neither reduce students’ workload (TM1) nor save them time (TM2). As regards help-seeking, on the one hand, monitoring feedback comprehension (MET1) had a significant positive correlation with making students use DSLab to obtain a more robust solution (HS2) and to test different solutions based on the tool quick feedback (HS3). On the other hand, regulate or change learning through rereading feedback (MET2) had no influence on help seeking (HS2, HS3). Moreover, the students’ perception that the use of DSLab made them the notion of assignment correctness too strict (MET3) has constrained several students from using DSLab to obtain a more robust solution (HS2) and to test different solutions based on the tool quick feedback (HS3). In addition, all MET strategies did not have any significant correlation with HS1, which means that these strategies have not encouraged students’ motivation to take full advantage of the tool feedback (HS1).

Finally, in the critical thinking (CT) strategies subscale, positive significant correlations were found with effort management, time management to a greater or lesser extent, and help-seeking to a considerable degree. As regards effort management, using CT strategies, students worked diligently both to apply prior knowledge to new implementation phases of their assignment (CT1) and to analyze and evaluate information of the tool feedback in a thoughtful manner (CT2), aiming at achieving high quality assignments (EM1, EM3). CT strategies also helped students stop thinking of giving up or doing only the easy parts, if they realized that the assignment was difficult for them (EM2). As concerns time management, the use of analysis and evaluation of information of the tool feedback in a thoughtful manner (CT2) had a significant relationship with time management, helping students both reduce workload (TM1) and save time (TM2). Yet, the use of the other CT strategy (apply prior knowledge to new situations–CT1) correlated positively though weakly with time management. As regards help-seeking, both CT strategies, namely “apply prior knowledge to new implementation phases” (CT1) and “analysis and evaluation of information of the tool feedback in a thoughtful manner” (CT2) had significant, though moderate, relationships with all three help-seeking strategies (enhancing students’ motivation to take full advantage of the tool feedback–HS1, using DSLab to obtain a more robust solution–HS2, or using DSLab to test different solutions based on the tool quick feedback–HS3).

The above results echo findings from other studies indicating that students who typically use more cognitive and metacognitive strategies (with the exception of rehearsal) tend to seek help when it is needed (Karabenick & Knapp, 1991). Improving the relationships between metacognitive strategy use and help-seeking could lead to adaptive help-seeking which involves self-reflection and self-related affective motivational factors (Newman, 2008) which, in turn, positively relates to academic achievement ((Zimmerman et al. 1992).

Conclusion and Limitations

This work analyzed the extent to which an online Distributed Systems Laboratory (DSLab) managed to enhance students’ self-regulation of behavior. It also explored the correlation between students’ cognitive, metacognitive and critical thinking strategy use and their self-regulation of behavior. Our results showed that when students upload, execute and assess their programming assignments in the distributed environment of DSLab, they perceive that:

  1. (1)

    DSLab enhances students’ self-regulation of behavior to a large extent. In particular, it promoted effort management considerably and allowed students to use help-seeking strategies in an effective and responsible way. However, students were not able to use time management strategies efficiently so that to reduce their workload and save time as they wished.

  2. (2)

    Regarding the interplay between students’ cognitive, metacognitive, critical thinking strategy use and self-regulation of behavior, in the context of our DSLab tool, cognitive elaboration and organizational strategies showed a significant relationship with most aspects of students’ self-regulation of behavior (time management and help-seeking). Most importantly, students using metacognitive and critical thinking strategies in DSLab had in general positive correlation with all three aspects of students’ self-regulation of behavior (effort and time management as well as help-seeking) to a greater or lesser degree.

An important conclusion of the above results is that if cognitive elaboration and organizational strategies can be used effectively in the DSLab environment they can contribute to reduce students’ workload and save them time. These are important considerations we have to take into account and reinforce further in future versions of DSLab. Our results also showed that the metacognitive strategy use needs to be further reinforced in order to provide a more efficient time management. In addition, critical thinking strategy use has shown to be crucial for achieving high quality assignments, so special care has to be taken in promoting its involvement in future versions of DSLab.

Limitations of the current work reveal the need for future research. First, the current research constitutes the first step in analyzing students’ self-regulation of behavior (effort management, time management and help-seeking) in a real, long-term learning situation in a systematic way. Certainly, more experimental studies are needed to confirm, better understand and extend these findings further. Second, an important issue that we did not deal with in this work is context and its regulation. Regulating context implies attempts to control and regulate the tasks and learning environment context in order to enhance students’ task and goal accomplishment (Corno, 1986; Kuhl, 1984; Pintrich, 2004). Our online environment follows a student-centered approach where students are able to manage the climate and structure of their environment. In fact, they are responsible of uploading and evaluating their programming assignments in DSLab, so they actually take the control and regulation of their tasks, whereas they also have to decide whether they will perform the tasks individually or they will form a group and elaborate the tasks collaboratively in pairs. Considering the mode of working individually or in group constitutes an additional variable, which is a limitation that can be addressed in future studies. As Zimmerman (2008) argues, when students need to achieve an efficient working environment, especially when they have to collaborate and interact with their peers at a stable base, they use specific monitoring methods that support collaborative learning through self-regulation, co-regulation, or socially shared regulation (Hadwin et al. 2011). Another important issue is regulation of motivation and affect, which includes attempts to control self-efficacy, affect and emotions through the use of various strategies (Panadero, 2017; Pintrich, 2004). Finally, we would like to highlight some implications for research and technology design beyond our specific tool. A very important issue for online environments aiming at improving students’ self-regulation and engagement is the design and production of intelligent interaction with the students through a variety of techniques, such as visualization of students’ working and learning process as well as facilitation of notifications, recommendations and personalized feedback in a timely manner. To this end, a new tendency to the design of online environments is to incorporate educational chatbots which can provide customized solutions to students’ queries whereas making it fun for students to learn (Quiroga et al. 2020).