Lessons Learned from 6 Years of a Remote Programming Challenge Activity with Automatic Supervision

Abstract

In an “Introduction to Programming” course dedicated to first-year students, many students tend to procrastinate and do not autonomously process step by step new topics taught over time. In response to that trend, a tool (called Café) was implemented to supervise a remote activity spread over the semester, by instantaneously correcting students’ exercises and providing feedback to guide them in refining their solutions. This paper presents and discusses the current impact of the system on students’ learning, based on six years of activity. The results validate the high potential of such a tool, but also highlight many students do not take advantage of that opportunity to boost their learning. That opens doors to some significant upgrades in the tool, mainly consisting in offering a closer guidance to students through a larger range of regular activities. While Café was initially standing as an isolated tool offering correction and feedbacks, this paper advocates for Café becoming an integral part of the course, leading to a consistent synergy between in-person and continuous remote learning.

1 Introduction

For first-year students, Higher Education is a new ground with higher requirements and more freedom compared to Secondary School. To support their learning, they need to self-regulate [20], as their success heavily relies on their ability to autonomously and actively engage in their learning process [19]. In particular, in our “Introduction to Programming” course, it is essential to stay on track over the semester as the topics are cumulative. However, in practice, many students have difficulties in managing the amount of time and quality of cognitive effort devoted to learning [4, 11].

From a context point of view, our country (i.e., Belgium) applies an open policy access to Higher Education. It results in large groups of first year students for which traditional classroom activities are organized. Moreover, students are fully free to take part (or not) in the academic activities, their only commitment being passing the final exam. Therefore, during the semester itself, we need to promote regular students’ training, handling the diversity and the large number of students, for a limited number of supervisors.

With this purpose in mind, six years ago, we developed a remote regular activity, the Programming Challenge Activity (Pca). The activity spans over the whole semester and is made up of six programming Challenges targeting to punctuate students’ learning by providing short-time goals. For supporting the Pca, we implemented a tool aiming to remotely guide students’ learning while maintaining supervisors’ workload feasible. More specifically, that tool, called Café (“Correction Automatique et Feedback pour les Ëtudiants” [9]) corrects students’ work and provides instantaneous personalized feedback and feedforward, based on their mistakes, encouraging reflection and self-regulation [3]. That new approach was the first step towards Blended Learning, combining face-to-face and computer-mediated instruction.

Overall, from the six last years, we see that our current remote system does involve some students in their learning while some others do not take it as an opportunity to learn. We also notice that despite that some students took part in this remote activity, they still demonstrate a deep lack of knowledge and skills during the final exam. In response to those main observations, we intend to refine our system in accordance with students’ actual needs.

To put in place those enhancements, we take a step back and bring the light on both strengths and weaknesses of the current version of Café in the context of the Pca. More specifically, in this paper, we closely analyze students’ participation to the remote activity over the semester. Then, we examine how students interact with the tool as well as how it impacts their learning. More precisely, we are interested in how students self-regulate to perform the Challenges with that automatic supervision and how their self-regulatory skills are related to their performance. In the context of this paper, we restrict ourselves to students’ time management to represent those self-regulatory skills. Finally, from our study, we identify the features of success that should be integrated in Café in the future in order to onboard more students over all the semester and further boost students’ learning.

2 Context

2.1 The Course

Café was introduced in the context of the course “Introduction to Programming” (abbreviated here in “CS1”), provided to first year students (Bachelor level). It is organized during the first semester of the academic year with exams in January, preceded by a 15-day study period. The CS1 course consists of theoretical lectures (ten sessions), practical sessions (exercises on paper - ten sessions), laboratory sessions (exercises in front of a computer - five sessions). Lessons and exercises sessions typically last two hours.

Over the semester, maintaining students on track is essential as the topics taught in previous sessions are often prerequisites of the new coming ones. Under that concern, we boost student engagement by building the course around Assessment for Learning (AfL) [10, 13]. In practice, a Mid-Term evaluation is organized during the first week of November for all courses of the semester [10] and six Programming Challenges are given over the semester, as illustrated in Fig. 1.

Fig. 1.

CS1 course timeline, highlighting the Challenges as well as their general timeframe.

2.2 The Programming Challenge Activity (Pca) Supported by Café

The Pca is made up of six Challenges. Regarding the content, a Challenge is a statement aligned to the chapter taught the week(s) before. Like the chapters, Challenges are cumulative, requiring a good level of understanding about the previous topics to be properly handled. Each Challenge consists in producing some pieces of code. For Challenges 2, 3 and 4, students must also provide some graphical reasoning by filling a given canvas.

Regarding the modalities, each Challenge from 1 to 5 represents 2% of the students’ final mark while the first Challenge (“Challenge 0”) just gives the opportunity to get used to the system. Students get 2 days to individually perform the Challenge. As depicted in Fig. 1, each Challenge is published on Wednesday 17:00 and submissions are allowed until Friday 19:00.

Students get three submission shots per Challenge, where the last attempt is considered in the final grade.

As shown in Fig. 2, each submission is instantaneously processed by Café that computes, highlights what should be adapted in the current submission (through the feedback), and provides pointers to the theoretical courses (through the feedforward). In this way, students get the opportunity to realize their misunderstanding and improve their subsequent submissions. Figure 2 also highlights that, as supervisors, in addition to be timesaving and scalable, such a system allows us to keep track of student’s behavior by collecting data related to their activity and performance.

Fig. 2.

Students’ interaction with Café in the context of the Pca.

3 Method

3.1 Data Sources

Data was collected over the six last years (from 2016 to 2021) according to the 3 P’s framework [18] that recommends to consistently analyze any pedagogical innovation by gathering and meshing three types of data reflecting dimensions of students’ learning experience therewith: Participation, Perception, and Performance data.

3.2 Participation Data

In this paper, Participation Data reflects (if and) how students interact with Café, in the context of the Pca. More precisely, for each Challenge, we recorded each student’s submission timestamp. From that timestamp, we can also easily derive the elapsed time between the moment the Challenge statement was published and the student’s first submission (e.g., A student submitted their work on Thursday at 16:20 while the current Challenge was published on Wednesday at 17:00. The corresponding elapsed time is about 23 h). In Sect. 4, the “elapsed hours” unit is used to represent the submission slot time.

3.3 Perception Data

For academic year 2021–2022, an anonymous survey was administered to students at the end of their exam. 71 students shared their opinion. The survey was made up of Likert scale questions, asking about their experience with Café in the context of the Pca.

Notice that for the previous year, a survey was also addressed to students, but it was not mandatory and sent after the exam (i.e., during the second semester), leading to few answers and an overrepresentation of opinions from students who were involved in the course. Because of that bias, those answers were not included in our analysis.

3.4 Performance Data

All the grades every student obtained for the different Challenges were recorded. Moreover, we are also interested in the mid-term and the exam grades, considering that they model at best how much students learnt from the course at a given point in time.

4 Results and Discussion

In this section, the goal is to assess the current version of Café through students’ experience over the Pca. To lead our analysis, we first identify how much students use Café by taking part in the Pca (Sect. 4.1) and how much it impacts on students’ success (Sect. 4.2). Then, we deepen our research by studying how students use the tool (Sect. 4.3) and how that learning behavior is related to their performance (Sect. 4.4).

Table 1. Number of students enrolled to the course from 2016 to 2021.

Fig. 3.

Evolution of the proportions of students taking part in the Challenges from 2016 to 2021. See Table 1 for the raw number of students per year.

4.1 How Students’ Participation Evolves over Time?

From Fig. 3, we can see that the participation varies quite similarly over the semester, from 2017 to 2021. Besides this, 2016 stands apart. Despite a participation decrease from Challenge 2 to Challenge 5 also occurred that year, we can notice that this reduction was slighter compared to the next years and the global participation remained quite high. When investigating deeper, we could note that in 2016, students outperformed in general, whatever the courses, compared to the other years.

Besides this, Fig. 3 depicts that, for each Challenge, there are always students who do not take part in it. Taking a closer look, we computed that 7% of students never participated to any Challenge. The possible explanations are that the Pca supported by Café is not attractive enough and/or the level of the Challenges is not adapted to some students. Another aspect to consider is that some students attend the course for a second time, meaning that they are already familiar with the course and may directly choose to handle it in their own way. Next to this, we computed that 43% of students participated to less than five Challenges, in accordance with Fig. 3 showing fewer participants in the last Challenges.

More precisely, the participation is the highest for Challenges 1 or 2, reaching a range between 77% and 85%. Participation to Challenge 0 is lower (despite its easiness to manage), probably due to the fact it does not contribute to the final grade. That suggests that some students’ work is driven by grades. To go further, we can observe that, every year, participation drops across the last four Challenges and falls to a range between 29% and 46% (2016 being put apart). More specifically, from 2018 to 2021, we can notice a recurrent significant decrease (by 20%) from Challenge 2 to Challenge 3. We can relate it to Mid-Term organized between those two Challenges (as depicted through the timeline in Fig. 1). Indeed, the failure rate in the Mid-Term is quite high in the CS1 course as well as in the other courses, leading to many students feeling demotivated. Moreover, the chapters are getting harder and harder over time, with many dependencies on the previous ones. That means that once students misunderstand some concepts, they cannot keep learning properly without reinforcing first those prerequisites seen previously. Both aspects combined likely lead some students to lose the track over time. That inference gets enforced by computing the correlation between the mid-term grades (reflecting students’ level after 3 Challenges) and the number of Challenges students took part in over the semester. The resulting Pearson coefficient is 0.51, held up by a p-value of ${4.68\text {e}-27}$, meaning that, for a given student, the higher their mid-term grades, the more they are stimulated to participate to the Challenges.

Fig. 4.

Students’ opinion about how much appropriate the Challenges level is.

From Fig. 4, we can observe that students’ opinions are quite divided. 64% of them see the Challenges as an opportunity to train (those being manageable) and 46% find the Challenges give a chance to get better through the feedback. On the opposite, for other students, the Challenges appears too difficult, and they cannot take advantage of the feedback.

To wrap up those last results, we saw that around 60% of students keep performing the Challenges over the semester. On the contrary, a non-negligeable number of students fall behind with the Pca. From those observations, we can draw two main learner profiles: one capturing the participants to the Challenges and another one referring to students who do not take part in it. A likely general root cause to this recurrent clustering is the large diversity of students’ profiles since there is no prerequisite to enter the cursus (due to open access policy in our country). We can also notice that the group of students who do not take part in the Challenges grows over time. Very likely, some students find the course hard. They do not see themselves succeeding in the mid-term evaluation and the Challenges and get eventually demotivated. The same phenomenon is observed in the other courses during the semester (e.g., Physics, Math). To overcome it, students need to regularly train, which makes Café necessary, seeing the large number of students that should be assisted. However, as it stands, Café does not seem to offer the learning experience some students need, leading them to stop taking part in the Pca Café is supporting. In further subsections, we analyze more closely how students handle the Challenges through Café and what is their performance, in order to catch how Café contributes to learning and understand why many students lose the track over the semester.

4.2 What’s the Impact of Café on Students’ Success?

Figure 5 reflects a linear relation r = 0.57 (Pearson coefficient), indicating that the more Challenges students tackle over the semester, the higher their grades in the exam. In particular, from Fig. 5, we can notice that students performing less than five Challenges usually do not outreach the average grade. Students need to tackle five (ideally six) Challenges to really forge ahead and maximize their chance to pass the exam. That demonstrates the interest of the Pca that covers and boosts the whole course through the six proposed Challenges. However, it is also important to recall that all participants chose to be participants, meaning that they tend to be more involved in general.

Fig. 5.

Correlation between the participation to the Challenges and the learning rate (aggregation over the six years of interest). The histograms (top and right of the graph) gives the number of students per X-Axis or Y-Axis value. Grades range within [0; 20], 10 being the success/failure threshold (illustrated by the horizontal dashed line).

Besides this, if we restrict our analysis only to the grades, those may appear quite low (most of them being below-average), even when students tackled five or six Challenges. A likely explanation is that some students rush in handling them (that trend being investigated in the next section). Another possible issue is that students cannot draw any lessons from the feedback that is provided. In fact, that last assumption is strengthened by some other studies showing that, more often than expected, students do not read feedbacks at all, especially if they perceive the task as too complicated [17].

In addition to this, we can qualify the impact of the Challenges and Café (both being linked to the other) by completing those pieces of data with some students’ opinions.

Fig. 6.

Students’ opinion about how much Café helps in learning in the context of the Challenges.

Figure 6 shows how much students agree on five assertions, the first two ones being related to the remote activity itself and the last three ones to Café. First, most students consider that the Challenges are a good indication on the skills they are expected to demonstrate and attest that those regular statements are a good preparation to the final exam. That underpins the purpose of Café that supports scalable regular activities over the semester. Next, regarding the three statements about Café itself, we can see that opinions tend to be more blurred. 69% of students claimed that the system encouraged them to refine their submission (recall that, for a given Challenge, a student can submit up to three times its solution), which suggests that they can process quite well the feedback they receive. It gets confirmed when we compute the average of the improvement rates of all students, across all Challenges, reaching 29%. To go further, through the next claim, we can notice that 60% of the students say they realized their shortcomings thanks to Café ’ feedbacks. However, only 42% think that Challenges effectively helped in understanding better the course, while 27% have no opinion about it and 20% believe it did not bring any added value in their learning. Notice that the rest of the students did not provide any opinion, meaning that, likely, they did not take part in the challenges. From those last three opinions, it seems that some students do not always connect their “local learning” (i.e., what they found out during the Challenge and used to improve their solution) to the global picture of the course. Some may only focus on maximizing their score on the Challenges without keeping track of their weaknesses that are being highlighted and overcome them in the future. The consequence is that we often see the same mistakes occurring across the Challenges, as well as in the exam.

All in all, Figs. 5 and 6 corroborate the conclusion from Sect. 4.1. First, it highlights the potential of Café in boosting learning. Indeed, the more Challenges students participate in, the higher their final exam grades. However, currently, the impact of Café is limited. On the one hand, many students do not think that Café really brings the light to the concepts of the course. That explains why about 40% of them stopped taking part in the Challenges over the semester. On the other hand, the global results of the exam remain quite low, even when students participated in all the Challenges. Those two observations raise the new question: “Why some students do not learn (enough) from their experience with Café?” To answer it, we will focus on students’ learning behavior across the Challenges in order to catch how they use that tool in practice.

4.3 How Students Manage Their Time to Handle the Challenges?

Figure 7 shows how the students’ first submissions are spread over time with respect to the time the Challenge was published (referred by 0 on the X-Axis). From that figure, we can note that a lot of students (from 38% to 72%) wait for the last day before sending their first submission. Moreover, we can notice that this behavior intensifies over time, which is reflected through the curves that shift downwards across the Challenges. More precisely, for the first two Challenges, more than 53% of the students submitted the first version of their work before the last day while only 28% did so for the last Challenge. In addition, when we focus on the last submission days, we can see that 30% of the students sent their first submission in the last two hours for Challenges 4 and 5. As the topics are getting more and more complex over time and the students more and more used to handle Challenges, we would have expected the opposite behavior. Here, it seems that students are increasingly rushing to solve the Challenges. A possible root cause is that students take more time to build their first submission since the Challenges get harder. If so, that means that many students spend several days in designing their solution before collecting any feedback. Some other explanation is that students get more tired (especially for the last Challenge occurring during the last week of the semester) and feel less motivated, leading them to procrastinate [1].

Fig. 7.

Cumulative students distribution over the elapsed hours from the moment the Challenge was released (Wednesday, 17:00) up to the deadline (Friday, 19:00). Results concern the first submission and are aggregated over the six years of interest.

Besides this, Fig. 7 also shows that, for each Challenge, for each day transition, some plateaus occur. Those reflect the night as well as the morning of the next day. From Wednesday to Thursday, the plateau even includes the afternoon. Regarding the night submissions, to complement Fig. 7, we computed the students’ proportion that submitted their work between 00:00 and 06:00. On average, over the different years and all the six Challenges, 5% of students handled at least one Challenge during the night. Next, the limited number of submissions in the mornings and Thursday’s afternoon can be explained by the fact that students are supposed to attend classes. Still, on average 24% of the submissions occurred during that period. That means that either they chose to not attend a course, either they “split their attention” between the Challenge and the course, likely leading to a lower-quality refreshed solution.

From those observations, we can say that students do not optimize their work conditions to handle the Challenges, which reduces their opportunity to properly understand the feedback and take benefit from their 3 submissions per Challenge.

Fig. 8.

Students’ opinion about how much Café fosters general regular work throughout the challenges.

Putting this data in perspective with the students’ perception, Fig. 8 shows that only 31% of students use the Challenges as springboards to boost their general work through time. Besides this, 58% of students felt the need to review the course, either because Café explicitly directed them to a specific topic or to understand better the feedback. However, very few students spontaneously extended their practice of the course by autonomously solving exercises.

In summary, from Figs. 7 and 8, we can say that Café does not really influence students’ work outside the scope of the Challenges, even when they see a Challenge coming (otherwise, we would likely observe earlier submissions since students would feel more at ease to solve them at the time Challenges are published). It suggests that the motivation of many students is mainly fed by close reward (referring to the grades) and feedback. Outside that context, many students do not spontaneously work on their own.

4.4 How is Students’ Self-regulation of Time Management Related to Their Performance?

Finally, now that we understand better how students manage their work time across the Challenges, we aim at evaluating the relation between their time management and their performance. Two perspectives are considered: a local one and a global one.

First, we check the “local students’ performance”, i.e., the performance that results from the current activity students are handling over the days of interest (the Challenges in that case). More precisely, for each Challenge, and for each elapsed hour, we grouped together all students who submitted at that time, and we computed the average of their final grade to the current Challenge. The resulting graph is given through Fig. 9, where the grades (ranging in [0; 20]) are depicted through a color. The mapping between the colors and the grades is given through the color bar on the right of the figure. That bar is centered in 15 in order to better highlight the difference between the grades related to the last day and the grades linked to the previous days.

Fig. 9.

Average Challenge grades (color bar centered at 15) at each submission time (X-Axis) for each Challenge (Y-Axis). Results are aggregated over the six years of interest. (Color figure online)

From Fig. 9, we can notice that the later the student submitted their work, the poorer the final solution. Those results enforce what was inferred before: working in last minute does not allow to properly think and integrate the feedback that is provided, leading students to miss the opportunity to really learn from the Challenges. Joining those results to the ones from Fig. 7, we can see that, despite low grades obtained in the previous Challenges, students generally keep waiting the last eight hours to submit their work and refresh it, without stepping back. That static behavior can be due to many factors like a motivation drop, a lack of self-regulatory skills, a recurrent underestimation of the expectations, or some “hidden” collaboration where a large group of students wait for some others to submit their work, so that they can take advantage from others’ feedback, without losing their own attempts. Moreover, like we saw in Fig. 7, some students submitted their work during the night, which led to a final poor solution in 47% of the time. However, it is important to notice that those poor solutions were not necessarily the ones submitted during the night since every student gets up to three trials and the time recorded here refers to the first submission.

Besides this, a more global view is given by modeling students’ time management and correlate it to their final skills level, reflected through the exam grades. To model students’ time management, we compute the average time of their first submission over the six Challenges, and we derive the elapsed average time from the Challenges publication. The idea behind that modeling process is that students’ time management across the six Challenges reflects their time management in general. Similarly, Hooshyar et al. [6] also used student’ assignment submission behavior to model students’ procrastination trend. The resulting relationship is illustrated in Fig. 10.

Fig. 10.

Correlation between “last minute” trade and learning rate (aggregation over the six years of interest). The histograms (top and right of the graph) gives the number of students per X or Y value. Grades range within [0; 20], 10 being the success/failure threshold (illustrated by the horizontal dashed line).

In accordance with previous results, that last figure shows that the later the students submitted the first version of their solution, the lower the exam grades they obtained, eventually. That observation can be extended by stating that the more time you take to process the topics, the more you will learn in long-term. That claim appears reliable seeing the very low p-value behind that analysis (see Fig. 10). That also fosters again the necessity to engage students along the semester through regular activities in order to reduce procrastination behavior and naturally integrate learning in the students’ day-to-day lifestyle.

5 Perspective and Conclusion

To wrap up, the whole results exposed in previous section reveal that Café can make the difference on students’ success, due to its purpose to supervise regular work. Automatic grading and feedback to assess students can increase their motivation to practice continuously (which is required to learn to program) [12]. However, currently in our course, many students soon or later fall behind anyway (which is reflected through the participation drop in the remote activity as well as through the low performance students demonstrate in the Pca and the exam). One reason is that many of them do not use the tool in an optimal way (working very close to the deadline, waiting for the challenge period to review the course, likely not digesting properly the feedback) leading many students to lose the track of the course, eventually.

From that observation, our goal now is to bring Café to an upper version such that it offers a more suitable experience to students. More specifically, we can define three global features of success a blended learning environment should have. First, the system needs to be attractive to drive participation. Next, its content should b e accessible for any student. That can be developed by making more flexible and achievable statements as well as by providing appropriate feedbacks. Finally, Café should clearly embed learning over time to make students fully aware of their learning progress with respect to the final course objectives and help them organize their time around the remote activities. Flexibility and time organization match with what Alavarez et al. [2] defined as requirements a computer system should meet in order to adequately support students in a Blended-Learning context.

Fig. 11.

Expansion of Café from its current version (Café 1.0) to the upper one (Café 2.0).

Figure 11 gives an overview on Café ’s upcoming evolution from its current version described before, through Fig. 2.

In particular, Fig. 11 reflects how Café 2.0. will expand and encapsulate the activities from end-to-end (instead of just handling submissions). The goal behind that is to build a more user-friendly system that better onboards the students, overcoming the recurrent participation drop. Rather than just punctually supporting Challenges submissions, we aim to turn Café into a platform that will be more integrated into the course [5].

Regarding the activities, besides the Challenges, students should get other opportunities to train with that automatic supervision. A first slight upgrade would be to give back access to Challenges after the deadline, without altering the grades. This way, students could take more time to assimilate and apply the feedback to refresh their solution. Further than this, we could also include into Café another type of activity [8] that would be fully facultative and always available from the time the topic of interest has been taught. In this way, students would be able to train under less pressure, at their own pace. More generally, they would also get more diversity in their learning activities, which is likely to facilitate the development of task value, especially their interest [14].

Next, regarding the automatic supervision, Café should offer a closer guidance in order to “keep students tuned” and naturally direct them to correctly and rigorously solve statements. Guidance can be set at different levels. First, we should define intra-statement guidance including more step-by-step resolution as well as a better error catcher and resulting feedback. Decomposing students’ resolution into pieces will make it more accessible. Furthermore, by zooming more on each resolution step, on one hand, students can acquire a better resolution structure, and, on the other hand, we can easily put in place specific short theoretical reminders, hints, and local feedbacks to guide the student towards the solution. Of course, the idea is to progressively relax that resolution framework in order to finally see the students handling properly problems on their own. Secondly, we should shape their time management and develop good habits in their working lifestyle, like already implemented by Su [16]. Typically, we could lock the platform during the night (as well as during course time). Furthermore, in response to large number of last-minute submissions (leading to poorer solutions, as depicted through Fig. 10), we should define some inter-statement guidance, so that students get a clear view on which tasks they achieved and what they should do next. That third feature appears essential to balance the self-paced learning environment supported by Café, where students are susceptible to procrastinate [7, 15]. As a first upgrade, we could create a dashboard (called the Progress Tracker on Fig. 11) through which they could visualize:

• A progress bar comparing topics that have been taught in classroom activities and their current activity on the different topics. In this way, students could better realize where they still need to put efforts.

• More specifically, the remote activities that are open and need to be performed. Typically, for the Challenges, the students would see how much time remains to achieve it.

In further versions, we could even recommend some specific statements based on each students’ level. However, that functionality involves many prerequisites, including the collection of more refined data (stored as Learning Analytics, as shown in Fig. 11). Data provides more transparency about individual students’ learning behavior and resulting performance. At a higher level, through this paper, it already gave us the direction to take to empower Café in order to keep more students in line with the course. Although, that step just sets the pace for further enhancements. By catching closer and closer students’ learning behavior, Café will be able to regulate better and better every student’s learning, forging a more and more optimal blended learning environment.