Study and Analysis on the Student Response System Adoption: Experimentation in a Programming Course

Abstract

Understanding the impact of different types of questions on learning is still an open problem in the literature. Many tools allow teachers to ask different kinds of questions, such as closed (multiple-choice, true/false, associations, etc.) and open (tag clouds, short notes). Our work aims to compare the effect of open and closed questions on student learning. To analyse the differences in learning results between the two types of question approaches, we carried an experiment in a Programming course (Programming 1 - Italian academic course in a Bachelor of Computer Science degree), dividing the students into two groups. During the first half of the course, we asked multiple-choice questions to the students in the first group, whereas we used an open-ended approach for the students of the second one. In the second part of the course, we inverted the modalities. For this work, carried out in the 2018/19 academic year, we used Mentimeter as Student Response System (SRS). We believe that identifying the right balance between explanation and verification and understanding the effects of the different evaluation types represent a significant contribution to the debate on formative evaluation in complex contexts. This contribution could be helpful in university classrooms where the high number of students does not allow easy interaction. Thanks to the data analysis and relative charts presented in this work, the authors share the learned lesson and offer several discussion points raising new questions that have to be addressed yet.

1 Introduction

This paper aims to compare the effect of open and closed questions on student learning.

To analyze the differences in learning results between the two types of question approaches, we carried an experiment in a computer science course (“Programming course”) held at the University of Cagliari. Thanks to the data analysis and relative charts presented in this work, the authors share the learned lesson and offer several discussion points raising new questions that have to be addressed yet.

In detail, in Sect. 2, readers will find the theoretical framework where the scientific question finds its foundation. Previous and related works that inspired the experimentation are listed and highlighted. In Sect. 3, the author discusses the method and the learning environment, giving more details on the tools, the different kinds of questions, and the course where the experiment took place. In Sect. 4, the measurement methodology is presented, while in Sect. 5, the data analysis is deeply discussed. Finally, in Sect. 6, the lesson learned and future work are given to the reader.

2 Literature Review and Theoretical Framework

Higher education classrooms are still dominated by teacher talk. However, research indicates that the most important task for teachers is to listen [1], understanding which prior knowledge students are using and the nature and extent of the gap between where they are and where they need to be [2]. Student Response Systems (SRS), now in the form of mobile applications, could represent an exciting opportunity to increase student involvement and cognitive activation [3,4,5]. SRS improves teachers’ conduct of formative assessments, identifies misconceptions, and fosters students to respond without peer pressure, exceptionally when their anonymity is preserved [6].

There has been a considerable amount of research work on the use of SRSs. These tools underline the importance of questions [7], teacher feedback management, and formative evaluation [1]. Mayer and colleagues [8] conducted a series of studies on the effect of immediate response to feedback and promoting active participation, detecting errors and misunderstandings, supporting knowledge organization of the selected material, and facilitating memorization and integration of the material with prior knowledge. Their work shows positive effects from asking students to answer questions during the lecture (ES = 0.40), probably due to students paying more attention to be able to answer questions as well as being constantly stimulated to be active.

Mobile-based SRSs are an evolution of clickers. Clickers only allowed closed answers, such as true-false or multiple choice. Today, applications such as Mentimeter allow teachers to also collect open answers in various formats (short texts and word lists). It is, therefore, a problem of how to choose between the different types of questions.

What are the differences in terms of involvement and learning? It is already known that answering closed-ended questions through a device improves student participation and performance. Conversely, few studies have been done on the use of SRS with open questions.

A survey design study [9] was conducted to investigate students’ engagement when using open-ended SRS during class time; findings of this study suggest that the use of open-ended questions positively impacts students’ engagement, motivation, and learning in an undergraduate-level course. This study also found interesting shifts in students’ perceptions about their role as learners and their instructor’s role in the classroom, but many questions still remain open.

We suppose that it comes from the fact that students are more focused on concepts and definitions to deal with pending verification. We believe that open-ended questions will increase the ability to rework the concepts [8], but, at the same time, it could increase the risk of cognitive load [10].

Several literature studies investigating the benefits of the SRS indicated that they improve students’ academic performance, at least in some limited areas [11, 12]. Students’ performance can easily be measured by learning achievement in specific topics, which is more related to mastery-based learning goals [13]. Moreover, students’ academic performance may be affected positively by their active engagement during the lecture [14].

Engagement consists of a strong connection between different school players: students, teachers, families, and schools. Student engagement is a crucial element of creating a positive learning environment, and it can take benefits from active learning. Felder and Brent [15] describe active learning as “anything course-related that all students in a classroom are called upon to do other than simply watching, listening, and taking notes”. Active learning can improve student engagement and can positively impact student learning.

Active learning focuses on how students learn, not just on what they learn. As discussed in [16], active learning is “generally defined as any instructional method that engages students in the learning process. In short, active learning requires students to do meaningful learning activities and think about what they are doing”. It is possible to put the active learning principles into action with the help of technology and, in particular, through student response systems (SRS).

The current consensus is that tools like SRS positively impact students’ cognition, attitudes, and behaviors in learning activities [17]. These aspects lead to better learning achievement, higher motivation to learn, higher self-efficacy, and more participation.

3 Method and Learning Environment

As discussed in Sect. 2, student involvement plays the main role in learning activities. To improve this, teachers can use several tools and techniques. One of the tools at their disposal is a student response system.

These tools are now available on web platforms, smartphones, and dedicated devices, thanks to software development. Their adoption could represent an exciting opportunity to increase student involvement and cognitive activation.

3.1 Mentimeter

Among many alternatives, we adopted for the entire experiment the presentation tool Mentimeter [18]. It includes this feature and grants large portability and anonymity. This proposed work started from three specific questions: i) “how can we improve the involvement of students during classes?”, ii) “Can we facilitate their learning experience?” and iii) “are questions at the end of the lesson enough?”.

We decided to set up an environment adopting a student response system as a support tool. Since understanding the impact of different questions on learning is still an open problem in the literature, we decided to use both open and closed questions.

3.2 Open and Closed Questions

Open questions allow participants to answer in their own words; closed questions require students to select a response from a specific set of options. One practical advantage of closed questions is that the answers are automatically stored as numeric or alphabetic values.

On the other side, they can be problematic from a measurement perspective if the predetermined categories are not exhaustive or influence students. On the contrary, the open ones do need to be coded into categories after the test. Our work aims to compare the effect of open and closed questions on student learning.

3.3 The Course

The experiment was carried out during the “Programming Language 1” course at the University of Cagliari within the Computer Science degree. Teachers have never used such tools during lectures.

Two modules make up the course: the first focuses on the theory of programming languages, while the second part comprises 24 lectures, 3 h each, focusing on laboratory activities. Fourteen of these twenty-four lectures concern the practical part of the laboratory (mainly coding activities).

These lectures are where the experiment took place, and it was a two-month length experiment. Course objectives are related to 1) reading C programs of small complexity and understanding why they are based on the code itself and the associated comments, 2) designing small C applications starting from the guidelines or building over existing code, 3) developing independently, for the design and implementation choices, the final project from the specifications provided, and 4) present the project at the end of the course by discussing the design and implementation choices.

The content of the course is relatively standard for a programming course. We discussed every aspect of the language, from the algorithm to the design of the program. Step by step, from the declaration of variables to the use of pointers, students learn to write and execute medium-complexity projects in the C language.

Every year, 150 students are enrolled in the Computer Sciences degree, a course of the first semester of the first year. For the experiment, and due to the available places in the laboratory, we decided to split the list of students into two groups using their student id: even ids have formed group A, and the others formed the B group. In the end, we made two groups with a total of one hundred and forty-five students. They had different backgrounds, only 10% of them were females, and the average age was 20. The two groups, having been formed by random extraction, are therefore to be considered equivalent.

Each group had a different teacher. We adopted the same program, same order of contents, same pace: two lessons every week. From the 1st lesson to the 7th one, the first group answered open questions; in the meantime, group B answered closed questions. We switched the modality after these seven lessons, so group A answered closed questions while group B opened ones. Changing the answer modality but not teachers allowed us to ensure that the teachers’ skills and behaviors were not influent during the entire experiment. This switch also allowed us to investigate the effect of changing the modality from one day to the other and see students’ reactions (some effects are discussed in Sect. 5.1).

It’s important to say that the same questions were provided; the only difference was the answer modality. For each lesson, we submitted three different questions concerning the explained topics. In total, we proposed and analysed 42 questions.

I.e., during lesson number six, we asked students what the aim of the “return” statement was. Group A answered without any proposed alternative, while students in group B selected one of the given alternatives: A) ends the execution of the function; B) it represents the type of the return value; C) Ends the execution of a function, and returns a value/the control to the calling function; D) Ends the executions of a void procedure and return the value to the main function;

To have a standard approach, we agreed on the following points before starting the experiment: i) have the Q/A section only at the end of the lesson, reserving 15 min: 1 min to give the answer and four minutes to discuss it; ii) propose three questions per lesson on the most complex topics; iii) keep code-related questions as simple as possible due to the constraint of 1 min; iv) let students answer anonymously.

4 Measurement and Methodology

One of the first critical points to prepare a valuable dataset for data analysis was to adopt an excellent way to compare such different modalities. Closed questions can be easily categorised, but open questions cannot. Given answers to open questions need to be coded into categories. For each collection of open question answers, the author labeled each answer as correct or wrong (Fig. 1).

Fig. 1.

An example of labelling of the answers given to the question: “Which types are allowed in a new declared Struct?”.

This part was one of the most time-consuming of the experiment, but this approach allowed authors to determine the number of correct answers regarding the students’ responses. Having such data allowed authors to analyse students’ engagement, the correctness of answers, solutions trends correlated with the grades given during their courses’ examinations.

5 Data Analysis

At the end of the experiment and data collection, a pre-processing phase allows authors to create a unique spreadsheet with all the raw data. Thanks to this spreadsheet, it has been possible to create several charts.

5.1 Students’ Participation

The first one Fig. 2 is related to the percentage of participants who answered the proposed questions—data represents the number of correct and wrong answers given by attendees. Answering was not mandatory; we left the decision to students. No pressure, only voluntary answers. In this chart, in light purple, you’ll see the second group, B. The light green is the first group.

After seven lessons, as we said before, we switched the modality. As you can see, the main difference is that group B, which started with closed questions and then switched to open questions, shows a negative trend. It started with 100%, which means that all students present in the laboratory answered the first question of the first lesson, then it reached its bottom during lesson number twelve. That lesson was quite complex, and only 25% of students answered the first question—the same behavior in the second and third. After that lesson, the trend continued pretty similarly to the other group.

On the contrary, group A maintains almost the same trend during the entire experiment.

Fig. 2.

Percentage of participant, number of answers in respect of attendees. (Color figure online)

This trend could be explained due to cognitive effort. Given that open questions require a higher effort from the students, they can be perceived as more demanding than closed ones. Passing from accessible to difficult questions will probably cause a sense of frustration in students; it leads them to not answer at all.

No changes seem to affect the first group that passed from difficult to easy ones. In general, we noted that closed questions guarantee higher participation of students. We can also state that open questions show a higher percentage of correct answers.

5.2 Correct Answers Comparisons

Comparing the correctness of the answers Fig. 3, we can see that there are no fundamental differences between the total score of the first and second groups. But if we look at the comparison of the two modalities, we’ll see that there is a significant difference between open and closed questions in both groups.

It’s worth noting that group B moved from 70% of correct answers to 40% after the switching point, dropping 30%. The drop is heavier than the one shown in group A, which is lower than 10%.

Fig. 3.

Comparison between general correct answers ratio and different modalities (OQ, CQ).

This trend could be correlated with the previous chart: with open questions, we have a low percentage of answers and, at the same time, a lower percentage of correct answers.

5.3 OQ and CQ Answers’ Correctness Trends

We then decided to analyse the trend of the two modalities Fig. 4. In orange, open questions; in blue the closed ones. As you can see, the majority of questions shows better result in closed questions. Only four times, out of 42 questions, the chart shows better results for open questions. In 15 of them, we see a significant difference between the two modalities.

Fig. 4.

Comparison between closed and open questions.

The result is more evident in next chart Fig. 5. We received 65% of correct answers in closed ones, while around 50% in open questions.

Fig. 5.

Comparison between closed and open questions (total average).

5.4 OQ and CQ Answers’ Correctness Trends

We then decided to compare the correctness of answers with the grades obtained by the students during the course Fig. 6.

It is essential to say that mid-term evaluation is a coding exercise, not a written test nor a quiz.

As you can see, we have group A on the left, on the right, group B. During the first part of the course, students in group A, which started with open questions, got higher grades regarding the correct answers given during the classes. There is a fundamental gap between these values.

This gap is almost invisible for the second section of the chart. Grades and correct answers percentage are in line.

This phenomenon also happens in group B; the higher gap occurred during the second part of the course, where they answered open questions.

Due to this analysis, we can say that both open or closed could be a tool to anticipate the learning evaluation phase, but they do not appear to differ in boosting the results.

Fig. 6.

Comparison between grades and correct answers.

We can also see Fig. 7 that grades and correct answers are pretty similar during the entire experiment for both groups. Data shows no significant differences. It means that their knowledge, at least the average, is the same even if they adopted different modalities regarding the two parts of the program.

Fig. 7.

Comparison between grades and correct answers (total average).

5.5 OQ and CQ Answers’ Correctness Trends

We also compared the results obtained using the student response system and the one we got the year before where we didn’t use them Fig. 8. In grey, you have the previous academic year; in yellow, the one of the experiments.

It’s visible that results improved massively during the first part, Q1, and Q2 exam, while in the second part, Q3 and Q4 exam, the result seems better without adopting such kind of systems.

Fig. 8.

Comparison between grades of two different academic years.

It is clear that during the first parts of the course, we discussed more accessible topics, more based on the fundamentals of programming languages.

For these topics, Mentimeter gives us an excellent hand for keeping students motivated and on track.

Thanks to its anonymity, students tend to actively participate and give honest answers which are always helpful for teachers to adjust in real time how they are approaching and providing information.

But the tool seems useless when the subject starts to be complex and requires more aspects related to problem-solving and coding by using complex approaches and skills.

It’s worth noting that we adopted the same program, content, examination approach, and difficulty of exams.

6 Conclusion and Future Work

After this experiment, we also can give several considerations. In general, if the answers are shown in real-time, the flock effect should affect students, and if the answers to open questions are shown in real-time, students tend to write comments or responses out of topic. Concerning the complexity of the question, if the question is complex, the number of replies drops more heavily in open questions. Student response systems help students avoid frustration at the beginning and engage them by giving them a chance to voice their knowledge and actively participate in the lecture.

Several discussions were raised during the Q/A sessions, and it was significant for the teacher to get real-time feedback. SRS increase knowledge retention, at least during the less complex part of the program. Last but not least, it requires more effort on the teacher’s side, so teachers have to be prepared for it.

This work could be used as a touchstone for further research and, at the same time, as a starting point to deepen the study of understanding the adoption of open and closed questions. It could also be used to facilitate learning and knowledge retention in the programming language teachings and more. A follow-up of this study was planned before the pandemic situation related to C19, but data have been discarded due to extremely different and intermittent setup (online and blended). A new experiment will take place during the next academic year.