Keywords

1 Introduction

There is a growing body of evidence which demonstrates that active learning pedagogies improve learning outcomes in a wide variety of courses, including introductory programming courses [2, 3, 10]. It is very natural to allow students to use their laptops in class during active learning sessions of an introductory computer science course. Many faculty, however, are wary of requiring computer use during class sessions since they feel students might become distracted.

In recent years we have seen many new developments in the way teaching and learning are accomplished inside and outside of the classroom. The last decade has seen research, development and adoption of new pedagogies, classroom technology and software applications. One such new pedagogy technique has been the ‘inverted’ or ‘flipped’ classroom in which static content is covered outside of class (through readings or videos) and class time is devoted to more interactive and engaging activities. Even though most approaches have leveraged the ubiquity of technology, flipping a classroom does not necessarily require the use of computers or other networked technology.

In this work we present our case study of partly-flipping a large CS1 class. The course was an Introduction to Programming in Java and C in which we used a partly-flipped pedagogy that combines both in-class lectures and in-class programming challenges often using a Think/Pair/Share technique [7]. Since the course was taught in two sections (of about 150 students each), we were able to design an experiment to evaluate the effect of two approaches to partly-flipping the classroom. The first approach is to require all students to bring a laptop or tablet to class and use their computers for various interactions, to answer questions and to solve coding challenges. The second approach is to ban computers from the classroom and to require students to solve problems with pen and paper and to be prepared to present and discuss their solution to the class if called upon.

In the computer-mediated sessions, students used two web-based applications, TeachBack [6] and Spinoza [1], to interact with the instructor and the other students while solving programming. In the non-computer sessions, we endeavored to replicate the same pedagogy using pen, paper, blackboards and the instructor’s computer projected on a screen. Both sessions covered exactly the same material and used exactly the same pedagogy. Students received nearly identical lectures and were given the same programming challenges. The only difference is that in one section students were allowed to use their computers to solve the programming problems, while in the other section they had to use pen and paper only.

In the following sections we present the experimental design that was used to compare the computer and non-computer approaches to the pedagogy. We then proceed to describe the pedagogy used and we compare the way it was implemented using the computer-mediated and pen-and-paper based approaches. Finally, we present the results of the experiment and discuss its implications for computer use in the partly flipped introductory programming classroom.

2 The Experiment

Introduction to Programming in Java and C is the first course in the Computer Science major in our department. Students who performed well in an equivalent CS1 course in high school may skip the course, but all other potential majors are required to take it. It was taught in two sections (self-selected by the students). One section had 136 students and the other had 148. Both sections had the same instructor, exams, homeworks, teaching assistants, and daily lesson plans. For both sections, we provided screen recordings of each class that students could review at their leisure.

The course was divided into 4 units, each lasting about 3 weeks. Each unit culminated in a 90-min exam that provided a summative assessment of student mastery of the material for that unit. In the first two units students were required to bring their computers to class and to interact with the instructor using TeachBack and Spinoza. Ten percent of their final grade was based on the number of TeachBack formative assessment questions they answered (whether the answers were correct or not). During Units 1 and 2, students were required to bring computers to class and use them to interact with the instructor and their peers. During Unit 3, computers were banned from Sect. 1 while still being required in Sect. 2. During Unit 4, the protocol was reversed: computers were required in Sect. 1 and banned in Sect. 2. This provided us with two units of control in which both sections used computers, and two experimental units where one section required computer use and the other banned its use.

3 The Active-Learning Pedagogy

Before each week of classes students were assigned topics or subtopics to read and as a weekly homework they were asked to submit a short reflection on what they learned and to discuss any confusing ideas in the reading. Each class had lectures intermixed with class-wide interactive activities. The lectures involved PowerPoint slides, notes from the class website, live coding demonstrations by the instructor, and visits to various websites. The interactive activities included short answer questions as well as programming challenges.

In this section we discuss the main pedagogical techniques used in the two versions of the class and along the way we introduce the TeachBack and Spinoza tools. TeachBack [6] is a web-application with three main features: a supervised back-channel forum (called the Forum) where students can ask and answer questions with each other and with TAs who are always present during classes, a pie chart and timeline plot (called the Feedback) where students can indicate if they are confused, engaged, or bored and can include a 50 character explanation of their affect and cognition (i.e. emotional and comprehension) states, and a clicker-type application (called the iResponder) which allows the instructor and TAs to collect and grade student answers to formative assessment questions during the class. Spinoza [1] is a web-based Java IDE that allows students to solve simple programming problems online and provides the instructor with a real-time view of the progress of the class with similar solutions grouped together.

3.1 PowerPoint Lecture Activity

Although the students were required to read the text before class, we often began a class with a PowerPoint overview of the main ideas presented in the readings. In the computer-based version of the class, students could view the PowerPoint slides on their computers and ask questions of the teaching assistants using the TeachBack Forum. In the pen-and-paper version they could print out the slides on paper before class and ask questions by raising their hands, which interrupted the class flow.

In the computer-mediated version of the class, we used a web-based Integrated Development Environment.

3.2 Live-Coding Activity

Another lecture-style activity is when the instructor solves or demonstrates a programming problem using a Java IDE and the class watches (or in the computer-mediated version, follows along). This can be made interactive by asking students to provide suggestions for how to solve the problem. In the computer-mediated version when students are following along with the coding using Spinoza and they encounter syntax errors they can interact with the TAs using the TeachBack Forum without interrupting the class.

3.3 Answering Student Questions During Class

In both versions of the class, students were encouraged to ask questions if they were confused. In the pen-and-paper version, students would raise their hands and engage with the instructor while the class paused. In the computer-mediated version, students used the Forum feature of TeachBack to ask questions online, and have their questions answered by TAs assigned to the course, or sometimes by other students who were monitoring the forum. The instructor would briefly review the forum with the class at the end of most activities.

3.4 Posing Questions for Students to Discuss and Answer

After a lecture activity, we would usually pose a series of questions and ask the students to think for a minute about a solution, then to talk with their neighbors about their solution, and finally to share their solutions with the class. Typical examples would be predicting the result of evaluating a snippet of code, or finding a bug in a piece of code shown on the projector. In the computer-based version, we used the iResponder feature of TeachBack. Figure 1 shows a typical activity in which the instructor projected a method on the screen and asked students to predict the return value for various calls. iResponder allows the instructor and TAs to not only see the solutions (grouped) but to grade them and assign points and comments. Once a sufficiently large number of students have submitted an answer, the instructor reviews the most common solutions and leads a short class discussion on the different approaches and the different kinds of errors. In the pen-and-paper version, it is difficult to determine how many students have completed the activity and it is hard to tell what the most common solutions and errors were. Students were motivated to solve problems in the pen-and-paper class by randomly selecting students to describe their solutions (possibly on the board or typing into the instructor’s computer).

Fig. 1.
figure 1

A typical iResponder screen in TeachBack.

Full size image

3.5 Programming Problems

In this activity, students are given a programming problem and asked to think about how they would solve it and then work with their neighbors to come up with a solution. For example, students could be asked to write a method with three integer parameters that returns true if the parameters all have different values.

In the computer-mediated version of the class, we used a web-based Integrated Development Environment (IDE) called Spinoza that allows instructors to quickly create a programming problem. Figure 2 shows the student view of a Spinoza programming problem which provides a description of the problem on the left, some initial scaffolding code in the center, a “Run” button below, space for the output on the right, and the results of an instructor supplied set of unit tests at the bottom. Students can then write, run, and debug the problem using the web-based IDE. Spinoza has an instructor’s view which shows the number of students that have hit the “Run” button and it groups the programs together based on a similarity function (ignoring white space, variables names, etc.). The instructor can see in real-time the most popular proposed solutions to the problem and can view and debug those solutions in front of the class. The debugging process itself can be formulated as a Think/Pair/Share model [7], where students try to find and discuss the bugs (both syntactic and logical) in small groups before sharing with the class.

Fig. 2.
figure 2

The student view of a Spinoza problem.

Full size image
Fig. 3.
figure 3

The Feedback tool in TeachBack.

Full size image

In the pen-and-paper version of the class, programming problems are displayed on the screen and students are asked to write their solutions on paper. The instructor then randomly selects students to share their solutions. The disadvantage of this approach is that the instructor doesn’t know what the most common solutions or errors are and the process of sharing a solution with the class is more time consuming.

3.6 Feedback

After new material has been introduced we often ask the students for feedback, typically at the end of an activity or class. We ask whether they are confused, bored, or engaged by the material and also ask for a short comment. In the computer-mediated version, this is done using the TeachBack Feedback feature, which displays a pie chart showing the three responses (Fig. 3). Hovering over one of the pie slices reveals a list of the comments students provided. We often find 20 %–50 % of students report feeling confused when a class introduces new material (e.g. arrays or the for-each loop). This provides an excellent opportunity to reassure them that it is natural to feel confused when learning new material. The comments also show what confused them or expand on their affect. At this point the instructor also clarifies the various confusion issues. Since it is so easy to get and analyze feedback from students using TeachBack, we often get feedback after each activity in a single class. TeachBack also provides an instructor/TA view of the daily progress of individual students using performance and participation statistics at an activity, lecture and course levels.

In the pen-and-paper version, we ask students to put this information on a small card or piece of paper, which is then reviewed by the instructor after the class. One disadvantage of this approach is that we can’t report the results until the following day and it can take 30 min to an hour to read through a few hundred separate comments.

4 Data Collection

After each unit, students were asked to complete a survey where they self-assessed their level of understanding of the material in that unit as well as their level of enjoyment of the material in that unit. In units 3 and 4 they were also asked to rate each of the different styles of pedagogy employed in terms of its effectiveness for their own learning.

We kept track of the number of students from each section that visited TAs during each of the units and asked students to estimate how many hours they spent working on the course outside of class. We also kept track of each student’s participation in various components of TeachBack during each class, each unit, and the semester. Finally, grades on the four unit quizzes as well as course grades were used to measure mastery of the material by unit and over the entire course.

5 Results

We found four main results from our analysis of the data which we discuss below:

5.1 The Use of Teachback/Spinoza in Class Does Not Harm Learning Outcomes

In Units 1 and 2, both sections used computers in class. In Unit 3, computers were banned in Sect. 1 and required in Sect. 2. In Unit 4, the reverse policy held, computers were required in Sect. 1 and banned in Sect. 2. We found that there were no statistical differences between the two sections during those units in terms of quiz scores, student satisfaction, student self-assessment of understanding, or student use of teaching assistants. From the surveys at the end of each unit, students self-reported their levels of learning and satisfaction in the range [1–5]. As seen in Figs. 4 and 5, the averages on each section do not indicate any significant influence from the changes of pedagogies in units 3 and 4.

Fig. 4.
figure 4

Average perceived enjoyment.

Full size image
Fig. 5.
figure 5

Average perceived understanding.

Full size image

Section 1 generally indicated a higher level of enjoyment, understanding, and mastery than Sect. 2, for all units, but that increased level of understanding was not statistically significant.

For example, in Fig. 5, the difference between the average understanding in unit 3 between Sects. 1 and 2 was 0.17 but the p-value for the two-tailed unpaired T-test for those means was 0.20 which is not significant. Likewise, in Fig. 4 the difference of average enjoyment for unit 4 between Sects. 1 and 2 was 0.23 but the p-value was 0.12, again indicating no significant difference. None of the apparent differences in Sects. 1 and 2 shown in these three figures was significant at the 0.10 level.

Fig. 6.
figure 6

Average end-of-unit quiz grades.

Full size image

If use of computers was especially distracting, we would expect to see Sect. 1 outperform Sect. 2 in Unit 3, and the opposite occur in Unit 4. Likewise, if the use of computers was highly beneficial we would expect Sect. 2 to outperform Sect. 1 in Unit 3, and the reverse in Unit 4. No such effects were found (Fig. 6).

5.2 Most Students Prefer Using Computers in Class

When asked about the two different styles of active learning - writing programs with your neighbors on paper versus writing programs on your computer while talking with your neighbors, the use of computers was thought to be more effective and the results are statistically significant. Students used a five point scale to rank effectiveness of learning from 1 = not effective to 5 = very effective. Solving programming problems with friends using pen-and-paper was ranked at 2.96/5 and solving programs using Spinoza with friends at 3.65/5 with a difference of 0.69. This is significant at the 0.001 level using a two-tailed paired T-test. The 95 % confidence interval of the difference is 0.5 to 0.88.

Below are some typical comments from students after unit 4. Here is a Sect. 1 student, happy to be able to use his computer again in class:

  • “I really enjoyed when we got to live code in class. It was helpful to either follow along with what [the professor] was typing or work on building up the program with the people around us. It allowed me to see what thought process has to go into building up a program.”

And here are comments from students in Sect. 2 explaining why they were disappointed about not being able to use computers in class:

  • “The lack of computers makes following along a lot less interesting and understanding class material becomes much more difficult.”

  • “Taking notes on paper and not being able to practice coding in class slowed down acquisition of the material greatly. It [took] much longer for this unit than others to master the material. I also disliked being asked to work in teams or to talk to people in class, but that’s because I’m shy …”

  • “We can’t use computer[s] to do real-time programing in class. To make it up, I have to go back home and watch the class recordings to brush my memory on what programing topics we went through in class that day. It is really time consuming.”

5.3 Some Students Were Distracted by Computers in Class

A close examination of the student comments about each unit demonstrated that there was a group of students who did not feel they learned as well with computers as they did without. Indeed, a few students would attend the lectures from the other section when the pedagogy was switched because they felt they could not learn well when required to interact with a computer in class. These were mostly students who reported being easily distracted in general. Below are some comments from students indicating what they liked about unit 4 when computers were not allowed.

  • “Not using a computer, it lead me to better concentrate.”

  • “Not being allowed to use our computers helped for concentration and focus.”

Most students, however, didn’t report being distracted by the use of computers in the class, contrary to the worries of many instructors. This observation is largely due to the nature of the pedagogy. The division of the class time into short interactive activities allowed students to always be engaged with the material, their peers or the instructor. There was no time for students to get side-tracked into distraction with non-class related endeavors.

5.4 Students Generally Approved of the Active Learning Approach

In general, students appreciated the pedagogy used in the class, whether or not we were using computers. Here are some illustrative comments.

  • “The class was very lenient towards our learning and it’s a great feeling to know that the teaching staff is very forgiving for us ‘newbies’. Learning is the number one goal.”

  • “I was forced to try to learn the material to the best of my ability beforehand to be as prepared as possible whether or not I was using my computer or notebook.”

6 Discussion

One of the main goals of the experimental design was to be able to test the effects of the use/non-use of computers in a CS1 class in a way that removed as many possible confounding variables as possible. Therefore, both sections had the same instructor, the same course plans for every day, and each section had an exposure to a unit where computers were banned while computers were required in the other section.

We observed that the students in Sect. 1 did better in all four units than the students in Sect. 2 in terms of quiz grades, self-estimation of understanding, and level of enjoyment of the course. Section 1 was an earlier class (11:00) and Sect. 2 conflicted with some other large classes (e.g. Intro to Economics). Therefore, the two populations of students may have been qualitatively different, e.g. early-risers with a tendency to take Economics courses. If we had taught Sect. 1 without computers and Sect. 2 with computers, then the fact that Sect. 1 had “better” students would have made it more difficult to tease out any potential effect of the treatment (banning versus requiring computers in class).

Our experimental design did allow us to show that in a class in which computer-use was alternately required and banned, the student performance was not different (in a statistically significant way) in the two kinds of units. On the other hand, this experimental design also tended to hide some potential effects that might appear in an experiment where treatments don’t alternate and multiple sections are assigned to computer-banned or computer-required treatments for the entire semester. In fact, we suspect that active-learning CS1 classrooms in which students are required to use TeachBack and Spinoza (or similar computer-based classroom technologies) for the entire semester may indeed have better learning outcomes than active-learning CS1 classroom that ban computer use. In this section, we provide some arguments in support of this claim.

The simplest critique of our experimental design is that although there was no statistically significant effect on the learning outcomes, it could be because the students in the computer-required (or the computer-banned) section may have had to work much harder to achieve the same performance as they did in the other section. We had one student, quoted above, who said they went back and reviewed the screen recordings every day during the computer-banned units so they could follow along with their computer and this took an enormous amount of time. It is hard however to accurately estimate how much time students are spending outside of class without some sort of automatic time-keeping protocol.

We hypothesize that the TeachBack Feedback mechanism in which students report if they are bored, confused, or engaged and then see the results for the entire class may have a beneficial effect on the students by easing their anxiety about being confused. When they see that half of the class is regularly confused when new material is introduced they may feel less fearful and anxious and hence be able to learn more effectively. In our experimental design both sections started out with 2 units of computer-required classes and hence both benefited from this effect, if it was indeed present. Similarly, when students use Spinoza to solve programming problems in class and then are able to see that many other students in the class are making the same mistakes, we suspect that this can also reduce anxiety and improve learning effectiveness.

Another drawback of our experimental design is that it doesn’t allows us to see if mandatory computer-use in class has any effects in future classes, e.g. does it improve retention in future CS courses (or not), do students in a computer-required course perform better (or worse) in the follow on courses. In our experiment both sections had three units in which computer-use was mandatory in class, and one unit in which it was banned. Any benefits either approach has over the other are averaged out over the semester and hence the experiment does not have any different impact on the two classes in years to come. This makes it impossible to study longer-term effects of computer use in the classroom.

Finally, since the protocol required that both courses use exactly the same pedagogy to the extent possible, we were not able to use the highly detailed and nuanced information that we had collected using TeachBack and Spinoza to find students who were at risk of doing poorly and target additional resources to them. In the computer-banned version of the course, the only way to assess student performance with our experimental design is through the homework and quiz grades. The quiz grades do indeed show who is doing poorly, but by then it is too late as quizzes are summative assessments. The homework grades are unreliable as they don’t show how much time students spent on the homework, nor how much help they received from friends or teaching assistants, nor what kinds of mistakes they were making when completing the homework. If a struggling student works hard enough and gets enough help, his or her homework can be nearly as good as the student who is doing well. TeachBack and Spinoza on the other hand capture much more information and can give daily snap-shots of student understanding at a much deeper level of detail.

7 Related Work

Amresh et al. [2] performed a recent study involving flipping an introductory computer science class, where students would watch prepared lecture videos before classes, and have interactive discussions in class. Through summative assessments, this flipped model was found to produce higher average test scores. However, due to many years of traditional classrooms, students found this new approach to be overwhelming at times, especially as the videos and reading became boring and less engaging. In regard to this, Bates and Galloway [3], point out that successful flipped classes require the acceptance and embracing of this new unstructured and contingent lecture approach where the instructor is a coach of learning. In this case study in an introductory physics class; students were assigned pre-class readings and quizzes, and class meetings involved discussions driven by clicker questions. An important factor for success is to have access to or create sufficient clicker questions for good discussions. If students can be motivated to complete the work outside of class, flipped classrooms can enable more and deeper understanding without necessarily covering less content. Since students are more exposed to the materials in pre-class and in-class activities, the flipped pedagogy has the advantages of developing life-long learners, increasing engagement during classes, and increasing interactions among students and the instructors [10].

Systems similar to the Spinoza system used in this study have been developed to facilitate teaching introductory programming classes. JavaBat [8] is a web application that helps students to build coding skills by providing immediate feedback to small problems in which they write code for the bodies of single methods. The system generates several tests (handwritten by the instructor) and shows students the results of those automatic tests. Students can specify a teacher who can then see their work and follow their progress, but the teacher cannot write comments or otherwise communicate with the students through the tool. Another system is Informa [5], a clicker software system for teaching introductory programming with Java. Informa has been used in flipped classrooms as a way to support active learning of programming skills. It supports several different types of questions, including problems requiring students to write Java code, but it does not run the students’ code and it is not web-based, it requires a Java app to be downloaded and installed. It also allows students to download and comment on other students’ solutions. Spinoza allows instructors and TAs to view and comment on student programs, but does not currently allow students to comment on other students’ code.

8 Conclusions

Our findings demonstrate the critical part played by the pedagogy in a classroom. As we pursued the same interactive pedagogy with and without computers, we were able to support two main results. First, the class can be taught using essentially the same thought and learning processes with or without computers. Secondly, contrary to most beliefs, allowing computer-use in class on its own causes distractions and poor learning outcomes for only a small subset of students. Those students could perhaps be given accommodations to not use their computers in class. Finally, this study therefore demonstrates that the use of computers in this class does not affect learning outcomes in any statistically significant way.

The key factor in student learning is the pedagogy itself. The highly interactive pedagogy we used encouraged students to maintain high levels of interaction, engagement and motivation with the material whether they used computers or not. We know from previous studies that active learning in flipped classes is a more effective pedagogy than straight PowerPoint lectures [2] and the results from this paper suggest that this pedagogy can be delivered either with or without a computer.

The various avenues of interaction offered by tools like TeachBack and Spinoza offer increased participation and involvement rates. But that is not all, like most computer-mediated communication tools, TeachBack and Spinoza allow content and conversations to be stored and accessed at later times. Both of these tools closely record students’ participation and performances across the various interactive tools. For instance, in TeachBack, these statistics are both aggregated and viewable at the activity, lecture and course levels, enabling close monitoring of students’ progress in the class. Moreover, this information is individually provided for each student. Non-summative assessment data of this nature can be useful in early detection of students in trouble as well as provide new avenues to learn best how to carter to students’ learning needs.

In addition, participants do not have to be in the same physical locations, and users can engage in multiple conversations at once. In a way, these tools liberate learning and teaching from constraints of time and distance [6, 9] where barriers such as distance, disabilities, shyness and cultural difficulties are overcome. Our proposed computer-mediated pedagogy features various interactive and engaging activities that do not give students the opportunities to get distracted. However, as we have discovered in this study, there are a few students who are ill equipped to handle computer-mediated interactions and online environments. Our results suggest that it might be worthwhile to offer two versions of the CS1 class, one which is fully computer-mediated providing the instructor with high quality and timely information about student performance, and one that is not computer-mediated to accommodate those students who are prone to distraction when given access to a computer in class. Another alternative approach is to teach a hybrid class, where computers are only allowed during certain in-class activities and are banned at other times, or to give accommodations to students who are highly distractible and have them work without computers in class.

9 Future Work

From the instructors’ point of view, the use of computer-mediated pedagogy does have many benefits. As mentioned above, it provides a detailed record of the activity of each student in the class including which questions they answered, whether their answers were correct, how they tried to solve a programming problem, what their level of confusion was during and after each activity, etc. In this experiment, we did not use this additional data to customize our support for individual students in the class. We strongly suspect that this detailed information about individual students could be used to provide individualized support for at-risk students in a way that would make a statistically significant difference in learning outcomes. In our future studies we plan to test this hypothesis, where we will add early detection and notification functionalities that would enable to the ultimate goal of providing needed early support to students in need. We also plan to analyze our data after the fact to see whether it we are able to accurately predict which students do poorly in the course based on this fine-grained data from TeachBack and Spinoza.

We also plan to rigorously compare various interactive pedagogies which mix mandatory computer-use and mandatory non-computer use in the same class. For example, a few students commented about being distracted with extended computer use or at times preferring traditional lecture activities. We plan to design experiments to compare the effectiveness of three types of classes: those in which computers are required the entire semester, those that ban computers the entire semester, and hybrid classes in which computer use is sometimes required and sometime banned, possibly even during a single class session.

Based on the successes of iResponder in facilitating quick formative assessment questions where students respond individually, we plan to add the functionality for group questions where students would respond in small groups. The goal is to encourage more supportive and motivational peer discussions in smaller groups. Such a feature would make it easier to run Think/Pair/Share exercises and track the effect of sharing on changing students answers. Moreover, there are many ways of forming these discussion groups and therefore it will be a good framework to compare the effectiveness of these various techniques in forming student groups.

Finally, we are planning to build SOTL tools into TeachBack and Spinoza that will allow the instructor to tag certain days by the pedagogy used and then automatically test for statistically significant differences in the in-class student performance. The goal would be to encourage faculty to think about ways they could vary their pedagogy and then easily look for any differences in learning outcomes.