Combinatorial tasks and outcome listing: Examining productive listing among undergraduate students

Abstract

Although counting problems are easy to state and provide rich, accessible problem-solving situations, there is much evidence that students struggle with solving counting problems correctly. With combinatorics (and the study of counting problems) becoming increasingly prevalent in K–12 and undergraduate curricula, there is a need for researchers to identify potentially significant factors that might have an effect on student success as they solve counting problems. We tested one such factor among undergraduate students—their systematic listing of what they were trying to count. We argue that even creating partial lists of the set of outcomes led to significant improvements in performance in students’ success on problems, implying that systematic listing may be worthwhile for students to engage in as they learn to count. Our findings suggest that more needs to be done to refine instructional interventions that facilitate listing. We discuss these findings and suggest avenues for further research.

1 Introduction and motivation

Enumerative combinatorics, or the solving of counting problems, has become increasingly prevalent in K–12 curricula (e.g., English, 2005) and in undergraduate mathematics courses. This increased attention is not surprising, as counting has practical applications in computer science and probability (e.g., Abrahamson, Janusz, & Wilensky, 2006; Shaughnessy, 1977). and counting problems are accessible but require critical mathematical thinking to solve (e.g., Kapur, 1970; Martin, 2001; Tucker, 2002). The mathematics education research community has seen increased attention on students’ combinatorial reasoning, with a number of researchers over the last two decades examining common errors, strategies, and ways of thinking related to students’ solving of counting problems (e.g., Annin & Lai, 2010; Batanero, Navarro-Pelayo, & Godino, 1997; Eizenberg & Zaslavsky, 2004; English, 1991, 2005; Halani, 2012; Lockwood, 2011, 2013, 2014; Lockwood, Swinyard, & Caughman, 2015; Maher, Powell, & Uptegrove, 2011; Tillema, 2013). In spite of the importance of combinatorial reasoning, a number of these studies suggest that, overwhelmingly, students face difficulties with solving counting problems correctly. Given such struggles, there is a substantial need for more investigations into effective ways to improve students’ solving of counting problems.

In particular, we have previously reported that knowledge of the set of outcomes is a key component of students’ combinatorial reasoning (Lockwood, 2013). and we have repeatedly argued that students may benefit from grounding their counting activity in sets of outcomes (Lockwood, 2014; Lockwood et al., 2015). Even more specifically, we recently made a case for the benefit of a set-oriented perspective toward counting, which is a way of thinking (in terms of Harel, 2008) about counting that “involves attending to sets of outcomes as an intrinsic component of solving counting problems” (Lockwood, 2014, p. 31). Here, we provided examples of students who had an underdeveloped, a developing, and a robust set-oriented perspective and established the value of such a perspective in solving counting problems. However, while such studies have provided qualitative evidence of the importance of focusing on sets of outcomes, there has not yet been studies that demonstrate that a focus on outcomes might have a statistically significant effect on students’ solving of counting problems. In this paper, we report on a study that examined the effects of having students engage in systematic listing—that is, to create an organized list (or even a partial list) of the outcomes they are trying to count. We answer the following research questions:

• 1) Does engaging in systematic listing have a statistically significant effect on students solving counting problems correctly?

• 2) What are features of productive or unproductive lists that students generate as they solve counting problems, and what insights can we gain about productive combinatorial listing?

The broad aim of the study is to contribute to research on students’ combinatorial reasoning, specifically investigating whether a focus on outcomes is positively correlated with solving counting problems correctly. As we will argue, even creating partial lists of outcomes positively affected performance in students’ success on counting problems, and this implies that systematic listing of outcomes may be a worthwhile activity for students as they learn to solve counting problems.

2 Literature review and theoretical perspective

2.1 Students’ difficulties with counting

Although there are some reports of success in which even young children display robust combinatorial thinking (e.g., English, 1991; Maher et al., 2011). for the most part, research on students’ work on counting problems shows that students substantially struggle with solving counting problems. Batanero, Godino, and Navarro-Pelayo (2005) highlight the need for researchers to explore and better understand such difficulties:

Combinatorics is a field that most pupils find very difficult. Two fundamental steps for making the learning of this subject easier are understanding the nature of pupils’ mistakes when solving combinatorial problems and identifying the variables that might influence this difficulty. (p. 182)

One piece of evidence of this difficulty is low overall success rates, with several researchers reporting findings in which postsecondary students solved well under 50 % of counting problems correctly (e.g., Eizenberg & Zaslavsky, 2004; Godino, Batanero, & Roa, 2005). In addition to low performance rates, there is evidence in the literature of how, specifically, students struggle. Batanero et al. (1997) listed several error types they found in students’ work, including errors of order, errors of repetition and using incorrect arithmetic operations. Eizenberg & Zaslavsky (2004) point out that because of the nature of counting problems and their very large numerical answers, counting problems can be particularly difficult to verify. Other researchers have highlighted specific mathematical features of counting problems that are especially difficult, such as issues of order (Batanero et al., 1997) and overcounting (Annin & Lai, 2010; Lockwood, 2014). Given the pervasive difficulties that students face, there is a need to identify potentially productive interventions that may help students solve counting problems more successfully.

2.2 Sets of outcomes

Theoretically, our focus on systematic listing stems from the idea that students may benefit from grounding their counting activity in the concrete set of outcomes they are trying to count. We use the term outcome because we conceive of the objects being counted as the result or output of a counting process, and we use the term desirable outcome of a counting problem to mean an outcome that satisfies the particular constraints of that problem. The study draws upon a model of students’ combinatorial thinking (Lockwood, 2013, Fig. 1). which proposes three basic components of students’ counting (expressions/formulas, counting processes, and sets of outcomes) and expounds upon the relationships between these components (the model is further elaborated in Lockwood et al., 2015). We define the set of outcomes as the “collection of objects being counted—those sets of elements that one can imagine being generated or enumerated by a counting process” (Lockwood, 2013, p. 253). In terms of the model, the idea of systematic listing, and especially the act of reflecting on how to create an organized list of outcomes that correctly answers a counting problem, lies in the relationship between counting processes and sets of outcomes.

Fig. 1

Lockwood’s (2013) model of students’ combinatorial thinking (p. 253)

A major motivation for the focus on outcomes is that to solve a counting problem correctly, we must know that we have counted each of the desirable outcomes exactly once. Correctly identifying exactly those outcomes that should be counted, and then counting them exactly once, is the crux of solving counting problems correctly. Students, however, can tend to gloss over the outcomes in favor of moving too quickly to formulas or quick techniques to solve counting problems (e.g., Kavousian, 2008). Although the focus on sets of outcomes stems primarily from our previous work (especially Lockwood, 2013, 2014). other researchers (English, 1991, 2005; Hadar & Hadass, 1981; Mamona-Downs & Downs, 2004; Shaughnessy, 1977) have acknowledged that emphasizing the set of outcomes could support counting activity. Our premise in this paper is that students may benefit from explicitly focusing on sets of outcomes via listing, and, broadly, our work is motivated by a goal to investigate whether (and if so, how) students may benefit from a focus on sets of outcomes.

2.3 Listing strategies

Here, we briefly frame listing within broader problem-solving literature, and we then discuss listing in a combinatorial context specifically. We are most interested in listing as a combinatorial tool in this paper, but we acknowledge that we may also gain some insights from listing in the more general context of problem solving.

Systematic listing as a problem-solving strategy

Although our focus is on the effects of listing in a combinatorial context specifically, the activity of listing fits more broadly into literature surrounding problem solving. The systematic nature of the listing we are investigating suggests that the students are intentional about the listing they do. This aligns with Polya’s (1945) well-known four-step problem-solving process of understanding the problem, devising a plan, carrying out the plan, and looking back (pp. 5–6), particularly the devising a plan step. In using systematic listing as a part of their problem-solving phase, students might have different reasons for creating a list of outcomes. They may want to construct a complete list, yielding the answer to the counting problem, or they may intend to create a partial list and then extrapolate patterns from that list to find a solution. In either case, a list that is constructed systematically would involve some structure that could shed light on the answer to the counting problem. On the other hand, sometimes students may not put much thought into their list and may simply write down outcomes in an unsystematic way. In contrast with the intentional construction of a systematic list, this kind of activity belies a lack of a well-designed problem-solving plan. Thus, the way in which students list, and the systematic or nonsystematic nature of their listing, could characterize their awareness of their overall problem-solving plans and how that listing fits into their solution.

The degree to which students consider how such activity fits into their overall solution is related to their control processes (e.g., Carlson & Bloom, 2005; Schoenfeld, 1992). Carlson and Bloom characterize control as encompassing “metacognition and monitoring and all associated behaviors” (p. 48), and they emphasize the important role that control plays in the problem-solving process. Schoenfeld (1992) also highlights the need for control and self-regulation in problem solving. He characterizes self-regulation in the following way: “Monitoring and accessing progress on line, and acting in response to the assessment of on-line progress, are the core components of self-regulation.” (p. 355). Our examination of systematic listing in counting can thus be framed more broadly in terms of students’ broader problem-solving processes, as systematic listing both fits within a stage in the problem-solving process and can be situated within overall counting activity during metacognitive reflection.

Systematic listing as a combinatorial tool

A significant challenge with solving counting problems in particular is to convince oneself that each of the desirable outcomes have been counted exactly once. Constructing a systematic, organized list can allow us to make convincing arguments about why we have counted all of the outcomes, and failure to construct a systematic list can cause problems for students. Batanero et al. (1997) include non-systematic listing as one of their errors, which “consists of trying to solve the problem by listing using trial and error, without a recursive procedure that leads to the formation of all the possibilities” (p. 192). Others have characterized this recursive procedure as an odometer strategy. English identified combinatorial strategies in her work with young children (age 4 to 9 years), describing the odometer strategy as among the most sophisticated of the strategies she observed. She defined this strategy as having a consistent and complete cyclical pattern with “a ‘constant’ or ‘pivotal’ item…Upon exhaustion (or apparent exhaustion) of the item, a new constant item is chosen and the process repeated” (p. 460). Building on English’s work, Halani (2012) identified an odometer way of thinking. Both of these researchers’ work suggests that there are overall listing strategies with which students engage, and some, like the odometer strategy, can be quite systematic. A major benefit of the odometer strategy is that it convincingly provides a rationale for why no outcome is missed. Our study is grounded in the fact that such organized listing strategies may be helpful in convincingly providing (justifiably) complete lists of outcomes. We hypothesize that systematic listing can give students a mechanism by which to convince themselves that they have all of the outcomes—something that is not trivial.

The section above is meant to highlight that although there is some mention of outcomes in the combinatorics education literature, the treatment of outcomes is largely implicit. That is, researchers have not set out to systematically test the effectiveness of students’ engagement with outcomes when counting. Similarly, while listing has been identified as a common strategy among students, studies have not targeted the effects of listing on combinatorial performance. Thus, while a variety of listing strategies has been documented, they have primarily emerged as one aspect of students’ strategies (English, 1991) or ways of thinking (Halani, 2012). and listing itself has not been explicitly targeted as a statistically significant factor in students’ solving of counting problems.

In this study, our goal is to show the effectiveness of one particular factor in solving counting problems—systematic listing—in helping students to count successfully. Here, we seek to accomplish this in two ways—first, by quantitatively reporting on the effectiveness of listing on students’ performances on counting problems, and second, by qualitatively detailing the nature of productive versus unproductive lists in undergraduates’ listing strategies. Because of the nature of our study, we cannot make claims about what this means for students’ thinking or reasoning; all we have is their performance and their listing activity. Nonetheless, this quantitative evidence is an important step toward showing the value of focusing on outcomes (Lockwood, 2014; Lockwood et al., 2015). and it provides motivation to study implications of these findings in terms of students’ thinking in subsequent work.

3 Methods

3.1 Participants and data collection

Forty-two undergraduate students participated in the study. These students were enrolled in an introductory psychology course at a large Midwestern university, and they received extra credit for their participation. Demographic information revealed varying degrees of experience with counting problems and suggested that almost all of the students had seen counting problems before (most typically in high school but not formally in college). The nature of counting problems, which does not require any mathematical prerequisites to solve, does not preclude students from being able to approach counting problems, and the psychology students represented novice counters. The students completed a written assessment consisting of counting tasks, which took about 60 min to complete. We administered the assessments to students across three different days. Students signed up for hour-long time slots during which they completed a written assessment with the following written prompts:

• Please answer the following questions.

• You have 60 min to work on these questions.

• Please at least start/try all of the problems.

• We are as interested in your problem solving process as we are in the final answer.

• Get as close to the final answer as you can without a calculator.

• Only write in gray areas [this was for Livescribe pen capture].

• Please show your work and indicate your final answer.

In a given day, the students took the survey in groups of 1–6 students at a time. The totals for the 3 days were 13, 19, and 10, respectively (totaling 42 students).

3.2 Tasks

The written assessment consisted of ten counting problems that would be accessible to novices, involving relatively simple applications of addition and multiplication. We chose tasks with a variety of sizes of sets of outcomes (some which could not easily be listed by hand) in order to allow us to see if even partial listing might help students count successfully. In all of the problems, even those in which listing all outcomes was not plausible, we hoped that students would be able to write down outcomes and perhaps use that listing to determine a useful pattern or structure. The name and statement of each task and the cardinality of the answer are outlined in Table 1 (if a reference is noted, the task was adapted from that book; otherwise, we developed the task).

Table 1 Assessment tasks

We used Livescribe pens to collect the data, which have technology that allows the students to write in pixelated notebooks that recorded written work and audio. The written work is then embedded into a pdf file, and that recording can be played back, revealing what was written and spoken in real time. This technology facilitates rich analysis of written data (without the same time investment that interviews require) as the researchers are able to trace exactly how and when particular aspects of a written response were inscribed. We did not analyze the audio-recordings as the students simply sat in the room quietly and completed the written assessment.

3.3 Analysis

Initially, the first author coded the responses according to correctness (correct or incorrect) and then coded the student responses according to four categories of listing: Fig. 2a–d exemplifies responses that received codes of no listing, articulation, partial listing, and complete listing, respectively. A code of no listing was given if there was no attempt at any kind of list. In terms of Lockwood’s (2013) model, a solution with no listing might involve a numerical value or expression, which may or may not suggest a counting process, but the answer is not explicitly related to the set of outcomes. A code of articulation emerged during analysis, as some responses involved more than only providing a formula, but they were not quite suggestive of even a partial list. This articulation code was given when a student wrote down at least one instance of what they were trying to count (one outcome) but did not actually create any kind of list. Here, the student may be attending to an outcome, but she does not acknowledge that an entire set of outcomes exists or may be meaningfully structured, nor does she establish a clear connection between how a counting process might generate that outcome. A code of partial listing was given if there was some evidence that the student created a list or partial list of the outcomes, but they may have not written the entire list correctly or may have truncated their listing when they identified a pattern. The partial listing suggests that a student has a counting process that generates outcomes in some way, thus establishing a relationship between counting processes and the set of outcomes, but this relationship is not fully realized or articulated. A code of complete listing meant a student provided a complete, correct list of the outcomes. Here, the list is generated via a counting process. The process need not be sophisticated, but the point is that the student is drawing on some counting process to create a complete list of outcomes. All problems were coded one problem at a time to maximize the consistency in coding per problem. The research team discussed questions about particular codes.

Fig. 2

a No listing. b Articulation. c Partial listing. d Complete listing

The quantitative findings suggested that listing could be potentially beneficial for students (discussed in Section 4), and we were thus motivated to look more closely at students’ work to learn more about what aspects of listing might be particularly helpful and why. For the qualitative analysis, then, we reviewed the pdfs of the students’ work and watched back through the real-time work, focusing especially on those solutions that had been coded as correct and involving partial or complete listing. We used the constant comparison method (Strauss & Corbin, 1998) to document features of lists that yielded correct results versus those that did not, and for each student’s work, we recorded activities and phenomena that shed light on the nature of productive listing. The goal was to examine what aspects of students’ listing behavior seemed to contribute to successful solutions. The Livescribe pens facilitated an in-depth look at the listing strategies of those students who listed successfully. Indeed, on a number of occasions, we were surprised to see the writing unfold in a different way than the static written work alone might have initially suggested, as examples will show.

4 Results

In this section, we present both quantitative and qualitative results. Together, these contribute to the overall narrative that certain listing behaviors and activities seem to be beneficial for students’ counting and that listing warrants more attention in combinatorics education research.

4.1 Quantitative results

In the following analysis, we only used problems where the answer was clearly correct or incorrect, and where the listing behavior was clear (a total of 352 problems—some problems were excluded because of poor Livescribe pen capture). To address research question 1 (Does engaging in systematic listing have a statistically significant effect on students’ solving counting problems correctly?), we measure student performance by the number of questions answered correctly. On the whole, students struggled to solve these problems correctly, with only 24 % (84/352) accuracy overall. Upon examining the students’ listing behavior, we discovered that listing had an overall positive effect on correctly solving a problem (here, we take listing as including a code of either partial or complete listing). We performed two tests to confirm this.

In the first test, we asked were students just as likely to get problems correct, regardless of whether they listed or did not list? In other words, given whether or not they listed, what is the likelihood that students got the problem correct? To answer this, we calculated, for each student, the proportion of problems on which they were correct out of those problems on which they listed, as well as the proportion of problems on which they were correct out of the all problems on which they did not list. This required looking only at those students who both listed and did not list (n = 39). Figure 3 shows a summary of these proportions, averaged across students. In both columns, the top of the columns (light blue) is the mean proportion of correct problems, compared to the bottom of the columns (dark blue), which is the mean proportion of incorrect problems. Summarizing across students, we find an average proportion of correct and listing to all listing in column 1 (mean = 0.29, SD = 0.28), and an average proportion of correct and not listing to all not listing in column 2 (mean = 0.11, SD = 0.21). Performing a paired t test where we consider each student individually, we found a significant difference between these two proportions (t(37) = 3.92, p < .00038). The significance we show here is the difference between the relative sizes of the top portions of each column to that column.

Fig. 3

First paired t test

In the second test, we asked a similar question: Were students just as likely to have listed on problems they got correct as those they got incorrect? In other words, given whether or not they got the problem correct, what is the likelihood that they listed? For each student (who had both correct and incorrect answers, n = 29), we calculated a proportion of the number of problems with listing out of number correct and number of problems with listing out of number incorrect. Summarizing across students, we find an average proportion of listing and correct to correct (mean = 0.69, SD = 0.38) and an average proportion of listing and incorrect to incorrect (mean = 0.40, SD = 0.26). Figure 4 summarizes these results, with the top portion in each column (light blue) being the mean proportion where students listed and the bottom portion (dark blue) being the mean proportion where they did not list. Applying a paired t test where we consider each student individually, we find a significant difference (t(29) = 5.32, p < .000011). Again, the important feature is the large difference in the relative size of the top portion of the column to the entire column.

Fig. 4

Second paired t test

In sum, the quantitative results show that listing behavior was positively correlated with correctly answering counting problems. While there is a correlation between listing and correctly answering a problem, we do not claim causation. We acknowledge that it may be the case that stronger students may naturally list, and that is why we see the positive correlation. However, regardless of whether success leads to listing or vice versa, the correlation is promising—if the more successful counters are listing, perhaps listing deserves more attention as a pedagogical focus. Given students’ clearly documented and sustained struggles with counting problems, these initial quantitative findings suggest that listing may be a valuable way to help students count more successfully. The fact that systematic listing of outcomes is positively correlated with performance provides much-needed quantitative evidence to support our previous claims about the importance of focusing on outcomes (Lockwood, 2013, 2014). We examined the students’ written work to study the students’ listing in more detail.

4.2 Qualitative results

These qualitative results stemmed from the basic quantitative findings, as discussed above, with the aim of answering research question 2 (What are features of productive or unproductive lists that students generate, and how does listing activity differ in the generation of productive versus unproductive lists?). In analyzing the students’ written work, we distinguish between productive lists and unproductive lists. We take productive lists to mean any lists, partial or complete, which were generated on a problem that the student eventually solved correctly. Unproductive lists are lists that were generated on a problem that was not solved correctly.¹ Because of the nature of our data, we cannot make conclusive statements about whether or not a particular list actually caused a student to answer a problem correctly. However, for analytic purposes, we found the productive versus unproductive distinction to be helpful as we tried to determine potential aspects of listing that seemed particularly beneficial for students’ counting. We first discuss features of productive lists (providing contrasting examples of unproductive lists), and then we present additional noteworthy aspects of listing that arose among multiple students. These qualitative results complement the quantitative results presented previously, helping to paint a clearer picture of precise ways in which listing seemed to be effective for students in some situations.

4.3 Features of productive lists

In this section, we discuss three key features of productive lists, which, while not necessarily present in every productive list, are representative of overall characteristics of productive lists.

Useful notation and appropriate modeling of outcomes

Students who wrote productive lists typically found a suitable notation that appropriately modeled an outcome. In some problems, when an outcome is fairly self-evident and can easily be written on the page literally (such as letter or number sequences), figuring out a meaningful notation may be trivial. Other problems may require extra effort to translate the outcome into something that can be written down and listed, and many problems require some translation of the problem into a useful notation. The lollipop problem, in which the outcomes being counted are sets of three students who receive lollipops, highlights the value of a usable notation and shows how students displayed a variety of notations even on the same problem. For example, Student 1² let the letters A, B, C, D, E, and F represent the six children. Her outcomes were sets of three letters (representing three students), which she listed lexicographically (Fig. 5).

Fig. 5

Student 1’s lollipop problem

Student 2 encoded the outcomes by creating a table that labeled columns 1–6 for the children (Fig. 6). The student represented an outcome as a row of three marks, with one mark in exactly one column, and the total number of rows gave the final answer. While other students used tables in a similar way, this table is particularly effective in its use of different marks for rows that have marks in different first columns. That is, for the sets that include child 1, the student wrote rows as simply three dots. When he shifted to those including column 2 but not 1, he wrote xs. Then, when writing the rows that include column 3 but not 2 or 1, he wrote small circles, and he finally wrote the last outcome with filled-in circles. This notation allowed for him to count up the rows, but it also helped him to keep his place in his listing and reflects a structure in the outcomes.

Fig. 6

Student 2’s lollipop problem

In contrast to these lists that show useful notations that facilitated a way to encode outcomes, on a few occasions, students’ notations seemed problematic and potentially had a negative impact on their solving of the problem. For example, in Student 3’s work on the lollipop problem (Fig. 7), her labeling of lollipops 1, 2, and 3 suggests that she was thinking of the lollipops as distinct. When she writes permutations of the numbers 1, 2, and 3 beneath six children, this suggests that she has not clearly articulated what constitutes an outcome. Her issue may not be merely one of notation—she may have some incorrect notion of what the problem is asking—but her lack of a clear notation certainly does not help. In comparison to other students’ useful notations on these problems, hers fail to convey much of the insightful, organized information that other notations afforded. Importantly, the notation did not facilitate a correct articulation of what constituted a desirable outcome. A key aspect of being able to create a productive list is to correctly model the outcomes, which often involves developing a useful and efficient notation.

Fig. 7

Student 3’s lollipop problem

Proper implementation of an organized strategy

Another feature of productive lists was that they often seemed to be developed with an intentional organizational strategy. Student 1’s work on the lollipop problem (Fig. 5) exemplifies the odometer strategy (English, 1991). In order to list the set of three letters, she began by holding the first and second elements constant and then cycling through the last elements in the order of how she initially wrote the six letters that represented the students. Then, after cycling through every possibility for the third letter, she changed the second letter and again cycled through the remaining options for the third letter. Once she had similarly cycled through each possibility for the second letter, she could move to the next choice for the first option and repeat the process. Because we could watch her make the list in real time, we know that she did in fact implement this process as she listed outcomes.

A similar organizational strategy was exemplified by a number of students in the CATTLE problem. These students did not create an entire list for this problem; instead, they wrote out some of the arrangements of the letters C, A, L, and E, identified a pattern, and used multiplication to calculate the total. For example, Student 4 wrote out the two Ts and arrangements of the letters A, C, L, and E. The student’s real-time listing shows an attempt on his part to remain organized and systematic. He first wrote TTACLE but then crossed it out, and we infer from the rest of his work that he sought to list alphabetically. Figure 8a shows he then proceeded to write the first alphabetical outcome, TTACEL, followed by TTAECL. As he was writing TTAELC, he seemed to realize that he had missed another outcome starting with “AC,” and so, he went back and added TTACLE to the top of his list, pairing it with TTACEL. Figure 8a shows him in the process of going back and adding TTACLE to the top of the list. He then proceeded to complete an alphabetical list of arrangements starting with A. He wrote one of the arrangements starting with C (TTCALE) but then seemed to notice a pattern. Figure 8b shows his final list, in which he noted there were six options for the starting letter, and he multiplied this 6 by 4 and then by 2 to yield the correct answer. There are two points to make about this example. First, we see a student using an organized strategy on a partial (as opposed to a complete) list that ends up being productive. Second, we see that the student was intentionally organized in his listing. He was careful as he listed those six options that begin with A, to the point of going back and adding an outcome where it best belonged within her scheme. This organized, near alphabetical pairing of certain outcomes helped ensure that he had all of the outcomes.

Fig. 8

a Student 4’s partial list of the CATTLE problem. b Student 4’s complete work on the CATTLE problem

Student 4’s work reflects the kind of metacognitive problem activity described by Carlson and Bloom (2005) and Schoenfeld (1992). We see the student, who is in the process of listing, pause and seemingly evaluate the list so far and then return to the top of the list to make a more organized list of the outcomes. His activity suggests self-regulation that resulted in a list with structure that convincingly justifies why all of the words starting with TTA are accounted for. In terms of Lockwood’s (2013) model, Student 4’s work suggests a connection between a counting process and the sets of outcomes—he implemented his listing so as to create an organized, structured list, even going as far as to adjust his listing to improve the organization of the outcomes.

In contrast to the organized lists above, we present two instances of unproductive lists. Unproductive lists most typically seem to have resulted from an organizational strategy (typically the odometer strategy) not being properly implemented. For example, on the lollipop problem, Student 5 modeled the children as the numbers 1–6, and an outcome is a set of three numbers that represents the three children who would get a lollipop (Fig. 9). Her list in Fig. 9 reflects a systematic odometer listing, and this work demonstrates some clear organization and additionally belies a particular structure. It also suggests that the student correctly interpreted what constituted an outcome (she did not overcount 231 and 132 as distinct outcomes). However, she neglected to proceed to the next step, including sets of children that do not include child 1. This example represents an incomplete implementation of the odometer strategy.

Fig. 9

Student 5’s lollipop problem

Another common observed phenomenon was for students to encapsulate the listing process erroneously by making incorrect assumptions about how the list would generalize. On the committee problem (Fig. 10), Student 6 correctly systematically listed all the committees containing an F, of which there are 15, demonstrating an odometer strategy in holding the second letter constant and cycling through each subsequent option for the third letter. However, once the student arrived at the answer of 15, he simply multiplied this answer by 7 (we conjecture that he may have reasoned there are seven options for what the “first” letter in his list could be). In situations like these, it is unclear whether the error represents some deeply held belief about what constituted a list of outcomes or is simply a matter of carelessness. Regardless, this example demonstrates a situation in which a student incorrectly assumed that a partial list could be extrapolated in a given way.

Fig. 10

Student 8’s committee problem

Evident structure

Some productive lists had an obvious structure that elicited a certain way of organizing the outcomes, and the structure evident in some students’ lists contributed to a convincing argument that all of the outcomes were being counted exactly once. In discussing the model, Lockwood (2013) notes that “a counting process can be seen as generating some set of outcomes” (p. 256) and goes on to say that “a counting process can impose a structure onto a set of outcomes (and, in fact, different counting processes can result in different structures)” (p. 256). The following examples demonstrate this phenomenon that a given counting process (or listing process, in this case) can structure outcomes in various ways, some of which are more or less helpful in justifying whether a list is complete.

Student 7’s work on the lollipop problem (Fig. 11) clearly yielded a list that suggests a correct sum, which the student wrote down: (5 + 4 + 3 + 2 + 1) + (4 + 3 + 2 + 1) + (3 + 2 + 1) + (2 + 1) + 1. The student encoded triangles as the students (distinct because they are in a line), with rows of circles under three of the triangles representing an outcome. She apparently made use of the odometer strategy by first holding constant the circles in the first two columns and then cycling the third through the remaining columns. Then, while still keeping the first entry static, she moved the second circle to the second triangle and cycled through all of the possibilities for the first triangle, continuing in this way while keeping the first item constant. The first sum of 4 + 3 + 2 + 1, then, includes all the outcomes with a circle under the first triangle (or with the first child receiving a lollipop). She then repeated the process to produce the remaining sums. It is noteworthy that the structure of the sum visually pops out of the list, making apparent how the student meaningfully organized the outcomes to count them effectively.

Fig. 11

Student 7’s lollipop problem

In contrast to Student 7’s work, on the same lollipop problem, Student 8 wrote the 20 outcomes in a 5 × 4 array (Fig. 12). Given the student’s subsequent writing of $\left(\begin{array}{l}6\\ {}3\end{array}\right)$, he may have already guessed or arrived at the answer and wanted the array to reflect the answer of 20. However, the array, while correctly representing an answer of 20, does not offer much insight into the structure of the list or why the student may be convinced that all of the outcomes are counted. Unlike the work in Fig. 11, there is no further insight gained by the structure of how the list is written. In fact, while on some problems an array might provide some insight, here the particular arrangement of the outcomes in the array obfuscates any relevant structure of the set of outcomes.

Fig. 12

Student 8’s lollipop problem

In sum, in terms of Lockwood’s (2013) model, having a list that highlights a particular structure is an effective way of connecting a counting process to a set of outcomes, and examining the set of outcomes can be an important means by which to be sure a student is counting correctly. Thus, a list with a transparent structure may provide more concrete evidence that the list may be correct, and creating such a list may be a productive verification strategy that could be investigated in subsequent studies.

4.4 Other insights into productive listing

In addition to identifying features of productive versus unproductive lists, we identified two themes across students’ work that shed light on productive listing: productive listing experience seemed to affect students’ work on other problems, and even partial listing proved to be beneficial.

Productive listing experience seemed to affect students’ work on other problems

Perhaps one of the more surprising results that came out of the qualitative analysis was to see the dynamic way in which some students’ work unfolded across problems. The real-time pdfs showed that students were much more creative and dynamic in their listing than can be seen simply from the written work on the page. This came out most pointedly as students moved back and forth between problems. On several occasions, it seemed that successful listing on one problem led students to go back and revisit previous problems, incorporating listing strategies to arrive at correct answers. As an example of this, Student 9 worked on the domino problem, initially drawing out dominos and listing [0,0] through [0,6] and writing ×6 = 36 (see the dark but not grayed out writing in Fig. 13). The grayed-out work would suggest that she then proceeded to list the rest of the dominos, cross out the duplicates, and arrive at the correct answer of 28. However, immediately following writing down 36, she moved on to the next problem. The dynamic recording reveals what was, to us, a surprising phenomenon, given the remaining written work on the page.

Fig. 13

Student 9’s initial domino problem

Student 9 then solved the lollipop problem, engaging in very careful and systematic listing (Fig. 14), ultimately arriving at the correct answer of 20 using a very well-organized list.

Fig. 14

Student 9’s lollipop problem

Upon completing the lollipop problem, she immediately returned to the domino problem, systematically listed all of the dominos, crossed out duplicates, and arrived at the correct answer of 28 instead of 36 (Fig. 15). While we cannot know for sure what caused this (because we did not ask her), it is interesting that directly following successful, systematic listing on the lollipop problem, she used listing to fix an initially incorrect answer. We conjecture that the student gained an important insight as she listed in the lollipop problem—namely, the lollipop problem does not count rearrangements of the same three students as distinct outcomes. That is, by listing out the lollipop possibilities, she knew that she did not want to count sets {1, 2, 3} and {2, 1, 3} distinctly toward the total. The same is true of dominos—domino [1,2] is the same as domino [2,1]—and her behavior on the domino problem suggests that the work on the lollipop problem led her to change her strategy on the domino problem. One might argue that it was not the act of listing itself that caused her realization, but rather that it might be caused by exposure to another problem with outcomes of a similar kind. However, we contend that the listing on the lollipop problem drew attention to the nature of outcomes in ways that trying to solve the problem without listing might not have done. Additionally, the student’s systematic, detailed list on the domino problem suggests that the act of listing itself was something she carried over from the lollipop problem and chose to use in her subsequent solution of the domino problem.

Fig. 15

Student 9’s correct domino problem

Other students similarly revisited a problem after perceived successful listing on another problem, suggesting that Student 9’s case was not unique. There may be other explanations for the students’ revisiting of problems, but we believe our interpretation that successful listing triggered new insight on previous problems is at least plausible. Because of this, we feel that the effect of successful listing on other problems is something that could be investigated more explicitly in further studies.

Even partial lists can be productive

The second theme we identified was that even partial listing proved beneficial for students on a number of occasions. That is, it did seem to be the case that students could, at times, productively arrive at the correct answer by creating only a partial list without having to list all of the outcomes completely. Indeed, of all of the correct answers in which students listed, 63.2 % of them were partial lists. Often, some type of encapsulation process was observed in the list, and while students at times overgeneralized (as in student 6’s work on the committee problem in Fig. 10), many students were able to identify and correctly use a pattern they saw in a partial list. Student 4’s work on the CATTLE problem above is one such example of a productive partial list.

As another example, Student 10’s work on the apples and oranges problem (Fig. 16) demonstrates an increasingly efficient process that emerged during the students’ work. Figure 16 shows that the student began a fairly detailed complete list. The first two columns show every possible combination of 0 oranges with 1–8 apples, as well as 0 apples with 1–8 oranges (note, this is insightful, as the 0–0 combination of apples and oranges is not allowed). By the fourth column, though, he has streamlined the process and no longer seems to feel the need to write the number of apples in each case. Subsequent columns become even more streamlined, and the process of listing each outcome is encapsulated and truncated. The student is still organized, displays a structure in the list, and arrives at the correct solution without having actually listed all 80 outcomes.

Fig. 16

Student 10’s apples and oranges problem

This finding that even partial lists can be productive is extremely important. If listing were only beneficial if the problem’s solution can easily be listed completely, the value of listing would have serious limitations, as most counting problems have solutions that cannot easily be physically listed by hand. It is unrealistic to claim that students might be able to list all desirable outcomes as a regular part of their solving of counting problems. The fact, then, that we have evidence that listing appears to be a useful strategy, even on problems which students may not choose to (or be able to) create complete lists, is promising for increasing student success in a variety of counting situations.

5 Discussion

5.1 Certain problems facilitated listing more than others

One point of discussion is that the number of instances in which students attempted to list, as well as whether such listing was productive, varied from problem to problem. Appendix 1 demonstrates this phenomenon and includes counts for each problem according to correctness and listing. For example, the CATTLE and ABCZZZZ problems are relatively comparable in the total number of outcomes that students would have to list (48 and 35, respectively), and yet, the CATTLE problem had a much higher success rate (13 correct solutions, 10 involving listing, as opposed to the ABCZZZZ problem with only 3 correct solutions, 2 involving listing). We suspect that one difference may be that the CATTLE problem has a clearer multiplicative structure, and this might affect the ease with which the listing process can be meaningfully encapsulated. Another difference might pertain to the nature of what was being counted—perhaps the constraint of the alphabetical letters in conjunction with the identical Zs felt overwhelming for students.

We highlight this distinction to suggest that certain problems did seem to elicit listing (partial or complete) more than others, and this may hold some implications for instruction. If we indeed accept the premise that listing can be a helpful factor in counting, and that it might be worthwhile for students to do at least some listing as they learn counting initially, then teachers should take into account what problems might best facilitate listing and at least be aware that not all problems are equally effective in eliciting listing. From a research perspective, the effects of certain problem features and problem types on students’ listing behavior warrant additional study.

5.2 Careful and deliberate work is important

A noteworthy factor in helping students to list productively seems related to having a careful disposition toward mathematical work (in this case, articulating outcomes and listing), which is in line with findings other researchers have shared (e.g., Hadar & Hadass, 1981; Lockwood, 2013, 2014). The nature of our data does not allow us to draw any conclusive findings about how to appropriately define being “careful,” or even whether students were intentionally being particularly cautious or careful in their work. However, analyzing the productive lists does suggest that successfully creating organized lists comes about through careful (as opposed to careless) listing of outcomes. We suspect that students who failed to implement organizational strategies were typically not necessarily making a mathematical error, but rather that they were, at times, not being careful and deliberate in their work. This is an aspect of listing, and counting more generally, that needs to be investigated further, perhaps by explicitly examining how metacognitive aspects of problem solving affect students’ counting. Again, Schoenfeld (1992) and Carlson and Bloom (2005) highlight the importance of such self-regulatory activity in problem solving, and students’ systematic listing in combinatorial tasks can be framed within these more general problem-solving principles and strategies.

5.3 Listing reinforces sets of outcomes

We finally observe that the results support the notion that a focus on sets of outcomes is important as has been previously proposed (Lockwood, 2013; Lockwood et al., 2015). Lockwood (2014) specifically notes the important outcomes, noting that it should be a priority “to reinforce the relationship between counting processes and sets of outcomes, and to help students integrate the set of outcomes as a fundamental aspect of their combinatorial thinking and activity” (p. 36). Systematic listing helps to accomplish this aim by orienting students with what constitutes a desirable outcome, ensuring that they understand what they are trying to count. Identifying structure and organizational techniques in a list reinforces productive counting processes that can help students generate patterns and avoid overcounting. Even more, our findings provide direct evidence that listing outcomes is one practical, concrete way in which outcomes can be made a focus of the activity of counting. The value of listing demonstrated in this study validates further work on sets of outcomes so that we might better understand ways in which outcomes might productively be used to help students count.

6 Conclusion and implications

Even though we did not test any instructional interventions in this study and instead are reporting on effects of students’ listing activity, we can still discuss some pedagogical implications. Because we have quantitative evidence that engaging in listing activity is correlated with correctly solving counting problems, and because we have qualitative insight into the nature of productive listing, a clear implication is that we should encourage students to list some outcomes as they solve counting problems. Instructors could tell students that there is value in even partially listing some outcomes, and they could design some simple tasks in which students can list and start to see the benefit of listing. A task like the domino problem used in this study highlights the value of listing in two ways—first, it demonstrates that the answer to a counting problem can be found through listing (sometimes more easily than trying to come up with a precise, closed formula), and second, it shows students how listing can help identify and prevent a potential overcounting situation (counting both the [1,2] and [2,1] domino as distinct). Exploring even a handful of such problems before learning and implementing more powerful formulas may orient students toward a productive set-oriented perspective toward counting (Lockwood, 2014). This is one example of a useful task, but our findings suggests that there is a need for more careful investigation of alternative methods of instruction to help students gain experience with systematic listing and, more importantly, to see clearly the benefits of listing. Especially given the fact that in some cases, listing on one problem affected students’ work on other problems, it seems promising to explicitly target whether or not (and if so, how) instructional interventions can be designed to help students appreciate listing. It may be instructive for students to see productive versus unproductive lists in other students’ work in order to reflect on how helpful lists might be generated.

Our aim in this study was to examine whether or not having students systematically list might be a significant factor in their solving of counting problems, thus providing evidence for the value focusing on outcomes in solving counting problems. Students at all levels struggle with correctly solving counting problems, and we have uncovered one particular factor that seems at least positively correlated with successful counting. Our study provides motivation to look more closely at how listing might relate to students’ thinking and learning about counting problems and also to develop and study instruction that might foster such productive listing activity.

Appendix 1

ᅟ The table below offers, for each problem, frequencies of student responses that were correct with listing, correct without listing, incorrect with listing, and incorrect without listing