There have been increasing calls in both the classroom assessment literature (e.g., Gardner 2006; Stobart 2008) and in the mathematics education literature (e.g., National Council of Teachers of Mathematics [NCTM] 1995, 2000a; Wiliam 2007) for teachers to shift their assessment practices. In both literatures, teachers are encouraged to incorporate a range of assessment practices that are responsive to student thinking and that promote learning. Concurrently, teachers are situated in a complex landscape of accountability in which they are often portrayed as technicians tasked with implementing prescribed curriculum, policies, and procedures (Cochran-Smith and Lytle 2009). Moreover, success is often measured by means of externally created, summative assessment tools, which neither facilitate nor measure teacher or student formative development. In the midst of these conflicting messages, teachers have their own beliefs and established classroom practices that inevitably interact with current thinking about mathematics education and assessment (Ball 2003; Wiliam 2007). In this paper, we examine how teachers navigate this difficult terrain through an analysis of teacher discussions within two professional communities of practice. Forty-two teachers of mathematics across two school districts in Ontario, Canada were members of these communities of practice.

Our goals in initiating this project were to support teachers as they engaged in dialogue and inquiry into their practice and the practice of others and to better understand these mathematics teachers’ experiences implementing new assessment ideas. We use the term assessment broadly to refer to the full range of assessment and evaluation practices that these teachers were engaged in including those practices sometimes referred to in the literature as either formative or summative and/or as assessment of, as, and for learning (Earl 2003). Our project provides an account of teachers working collaboratively over a period of two school years to further develop their classroom assessment practice. The purpose of the study was not necessarily to investigate a professional development initiative but rather to acknowledge the value of communities of practice (Carr & Kemmis 1986; Cochran-Smith and Lytle 2009; Kincheloe 2003) and to use the community of practice forum to investigate teachers’ practice and inquiry into assessment.

In this paper, we describe some of the complex assessment dilemmas these teachers were grappling with. We endeavor to better understand these dilemmas by using an analytic framework based on the work of Windschitl (2002). Our analysis provides insights into ways of supporting teachers as they continue to develop their assessment practice. These insights will be of interest to classroom teachers, those working with teachers to support their ongoing development in classroom assessment, researchers of classroom assessment, and policy makers seeking to influence assessment practice.

1 Theoretical perspectives

Current perspectives on classroom assessment that draw on cognitive, constructivist, and sociocultural views of learning have shifted from a view of assessment as an event that objectively measures the acquisition of knowledge toward a view of assessment as a social practice that provides continual information to support student learning (Gipps 1999; Lund 2008; Shepard 2001). Instruction and assessment are viewed as seamless processes that support students’ learning through continuous feedback to teachers and students. Current thinking in classroom assessment also recognizes that learning is multi-dimensional and that complex processes require innovative and varied assessment practices (Brookhart 2003; Delandshere and Petrosky 1998; Gipps 1999). Rather than relying solely on summative end-of-unit, paper-and-pencil tests, students should be afforded multiple opportunities throughout the learning process to demonstrate what they know and can do. Teachers are encouraged to use formative assessment strategies such as observation and questioning throughout the learning process to better understand their students’ learning and decide on subsequent instructional activities (Moss and Brookhart, 2009). In addition, current approaches to classroom assessment emphasize the role of the student in the assessment process, include students in developing and applying assessment criteria, provide opportunities for students to develop peer- and self-assessment skills, and to use feedback to improve their learning (Moss and Brookhart, 2009; Shepard, 2001).

Developing new assessment practices is also essential in implementing inquiry-based approaches to mathematics teaching and learning (Earl 2003; Graue and Smith 1996; NCTM, 1995; Wiliam, 2007). Contemporary views in mathematics education prompt classroom practices that value mathematical inquiry as a way to engage learners in mathematical thinking and deepen their understanding of mathematical concepts (Forman 2003; NCTM 1989, 2000a; Sfard 2003). As such, students are engaged in mathematical investigations, create a variety of ways to represent and connect mathematical ideas, make conjectures, participate in discussions, and make sense of mathematical claims. As teachers encourage students to engage in these mathematical processes, it is evident that traditional mathematics assessment strategies such as paper-and-pencil tests will not suffice (Gearhart and Saxe 2004; Romagnano 2001; Suurtamm 2004; Suurtamm et al. 2010). Teachers need to incorporate a variety of assessment practices in order to assess the range of mathematical activity occurring in their classrooms. The emphasis on problem solving and communicating mathematical thinking prompts teachers to find new ways of assessing these complex tasks. Hence, they are encouraged to develop and implement alternate forms of assessment such as performance tasks, portfolios, conferencing, observation, and student presentations (NCTM 1995, 2000). Furthermore, teachers are encouraged to use formative assessment. They are urged to imbed assessment in instruction, to focus on student thinking as students share problem-solving strategies and solutions, and to provide meaningful feedback on day-to-day work to improve student learning.

The call for teachers to shift their assessment practice to align with these perspectives may pose challenges for teachers. These assessment practices may be unfamiliar to teachers and may test their deeply held notions about both the purpose of assessment and the nature of mathematics teaching and learning (Ball 2003; Black and Wiliam 1998; Shepard 2001). Moreover, current views of effective mathematics education and assessment are set against a backdrop of test-based accountability which places particular demands on teachers and may contradict messages being conveyed both about assessment and about mathematics teaching and learning. In many jurisdictions, student success is measured less by the thoughtful ways that teachers come to understand and extend students’ mathematical understanding and more by the targets students hit on standardized tests (Cochran-Smith and Lytle 2009). As one example, in a three-year study conducted in Chicago, Lipman (2009) found that two of four elementary schools dropped inquiry-based approaches to mathematics in favor of more procedural approaches to problem solving in an effort to raise scores on standardized tests. These observations contributed to Lipman’s conclusion that accountability testing can hinder teachers’ capacity to develop the professional judgement needed to engage in complex pedagogical and curricular practices.

Teachers’ opportunities for dialogue and collaborative work are well recognized as ways of encouraging change in classroom practice and supporting the implementation of new ideas (Cochran-Smith and Lytle 2009: Fullan 2001; Hargreaves 2009; Lachance and Confrey 2003; Webb and Jones 2009). We see the value of collaboration as grounded in work on situated learning and communities of practice that suggest that social practice is the primary, generative source of learning (Lave and Wenger 1991; Wenger 1998). However, in many cases, learning communities have been set up in school districts as part of the rhetoric of school reform and often have an external agenda that may have little meaning for teachers (Hargreaves 2009). In contrast, in a community of practice that is meaningful for teachers, dialogue needs to be practice-based and practitioner-directed (Lee and Shaari 2012). The resulting exploration of practice can adapt to the immediate learning priorities of the teachers, thereby enabling rich insights to emerge. In this project, we sought to provide such a forum wherein teachers could explore their experiences with assessment, engage in dynamic and iterative discussions about topics that are meaningful to their assessment practice, and try new assessment ideas. As researchers, we sought to better understand the action and thinking of these teachers as they implemented new assessment approaches and as they participated in these communities of practice.

2 Context of the study

In Ontario,Footnote 1 recent policy documents and ongoing educational reform efforts reflect current thinking about mathematics teaching and learning and about classroom assessment practice. The provincial mathematics curriculum documents that teachers are required to follow serve essentially the same purpose as standards in the USA. The Ontario curriculum encourages teachers to engage students in a mathematical activity through problem solving and collaborative investigation. These activities help to support students’ procedural and conceptual understanding of mathematical ideas (OME 2005a, 2005b, 2007). The curriculum also includes important messages about classroom assessment that align with current assessment literature. In addition, a recent provincial assessment policy document, Growing Success (OME 2010), establishes principles to guide teachers in their assessment practice. This document discusses the multiple purposes of assessment and provides descriptions of assessment for learning (AfL), assessment as learning (AaL), and assessment of learning (AoL). In addition, Growing Success distinguishes between assessment (i.e., the process of gathering information about how well students are achieving the curriculum expectations) and evaluation which is defined as “the process of judging the quality of student learning on the basis of established criteria” (2010, p.147).Footnote 2 Teachers are encouraged to use a variety of assessment tools and to provide students with multiple opportunities to show what they know and can do. The value of providing students with ongoing feedback and developing students’ self-assessment skills is also stressed. Another important dimension of this document is that it explicitly acknowledges the role of professional judgement in effective assessment practice: “the professional judgement of the teacher, acting within the policies and guidelines established by the ministry and board, is critical in determining the strategy that will most benefit student learning” (p. 46).

In many jurisdictions, prescriptive policies including highly routinized and semi-scripted curricula are put into place to promote consistency from teacher to teacher, school to school, and district to district. Such policies often prompt confusion, encourage comparison, and may discourage innovative teachers from trying out new ideas (Hargreaves 1994, 2009). In his 2011 American Educational Research Association Distinguished Lecture, Luke (2011) highlighted reform efforts in several jurisdictions and observed that Ontario’s education reform has been characterized by moderate levels of prescription regarding assessment and curriculum and by a low to moderate emphasis on large-scale assessment. While Ontario does have province-wide assessments in mathematics, reading and writing in Grades 3 and 6 and in mathematics in Grade 9, these assessments provide information for system-wide accountability purposes and have low-stakes for students. The assessment that has the most direct impact on students is a literacy test, administered in Grade 10, which students are required to successfully complete in order to graduate. Success on the other province-wide assessments is not required for course credit or grade promotion. Luke also notes the extensive investment in teacher development that has taken place in Ontario. He observes that Ontario has “prioritized the expansion of adaptive professional expertise rather than the production of routinized teaching” (p. 374). Consistent with Luke’s observation, Growing Success (OME 2010) recognizes the need for teachers to engage in ongoing learning in order to implement new assessment policies. Teachers are encouraged to set goals for their learning and to work collaboratively with their peers to receive and provide feedback about their assessment practice (OME 2010, p.36). Recent studies demonstrate that Ontario teachers have had varying degrees of support in shifting their assessment practice and that many still wrestle with the implementation of new assessment ideas in their day-to-day practice (Suurtamm and Graves 2007; Suurtamm et al. 2010; Volante and Beckett 2011).

3 Analytic framework

Both the literature we have reviewed and our experiences working with teachers in various contexts suggest that as teachers incorporate new assessment practices and implement inquiry-based approaches to mathematics teaching and learning, they are likely to face multiple and varied challenges. In this study, we frame the challenges that were discussed in the communities of practice as dilemmas of practice. Focusing on these dilemmas of practice enables us to value the complexity of educational change, to better understand the change process, and to suggest ways that teachers can be supported as they further develop their practice (Adler 1998; Windschitl 2002). The analytic framework we used to better understand these teachers’ dilemmas is adapted from one developed by Windschitl (2002). Windschitl used a framework of four types of dilemmas (conceptual, pedagogical, cultural, and political) to put forth a phenomenological perspective on what it means to enact constructivist teaching methodologies. This phenomenological perspective recognizes that enacting constructivist methodologies is not a matter of simply applying instructional strategies but rather must consider the multiple contexts of teaching and the complex web of beliefs and concerns of a variety of stakeholders including teachers, administrators, students, and parents. The enactment of new practices brings forth ambiguities, tensions, and compromises that need to be negotiated in practice. These dilemmas are important aspects of teachers’ intellectual and lived experience and play a key role in day-to-day classroom practice.

We see a strong parallel between Windschitl’s (2002) characterization of the enactment of constructivist teaching methodologies and our observations of the enactment of new approaches to classroom assessment. We found that using an adaptation of Windschitl’s framework suits the situation of teachers introducing new assessment practices in their mathematics classrooms and helps us to more closely examine the types of dilemmas teachers face. We first outline Windschitl’s definitions of the four dilemmas and then present our adaptation of his framework. In the “Method” section, we discuss how we used this adapted framework in the analysis of our data.

Windschitl (2002) presents conceptual dilemmas as arising from teachers’ grappling with the philosophical, psychological, and epistemological underpinnings of constructivism. For instance, teachers may question whether particular disciplinary knowledge can be constructed by students or needs to be explicitly taught. Pedagogical dilemmas arise from attempts to design learning experiences that are based on a constructivist philosophy and to adopt strategies such as managing new kinds of discourse or facilitating students’ collaborative work. Cultural dilemmas emerge from attempts to shift the classroom culture which might include a reorientation of the roles of teachers and students in constructivist learning contexts or managing the various expectations of teachers and students. As an example, teachers may struggle with how to develop a classroom culture where students take responsibility for their own learning. Political dilemmas emerge when constructivist ideas meet stakeholder norms and policies that may appear to conflict with constructivist ideas. For instance, teachers may wonder whether constructivist ways of teaching will adequately prepare students for standardized testing situations. While highlighting each type of dilemma can be very useful, Windschitl points out that dilemmas are inherently complex and may not necessarily fall neatly into one of these four domains. Therefore, the overlap and interconnections between dilemmas must also be examined.

The adaptation of Windschitl’s (2002) framework to address the dilemmas teachers face when implementing new assessment ideas emerged primarily from our previous research with teachers where we were able to see examples of each category. We see conceptual dilemmas in assessment arise as teachers attempt to understand the conceptual underpinnings of current views of assessment and of mathematics teaching and learning. For instance, teachers often are not presented with the rationale for making changes in assessment practice, and they question whether the changes will improve student learning. They grapple with such things as the different purposes of assessment, the role of formative assessment, the value of aligning instruction and assessment, or what it means to understand mathematics. Pedagogical dilemmas arise as teachers create and enact new assessment opportunities. These dilemmas are often connected to how to create assessment tasks, strategies, and tools, and they may occur as teachers design mathematics activities, determine ways of recording, or work to find time for meeting with students and providing feedback. Cultural dilemmas focus on changes in classroom and school culture with regard to assessment practice. Windschitl seems to restrict this category to the cultural changes in the classroom experienced by teachers and students and places other stakeholders in the political dilemma category. In our adaptation, we see cultural dilemmas as those that arise within the broader school culture including administrators, parents, and other stakeholders. We see this broader cultural context as strongly influencing or even co-constructing the classroom culture. Teachers may face dilemmas when their new assessment practices threaten existing cultural practices within a school or department setting or challenge parents’ notions of assessment. Political dilemmas arise when teachers try to align their thinking and practice with provincial, district, and school policies around assessment, particularly with regard to accountability. For instance, teachers may be trying to make sense of a new assessment policy that may conflict with the way they were thinking about assessment. Table 1 summarizes our definitions of the dilemma categories in assessment and offers additional examples to provide more clarity.

Table 1 Definitions and examples of assessment dilemmas
Full size table

4 Method

We initiated this study by facilitating the creation of communities of practice in two Ontario school districts. District A is a mid-sized, urban, and suburban district serving approximately 120,000 students, and District B is a small, largely rural school district serving approximate 5,000 students. The project began through our collaboration with one mathematics leader in each district. Claire, from District A, was a mathematics department head in a secondary school, and Julia, from District B, had previously been a mathematics teacher but was in the role of a district coordinator responsible for student achievement. Claire and Julia sent invitations to elementary and secondary teachers interested in enhancing their classroom assessment practice in mathematics and in meeting with others to discuss their assessment practice. They also assisted us with establishing the communities of practice by helping to plan meetings throughout the project and by participating in the sessions.

At the initial meeting in each district, we addressed questions about the project, obtained participants’ consent, and provided an opportunity for participants to become acquainted with the researchers and with one another. The participants also began to share aspects of their assessment practice. We asked each participant to post an assessment challenge on chart paper and then encouraged them to form small discussion groups based on similar challenges. We also negotiated a tentative schedule of future meetings to occur every 4–8 weeks depending on constraints such as school breaks, examinations, or reporting periods. The initial meeting in each district took place in November 2008, and we continued to meet with these communities of practice until May 2010. Each meeting lasted approximately 2 h.

In District A, we had eight meetings over the 2 years. These were held after school hours. While a total of 16 teachers participated in the District A community of practice, there was generally a core of seven teachers attending most meetings. In District B, a total of 26 teachers participated. The meetings were held during school hours, and the school district paid for coverage of the participating teachers’ classes while they attended the meetings. In the first year, there were three meetings with a total of 12 teachers participating, though attendance varied with approximately 10 teachers at most meetings. During the second year, 14 additional teachers joined bringing the group to a total of 26. As a result, the group was divided into two communities of practice based on geographic location within the district. In that year, each of the District B communities of practice met three times. Thus, over the 2 years, nine meetings were held in District B. In summary, 42 teachers participated in this research, and we gathered data from a total of 17 meetings.

For both districts, in each meeting, we focused on sharing assessment practices and challenges. As the project proceeded, teachers often reported their experiences trying out ideas that had been generated at previous meetings, and they brought samples of performance tasks, recording sheets, or rubrics and discussed how they were using them. As researchers, we also shared assessment strategies that we had used, either in our previous elementary or secondary teaching experience or in our current university teaching. Over time, the teachers appeared comfortable enough to bring drafts of assessment materials to the meeting so that they were able to get feedback from the group. During the second year of the project, each group also used one of two assessment resources depending on the grade focus of the group (National Council of Teachers of Mathematics NCTM. 2000, 2001). Short readings from these resources were used, at times, to stimulate a discussion. These resources were also used as the basis for critically examining the design of an assessment instrument, such as a rubric, without critiquing the work of any individual participant. Thus, while we provided some structures for the meetings by initiating discussion with a question or a brief reading, the focus of each meeting emerged mostly from the teachers.

The meetings were audio recorded, and the assessment materials the teachers shared were gathered. After transcribing the recordings, our first step in the analysis process was to read through each transcript in order to separate it into a series of conversations. Individual conversations became the unit of analysis, and these conversations were considered during the layers of analysis that we conducted. In the first layer, we focused on assessment practices and gathered conversations within assessment practice categories such as observation, portfolios, checklists, or rubrics. This process enabled us to describe the practices these teachers were engaged in. It also reacquainted us with the richness of the conversations that had taken place and suggested some of the assessment issues and dilemmas that would emerge in our second layer of analysis.

For the second layer, we used a process of interaction analysis (Jordan and Henderson 1995) so that we could together make meaning of the conversations and issues that were discussed. We each read the conversations and identified those that brought forth dilemmas for these teachers. For each of these conversations, one researcher would write a short synopsis. Then, using our adaptation of Windschitl’s framework, the researcher would identify the types of dilemma and provide a justification. Throughout this stage of the process, we met to discuss the conversations and the synopses in order to come to a consensus as to meaning and categorization. As we placed discussions within the framework, we noted many conversations that overlapped categories. We further explore this observation in our discussion of the findings.

5 Findings

To present our findings, we draw on our first layer of analysis and provide a summary of the assessment practices these teachers were using. Using our second layer of analysis, we then consider the dilemmas that emerged for these teachers as they incorporated new assessment ideas into their practice. We provide not only examples of conversations for each dilemma type but also show that many of the conversations demonstrate the interconnections between the dilemma types. We also want to reiterate that the goal of the project and of our analysis using Windschitl’s framework was not to evaluate teachers’ assessment practices. Rather, we wanted to gain an understanding of these teachers’ experiences and of the dilemmas that emerged in their practice.

5.1 Teachers’ assessment practices

Our first layer of analysis suggests that the teachers in the study are using a variety of assessment strategies to get a sense of the students’ understanding and to provide feedback to the students and themselves. Their assessments took a variety of forms such as small quizzes, math journals, conferencing, and observation. For instance, one Grade 6 teacher in District B spent some time focusing on encouraging students to explain their solutions and write answers in a way that could be understood by classmates. Our conversations with the teachers revealed to us that the teachers wanted to make the students’ mathematical thinking more apparent to both students and teachers. A Grade 7/8 teacher in District A talked about the ways that she engaged students in a group activity where they were creating algebraic patterns for one another to “break”. She explained that while the students were busy creating and decoding algebraic patterns, she was freed up to listen to students’ thinking. A Grade 9 teacher in District B chose a previously assigned homework question and asked students to rewrite their responses with an emphasis on explaining their thinking. He then provided students with descriptive feedback on their mathematical understanding and the clarity with which they had explained their solution. At times, rather than the teacher providing feedback, he worked with students to help them provide descriptive feedback to one another.

Teachers were also working on ways of designing new assessment strategies and tools such as creating ways to record their observations or developing rubrics where students were involved in determining the criteria so that the rubric is more useful to students. Many discussions centered around creating summative assessment tasks that mirrored the types of problem-solving tasks these teachers incorporated in their classroom practice. These summative tasks often took the form of performance tasks that were spread over several days.

5.2 Dilemmas in assessment practice

In this section, we discuss the dilemmas of practice that emerged from our second layer of analysis. We revisit the four dilemma categories presented in Table 1 and demonstrate how they connect with the data analysis. For each category, we give a brief summary of the dilemma category, describe some specific dilemmas from our data that connect with the category, and provide an example of at least one conversation within each category. Subsequently, we discuss some of the ways that these dilemmas are interconnected.

5.2.1 Conceptual dilemmas in assessment

Conceptual dilemmas in assessment arise as teachers attempt to understand the conceptual underpinnings of inquiry-based mathematics teaching and learning and of current views of assessment. We found that conceptual dilemmas arose quite frequently in these communities of practice. We present one example of a conceptual dilemma conversation centered around current views of assessment followed by an example of a conceptual dilemma conversation related to inquiry-based mathematics teaching and learning that has implications for assessment.

In several conversations in District B, teachers were coming to grips with the notion of assessing students by assigning levels. In Ontario, curriculum documents present a generic assessment rubric called the Achievement Chart that provides descriptors for four levels of achievement (Levels 1–4) across each of four categories of knowledge and skills (OME 2005a, 2005b, 2007). While the Achievement Chart is designed to guide teachers in their assessments of students, teachers in both districts saw the issue of assessing by levels as a challenge. In District B, some of the concerns associated with the Achievement Chart arose at our first meeting. The discussion at this meeting suggested that teachers were creating different questions intended to assess each level of achievement. In other words, they created Level 1 questions, Level 2 questions, and so on. As one teacher indicates;

So, for these questions, I call them ‘Level 3 questions’ and the kids can get up to a three on that stuff. The only way for them to get a Level 4 then is if they choose a Level 4 question from the problems at the end. (Anita, District B November 17, 2008).

There was a further discussion about creating and scoring these “leveled” questions, and it became apparent that several teachers felt that they needed to have questions at each level of achievement in order to assign levels to students. During the next meeting 2 months later, teachers were working in small groups using tasks and samples of student work taken from the province-wide assessment. The teachers were examining the samples of student work and were asked what they thought the assessment criteria for each level might be. The issue of leveled questions came up in a small group discussion as they looked at one question and the student work samples.

Richard: So I bet you that would be a Level 4.

Jim: Yeah. So ultimately it was a Level four question in the first place, right?

Darren: Like a Level four answer you mean?

Richard: A Level four answer.

Jim: Since they’re able to solve it, am I misinterpreting it as a Level 4 question, then?

Richard: No. See, there’s answers that are Level 1, 2, 3, 4, right?

Jim: Yeah.

Richard: The question isn’t really Level 4.

Jim: (Pause) Okay.

Richard: The question can, it can achieve any of the levels.

Jim: (Pause) Okay. You know and this is a good eye opener cause then it makes a little more sense, too, with what we’ve been doing in our program. (District B, January 29, 2009).

Unfortunately, reading this transcript may not do justice to the a-ha moment that Jim had when he realized that it is not the question that is leveled, but the student responses that are assessed in levels. Jim’s realization could certainly be noticed in the recording both by the duration of the pause before he said each “okay” as well as in the tone of his voice. It was a turning point as he realized that a well-designed question can elicit a variety of levels of a student response. The ensuing discussion with the whole group indicated that many of these teachers had collaboratively worked through a conceptual shift about the idea that an effective question can elicit a full range of student solutions.

Amidst conversations about assessment, conceptual dilemmas of what mathematics is and how students learn mathematics often surfaced. Teachers talked about what inquiry learning means to them, the value of students struggling with a problem, and just how much prompting or scaffolding is appropriate. Often, the nature of learning was at the heart of the discussion. For instance, there was an extensive discussion in District A, as one teacher struggled with how to provide questions that “move students from A to B”. This prompted another teacher to question whether learning was linear (moving from A to B) or whether it was more of a “landscape” of concept connections where “things are all interconnected and building on each other” (District A, November 28, 2008). What followed was a rich discussion about what it means to understand mathematical ideas and consequently to assess student thinking.

Evidence of a number of other conceptual dilemmas emerged during the meetings. Discussion, clarification, and negotiation of conceptual dilemmas was thread throughout the conversations and, over time, participants felt comfortable sharing their understandings as well as respectfully challenging different ways of thinking.

5.2.2 Pedagogical dilemmas in assessment

Pedagogical dilemmas in assessment arise as teachers deal with the “how” of enacting current assessment ideas. Our analysis suggests that teachers experience numerous and varied pedagogical dilemmas that include the design of assessment activities, strategies and tools, finding time for ongoing assessment, or finding ways to record observations and conversations with students. For instance, dilemmas surfaced as teachers sought to find time to create opportunities for assessing mathematical inquiry; “I mean I’d love to give them enough thinking opportunities. I’m trying all the time, but I’m not assessing it as often or not assessing enough of it.” (Sharon, District A, May 5, 2010).

There were recurring conversations about finding ways to record student responses and teacher observations during problem-solving activities. In District A, this arose in the first session as seen in the following excerpt:

Jason: This is my number one question ‘Do you need to record that somewhere?’ Number two—‘How should you record that?’ and number three, you know ‘What will we do with it if we had it recorded?’

Barbara: Yeah, yeah, exactly and, I mean, it’s not always going to be something you would use to move forward. It depends on what it is, right.

Jason: Yeah.

Barbara: But, but if we’re using formative discussion or formative assessment, then you do want to use it to figure out where you go next, whether it’s where you go next with that student or with everybody. And I think that’s really an issue. Like how do you keep track of that? (District A, November 28, 2008).

This issue of recording and tracking the information that teachers receive from observing problem-solving situations recurred during several conversations. Teachers brought in different recording sheets that they used so that through sharing their strategies, they could refine this practice. For instance, during the second year of the project, Jason revisited the issue of recording by bringing in a recording sheet one of his colleagues was using that he wanted to share with the group.

Jason: So, I really want to go deeper into this whole idea of the assessment for learning, in terms of having a record of the ongoing conversations we have with kids while they're learning. And so we don’t know if this [the recording sheet] is the right way because it's quite a bit of tracking and detail, but we’re experimenting with it and we’re trying it out. (District A, January 21, 2010).

The group was interested in the way students’ conversations were recorded as it involved a coding system where the teacher could quickly jot down anecdotal notes about their students’ mathematical thinking.

Many other conversations that suggest pedagogical dilemmas took place in our meetings in the two districts. Conversations occurred in both districts about designing rubrics that clearly communicate assessment criteria to students. Teachers also grappled with developing and trusting students’ skills in self and peer assessment. Many teachers focused on working with students to give written feedback to one another and often struggled with ways to help students provide meaningful peer assessment. Teachers brought in samples of strategies and tools they were using; and in some cases, they decided to work together on the design of a tool, try it out in their classroom, and bring back student work and feedback to share with the group.

5.2.3 Cultural dilemmas in assessment

Cultural dilemmas in assessment focus on changes in classroom and school culture with regard to assessment practice. In our study, dilemmas often arose when new assessment practices threatened existing cultural practices. There were often discussions about student expectations with respect to classroom practices, and specifically with respect to marks. Teachers also wrestled with colleagues’ concerns about new approaches to assessment, the role of consistency in assessment practices among department members, and the parents’ and administrator’s understanding of assessment.

Teachers often discussed the challenges they faced in shifting the classroom culture so that students could become comfortable with new classroom practices. One such discussion in District A revolved around the use of concrete materials, or manipulatives, in problem-solving activities. Claire began the discussion by describing a situation where she had Grade 9 students work on a problem-solving activity and had a container of square tiles available at the side of the classroom for students to use. She remarked that while students readily got up from their seats to get a calculator, ruler, or graph paper, none of the students got up to get the tiles. On another occasion she tried something different. She placed a small bag of tiles on each group’s table and observed as the students eagerly used the tiles to help them solve the problem.

There must have been something about the fact that they had to go get them that didn’t, that felt like they might be saying “I can’t do this without the tiles”. Whereas when everybody had them, everybody used them. (Claire, District A, June 9, 2009).

Several others talked about the cultural shift as their students accepted that they could use concrete materials in both instructional and assessment tasks. As Demi explained of her Grade 10 students;

I tried to introduce algebra tiles. And this year we used the algebra tiles to factor and one of the kids said to me “If I use the tiles, does that make me dumb?” And I said “No, we all have different avenues to see things . . . It's, you know, it’s just we all learn differently”. Now that you say that, it would be nice if I could give them each a little kit of the algebra tiles . . . And then the other big surprise for them was “Are you going to let us use this on the test or the exam?” I said “Well, yeah, that’s perfectly fine because you are still showing me your understanding.” So it's a big step for me to let them, you know, show me how they can do it without prescribing one way. (Demi, District A, June 9, 2009).

This quotation demonstrates the classroom cultural shifts that take place for the students and for the teacher. In this conversation, the teachers went on to discuss other assessment opportunities they provide for students to show what they know and can do.

Cultural dilemmas also occurred with interactions with colleagues or administrators. The issue of consistency and colleagues’ expectations was a recurring theme. Many of the participants felt fairly isolated in their efforts to try to find or develop alternate forms of assessment. For instance, in a discussion in District A, teachers were talking of the difficulty of incorporating alternate forms of assessment rather than only using a unit test, particularly when their colleagues were not doing so. Barbara, a secondary mathematics teacher, mentioned that she found it much easier to engage in alternative types of assessment, such as performance tasks or student presentations, when she is teaching a course that is not being taught by other teachers in the department.

I find it easiest in a class that’s very different anyway where I don’t have to worry about being consistent with other teachers or anything else. Like my gifted class, I find it much more easy to be flexible and try different things. (Barbara, District A, November 26, 2009).

In another conversation, Jason questioned what was meant by consistency.

I can think of consistency meaning a couple of things. One, it means “Does an A in my class mean the same as an A in someone else’s class?” Does it mean the different ways in which the final percent grade can be determined? Or does it mean that the assessment is consistent in terms of its ability to actually measure the learning outcome? (Jason, District A, November 28, 2008).

Issues of consistency relate to the cultural dilemmas that teachers face as they work within schools and mathematics departments where other teachers may be using more traditional forms of assessment and the participants are often challenged by colleagues about what they are doing and why. For the sake of consistency across classes, teachers may be discouraged from using new assessment approaches that others may be unwilling to use.

Teachers also talked about the role of parents in changing assessment practices. Many parents are much more familiar with a test at the end of a unit and may lack confidence in new forms of assessment. In one discussion, Brian mentions that parents often have trouble interpreting a rubric that accompanies a task and seeing how it is used to assess student work. He remarks,

We sometimes have to deal with a parent who is questioning a student obtaining one mark versus another. For them, a rubric is not clear enough as to how to distinguish between marks and that's getting back to the fact that now we’re having to create the rubric so clear as to how to mark it that you're giving away exactly how the problem is to be solved. (Brian, District A, October 29, 2009).

This comment led to a discussion of the design of rubrics for problem-solving activities that could provide clarity to both students and parents without being overly prescriptive.

5.2.4 Political dilemmas in assessment

Political dilemmas in assessment arose as teachers’ thinking and practice were juxtaposed with provincial, district, and school assessment policies. Many discussions focused on understanding and interpreting policy about grading and reporting, such as matching assessment levels used on rubrics with required report card percentage grades. For instance, during a discussion of reporting with elementary and middle school teachers in District B, teachers talked about particular ways that they are instructed to make comments on a report card. Rather than being able to write their own comments to accompany a mark, they must choose from an established list of report card comments. As one Grade 6 teacher comments;

The way I would like to write my report cards and the way I’m expected to write my report cards are two very different things. I often feel really bad signing my name to something when I don’t agree with the way it is written. You know, it’s written with Ministry [of Education] words that I would never say if I was speaking face-to-face with a parent. Half of the parents don’t understand it. When they see you know, ‘with considerable effectiveness’ eighteen times throughout the report card, they just tune it out. They don’t even read that. . . I’d like to use language they would understand but we’re told that we can’t do that (Melanie, District B, April 27, 2010).

The conversation continued with others commenting on the value of using student-friendly language in rubrics in class and parent-friendly language in reporting. Many teachers felt constrained by the feeling that they had to use language consistent with that used by the Ministry of Education.

There were many discussions in both districts over the course of the 2 years about the “Achievement Chart,” which (as previously noted) is found in all Ontario curriculum documents and is intended as a tool to guide the assessment of student work. The Achievement Chart is divided into four categories of knowledge and skills, and for each of these categories, descriptors for four levels of achievement are provided. The four categories of knowledge and skills are the following: knowledge and understanding, thinking, communication, and application. In some schools, particularly at the secondary level, teachers are expected to organize their marks into these four categories. At the very least, the expectation is that teachers will provide assessments that cover these four categories. From the very first meetings in each district, teachers claimed that they were expected to identify assessment questions and/or tasks to match each of the categories. They saw this as a challenge as they felt they needed several tasks or questions for each category, and they also saw that many of the questions or tasks that they designed spanned more than one category. One conversation in District A began with Bob suggesting

It’s [the Achievement Chart] pretty much a straightjacket, I think, and it doesn’t make any sense. I mean, people are spending so much time trying to figure out ‘Is this an application?’ or ‘Is this a knowledge item?’. There’s no way you can ever decide whether something is one or the other. (Bob, District A, November 28, 2008).

Bob sees no real reason to separate assessment and marks into these categories, and he feels that he has been incorporating and interconnecting these categories in his work for quite some time. He finds the policy directing him to separate them to be counterproductive. However, there was some disagreement among the teachers in this conversation. One speaker stated that she finds it helpful to differentiate where students are doing well and where they need assistance, and the categories help her with this differentiation. Another teacher claimed that this policy encourages teachers to consider incorporating more than just knowledge into the teaching and learning of mathematics. There was also a discussion about what the categories mean, particularly the thinking category. The participants went on to discuss the definition of thinking, ways to assess thinking, and what formative assessment of thinking might look like.

Teachers also struggled with the role of province-wide accountability assessments in their classroom practice. While the teachers involved in the study appeared to have progressive ideas and practices with respect to assessment, many shared their practice of preparing students for province-wide assessments by practicing multiple-choice questions. In addition, while all of the teachers felt that they had a good sense of what their students knew and could do, many felt that the results of the provincial assessment were the measures valued by parents, school administrators, and the general public.

5.3 Interconnections among types of dilemmas

Like Windschitl, we do not see the categories of dilemmas as mutually exclusive and have several examples that show the interconnectedness of these categories. For instance, several conversations in District A focused on the alignment of collaborative learning situations and assessment practices. One such conversation about collaborative assessment appears to have participants grappling with pedagogical, conceptual, and political dilemmas. The conversation began with Claire indicating that she finds it difficult for students to work on their own in an assessment situation since they typically work together on tasks in instructional situations. She wonders whether they could work collaboratively on an assessment task but struggles with how she would assess individual students if the solution to a problem was a group effort. She also believes that the assessment messages in the policy documents require that summative assessments evaluate individual student achievement only. Others in the group discussed how separating students for an assessment task seems artificial and counter to how the students are used to working. Some of the participants shared the ways that they incorporate some group work in their assessment. Nancy, a secondary mathematics teacher, explains;

And so, when the students are working really well together, and that’s how they’re learning, is it fair for a summative assessment to force them to generate their own independent product? And we all kind of thought that we do things, when we do performance tasks, that allow the students to work together, like giving them an opportunity to brainstorm [collaboratively] and then go off and create their own individual product. (Nancy, District A, November 28, 2008).

Barbara followed Nancy’s comment by describing how her use of student collaboration in an assessment has evolved.

Something that’s been happening to me just maybe in the last 2 years . . . where there’s a real challenging problem-solving activity, where they [students] have to sit down and figure out how to solve it, and it incorporates maybe more than just one unit. I start them off in pairs usually, and what happens is that I’m always going to separate them, right? It’s always you start together in pairs for the first 10, 15 min, and then you’ll do the rest by yourselves because you’ll consolidate some ideas. But what happens is, when the 10 or 15 min is up, they’re still working so deeply, . . . and they’re getting excited because one kid will say ‘Well, okay, but I’m stuck’ and the other kid will say ‘I couldn’t get to there, but now I know what to do’. And they’re so excited about what’s going on that I often end up not separating them, just because, you know what? ‘You’re working so well together, just go ahead and finish that way’. And part of me says ‘Well, should it have been independent?’ And then I figure so many other summative assessments, my tests are definitely independent, so maybe it’s okay in problem solving or in thinking if they end up doing it together. (Barbara, District A, November 28, 2008).

Nancy added that she often interviews individual students about their solution. She explains,

In spoken interviews afterwards, I can determine the extent of understanding of each of the people. It’s a probing interview, ‘So, tell me about what you said to be a slope there and, and what about this and what if that happened?’ That kind of talk, that kind of interview is the most insightful. (Nancy, District A, November 28, 2008).

We see a variety of types of dilemmas here. There are pedagogical dilemmas about how one arranges students who are working on an assessment task. There are conceptual dilemmas as participants’ grapple with the idea that if instructional tasks are done collaboratively, then assessment tasks should reflect that collaboration. This challenges the teachers’ thinking about the alignment of curriculum and assessment. For some, like Claire, collaborative assessment raises questions about feeling comfortable with what an individual student really knows. This tends to bring the discussion back to pedagogical issues as participants offer ways of getting at individual student’s understanding. In addition, the conversation is framed within a political context where teachers interpret policy documents as stating that formative assessment can include collaborative tasks, but summative assessment and evaluation must be based on individual student work.

This is just one of the many examples of conversations where different categories of dilemmas are interconnected. In fact, it is difficult to label each conversation with just one dilemma category. In the examples we provided within our description of each category, the reader probably started seeing other categories of dilemmas. For instance, in the discussion of consistency in the section on cultural dilemmas, Jason’s discussion also brings forth conceptual dilemmas about the meaning of consistency. And, in the section on political dilemmas, the discussion of the Achievement Chart categories is also framed by conceptual dilemmas, particularly around the concern over “what is thinking?.”

6 Discussion

This study provides many valuable insights into teachers’ thinking and actions as they implement new ideas in assessment. In our discussion, we focus on three areas of insight that emerge from this research: the value of using our adaptation of Windschitl’s framework to better understand dilemmas in assessment practice, the role of coherence in messages about classroom assessment, and the potential for communities of practice to enable teachers to further develop their assessment practice and expand their capacity for professional judgement.

6.1 Insights about the dilemmas’ framework

We find that the adapted framework helps us to better understand the complex process of changing assessment practice. Being able to parse out conversations into categories helps to unravel the struggles that teachers grapple with and focuses attention on particular types of dilemmas. Developing an in-depth understanding of the conceptual, pedagogical, cultural, and political dimensions of the dilemmas that teachers face is the first step in better understanding the complexity of changing assessment practice. For instance, the pedagogical dilemmas that emerge as teachers work to develop effective assessment strategies and tools are qualitatively different than the political dilemmas teachers face as they implement assessment policy. Acknowledging and understanding each type of dilemma is particularly valuable in helping us to recognize that teachers’ work needs to be supported in different ways and also in determining what those supports might look like. Pedagogical dilemmas that deal with access to resources or the specifics of assessment practice may be able to be addressed through workshops or enhanced support for sharing resources. Cultural dilemmas, such as dealing with the differing views of assessment held by students, parents, and teachers, may be addressed through administrative support as well as ongoing forms of communication that make classroom assessment practice more transparent to everyone. Recognizing the different types of dilemmas and that each needs to be supported in a variety of ways is essential for those who design policy, professional development, and other supports for the implementation of new assessment ideas.

Moreover, while our observations are based on the experiences of a small group of teachers in two Ontario districts, the categories in the framework seem to reflect observations made in other studies of teachers’ assessment practice. For instance, Webb and Jones (2009) demonstrate that changing assessment practice to genuinely support student learning constitutes a major change in a classroom culture. Their observations of the challenges faced by six primary teachers and their classes in the UK are quite similar to the cultural dilemmas we observed. Earl et al. (2011), drawing on a number of studies of assessment practice in Canada, observe that teachers may begin to change their assessment practice by trying out new tools and techniques, but a more fundamental shift in thinking and beliefs about assessment is needed to enable teachers to make adaptive assessment decisions in their classrooms. They highlight the need for teachers to move beyond a surface-level understanding of an assessment that supports student learning to engage in a deeper understanding of these concepts. The discussion by Earl et al. (2011) closely parallels the distinctions between pedagogical and conceptual dilemmas in our study. Being able to identify the different dimensions of dilemmas using the adapted framework helps to identify aspects of the change process and points to the multiple and diverse ways that change must be supported.

Along with looking at the dilemma types individually, the framework also demonstrates that high-quality assessment lies at the intersection of political, cultural, conceptual, and pedagogical issues. Our data show that many conversations include more than one dilemma category. While distilling the experiences into the four categories emphasizes the different layers of concern that teachers face, considering the interconnectedness of these layers foregrounds the complexity of implementing and supporting new practices in classroom contexts. In other words, professional development that provides new assessment strategies and tools, perhaps addressing pedagogical dilemmas, will not suffice as issues of classroom, department, school, and community cultures may also need to be addressed. Likewise, grappling with teachers’ conceptual dilemmas by developing their understanding of a sound assessment practice may clash with district, state or provincial, or federal assessment policies regarding a large-scale accountability. These sorts of clashes need to be acknowledged and addressed. Furthermore, policy makers who mandate new assessment practices need to be aware of the implications politically, conceptually, pedagogically, and culturally and recognize that a mandate alone will not suffice.

6.2 The role of coherence

Perhaps, coming most clearly from our analysis of the interconnections between dilemmas, this framework demonstrates the pivotal role of coherence in supporting the development of new assessment practices. Our observations underscore the importance of coherence in the messages teachers receive from various sources including current thinking in assessment, policy documents, professional development activities, existing cultural practices, and available teacher resources. Similar messages within and between curriculum documents and assessment policies and across the policies that are established at the school, district, and provincial levels are one key dimension of coherence. At the same time, policy documents need to echo current ways of thinking about assessment and to reflect the views of assessment that teachers are encountering in research-based professional development initiatives and publications. Coherence is further enhanced when similar messages are given to administrators, teachers, parents, and students with the goal of developing everyone’s conceptual understanding of assessment. The importance of a coherent assessment system is emphasized in NCTM’s Assessment Standards document (1995). The Coherence Standard is one of six mathematics assessment standards, and it emphasizes alignment between curriculum, instruction, and assessment. Furthermore, it suggests that assessment should not produce conflicting messages.

In this research project, we found that coherence was an essential support for the participating teachers as they worked to develop their assessment practice. For the most part, the teachers in this study were receiving similar messages from the Ontario curriculum, assessment policy documents such as Growing SuccessFootnote 3 (OME 2010), various teacher resources they were using, and in professional development opportunities. As a result, many of the things they were doing collaboratively in their departments or schools were aligned with current thinking about assessment and facilitated the development of these teachers’ professional judgement with regard to assessment. We also saw that in those instances where a policy was misunderstood or was mandated without providing teachers with an opportunity to understand the meaning or rationale of the policy, the ambiguity generated dilemmas for the teachers. Our observations demonstrate how coherence supports teachers in further developing their assessment practice and suggest that a lack of coherence is likely to exacerbate the dilemmas teachers experience as they implement new assessment strategies. We are encouraged that the recently released Growing Success policy document is consistent with current thinking about assessment and brings greater coherence across curriculum and policy documents in Ontario. Furthermore, it acknowledges the critical role of the teachers’ professional judgement in classroom assessment practice.

6.3 The value of communities of practice

Many researchers have shown that developing a deep conceptual understanding of assessment that supports student learning requires ongoing professional development (e.g., Black and Wiliam 1998; Earl et al. 2011; Webb and Jones 2009). Changing assessment practice is more than a matter of implementing new tools and techniques or of transporting an intervention or best practice from one jurisdiction into a new setting. We take the position that teachers develop new ideas through their own stance of inquiry (Cochran-Smith and Lytle 2009) by exploring issues within their own context and wrestling with these issues with others. Valuing and respecting teachers’ professionalism, commitment, knowledge, and judgment is critical in supporting such inquiry (Hargreaves and Fullan 2012; Lipman 2009). A community of practice approach has many characteristics that support the development of teachers’ professional judgement. Here, we focus on two particular ways that we saw teachers supported through their work in communities of practice: sustained interaction and a practice-based, teacher-directed dialogue.

The sustained collaboration over a 2-year time period that took place in these communities of practice provided many benefits. Attending several meetings over time gave participants the chance to develop trust and to become comfortable sharing their assessment practices The sharing of assessment strategies not only provided rich data, but the participants also found that through discussion, they learned new ways of assessing students and challenged their own thinking about what assessment means and could look like. In addition, participants had time to try out ideas in their own classroom, bring their experiences back to the community of practice, and collaboratively work to refine their assessment tools and strategies. We saw this happen a number of times. Further, sustained collaboration makes it possible for participants to develop understanding beyond a surface level and begin to embrace the deep conceptual ideas that are included in new approaches to assessment. At the outset of the project, we found that conceptual understanding of assessment was varied for the teachers in these communities of practice. We gained insights into how this conceptual understanding evolves through teachers engaging in dialogue, questioning one another, and having the chance to revisit earlier conversations after time to reflect on the ideas. We reported the example of the ‘a-ha moment’ with regard to the four levels applying to a students’ response rather than to the question itself. This conceptual learning took place across several meetings, and we saw many other examples of this process during our study.

Another characteristic of these communities of practice that supported teacher development is that they were practice-based and practitioner-directed. While we sometimes prompted discussion with a question or a brief reading, the topics discussed related to the immediate learning priorities of the teachers. Not only was each topic teacher-directed, but also the conversation that ensued was facilitated in a non-evaluative manner. During the meetings, we refrained from judging teacher’s comments or evaluating their current understanding of either classroom assessment practices or mathematics teaching and learning. We also actively worked to ensure that all of the teachers could have a voice. Adopting this non-evaluative stance and encouraging teachers to share their views seemed to enable the teachers in these communities of practice to come to new understandings themselves. In essence, the communities of practice provided rich opportunities for each teacher to continue to develop their professional judgment. We suspect that such an approach may result in more meaningful changes in teachers’ classroom assessment practice than would result from an approach where assessment experts transmit theoretical ideas to teachers. As Michelle mentioned at our final meeting, the dialogue we engaged in as we came together seemed to be key:

What we have is respectful dialogue. There is not always agreement, although there is a lot, but the disagreements are always to further delve into issues as opposed to, you know, coming to a place of agreement. I just enjoyed the dialogue, I really, really have. (Michelle, District A, May 10, 2010).

We see communities of practice as an ideal forum for moving beyond tools and techniques, for developing deep conceptual understandings, for working through dilemmas in assessment practice, and ultimately for enhancing the teachers’ capacity for professional judgment. This sort of sustained, practice-based, teacher-led initiative can be very effective in moving teachers beyond a surface level of change in their assessment practice.

7 Concluding comments

Our work has implications for both research and practice. We provide a rich description of teachers engaged in a change process concerning assessment. Many of the areas that teachers touched on are rich sources for further research, such as issues of consistency, or assessment of collaborative inquiry. Issues such as these require further understanding, and additional research would be beneficial. Additionally, we suspect that the dilemmas’ framework may be useful to other researchers when examining new initiatives in teacher practice. Identifying dilemmas, determining how the dilemmas interact, and examining how teachers are supported in negotiating dilemmas are important components to implementing new initiatives. We found that Windschitl’s (2002) framework helped us to examine the dilemmas in more depth and to recognize the variety of ways that the implementation of new ideas needs to be supported. Our work also provides an example of effective research methods that focus on teacher practice. The community of practice provided a forum for sustained professional development and dialogue as well as a rich research setting. While we were able to identify certain components that facilitated this, such as connections to practice and a non-evaluative stance, each of these components could be the subject for more detailed research.

In terms of implications for practice, the insights gained through our analysis are of interest to teachers, those working with teachers, and policy makers seeking to influence assessment practice. We have had opportunities to share our work using this framework with groups of these stakeholders. In a 3-day mathematics education leadership conference, 300 participants were exposed to the dilemmas’ framework. We engaged them in a discussion about examples of types of dilemmas and ways that they could be supported. The majority of the conference audience was mathematics resource teachers, mathematics teaching coaches, school district consultants, or those responsible for mathematics teacher education. Feedback from these discussions indicated that the framework gave them ways of considering the complex task of professional development around assessment. Relating to the framework helped them realize that in their work with teachers, they need to consider a multi-pronged approach to deal with the political, cultural, conceptual, and pedagogical aspects of implementing new ideas. As the 3-day conference was concluding, we could hear the categories of Windschitl’s framework as part of the participants’ vocabulary. The ease with which they adopted these ideas told us that our work was also useful for their work with teachers.

In this paper, we have described some of the complex assessment dilemmas that teachers face as they implement new assessment practices. While some might think that dilemmas are problematic and need to be solved, we saw discussions of these dilemmas as generative. The intense and sustained work that our teacher participants engaged in as they wrestled with important assessment issues helped to break down the isolation that teachers often feel. The discussions of dilemmas allowed teachers to share and develop new practices, clarify their thinking, and find new approaches to negotiating their dilemmas. We saw how the iterative dynamic of moving between meetings, classrooms (for the teachers), and the review of the data (for us) created a synergy and enabled a deeper understanding of assessment issues for all of us. This work emphasizes the value of engaging teachers in communities of practice and recognizing the important role that teachers’ professional judgment plays in the assessment process.