Online Curriculum Modules for Preparing Teachers to Teach Statistics: Design, Implementation, and Results

Abstract

The Enhancing Statistics Teacher Education through E-Modules (ESTEEM) project created online curriculum materials to support preservice secondary mathematics teachers who are learning to teach statistics. The materials are based on key design principles drawn from literature in online learning, statistics education, and mathematics teacher education. The materials include many opportunities to engage in data investigations using the Common Online Data Analysis Platform (CODAP). The modules were innovatively designed to provide flexibility for use in a variety of course contexts within different learning management systems (LMSs), and mathematics teacher educators (MTEs) can adapt materials to meet the needs of their students. The ESTEEM modules were implemented at 27 unique institutions across the United States, and data was collected from MTEs and their students, primarily preservice teachers. The MTEs and teachers reported materials were easy to access and navigate online, the materials helped prepare them to teach statistics, and the videos in the materials helped to understand issues that arise in the teaching of statistics. MTEs reported that the ESTEEM modules provided a critical emphasis on key ideas in teaching statistics and contributed to their own professional growth. Teachers reported they improved their confidence in teaching statistics, learned how to utilize technology when teaching statistics, engage in the statistical investigation cycle, understand students’ statistical thinking, and value a framework to support students’ approaches to statistical investigations. Although our materials were designed for online asynchronous instruction, MTEs implemented them in face-to-face settings, synchronous online sessions, and asynchronously. One of the major impacts of our project is the MTEs’ appreciation for the embedded nature of materials within their LMSs and how easily they could adapt and supplement the materials to meet their course needs. However, the paper also provides lessons learned and implications for providing materials through LMS integration and modularized materials for preparing teachers for teaching statistics.

Online teacher education and preparing teachers to teach statistics are two areas of growth needed in mathematics teacher education. Undergraduate teacher education programs traditionally include mostly face-to-face courses, and many teacher educators have reported a lack of preparation and readiness to use online modalities in their instruction (Downing and Dyment 2013; Holmes and Prieto-Rodriguez 2018). Recently though, some have used flipped instruction, hybrid courses, and synchronous and asynchronous online courses in mathematics teacher education (e.g., Harrison et al. 2018; Hjalmarson 2015; Starling and Lee 2015), and during the COVID-19 pandemic, most mathematics teacher educators (MTEs) had to quickly convert courses into online or hybrid formats.

While statistics and data science now play a larger role in secondary school curricula, many in-service and preservice teachers are inadequately prepared to teach statistics (e.g., Burrill and Biehler 2011; Lovett and Lee 2017). MTEs, often lacking experience and expertise in statistics education and the use of technology for investigating data, need access to teacher education materials focused on preparing teachers for teaching statistics (Franklin et al. 2015), along with networking and support in implementing such materials.

In this chapter, we share the evolution and impact of a curriculum project designed to fill a gap in teacher education materials focusing on statistics education and to innovate an approach to assist MTEs in utilizing online instructional methods. We structure the chapter around: (1) design of online modules for statistics teacher education, (2) implementation of modules, and (3) results of research on the effectiveness of the modules.

4.1 Principles to Guide Design of Online Modules

The Enhancing Statistics Teacher Education through E-Modules (ESTEEM) project, funded by the National Science Foundation (DUE 1625713), began in 2016 to develop online modules designed to support prospective secondary mathematics teachers to learn to teach statistics. Mathematics teacher preparation programs vary widely, and statistical content and pedagogy may be introduced in a number of different courses, such as a general mathematics methods course, a course that focuses on teaching and learning statistics, a statistics content course, or courses focused on technology for teaching mathematics. Course modalities also vary greatly across programs, and there is an increased need for resources that support online learning as a result of the COVID-19 pandemic. Consequently, MTEs desire curriculum materials that can easily be embedded into diverse programs and courses and that are easily adaptable to meet the needs of their students. Learning management systems (LMSs) are commonly used by institutions and provide accessibility for multimedia materials such as readings, videos, images, slide decks, and interactivity options such as quizzes with feedback, discussion boards and synchronous communications such as video-based learning environments (e.g., Blackboard Collaborate, ZOOM) and live chat (McBrien et al. 2009; Park 2015). Three fundamental design principles used in ESTEEM are to provide MTEs with online instructional materials that are:

• 1. Modular and adaptable

• 2. Easily accessible and integrated into LMSs

• 3. Interactive in asynchronous or synchronous modalities

The ESTEEM materials’ modular approach and its creation of e-modules for import into LMSs meet MTEs’ expectations of adaptability and accessibility and can promote interactivity online. As the ESTEEM modules were designed, we recognized that MTEs needed to easily access and share the modules with their students. The ESTEEM modules are designed within the three most commonly used LMSs in the United States - Moodle, Blackboard, and Canvas- as well as a common cartridge package that can be imported into other LMSs. A common cartridge is essentially a format for exchanging content between LMSs, so these systems have a way to interpret the digital learning content and how it is organized (see http://www.imsglobal.org/activity/common-cartridge). The ESTEEM modules, packaged for portability, can be downloaded from a web portal upon registration (https://go.ncsu.edu/esteem). Sharing modules through LMSs gives greater autonomy to MTEs by allowing them to add, modify, or delete content to meet the needs of their local learning contexts. Our materials are shared under a Creative Commons Attribution Non-commercial Share-alike license. We believe that having the flexibility to modify content was important for teacher education since statistics content and pedagogy are commonly addressed in different courses within secondary teacher preparation programs.

Several additional design principles based on research from statistics education, mathematics teacher education, and online teaching and learning guided the development of the modules (Hudson et al. 2018). We highlight each of these below.

4.1.1 Use of a Free Web-Based Tool for Learning with Data

We intentionally chose to use the Common Online Data Analysis Platform (CODAP, https://codap.concord.org) as the primary data analysis tool utilized in the ESTEEM materials because it promotes exploratory data analysis, is based on research regarding how students learn with data, and is free and accessible (Finzer and Damelin 2018; Mojica et al. 2019). Ease of use, no cost, and accessibility via a web browser are also of paramount importance for use in K-12 settings. Mathematics teachers may use tools they learned about during their teacher preparation but strongly prefer tools that are free and accessible (McCulloch et al. 2018). CODAP also has strong visualization capabilities and allows users to dynamically link multiple representations and explore relationships among variables.

4.1.2 Teachers Learn by Doing Data Investigations

Mathematics teacher preparation often emphasizes the importance of developing specialized knowledge for teaching that includes a deeper understanding of mathematics and statistics content (e.g., Groth 2013; Hill et al. 2008). This specialized knowledge can be developed through teachers’ engagement with mathematics and statistics tasks as learners themselves. Through such experiences, particularly with technology, teachers have opportunities to revisit concepts they had opportunities to learn in prior experiences in K-12 or college and deepen their understandings in ways that can build specialized knowledge useful for teaching (e.g., Lee and Hollebrands 2011; Wilson et al. 2011). Lee et al. (2014) and Pulis and Lee (2015) have shown that teachers’ use of dynamic statistics technology tools to investigate multivariate data enhances their statistical problem-solving skills. We follow recent suggestions (Franklin et al. 2015; Gould and Cetinkaya–Rundel 2014; Hayden 2015) for teachers to have multiple opportunities to investigate data sets that are large, multivariate, and from real sources. Figure 4.1 illustrates CODAP’s visualization and linking capabilities using data about roller coasters.

Fig. 4.1

Exploring multivariate data about roller coasters using CODAP

4.1.3 Use Frameworks Common in Mathematics Teacher Education

Since the ESTEEM materials prepare mathematics teachers to teach statistics, we felt it was important to incorporate frameworks commonly used in mathematics teacher education. One such framework is the professional noticing of children’s mathematical thinking (Jacobs et al. 2010). Professional noticing consists of three related skills: (a) attending to a student’s strategies as they reason mathematically or statistically, (b) interpreting the student’s strategies, and (c) deciding how to respond to the student based on the student’s understanding. The ESTEEM materials aim to develop in teachers all three skills of professional noticing for the specific content area of statistics—an area that has not received much attention in the work on professional noticing in mathematics teacher education to this point.

Another framework we drew upon when developing the ESTEEM materials is the Five Practices model for productive classroom discourse centered around engagement in meaningful tasks (Smith and Stein 2011). The five practices emphasized in this model are anticipating students’ responses to a task, monitoring students’ responses to a task, selecting specific students to present mathematical ideas, sequencing students’ responses that will be publicly displayed, and connecting between student responses as well as to key ideas. The groundwork for implementing this model also involves setting instructional goals and selecting an appropriate task. The ESTEEM materials attend to all of these aspects of using statistical investigation tasks with additional attention given to characteristics of an effective statistical task launch.

4.1.4 Incorporate Representations of Practice

Another design principle of the ESTEEM materials was to incorporate representations of the practice of teaching and learning statistics throughout our materials. These allow teachers to have a shared common experience of viewing statistical instruction, critically analyze teachers’ practice (e.g., Seago and Mumme 2002; Sowder 2007), examine students’ use of technology to support data investigations (Wilson et al. 2011), and develop professional noticing skills. For example, we include videos of secondary classrooms and individual student talk-alouds where teaching and learning of statistics are occurring. When real classroom video is not available, new capabilities in the creation of animated videos, such as complete stop-motion animations, can create engaging learning opportunities for teachers (e.g., Herbst and Chazan 2020; Herbst and Kosko 2014; Laaser and Toloza 2017). Thus, we also captured and used videos from real classrooms to produce several animated videos depicting teachers and students engaged in statistics tasks.

4.1.5 Promote Learning through Multiple Perspectives

Our team of authors bring a wealth of experience in teaching and research in statistics education to inform the design of our materials. We ensure that the brief readings and videos represent syntheses of literature and guidelines from professional organizations (e.g., American Statistical Association, National Council of Teachers of Mathematics) from a variety of perspectives and that all works are cited. The use of representations of practice that include teachers and students bring those perspectives and voices into learning opportunities for teachers using ESTEEM materials. We also take advantage of the multimedia format of our materials to offer video-based conversations with teachers and experts in panel-like discussions where teachers can listen to and consider the perspectives of those with experience in teaching statistics and designing curricula and software tools. In a study examining shifts in teachers’ perspectives and practices in teaching statistics, Lee et al. (2020) identified expert panel videos as a critical learning experience for reflection and change about one’s own perspectives on teaching statistics. Lastly, the ESTEEM modules are designed to include repeated opportunities for teachers to learn from each others’ perspectives and participate within a student-driven virtual learning environment, including interacting with one another through online discussions (Park 2015; Revere and Kovach 2011).

4.2 Online Modules for Preparing to Teach Statistics

4.2.1 Overview of Learning Goals and Opportunities

The ESTEEM materials consist of three interconnected modules and two independent assignments, as shown in Fig. 4.2. Collectively, the modules aim to develop teachers’ critical understanding about the differences between mathematics and statistics and key statistical concepts, abilities to use CODAP to investigate real-world phenomena with bigger data, professional noticing of students’ thinking about statistics, abilities to make and evaluate data-based claims and arguments themselves and how to navigate data-based discourse in classrooms, and competencies in planning for and leading students in data investigations.

Fig. 4.2

Structure of ESTEEM online modules and assignments

The Foundation in Statistics Teaching Module (Module 1) emphasizes the differences between mathematics and statistics as well as how to support students in learning to reason statistically. It includes activities concerning launching statistical tasks, the statistical investigation cycle, and fostering discussions around students’ statistical thinking utilizing the Five Practices model (Smith and Stein 2011). Teachers are also introduced to CODAP in this module, using it themselves to analyze data concerning roller coasters as well as analyzing videos of classroom lessons where students are using CODAP. Module 1 provides the minimal learning experiences that teachers need to develop key understandings and strategies for supporting students’ engagement in statistical investigations. This is a purposeful design of the ESTEEM materials so that if MTEs only have about 2–3 weeks in a course to devote to preparing teachers to teach statistics, this module will provide a minimal foundation. Module 1 is also a prerequisite to the other two modules, the Teaching Inferential Reasoning Module (Module A) and the Teaching Statistical Association Module (Module B).

Module A focuses on how questions, modeling processes, simulation tools, and tasks can support students to reason inferentially. Teachers use CODAP’s Sampler plugin to investigate inferential questions, make claims based on simulated data, and develop an understanding of sampling distributions. Module B focuses on preparing teachers to teach statistical association, including both quantitative and categorical association. The module shares typical conceptions students have when considering whether variables are associated and develops teachers’ professional noticing skills regarding students’ thinking about the association. Teachers extend their use of CODAP by learning to add attributes to a data set, create two-way binned plots and segmented bar graphs, model data with a least-squares line, and create and analyze residual plots.

4.2.2 Organization of Modules

Each module has a common organizational structure. A module is split into two parts, and each part is expected to correspond to about 5–8 h of learning opportunities for teachers. Each activity in the ESTEEM materials is assigned a three-digit code which designates the module, part, and activity. For example, activity A.1.b signifies that the activity is in Module A, Part 1, Activity b. Although activities are organized sequentially within each part, teacher educators have the flexibility to reorganize and rename activities within their LMS for their course. A table of contents of all ESTEEM activities is provided in Appendix 1.

Each part of a module is divided into three sections: Read and Watch, Engage with Data, and Synthesize and Apply. The Read and Watch materials are further separated into two types of content: Essential Materials and Learn from Practice. Read and Watch: Essential Materials include readings and videos that introduce fundamental concepts about the teaching and learning of statistics (e.g., brief reading about differences between mathematics and statistics, reading and video introducing key aspects of inferential reasoning). The Read and Watch: Learn from Practice materials consist of documentation of the teaching and learning of statistics, such as videos of secondary classrooms engaging in statistical investigations (example in Fig. 4.3), an animation video that demonstrates common student approaches to placing an informal line of best fit (as shown in Fig. 4.4), videos of students creating graphs to display bivariate categorical data, and videos of educators discussing tasks they use to teach inferential reasoning.

Fig. 4.3

Video lesson on sampling distributions involving multiple technologies (Activity 1.2.f)

Fig. 4.4

Animation video of a class investigating the placement of an informal line of best fit (Activity B.2.e)

The second section of each module part involves Engage with Data activities, which are statistics investigations teachers complete using CODAP. These activities include multimedia components (videos, images) to contextualize the investigation, a CODAP file which includes the data to be analyzed, and handouts in PDF and Word versions that can be downloaded and used directly by teachers. Teachers investigate a variety of statistical questions in these activities, including questions about roller coasters, healthiness of granola bars, carbon emissions (see Appendix 2), and fuel efficiency of vehicles, and engage in model-building and simulating outcomes from contexts such as dice and predicting soccer wins (see Fig. 4.5). Some Engage with Data activities also focus on pedagogical and technological aspects of data investigations.

Fig. 4.5

Sample model and simulated data in CODAP (Activity A.2.e)

Synthesize and Apply activities are the final section of each part of a module. These activities ask teachers to reflect on and apply what they have learned to the practice of teaching. Often, these activities explicitly use the aforementioned frameworks commonly used in mathematics teacher education of professional noticing (Jacobs et al. 2010) and the Five Practices model for productive classroom discourse (Smith and Stein 2011). For example, in Activity 1.2.i., teachers read a summary of the Five Practices model and then apply their understanding of the model to analyze a video of a teacher orchestrating a whole class discussion using students’ work in CODAP (see Fig. 4.6). Other Synthesize and Apply activities expect teachers to participate in a discussion board to analyze and discuss characteristics of different statistical tasks, design the launch for a statistical investigation, and craft responses to students whose thinking has been shared in videos, all of which align with the noted frameworks.

Fig. 4.6

Focus on orchestrating statistical discourse using students’ work in CODAP (Activity 1.2.i)

4.2.3 Assignments to Demonstrate Competency

In addition to the three modules, the ESTEEM materials also include two assignments for teachers to demonstrate their competency in engaging in statistics investigations and planning to teach statistics: the Screencast Assignment and the Task Design Assignment. The Screencast Assignment may be used at any point after the completion of Module 1. In this assignment, teachers record the actions on their computer screen and talk aloud as they complete a new data investigation in CODAP. These screencasts reveal how the teachers use CODAP and provide an opportunity to communicate their statistical thinking. The Task Design Assignment is intended to be assigned after completing Module 1 and at least one of the other two modules. In this assignment, the teachers design a CODAP-based statistical task and create a plan for implementing the task. The assignment consists of six parts: (1) background information including alignment to standards, student learning objectives, and a link to a CODAP file; (2) a plan for how the task will be launched with students; (3) the task as it will be posed to students; (4) anticipated student responses to the task; (5) a description of the intended implementation, including how the teacher intends to scaffold students’ thinking and use students’ work to discuss the task; and (6) a reflection where the teachers explain the choices they made in developing the task and identify what they learned in developing the task.

4.3 Implementation of Modules

We actively recruited mathematics and statistics teacher educators to participate in professional development workshops and subsequently implement ESTEEM material in their university course(s). Forty-five MTEs participated in at least one workshop during 2018 or 2019. Between the spring of 2018 and summer of 2020, 30 of these MTEs participated in a study in which data was collected. The MTEs taught at 27 institutions across the United States that implemented ESTEEM materials in a variety of undergraduate and graduate courses (48 courses total). These courses included 804 enrolled students, most of whom were preservice mathematics teachers and some in-service teachers or general statistics students. For simplicity in reporting, we refer to students enrolled in these courses as teachers and the postsecondary educators as MTEs. Most MTEs received professional development through a one-day workshop, and some participated in additional online webinars. Our research examines implementation across settings and ways materials impacted MTEs’ and teachers’ learning about teaching statistics.

4.3.1 Data Collection During Implementation

In 31 courses, 298 teachers took a self-efficacy survey that measured their before and after confidence levels for teaching statistics (Harrell-Williams et al. 2019), and teachers and MTEs completed a post-implementation survey about their learning experiences. The self-efficacy to teach statistics survey was given in a retrospective format so that teachers were only asked to engage in the survey once. The retrospective version of the survey has been shown to have a similar structure, validity, and response trends as the version of the survey given in pre-post format (Harrell-Williams et al. 2020). The post-implementation surveys for MTEs and teachers included Likert scale ratings about impressions of materials, open-ended feedback about most impactful learning experiences, and suggestions for improvement. In addition, MTEs indicated in a post-implementation survey which module activities they implemented and whether teachers in their course engaged in an activity in a face-to-face, online synchronous, or online asynchronous setting. Other data sources included interviews with MTEs (n = 25) and a sample of volunteer teachers (n = 16), statistics tasks designed by teachers, and other assignments.

4.3.2 Use of Modules and Activities

Of the 48 courses in which MTEs used any part of the ESTEEM modules, we have post-implementation survey results from 46 courses. Even though ESTEEM materials are designed for secondary preservice mathematics teachers, 37% of courses (n = 17) included elementary preservice teachers, and 7% of courses (n = 3) were undergraduate statistics courses serving different disciplines. LMSs used in courses included Blackboard (39%), Canvas (26%), and Moodle (17%). The move to remote instruction experienced by many universities in the spring and summer of 2020 increased the number of MTEs who used our materials in either a hybrid (35% of courses) or completely online setting (20% of courses), though these modes of instruction still occurred less often than courses taught in face-to-face settings (45% of courses).

When planning for a course, MTEs indicated which modules they intended to use or partially use in 46 different courses. Once MTEs selected which module(s) were appropriate for a course, 65% of the courses implemented 80% or more of the activities in those modules. Only 7% of the courses implemented less than 40% of the activities within their selected modules, and these courses tended to be general statistics courses or content courses for elementary preservice teachers. Both parts of the Foundation in Statistics Teaching Module (Module 1) were available to all MTEs (n = 46 courses). Parts 1 of the Teaching Inferential Reasoning Module (Module A) and the Teaching Statistical Association Module (Module B) were available starting in spring of 2019, with the potential to be used in 30 of the 46 courses. Part 2 of Module A and Module B were available starting in the fall of 2019, with only 17 courses having the potential to implement.

Regardless of the mode of instruction for the course, MTEs may have implemented activities in a face-to-face setting, a synchronous online setting (e.g., video conferencing tools such as Zoom or Google Meet), or an asynchronous online setting (e.g., as an assignment posted in their LMSs to be done without a MTE’s interaction). Figure 4.7 shows the number of courses in which each activity across the three modules was used and the modality in which they were used: 46 courses used activities from Foundations in Statistics Teaching, 17 courses used Teaching Inferential Reasoning activities, and 14 used activities from Teaching Statistical Association. The activity numbers on the horizontal axes in Fig. 4.7 correspond with the activities described in Appendix 1.

Fig. 4.7

Use and modality of ESTEEM activities implemented from fall of 2018 to summer of 2020

Data investigations using CODAP were highly used in courses (1.1.g, 1.2.g, A.1.f, A.2.e, A.2.f, B.1.e, B.2.f) and often used in face-to-face settings (red in Fig. 4.7) or online synchronous sessions (green). Many MTEs also chose to engage their classes with the videos in 1.1.f and B.1.a during in-person class sessions—videos that were brief and showed students and teachers discussing critical concepts in statistics (investigative cycles and categorical association). However, many activities that included longer videos of expert teacher discussions or videos showing students and teachers engaged in statistics were often used asynchronously (orange bars in Fig. 4.7) through independent work (e.g., 1.1.e, 1.2.d, A.1.d, B.1.c, B.2.d). Even when MTEs were using other activities from a module, some of the discussion board activities were used much less often than other types of activities (e.g., 1.1.i, 1.2.j, A.1.g, A.2.g, B.1.f, B.2.h). However, it is also clear that a few MTEs used the activities from a discussion board to structure an in-class experience or discussion with their teachers, as indicated by the small red bars for those activities.

4.4 Effectiveness of Modules

We used several data sources from ESTEEM material implementations to examine the effectiveness of the materials, from the perspective of MTEs and teachers in their courses.

4.4.1 Feedback from MTEs and Teachers

We collected post-implementation feedback from almost all MTEs in the 48 courses (96% of MTEs, n = 46) but had a much lower response rate from teachers (41% of the 804 enrolled, n = 329). After examining MTEs and teachers’ responses to these post-implementation surveys, ESTEEM materials were found to be effective in several ways. MTEs and teachers had overall positive impressions of materials, where 89% of MTEs and 87% of teachers strongly agreed or agreed that the materials were easy to access and navigate online. In post-implementation MTEs’ interviews (n = 25), many MTEs commented that it was “very easy” to use ESTEEM materials in their own LMSs. Eighty-five percent of MTEs and 74% of teachers strongly agreed or agreed they felt more prepared to teach statistics after using ESTEEM materials. A majority of MTEs and teachers strongly agreed or agreed that the materials were effective in developing their statistical knowledge and knowledge about teaching statistics, particularly in relation to using CODAP to engage in statistical investigations and to analyze multivariate data (see Table 4.2).

Table 4.2 Ways ESTEEM materials were useful, according to MTEs and teachers

Both MTEs and teachers overwhelmingly found ESTEEM-designed CODAP materials to be useful resources that they would implement in their own teaching (91% and 82% strongly agreed or agreed, respectively). They indicated that videos showing students and teachers working with statistics tasks and online statistics tools were useful in understanding classroom issues related to teaching and learning statistics (83% and 76% strongly agreed or agreed, respectively), as well as videos of discussion with teachers and statistics education experts (75% and 64% strongly agreed or agreed, respectively). From open-ended comments and interviews, we also learned that teachers highly valued the opportunity to observe authentic learning through multiple video resources (i.e., real classroom videos, animated videos, and teacher reflection videos). Videos showing students and teachers working on statistics tasks were particularly helpful in illustrating how to implement the ESTEEM data activities in classrooms, as well as teachers’ pedagogical strategies for exposing students to less familiar contexts in a data investigation (e.g., noticing characteristics of roller coasters while experiencing a point of view video of a roller coaster ride).

We next share results related to specific themes regarding MTEs’ and teachers’ impressions of the effectiveness of the ESTEEM materials based on responses to open-ended questions in the post-implementation survey and interviews. For MTEs, we share both positive and negative feedback. We briefly describe revisions to the materials based on negative feedback.

4.4.1.1 MTEs’ Impressions of Materials

Overall, MTEs felt ESTEEM materials were of high quality and compared very favorably to other teacher education materials. They indicated that the materials provide a much-needed emphasis on the key ideas in statistics and they fills “a dire need” for material with a focus on teaching statistical practices. For many, ESTEEM materials were a missing puzzle piece that supported them in filling existing gaps in their curriculum. One MTE stated,

I felt like in the past,... I was really struggling to find resources that would give my teachers experience not just doing the statistics their future students would do, but to think about teaching the statistics after they had done the activity or seen a piece of the activity and to step back and think, "What are the teaching principles behind this?" These materials really helped fill in those gaps for me as far as providing a tool-kit for me and teaching the course.

Others found the materials to be cohesive, well-thought-out, organized, and well-packaged, enabling smooth integration into existing courses. Most MTEs reported little difficulty in integrating modules into their LMSs. One MTE elaborated in the following way:

I could access everything right from that Moodle course. So, that was so nice not to have separate files for all the documents I gave them. Or just all the links could take me to anywhere where I needed to go in the materials themselves. Super helpful. Super timesaving.

While a small number of Blackboard users experienced some technical difficulties, almost all other MTEs expressed positive experiences in using ESTEEM materials in their LMSs.

MTEs also reported that ESTEEM materials contributed to their own professional growth. Although many MTEs felt they had a strong understanding of statistical content knowledge, many expressed in interviews that they had not previously felt prepared to teach preservice teachers how to teach statistics. MTEs believed ESTEEM resources better equipped them to prepare teachers to teach statistics.

While MTEs overall reported that the ESTEEM materials were of high quality and useful, some reported challenges in using the materials. Five themes emerged around the following: time constraints, teachers’ knowledge, discussion forums, videos, and technology issues. One of the greatest challenges in MTEs implementing the materials was that they felt constrained by time. Some felt that there was too much material to get through in their already packed semesters. Other MTEs indicated that their teachers lacked sufficient statistical content knowledge to engage with the materials and they needed more support in helping teachers develop this knowledge, sometimes indicating that teachers did not get as much from specific resources (e.g., videos) because teachers’ content knowledge was not sophisticated enough. An MTE also indicated that teachers’ pedagogical knowledge was sometimes limited because they did not have enough field experiences or professional maturity to appreciate some resources (e.g., videos).

Some MTEs reported discussion forums did not facilitate meaningful discourse in their courses. As a result, the ESTEEM team reviewed discussion forum prompts, particularly those which featured representations of practice (e.g., classroom video, video of a student working with technology) and modified prompts to focus the discussion. When able, we added background information to provide more context.

While most MTEs had positive impressions of the video resources in the materials, some indicated that it was sometimes difficult for teachers to take away the message of the video. Other MTEs indicated some videos were too long. Yet others pointed to the vast amount of video resources, indicating there were too many. Rewording discussion prompts around videos as described was also used to address this issue. Additionally, we enhanced longer videos by using PlayPosit, an online tool that allows an instructor to embed questions and interactions in videos to provide a more interactive experience for the learner, along with feedback. This also focused on important ideas from the videos and made them more explicit for teachers.

A small number of MTEs reported technical issues; however, this did not interfere with their use of the materials. Issues typically related to importing materials into MTEs’ specific LMSs, like being unable to import without IT assistance, extra assignments being imported that could not be deleted, or issues with their grade books. Others found it cumbersome to select specific materials to import when they did not want all the materials. Almost all issues were reported by Blackboard users. Some MTEs were not able to easily change the format of assignments (e.g., PDF to Word or Google doc). Issues were usually quickly resolved with assistance from a member of our team or IT at their institution. The ESTEEM team updated written instructions for importing materials specific to MTEs’ LMSs for Blackboard, Canvas, and Moodle and also created how-to videos for importing materials into an existing course.

MTEs also made a couple of recommendations for implementing the materials. They advocated for the creation of an implementation guide and establishing a community of implementers. While creating a full implementation guide was beyond the scope of the project, the ESTEEM team engaged in several activities to address these needs. The team designed and implemented five free webinars to date to support implementers of the ESTEEM materials. Three were held specifically for MTEs who had already implemented materials, where we focused on needs they identified from a survey. In these webinars, a few implementing MTEs participated and shared their own experiences, providing advice for implementation. Additionally, we included MTEs who had field-tested our materials in a professional development workshop to support new MTEs to offer implementation advice. We also provided implementation advice for specific contexts (i.e., face-to-face, online, hybrid) during this workshop. The workshops and webinars were spaces for establishing and building a community of users.

4.4.1.2 Teachers’ Impressions of Materials

Based on teachers’ post-implementation surveys, the vast majority strongly agreed or agreed that all materials were useful for their learning about various aspects of statistics teaching (76%) and increasing their own content knowledge (75%). In responses to open-ended questions from the post-implementation survey and interviews, five themes emerged in relation to what teachers learned from the ESTEEM materials, including their experience exploring data with CODAP. In general, teachers believed that the ESTEEM materials developed their understanding of statistics and provided them with valuable resources for teaching, particularly how to engage students in statistical tasks using technology as a tool to teach statistics.

Teachers also noted the importance of engaging students in the full statistical investigation cycle and in data collection. They appreciated the power of having their future students explore data through CODAP and overwhelmingly identified CODAP as the resource that had the biggest impact on their thinking, where they identified their role as asking good questions to students and facilitating while students themselves engage in investigations. A preservice teacher expressed the importance of engaging students in the investigative cycle:

Students are involved when they feel they are engaged in the process. They are willing to look for data, analyze it, and find the results of data if they can relate to it. You want to have a statistics lesson that allows students to participate in all steps of the process.

Teachers also indicated they learned it is critical to make statistics relevant to students by using real data and about the power of statistics in solving real-world problems, and some emphasized the importance of context. One teacher from 2019 discussed:

The biggest lesson learned from these materials that will inform my teaching of statistics is to have the students work with real-world data, preferably data that they retrieved themselves. Have them pose a question they are interested in finding an answer to, then have them collect data and analyze it using technology, and finally, interpret what the data means and make a connection to the original question.

Many teachers also noted gaining a deeper understanding of students’ thinking about statistics, as one described:

I learned that students, no matter their age, can understand statistical concepts. Statistics is much more than I thought it was before, and I could see that through viewing the videos of students interacting. Students can look at things, make assumptions, and get interested in learning more about the data before them. They can investigate on their own and find out more information without having to [be] walked through the process.

Many teachers reflected on the value of the Students’ Approaches to Statistical Investigations framework, which explains how students’ statistical sophistication develops across three levels in relation to the investigation cycle (first introduced in Activity 1.2.b), as a resource for better understanding how students develop their statistical thinking. One teacher discussed the value of the framework in the following way:

I really liked the handout we got that broke down the different levels of thinking students can have with the different aspects of statistics. Posing questions, collecting data, analyzing data, and making conclusions. I learned that students, independent of the grade level, can all be at different levels of statistical thinking, and it is important to always start at their level and build them up from there.

And for some teachers, they learned how statistical thinking differs from mathematical thinking and about common student misconceptions, as one MTE stated,

We had some very good discussions about how math and [statistics] are related [and] the differences between math and [statistics]. I don’t think that’s something a lot of them had considered closely before. So the discussion that we had related to our reading; that was really critical to fostering their thinking about how those are different and how they should be taught differently.

4.4.2 Impact on Teachers’ Self-Efficacy to Teach Statistics

Teachers’ self-efficacy to teach statistics increased following the use of the ESTEEM materials. In interviews, many teachers described themselves as “math” people rather than “statistics” people before engaging with the ESTEEM materials. The ESTEEM materials provided a safe context to learn and explore, and teachers felt they experienced increased confidence with statistics as a result of their engagement with the ESTEEM materials, helping them learn to both appreciate and apply statistics. Other teachers expressed more comfort in conducting statistical investigations with large data sets as a result of ESTEEM’s Engage with Data experiences.

Results from the self-efficacy to teach statistics (SETS) survey also indicated that teachers’ confidence to teach statistics significantly increased after engaging with the ESTEEM materials. The SETS survey asks teachers to rate their confidence to teach students 44 different, specific, statistical topics, ranging from less sophisticated topics to more sophisticated topics. Responses on the survey were on a scale from 1 (not at all confident) to 6 (completely confident). Collectively, 298 preservice and in-service teachers in 31 courses using the ESTEEM materials completed the retrospective SETS survey, typically within 1–8 weeks after completing the ESTEEM materials in the course. Across all topics, teachers’ mean confidence increased by 1.22 points, from a mean of 2.85 (before) to 4.07 (after), which represents a large gain in confidence to teach statistics.

4.4.3 Designing Tasks with Data Investigations

Courses targeted at preparing teachers often implemented the Task Design Assignment. We examined products of this assignment, including 73 CODAP-enhanced data investigation tasks and accompanying task “launches” (Casey et al. 2020a, b; Hudson et al. 2020). In the task launches, the majority of teachers oriented students to the context of the data (68%) and prompted students to make a personal connection with the data’s context (53%). Few (25%) of the task launches, however, motivated a need for a driving statistical question that would guide engagement in the task. Analysis of the tasks themselves showed that most tasks incorporated three key aspects for teaching statistics espoused in ESTEEM materials: analysis of large, multivariate, real data sets; connection to the data’s context throughout the task, including through task presentation and prompting of students to connect their work to the data’s context; and engagement in multiple phases of the statistical investigation cycle. The tasks in general often lacked a driving statistical question, which resulted in tasks that consisted of a series of disconnected prompts without a clear statistical purpose. In addition, many tasks did not honor differences between mathematics and statistics, asking students to prove things based on their analysis or to make predictions that are “exact solutions” rather than estimates with a margin of error.

4.5 Discussion and Implications

Online modes of instruction have become increasingly available in mathematics teacher preparation programs. Graduate programs designed for initial licensure in secondary mathematics and for supporting in-service teachers in developing their mathematics teaching practices were some of the first to utilize synchronous and asynchronous course modalities. Our project was primarily targeting undergraduate mathematics teacher education courses. We were innovating solutions for two gaps in undergraduate secondary mathematics teacher preparation: use of online modalities for instruction and access to high-quality statistics teacher preparation materials. We discuss successes, struggles, and implications for each of these innovations.

4.5.1 Flexibility in Online Instructional Modules

It is clear that MTEs found great value in the ability to easily import ESTEEM materials into their courses and integrate them with other materials in their LMSs. MTEs made different choices about which activities to use in their classes that matched the needs of their students and the goals of their course or program. In accord with findings from Holmes and Prieto-Rodriguez (2018), ESTEEM-using MTEs appreciated the flexibility to rearrange materials within their LMSs. They appreciated that they did not have to go to different sites themselves to find codes to embed videos, upload PDFs, or find CODAP documents, and their teachers did not have to leave their course LMSs to engage with materials, except when opening CODAP in a new browser tab.

Although the ESTEEM modules were designed to provide accessibility and interactivity in online asynchronous settings (Holmes and Prieto-Rodriguez 2018; Park 2015), MTEs found ways to modify the materials into face-to-face or online synchronous activities and also used many online activities as asynchronous homework that students accessed through the LMSs outside of the in-person setting. Since March 2020 and an abrupt shift to online instruction at many institutions, we have seen a greater interest in our materials and use of them in online or hybrid settings. Very few courses which used ESTEEM were completely online (20%), and our data indicate that very few activities were used in synchronous sessions online (small green bars in Fig. 4.7). In the next phase of our project, we will be expanding support for online instruction and are curious to investigate how MTEs engage teachers in completely online courses, synchronously or asynchronously. For example, we know that various features of video-based synchronous learning environments are highly effective in promoting student engagement: polling, emoticons, hand raising, chats, shared whiteboards, and screen sharing (McBrien et al. 2009; Starling and Lee 2015). We have not yet collected data on how features of synchronous learning environments may impact MTEs and teachers’ learning experiences with the ESTEEM materials.

Publishers of college textbooks often give MTEs ways to create links between an instructor’s course site in LMSs and an external site housed with the publisher containing electronic material (applets, videos, readings, data sets) and assessments to support instruction with an adopted textbook. Accessibility of material in an organized, structured manner within LMSs is highly valued by future teachers and MTEs (Holmes and Prieto-Rodriguez 2018). When materials are distributed in LMSs in a way that MTEs then have direct control to reorder, rename, modify, or supplement, it can empower them to actually learn more about how features of LMSs and the organization of materials could support their work as teacher educators. We encourage more teacher education curriculum projects to consider the development of online materials with Creative Commons licensing and distribution in a ready-to-use format that can be easily integrated and modified by MTEs in LMSs to fit the unique needs of a course.

4.5.2 Constraints in Using LMSs Distribution

Our project aimed to use the common cartridge standard for working with digital learning content. Our approach to packaging materials in a way that can be used across different LMSs resulted in a necessary constraint in the types of elements (or activities) that could be used. We were limited to using only those elements that are supported across LMSs: pages, assignments, quizzes, and discussion forums. Even though we included a few instances of an interactive video using a third-party tool like PlayPosit, the use of such a tool had limited benefit for an instructor, who could not view their teachers’ responses to video prompts. Even when using these elements, the import from the common cartridge did not appear well organized in all LMSs. For example, when the common cartridge was imported into Blackboard, the import included empty folders and unorganized pages. Additionally, importing into different versions of the same LMSs was sometimes an issue. Our team needed to have access and the skills to build materials directly in three LMSs (Moodle, Blackboard, and Canvas). Thus, any revisions to materials created a domino effect of changes that had to be made across all three LMS exports. 

Although MTEs appreciated the access of materials in LMSs and the flexibility to modify materials as needed, they also cited issues with importing modules into their LMSs. For example, when a Blackboard user who only planned to implement the Foundation module imported material into a course, the import included all three modules and both assignments. Although we made a design decision to package all three modules together, this decision created additional work for MTEs who did not intend to implement all modules. An MTE would need to hide or delete unused materials and delete unneeded assignments that were auto-added to an LMSs’ gradebook. Instead, in future iterations of our materials and as a suggestion for others developing online materials, MTEs should be able to select which materials they wish to export and import into their courses. This would require more careful thought in an MTE’s planning process for a course but less work in structuring, organizing, and integrating materials in their LMSs.

4.5.3 Improving Preparation for Teaching Statistics

MTEs integrated ESTEEM materials into a variety of undergraduate and some graduate courses that contribute to teachers’ preparedness to teach statistics. MTEs used materials in courses such as Teaching Mathematics with Technology (typically serving teachers for grades 6–12), Statistics for Teachers, Secondary Mathematics Methods (9–12 or 6–12), Elementary Mathematics Methods (K-5 or K-8), and Introductory Statistics. Teacher preparation programs vary greatly across institutions, and MTEs report it was easy to find courses in their program where the modules fit well, at least partially, for the statistical and pedagogical goals of a course.

Multiple MTEs who used the ESTEEM materials stated that oftentimes, their teachers did not have the prerequisite statistics content knowledge needed to meaningfully engage in ESTEEM activities. We purposefully designed ESTEEM materials to focus on developing teachers’ pedagogical knowledge for teaching statistics since there are few secondary mathematics teacher education materials that do so and the development of this knowledge is crucial for teacher development. Based on feedback from MTEs that used the ESTEEM materials and this design decision, MTEs may need to provide additional learning opportunities for teachers to ensure they have the statistical knowledge needed to engage in ESTEEM activities.

Teachers that engaged with ESTEEM materials built their confidence and reported feeling better about their statistical understandings, comfort in exploring multivariate data on their own with CODAP, and knowing ways to engage students in key statistical practices and data investigations using technology. MTEs also felt better about their ability to prepare teachers to teach statistics, particularly related to the use of CODAP and understanding how to promote statistical discourse. Thus, the ESTEEM materials can be used to help MTEs meet the recommendations in the Statistical Education for Teachers report (Franklin et al. 2015). Both MTEs and teachers had positive experiences with CODAP and viewing videos of its use in classrooms that appeared to impact their perspective and comfort with investigating large multivariate data sets using linked representations. Disaggregating feedback data based on the learning activities MTEs and teachers used is an important next step in further research to unpack relationships between ESTEEM activities utilized in a course and reported impressions and growth in confidence.

Though teachers were able to demonstrate an ability to design tasks that can give students appropriate experiences in the age of the “data deluge” (Gould and Cetinkaya–Rundel 2014) by using large real data sets and connecting statistical thinking to the context of data, they still struggled with keeping a focus on statistical practices, investigations, and uncertainty in claims. Instead, some still focused on procedures and exact answers and seemed to not be able to put into action what they had an opportunity to learn about the differences between mathematics and statistics. What we don’t know yet, and will be the focus of future research, is which ESTEEM activities seem to be connected with stronger task design. For example, we noticed that while most teachers had several opportunities to engage with statistics tasks themselves as learners, several activities in our modules that are specifically targeted at developing pedagogical strategies for designing worthwhile statistics tasks were used less often than other activities that focus on content (e.g., see Fig. 4.7, 1.2.j, A.2.d, A.2.g, B.2.h). Some courses focused more on content, so these differences may be a result of the goals of a course. This impels us to dig into our data further and perhaps design a comparative study of tasks designed by teachers who did experience those pedagogy-focused activities and those who did not.

MTEs can readily integrate ESTEEM materials into a variety of courses to help ensure future teachers are prepared for teaching statistics. We have learned that teachers are gaining better understanding of the nature of statistics by experiencing data investigations with CODAP and critically examining videos of students and teachers engaged with statistics tasks. The ESTEEM videos are some of the few available that include a focus on teaching secondary statistics with exploratory technology tools. Teacher educators, however, may need further assistance in ensuring that their choice of activities to implement may matter and impact what teachers are ready to do as statistics teachers. For mathematics teacher education programs that have multiple courses, there may also need to be concerted effort to weave the set of ESTEEM materials into more than one course in a way that can build teachers’ sophistication in statistical understandings and pedagogy across more than one semester.

Appendices

Appendix 1: ESTEEM Table of Contents

Full Annotated Table of Contents is available at https://fi-esteem.s3.amazonaws.com/lmsbackups/annotated_table_of_contents.pdf

Module 1: Foundation in Statistics Teaching

The focus of this module is the core ideas about teaching statistics and how to support students’ investigations. It is highly recommended that teachers complete this module before being introduced to materials from either of the other two modules (Module A or Module B). Module 1 is organized into two parts.

Module 1.1 What is statistics and how should we teach it?

This module focuses on learning to engage in statistical investigations using technology (CODAP) and developing knowledge and skills for planning lessons for teaching statistics. The emphasis is on understanding how statistics is different from other areas in the mathematics curriculum and how students can develop statistical ways of reasoning. Read and Watch Essential Materials 1.1.a How is Statistics Different from Mathematics? (Reading) 1.1.b Statistical Investigations and Habits of Mind (Video, Reading, Diagram) 1.1.c Considering the Importance of Teaching Statistics (Video, Reading) 1.1.d Quiz on Read & Watch Material (Quiz: 6 multiple-choice questions & 2 free-response questions) Learn from Practice 1.1.e Teaching Statistics in the Mathematics Curriculum (Video) 1.1.f Statistical Investigation Cycle in a Classroom (Video) Engage with Data 1.1.g Investigating Older Roller Coasters in the US (Video, CODAP Data Investigation) Synthesize and Apply 1.1.h Discuss Learning Statistics through Investigations with Real Data (Discussion Forum) 1.1.i Using an Online Data Analysis Tool (Discussion Forum)

Module 1.2 What is statistics and how should we teach it?

In this module, teachers will learn about a framework that can guide them in supporting students’ statistical reasoning, including designing tasks and making sense of students’ work. Teachers will also engage in a statistical investigation from the previous module, with a larger data set. Read and Watch Essential Materials 1.2.a Supports for Learning to Do Statistical Investigations (Readings, 2 Videos) 1.2.b A Guiding Framework for Teaching Statistics (Reading, 2 Videos) 1.2.c Tasks as Opportunities for Statistical Learning (Table, Video) 1.2.d Read & Watch Quiz (Quiz: 8 multiple choice questions) Learn from Practice 1.2.e Expert Teacher Interview on Tools & Resources (Video) 1.2.f Teaching Statistics Using Multiple Technologies (Video) Engage with Data 1.2.g Investigating More Roller Coasters (CODAP Data Investigation) 1.2.h Examining Students’ Work on the Roller Coaster Task (Video, Discussion Forum) Synthesize and Apply 1.2.i Supporting Statistical Discourse with the Roller Coaster Task (Reading, Video, Reflection Paper) 1.2.j Analyze Tasks and Discuss (Discussion Forum)

Module A: Teaching Inferential Reasoning

This module focuses on ideas around supporting the development of students’ inferential reasoning. Teachers should be familiar with the ideas in Module 1 prior to being introduced to this module. Module A is organized into two parts.

Module A.1 Promoting and supporting inferential reasoning

The first part of module A focuses on developing essential understandings of how to support inferential reasoning. Materials will help teachers consider why it is important to teach students to reason inferentially and how questions, modeling processes, simulation tools, and tasks can support students’ development of inferential reasoning. Read and Watch Essential Materials A.1.a What is Inferential Reasoning? (Reading, 2 Reflection Questions A.1.b Promoting Key Aspects of Inferential Reasoning (Video, Reading) A.1.c Using Models to Build Inferential Reasoning (Reading, CODAP Sampler Activity, Reading, Video) Learn from Practice A.1.d Considering the Importance of Inferential Reasoning (Interactive Video) A.1.e Anchoring Inference in a Cycle of Investigation (Video) Engage with Data A.1.f Investigating Fairness of Dice (CODAP Data Investigation) Synthesize and Apply A.1.g Comparing Use of Models in Tasks for Inferential Reasoning (Discussion Forum) A.1.h Analyzing Students’ Work on Schoolopoly (2 Interactive Animation Videos, Discussion Forum)

Module A.2 Using models and repeated samples to develop inferential reasoning

The second part of module A focuses on how models and repeated sampling can be used to support inferential reasoning. Materials will delve deeper into critical understandings related to sampling distributions and how learning experiences can assist students in developing inferential reasoning. Read and Watch Essential Materials A.2.a Critical Role of Samples, Sampling, and Sampling Distributions (Reading, Video A.2.b Attention to Sampling Variability and Sample Size (Reading, CODAP Activity, Reading) Learn from Practice A.2.c Using Repeated Sampling to Introduce Sampling Distributions (2 Videos, 4 Reflection Questions) A.2.d Statistics Tasks to Promote Inferential Reasoning (Video) Engage with Data A.2.e Investigating the Success of Paul the Octopus (CODAP Data Investigation) A.2.f Investigating Carbon Dioxide Emissions in Vehicles (CODAP Data Investigation) Synthesize and Apply A.2.g Discussing Launching a Task to Support Inferential Reasoning (Discussion Forum) A.2.h Applying Modeling and Simulation to a Probability Comparison Task (Written Response)

Module B: Teaching Statistical Association

This module focuses on ideas around the teaching and learning of statistical association. Teachers should be familiar with the ideas in Module 1 prior to being introduced to this module. Module B is organized into two parts.

Module B.1 Statistical association of categorical variables

The first part of module B focuses on association of categorical variables. Materials assist teachers in developing critical understandings related to graphs and measures used to describe association between categorical variables and how learning experiences can assist students in developing reasoning about association. Read and Watch Essential Materials B.1.a Investigating Categorical Variables in CODAP (Reading, 2 Videos) B.1.b Common Student Approaches when Analyzing Bivariate Categorical Data (Reading, Videos) B.1.c Quiz on Read & Watch material (Quiz: 7 multiple choice and 1 open-ended questions) Learn from Practice B.1.d Student-created Graphs of Bivariate Categorical Data (Video, Discussion Forum) Engage with Data B.1.e Investigating Data about Granola Bars (CODAP Data Investigation) Synthesize and Apply B.1.f Discuss Representations of Bivariate Categorical Data (Discussion Forum) B.1.g Students’ Reasoning about a Segmented Bar Graph (Activity; 5 Videos, Written Response)

Module B.2 Statistical association of quantitative variables

The focus of the second part of module B is association of quantitative variables. Materials assist teachers in developing critical understandings related to graphs and measures used to describe association between quantitative variables and how learning experiences can assist students in developing reasoning about association. Read and Watch Essential Materials B.2.a Introducing Students to the Topic of Statistical Association (Activity, Video) B.2.b Measures of Association and Lines of Best Fit (Video) B.2.c Distinguishing Between Correlation and Causation (Reading) B.2.d Quiz on Read and Watch Materials (Quiz: 6 multiple-choice and 1 open-ended questions) Learn from Practice B.2.e Considering Student Approaches to Placing the Informal Line of Best Fit (Activity, 2 Animated Videos) Engage with Data B.2.f Investigating Data about Vehicles (CODAP Data Investigation) B.2.g Teaching Statistics with CODAP (Video) Synthesize and Apply B.2.h Discuss Differences between Mathematics and Statistics in the Study of Association (Discussion Forum) B.2.i Investigating Data from the Census at School Random Sampler (readings, CODAP Data Investigation)

Screencast Assignment

This assignment allows teachers to illustrate their ability to conduct a statistical investigation with larger, multivariate data sets in CODAP. Teachers record themselves with screencast software while conducting a statistical investigation and explain their thinking throughout the process. This assignment can be done after any of the modules.

Task Design Assignment

The purpose of this assignment is to design a task that illustrates how one can develop students’ statistical thinking utilizing CODAP as a tool. This assignment can be completedafterModule A and/or Module B.

Appendix 2: Sample Multimedia Page from Teaching Inferential Reasoning Module