Keywords

1 Introduction

Statistical literacy is typically considered to involve the ability to read and interpret statistical information, often in everyday contexts, and draws on understanding of numeracy, statistics, general literacy and data presentation (see, e.g. Gal 2002). The importance of statistical literacy for all has been highlighted by statisticians (e.g. Wallman 1993), and its scope within the school curriculum and appropriate pedagogical approaches have received attention (e.g. Watson 2006). What is being referred to there is knowledge sufficient to make sense of data and statistical information that are likely to be encountered in the broad community (e.g. in the media). However, there may be specific aspects of statistics and reporting styles that are specific to particular sectors of the workforce. We use the term professional statistical literacy to refer to this slightly more specialised knowledge.

An increased push for measuring educational outcomes and a growth in institutional capacity to generate and analyse large data sets have resulted in schools having to deal with extensive statistical information about student outcomes and similar data. The government expectation is that teachers will use this information to inform decisions regarding school planning and teaching practice, and “develop a more objective view about the performance of their students compared to those in other schools and in relation to state-wide standards” (Ministerial Council on Education, Employment, Training and Youth Affairs, n.d.). Clearly a necessary requirement for successful data-driven choices is that school principals and teachers have sufficient statistical literacy to interpret such data. However, a Statistical Society of Australia Inc. (2005) report pointed out that statistics has a poor image and profile among students, parents and the general public. Indeed, negativity towards statistical information and lack of confidence in analysing statistical data may discourage education personnel from other than cursory engagement with such information. Tom Alegounaris, a board member of the Australian Curriculum and Assessment Authority and president of the NSW Board of Studies, commented that “teachers lack the expertise to analyse student results” and “were seen to resist using data”, arguing further that “teachers had to discard their ‘phobia’ of data” (Ferrari 2011). Michelle Bruniges, Director-General of Education in NSW, similarly suggests that “school improvement is being held back because many teachers lack confidence and skills to analyse National Assessment Program for Literacy and Numeracy (NAPLAN) student test data” (Milburn 2012).

This chapter examines the issues associated with statistical literacy in the education workplace and reports on the trial of a workshop for teachers. This workshop was developed in order to address previously identified barriers and misconceptions associated with analysing and interpreting system reports of student assessment (hereafter referred to as SRSA). The chapter draws attention to conceptual and attitudinal issues that need to be addressed in any statistics courses designed for pre-service or practicing teachers.

2 Evidence Base for Teachers’ Statistical Literacy Workshop

In researching professional statistical literacy in the education sector, we chose to focus on the system reports associated with student assessment in Victoria, specifically reports from the National Assessment Program for Literacy and Numeracy (NAPLAN) and the Victorian Certificate of Education (VCE) data service. Prior to the development of a workshop for teachers, data were collected from 938 Victorian primary and secondary teachers as summarised in Table 1. These data were intended to give us an indication of both attitudinal factors and statistical literacy issues that might impact on teachers’ work with SRSA.

Table 1 Data collected to provide an evidence base for the design of the workshop for teachers
Full size table

2.1 Underpinning Frameworks and Results

The surveys mentioned in Table 1 had items investigating teachers’ attitudes towards data, and their understanding of how to interpret SRSA. The background for and results from these sections are discussed in what follows; they were particularly relevant to informing the design of the professional statistical literacy workshop.

2.1.1 Theory of Planned Behaviour

Gal (2002) specifically highlighted the role of dispositions in contributing to whether or not individuals will choose to “activate” their statistical knowledge. A series of items framed by Ajzen’s (1991) Theory of Planned Behaviour (TPB) probed teachers’ attitudes, subjective norms and perceived behavioural controls that may act as enablers or barriers for their intention to engage with SRSA (see Pierce and Chick 2011a). Ajzen (1991) proposed the TPB as a framework for studying intention to change behaviour, and identified three key components that determine that intention:

The first is the attitude toward the behaviour and refers to the degree to which a person has a favourable or unfavourable evaluation or appraisal of the behaviour in question. The second predictor is a social factor termed subjective norm; it refers to the perceived social pressure to perform or not to perform the behaviour. The third antecedent of intention is the degree of perceived behavioural control which … refers to the perceived ease or difficulty of performing the behaviour and it is assumed to reflect past experience as well as anticipated impediments and obstacles. (p. 188, emphases added)

He argued that favourable attitudes and subjective norms, together with greater perceived behavioural control, would result in a stronger intention to perform the associated behaviour. This has been supported by research (see, e.g. Armitage and Conner 2001), showing that the three components are strongly predictive of behavioural intent, which, in turn, can account for a considerable proportion of variance in actual behaviour.

Table 2 shows the results from half of the TPB questions used in the survey conducted with Group 3, focusing on attitudes, signified by AT, and behavioural controls, signified by BC. These indicated that most teachers saw SRSA as valuable for planning at a school, curriculum and lesson level (2.3, 2.5, 2.6), and that they did not think that they had a problem interpreting the SRSA (2.7). Although these results were generally positive, there were significant numbers of neutral and disagreeing responses.

Table 2 Likert items probing teachers’ attitudes and perceptions towards SRSA and planning
Full size table

2.1.2 Framework for Professional Statistical Literacy

A second series of items on the Group 3 survey focused on teachers’ professional statistical literacy. Using a framework proposed by Pierce and Chick (2011b)—building on earlier work by Curcio (1987), Gal (2002) and Watson (2006)—these questions assessed teachers’ ability to interpret data in a professional situation. This framework (shown in Fig. 1) acknowledges that effective data interpretation requires attention at multiple levels in a hierarchy. The lowest level, reading values, involves understanding features such as keys, scale and graph type, together with the capacity to read specific data points on the graph or table. The second level, comparing values, requires attention across multiple facets of a graph or across one or more representations (graphs or tables). Finally, the third level, analysing the data set, involves considering the data as a whole entity, including interpreting variation, and attending to the significance of results. The framework also acknowledges the role of context, in association with the three levels of technical facility. Professional context concerns information relevant to the profession and needed to interpret the data set (e.g. meaning of specialist terms such as “band”). The second, local context, comprises knowledge about the situation or context that gave rise to the data that is not evident in the data set alone (e.g. knowledge of the local school situation that may have affected test results). These two context components may overlap, hence the dashed line between them in Fig. 1. The structure of this framework for professional statistical literacy was verified in Pierce, Chick, Watson, Les and Dalton (2014).

Fig. 1
figure 1

A framework for considering professional statistical literacy (Pierce and Chick 2011b, p. 633)

Full size image

It should be noted that in Victoria the most common graphic used in SRSA is a box plot, with whiskers extending only to the 10th and 90th percentiles because testing is considered unreliable at the extremes. Since the graphic does not represent the full distribution of results, there is potential for misinterpretation; furthermore, research (see, e.g. Pfannkuch 2006) tells us that students commonly exhibit confusion between the frequency and density of data points in a box plot. In the survey items focusing on these aspects, misconceptions were found to be prevalent, implying that teachers’ confidence in their capacity to interpret SRSA—as revealed by their responses to 2.7 in Table 2—was not well founded (see Pierce and Chick 2013, for further details). Specifically, whereas most teachers could correctly read values from tables and identify a school’s weakest area from a graphic, more than 70 % misinterpreted certain aspects of box plots. Furthermore, data collected through focus groups and surveys (from Groups 2 and 3 in Table 1) suggested a need to check and address participants’ conceptual understanding of data and graphics before focusing on interpretation and consequent workplace decision making. It was also clear from Group 2 responses and focus group discussions that a lack of appropriate vocabulary hindered teachers’ ability to describe and compare distributions of results.

3 The Teachers’ Statistical Literacy Workshop

3.1 Learning Objectives for the Workshop

The information obtained about teachers’ attitudes and statistical literacy informed our objectives—in terms of the messages to be emphasised and the choice of content to be targeted—in the half-day professional learning workshop that we designed for teachers to improve attitudes and statistical literacy. The data suggested that a significant minority of teachers did not feel that the data were valuable nor that they provided useful information about their students (items 2.1 and 2.4, Table 2), and that a majority did not understand the fundamental construction of a box plot (a typical set of SRSA boxplots is shown in Fig. 2). These results implied that any professional learning workshop clearly needed to address both attitudes and competence.

Fig. 2
figure 2

The hypothetical school summary report showing the results of the hypothetical class’s results (the right-most box plot in each group of three) compared with national and state results (leftmost and middle, respectively)

Full size image

The structure and nature of the program developed paid attention to previous research on the elements of successful professional learning programs. Ingvarson et al. (2005), for example, examined four studies on teacher professional development undertaken by the Australian Government Quality Teacher Programme. In total this encompassed 80 professional development activities and 3,250 teachers. They found that “consistent significant direct effects were found across the four studies for the impact of content focus, active learning, and follow-up on knowledge and professional community” (p. 1). These findings—together with the recommendations of Martin (2008, 2010) based on his experience of providing statistical training in a variety of industries—informed the development of the “Making sense of data” workshop for teachers described below. Martin (2008, 2010) drew attention to the deep learning that can take place when statistical knowledge is set meaningfully in the participants’ workplace context so that the new knowledge and the skills being taught relate to real workplace needs. For these reasons the workshop focused on the detail and format of the statistical reports most commonly sent to Victorian schools from the various government education authorities.

Based on our research results and the above professional learning principles—a focus on content, active learning and attention to participants’ practice—a workshop was designed to actively engage teachers with the statistical content of SRSA. The learning objectives for the workshop were that teachers should be able to:

  1. 1.

    Demonstrate understanding that the box plots in the NAPLAN reports provided to their schools only represent the middle 80 % of the cohort since the whiskers extend only to the 10th and 90th percentiles, and, as a consequence, interpret box plots taking into account that the weakest and strongest students are not represented by this particular graphic.

  2. 2.

    Demonstrate understanding that the “fences” of a box plot divide the cohort into quarters so that the length of each section gives an indication of density not frequency, and, as a consequence, interpret box plots appropriately avoiding such misconceptions as “there are too many students in the tail”.

  3. 3.

    Engage with the data because they realise it tells them something about their students.

  4. 4.

    Make use of the data to inform planning for teaching by identifying patterns in students’ strengths and weaknesses.

3.2 Workshop Structure and Tasks

The activities were designed to actively involve teachers with relevant scenarios that targeted key concepts. At the beginning of the session teachers were introduced to a hypothetical class of 30 students with a School Summary Report (see Fig. 2) and individual SRSA results. Each student in the class was depicted on a separate narrow card as an image with individual assessment data (one such student is shown in Fig. 3, together with his NAPLAN data). The class size of 30 was chosen not only for its realistic estimate of Victorian class size, but also to allow simple determination of the top and bottom 10 % of the cohort together with the location of the median.

Fig. 3
figure 3

One of the set of 30 hypothetical students used as a data set for the professional learning workshop with individual NAPLAN data results. Data recorded in the text below the student figure were James’ NAPLAN results — Reading 523.4, Writing: 582.4, Spelling: 594.8, Grammar & Punctuation: 557.2, Numeracy: 541.1

Full size image

3.2.1 Task 1: From Students’ Scores to Box Plots

The teachers, working in groups of three or four, used the image cards and their data (Fig. 3) to plot the distribution of Reading scores on a large number line. The cards themselves were used as data points (see Fig. 4). Using this plot, they then built a NAPLAN-like box plot by (1) dividing the group into quarters, (2) placing a red box over the middle 50 %, (3) turning over the top and bottom 10 % of images to hide those students and their data (while still leaving a place holder in the plot) and (4) extending whiskers to the furthest visible students (see Fig. 4). It should be noted that dividing the class into quarters was mildly problematic for some teachers because the class size was 30. The statistical solution to this was discussed briefly, but the main emphasis was on building the idea that each quarter contains equal numbers of students. This exercise used images of students, not just points, in order to reinforce the message that this data provide information about students, thus linking the abstract box plot graphic image to concrete information. The box plot construction exercise emphasised, first, the density rather than frequency of students represented by the sections of the plot and, second, the importance of paying attention to the key, which on Victorian SRSA notes that the whiskers extend to the 10th and 90th percentiles.

Fig. 4
figure 4

The distribution of reading scores for the set of 30 hypothetical students used as a data set for the professional learning workshop, with the associated boxplot

Full size image

3.2.2 Task 2: From Box Plots to Students’ Scores

Given that the background research data had suggested teachers were confused regarding the density vs. frequency issue, the workshop included an activity that was essentially the reverse of Task 1, in which it was highlighted that a given box plot could represent any number of possible distributions. Each group was given a new box plot and asked to plot student image cards (with no scores given) in a possible distribution to fit the box plot. The variety of distributions suggested from the different groups allowed discussion of possible and impossible solutions. The activity also afforded explicit consideration of possible ranges, densities, and skewness.

3.2.3 Task 3: Interpreting and Discussing Box Plots

In the third task participants were asked to consider a series of statements describing Example College’s Numeracy results. The wordings used in the descriptions—good or otherwise—were based on actual descriptions given by teachers to an open-response survey item in the evidence-base phase described earlier. These descriptions, together with our observations in the early phase of data collection with Group 2, revealed that many teachers appeared to have difficulty using appropriate language to describe graphical information and statistical results. The workshop task was intended to address these difficulties and focus on descriptive precision and appropriate vocabulary. In the workshop each group of teachers was given ten statements to consider (see Fig. 5). The task was to (1) identify and discard incorrect statements; (2) rank the remaining statements in terms of helpfulness and then (3) write their own summary of one of the SRSA box plot reports.

Fig. 5
figure 5

Teacher statements describing Example College’s Numeracy results

Full size image

3.2.4 Task 4: Analysis of Real Data

After working on box plots and allowing teachers to articulate global impressions of the Example College group’s abilities (in relation to the State across five tests), the focus of the workshop shifted to having teachers examine students’ responses to individual NAPLAN test items via what is known as an Item Analysis Report. This report provides statistics about school, state and national performance on specific questions used in student tests. This activity was introduced through examples discussing the absolute and relative differences between results (e.g. determining whether or not it is significant that a school has only 56 % of its students getting a question correct compared to 61 % of the students in the whole state), with a focus on practical local measures of similarity and difference. This led to the final half of the workshop, which provided an opportunity for each school group to work on interpreting its own school SRSA.

4 Outcomes

4.1 Participants and Method

The workshop participants were a subset of Group 2 in Table 1, with the same teachers who had provided the survey and focus group data, but chosen from just one of each of the primary and secondary schools in the networks previously involved (this choice of school was random, leading to a selection of participants from 10 schools out of the original 20). Five workshops were conducted, one for each of the five pairs of primary and secondary schools. At least 4 weeks after taking part in their single half-day workshop, the participants—together with the other Group 2 teachers from those schools who had not attended the workshop—completed an online survey targeting attitudes, perceptions and statistical literacy with respect to SRSA, similar to the initial paper-based survey they had taken to provide the evidence base for the workshop design. Allowing for teacher attrition over time and non-participation in the follow-up survey, 82 of the original 150 teachers completed the survey (noting that only 123 were still available to participate), and 45 of these attended the workshop.

4.2 Workshop Outcomes

The outcomes of the workshop were noted through the workshop observations of the authors, and via more formal feedback from participants via the follow-up survey. Those involved in presenting and organising the sessions (the authors, along with representatives from the project’s partners) were aware of the impact of the workshops as they progressed. After each presentation the project team noted the strong evidence for the teachers’ clear need for conceptual activities related to box plots. In one group, the assumed data expert from the school was observed to be confidently making erroneous statements about some aspects of the data set. This helped confirm the researchers’ decision to include a workshop activity where the validity of statements made about data could be examined.

There was informal but audible appreciation from the participants as they came to understand the box plot representation, with many indicating that they had not had such understanding prior to the workshop. This feedback indicated that some participants realised for the first time, for example, that a “long tail” on a box plot indicates diversity of student results rather than large numbers of students; that the box does not represent the whole class; and that bounds at the 10th and 90th percentiles mean that the character and number of students not represented within the displayed box plots is almost certainly of practical significance for teaching. Participants also commented that working with images of students prompted mental links to their own students, and that they now had a greater perception that such data could be relevant to their practice as teachers. The observers also noted that participants were able to use this understanding to critique descriptions of distributions, and could appropriately apply the principles learnt to an analysis of their own school reports.

The researchers also noted a high level of engagement from participating teachers throughout each of the workshop sessions. Working in groups of three to four, the participants were observed to make group decisions about all aspects of the activity where a box plot was built from the data on the figures’ cards. In response to a final comments question in the online post-workshop survey mentioned above, many commented favourably on the value of that exercise:

I thought the hands on method (using the actual number lines & cards) was excellent & increased my understanding of box & whisker graphs and data analysis.

I found the explanations of how to produce and read box plots really helpful. The manipulatives were great.

Teachers’ general perceptions of the workshops were overwhelmingly positive. Some comments mentioned increased confidence, greater understanding of box plots, and use of (or intention to use) the materials and ideas with other staff in participants’ schools. Indeed, some participants requested access to the materials so they could replicate the box plot activities at future staff meetings. The comments also indicate that principals and teachers at both primary and secondary levels responded favourably to the simple hands-on learning activities creating box plots. Typical of the comments are the following:

Excellent opportunity to finally learn how to properly analyse box and whisker plots (hopefully)—we dedicated a 2-hour staff meeting to sharing this information with the rest of our staff. Feel much more confident when asked what comments I can make in response to data when presented to me.

It was good also to work with secondary teachers which gave a slightly different perspective (although we used our own school data).

Was really good to discuss and interpret the data, and have practical tasks to complete relating to data analysis. Made you think! Some good ideas to take back to staff on how to analyse data more accurately.

Some of the comments indicated why changes to statistical literacy understanding may have been limited for participants. Three comments in particular indicated that even a focused, hands-on workshop may be insufficient to ensure on-going understanding.

It was excellent. However, you tend to forget some of the stuff because you are only exposed to it for a short period of time. You need to be exposed to it for a few more sessions to not only become confident with it and ask questions, but then become so familiar with it that it becomes second nature.

I think I would need a week of intense professional development to be able to fully understand how to read these graphs. While I get exposure to this and I am getting a better understanding as years pass I still struggle on some aspects.

I find data interpretation to be quite challenging, and would like to attend further PDs on this subject if they should arise.

4.3 TPB Factors and Statistical Literacy

The delayed post-workshop survey allowed some comparisons between workshop participants and non-workshop participants. On average those who had attended the workshop had higher scores on attitude items and lower scores for perceived barriers than those who had not attended the workshop, indicating the likelihood of improved attitudes and perceptions among participants. The statistical literacy questions on the survey were scored, using a simple partial credit scale, and it was found that the statistical literacy scores for those who attended the workshop (W) were higher than for those who had not had this experience (NW) but this difference is not statistically signifi-cant: \( \left({\overline{x}}_W=26.4,{s}_w=7.5;{\overline{x}}_{NW}=24.5,{s}_{NW}=9.1;{t}_{df=79}=0.975,p=0.3327\right) \). However, the results in Table 3 suggest that although the frequency density misconception had been corrected for some—but by no means all—teachers (see the data in Row 3.2), there remained confusion about some aspects of the presentation of box plots. This was caused by not reading the key and difficulties interpreting the consequences of a “reduced” box plot (with 10 %/90 % whiskers) (see Row 3.3). It may be that the image of the box plot is too powerful and hinders individuals’ capacity to (a) remember that some data are not represented by components of the plot and (b) recognise the possible range of such “invisible” data.

Table 3 Percentage of teachers choosing the truth or otherwise of given statements in the post-workshop survey, from those who had (W) and had not (NW) attended the workshop
Full size table

5 Conclusions and Implications

The workshop described here certainly appears to have had a positive effect on attitudes and perceptions, as well as having improved some aspects of teachers’ professional statistical literacy. Unfortunately, at least one significant misconception persisted despite the attention given to what is and is not shown in box plots, and one teacher claimed to be “still baffled” after the program. Another comment reflected the mixed success of the program: “[The activities] were worthwhile, interactive, hands on, well paced, informative and I learned lots (but obviously not enough to pass this ‘test’!)”. For many teachers, reading box plots is not yet “second nature”, although it is acknowledged that the peculiarities of the non-standard format used in Victoria’s particular box plot representations may exacerbate the problem. Nevertheless, the teachers acknowledged the effectiveness of the materials and activities used in the workshop for developing understanding.

For those with statistical experience and expertise, many of the concepts associated with professional statistical literacy for teachers and highlighted above are relatively elementary. On the other hand, for those who deal with data only a few times a year and with limited prior knowledge, the outcomes from the evidence-based research and the post-workshop survey suggest that there are some areas that many teachers do not understand sufficiently. In any case, it appears important to provide timely access to reminders of what others may regard as basic concepts. For these teachers, extended programs and/or refresher programs may be of benefit. Following the qualified success of the workshop, the researchers are interested in examining ways of providing targeted statistical literacy professional learning for teachers in a timely and accessible way, without having to formally timetable a face-to-face presentation for groups. The authors have used their experiences with the workshops to create a series of short online packages, each requiring just a few minutes of interaction. The packages have been designed to emulate some of the practical activities, but are accessible for teachers’ use at their convenience. These have been trialled with some groups of teachers, with initially encouraging signs, although the formal research into their effectiveness is not yet complete.

In order for teachers to use data effectively in their planning, it is essential that they acquire the kind of fluency that allows them to understand how the abstract data representations depict their very real class or school. In addition, they need to understand where each student is or could be located within the data. This means that the critical misconceptions explored here—the density/frequency issue and the all-the-data-are-within-the-plot assumption—need to be overcome. Equally importantly, a shift in attitudes, confidence and perceptions about the data’s value and comprehensibility is also likely to lead to greater and better use being made of such school assessment data.

6 Note

Developed from a paper presented at Eighth Australian Conference on Teaching Statistics, July 2012, Adelaide, Australia.

This chapter is refereed.