Keywords

Iterative Best Evidence Syntheses

New Zealand’s Iterative Best Evidence Synthesis (BES) Programme leads collaborative knowledge building and use in education by taking a brokerage role across policy, research and practice communities (Alton-Lee 2007). The touchstone of the programmeFootnote 1 is its attention to selecting and synthesising evidence about influences on valued outcomes for diverse learners with a focus on what makes a bigger difference in the outcomes for all students. There are two major purposes of the programme: first, to be catalytic in advancing the use of research as a resource for continuous improvement; and second, to highlight and promote the kind of high impact research and development in education that has refined effective approaches for diverse learners through cycles of collaborative implementation with teachers. It is not surprising that such forms of research and development can make a much bigger difference in education.

The use of effect sizes and the results of meta-analyses provide educators with invaluable information about where investment can make a bigger difference. For teachers, school leaders and policy makers this is a significant resource because it shows where working smarter not harder can make a bigger difference too. Hattie (2009) has been able to analyse national data from the Assessment Tools for Teaching and Learning database to ascertain the average student gain for a year of teaching in New Zealand across three subjects:

In our own New Zealand studies, we have estimated the yearly effect in reading, mathematics, and writing from years 4 to 13 (N = 83,751) is .35 – although this is not linear. (p. 17)

Accordingly, in the best evidence synthesis iterations (BESs) we use an effect size of d = 0.35 as an indicative benchmark for the business-as-usual contribution of teaching over a year. This enables teachers to make initial judgements about potential impact for investment of their time. For example, in his synthesis of over 800 meta-analyses Hattie (2009) found an overall effect of d = 0.74 across two meta-analyses of 38 studies for reciprocal teaching approaches to improve reading comprehension. Reciprocal teaching is a way of specifically teaching reading comprehension originally developed by Palincsar and Brown (1984). Reciprocal teaching involves four ‘thinking skills’ that have been shown to improve comprehension and self-monitoring: clarifying, questioning, summarising, and predicting. The teacher uses explicit teaching, including modelling, repeated practice, and feedback to coach students into taking over leadership roles in using each skill in turn in a small group.

There are several New Zealand postgraduate thesesFootnote 2 showing where the use of reciprocal teaching has been implemented over relatively short periods of time (intensively over around 5 weeks) resulting in accelerated achievement, student self-regulation, and peer leadership, using metacognitive and strategy skills. One New Zealand implementation for first year secondary students produced an effect size of d = 1.10 after students had 12–16 reciprocal teaching sessions (with no effect at all for a shorter comparison intervention) (see Westera 2002).

Because the results of meta-analyses also show us where educational influences have a negative impact on student outcomes, it can be a resource in developing more ethical educational practices. We have waited too long for a ‘first do no harm’ in education principle. A meta-analysis for the School leadership and student outcomes: Identifying what works and why best evidence synthesis iteration shows both where many business-as-usual practices have negative effects overall, but also where high impact research and development can have very high positive effects (Fig. 27.1).

Fig. 27.1
figure 1

Findings of a meta-analysis of research on the educational impact of making connections between schools, families/Whānau and communities (See Robinson et al. 2009, Chapter 7 http://www.educationcounts.govt.nz/goto/BES)

Full size image

Knowing that a certain approach makes a bigger difference is insufficient. One worst case scenario is that policy takes an ‘evidence-based approach’ in which magical thinking is used around the implementation of ‘what works’ evidence. Taking an ‘evidence-based’ approach needs to be as assiduous in the process of change as it is in the focus of change if improvement is to occur. For improvement to follow then educators need to know why and how some approaches make a much bigger difference for valued outcomes for diverse learners. These are questions that demand theory, qualitative information, and an ecological approach to the multi-level and systemic changes needed to support improved educational practice. They are also helpfully informed by cases where effect sizes are high and the qualitative information illuminates the why and the how.

Examples of Best Evidence Syntheses

Through its series of best evidence syntheses focused on the outcomes-linked evidence about pedagogy, teacher professional learning and development, and educational leadership influences on valued student outcomes, the Iterative BES Programme is progressively explaining the how of educational improvement processes.

Four BESs were initially developed as part of the New Zealand Ministry of Education’s medium term strategy policy work:

  1. 1.

    Quality Teaching for Diverse Students in Schooling (Alton-Lee 2003).

  2. 2.

    Community and Family Influences on Children’s Achievement (Biddulph and Biddulph 2003).

  3. 3.

    Quality Teaching: Early Foundations (Farquhar 2003).

  4. 4.

    Professional Development in Early Childhood Settings (Mitchell and Cubey 2003).

In 2004 the Guidelines for Generating a Best Evidence Synthesis Iteration were developed (Haig 2004) using a realist synthesis framework in consultation with three national advisory groups in New Zealand: the BES Standards Reference Group, the BES Māori Educational Research Advisory Group, and the BES Pasifika Educational Research Advisory Group. These Guidelines are available on the BES website and govern the methodological approach to best evidence synthesis development.

In 2005 the Iterative Best Evidence Synthesis Programme was formally established within the Ministry of Education and four subsequent BES iterations have been developed using the Guidelines and extensive engagement with stakeholders; representatives of whom have helped to manage and formatively quality assure the BES developments:

  1. 1

    Effective Pedagogy in Mathematics/Pāngarau (Anthony and Walshaw 2007).

  2. 2

    Teacher Professional Learning and Development (Timperley et al. 2007).

  3. 3

    Effective Pedagogy in Social Sciences/Tikanga ā Iwi (Aitken and Sinnema 2008).

  4. 4

    School Leadership and Student Outcomes: Identifying What Works and Why Best Evidence Synthesis (Robinson et al. 2009).

In recognition of their contribution to definitive knowledge about effective educational practices and policies the International Academy of Education has commissioned summaries of all four of the BESs generated since the formal establishment of the programme to be lodged on the UNESCO website. At the time of writing two summaries are available in English, Te Reo Māori, and progressively in other languages, at http://www.ibe.unesco.org/en/services/publications/educational-practices.html. These are Anthony and Walshaw (2009) on Effective Pedagogy in Mathematics, and Timperley (2008) on Teacher professional learning and development.

Perhaps the most critical finding across all of the best evidence syntheses is the effect size arising out of a meta-analysis carried out for the School Leadership and Student Outcomes BES, that revealed an effect of d = 0.84 for leadership practices involved in promoting and/or participating in teacher professional learning and development. In turn the effect sizes for the highest impact professional development in the Teacher Professional Learning and Development BES show effect sizes for achievement gains that are equivalent to more than 2 year’s progress in 1 year across student populations, and for the 20 % of lowest achieving students the equivalent of 3–4 years’ progress in 1 year. Again the qualitative data and emerging theory that explain how and why such professional development was so effective matters. Accordingly the best evidence syntheses illuminate through case and vignette what was distinctive about highly effective practice.

However, just because there is evidence about an approach making a bigger difference in one context does not guarantee that it will in another. Accordingly, the Iterative BES Programme proposes that teachers, leaders and policy makers take an inquiry mindset to using the evidence (See Fig. 27.2).

Fig. 27.2
figure 2

Teacher and leader inquiry and knowledge building cycle (Timperley 2008, pp. 26–27)

Full size image

Concluding Comments

Critical to the Iterative BES approach is drawing upon a range of expertise from formative quality assurers in the research community and working collaboratively with education stakeholders. Principal and other middle leader representatives, teacher union representatives and others advise throughout BES development on how to shape the scope, usefulness, and accessibility of the best evidence syntheses, and critique work in progress. These processes provide a forum for constructive albeit heated debate and help to strengthen the quality, trustworthiness and usefulness of the syntheses. They also help to advance use of the findings through building trust and ownership with those for whom the knowledge is intended. Such ownership is represented in stakeholder forewords to the documents.

PPTA (the New Zealand Secondary Teachers’ Union) welcomes this latest Best Evidence Synthesis as a significant contribution to our understanding of the role of professional learning in assisting teachers to develop their practice…

Robin Duff (President) PPTA

(Timperley et al. 2007, xii)

Depending on the area of BES focus, meta-analyses are not always available to inform or able to be constructed to inform the comparative magnitude of impact analyses. Where they can be used, and illuminated by cases and theory, they provide an invaluable resource for educational improvement that can benefit not only students, but also teachers, and educational leaders who are able to build on what has been learned, and further advance our understanding of what makes a bigger difference in education.