Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Study Aims and Research Questions

Alignment between educational goals, intended and implemented curricula and educational outcomes is considered as a characteristic of effective education. The expectation is that better alignment leads to better student performance. The concept of Opportunity to Learn, abbreviated as OTL, is commonly used to compare content covered, as part of the implemented curriculum, with student achievement. As such it is to be seen as a facet of the broader concept of “alignment”. One of the aims of this study is to further clarify these concepts, identify how they have been used in research studies, and are employed in practice. Although opportunity to learn was originally studied within the context of curriculum research, it has also obtained a place in educational effectiveness research. Within this research orientation OTL is seen as “an effectiveness enhancing condition” and can be compared with other such factors for its influence on student achievement. As a matter of fact, results of meta-analyses would suggest that OTL has a relatively substantial average-effect size when it is compared to other effectiveness enhancing conditions, such as learning time and instructional leadership (Scheerens 2016). Yet, the number of meta-analyses and review studies on the effects of OTL is rather limited. This study seeks to make a step towards updating the state of the art, by means of a search for meta-analyses and recent primary studies.

Several recent trends in the ongoing global efforts to improve the quality of education provide further perspective to assessing the state of the art on OTL, these are alignment within the context of systemic reform, the accountability movement, and task related cooperation between teachers.

Alignment Within the Context of Systemic Reform

In influential reports by the OECD and McKinsey the quality of educational systems is considered in systemic terms, as a whole of impulses and mechanisms at system, school and classroom level (OECD 2010; McKinsey & Company 2010). Alignment between levels in various functional domains is a key concept in finding out why certain educational systems do better than others. The expectation is that systems do better when aims, objectives, curricula and assessment programs are well-aligned. The conceptual analysis in this report, starting out from OTL intends to further clarify the complexity of alignment between “curricular elements” and opens up discussion on alternative interpretations, for example by comparing proactive structuring and retroactive planning.

Accountability and Its Influence on Teaching

As indicated in the above, OTL originates from curriculum theory and research. According to a pro-active logic, aims are operationalized to standards, worked out as intended curricula, which are expected to be implemented with a certain fidelity, and finally evaluated and assessed, by means of examinations and formative and summative assessment. This is still a valid logic, although developments in the direction of greater school and teacher autonomy may give rise to a different orientation. More curricular autonomy that goes together with a more prominent role of “high stakes” testing might lead to situations where teaching gets more direction from alignment to the assessment programs than from references to rather global and “open” curricula. A negative interpretation from this phenomenon is “teaching to the test”. A more positive interpretation is described by terms like “exam preparation” and “instructional alignment” (Popham 2003; Sturman 2011; Polikoff and Porter 2014). One of the challenges of this study is to provide suggestions for legitimate test preparation, while avoiding harmful interpretations in “teaching to the test”.

Task Related Cooperation Between Teachers

The teacher has a key role in realizing “opportunity to learn”; the choice and use of textbooks may be one issue in how this plays out. Another medium is teacher training and professional development of teachers. Recent studies in the realm of teacher training effectiveness underline the importance of teacher content knowledge and pedagogical content knowledge (Baumert et al. 2010; Blömeke et al. 2014; Scheerens and Bloemeke 2016). Within the context of continuous “on the job” professional development teacher cooperation and “peer learning” have obtained high profile (e.g. Thurlings and den Brok 2014). Results of meta-analyses underline the importance of task related work in order to make teacher cooperation effective (Lomos et al. 2011). The results of this study will be used to provide suggestions for placing OTL and instructional alignment on the agenda of task related teacher cooperation.

The general objectives of the review study are to create more clarity about the conceptualization of OTL within a broader framework of educational alignment and to assess the available research evidence about OTL effectiveness. For this latter objective the focus is on the positive significance of OTL effects, effect sizes, and the degree to which OTL effect are related to contextual conditions, such as subject matter area, grade level and national context where the study was conducted.

More specifically the following research questions are addressed:

  1. (1)

    Which facets are to be distinguished in clarifying the overall concept of OTL, and how are these to be placed as part of more general models of educational alignment and systemic reform?

  2. (2)

    What is the average effect size of OTL (association of OTL with student achievement outcomes), as evident from available meta-analyses, review studies, secondary analyses of international data-sets and (recent) primary research studies?

  3. (3)

    What are the implications of the results on (1) and (2) for educational policy and practice?

Methods

The study approach consists of a conceptual analysis, based on literature review. Review of research literature: meta-analyses, research reviews and primary research studies, and secondary analyses on data from international assessment studies, TIMSS and PISA.

Conceptual Analysis

The following issues are addressed

  • The definition of OTL. OTL will be defined from the perspective of three research traditions: curriculum research, educational effectiveness research and (international) student assessment.

  • Embedding OTL in a broader framework of “educational alignment”.

  • Alternative ways to measure OTL (in terms of research methods, respondents, content focus and/or focus on psychological operations that students are expected to master).

  • The role of teachers in realizing OTL.

Literature Search

First of all, an inventory will be made of available meta-analyses with respect to OTL and instructional alignment. Next, from the available international assessment study reports one or two examples will be selected for secondary analyses of OTL effects. Finally, a systematic search of recent primary OTL effectiveness studies will be carried out. A set of explicit selection criteria will be used to arrive at a set of relevant studies with sufficient research quality.

Analyses of Research Literature and Available (International) Data Sets

A narrative review will provide a summary of the identified review studies and meta-analyses on OTL effectiveness. Average effect sizes from these meta-analyses will be compared with similar results for other effectiveness enhancing conditions, like learning time, educational leadership, teacher cooperation and evaluation at school level. The data from international assessment studies yield descriptions of the way OTL was measured in these studies, as well as effect sizes (OTL associated with student achievement) within and between countries. The individual research studies identified by means of the systematic searches, and application of the selection criteria will be schematically summarized. Basic analyses of the tabulated descriptions provide information about the proportion of studies in which OTL had a positive and significant effect on student achievement (a so called vote-count analysis), grade-levels addressed in the studies, subject matter area in which OTL was measured and nationality of the study. Vote counts found for OTL in this study are compared to vote counts for other effectiveness enhancing variables, computed in other review studies.

Exploration on How OTL and Educational Alignment Are Addressed in the Practice of Dutch Primary Education

This exploration is based on a limited number of interviews with experts and officials in the areas of curriculum development, educational testing, and textbook production. Preliminary results will be discussed with a panel of teachers.

Structure of the Report

In the second chapter a conceptual analysis of Opportunity to Learn (OTL) is given, covering also related terms, such as instructional alignment and test preparation. The OTL issue is highlighted from three educational research traditions: educational effectiveness research, curriculum research and achievement test development. The conceptual analysis leads to pinpointing OTL as a specific type of alignment in educational systems; a taxonomy of alignment forms is presented. Next, different facets of the way OTL is measured empirically are discussed. The conceptual analysis is given further theoretical depth, by discussing De Groot’s (1986) integrative model of didactic and evaluative operationalization. Reflecting on this model brings the alignment issue in a systemic perspective, leading up to the conjecture that alignment, OTL and test preparation aim for integration in organizational structures that are often to be characterized as loosely coupled.

In the third chapter and inventory of meta-analyses of OTL effects (association between measures of OTL and instructional alignment with cognitive achievement outcomes) is presented. This leads to a first impression of the average magnitude of OTL effects. Next, seven case-study descriptions of illustrative OTL research studies are given, spanning four decades of research. The illustrative studies provide an impression of the diversity in emphasis, with exposure of content taught, and alignment between different curriculum elements (like standards, textbooks, taught content and tested content) as two different kinds of independent variables. One of the meta-studies is more specifically oriented to implications of high stakes test for content selection in teaching.

In the fourth chapter an overview is given of about 50 primary studies, conducted during the last twenty years. Schematic summary descriptions are put together in a table. Although quantitative meta-analysis of these studies is beyond the scope of this review study, some basic summary tables are produced to provide an overall orientation on how OTL has been researched and what can be concluded about its effectiveness.

In the fifth chapter secondary analyses based on data from international studies are presented.

In the sixth and concluding chapter conclusions are drawn, and the relevance for educational science and policy and practice is considered. Illustrations will be provided that are drawn from the exploration of policy and practices in the Netherlands. The chapter leads up to recommendations for educational policy planners and teachers.