Keywords

3.1 Introduction

The goal of our research is to provide a multilevel empirical comparison of two accountability policies—in France and in Quebec—which we have characterized as “soft” or “reflexive.” These policies receive less attention in the international literature than policies in English-speaking countries. In this chapter, we describe our theoretical framework and what we mean by a multilevel approach to educational policies. This approach intends to consider the global influences which have played a role in their genesis but in situating these influences in their interactions with other more deep-rooted national institutional or political processes. Such processes shape and affect their manifestation in a specific trajectory embedded in the more long-term temporality of national education institutions and policies. However, this trajectory leading to vernacular forms of globalization (rather than a convergence of policies deriving from irresistible global discourse or forces) is also shaped by the processes of policy implementation at the intermediate level (regulatory authorities falling between the central authorities and local schools) or at the local level. More precisely, the orientations and local meaning of the policy are developed through the mediations which occur at the level of school boards (SBs) in Quebec and académies in France, when they respond to institutional expectations or prescriptions to “steer” or “manage” schools based on their results. In addition, they are conditioned by the instrumentation of this policy and by local (non) usages of devices and tools developed in the schools, depending on local contexts and interpretation—significant and strategic—of the sense of the policy which is being developed. These mediations and instrumentations are embedded in local contexts and configurations of actors. They are inscribed in pre-existing institutional arrangements and norms which may prove to be resources as much as constraints in implementing the policy. Over time, they will in turn shape the trajectory of the policy, which is not merely the fruit of an interface between the national and the global. National/regional/local mediations and the choices of instrumentation at an intermediate or local level are also significant in the orientation and meaning of these trajectories.

Thus, our analytical framework based on a three-way conceptual approach—trajectory, mediation, and instrumentation—can be further developed by, on the one hand, situating it with respect to the key analytical issues of accountability in education (Sect. 3.2) and, on the other hand, providing further details about our theoretical sources which together comprise our analytical “toolbox” (Sect. 3.3). Finally, in Sect. 3.3.1, we will develop the three conceptual strands proposed—trajectory, mediations, and instrumentation—before pointing out some limitations entailed by the use of such concepts (Sect. 3.3.2).

3.2 Accountability in Education

Having defined the generic notion of accountability, we will explore four research trends which have arisen around the analysis of accountability policies in education and are rarely articulated in the literature: (1) a typological approach to actual accountability policies in terms of their orientations and the tools employed, (2) an analysis of meanings and macrosocial or global sources of the transformations in the governance and regulation of education systems that these policies incur, (3) an econometric evaluation of some effects of accountability policies on academic performance in terms of efficacy or equity, and (4) an analysis of the implementation and reception of these accountability devices at the level of local or intermediate actors and organizations.

3.2.1 Accountability: A Multilayered Concept

Accountability is an “appealing but elusive concept” (Bovens, 2007, p. 447) which assumes many meanings and refers to various practices depending on the social field or the academic discipline concerned (political science, financial accounting, public administration, or education). Reviewing the literature, Lindberg (2013) has counted “over 100 different ‘subtypes’ and usages” of the term.

This English term is related historically and semantically to the term “accounting” (Bovens, 2007) and, as such, is difficult to translate into other languages such as French (Broadfoot, 2000) or Slavic languages (Veselý, 2013). Indeed, it has a long tradition in political science, where it refers to the idea that “when a decision-making power is transferred from a principal (e.g. the citizens) to an agent (e.g. the government), there must be a mechanism in place for holding the agent accountable for their decisions, and tools for sanctions” (Lindberg, 2013, p. 203).

Thus, the political term is related to representatives’ obligation to justify themselves to the constituencies who elect them (at least in democratic regimes). At the same time, higher administrative officers (and their agents following the chain of command) should be held directly accountable by the government (sometimes the legislature). In these terms, accountability becomes a democratic tool for monitoring and controlling government and, furthermore, the administration’s actions.

This meaning of accountability has broadened and also become more ambiguous, both in the academic literature and in public discourse, as public administration reforms have been undertaken in various countries under the influence of New Public Management (Hood, 1991), which situates accountability as a central device of good governance. In this context, the notion of accountability has become very popular but also polysemous. It has either become a synonym for loosely defined political goals (such as “good governance” or “democracy”) or has been related to various “mechanisms for controlling and ensuring quality in public institutions” (Veselý, 2013, p. 5). However, Bovens proposes considering accountability not as a virtue, but as an analytical tool defined as “a relationship between an actor and a forum, in which the actor has an obligation to explain and to justify his or her conduct, the forum can pose questions and pass judgments, and the actor may face consequences” (Bovens, 2007, p. 450).

The advantage of this definition is that it can be used as an analytical tool structured around five operational questions to analyze and classify accountability features and tools:

  • (1) Who is accountable? Who should render accounts? It could be organizations, such as political institutions (government and government administration), public sector organizations (schools), or individuals (politicians, higher officers, school principals, teachers, etc.).

  • (2) To whom (which forum) is the account to be rendered? In answer to this question, Bovens distinguishes various types of accountability: (a) political accountability, when the forum comprises political actors or institutions, such as governments or constituencies; (b) managerial or administrative accountability, when the forum is the administration or public service organization hierarchy; (c) professional accountability, when the forum is a professional body, an audit office, or “chartered accountants” and accountability relates an actor to a professional peer; (d) social (or market) accountability, when the forum comprises actors from civil society, users of a service, clients, or interest groups (stakeholders); and (e) finally, legal accountability, when the forum consists of courts.

  • (3) What is the relation (or the type of obligation) between the actor and the forum? A vertical relation refers “to the situation where the forum formally wields power over the actor, perhaps due to the hierarchical relationship between actor and forum” (Bovens, 2007, p. 460); horizontal accountability occurs when an account is given to stakeholders without formal obligation, as is the case in social accountability where relations are based on a social or moral obligation. Finally, a diagonal relation is an intermediate form (as is the case of accounts due to an audit or inspection body, without direct hierarchical power over the actor who is being held to account).

  • (4) and (5) Which aspect of his conduct (financial, procedural, and so forth) is the actor obliged to explain and provide justification? It could be based on inputs, processes, outputs, or effects of the actions taken. Moreover, these aspects can be evaluated on the basis of different criteria (i.e., equity, effectiveness, efficiency, conformity, transparency, or democracy) and diverse methods.

  • This polysemous notion of accountability has been used in various ways in education, as in other fields, and has paved the way for different approaches. We distinguish four of them in this chapter, given the focus of our research.

3.2.2 A Typological Approach to Orientations of Accountability Policies in Education

The first approach focuses on the specific features of accountability policies in the education field and leads to precisions concerning, first, the content of the new accountability policies in education and, then, how these policies vary across systems and countries. While accountability often refers to a social practice—“to be held to account” (Broadfoot, 2000; Leithwood et al., 1999; Jaafar & Anderson, 2007; Kogan, 1988; Leithwood et al., 1999)—various accountability approaches and tools are defined or compared by raising the same questions as Bovens: Who is accountable? To whom is the account owed? What is being accounted for? What are the consequences of providing an account?

Thus, in 1988, Kogan (1988) proposed a typology of models of accountability in education, based partly on the entity which exercises control, the actors to whom accounts must be rendered, and partly on the values and principles underpinning the legitimacy of accountability. In the model of public and hierarchical control (“public state control and managerialism”), control is exercised by an institutional authority (elected political representatives, or bureaucrats) which “holds people responsible,” and the accountability is justified in the name of “liberal democratic” principles (political or administrative responsibility in a democratic state). In the “professional” model, control is delegated to a professional group, due to their expertise, and this group is then held responsible for the self-regulation of its members. Finally, in the “consumerist” model, it is the users/partners or the clients who exercise the right of control, in the name of a liberal market philosophy.

In an international literature review, Leithwood (Leithwood & Earl, 2000; Leithwood et al. 1999) describe a large panorama of accountability approaches and tools: market-based, decentralized decision-making, professional, and managerial accountability. Accounts are to be given by teachers, principals, or schools to various forums (a professional order, a professional community, a local community, a district, a state, or parents) about either the processes or the outputs of education, with various consequences.

In these typologies, accountability can concern inputs and processes, as well as outputs. However, in recent policies, mostly in the USA and England, accountability in education is increasingly understood in a narrower sense. Accountability is related to accounts to be given by schools and teachers about results and outputs, taking into account targets (in terms of qualification rate and performances in external exams) and standards (related to curriculum or evaluation) determined at the central level (the ministry or state level). These accounts are to be given to the chain of command (district or state) or to parents, on the basis of school and student performance assessment, related to various indicators of results.

This “new” accountability in education or a “performance-based accountability” system (Carnoy & Loeb, 2002) implies a shift from input-based political regulation toward output-based regulation, where student results constitute the linchpin of accountability systems:

In principle, focusing on student performance should move states away from input regulations—judging schools based on the number of books in the library and the proportion of certified staff, for example—toward a model of steering by results—using rewards, sanctions, and assistance to move schools toward higher levels of performance. In other words, the educational accountability should focus schools’ attention less on compliance with rules and more on increasing learning for students. (Elmore et al. 1996, p. 65 in Linn, 2000, p. 12)

Thus, performance-based accountability includes four elements: (1) standards (what students should learn); (2) a testing system (or large-scale assessment system), usually administered by an external body; (3) public information about test results and an account explaining their sources or causes; and (4) positive or negative consequences for schools (Harris & Herrington, 2006).

These performance-based accountability systems share common features in reference to the questions proposed by Bovens:

  • Accounts are to be given by individuals (teachers, principals, or administrators) or organizations (schools or districts) to various organizations in the chain of command, such as districts or states, or to specialized agencies, such as inspection bodies.

  • The accountability relationship is vertical. Less frequently, the relationship can be horizontal, geared toward the local community.

  • Accountability is mostly based upon results or outputs of organizations (qualification and/or retention rate, students’ performances in external assessments in key grades and subjects, etc.). However, traditional objects of accountability (with respect to rules and procedures and the use of budgets) do not disappear.

    These outputs are evaluated according to certain standards and measurable objectives (indicators, targets, and benchmarks), and the actors held to account have to explain or justify potential gaps between their results and these standards to the forum.

  • Finally, the actors might have to face various consequences (symbolic or material) following this account. In particular, the literature has often contrasted high-stakes and low-stakes accountability devices (Carnoy & Loeb, 2002).

According to Figlio and Loeb (2011, p. 385), “the most-developed accountability systems operate in the U.S., England, and Chile and they are also the systems on which the overwhelming majority of academic research has been based.” However, in other parts of the world, similar policies have been given other labels, such as steuerung policies (Altrichter, Heinrich, & Soukup-Altrichter, 2011), “steering policies” (de Landsheere, 1994), “testing regimes” (Lingard, Martino, & Rezai-Rashti, 2013), and “evaluation policies” (Buisson-Fenet & Pons, 2014).

Various typologies have been developed to distinguish among these new accountability policies. Harris and Herrington (2006), for example, distinguish two types of accountability characterizing some aspects of educational policies adopted in the USA in the last decades (1990–2005): government-based accountability and market-based accountability. Government-based accountability corresponds to “government efforts to measure the outcomes of students and schools, especially on the basis of student test scores, and to provide explicit rewards and punishments based on these measures” (Harris & Herrington, 2006, p. 217). Market-based accountability corresponds to policies providing parents with greater school choice. The basic assumption is that giving parents greater choice regarding the school attended by their children is the best way to develop competition between schools and enhance the quality of public schooling. These types of accountability tend to be in addition to traditional local public accountability, where school principals are held to account by district administrations, and elected SBs are accountable to their constituencies.

The accountability policies established by various education systems in the USA were also differentiated according to the level of sanctions and incentives adopted to encourage or constrain schools and teachers to develop their educational capacities and improve their students’ results (Carnoy & Loeb, 2002Harris & Herrington, 2006). “High stakes” or “strong” forms of accountability go hand in hand with sanctions or incentives with serious consequences for the actorsFootnote 1; conversely, when these mechanisms of sanctions are less severe or absent, the literature refers to this as “low stakes” or “soft” accountability. In particular, such distinctions are used to differentiate among various US policies, in order to analyze their effects.

In a broader sense, Nathalie Mons and Vincent Dupriez distinguish accountability policies based on the theory of regulation that they incorporate in their orientations. Thus, they contrast “hard” accountability systems based on high-stakes sanctions, typical of certain US states, with “softer” and “reflexive” systems of accountability (Dupriez & Mons, 2011). In these systems, developed especially in France, Austria, and Belgium, it is a matter of confronting the organization or the teaching professional with their results, based on evaluation devices applied to the education system or individual schools, and then encouraging them to reflect on these and improve their practices through various professional training and support measures (Mons & Dupriez, 2010).

Maroy and Voisin (2014) and Maroy (2015) also propose a theoretical typology of the diversity of rationales and policy tools used by accountability policies in various education systems. This typology is based on four dimensions and is an attempt to combine the criteria used by earlier typologies. Two bear on the characteristics of policy tools deployed to implement policies (the degree to which measures are aligned and the implications of accountability for the actors) and two others on the theory of change embedded in policy tools (the conception of the actors targeted by the policy and the theory of change concerning their behavior).

Concerning the degree of alignment (strong or weak) between the tools and levels of action of education systems (central, intermediate, or local), there can be a tight and narrow coupling between tools and levels of action (when standards set criteria and provide guidelines that should orient local practices). In contrast, weak alignment involves instruments that are loosely coupled with one another and/or between levels of action. Another dimension to the tools is that of the nature and strength of the consequences faced by actors (high or low stakes).

The two other dimensions of the typology bear on the theory of change and regulation underpinning policies. This theory is not necessarily made explicit in a developed discourse, although this may sometimes be the case.Footnote 2 Change theories are often embedded in the policy tools by which the policy is operationalized (Lascoumes & Le Galès, 2004). Change theory involves, on the one hand, the conception of the actor that the policy intends to regulate (viewed as a rational and utilitarian actor or instead as a “reflexive” and socially embedded actor) and, on the other hand, the external or internal character of measures or dispositions by which an educational authority seeks to change or regulate the behavior of a local actor. In certain types of accountability policy, policy ontology makes external measures key factors in the process of change and regulation—they are the pragmatic supports that tend to condition the orientation of individual or collective conduct from the outside. In contrast, other types of accountability policies grant greater importance to the interiority of actors, their ethos, and internalized dispositions, as key vectors and mediations in the process of improving school performance (Boltanski & Chiapello, 1999; Mangez, 2001).

Combining these dimensions, Maroy and Voisin distinguish four approaches (see Table 3.1) underlying accountability policies: regulation through strict accountability, regulation through neo-bureaucratic accountability, regulation through reflexive responsibilization and accountability, and regulation through soft accountability.

Table 3.1 Four approaches to regulation by results
Full size table

Finally, from our perspective, performance-based accountability policies or steering by results policies (Linn, 2000) intend to modify (to a lesser or greater extent) the set of coordination and control mechanisms established by the state and educational authorities in order to orient or regulate the behaviors of local actors (e.g., teachers and school principals). In this sense, these policies entail new forms of political regulationFootnote 3 of the education system to improve the system’s effectiveness or equity (Maroy, 2009). As such, accountability policies share four common traits (Maroy, 2013): (1) they are embedded in a new policy paradigm whereby the school is conceived no longer as a core institution within society, but as a performative system of production (Ball, 2003b); (2) operational objectives of the school policy and system may be expressed in quantifiable data, which, in turn, become the standards and targets for the system; (3) various tools to assess student achievement are central to evaluating the outputs of the system; and (4) individual or collective actors at different levels of the system are held accountable for these results, with various consequences for them. Moreover, accountability policies are based on common policy tools (standards, assessment and testing tools, accountability mechanisms, consequences, incentives, sanctions, and rewards), even if the features and range of instruments used for accountability purposes can vary widely from one education system to another.

These typological approaches were very useful for us in identifying the constitutive properties of accountability policies in education, as mentioned above, to reflect on the specificities of our cases and to better position our contribution to the international literature (for instance, by providing a multilevel analysis whereas, in this academic literature, new accountability systems are often related to macro and global changes). Nevertheless, if these typologies remain the main purpose of the research and if they are simplified in the analysis, they can indirectly and paradoxically contribute to impoverishing the interpretation of isolated cases and the lessons drawn from them. For instance, is the French accountability policy only “reflexive?” Is it sufficient to define France as a low-stake accountability system? Is France bound to be regarded as an exception or a marginal case or can we learn other things from its analysis? On the other hand, is the Quebec accountability policy only defined by neo-bureaucratic features related to the successive bills on RBM?

That is why we have considered this first body of literature as a strong invitation to go beyond the typological way of thinking about accountability policies. We will paint a more complex picture, taking into account the mediations and local uses of the policy tools, as well as their evolution related to the trajectories of the policies.

3.2.3 The Macrosocial and Global Sources of Transformations of Modes of Governance and Regulation of Education Systems

Accountability policies in education cannot be isolated from other public policies (such as decentralization, school autonomy, and school choice) adopted in many countries and education systems in the last two or three decades. These reforms of school governance and regulation should be related to major evolutions subjecting governments to pressures or demands, in an at least partially converging sense, especially in the OECD industrialized countries (Ball, 1998).

  • The development of economic globalization has accentuated business demands for greater efficiency in public education systems but also for greater attention to the economy’s needs for particular skills. In this context, discourse about the needs of the “new knowledge economy” has led to an emphasis on the need to improve the public education system’s effectiveness and efficiency (Brown, Lauder, & Ashton, 2008).

  • The welfare state’s crisis of legitimacy and funding and the rise of neoliberal political paradigms have raised questions about bureaucratic modes of managing public action and have led to an adoption of managerial concerns, heretofore characteristic of the private sector (preoccupations with efficiency and accountability), into the public sector. Indeed, the principles of New Public Management (Hood, 1996) have had a major influence on public administration reforms and the development of decentralization and accountability policies in education. Nevertheless, it should be noted that these models have been implemented and recontextualized in various ways (see below, Chap. 5).

  • An increasing social demand on the part of the middle classes has also emerged, favoring more quality, choice, and the individualization of education pathways. Aside from the influence of the increasing individualization of social ties, this demand has its source in middle-class anxiety in the face of the erosion of their social and professional positions (Ball, 2003a).

  • There is also a phenomenon of globalization of education policies, at least in the form of the diffusion of reference models by various bodies, feeding the construction of new policy tools (in particular NPM and diverse post-bureaucratic models of governance (Maroy, 2012). Such models sometimes serve to inspire and sometimes to legitimize the construction of national policies, notably through the circulation of ideas favored or initiated by transnational organizations (OECD, European Union, etc.) and policy networks (Ball, 2012; Rizvi & Lingard, 2000).

More broadly, a number of studies have attempted to characterize the direction and meaning of the ongoing changes in contemporary education policies and the factors and processes underpinning these changes, in connection with the development of a performance-based accountability policy. Thus, several researchers have insisted on the shift from government to governance and the development of data-based governance tools (Normand & Derouet, 2009; Ozga, 2009). Others stress changes in normativity and theorize about the emergence of an “obligation of results” (Demailly, 2001; Lessard & Meirieu, 2008) and a new moral “performative” economy which affects expectations toward education professionals (Ball, 2003a).

English-speaking literature emphasizes the rise of a neoliberal logic in the governance of education (Whitty, Power, & Halpin, 1998) that of an “Evaluative State” and accountability (Broadfoot, 2000), the globalization of educational governance, and the interpenetration of scales in an increasingly complex multigovernance (Dale & Robertson, 2002) or the changing nature of the state (from the Keynesian welfare state to a Schumpeterian workfare state; Ball, 1997).

Therefore, these works offer partially divergent and complementary interpretations of the reasons behind these transformations. These reports of accountability policies are interesting and heuristic but difficult to test empirically or to hierarchize. These works usually look at the macroscopic level or favor an analysis of the “discourse,” referentials, or transnational policy paradigms which underlie the evolution of education policies. They do not document their concrete and progressive implementation in the systems concerned, which often involve significant hybridization and bricolage (Campbell, 2004; van Zanten, 2008).

Now, only such a detailed analysis of the construction and implementation of these policies in concrete education systems can reveal their recontextualization, the plurality of logics at work, and their tensions and inconsistencies, once these policies are examined over time and through the complexity of the mediations and usages at the intermediate or local level. Our research strives to achieve such an analysis, based on the sociology of public action, attentive to the diversity of actors and levels which contribute to the fabrication of public action (Commaille, 2006) while looking closely at the institutional processes which might condition or filter the action.

In other words, these external or internal evolutions and demands propelling governments toward the development of performance-based accountability systems do not lead to a single model of accountability policies. This is due to many factors that we will examine in this book, especially in Chap. 5. First of all, transnational models of accountability policies are recontextualized and translated into national policies in various ways. The international model could be subjected to idiosyncratic normative and cognitive “bricolage” that adapts and hybridizes the “pure model” into more legitimate models in specific societal contexts (Steiner-Khamsi & Waldow, 2012). Moreover, there are political struggles concerning the model to put in place. Finally, the actual policy trajectory of accountability policies is often conditioned by the socioeconomic context of the country and the path dependencies to societal or local institutions (Streeck & Thelen, 2005).

3.2.4 The Econometric Evaluation of the Effects of Accountability on Academic Performance

A number of works, drawing on an econometric and systemic analysis of school systems, aim to highlight the impact of accountability devices on student results.Footnote 4 While the rhetoric surrounding the development of accountability policies brings up their supposed positive impact on effectiveness and equity in education systems, evidence from the research is rather controversial and not so clear-cut.

How effective are they in terms of improving performance? To what extent can they result in more equity in the school system? In order to have a more comprehensive picture, we will also discuss some unintended effects of high-stakes accountability policies highlighted by this literature. As we will see, it is difficult to give a clear-cut answer to these questions. In this regard, we agree with Lee’s (2008) recommendation: “(…) educational policy makers and practitioners should be cautioned against relying exclusively on research that is consistent with their ideological positions to support or criticize the current high-stakes testing policy movement. They should become aware of potential biases arising from the uncertainty and variability of evidence in the literature” (p. 629).

Thus, we present the conclusions of extensive literature reviews and longitudinal or macro statistical studies based on the Federal National Assessment of Educational Progress (NAEP) in the USA, state-driven tests, district-driven tests (Chicago, New York), and meta-analyzes that try to isolate and discuss the impact of high-stakes accountability systems on student performance and achievement gaps.

3.2.4.1 Effectiveness and High-Stakes Accountability Systems

3.2.4.1.1 Slight Positive Effects on Student Performance but Variable and Unstable Over Time

The main findings of US econometric studies suggest that high-stakes accountability systems, particularly “more stringent accountability systems” (Harris & Herrington, 2006), have a positive effect on student achievement (Figlio & Loeb, 2011; Hanushek & Raymond, 2005; Harris & Herrington, 2006; Jacob, 2005; Lee, 2008). Yet this effect remains moderate (Lee, 2008). Moreover, there are important variations with respect to school grades and disciplines (Carnoy & Loeb, 2002; Chiang, 2009; Figlio & Loeb, 2011; Jacob, 2005; Lee, 2008; Mons, 2009). Achievement gains are greater for higher school levels, possibly because these grades are offered the largest incentives in order to improve student performance (Jacob, 2005, p. 772). They also tend to be greater in mathematics than in reading or other disciplines tested (Jacob, 2005; Lee, 2008; Treisman & Fuller, 2001 in Mons, 2009; Dee and Jacob 2009 in Figlio & Loeb, 2011).

Furthermore, achievement gains do not remain stable over time (Chiang, 2009; Lee, 2008; Mons, 2009). In this respect, as pointed out by Lee (2008, p. 619): “the volatility of gain scores requires that one look at changes in performance over the long run.”

3.2.4.1.2 Varied Results Depending on Specific Mechanisms and Tools

The impact on student achievement also differs depending on the specific mechanisms and tools at the heart of accountability systems. Three main instruments and mechanisms are usually studied: (1) the use of information for accountability purposes at the school level (Bruns, Filmer, & Patrinos, 2011), report cards, publication of test results, and ranking of schools; (2) the introduction of high-stakes testing; and (3) the system of incentives, sanctions, and rewards targeted at different levels (teachers, schools, and district levels).

Considering the effect of report cards (the publication of information regarding school results, possibly with student characteristics and breakdowns by subgroups), the assumption is that publishing student and school results allows the public to identify low-performing schools and districts and possibly leads to better performance. The “scarlet letter” effect “would suggest that educators wish to avoid and will respond to stigmatization regardless of other incentives” (Harris & Herrington, 2006, p. 220). However, analysis of the core literature leads to mixed conclusions: as an example, Bishop et al. (2006 in Harris & Herrington, 2006), Hanushek and Raymond (2005), and Carnoy and Loeb (2002) found that the gains in NAEP are greater for states that use report cards, but Harris and Herrington (2004 in Harris & Herrington, 2006) found no effect. Finally, it is difficult to reach a definitive conclusion on the positive effect of report cards on achievement gains.

Conversely, high-stakes testing, particularly promotion and graduation exams (PGE) and state high school exit examinations (HSEEs), seems to have a positive impact on student achievement (Carnoy & Loeb, 2002; Harris & Herrington, 2006; and Neill 1998 in Grodsky, Warren, & Kalogrides, 2009). For Harris and Herrington (2006) indeed, PGE is a “key player” in enhanced student achievement.

Research consistently shows that incentives, sanctions, and rewards for schools and individuals attached to high-stakes testing may have the potential to increase student achievement. This is the conclusion of system-wide studies carried out by Chiang (2009) and Reback (2008) (as well as Hanushek and Raymond’s (2005) cross-state analysis and Harris and Herrington’s (2006) literature review).

Nevertheless, there is strong evidence that the introduction of sanctions linked to high-stakes testing has also led to the development of certain strategic behavior at the school or individual levels. Consequently, many authors have expressed concerns about this mechanism.

3.2.4.2 Equity and High-Stakes Accountability Systems

One of the key objectives in the introduction of high-stakes accountability systems in the USA is also to reduce the achievement gap between students in a context of strong achievement inequalities (Carnoy & Loeb, 2002; Harris & Herrington, 2006), where underperforming groups are traditionally subgroups of students “defined by their race, income and disability status” (Figlio & Loeb, 2011, p. 395).

3.2.4.2.1 A Plausible Negative Impact with Variations Depending on Social and Ethnic Subgroups

Leaving aside some discrepancies among study conclusions, a consensus emerges regarding the lack of effect of high-stakes accountability policies to narrow the achievement gap (Figlio & Loeb, 2011; Hanushek & Raymond, 2005; Harris & Herrington, 2006; Lee, 2008). There is little evidence that accountability policy has a positive impact on equity. On the contrary, it even tends to increase the performance gap between schools and students, mostly between advantaged and disadvantaged. Hanushek and Raymond’s (2005) conclusions show that the black-white achievement gap has widened since the introduction of high-stakes accountability systems, while Dee’s (2002 in Carnoy & Loeb, 2002) conclusions highlight “reductions in educational attainment, particularly for black students.”

In a market context, with competition between schools, high-stakes accountability mechanisms also tend to increase social and ethnic segregation between schools (Hanushek & Raymond, 2005) and, as a consequence, the stigmatization of low-performing schools and students. Studies have shown that such mechanisms lower the capacity of low-performing schools to attract and retain highly qualified teachers (Figlio & Loeb, 2011; Finnigan & Gross, 2007; Wolf & Janssens, 2007) and consistently increase staff turnover (Figlio & Loeb, 2011). Such evidence matters if we assume that teacher excellence is one of the key factors in the quality of schooling and that school segregation is negatively correlated with equity in education systems (Demeuse & Baye, 2009; Dumay, Dupriez, & Maroy, 2010).

3.2.4.3 Strategic Behaviors, Gaming Practices, and Unintended Effects

Many authors have also highlighted the fact that high-stakes accountability systems, particularly sanctions and rewards, lead to the development of strategic behaviors and provide incentives for actors (at the state, district, school, or individual level) “to game the system” (Heilig & Darling-Hammond, 2008; Lee, 2010). Under accountability pressure, actors tend to adopt gaming strategies to artificially improve test outcomes. These gaming strategies could range from the falsification of student and school results and the exclusion of low-performing students from testing and from school to the increase of student retention in lower grades, as well as the placement of weaker students in special needs education or “limited English proficiency” categories, resulting in their ineligibility for high-stakes testing (Figlio & Loeb, 2011; Ladd & Lauen, 2010; Linn, 2000; Mons, 2009; Webb, 2005).

In a context of high-stakes testing, “given the consequences attached to test performance in certain subjects, one might expect teachers and students to shift resources and attention toward subjects included in the accountability program” (Jacob, 2005, p. 786). The very well-documented phenomenon of “teaching to the test” consists of concentrating learning activities and taught contents on test preparation. It leads to a focus on short-term learning outcomes and a reduction of the taught curriculum, particularly for low-performing students. The shift could also result in increased attention paid to students who are more likely to improve their performance, the ones just below proficiency level (Ladd & Lauen, 2010). Moreover, the threat of sanctions also affects teacher motivation and increases teacher and student stress (Finnigan & Gross, 2007; Mons, 2009). These collateral effects of accountability policies could lead to a downward cycle.

In conclusion, we can say that finding strong evidence of the efficacy of these accountability policies remains problematic since (1) empirical studies are conducted from various theoretical and epistemological perspectives and (2) most comparative evaluation studies focus on the effects of high-stakes accountability systems, mainly in the USA. Moreover, because isolating the effects of system-wide reforms represents a methodological challenge, studies often focus on key-related tools such as high-stakes testing and incentives for actors; and (3) fewer studies have looked at soft accountability or reflexive accountability systems.

Furthermore, a number of these studies focus on the officially expected effects in the main subjects for which external “tests” exist (mathematics, sciences, and mother tongue); and some of the positive results of these policies could be related to unexpected or perverse effects of strategies to game the system adopted by students, teachers, or managers in school authorities, in a context of “high-stakes accountability” (Mons, 2009).

More broadly, while these studies identify some effects, they are incapable of theorizing about the processes through which performance-based accountability policies produce these changes in schools and classrooms. In other words, their only theory to explain the action of students or teachers is narrowly utilitarian and economic. Moreover, they assume that the same accountability policies are applied uniformly in all regions and localities of the same school system, although it would be important to contrast the policies according to their theories of action and their instrumentation and, above all, their local implementation.

Indeed, an entire research trend in the analysis of public action in education has shown that local interpretations and translations of a national policy at the intermediate level deserve attention (Cattonar, Lessard, & Maroy, 2010; Lessard & Desjardins, 2009; Maroy & Demailly, 2004). The same can be said for micro-social processes which contribute to shaping the local reception and enactment by a SB, a local administration in a French académie, or a school. Policies are not only constructed at the central level. Local and intermediate actors, both collective and individual, matter (Coburn, 2001). This activity of “sensemaking” of reforms is not only influenced by actors’ representations, as well as their cognitive and normative frames (Spillane, Reiser, & Reimer, 2002), but also by organizational and social contexts: the modes of leadership and collaboration in the schools (Dumay, 2009) and the social and academic composition of schools. We will now turn to this type of research.

3.2.5 Analysis of the Implementation and Local Reception of Accountability Policies

These studies concentrate on the implementation of accountability policies. They draw on a variety of theoretical frameworks (micro-political, neo-institutionalist, or socio-cognitive) and adopt approaches that either claim to be neutral or, based on more normative stances, with critical perspectives or on the contrary, to advocate decision-making and implementation. These works develop a sufficiently broad framework to account for the processes by which accountability policies produce their effects at the local or intermediate level and, in this sense, go beyond econometric studies.

Thus, the policies of external evaluation and steering by results (pilotage par les résultats) were analyzed, especially in France, from a micro-political perspective focused on the power relationships among actors (Ball, 1987; Friedberg, 1993), because “evaluation is not merely a technical practice; it is a strategic and political practice” (Demailly, 2003, p. 116). Indeed, there are stakes for local actors who experience the advantages and disadvantages of the implementation and effects of such evaluation policies. Consequently, the implementation of evaluation and accountability devices is analyzed as a process of negotiation inscribed in power relationships. Thus, Demailly, Gadrey, Deubel, and Verdière (1998) observed the implementation of an “audit” within the académie of Lille (France) and describe the use of evaluation by teachers, school heads, and inspectors. Barrère (2006) approached this topic from the viewpoint of school heads. Kelchtermans and Vandenberghe (1998) also applied a “micro-political” approach to the use of evaluation in schools in Flanders. Actors have multiple interests (material, professional, organizational, and cultural) which should be taken into consideration in order to understand why an inspectors’ audit is well received or rejected. Such strategic studies of the reception of performance-based accountability policies allow for an analysis of collateral and unexpected effects on schools or their professionals. Power relationships are sometimes embedded in multiple interdependent networks that exist among intermediate actors. Such configurations of local public actions largely predetermine the form and intensity of the enrolment power of accountability instruments which are implemented, for example, in French académies (Buisson-Fenet & Pons, 2012).

Studies on the reception of soft accountability policies have emerged recently (in France, Belgium, and Switzerland). Following what has been labeled as the “implementation turning point” (Lascoumes & Le Galès, 2007; Lessard & Carpentier, 2015; van Zanten, 2014), they focus on teachers and management at the school level. The heart of the question is often the nature of teachers’ experience and their relationship to reforms, external evaluation, and accountability (Cattonar, Dumay, & Maroy, 2013; Yerly, 2014). In addition, studies also examined school heads’ strategies, their use of new policy tools, their adoption or rejection of new pedagogical roles toward teachers (Barbana, Dellisse, Dumay, & Dupriez, 2016; Barrère, 2006; Barrère, 2009; Dupriez & Malet, 2013), the analysis of public controversies around external evaluation practices and their local repercussions (Dutercq, 2001; Dutercq & Lanéelle, 2013), and, more broadly, the effects of New Public Management on professionals (Demazière, Lessard, & Morrissette, 2013).

More rarely has the role of actors at the level of intermediate regulatory bodies been investigated, especially to demonstrate that voluntary and incremental enrolment of schools can be more effective than bureaucratic coordination in fostering efficient implementation of the policy, in particular from the perspective of pedagogical monitoring by school heads following external evaluations (Dumay, Cattonar, Maroy, & Mangez, 2013; Maroy, Mangez, Dumay, & Cattonar, 2012).

Generally, these French studies assume an axiologically neutral stance, even if their relationship to the reforms and policies may vary. A number of more critical works have also emerged in recent years (Garcia & Montagne, 2011). Sometimes claiming to stem from a new “critical sociology,” they tend to consider accountability policies as proof of the submission of education to the principles of neoliberalism, according to which the state itself acts at the behest of the market. This submission would be accompanied by a mishandled education institution’s internalization of the social norm of capitalism, according to which schools must now produce students equipped with the skills expected by the labor market, rather than train the autonomous citizens of tomorrow (e.g., Laval, Vergne, Clément, & Dreux, 2012). Another approach consists of revealing the phenomena of domination at work in New Public Management, as well as the perverse effects of this doctrine, on the basis of empirical studies or through more philosophical reflections.Footnote 5

A very great number of studies on the implementation or the enactment of various accountability policies have been conducted and published in the English-speaking literature.Footnote 6 Therefore, we will restrict ourselves to merely indicating broad trends, without undertaking a comprehensive literature review (for a review, see Falabella, 2014; Verger & Parcerisa, 2017).

North American studies analyze diverse strategies for implementing accountability policies, with a focus on the intermediate level (school districts and intermediate bodies), on school management, or on the reception and cognitive and practical appropriation of these policies by teachers.

Therefore, the roles of intermediate levels, of states or school districts and their managers in the management and monitoring of change expected by US federal policies—in particular the No Child Left Behind Act, Race to the Top, or Comprehensive School Reform Program—have been the subject of numerous studies, especially from the perspective of improving their efficacy (Shannon & Bylsma, 2004). Thus, Sykes and colleagues stress the multiple roles (policy initiators, interpreters, and enactors) that districts can play in the implementation of policies and the improvement of students’ learning (Sykes, Schneider, & Plank, 2009), while others raise questions about how individuals in districts make sense of evidence-based practices promoted in schools or districts by current US policies (Coburn & Talbert, 2006). Amanda Datnow stresses the varying coordination of a number of policies and levels of power (federal, state, and district) and its impact on the implementation of accountability policies, especially as concerns “underperforming” schools. She shows that the desirable “co-construction” of policies by these different levels of power may not happen (Datnow, 2006; Datnow & Park, 2009). From an analysis of micro-policy games, Betty Malen notes the diversity in school districts’ strategies for implementing a governmental policy (Malen, 2006). James Spillane and colleagues also stress the importance of the “sensemaking” of reforms by school district managers and their variable significance (Spillane et al., 2002; Spillane, Reiser, & Gomez, 2006). Meredith Honig draws theoretical conclusions from such studies, highlighting the complexity of policy implementation and the necessity of adopting a contingent and dynamic approach which simultaneously takes account of the local contexts, the actual populations concerned, and the content of the policy (Honig, 2006).

Moreover, the analysis of the role of school heads in the implementation of reforms has also been widely analyzed, especially in research on “school effectiveness” and “school improvement.” In particular, through either normative or empirical analysis, attention is brought to the exercise of instructional or transformational leadership which principals are supposed to be developing in a context of increasing accountability (Leithwood (2001), Hallinger (2003), and Scheerens (2012)). Thru the presentation of a large range of national case studies (Asian, European, Oceania, American countries), Easley II and Tulowitzki (2016) are questioning national accountability policies from the point of view of the challenges and possibilities they offer for school leadership. Other studies question whether accountability policies disseminated by inspectorates contribute to improving the functioning of schools and of leadership (Altrichter & Kemethofer, 2015). The role of school heads as mediators between external expectations or pressures (quality, accountability, competition) and internal context of schools has been emphasized in the US context (Spillane & Kenney, 2012), in Nordic countries (Moos, 2009; Skedsmo & Mausethagen, 2016), in Chile (Weinstein, Raczynski, & Hernández, 2016), and in the European context (Ball & Maroy, 2009). The process of sensemaking by school leaders has been emphasized (Spillane et al., 2002).

More recently, it is the mediation of organizational instruments and routines (Spillane, 2012) put in place at the school level that are the focus of attention, in terms of their effects on the process of change and policy implementation. More specifically, databases and their uses are the center of increasing attention (Coburn & Turner, 2012; Spillane, Parise, & Sherer, 2011). Indeed, the role of “data infrastructure” and “social technologies” (as large-scale assessment and accountability systems) is also more and more recognized as key drivers of change within various global or national policies (Sellar, 2015; Imsen, Blossing, & Moos, 2017, for Nordic countries).

The context of the school is also the object of studies underlining the differentiated effects of accountability policies on the response of the schools, according to their status (in probation or well performing) (Diamond & Spillane, 2004; Mintrop, 2004; Mintrop & Suderman, 2009 in US context) or the market position or the intake of the school (in Chile, Falabella, 2014; in Francophone Belgium, Barbana et al., 2016).

Some recent research conducted by Braun, Ball, and Maguire has both an empirical and theoretical goal. Conducting four school case studies in England, they aim to produce “a theory of the policy enactment,” showing the active role of local actors in the interpretation and the translation of competing policies, among them accountability policies. Beyond the active “interpretation” of discourses by local actors, they insist on the translation of the policies into moving local texts, tools, and artifacts. Various positions of actors and the role of local school contexts (intake, professional, institutional) are also underlined (Ball, Maguire, Braun, & Hoskins, 2011; Braun, Maguire, & Ball, 2010; Braun, Ball, & Maguire, 2011; Braun, Ball, Maguire, & Hoskins, 2011; Maguire, Braun, & Ball, 2011).

As for the most critical works on accountability policies (in the USA, the UK, or Europe), they underscore the significance of the implementation and monitoring of “instruments of surveillance” (Bushnell, 2003; Ranson, 2003) and of regulation by numbers (Ball, 2015; Grek, 2008). These devices are based, to varying degrees, on hierarchical and vertical relations or may be more widely distributed among all the actors. For instance, certain studies highlight an approach whereby inspection bodies stigmatize underperforming schools (Thrupp, 1998), while others stress the stakes and symbolic violence at play in the more insidious process of collective self-evaluation from which schools cannot escape (Ozga, 2009; Ozga & Grek, 2012).

A number of studies have also examined the effects of accountability policies on teachers’ professionalism, in particular, the usage by teachers of evaluations of their students’ results, and their actual effects on pedagogical practices. In a qualitative meta-synthesis of studies in the high-stakes US context, Au shows a predominant trend toward the contraction and fragmentation of the taught curriculum and also teacher-centered pedagogy (Au, 2007). Other literature reviews stress negative impacts as teaching to the test, and cheating, and, to varying degrees, tighter coupling (or decoupling) of institutional expectations and practices (Hellrung & Hartig, 2013; Maier, 2010; Rozenwajn & Dumay, 2014). Other studies focus on the redefinition of the roles and identities which institutions expect of teachers but also of school leaders, both in North America (Anderson & Cohen, 2015; Hall & McGinity, 2015) and in the EU (Ball, 2003a; Braun et al., 2010; Czerniawski, 2011; Evetts, 2008; Gewirtz, Mahony, Hextall, & Cribb, 2008; Muller & Hernandez, 2010; Osborn, McNess, Broadfoot, 2000). Similar research is emerging in the Francophone European context (Dupriez & Malet, 2013; Dutercq & Maroy, 2017).

This type of analysis emphasizes the normative transformations of teaching professionalism and, more broadly, shifts in the meaning of local educational action resulting from the interplay of discourses from central authorities, as well as artifacts for the local enactment of tools of governance (Maguire et al., 2011). The transformations of actors’ social relations and identities are here considered as the product of shifts, both cognitive and normative, which affect actors’ possible and legitimate fields of action and, therefore, their ability to act in their daily professional lives.

Therefore, these studies all have something to contribute since they underscore the need to analyze the enactment of accountability policies to enhance our understanding of their modus operandi. They attach great importance to the strategic, cognitive, and normative dimensions of the processes at work in the implementation of performance-based accountability policies, taking also into account the school’s contexts. Moreover, the effects investigated concern the instrumental aspect and efficacy of school systems and organizations (especially with the question of coupling/decoupling) in the case of research oriented toward an effective improvement of the school (and student learning), while more critical research, or that inspired by neo-institutionalist sociology or the sociology of professional groups, emphasize the transformation of the institutional foundations of education and the transformation and reshaping of professional identities that these transformations entail for education personnel. More broadly, this critical approach also examines the political and policy significance of new modes of governance that these policies entail. In fact, this research stresses a key dimension of the analysis: evaluation or accountability practices involve political conceptions of governance of the school system, visions of the management of a local school, and normative conceptions of education in general (goals, means, and conceptions of justice and of efficacy) which are enacted by the discourses and practices of agents responsible for their implementation.

Nonetheless, these studies suffer from a twofold limitation. The first is that they are usually centered on a single level of analysis (intermediate or local) and tend sometimes to be satisfied with a summary presentation of the content of the policy and of the institutional environment under analysis. This could either result in exaggerating the changes under way (or underestimating them) or in attributing most of the process of change to a single level of analysis and action, thus neglecting part of the picture.

Another shortcoming is that some of them adopt a normative stance—either critical or one of problem-solving to help improve the decision-making, which tends then to either accentuate the effects of certain transformations in order to denounce them or minimize the change and focus attention on all the obstacles to change. The analysis of the sensemaking processes or the mediations of organizational tools or routines necessary for successful policy implementation could be flawed in being to some degree “value-free.” They study cognitive and institutional mediations concerning the policy implementation, without always asking about the political, social, and educational effects or meanings of the changes at play.

3.3 The Theoretical Ambition of a Multilevel Approach

3.3.1 Challenges and Analytical Issues

Our overview of the literature on the issue has brought out the fact that research on changes in modes of governance at the global level has remained relatively macroscopic and centered, above all, on paradigms inspiring the policies. The weakness part of this research lies in the reification or simplification of the processes of change which are apprehended as top-down movements while overlooking the mediations at work at all levels—especially at the intermediate and local level—and the various paths to effective change of organizations and education institutions. Sometimes generalizing from particular national cases, especially cases from English-speaking countries, they are also not sufficiently attentive to the empirical diversity of current policies, which existing typologies nevertheless serve to highlight. The latter, on the other hand, have a static character which tends to neglect or overlook the processes at work in policy paradigms’ shifts, or the transformations of instruments and implementation processes.

As for the qualitative studies, they contextualize and complexify the analysis of the implementation/enactment of these policies, emphasizing the role of local actors, the sensemaking of school leaders and teachers, and the role of schools’ contexts on the attended or unattended effects of the policy. They underline the effects of the policies on pedagogical practices or curriculum or on the professional identities or social relations within schools. However, they are usually limited to the most local reception of the policy or are concentrated on a single level of change, even though some researchers try to articulate theoretically the interplay of various levels of analysis (Braun, Ball, & Maguire, 2011; Braun, Ball, Maguire, et al., 2011). Furthermore, they are rarely comparative (Muller & Hernandez, 2010 or Maier, 2010 being exceptions) and do not allow us to consider variations in these policies and their forms in different educational and policy contexts.

For their part, critical studies are sometimes flawed in seeking to illustrate on a local level a dynamic of macrosocial transformation (the discourse of neoliberalism, of New Public Management, etc.) which is then overestimated. Thus, an entire literature insists on multiple turning pointsFootnote 7 affecting the world of education while overestimating the homogeneity of the local impact or unequivocal character of such turning points.

Finally, econometric studies shed no much light on the modus operandi and conditions of the legitimacy and efficacy of different types of accountability policies, and they often remain quite controversial when it comes to their effects, in terms of both equity and efficacy.

Our research project aims to go beyond these limits and offer a comparative analysis of the implementation processes of performance-based accountability in the French and Quebec secondary school systems since the early 2000s. The comparison will allow us to highlight the role of national, intermediate, and local mediations in shaping the direction (orientation and meaning) of these policies.

We intend to propose a sufficiently generic theoretical framework that will be applicable to different empirical contexts, allowing all possible flexibility, to be able to take into account any particularly strong dimension (e.g., the professional dimension of the instrumentation in France). This framework is structured so as to consider a number of levels of analysis.Footnote 8 Thus, it aims to go further than static and typological analyses, emphasizing trajectories, in which factors behind inertia or path dependencies, as well as factors and actors behind change—both in terms of institutions and of the actors in the game—are all taken into account.

3.3.2 A Plural Toolbox

We use a plural theoretical toolbox for the analysis of education policy (making and implementation), combining two main perspectives.

The first is the French sociologie de l’action publique which tends to emphasize actors’ games at various levels and scenes that matter for the making and implementation of policy, underlining that these actors have various interests, sources of power, and identities, that they could interpret and make sense of the policy in various ways, and that in the policy process, political processes—impositions, compromises, struggles, etc.—are very important. Being sometimes regarded as the French version of policy analysis in political science (Halpern, Hassenteufel, & Zittoun, 2018)—even if it is developing in several European countries—this sociology of public action integrates many different theoretical approaches of the policy process, which all have in common an extension of policy analysis from the action of governments to the numerous policy actors contributing to producing public action. This shift led French researchers to profoundly ground their analysis in the sociological tradition. Initially strongly focused on cognitive and organizational processes (Smith, 1999), this rapidly diversified from the 1980s and, since the end of the 1990s, covered a growing number of policy sectors such as education (Buisson-Fenet, 2007; Commaille & Jobert, 1998; Maroy & Doray, 2008; Muller, 2000; van Zanten, 2014).

The second one is the North American neo-institutionalist approach, either organizational or sociohistorical. It stresses that institutions—understood broadly as rules, norms, and cognitive frames (Scott, 1995)—matter, due to the obstacles or barriers that they can represent for certain policy solutions or political games while at the same time supplying resources for action. Institutional arrangements are constraints, resources, and objects of public action.

The tension between these two analytical traditions,Footnote 9 which we hope proves productive, has allowed us to identify three research issues in order to take account of variations in performance-based accountability in each education system through a comparative analysis of (1) the trajectory of these policies; (2) mediations in their national, regional, or local construction; and (3) policy tools used to operationalize the public action or policy. These objectives lead to three conceptual and empirical entry points for investigation (trajectories, mediations, and instrumentation) which are developed in the following Sect. (3.3.1). More specifically, as concerns the trajectory of these policies, we have paid close attention to the internal processes of the political systems (cognitive and policy bricolage; political games and alliances but also institutional processes) which have affected the national translation and hybridization of transnational models (Maroy, 2006; Steiner-Khamsi, 2004), in each society’s particular, historical context. The analysis of mediations conditioning the implementation of performance-based accountability draws on Francophone research concerned with the usage and local reception of external evaluation policies and studies giving due importance to strategic and normative dimensions of local actors’ behaviors. We will also refer to North American theories concerning the “implementation of educational policies” (Honig, 2006), with an emphasis on micro-political theories (Malen, 2006) and a neo-institutionalist framework, which theorize the conditions, stages, and actors engaged in the processes of deinstitutionalization/reinstitutionalization of organizational or pedagogical practices (Greenwood, Suddaby, & Hinings, 2002; Powell & Di Maggio, 1991) and their conditions of local legitimization (Suchman, 1995). Finally, the analysis of instrumentation is based on a plural sociology of policy instruments (Lascoumes & Le Galès, 2004; Pons, 2010). Indeed, accountability policies are strongly equipped and based on tools which appear as techniques: tests, indicators, “practical guides,” but also “conventions” and “contracts” between the partners of public action. Now, all these tools must not be considered merely from a functional perspective, reducing them to neutral techniques or management tools, but as components of public action. Certain of these tools (indicators and quantified evaluations of school results) constitute what Pons and van Zanten (2007) refer to as “knowledge-based regulation tools” which, in the field of education, become instruments of “governance by numbers” (Ozga, 2009). These tools should be analyzed in their constitutive dimensions (content, cognitive, and normative orientations) but also in their uses by actors at different levels of action (Akrich, Callon, & Latour, 2006).

3.4 Three Conceptual and Empirical Entry Points

Our theoretical framework is based on three concepts, the construction and stabilization of which have required drawing upon various analytical notions and traditions. Each conceptual entry point allows us to further our study and comprehension of the other two. We could even contend that, according to the questioning chosen, each is liable to subsume the others: for example, it is necessary to work on the processes of mediation and instrumentation to grasp the trajectory of a policy of accountability; it is difficult to interpret the processes of mediation at work without situating them in the trajectories of public action deployed at different levels, since the researcher risks forgetting an important dimension of the analysis of the processes of instrumentation of accountability at the school level if account is not taken of mediations at work at all system levels (e.g., the policy choices of intermediate authorities, etc.). Thus, this theoretical framework is not conceived as a “succession” of three conceptual entry points but rather as their articulation, if possible, dialectical.

3.4.1 Trajectory

Our research first rests on the concept of education policy trajectories (Ball, 1994; Ball, 1997). This approachFootnote 10 aims to grasp the various processes and mechanisms by which networks of national and transnational actors piece together and reconstruct institutional and practical principles from the national repertoire, selectively and each time anew. Along with these principles, they may also choose elements from the transnational repertoire of ideas and public action tools, such as the “quasi-market” and “New Public Management” (Hood, 1991). From a methodological point of view, this public policy trajectory approach “trace(s) through the development, formation and realization of those policies from the context of influence, through policy text production, to practices and outcomes” (Ball, 1997, p. 266). The trajectory of public policy is, thus, observed within a given political context and school system, increasingly transnational in scope (Dale, 2006). Yet a trajectory is also comprised of effective policy orientations, from the policy’s gestation period to its daily, local implementation. These orientations are the product of multiple recontextualizations of the policy’s “text” by actors situated at several levels of action. The policy’s trajectory is, therefore, tied to its successive interpretations and to its outcomes, particularly in terms of social justice and equality.

We argue that a policy’s trajectory depends on the combination of three processes, often intertwined empirically but separate from an analytical standpoint: (1) path dependence on earlier choices, due not only to the viscosity of institutions (Pierson, 1994; Pierson, 2001) but also to actor mobilizations (Mahoney & Thelen, 2010; Pierson, 1994); (2) a bricolage construction, whereby education action is developed in the context of negotiations and struggles among actors; and (3) the interpretation, by certain national actors, of policy ideas and instruments circulating on a transnational level. These three factors help to explain the singularity of education policy at a time of globalization and transnational circulation.

First, public action in education is linked to specific, evolving domestic problematization and to the sedimentation of tools that may have various sources and are part of long-term transformations. As in other policy sectors, education policy construction is shaped by existing institutions: formal rules and shared norms but also ideas and cognitive categories (Scott, 1995). Paul Pierson’s groundbreaking 1994 research showed that the initial choices made when a policy is introduced restrict the breadth of choices available later, making marginal or incremental changes more likely than radical ones. This self-reinforcement of initial choices, or policy lock-in, results from institutional locks, such as formal decision-making tools or vetoes capable of blocking the decision-making process, which may anticipate the change. In addition to the powers of formal institutions, the political costs of change are another obstacle to its realization. It has been documented that voters are more sensitive to the costs of a given policy change—that is, to the losses to which a change may lead—than to a policy’s potential benefits. In this sense, politicians are more often dissuaded electorally from introducing policy change than the reverse. Finally, policy lock-in is also tied to actions of those in favor of conserving the status quo (Mahoney & Thelen, 2010). These actors are diverse in nature, coming from inside administrations as well as outside: users, beneficiaries, union representatives, etc. In sum, if public policy may be characterized by change, then choices made in the initial period of its implementation are likely to influence the definition of change trajectories and will play a role in shaping them (Pierson, 2001).

Next, when we look closely at development and change in public action, it becomes clear that domestic policymaking is often a continuous bricolage. For Claude Levi-Strauss (Lévi-Strauss, 1962, p. 27), the latter is made by handymen whose “instrumental universe is enclosed” and whose “universe of instruments is closed and the rules of his game are always to make do with ‘whatever is at hand’, that is to say with a set of tools and materials which is always finite and is also heterogeneous because what it contains bears no relation to the current project, or indeed to any particular project, but is the contingent result of all the occasions there have been to renew or enrich the stock or to maintain it with the remains of previous constructions or destructions.” In policy analysis, bricolage means that public action is a dynamic assembly of heterogeneous elements in their natural or evolving state; such elements are not necessarily designed to be tied together. Examples include existing institutional arrangements, institutional change incited by certain actors, new managerial tools, discursive formalizations, competing policy narratives, and ideas taken from repertoires and transnational reference sources. This composite construction is related to the strictly political process of struggle and negotiation inside different arenas (political, scientific, and administrative) or social spaces within civil society. In these spaces, public problems are defined and developed, and policy is meant to deal with them and offer “solutions.” Indeed, for Philippe Zittoun (2013), this bricolage is a key stage of the political process of policymaking, and it consists of performing five main coupling operations when providing a policy solution: designating the policy solution and its owners; identifying its consequences and its public; linking this solution to a pre-existing policy problem; including it in a policy which is expected to change; and associating it with guiding values and principles. Theories of change and guidelines for public action are developed, and these ideas are transformed into actual policy instruments. Actors from the political, institutional, and academic fields—or other “stakeholders” in the education sphere—hold sway in these forums (Fouilleux, 2000). Finally, this bricolage may involve a kind of innovation whereby existing elements are combined in an innovative fashion on the basis of tools, practices, discourse, or institutional principles and are driven by an instrumental logic of efficiency and/or by a symbolic search for legitimacy and social acceptance (Campbell, 2004). Bricolage may thus be cognitive, institutional, or political, depending on the elements it incorporates and assembles.

Lastly, in an increasingly transnational context, bricolage is not based on exclusively national repertoires of ideas, tools, or objectives. It is also based on the translation of ideas, narratives, instruments, and approaches discussed or introduced elsewhere and which lead to redefinitions or rearticulations in light of rules, values, conventions, and practices which have already been institutionalized (Callon, 1986; Campbell, 2004; Dale, 2006). These elements may be used voluntarily by certain national actors in a bilateral borrowing process or by learning from other systems (Dale, 1999), but additional processes may also be at work given growing transnational interdependence. Recontextualizations and translations of these tools and models may lead to “hybridization” (van Zanten, 2002), tied to the trajectory of public action and, more broadly speaking, to existing institutions (rules, norms, and cognitive categories).

Consequently, public policy is made up of compromises and “assemblages”—more or less stable arrangements that have been reworked, tweaked, and pieced together in the wake of complex processes of political actions, writing, dissemination, and translation, which end up in practice as something new (Ball, 1998). The trajectory of education policy—here, accountability policy—is, thus, the product of empirically intertwined dynamics of path dependence, bricolage, and interpretation. The resulting combination of change mechanisms and inertia makes way for trajectories which may lead to gradual but significant changeFootnote 11 or trajectories which tend toward marginal change (Streeck & Thelen, 2005). More rarely, the trajectory may also be marked by a decisive turn.

3.4.2 Mediation

The second concept that we draw upon is that of “mediation.” The fundamental idea behind this notion is that education policies, here accountability policies, are co-constructed on a permanent basis and at all levels of public action by different actors and organizations who participate in their conception and in their implementation. These actors and organizations transfer, translate, and contextualize these policies, depending on (1) the policy trajectories in which they are embedded and (2) the policy tools at their disposal—the other two concepts of our theoretical framework. In addition, mediation also depends on the “local educational orders” (Ben Ayed, 2009) in which they evolve and which themselves depend on (1) the institutional context and environment, (2) actors’ dominant professional ethos, (3) local problematizations of education issues and policies, and, finally, (4) local configurations of actors (interests, power relations, and multiple interdependencies).

So, we have to comprehend this idea of mediation in a dual sense: not only is each actor potentially an intermediary between others in a context of densification and complexification of public action, but, in addition, more fundamentally, each proceeds to a specific task of mediation which aims to orient the conduct of the actor’s interlocutors in various ways and, thus, produces a form of autonomous regulation in the system of public action (Reynaud, 1988). Christian Maroy has highlighted this on a number of occasions in his work on intermediary regulations in European school systems. In his view, these intermediary regulations “emanate from public authorities or steering bodies and/or networks of actors (private and/or public) which seek to orient the conduct of families, school principals or teachers, in various ways, often within a designated area. Their regulatory action can bear on diverse objects. […] These actions may be understood as control regulations which entail translations and transfers between the central regulations and the schools. They may also be considered as autonomous regulations” (Maroy, 2006, p. 18). Nonetheless, this reasoning may be applied to all levels of the school system since mediations are also at work at the central national level among different groups of actors, as well as, at the school level, among local actors.

The issue is then to understand accountability policies “as they are enacted” and the diverse logics behind the co-construction of public action by the various stakeholders (Datnow & Park, 2009). More specifically, for us, it is a matter of determining (1) how national policies fit more or less successfully in the pre-existing spaces of autonomous regulation and configurations of local public action; (2) the logics of mediation at work; (3) the role of routines (institutional, organizational, professional, cognitive, etc.) in the translation and implementation of accountability actions and tools; and, finally, (4) the arguments, ideas, and knowledge mobilized to effect these mediations. To this end, we have combined three complementary types of research.

First, a set of studies which we mobilize in a dual micro-political and institutionalist perspective has allowed us to conceptualize the logics behind institutional mediation at work during the implementation of performance-based accountability policies by agents of intermediate regulation (Maroy, 2006)—Quebec SBs and French académies—and by actors at the school level. We formulated two structuring hypotheses to examine how these agents and actors “respond” to “institutional change mechanisms” of which they were the main recipients, whether these mechanisms are coercive,Footnote 12 normative,Footnote 13 or mimeticFootnote 14 (DiMaggio & Powell, 1983). The first hypothesis considers that the institutional pressures for change lead to an isomorphism of intermediary bodies. However, the elements of similarity effectively resulting from such isomorphism remain to be further studied (e.g., isomorphism in the goals, operational objectives, tools, etc.). The second hypothesis considers that these pressures give rise to a diversity of mediation logics. Two typologies were especially used to account for this diversity: the typology developed by Boxenbaum and Jonsson (2008) which distinguishes five possibilities of strategic responses on the part of actors and organizations with respect to external institutional pressures (conformity or acquiescence, compromise, avoidance, defiance, and manipulation) and, to a larger extent, the typology developed by Betty Malen (2006), which describes four logics of organizational response to a policy (dilution, appropriation, nullification, and amplification).

Second, the research on “gaming” strategies of actors facing the introduction of performance indicators, especially Christopher Hood’s work (Hood, 2006; Hood, 2007 and Hood, 2012), has allowed us to position our case vis-à-vis different types of management by numbers (the target system, ranking system, and intelligence system), different behaviors typical of actors considered in the literature (“saints,” “honest triers,” “reactive gamers,” and “rational maniacs”), and, finally, different types of organizational culture (hierarchical, individualistic, or egalitarian).

While these works have allowed our research team to frame a collective reflection and have served in the investigation as valuable safeguards (in developing an interview grid, analyzing empirical material, etc.), they were not all drawn upon to the same degree, depending on the particular phases of the inquiry and the specific contexts under study.

Hood’s contributions were only mobilized to some extent for particular local situations in the empirical analytical phases of our cases (for intermediate entities or schools), for instance, when analyzing the gaming strategy deployed by a category of actors in an académie, in the implementation of a particular public action instrument. However, in the context of this research, they did not allow for a great degree of generalization or a positioning of the French and Quebec cases within the proposed typologies.

The micro-political and institutionalist approach, in contrast, clearly allowed us to position our cases (intermediate entities and schools) within broad analytical categories (logic of amplification, dilution, appropriation, etc.). In this sense, it has opened up horizons for analysis, in sometimes allowing unexpected groupings (e.g., a specific SB with a specific académie), leading us to delve further in our analysis of factors explaining certain observed mediation approaches. Yet it has not always provided sufficient avenues to fully account for these factors. Consequently, we had to supplement this analysis with investigations into the professional ethos of actors or local configurationsFootnote 15 of public action to understand the structuring of local educational orders.

3.4.3 Instrumentation

The third concept at the heart in this research is that of instrumentation. This stems from the notion of policy instrument which corresponds to an “identifiable method through which collective action is structured to address a public problem” (Salamon, 2002, p. 19) or to a “an apparatus that is both technical and social, that organizes specific social relations between the state and those it is addressed to, according to the representations and meanings it carries” (Lascoumes & Le Galès, 2004, 13). The notion of policy instrument appeared in political science thanks, in particular, to the works of Christopher Hood (1986) who distinguishes two broad types of tools: “detecting tools” which allow governments to collect information emanating from society and “effecting tools” which aim to influence the functioning of the latter.

The notion of policy instrument has given rise to various conceptions (concerning the definition of the instrument and its links to other similar notions, such as tool, technique, or even device)Footnote 16 and to a multitude of typologies which we will not develop here for two main reasons. On the one hand, these definitions and category distinctions are rarely logically consistent, and above all, they have little relevance for the professionals themselves. On the other hand, their heuristic value resides rather in the processes that they highlight: in particular the growing weight of these instruments (especially statistics), in a context of technologization of public action and pluralization of the high ideals and values structuring school policies and practices (Derouet, 2000).

More than the instruments themselves, it is then a matter of reflecting on the dynamics and effects of processes of instrumentation. The latter may be defined as “the set of problems posed by the choice, [the implementation] and the usage of instruments ([tools,] techniques, means of operating, and devices) which allow governmental action to materialize and be operationalized” (Lascoumes & Le Galès, 2004, p. 12). For these authors, instrumentation is likely to produce a number of effects on the course of public action such as effects of inertia, the process whereby these issues are turned into policy problems and, finally, a particular problematization of the latter. These effects echo the two other conceptual entry points which we have retained for our research (trajectory and mediation).

To study the instrumentation of accountability policies in France and in Quebec, we therefore had to:

  1. 1.

    Carry out a constitutive analysis of these tools and instruments, their evolution, their forms, and their content, especially from the point of view of the cognitive and normative frameworks that they incorporate (instrument morphologies).

  2. 2.

    Discern their usages, reappropriations, and reorganizations by the various actors concerned with their implementation at different levels of public action while asking ourselves to what ends, with what modalities of implementation, and with what forms of monitoring in managerial or pedagogical practices (careers of instruments) they are implemented.

  3. 3.

    Study the controversies surrounding the tools, especially where there are deviations from the norm, unusual uses, and debates on the choice of tools, their hierarchy, their usages, their effects, or the content of tools, to question the bases of their legitimacy (legitimacy of instruments).

  4. 4.

    Finally, analyze the social and cognitive effects on the internal social relationships and on the knowledge, qualification, and highlighting (zones of transparency or opacity) of educational, pedagogical, and organizational realities of the school (the social significance of instruments).

To do so, we made two main theoretical choices. First, we dismissed three approaches of instrumentation often criticized in the specialized literature (Lascoumes & Simard, 2011). The first one, normative, aims to define the modalities of an effective implementation of instruments and of “good” public governance. The second, functionalist, aims to choose among available instruments those which will permit the effective resolution of problems. The third, cybernetic, considers the instruments principally from the perspective of improving the capacities to control and regulate a system.Footnote 17

Then we deliberately opted for a pluralist and multifactorial approach to instrumentation, aiming to combine three main analytical traditions (institutionalist, cognitive, and professional), rather than develop one of them in depth, which usually involves focusing on one key explanatory variable, at the risk of minimizing the impact of others.

The first tradition of analysis is institutionalist. For Lascoumes and Simard (2011), this approach requires considering the instrument:

in its most informal, symbolic and cognitive dimensions. The instrument as an institution is […] approached from a perspective of analysis of power, of the shaping of social realities it involves, but also of the pedagogical and framing actions and sometimes the manipulation it entails. These works show to what extent these instruments have cognitive and behavioral control effects. This literature strives to retrace their history and their impacts based on discourses and most of all, on practices. From this perspective, sociological institutionalism leads to an emphasis on two phenomena. On the one hand, it considers the question of the cognitive frameworks, both general and specific, to which the instrument is related. The latter is embedded in a general ruler/ruled power relationship, which it establishes to ensure an operational regulation of particular sectoral domains. […] On the other hand, this perspective strives to characterize the development and the regular renegotiation of conventions on which the instrument is based. The analysis of the degrees and forms of this plasticity is based on an examination of the internal properties of the instrument (its technical and logical constraints), as well as that of the expected and unexpected effects arising from its appropriations by various actors. From this perspective, public action is a sociopolitical space constructed as much by instruments and techniques of regulation as by actors’ beliefs and strategies. (Lascoumes & Simard, 2011, p. 17)

This approach provides a number of advantages: it is consistent with our other conceptual entry points (trajectory and mediation) and with the general theoretical framework, and it allows us to draw productively and cumulatively upon a great number of conceptual tools.

Moreover, it can easily be articulated with a second related approach, the cognitive approach. The latter considers processes of instrumentation in the light of the properties and dynamics of the production of ideas, representations, images, knowledge, discourse, categories of public action, etc. from which they stem and which they contribute to redefining. Then, instrumentation is mainly seen as the result of political struggles between actors (organized in “coalitions,” “epistemic communities,” and so on) to define the ideas, representations, discourse, etc. which are dominant relative to this instrument or combination of instruments.

However, the combination of the institutionalist and cognitive approaches risks masking or minimizing the role of other types of explanatory factors of instrumentation, in particular, in school systems such as in France where, for a number of reasons, these policies are less formalized and codified. This is why we felt the need to combine these two approaches with a third tradition which brings to the fore the weight of professional groups (their identities, their skills, their modes of legitimization, and their struggles for territory or jurisdiction) in understanding local (non) usages and appropriations of instruments. In some education systems marked by neo-corporatism, the understanding of competition between professional groups may prove central to retracing more or less convergent dynamics of the instrumentation of an accountability policy (Pons, 2010). This perspective invites us to raise questions about the margins of maneuver provided by instrumentation to the different work collectives and “established” professional groups. Certain instruments may be inseparable from, indeed consubstantial with, the actors which in return they contribute to make exist and those who elaborate them and those who control their implementation. Moreover, we may go so far as to consider the emergence of “new professions” around taking charge of new specific instrumentation (such as statistical services) or at least take account of the reconfigurations occurring within established professions. In contrast, other instruments support a (re)distribution of power among professionals or at least their rearrangement (Buisson-Fenet & La Naour, 2008).

Let us make clear that these different approaches to instrumentation were drawn upon to varying degrees, depending on the particular case. For example, the approach emphasizing the professional dimension seemed more relevant in the French case. This option is consistent with our desire to adopt a pluralistic approach, so as not to neglect the specificities of each school system. Yet it involved choices in the presentation of the following chapters, to ensure a balanced comparison of cases.

3.5 Implementation of the Theoretical Framework

Our theoretical framework is based on three interrelated conceptual entry points (trajectory, mediation, and instrumentation) which are the focus of the analysis and which are each constructed from a combination of multiple theoretical approaches. The latter may sometimes converge strongly (as in the case of neo-institutionalist analyses) and sometimes much less except in a deep analytical deconstruction of the process of public action by the researcher.

This is of interest for at least two reasons. The first, empirical, is that of simultaneously complexifying and systematizing the analysis of the implementation of accountability policies in the two school systems under study. The second, more theoretical, is to confront—we hope in a productive tensioning—two theoretical traditions: one, neo-institutionalism, which, despite the plurality of its variants (sociohistorical, sociological, discursive, related to rational choice, etc.), often allows us to track structural effects common to different school systems, and the other, the sociology of public action, which indeed provides tools to reflect on phenomena of convergence or isomorphism but which also and above all offers numerous conceptual instruments to study the many processes of fragmentation and differentiation of public action.Footnote 18

However, the application of such a framework and, further still, its restitution in a “linear” fashion in this book are not easy. Since this theoretical framework is fundamentally interactive, it would require to constantly interlinking the various analytical dimensions in the writing process itself, at the risk of making each one, and the objects to which it applies, difficult to comprehend. This risk is even greater given that our research is multilevel, resulting in a double entry table (see Table 3.2).

Table 3.2 Theoretical structure of the book
Full size table

Therefore, in this book, we have chosen to focus on three cells in this table, corresponding to three important stages of the analysis: the study of the trajectory of two accountability policies at the national level (Chap. 5); the analysis of mediation processes at play at the intermediate level of académies (France) and SBs (Quebec) (Chap. 6); and, finally, the study of the local implementation of tools and policy instruments that constitute the policy at the school level (Chap. 7).

This choice has the advantage of avoiding the repetitions inevitable in a presentation structured by levels of public action or by conceptual entry point and, therefore, of contributing to a more accessible presentation of a nine-dimensional research. Nonetheless, it has the major drawback of reproducing, at least indirectly or implicitly, a top down vision of public action in the realm of accountability (and the forms of globalization that this accountability favors), quite the contrary of our research intention. This is why, in the very writing of these chapters, without renouncing the benefit of the demonstration of a constitutive dimension in our theoretical framework at the level of a given public action, as much as possible, we stress the interactions at play with other levels and dimensions. In theoretical terms, these links will be facilitated by our conceptual choices. For example, studying the interpretation at the national level of international or supranational watchwords and imperatives invites, in fact, an examination of mediation processes, mechanisms of path dependency, and incremental institutional changes which characterize a trajectory of accountability policy which often involve policy instruments and, therefore, require thinking also in terms of instrumentation. Instruments condense power relationships, and their implementation involves various mediation processes concerning their purposes and properties and, beyond that, the policy on the whole. Finally, and more empirically, these links sometimes appear on their own. On a number of occasions, for example, the national trajectory of accountability policies underwent bifurcations due to the evolution of the policy at subnational levels; it is impossible to consider the instrumentation of these policies at the school level without linking them to political and administrative mediations by intermediate authorities, mediations which are themselves a function of a certain trajectory of the national policy, etc. However, these links in the analysis depend on possibilities offered by the investigative methodology selected, which we will describe in detail in the next chapter.