Methodology

Abstract

The purpose of this chapter is to make explicit the methodological choices at the origin of our research project. The first section comes back to its academic origins. The second describes the design of the comparison. The latter was more oriented to cases rather than variables according to Ragin’s distinction. Our overall research strategy was based on the disconfirmation of a most likely case. We chose a policy—accountability policy—which can be seen as a typical example of education policy promoted by transnational organizations to show that, even in this most likely case, governing changes at work are not unilateral but fundamentally multilevel and depend on the policy trajectory of each system, on a series of mediation processes occurring at various levels, and on the different logics of instrumentation at stake. If these aspects are empirically proven, then it can be argued that it is also true in least likely cases. The third section details the various analytical conventions to be defined throughout the research process (terminology, selection of cases, modes of collaboration within the project). The last two sections deal with the data and material collected through qualitative methods (mainly interviews, observations, and content analyses of documents) and how they were analyzed.

4.1 Introduction

As an international colloquium organized in October 2014 by the Centre of International Research and Study from the University of Montreal (CERIUM) pointed out, comparisons by social scientists of France and Quebec have multiplied in recent decades.¹ According to its organizers, this proliferation raises several issues.

First, these works tend to focus on specific topics, such as the nation-state, models of secularity, or social and urban policies, and to neglect others, such as social institutions, political parties, or social movements. In this landscape, works on education and education policies in a broad sense (including higher education) seem to be in an intermediate position.

Second, this selective development is not only the consequence of the evolution of academic disciplines themselves. It is also linked to particular political and social uses of the comparison: since the end of the 1970s, comparing Quebec with France instead of with other Canadian provinces, for instance, does not always come from a purely academic concern; it may also be a part of a wider political purpose. Conversely, in France, Quebec is often perceived by political leaders and administrative elites as an “easily understandable America.”² Thus, there may be various political assumptions in the French-Quebec comparisons, and education should be no exception.

Third, although Quebec and France appear close, as exemplified by their common French language, their apparent similarities may hide significant differences on the two sides of the Atlantic, with regard to their academic traditions and methods but also the design of some policy sectors. Hence, the risk from this kind of comparison is in underestimating certain challenges.

The purpose of this chapter is precisely to make explicit the epistemological and methodological choices of the two national teams involved in this research. The first section comes back to its academic origins. The second describes the overall design of the comparison. The third details the various analytical conventions to be defined throughout the research process, and the last two deal with the data and materials collected and how they were analyzed.

4.2 The Origins: The (New)AGE Project

This book is based on 4 years of qualitative research conducted between March 2012 and December 2015. This research was funded by the French national research agency, ANR,³ and by the Quebec research institute for society and culture, FQRSC,⁴ within the context of a selective call for projects specifically devoted to France-Quebec comparisons in general (not only in the education sector).

It is important to underscore that there were never any political objectives beyond the greater understanding of policy processes at work in France and Quebec. The ANR-FQRSC program was strictly devoted to the advancement of knowledge, even if the application forms required a prediction of the research’s social impact. Furthermore, the funding organizations never intervened in the research, either in its design or its targeted dissemination.

The topic, accountability policies, was chosen by the directors of the project as a continuation of former research/publications and collaboration. Indeed, several members of the project had already worked on the question of accountability, governance, and evaluation, both individually and in collaboration in earlier European research projects. Our idea was to benefit from this specialization and the internal epistemological and methodological coherence of a small team of researchers who had already worked together and who shared similar theoretical approaches, to extend and accumulate knowledge on accountability issues and processes.

We should also specify that this project was not based on normative assumptions concerning the need to improve accountability mechanisms in each system, for instance, or on the desire to modernize these accountability policies. The title of the project, (New)AGE, for “(New) Accountability and Governance in Education,” was conceived to stress the ambivalence of these so-called novelties. Moreover, the research teams were careful not to reproduce implicit assumptions, sometimes taken for granted by our interlocutors, on the position that France or Quebec should adopt in this comparison. For example, on the French side, it was not assumed that France, in which the accountability policy is less codified overall, should take the Quebec path, and on the Quebec side, that Quebec policy should be regarded as a policy model that should be exported to France.

4.3 General Design of the Comparison

Our overall research strategy is based on the disconfirmation of a most likely case (Gerring, 2007; Lijphart, 1971). As argued elsewhere (Maroy, Pons, & Dupuy, 2017), globalization is often depicted as a relatively uniform top-down, instrumental, normative, and mainly cognitive process, imposed “from above” upon the traditional education authorities which would pave the way for (1) new policy scapes, (2) the neoliberalization of education, and (3) new scalar politics on a global level (Lingard & Rawolle, 2011). Our strategy consists in testing this theory, or more precisely this series of theories, by choosing a policy—accountability policy—which can be seen as a typical example of education policy promoted by transnational organizations and showing that, even in this most likely case, governing changes at work are not unilateral but fundamentally multilevel and depend on the policy trajectory of each system, on a series of mediation processes occurring at various levels, and on the different logics of instrumentation at stake. If these aspects are empirically proven, then it can be argued that it is also true in least likely cases.

Our strategy of comparison was more oriented to cases rather than variables (Ragin, 1987). The idea was to analyze two cases in-depth, through an inductive, qualitative, comprehensive, and interpretative approach; to understand the complexity of each case and highlight its dynamic links, rather than adopt a deductive, quantitative, statistical, and explanatory approach to marginal effects; to confirm or inform a previous theory; or to rank or even eliminate explanatory factors (Giraud, 2003). The goal was to seriously consider this complexity and offer new, or at least different, complex causal relations to the theory. This kind of comparison risks resulting in merely a juxtaposition of case studies, on the basis of rather inconsistent analytical categories, and not venturing beyond the particularities of cases to reflect on universal or transversal aspects. That is why we always chose to present our cases through analytical dimensions across France and Quebec.

Last, we decided to compare policy in two different national contexts in terms of political foundations, structures, potential exposure to international influence, and dominant regulatory modes. The French education system is characterized by a high degree of administrative centralization, an emphasis and reliance on ministerial circulars as a mode of interdepartmental communication and regulation, and resistance to external influence, especially that of New Public Management (Bezes, 2009). By contrast, in Quebec, while the main administrative, curricular, and published pedagogical guidelines were centralized at the state level more recently (1960), the local governing bodies (school boards) were present at the inception of the educational initiative and have always had room for maneuver. Quebec society emphasizes education as a tool for individual and collective development, which is related to the objective of sustaining a distinct society within Canada (Rocher, 2004). This is also linked to the aim to reinforce the role of the state in education, in comparison to other Canadian provinces. However, there is no such thing as a resistance to transnational “public management” discourse within Quebec public administration (Dufour, 2012). The variations between these two cases will allow us to test the explanatory capacity of the three dimensions of our theoretical framework (trajectory, mediation, and instrumentation).

Choosing these two systems allowed us to design a most different system comparison (especially on the topics of multilevel governance) but also to extend the empirical coverage of the international literature on accountability in education, which tends to focus on systems where accountability mechanisms are highly developed, such as the USA, England, or Chile, by exploring cases where accountability is still a low-stakes issue (Maroy, 2015).

4.4 Research Conventions

Comparison depends on comparability. The latter was progressively shaped by a series of research conventions we adopted as a team.

4.4.1 Toward a Common Language

It was first necessary to agree upon a common language. The research was done in a French-speaking context, and it was not always possible to find a direct equivalent of accountability as a notion, as a process, and as a policy. As a concept, it was finally decided to think of accountability as a relationship, as Mark Bovens (2007, 2010) invites us to do.⁵ This definition was particularly useful in studying the concrete forms taken by accountability mechanisms which are effectively implemented in Quebec and France, without overestimating the importance of a particular theoretical model or accepting at face value profuse policy speeches on the topic. As a process, we rapidly focused on specific accounts: the outcomes, results, or performances of the school system and not other forms of accounts that the actor may justify in an accountability process, such as financial choices or process compliancy, for instance. As a policy, we finally defined accountability policies as policies whose purpose is to favor the implementation of governance, or more precisely an institutional regulation, based on results (politique de gouvernance par les résultats). This orientation rapidly led some members of the project to work on a new typology of accountability policies which more effectively took into account the diversity of accountability systems and the role of tools and instruments in these policies (Maroy & Voisin, 2014).

4.4.2 Identifying Comparable Empirical Objects

The key challenge, then, was to clearly define the empirical focus of the research in each system. In Quebec, accountability policy, as defined above, clearly referred to the “results-based management” (RBM) that has been explicitly and regularly implemented in public administrations (not only in education).

Results-based management appeared on the political agenda in the early 2000s and was soon converted into a reform bill (Bill 82).

From that perspective, we see RBM as an “institutional change mechanism.” In other words, RBM is promoting new institutional features, among which the most important are new public policy tools. Temporalities are at the center of our analysis since the changes (or non-changes) we were seeking could only be observed over time. From that perspective, we analyzed the “genesis” and the “trajectory” of the RBM policy in education. Through documentary analysis and a review of the literature, we looked for the gradual emergence of ideas, values, and rules linked to the RBM policy. We then saw how multiple policy actors were (co)constructing the policy itself through different “forums” (Fouilleux, 2000).

For the second research axis, we turned our lens toward policy implementation at the intermediate and local level of the school system. Even if we establish a methodological demarcation line, we consider that the implementation process is embedded in the policy co-construction process. In other words, the policy itself is not considered a “final product” when implementation occurs. We were still looking for changes, specifically what we call “mediation logic” by the intermediate actors (see Chap. 3).

Finally, we extended our research to a third axis complementary to the previous two: RBM policy tools. Those tools are well defined in the Education Act and their use is mandatory. Therefore, we quickly identified a core set of policy tools to investigate at each level—mostly based on the plan/contract/accountability report classical process. However, at the stage of defining the research problem, we found that we must add to these “official” tools all the mechanisms developed at the intermediate level in order to collect the data and monitor the organization’s performances. From our perspective, this broader range of tools constitutes the RBM toolset. We emphasized those policy tools at each level because they can be considered as change trackers (see Table 4.1). Most tools are artifacts that enable a diachronic analysis and, therefore, their study complements the other data examined. Yet, above all, policy tools can be studied on their own as “institutions” encompassing their own sets of ideas, values, and rules and interacting with organizational actors.

Table 4.1 Studying Quebec “results-based management”

In France, there was no obvious equivalent to the Quebec experience. The French accountability policy is not so formalized and codified, especially because there is still a burning policy debate among institutional actors on the legitimacy of imposing an obligation of results on professionals. Consequently, the translations of accountability available in the French policy debate all have their particularities and political meanings. As a set of press dispatches illustrates (see Sect. 4.5), the expression “obligation of results” is used from a rather critical perspective by professionals or scholars; “results-based regulation” remains an academic expression without any institutional implications; “administrative responsibility” refers to various—and often past—processes of accountability and not to current transformations; pleas for “responsibilization” in the policy debate do not refer to an accountability process but rather reflect moral indignation, etc. Finally, it was the notion of “steering by results” that seemed to be the best equivalent to the Quebec policy and to other forms of accountability policies in English-speaking countries. Nevertheless, this notion is mainly an administrative one, and it is not always explicitly integrated into a formal policy. The implementation of the different policy tools (projects, contracts, and evaluations) at the origin of this “steering by results” is very uneven from one period and territory to another. This explains why French field research consisted of the systematization of several case studies and the analysis of various elements falling within the main research subject (see Table 4.2).

Table 4.2 Studying the French “steering by results”: its constituent elements

4.4.3 Designing Field Research Studies

Another series of conventions concerns the design of the research fields at the intermediate and local levels. In both France and Quebec, the two teams selected intermediate authorities and schools. We first set up criteria for the desired diversity for each of the organizations studied (see Table 4.3) in terms of (1) the academic results and (2) the socioeconomic characteristics of its student population. Furthermore, their different stance on accountability policy was also taken into account in the French cases. The process of selecting the organizations was another matter and differs widely between our two national cases. In Quebec, this process was a little more haphazard due to field accessibility problems (especially at the school level). However, in both cases, we were able to meet the original diversity criteria to a considerable extent.

Table 4.3 Designing the research fields

In Quebec, since the SBs govern the schools under their jurisdiction, we have made them our first point of entry into the field. We have chosen to confine ourselves to the SB of the Francophone network in order to facilitate comparison with the French cases and because it is this network which provides schooling to the vast majority of students. We contacted a sample of 30 Francophone SBs (out of 60 in total). Some SBs were set aside in order not to hinder other similar research being conducted simultaneously or due to their atypical characteristics (notably the size of the organization). Moreover, a factor of relative geographical proximity (related to our ability to send research teams) guided this first selection.⁶ The SBs participated on a voluntary basis. In the end, 4 out of 30 SBs agreed to participate in the research. We used our preliminary analysis of the genesis of the law within the Quebec political field as a framework to guide our empirical choices, both to frame our interviews and for our interpretation of the data.

When selecting schools, only the Northern SB gave us full access to their secondary schools and staff. We were able to access schools that are all performing relatively well, with a few socioeconomic and demographic variations: a rural and economically advantaged school, a privileged suburban school, and a slightly disadvantaged suburban school. We were also able to interview several Southern SB teachers in further research on teachers’ perceptions of RBM. However, we were unable to obtain official access to those schools from the SB authority. Access to teachers was made possible by the local teachers’ unions. In one case (Meadow School), we completed our case study with an interview previously done with the school principal, but, in the other two schools, it was impossible to interview school principals. Therefore, we only selected Meadow School for our Southern SB sample; the two other school interviews are referred to as contextual elements. Despite our best efforts and previous agreements with the Eastern and Western SB, we were denied access to their schools. Obviously, this was not our initial plan.

Those difficulties in three SBs out of four can be explained by the sensitive aspect of the research on RBM in education. RBM has encountered much resistance from teachers but also from some principals. In certain SBs, as shown in the interview with middle managers, the implementation process was particularly tense. Moreover, while we were conducting our fieldwork, the teachers’ unions were negotiating their collective labor agreement, adding another level of sensitivity. Teachers’ union representatives, SB officials, and school principals were all trying to control the narrative on RBM implementation. Despite those difficulties and the various channels of access to the field,⁷ the four schools used as case studies were rich in information, especially from the point of view of the use of tools, and actors’ interactions with tools, as we were able to see plenty of either positive or negative interactions. Moreover, in all cases, we triangulated data collected from interviews with both staff and managerial teams, in addition to documentary sources. In the end, the data gathered gave us the empirical material needed for this last stage of the research.

In France, the selection of local and intermediate organizations was relatively complex because there are a lot of administrative levels corresponding to different responsibilities in primary and secondary education. Which is the relevant intermediate authority: the rectorat at the regional level or the former inspection d’académie at the departmental one? Three rectorats d’académie were selected according to the criteria mentioned above (results, population, and accountability policy) but also according to the location of the team members and the institutional links they could have developed with regional authorities. In each académie, three lycées (upper secondary schools) were selected, two from public education and one from the private sector.

These choices raise two major issues. The first concerns the very different scales of the intermediate authorities in France and Quebec. For example, the Western académie alone has as many students in primary and secondary education as Quebec as a whole (more than one million). Second, the French team focused on upper secondary education (compulsory or post-compulsory), whereas the results-based management in Quebec was a comprehensive policy covering both primary and secondary education.

Nevertheless, despite differences between primary and secondary education and differences in size, this comparison remained optimal for various reasons. First, a multilevel approach within French académies that would have taken into account both primary and secondary education, as well as the local, departmental, and regional levels, was not realistic within the research project format (five scholars working on the French policy for 4 years). The three académies encompass nine departments, and there are about a hundred local districts in primary education in the Eastern académie alone. Thus, it was very difficult to design a qualitative field study that would have embraced all these levels, even on the basis of relevant and representative sampling processes. It was even less feasible as negotiations would have had to be undertaken with the relevant authorities at every institutional level in order to access data. This was all the more problematic as the implementation of accountability policy tools is very uneven from one territory or period to another.

Second, since 2012, there has been a strong movement of reconcentration of powers at the regional level of the académies within French school administrations, so that the rectorats are now the key institutional intermediate level of the system. And specifically, we wanted to focus on the decisive intermediate level in mediation in each system. So even if, empirically, these intermediate levels are institutionally very different in their status and size, theoretically, they should play the same key institutional mediation role in the implementation of policies.

Third, it is in upper secondary education that performance indicators and accountability mechanisms are most developed: even if it is always interesting to study cases in which accountability measures are less intensively implemented, it was relevant to focus on this level for the purpose of comparison. Fourth, this focus on rectorats-lycées did not prevent us from studying the implementation of accountability tools at other levels since, according to the internal governance frameworks of each académie, powers may be given to the directors of departmental services.

4.4.4 Organizing the Academic Dialogue Within the Project

To adopt this common language, identify comparable objectives, develop similar research field designs, and then implement the research consistently, we had to adopt a specific method of coordination and academic dialogue between and within each national research team. Beyond the traditional and expected annual meetings of the teams and the numerous videoconferences that we organized, we decided to provide and disseminate a series of internal academic documents. The latter were necessary to systematize communication among members of the project, disseminate its key orientations, and agree on provisional conclusions or findings.

For each theoretical dimension of the project (see Chap. 3), the directors provided a theoretical note to frame the field research. These notes recalled the grid of theoretical questions mentioned in the project and stressed the main components of the theoretical framework adopted, as well as their general implications for the field research study. These notes were always discussed collectively so that different members of the project could enrich the global conceptualization of the research according to their areas of expertise.

Consequently, these notes were very helpful in developing methodological documents within each team, such as interview grids. These collective documents disseminated within and between each national team specified (1) the theoretical questions retained; (2) their implications for fieldwork, for instance, the necessity, when studying the instrumentation of an accountability policy, of stressing the various institutional and technical steps of the implementation of a specific mechanism, even if this is sometimes boring and distasteful; (3) the strategy of research adopted, for example, in interviews asking people to first talk about their professional trajectory and their everyday work to give them due recognition and build mutual confidence; and (4) the list of questions (interview grids, for instance).

The field research on mediation and instrumentation processes (the second and third research dimensions) resulted in a series of national case studies (for schools and intermediate authorities) that were systematically sent to the other national teams (sometimes in summary form) with additional documents, such as formal descriptions of the role of these intermediate authorities within the school system and summaries of the main statistical indicators available or of the cases themselves. These documents were systematically commented upon by a person from the other national team to ensure a comparative aspect in the conception of each case study.

These elements were discussed throughout the project in various internal seminars but also in two international symposiums that we organized in Paris in May 2015 and in Montreal in May 2016. The latter provided opportunities to compare the (New)AGE main findings with recent research on that issue in other countries or policy fields.

4.5 Data and Materials

In Quebec and France, we used different qualitative methods to collect appropriate data: analyses of various types of documents (official texts, institutional documents, parliamentary debates, press releases, and newspaper articles), observations in schools and school administrations (when possible), and interviews with actors at different institutional positions within the education system. As mentioned above, the implementation of such methods was based on various methodological framing notes to make comparison possible. Nevertheless, given the specificity of each policy and each system, on the one hand, and the particular administrative conditions of access to schools and school administrations on the other, differences were introduced in our methodological plan so that totally symmetrical datasets were impossible.

In Quebec, the RBM is a highly formalized policy based on several legislative texts. Thus, it was logical to center a significant portion of our methodology on the origin and political life of these laws.

First, we reviewed the literature (1988–2000) on the development of results-based management in Quebec and its precursors. The choice of documentary corpus for the synthesis of the empirical literature on RBM was carried out in several stages. We first did extensive subject-matter research. We consulted fellow experts to help us define this corpus. We then selected the most representative texts of the period (according to expert input and our own research topic—see Sect. 4.6). The final corpus contains a total of 23 references: research reports and articles in both academic and professional journals. To this corpus, we added the evolution of the law on education since its last revision (1988).

Yet, the key period for the development of results-based management in education is that of the 2000s. Our main objective for that period was to gain an understanding of which ideas and actors were influential in the construction of the policy. For this purpose, we focused on a dataset consisting of three kinds of primary sources: (1) the transcript of the political debates in parliamentary committees and in the National Assembly⁸ concerning RBM education bills (124 in 2002 and 88 in 2008) (we focused on speeches given by the representatives of the three political parties (the governing party and two opposition parties) in charge of education-related issues (7 selected in total for the whole period)); (2) the memoranda submitted by the professional and parent associations concerning these bills (13 were selected from 8 different associations representing the most influential groups); and (3) the official legislative documents, that is, both the bill submitted and the bill adopted which modified the law. We consider that those data sources incorporate most of the debate in what we call the “political forum” driving policy construction.

Finally, we also used a corpus of articles and op-ed pieces for the 2000–2012 period to analyze the debate outside the political forum, especially how RBM-related issues appeared on the political agenda. Through those articles, we were looking for a common basis of the discourse on RBM policy in education, embodying in some way the “production of ideas” (which explains the specific analytical method used for this data; see Sect. 4.6). Three major French-language dailies, Le Devoir, La Presse, and Le Soleil, were selected for consistency but also for practical issues of accessibility to sources and materials. Text sampling was based on an incremental keyword selection process (by adding specific keywords to narrow down the sample) by means of a computer search engine. We then revised the entire corpus selected in order to eliminate texts that were not relevant for the analysis. Three hundred forty-five texts were selected in this way.

At the intermediate level, the data were collected in the four SBs previously selected. The data come mainly from semi-directive interviews (n = 57) conducted with key stakeholders in the administrative and political structure of each SB (see Table 4.4). Interviews were conducted over a 14-month period. Interview guides were derived from a set of 15 common topics:

• The general presentation of the SB

• The evolution of the SB context

• The organizational structure of the SB

• Perception of educational success

• Overall perception of the results-based management policy

• Policy tools content

• Policy tools construction

• Implementation of RBM

• Data production and use

• Accountability processes

• SB results monitoring and evaluation by the Ministry of Education

• School results monitoring and evaluation by the SB

• Assessment tools

• Perception of the parents

• Perceptions of SB democracy

Table 4.4 Intermediate and local level interviews: sample composition

In order to reflect the professional reality of each actor and reach our own empirical objectives, we focused on specific topics for each stakeholder, topics related to their organizational function or knowledge (see Table 4.4). In rare cases (Southern SB and Western SB), we were also able to make some observations on the deliberation mechanism related to RBM.

Finally, we also had access to the content of the various policy tools central to the implementation of RBM at the intermediate level. We did a content analysis of the current strategic planning (at the time of the inquiry) of each SB studied. In some cases (when available; Eastern SB and Northern SB), previous strategic plans were also included in the analysis. In addition, we included in our policy tools dataset the Partnership Agreements of the SBs.

At the school level, we mainly focused on the context of the school, the specific RBM implementation strategy, and, in particular, the “policy tools” axis of our research (see Chap. 3). Besides the four semi-directive interviews designed for the SB level, conducted with school principals at the Southern SB and Northern SB in the first stage of our research, we also conducted another series of semi-directive interviews with school principals and assistant principals (n = 7) of the Northern SB’s secondary schools to which we had access. Those schools were examined as case studies in their own right. We analyzed different policy tools from those schools: their management agreements with the Northern SB, their annual public reports, and a sample of the representation of significant academic performance issues for those schools (mostly in the form of bar graphs and results spreadsheets). Moreover, the teachers’ interviews (Northern and Southern SB) were also used to analyze the RBM policy tools in use and the impact on their work.

In France, as we will see, steering by results led to less concentrated, more widely spread public action. Studying it involved going back and examining the origin of various instruments and ideas but also measuring its empirical effectiveness beyond the different types of discourse, whose actual institutional base may vary in strength. That is why we started first at the national level, with the analysis of a group of 621 dispatches on questions of governance by results. These were published between March 1998 and July 2012 by a press agency specializing in education (AEF).⁹ In addition, a collection of 493 articles published in institutional and professional journals was also studied.¹⁰ These were selected based on three criteria: their status, their role in rendering visible certain problems related to our subject, and the characteristics of their readership. We also examined a corpus of official texts from the French Ministry of National Education, part of which (the ministerial circulars issued at the start of the new school year) was analyzed using lexicometry (with Lexico software). Moreover 19 in-depth interviews were carried out with national actors on the question of steering by results. These actors were identified in the corpus. This material also allowed for an analysis of the trajectory of steering by results in France and made it possible to examine its implementation through a case study: the implementation of a contract between the Ministry of National Education’s central administration and the rectorats.

The same kind of research design was reproduced at the intermediate and local levels. As far as intermediate administrative authorities within académies are concerned, 30 interviews with administrative senior managers and office holders were cross-checked with (1) three datasets of AEF press dispatches¹¹; (2) an academic literature review of research specifically devoted to these territories; (3) the collection of various ministerial documents on the three académies (their profiles, their projects, their contracts, their key statistical features, sometimes their history, their evaluation by general inspectors, and so on); (4) some local archives when they were relevant; and (5), when possible, various observations of management meetings. At the public school level, we conducted 32 semi-structured interviews with principals, teachers, bursars, and education counselors. These interviews were triangulated with observations of council meetings (boards of trustees and education councils), numerous institutional documents (official circulars edited by the rectorats, administrative files conceived for the steering procedures, contracts, projects, evaluation reports, and statistical indicators), and, when relevant, datasets of local press articles.

The success of a qualitative inquiry of this kind in various territories depends greatly on the ability of researchers first to access the field of investigation and then, if possible, to control the sources of bias throughout the research process. In France, surprisingly, this access was easy to obtain at all levels of the system. This is, for instance, illustrated by the fact that the team could carry out observations, which are in general a sensitive topic. This may be the result of the research team’s choice of always negotiating this access at the top decision-making level in a given administrative environment, the confidence inspired by the members of the project who have sometimes specialized in these issues for several years and have established a professional network with respect to these topics, or, more simply, the willingness of the ministry to communicate its recent initiatives and get feedback from the research team in a system in which such feedback is scarce.

In Quebec, as stated above, several pitfalls related to access to our research fields appeared in the course of our investigation. Teachers were in the period of renewal of their collective labor agreement which made certain stakeholders more reluctant to engage in our research project and compromised access to several schools. Moreover, the research topic, education accountability, was sensitive among stakeholders themselves. Finally, this topic was not directly helpful for either SB managers or school principals. In other words, the (New)AGE project was intended as fundamental research on accountability, with no immediate benefits for participating actors. Even if the SB’s officials were aware of that when agreeing to participate in the research, it might have made it harder to sell it to all stakeholders. Combining sensitive research topics and a busy schedule for the SBs (which, in addition, were suffering from harsh budgetary cuts) and the fieldwork becomes difficult. However, those issues do not in any way minimize the investment in the project of a large number of individuals or lessen the quality of the information we managed to collect.

4.6 Data Analysis

In France, the analytical process can be divided into three stages. The first is encoding. The interviews were nearly all recorded and were either transcribed in their entirety or summarized in chronothematic tables. All the interviews were then encoded and synthesized in transversal thematic files (“governance,” “instrumentation,” “policy speeches on results,” and so on) for purposes of analysis and comparison. The same method was used for observations: all the significant passages of the observation reports were encoded and integrated in the same transversal thematic files. The ministerial circulars that we analyzed with the Lexico software were divided into various sections (one section per circular) and codified with five keys (date, minister in power, signatories, number of pages, and the prevalence of governance by results in the structure of the text). They were then formatted for software analysis (orthographic checking, syntax checking, change of specific signs, and so on). In contrast, the press dispatches were simply listed and directly integrated into overall chronologies.

The second stage is data analysis itself. As far as the French interviews and observations are concerned, we limited ourselves to the grid of questions identified in the academic project and repeated in the various theoretical framing notes mentioned in Sect. 4.4.4 (see also Chap. 3). The press dispatches enabled us to analyze policy trajectory at various institutional levels, to map the policy debate on accountability, to be informed of some key policy documents, and to identify relevant interlocutors. The lexicometric analysis of ministerial circulars remained modest: we only produced the contingency table, simple tabulations of specific graphic forms, qualitative analysis of the environment’s surrounding key terms, and cross-tabulated results according to key variables, such as the date. This is due principally to the infrequent occurrence of our key terms (“steering,” “results,” “project,” “contract,” and so on) which prevented us engaging in more complex methods of classification and suggested a more literary and qualitative approach. Thirdly, we produced a series of internal working documents in which we detailed our methodology and synthesized our main intermediate findings: two research reports on the national policy trajectory in each system, seven case studies on the implementation of the two accountability policies by intermediate policy actors (three académies and four SBs), and six public school case studies in France and four more in Quebec. These documents were structured on the basis of our grid of theoretical questions, but their formal presentation was adapted to the specificities of each national context. In France, since the académies as territories and as policy producers have been largely neglected by scholars and since the implementation of accountability mechanisms was uneven from one territory and period to another, it was necessary to have a comprehensive approach and to analyze the history, the morphology, and the general features of these académies, to trace the evolution of the steering processes since the end of the 1990s, to synthesize the governance at work at the time of the fieldwork, to appreciate the degree of instrumentation of the accountability policy, to recapitulate the main policy speeches on the topic, and to resituate all these elements in the overall regulation of the académies which may or may not follow an increasing accountability trend. The choices made for the analysis of school policies are similar. We started with a general presentation of each school (geographical context, history, size, population, curriculum, and results) and focused on the institutional regulation of each individual school (by the académie and by its leader through a specific local school policy), on the more or less strong instrumentation of its policy and on the accountability relationships within the school and their more or less effective integration in systems of broader work relations.

Lastly, as mentioned above, we decided to systematize the collective discussion of each working document during meetings and internal seminars and to always introduce this collective reflection with a discussion of the contribution by a member of the other national team in order to favor comparison building and improve the presentation of results.

In Quebec, different methods of analysis were employed to process the selected material. The analysis of the corpus of empirical research texts on RBM for the period 1988–2000 focuses on the synthesis and evaluation of each text based on an analytical reading grid. The reading grid of this corpus aimed at identifying the main elements of each text: research objectives, thesis, the central issues, and the principal elements of the argument. The analytical grid developed earlier was drawn upon to identify elements of interpretation. Selected texts were considered as working material. They served as a basis for our analysis of the policy trajectory. This work was an exegesis: we were putting forward ideas, extending others, and questioning some interpretations. Our goal here was to offer a general perspective on the orientations of Quebec policies and the context in which RBM emerged, while being careful not to overestimate the coherence of this policy network. Finally, we carried out content analysis of specific articles of the Education Act at different points in time (articles related to governance topics mostly in the decade 2000), allowing us to follow RBM trajectory (even though there is much more to it than simply its legalistic aspect).

The analysis of the RBM policy production through the “policy forum” is centered on its cognitive and strategic dimension. This leads to various analytical methods. We used NVivo to do a thematic analysis of the spoken and written data concerning the RBM-related bills. We first analyzed the different “policy narratives” (Radaelli, 2000) used in the education policy forum. Policy narratives are structured around a set of key elements: the main problems of the education system as framed by the actors, the development of set actions and solutions, and a set of anticipated consequences arising from the actual implementation or non-implementation of these solutions. This structure guided our NVivo coding. We then used this coding to reconstruct the main emerging narrative about RBM. This was done by aggregating the different points addressed by the actors on RBM-related topics. Several themes and sub-themes emerged; each sub-theme could relate to the different “phases” of the narrative (problem, action/solution, and consequence). Each theme was a puzzle piece contributing to one causal story representing one policy narrative. The shared visions of actors on specific themes, as well as their opposing positions, were highlighted in this process.

At the same time, we carried out a “strategic analysis” (based on interests and perceived wins and losses) of the main actors’ positions within the Education Policy Forum. To complement this analysis, we performed a content analysis of the evolution of Bills 124 and 88 (from inception to the final stage of adoption) to see how they evolved and, especially, which actors had more weight in promoting those changes during the adoption process. The strategic reading of actors’ positions also made it possible to discern tensions between different “advocacy coalitions” (Sabatier & Jenkins-Smith, 1993). The identification of those coalitions was done through highlighting positional fractures among actors and cross-referencing those fractures to different stakes/issues related to RBM.

For the intermediate level, we mostly focused on our extensive set of interviews to elaborate case studies for each SB. Using NVivo software again, we developed a descriptive coding grid based on the interview guide and an analytical coding grid grounded in the new institutionalism and micro-political theoretical approach (discussed in Chap. 3). This coding was, thus, based on the empirical goals associated with the interview guide: (1) determining the organizational and institutional context of each SB and (2) highlighting the process of implementing RBM by the SBs. Following the coding, we produced analytical memos to describe the coding categories for each of the actors interviewed. These were the basis of our case studies. They were complemented by content analysis of the SBs’ strategic planning and an analysis of contextual data (academic performances and socioeconomic levels) for each SB. The purpose of the case studies was to highlight the policy mediation logics at work within SBs and the social and organizational features that may influence SB responses to the ministry requirements linked to RBM.

Finally, we used a similar analytical strategy regarding the policy tools. We relied on the interviews conducted with the Northern SB schools and Southern SB Meadow School teachers and principals. We used the same dual coding grid in order to isolate the different elements of the interview and then to analyze them using a second coding grid. In this case, we focused on actors’ perceptions and uses of policy tools. In considering policy tools as institutions, we tried to discern their cognitive dimension (the comparisons they suggest and the ideas they contain) and their normative dimensions (the specific forms of “justice” they entail and the various forms of accountability for different school actors they rely on). We replicated this grid to analyze the policy tools themselves. We also did a standard content analysis of each tool. We then extended our interview coding grid to try to understand the tools in use within the school. More specifically, we focused our analysis on the interactions involved in the implementation of these tools. We looked for different forms of interactions, especially understood as a test for an actor. Those tests are public and could arguably be linked to some sort of reflection or critique. We attempted to isolate those reflections in the discourse of the interviewees. Finally, our last analytical goals were to identify any effects of those tools for the actors and the organization. We distinguished pragmatic effects (influences on action) from cognitive effects (influences on representation).

4.7 Controlling the Processes, Refreshing Perspectives

Finally, as far as methodology is concerned, this comparative research involved constantly formalizing and controlling, through specific collective processes, the progressive production of a comparative view of research subjects (governance, accountability, evaluation, contracts, and so on) that we had already analyzed frequently, using national perspectives or specialized approaches. This helped us to make our research focus more explicit, to test hypotheses, to highlight surprising regularities (such as, for instance, the reinforcement of state power in both cases; see Chap. 5), and to discuss the relevance of a series of theories on globalization, governing changes, and the effects of New Public Management. Yet more fundamentally, it clearly led us, individually and collectively, to distance ourselves from classical, immediate, or traditional classifications about our own cases and to renew our appreciation of our own education system. In this methodological process, the collective elaboration of theoretical notes, the choice made to discuss national results by members from the other team as often as possible, and the constitution of binational teams of researchers when presenting results all played a decisive role.