1 Introduction

The Organisation for Economic Co-operation and Development’s (OECD’s) Programme for International Student Assessment (PISA) aims to produce “a new basis for policy dialogue and for collaboration in defining and implementing educational goals, in innovative ways that reflect judgments about the skills that are relevant to adult life” (OECD, 2013, p. 13) by providing

  • Basic indicators that represent a baseline profile of the knowledge and skills of students.

  • Indicators derived from the contextual questionnaire that show how such skills relate to important demographic, social, economic and educational variables.

  • Indicators on trends that emerge from the ongoing nature of the data collection and that show changes in outcome levels and distributions, and in relationships between student-level and school-level background variables and outcomes. (p. 16)

Officially, PISA responds to a regime of economic development as a service to economic production by, for example, providing information about workforce knowledge bases, skill formation, allocation of educational resources, efficiency of markets for skill formation and training (Hirsch, 2002). The growing number of studies from within different academic areas and traditions in educational research indicates strongly the significance attributed to PISA for national educational systems, schools and, perhaps more importantly, the global discourse on knowledge, which also affects national and international academic research in education. Given this widespread influence, it is important to develop a critical understanding of what PISA is, what it does and how.

In a sociologically grounded, systematic analysis of the reception of PISA in six European countries, Pons (2012) focuses not on “PISA shock”—the popular media discourse that focuses primarily on how each country is positioned in the PISA rankings, producing and disseminating “shocking” headlines—but on how better to understand the forms and the mechanisms of the “continuous social fabrication of the legitimate ‘PISA knowledge’” in each national policy context (p. 207). The word “fabrication” here conveys the idea that “[a]ccountability regimes such as PISA produce…selective accounts that represent results and outcomes in the best possible light (Ozga, 2012a, p. 167). On the basis of these national case studies, Pons was able to show that new forms of knowledge, what he termed “knowledge for learning”, which is knowledge inspired by the strict content of the PISA survey, beyond the description of the general and gross results, were rare in each country for the period studied (2001–2008). In particular, he documents the “lack of institutional and cognitive spaces of enrolment and interest-building…in which PISA results could be analysed, translated and reinvested by people” (Pons, 2012, p. 207). In contrast to this paucity of knowledge for learning by national knowledge producers, he identified a high number of arguments based on PISA, what he calls “knowledge for policy”, used in the national public debates to argue for all kinds of opinions, stances and reforms. These arguments, formed in particular cognitive and institutional contexts, have progressively formed specific bodies of knowledge for policy and possess both internal structures (their formation and logic) and external structures relating to “why each actor is encouraged to use them” (p. 223).

The purpose of the present paper was to propose an approach to analysing the PISA regime, with specific regard to mathematics, that can address not only the knowledge structures produced by the regime but also the ways in which students, teachers and other agents may be produced as subjects. In the next section, we clarify what we mean by the “PISA mathematics regime”, taken as an exemplar of global trends in education policy formation (Ball, 2008).Footnote 1 We then outline and discuss the ways in which PISA has been taken up and responded to in public debate, policy and research. We propose a further mode of critique of PISA that allows us to analyse first its knowledge structures, making use of Bernstein’s theory on the formation of pedagogical discourse, and, then, drawing on Foucault’s last work on the self-constitution of the self, the ways these knowledge forms provide spaces for subjective experience. This mode of critique is illustrated by applying it to an example text drawn from the PISA mathematics regime. Out of these discussions, numerous theoretical concerns and difficulties arise: finally, therefore, we identify some of these as directions for follow-up research. With these aims in mind, we argue that our approach to the analysis of the PISA regime may provide a means of explaining the formation and logic of knowledge for policy. Of necessity, this is only an outline of our approach. Given the obvious limitations of space, however, we emphasise at the outset that the work reported here is offered principally as an indication of the kinds of analyses required to engage the mathematics education research community.

2 The PISA mathematics regime

Before proceeding, we want to be more specific about what we mean by the PISA mathematics regime. We take the PISA regime in general to comprise the texts and technologies produced by the OECD in the name of PISA, but also the discourses and practices produced by these in the realms of policy, practice and research. In the words of Carvalho (2012), PISA is “a complex of activities, objects and actors that generates diverse resources for social action in various social spaces” (p. 173). The PISA mathematics regime is the subset of this complex that serves to generate resources for action specifically in relation to mathematics and mathematics education.

The PISA regime includes texts produced for a variety of audiences directly by OECD and its agents at both international and national levels, for example

  • The tests themselves

  • Reports on the outcomes of testing

  • Technical reports

  • Policy statements

  • Documents produced for audiences of teachers, students and parents, including exemplification of the test materials and assessment rubrics

It also includes the technologies that lie behind the production of knowledge, including methods involved in

  • Constructing test instruments and test items

  • Population sampling

  • Definition and maintenance of quality of measurements

  • Data analysis

  • Formulation and dissemination of indicators and results

Together, these texts and technologies construct ways of thinking about mathematics, mathematical knowledge and mathematics education; about students, teachers and educational processes; and about how these are related to social and economic activity.

It is important to note at this point that PISA itself defines mathematics, or rather “mathematical literacy”, in terms that are consistent with the overall focus on preparation for adult life

Mathematical literacy: An individual’s capacity to formulate, employ, and interpret mathematics in a variety of contexts. It includes reasoning mathematically and using mathematical concepts, procedures, facts and tools to describe, explain and predict phenomena. It assists individuals to recognise the role that mathematics plays in the world and to make the well-founded judgments and decisions needed by constructive, engaged and reflective citizens. (OECD, 2013, p. 17)

The domain of mathematical literacy (the domain to be assessed in the PISA tests) is defined here in terms of “mathematics”, and there is an explicit orientation towards the application of knowledge learnt in school rather than the mastery of curricular content. However, elsewhere in the same document, it is “mathematics” that is being tested. There is thus slippage between these two constructs and an ambiguity about the distinction between mastery and application.

3 The reception of PISA in public debate, policy and research

The PISA regime provides resources that are integrated into existing ways of thinking about education. For example, Pons (2012) notes that the public discussion surrounding PISA in different countries to a large extent reflects “long-existing trends and characteristics of each national public debate in education” (p. 213). Nevertheless, as PISA resources are recruited, the debates are shaped by them.

the current PISA is commonly referred to and acknowledged as a valid and reliable resource for education policies and public action. In fact, one might say that PISA has achieved the condition of an obligatory point of passage (Callon, 1986; Lascoumes & Le Galès, 2007) for policy-making: it becomes an unavoidable (and ‘obvious’) provider of information ‘based on proof’, a tool that creates and allows the creation of new problems and imagined new tomorrows. (Carvalho, 2012, p. 183)

PISA is usually referenced, especially in the policy field, as a “global script” that is performed in national contexts (Ozga, 2012a, p. 166). However, as we indicated in the introduction, critical researchers document that PISA has brought about relatively few additional primary or secondary analyses of the data by national researchers, evaluators and experts. In contrast, it has been used extensively to support all kinds of arguments in national public debates (Pons, 2012). Furthermore, whilst similar kinds of argument can be found across countries, the arguments also depend on the interests, representations and cognitive predispositions of the numerous actors who individually or collectively support them (Pons, 2012). In many countries in Europe, PISA results have paved the way for significant educational reforms. For example, Ertl (2006) discusses three main areas that have been affected by the reception of PISA in Germany: the political discourse, changes in which have led to a wide-ranging reform agenda, including the introduction of national educational standards; curriculum development processes, where PISA has led to the growing importance of principles such as outcome control, orientation towards competences and external assessment; and the academic discourse, where there has been a reorientation of educational studies towards greater emphasis on the empirical research of pedagogic practice. PISA, thus, may be seen to drive reforms at a national level, though these are strongly shaped by preexisting local arguments and predispositions.

Mangez and Hilgers (2012), analysing the role of PISA as a “cultural product”, argue that the development of PISA is transforming the very form and shape of the field of knowledge at a global level. They show how the development of PISA has brought about a subordination of educational forms of cultural production to transnational “political and economic interests”, whilst the diffusion of PISA, in a sense, dissolves the very boundaries of (national) education policy fields and progresses through a “heteronomous definition of education” (p. 189). Educational researchers, internationally, have described PISA effects as the encroachment of markets into curriculum formation and as changes in the regimes of accountability, from professional and political to “technical accountability” (Ozga, 2012a, p. 167). The connection between PISA and the marketisation of curriculum formation is evident in the fact that Pearson, the transnational publisher of academic texts and standardized tests, has recently won a contract for generating PISA 2014 test items—furthering Pearson’s efforts to “control education” (see Gutstein, 2012).

The comparative study of Neumann et al. (2012) is somewhat different in emphasis. It discusses the specificities of the reception process in two post-communist countries, Hungary and Romania, with attention to the institutional changes facilitated by PISA in public administration and in the emergent instruments of regulation. These authors understand PISA as a particularly successful carrier of global ideologies of education regulation. Their aim was to identify the convergences and divergences in the patterns of the reception of PISA, in light of country-specific socio-histories of educational assessment. They argue that comparative data generate vernacular translations of problems and policy solutions, and “PISA’s local meanings unfold and alter in the national context of political, scientific and symbolic struggles” (p. 228). Interestingly, Hungary borrowed the concept and methodology from PISA and fabricated the national Assessment of Basic Competences as a tool to push the idea of schools’ self-monitoring, with Hungarian actors presenting themselves in the national policy domain “as messengers of the civilizing achievement of evidence-based policy making” (p. 237). In contrast, Romania considered PISA “as a diplomatic obligation” (p. 237). Romanian actors conceived it as a bureaucratic task, thereby resisting the implied policy consequences, and PISA had had a limited effect on shaping the logic of national education regulation in that country.

3.1 PISA in mathematics education research

Looking specifically at the field of mathematics education research, we ask how the PISA mathematics regime is shaping the practices of researchers. A search within the papers presented at the 2011 Congress of the European Society for Research in Mathematics Education (CERME7) and the 2012 Conference of the International Group for the Psychology of Mathematics Education (PME36) reveals a large number of references to PISA (and other large-scale international studies such as TIMSS).Footnote 2 The majority of these deploy PISA in a way similar to that suggested by Carvalho (2012) for policy-making—as an “obligatory point of passage”. The outcomes of PISA are cited uncritically (and generally briefly) as part of the rationale for the research, for example

  • Contextualising the research in response to a curriculum reform motivated by a government response to national PISA outcomes;

  • Identifying weaknesses in national performance, as defined by PISA outcomes, that “need” to be addressed by research and development; and

  • Justifying the choice of countries to consider in international comparative studies by reference to their relative positions in PISA rankings.

A smaller number of studies make more substantive use of the constructs and values of the PISA mathematics regime in framing their research, for example

  • Making use of PISA results of various kinds as benchmarks for interpreting their own data;

  • Framing the researchers’ theoretical and analytical approach in terms of PISA constructs (including, in particular, the definition of mathematical competences and the assessment framework for mathematical literacy);

  • Using PISA tasks as a tool for teacher professional development; and

  • Analysing PISA data in greater detail than that provided by OECD-published analyses.

The uncritical adoption of PISA constructs by such studies naturalises these as legitimate ways of describing mathematics and mathematics education practices.

Only one paper presented at CERME7 (and none at PME36) adopted any kind of critical stance towards PISA or its use. Cabassut and Villette (2011) having used the PISA definition of modelling to frame their study, then use their findings to challenge the kind of interpretation of PISA results at the national level that leads to policy makers trying to follow the systemic approaches of high-performing nations such as Finland. They suggest that looking at disaggregated data reveals differences within nations. This is thus a challenge to the local policy manifestation of the PISA mathematics regime rather than to PISA as a whole.

A similar search within the papers presented at the 2013 International Conference of Mathematics Education and Society (MES7), a conference at which many participants adopt sociological frameworks and critical perspectives, shows just two references to PISA. One of these (Norén & Björklund Boistrup, 2013) includes a critique of the practice (by PISA/OECD as just one agency among many) of categorizing students into groups. The other (Braathe & Otterstad, 2013) presents a critical analysis of how neo-liberal discourses, such as those associated with PISA, have been taken up in teacher education in Norway. Whilst the absence of uncritical references to PISA in the papers presented at MES7 suggests that this group of researchers stands outside the mainstream adoption and legitimation of PISA constructs, the paucity of explicit critique nevertheless suggests a lack of general recognition of the extent to which the PISA mathematics regime is shaping mathematics education and mathematics education research.

However, it is important to note that PISA and other large-scale international surveys have also leveraged debates that run counter to precisely the bureaucratic and market agendas they are often thought to support. For example, Oppedisano and Turati (2012) make use of econometric techniques on PISA survey data to investigate educational inequality (in this case in reading literacy rather than in mathematics), identifying a reduction between 2000 and 2006 in the levels of inequality in countries with decentralised educational systems and increased inequality in those with centralised systems. A comparable challenge is offered by Jurdak (2009), who uses TIMSS data to argue that some inequities in mathematics education are a result of factors such as levels of parental education and national levels of economic development that are “out of the reach of math educators and even national governments” (p. 153) and thus not susceptible to policy change. Taking an educational assessment approach, Doig (2006) argues that the information provided by the standard summative reporting methods of large-scale testing regimes such as PISA can have little effect at the level of mathematics teachers’ practice. He proposes alternative ways of using the data to inform teaching and enhance learning whilst noting that these would require substantial programmes of professional development and might need to overcome teachers’ existing perception that large-scale assessments hold little information of value to them or their students. Such studies give empirical and theoretical impetus to redefining educational problems in ways that suggest that alternative agendas can be enrolled within the logic of transnational bureaucratic marketisation.Footnote 3

We thus observe that the tendencies characterised by Carvalho (2012) and Ozga (2012a) as fabricatory are not unidirectional. They can also foreshadow forms of resistance immanent to the logic of these same fabricatory processes. For as Foucault (1978) suggests, “[w]here there is power, there is resistance, and yet, or rather consequently, this resistance is never in a position of exteriority in relation to power” (p. 95). And also, “[resistances] are the odd term in relations of power; they are inscribed in the latter as an irreducible opposite” (Foucault, 1978, p. 96). So for Foucault, the potency of power exists in the dimension of its logical reversibility. This observation frames the approach we seek to develop, with the aim of investigating PISA from the perspective of how its actions on mathematics curriculum and policy raise issues about its reception by both institutions and individuals.

Though there are numerous ways to approach this problem, we seek an approach that (1) details the formation of knowledge structures and (2) opens the black box of practices of the self. We note that, currently, work at the level of knowledge is mostly at a highly general level and we seek a theoretical frame capable of a high degree of specialization to education—and to mathematics education in particular. We need the capability to name and get to terms with the enormous variety of national, economic, cultural and micro-cultural variation documented in recent mathematics education research. To our knowledge, work at the second level, whereby knowledge agents (policy makers, researchers, teachers, students, parents) respond to the production of knowledges in their specificity in order to produce themselves, in Foucault’s terms, as legitimate subjects of the PISA mathematics regime, and self-organization and empowerment, is less understood. This analysis must be detailed and thoroughly situated in knowledge of the context if it is to show why, in relation to themselves, individuals act as they do under and against the PISA mathematics regime. Our analyses are thus oriented towards the interrelations between the PISA mathematics regime and practices of self.

4 Modes of analysis of PISA

The academic literature related to PISA is steadily growing. In this section, we shall describe four main modes of response that we have identified in this work, arguing that these constitute a space of critique framing the horizon of action of academic researchers vis-à-vis PISA. This will allow us to locate our own approach in relation to this space.

4.1 Further analysis of PISA data

A major group of research studies makes use of intra/international data sets to identify factors correlated with the different PISA test outcomes. These findings add to in-country curriculum debates, policy formation and implementation. For instance, focusing on teaching resources and teacher expertise, Ammermueller (2007) argues that, in Finland, levels of professional formal education of teachers might explain the relative superiority and narrower distribution (and thus possibly greater educational equality) of Finnish results compared to German scores and score distributions. Bodin (2005), writing from the French perspective, addresses matters of a similar kind. Analysing data at the school level Fuchs and Wössmann (2008) found that “institutional variation” accounts for 85 % of between-country differences, identifying correlation between high levels of school autonomy and higher levels of student performance.

Other studies use PISA’s comparative findings to motivate and give credence to debates about the governance of the teaching profession. For example, Malaty (2009) uses analysis showing that Finland’s superior results in the PISA tests are correlated with the quality of pre-service and in-service teacher education, positive characteristics of the “culture of the teaching profession”, and broad-based support in Finnish society for school education to make a plausible case for the involvement of higher education institutions in schemes to strengthen the mathematics proficiencies of primary school teachers.

Research in this first mode makes use of PISA data to address national and international issues of direct concern. It does not pose any critical challenge to the PISA regime. This contrasts with the other three modes, all of which question the methods and/or effects of the regime.

4.2 Critique from an educational assessment perspective

Critique from the perspective of educational assessment centres on issues of validity of various kinds (e.g., face, content, construct, parallel, consequential; Messick, 1989) and reliability. Studies in this category tend to consider PISA from within a test theory perspective and examine the construction of PISA instruments (test and questionnaire items and instruments) and their technical application. These are taken up as technical issues of reliability and validity, sampling and bias (e.g., McGaw, 2006, 2008; Wiliam, 2008). At issue are the ways items are tested, tests administered and test scores interpreted (Goldstein, 2004). For example, the debates relate to the application of item response theory (the appropriateness and application of Rasch modelling techniques), disputes about the dimensionality of item responses, and so on.

In this category, we can include Dohn (2007) who picks up a question relating to the construct validity (but see Tsatsaroni & Evans, this issue) of PISA test items: What does it mean to be competent in real-life situations? Dohn raises a serious methodological objection: PISA makes a faulty attempt to draw conclusions about absolute levels of knowledge and skills on the basis of an evaluation of students that is defined using methodological/statistical decisions made in accordance with the way one constructs a relative evaluation. She concludes her analysis by claiming that “PISA gives a relatively reliable assessment of ‘knowledge and skills for PISA’, that is, of how well students exercise competence within the PISA focus areas in one—and only one—‘real life’ situation, the PISA test situation” (p. 14), but that it cannot be assumed to be an adequate instrument for assessing the competence necessary in dealing with the variety of situations of life.

4.3 Critique focusing on cultural bias, distributional effects and equity

The third mode of critique we identify again focuses on the validity of PISA and other large-scale assessment regimes, but from the perspective of cultural bias and inequity. Cultural biases and fairness in large-scale assessment systems are clearly of relevance to OECD’s PISA, but pose considerable challenges. As Stobart (2005) argues, with reference to large-scale assessment systems within multicultural societies, “fairness is essentially a social process and judgement” (p. 285). Whilst the ways that assessment systems recognise and handle cultural and social differences are critical, it is also essential to consider issues of fairness in the impact of an assessment regime on the interpretation and implementation of a curriculum.

Sources of cultural bias identified by Nardi (2008) in a study of literacy tests raise issues about the validity and reliability of the PISA items. Cultural biases are present when students are disadvantaged by being asked for skills which have not been taught to them, but also when the abstract curriculum envisaged by the OECD is narrower than those used in reality. Nardi also identifies bias related to translation and related disadvantages for non-Anglophone countries. The cultural and social assumptions of large-scale testing have also been the subject of the research done by Cooper and Dunne (2000). This research focused on national assessments of mathematics in the UK, which, much like PISA, use “realistic” test items. Their findings and interpretations are of crucial importance as they focus on knowledge forms, their social (class, gender, race) assumptions, and their distributional consequences for students’ success and failure. More specifically, their research revealed that children from some social groups are likely to have “a better feel for this game” than others, showing that many children, unclear of the requirements of “realistic” test items, fail to demonstrate mathematical knowledge and understanding that they actually possess.

4.4 Critique focusing on “soft governance”

The final mode of critique relates to PISA’s role in “soft governance”. Soft governance or soft power is a form of governing which is not statutory or nationally bounded. It involves “the replacement of traditional, bureaucratically organised, command and control systems with networks of relationships, in which cooperation and coordination must be constantly negotiated and managed…and which rely on a mix of particular policy technologies, and on constant work by policy actors to maintain connections and coherence in re-spatialised governing relations” (Ozga, 2012b, p. 440). According to Ozga (2012a), “education and learning policies attempt to regulate and manage systems and populations in networked, rapidly moving and changing national and transnational contexts” (p. 166). She and her collaborators have been concerned in particular about PISA as an aspect of a “European project”, within a context of increasing cooperation between the European Commission and the OECD in the production of educational data and in “governing through data” (Grek et al., 2009; Lawn, 2006; Ozga, 2012b). Ozga (2012a) argues that PISA needs to be understood as one policy thread towards the “respatialisation” of “Europe” through the “redesign of institutions, the organization of networks and the flow of comparative knowledge and data” (p. 167).

Ozga argues that PISA is conducted through the systematic “fabrication” of new “accountability regimes” (Ozga, 2012a; Carvalho, 2012). But these regimes are only responsive to selective accounts of performance, the refiguring of evaluation/self-evaluation as the “conduct of conduct” (Foucault, 1991a) and the installation of the citizen as consumer. So PISA is here perceived as a knowledge-based and knowledge-generating regulatory tool: it conveys different ways of imagining and doing education and social relations, and at the same time, it plays a crucial part in the coordination of education policies and public action (Carvalho, 2012). As an instrument of public policy, PISA “constitutes a condensed form of knowledge about social control and ways of exercising it” (Lascoumes & Le Gales, 2007, p. 3, in Carvalho, 2012, p. 184).

In general, this mode of critique aims to reveal the ideas that PISA disseminates as main vehicles in regulatory processes. These include the idea of the primacy of the rational and a data-based model for coordinating action in the educational sector, as opposed to opinion-based or ideological coordination; the freedom of decision makers to be involved and to exercise mutual surveillance, materially and symbolically, as an effective practice; and a systematic assessment of student performances as a trustworthy resource for the steering of educational systems (Carvalho, 2012). These critiques describe the techniques of post-bureaucratic (and economic) rationalism as a dimension of government. Nevertheless, although they are concerned with questions around “useful” and “valued” forms of knowledge in policy instruments like PISA, they do not tell us a great deal about the specific knowledge structures considered legitimate or how such educational knowledge forms can construct subjective experiences, organise practices and provide techniques for the construction of the self.

5 Opening up the space of critique

In order to develop an approach to critique that enables us to answer in more specific terms the question of what PISA does, we turn first to Bernstein’s (1990, 2000) theory of the pedagogic device. This directs us to see that both the PISA texts and the modes of critique briefly discussed in the previous section relate to the acceptance/non-acceptance of particular rules for the production of pedagogic discourse and knowledge structures. We outline below a theorization of PISA taking up Bernstein’s theory of the pedagogical device as an analytical resource, illustrating the analytical method with an example PISA text.

However, in order to describe more adequately what PISA does, we also need to have a theory of how pedagogic discourse translates into material practice, processes of individual actionFootnote 4 and the construction of the self, a level of analysis which is still underdeveloped in Bernstein’s theory. This leads to the development and brief illustration of a further mode of critique of PISA, drawing on Foucault.

5.1 Analysis of knowledge structures

In this critique, the focus is on the rules underlying the knowledge structures of the PISA mathematics regime. Bernstein argues that it is possible to give an account of such rules. They are given by what he calls the pedagogic device.

Bernstein’s theory works at two levels: discourse and practice. Concerning the first, he draws attention to the ways in which what he calls pedagogic discourse is constituted through recontextualisation. It is important to understand that discourse does not refer to a concrete practice, but to the constitutive principles determining the rules and criteria of legitimate communications relating to content experienced at the level of practice. Two features of Bernstein’s theory are crucial for our analysis. First, Bernstein offers very useful detailed suggestions about how the pedagogic device is realised in practice within diverse concrete circumstances. In other words, using his technique, we are able to analyse the effects of knowledge transformations. He hypothesizes three sets of meta-rules for the production of this discourse: rules of knowledge distribution, rules of recontextualisation and rules of evaluation. We illustrate these in the example below.

Bernstein’s theory of the pedagogic device also helps us better understand the factors at issue in debates over PISA. This second major advantage can be realised by analysis directly on the elements of the PISA regime itself and indirectly on the critiques and the debates. This means we are able to mobilise all items, instruments, design, interpretations, evaluations, directly related to PISA, as well as the peripheral communications (fabrications), critiques, and so on, as our object of analysis.

To elaborate this important point, Bernstein introduces the notion of the recontextualisation of one kind of knowledge into another within specified fields. Fields pertinent to his studies are the official recontextualising field and the pedagogic or practitioners’ recontextualising field (e.g., middle-level policy makers; Morgan, Tsatsaroni, & Lerman, 2002; Singh et al., 2013). These fields lay claim to ownership of the pedagogic device, the pedagogic discourse it produces, and thus the possible orders and kinds of communication deemed valid. This, in turn, Bernstein hypothesizes, has implications for the structuring of practice.

The pedagogic device refers to the rules of discourse by which the entire complex of the mathematics curriculum (including classroom content knowledge, teaching programmes, assessment schemes and so on) is produced. This offers us analytical tools to address the question of the forms of knowledge legitimated by the PISA mathematics regime. We illustrate this in the following example.

5.2 Example: analysis of pedagogic knowledge structures

In this paper, we have space only to offer an illustrative example of direct analysis of one element of the PISA mathematics regime. This analysis is conducted at the item level, choosing an illustrative question and scoring rubric taken from sample questions published by the OECD (OECD, 2009). Whilst the details emerging from the analysis are clearly specific to the question chosen, the analytic approach is general. Our focus in this article is primarily methodological, demonstrating how the form of analysis can identify knowledge structures underpinning the PISA mathematics regime, but not claiming that our resulting analysis is comprehensive. We have chosen to take a test item as our example text because curriculum tasks are the most concrete expression of the pedagogic discourse, projecting a model (albeit partial) of the curriculum and of pedagogic practices. The specific item, shown in Fig. 1, has a structure typical of PISA items and exemplifies what PISA defines as “mathematical literacy”.

Fig. 1
figure 1

Sample item and scoring rubric. Reproduced with permission, (OECD, 2009), Take the Test: Sample Questions from OECD's PISA Assessments, PISA, OECD Publishing, http://dx.doi.org/10.1787/9789264050815-en

Full size image

We start the analysis by considering the three components of Bernstein’s pedagogic device: the rules for the distribution, recontextualisation and evaluation.

5.2.1 Distribution

PISA’s discourse of mathematics has restyled mathematics as “mathematical literacy”, defined as the capacity to use mathematics in context rather than as dealing directly with the knowledge and skills of school mathematics. This distinguishes a particular form of knowledge and order of meanings to be within the domain of the PISA regime: knowledge is neither the abstract, esoteric knowledge of school mathematics (or indeed that of the field of production in the academy) nor is it strictly everyday knowledge but is the adaptation of school mathematics knowledge to the situations of citizenship and consumption encountered in “the world”. The PISA assessment scheme further classifies contexts into four categories of “situation type” that are represented at the level of assessment items as personal, educational/occupational (defined as “school life, work life and leisure”), public (related to the local community or society), or scientific (concerning mathematics itself or the use of mathematics in other academic fields). These situation types are described as being in order of increasing “distance from the students” (OECD, 2013, p. 32), thereby proposing possible identities and trajectories for individuals and groups. The item we have chosen to illustrate our analytic approach is an example of a public situation. It has a structure common to PISA items: a stem presenting the student with information about a “real-life situation”; a set of questions related to this context; and a scoring rubric for each question indicating for the assessor how they should allocate marks to student answers. In this case, the stem of the question seems to draw on a genre of popular science magazines, referencing the authority of “many scientists” as a justification for the relevance of the topic of CO2 emissions. The graphical representation, whilst related in some ways to conventional representations that may have been encountered within the school curriculum, also has non-standard features (e.g., lack of an explicit vertical axis, combination of representations of absolute and relative measures on the same diagram); again, the idiosyncratic form of visual representation is characteristic of popular science media. The question is thus initially framed as concerning forms of knowledge available to the readership of such magazines (possibly educated middle class) rather than the knowledge of school mathematics.

5.2.2 Recontextualisation

Whilst the stem of the item might best be described as an instance of popular science discourse, presenting quantified information about an environmental issue that might be interpreted as related to national and international policy and citizenship concerns, as the item starts to pose specific questions, the knowledge that the student is required to demonstrate quickly moves away from the scientific and real-world reference, engaging them with a “pure” mathematical calculation. Even in part 44.1, however, the student’s task is not simply to calculate but to “show the calculation to demonstrate…”. The statement of this task suggests that the activities demanded of the student are communication (show, demonstrate), whilst the nominalisation calculation presents the mathematical process of calculating as an agent-less object. The 11 % result of this calculation is obtained using the passive voice, again obscuring agency. Thus, the student is distanced from the mathematical performance (“the calculation” is presented as having an unproblematic existence) and communication appears to be the privileged activity. Despite this, as the rubric shows, in order to earn full credit, the student must actually perform the necessary calculations correctly. On the one hand, marks are deducted if the student offers correct arithmetic expressions but calculates incorrectly—thus contradicting the distancing from calculation created by the question itself. On the other hand, most tellingly, partial marks are allocated for correct calculations, even where the student has proposed an inappropriate arithmetic formulation (e.g., dividing by the wrong value when calculating percentage change), thus contradicting the apparent valuing of correspondence to the “real world”.

In part 44.2, a narrative is introduced involving not only the information provided in the item stem but also knowledge of national and international citizenship—state membership of the EU. This is followed by the question: “Do you agree with Mandy when she says this is not possible? Give an explanation to support your answer”. The validity of the response cited in the assessor’s rubric depends on whether the lexical marker “possible” is taken to mean “possible in principle” (the “of necessity” interpretation) or “possible in fact” (the “contingent” interpretation). If it is the first, then Mandy is wrong precisely for the reason given in the rubric. However, there could be other evidence, external to the information provided in the item, that would give reason to believe that, for the specified period, the net change of the EU’s CO2 emissions, excluding those of the Netherlands and Germany in this period, was negative, or at least less than a total increase of 4 million tons (this quantity being the salient difference between the decline of the EU total output of CO2 in this period and the net decline in the output of Germany and the Netherlands combined). An answer based on such external knowledge would not receive any credit—despite entirely corresponding to the “real-world” reality. The allocation of marks privileges abstract necessity over real-world contingency.

Again, in part 44.3, in spite of the apparent focus on a “discussion”, the scoring rubric makes it clear that only “mathematical” approaches that draw on the information provided in the item may be considered valid. Other possible lines of argument, such as drawing on knowledge or evidence exterior to the question (which could certainly be considered valid in some “real-world” discussions), are excluded.

In all parts of the question, therefore, allocation of the full score depends on the student recognising the hidden assumptions embedded in the recontextualisation of mathematical knowledge in this assessment task, and which contradict the apparent valuing of real-world knowledge and the explicit privileging of communication over performance. This analysis is consistent with that offered by Cooper and Dunne (2000) of assessment items from UK national tests. Responses that in the actual “real-world” context might be considered valid are not legitimated by the PISA assessment scheme unless they make use of the kind of mathematics found in the school curriculum. In spite of PISA’s denial that they are assessing the knowledge and skills of school mathematics, it is these knowledge and skills that are valued by the marking rubric.

5.2.3 Evaluation

Question 44.1 begins: “In the diagram you can read…”. Here, the student is addressed directly as an apparently successful student who is able to interpret the diagram as required. This particular “you” may be read as the general “one” of more formal speech. Yet, simultaneously, the heightened modality of “you can read”, and the fact that the correct interpretation is explicitly provided, opens up the possibility that a particular “you” actually is not able to “read”, as stated, and therefore needs the help implicitly offered in order to answer the question. This “you”, a second “you”, brings to the question a real “you”, a personal “you”, one that might be taken by the student as “me”. The student then faces the instruction “Show…”. This, together with the use of the definite article “the calculation” suggesting a single possible correct answer, is a common formulation within traditional forms of pedagogy and, thus, by making the evaluation criteria more or less explicit, begins to demonstrate how the individual student is intended to negotiate their identity in the face of ambiguity among these possible “you”s.

In contrast, part 44.2 does not demand that the student engage with an explicitly mathematical task. Instead, a hypothetical “Mandy” is introduced—named informally by her first name and hence presumably to be taken as a peer or fellow student with whom the student is expected to identify. The student is asked to express an opinion, passing judgment on Mandy’s claim to have found a mistake, and to explain this opinion. This extends the domain of the instructional dimension of discourse beyond the mathematical skills of reading a diagram and performing a calculation to reasoning and argumentation. At the same time, the regulative dimension of discourse demands that the student should engage with others (even though only hypothetically in this test situation), not just participate as an individual.Footnote 5 The student is apparently afforded some authority in relation to the content and to her peers. This authority is, however, illusory, as the student’s judgment is only validated if it conforms to the “correct argumentation” indicated in the rubric. Similarly, in part 44.3, the single correct answer demanded by the scoring rubric is not signaled unambiguously to the student, who is invited to “give two possible ‘correct’ answers”. In this case, again, the absence of a definite article allows the possibility of more than two valid possibilities, whilst the inverted commas around “correct” marks it as contingent and unlike school mathematics where there may only be one interpretation of “correctness”.

This task, therefore, embodies a form of pedagogic practice that has apparently contradictory criteria of evaluation. In the instructional discourse, the student is addressed directly, as if free to express her own opinion, yet the criterion of evaluation (there is only one correct answer) is explicit and (if sample scoring rubrics such as these are shared by teachers and students) is likely to regulate the student’s response. In contrast, the regulative dimension of the discourse seems to position the student as one who can engage in argumentation and pass judgement on her peers. It appears, therefore, as if she is afforded some degree of authority in relation to the content and the process. Whilst the contradictory criteria might be confusing and intimidating for some students, depending on their social characteristics and school/learning experiences, the strengthened components of the regulative dimensions of discourse might be indicative of an emergent modality of pedagogy that involves also the project of identity change (and subjectivation), “ a project to shape the subjectivities, to create a citizen/consumer that increasingly ‘governs itself’ in the desired ways” (Beck, 2008, p. 132).

5.3 Towards analysis of practices of the self

As previously stated, the main purpose of this paper was to develop an approach to analyse the knowledge structures of the PISA mathematics regime, and to identify the tactics whereby individuals are recruited within it. Thus far, we have argued that Bernstein’s perspective provides recourse to techniques to make visible the “instrinsic grammar” (Bernstein, 2000, p. 38) whereby the PISA mathematics regime is produced as a body of pedagogic knowledge.

This recalls Foucault’s (1991a) notion of rationality whereby the regularities of formative discourse are analysed according to two axes:

on the one hand, that of codification/prescription (how it forms an ensemble of rules, procedures, means to an end, etc.), and on the other, that of true or false formulation (how it determines a domain of objects about which it is possible to articulate true or false propositions). (p. 79)

This perspective helps to examine how knowledge effects in their specificity can illuminate how individuals are encouraged to take up these effects. Following Foucault’s later work, Bernstein-inspired critique needs to be enriched in ways that enable researchers to explain agentic resistance to the PISA mathematics regime based on the rationality of PISA, not merely its negation.

Before proceeding, it is important to clarify Foucault’s idea of rationality which does not refer to an inner coherence of discourse, requiring from the researcher to uncover a single coherent truth and to provide explanations in terms of causality or structure. Foucault (1991a) himself states

I don’t believe one can speak of an intrinsic notion of ‘rationalisation’ without on the one hand positing an absolute value inherent in reason, and on the other taking the risk of applying the term empirically in a completely arbitrary way. I think one must restrict one’s use of this word to an instrumental and relative meaning. The ceremony of public torture isn’t in itself more rational than imprisonment in a cell; but it’s irrational in terms of a type of penal practice which involves new ways of envisaging the effects to be produced by the penalty imposed, new ways of calculating its utility, justifying it, graduating it, etc. One isn’t assessing things in terms of an absolute against which they could be evaluated as constituting more or less perfect forms of rationality, but rather examining how forms of rationality inscribe themselves in practices, and what role they play within them, because it’s true that ‘practices’ don’t exist without a certain regime of rationality. (p. 79)

There are two crucial differences between the accounts of Bernstein and Foucault. Bernstein (2000) takes his analyses as revealing the “hidden voice of pedagogic discourse” (p. 38). This predisposes his analyses to conclude that the self-regulated subject is essentially an expression of neoliberal discourse formed within the PISA regime. This conclusion is common, also, to the fourth mode of critique identified earlier, with its focus on PISA’s role in “soft governance”. Foucault’s perspective would expressly reject the unidirectional nature of such a conclusion. Rather, it would recognise a bidirectional relationship between the rationality of a discourse and its productivity—the possibilities it opens up for alternative accounts and selves. For example, in his studies on the “genealogy of the modern state”, Foucault attempts to show how the modern sovereign state—this political rationality—and the modern autonomous individual “co-determine each other’s emergence” (Lemke, 2001, p. 191).

Second, though Bernstein (2000) argues that his theory is not deterministic for the logical reason that the object of study, symbolic control, “cannot control what it is set up to control” (p. 38), it nevertheless sidesteps the issue of how the realization of pedagogic discourse ultimately connects with practice. However, Bernstein’s (2000) discussion of two opposing discursive mechanisms of pedagogic discourse, namely “introjection” and “projection” that presuppose different forms of engagement for individuals, and more generally his idea that different modalities of pedagogy afford the individual different spaces of freedom, brings us closer to the idea of Foucault. In fact, as Olson (2008) explains, Foucault’s writings contain and keep in tension two different interpretations of (governmental) rationality. One “holds that the way the state [or other agencies] articulates social relations, structures individual lives, and deploys power is to a great extent a function of what is thinkable within its particular political culture” (p. 333). In other words, though forms of rationality are not absolute or monolithic, nevertheless, they seem to have some independence from the practices within which they are inscribed and they seem to exercise some kind of formative influence on those practices, determining what is possible or reasonable within a certain domain of action. The other interpretation holds that “the rationality of governmental practices is ‘on the surface’ in an important way, available for the intentional use of people organizing social life. It is informed by the sets of technologies that happen to be available at any given time” (pp. 334–335). As Foucault himself (1991a) puts it,

The rational schemas of the prison, the hospital or the asylum are not general principles which can be rediscovered only through the historian’s retrospective interpretation. They are explicit programmes; we are dealing with sets of calculated, reasoned prescriptions in terms of which institutions are meant to be reorganized, spaces arranged, behaviours regulated. If they have an identity, it is that of a programming left in abeyance, not that of a general but hidden meaning. (p. 80; emphasis in the original)

This brief reference to Foucault, supplementing the Bernsteinian lens, allows us to characterise the PISA mathematics regime as a system/form of rationality. Crucially, however, we have not assumed that such a rationality can be erected as a universal totality. On the contrary, pedagogic discourse, though having general characteristics, is realised concretely within material practices. Thus, this approach generates a theoretical lens that accounts for the plasticity of the knowledge effects referred to by Carvalho (2012) and their remarkable capacity for localisation as documented by Neumann et al. (2012).

The analysis of knowledge structures using Bernstein’s pedagogic device as illustrated above includes identification of possible subject positions formed within the discourse of the PISA mathematics regime. However, it does not help us to address the issue of how individual subjects may take up, resist or move between such positions. We thus turn to Foucault’s discussion of “self” and discuss how it enables us to address the issue of how individuals are recruited by or enabled to stand aside from knowledge effects.

Foucault’s method consists of two dimensions (Howarth, 2000). In the archaeological dimension, his method of analysis is to bracket the human subject in order to see how the formal rules associated with discursive practice can demonstrate the formation of knowledge objects and statements. This trajectory involves a process Foucault refers to as objectivation. The analysis in the genealogical dimension is conducted simultaneously with archaeological analysis. However, unlike archaeological analysis, it aims at the strategic reinsertion of the subject. He uses the notion of “technologies of self” (Foucault, 1988) to refer to practices by which the self governs the self with respect to determinate forms of objectivation, a process he calls “subjectivation”. As already indicated, it is a characteristic of Foucault’s analysis that government of the self is integrally connected with government of the state. He refers to this as “governmentality” (Foucault, 1991b), and one of the great insights in his work on governmentality was the critical link he observed in liberalism—the object of his inquiries—between the governance of the self and government of the state, understood as the exercise of political sovereignty over a territory and its population (Lemke, 2001; Peters, 2009; Rabinow, 1984). The interplay between objectivation/subjectivation and governmentality Foucault characterises as a “game”, a durable preoccupation impacting both practice and discourse. He refers to this preoccupation as a “game of truth”. In order to understand governmental rationality, Foucault argues, we must understand how people govern themselves and others; that is to say, we have to see ensembles of practices as “different regimes of ‘jurisdiction’ and ‘verification’” (Foucault, 1991a, p. 79). As Olson (2008, p. 331) emphasises, governing is a matter of enunciating truths in the form of “codes” or “codifications” and “true discourses”. A code regulates ways of doing things (e.g., how people are to be graded and examined), it is “[t]he generalised formula that is used to partition spaces, organise bodies, and classify identities” (p. 332). True discourses “serve to found, justify and provide reasons and principles for these ways of doing things” (Foucault, 1991a, p. 79); “they are the fabric of a socially constructed, shared understanding of the world, which is in turn the means through which people and things are governed” (Olson, 2008, p. 332).

5.4 Example: analysis of PISA as a discursive field of agentic action

We return now to our example in order to consider briefly how the evaluative rules of the PISA regime and the discourse they give rise to register techniques of subjectivation whereby the student self is to become a legitimate subject of the concrete knowledge effect, elucidated previously in the Bernsteinian account. To this, there are not easy methodological answers. Foucault considered this question to be the equivalent of the problem of how to reinsert the subject into the analysis.

In part 44.2 of our example, we noted two junctures of decision making the student must or provisionally might face: between “necessity” and “contingency”, and between “you” as an abstract given and “you” as socio-culturally contingent (“me”). In relation to these, we might ask: what self do these choices propose as a legitimate subject within the discourse? But this would lead us back to the Bernsteinian analysis, conceptualising the self as a given knowledge effect, determined by the pedagogic discourse. Rather, we would wish to investigate in practice how people make use of the spaces created by the discourse in order to create and recreate their selves. Foucault would be inclined to start the other way around, asking in what way is PISA contingent as a self effect? It can be argued that this question opens the way to immanent agentic action, requiring us to explore (cf. Carvalho, 2012) in detail the political work presupposed; the agencies, agents and actors involved in its ascendancy and development; and the conditions under which the crystallisation of the PISA regime becomes a field of action in global education, individuals, groups, institutions and nations. From an empirical research perspective, to study subjectivation in Foucault’s sense would require not only analysis of the processes of construction of the PISA discourse from the point of view of agentic action (researchers, policy makers, all those involved with the assemblance of PISA) but also detailed analysis at the level of realisation of the pedagogic discourse and at the level of practice (schools, students, head teachers, parents, school consultants, etc.) that help to sustain (or disregard) this regime. For instance, we noted earlier in our analysis of the illustrative example that 44.2 employs a seemingly naive/playful narrative about a hypothetical “Mandy”. This narrative inscribes the practice of “peer assessment”—a current feature of curricula and textbooks in many countries—as part of the technologies of self-regulation and of governing of self and others. Archaeological and genealogical analysis about the reinvention of this older technique and the reconfiguring of the field of assessment and evaluation in its entanglements with knowledge-based regulation tools such as PISA open up a field of inquiry that promises to be very productive concerning questions of symbolic control and agentic action.

6 Final remarks

This paper has focused on the PISA regime—with specific regard to mathematics—understood as an exemplar of global trends in education policy formation. We discussed selective literature describing how PISA has been taken up and responded to in public debate, policy and research, identifying the extent to which PISA simultaneously shapes and serves global and local discourses of education, including in particular those of mathematics education. We also outlined different approaches taken by researchers to the study and critique of the significance and the effects of PISA. Within the field of mathematics education research, we found remarkably little critical response to PISA, even among those researchers with relatively high interest in social and sociological issues. By setting out an agenda to identify what the PISA regime achieves and its significance for mathematics education, we hope to stimulate further critical study of its actions and effects.

The contribution of the present paper to this endeavor is to propose an approach to the study and critique of PISA which aims to address simultaneously (pedagogical) knowledge structures and agentic action, identifying not just the positions the regime makes available for actors but also the technologies of the self through which individuals may take up or resist such positions. This approach has been exemplified through the analysis of a single text. As we have noted, we consider PISA as just one regime, albeit one that, because of its scope and influence, has especially wide significance. We would contend that the analytic approach we have proposed and illustrated has wider application to other international and national curriculum and assessment regimes.

Emphasising again that the work reported here is offered principally as an indication of the kinds of analyses required to engage the mathematics education research community, we will conclude with three remarks.

First, concerning political, pedagogical and socio-scientific responses to PISA, Carvalho’s (2012) conception of PISA as a complex of activities and actors generating diverse resources for social action in various social spaces commits us to a detailed analysis of how different fields of action are reconfigured, reconstituting and, simultaneously, deconstructing the field of symbolic control, in power games; enrolling and engaging agencies, agents and actors; provoking resistance and forming counter-discourses; solidifying spaces of action and transgressing boundaries; and opening new and “free” spaces for determination and self-regulation.

Second, the attempt to produce a complementary critical approach to PISA, based on some commonalities in the perspectives of Bernstein and Foucault, such as the rules of formation of discourses, but also significant differences in the ways they have inquired into knowledge/power relations, we believe, has been proven to be very productive. Analysis of the logic of PISA, what Bernstein would call the inner logic of emerging practices, as a consequence and response to global policy discourses on new forms of knowledge and pedagogy within the field of education and within the field of symbolic control, and analysis of current political rationalities, and the rationality of PISA, as proposed by the work of Foucault, point to “a range of hybrid possibilities” (Muller, 1998, p. 191) creating a complex field of contestation, which deserves detailed analysis within a systematic programme of research. This is very important in education research and especially in the interstices between specific knowledge production fields in education (like mathematics education research) and the field of assessment and evaluation. In particular, in such newly formed and still fluid spaces, it is important to identify and describe the nuances in the struggle for the “soul of the worker/learner/citizen” of our times. This entails, on the one hand, detailing the kinds of scripts developed and pursued within the PISA regime, as it is manifested in various local contexts. On the other hand, it is also necessary to develop an understanding of how individuals and collective actors engage in action under the global script of the “well-tempered learner” (Muller, 1998). The conclusion that PISA is simply a manifestation of the “ideology of neoliberalism” cannot be the final word.

Finally, the changes in curricula and pedagogies brought about by the action of global policy networks and technologies of power are rekindling interest in this very important area of education research (Seddon & Levin, 2013). Systematic and principled research within the field of mathematics education is required, recognising how researchers themselves are already repositioned in transnational spaces.