1 Introduction

‘Cross-national learning assessments’ refer to evaluation of learning that include multiple countries utilize tests that are uniform and standardized in terms of content, administration process, timing and scoring (Addey 2014; Mulongo 2014; UNESCO 2017). These assessments also involve large samples of test takers who are scientifically selected from comparable populations across countries (Wagner et al. 2012). The assessments can also be said to be comparative multi-country studies which contain a strong assessment component and utilize strictly equivalent definitions of target populations and standardized procedure. The objective of the assessments is to measure student achievement (UNESCO 2006).

Currently, the main cross-national learning assessments include the International Association for the Evaluation of Education Achievement (IEA) (e.g. Trends in International Mathematics and Science Study, TIMSS, and Progress in International Reading Literacy Study, PIRLS), the Convention on the Organization for Economic Co-operation and Development (OECD) (e.g. Program for International Student Assessment, PISA), the Laboratoria Latinoamericano de Evaluacion de la Calidadde la Education (LLECE) (e.g. TERCE), Programme d’Analyse des Systèmes Educatifs des Pays de la CONFEMEN (PASEC), Southern and Eastern Africa Consortium for Monitoring Education Quality (SACMEQ) and the Monitoring Learning Achievement (MLA) project (Addey 2014; Hanushek and Woessmann 2012; Kellaghan et al. 2009; Mulongo 2014).

Knight et al. (2012) point out that the main objective of conducting learning assessments is not just to avail information on a country’s educational outcomes, but to support policy—makers and other stakeholders in the education system—and to provide tools to inform policy formulation and resourcing decisions. Furthermore, UNESCO (2005) posits that the principal aims of these international/regional assessment programmes are to provide data for educational monitoring, particularly of Education For All (EFA) goals, as input into policy making in multilateral institutions and funding agencies. UNESCO (2005) further indicates that the aim is to achieve tables of literacy rates which are more reliable, which mean something in terms of the national reality which they represent, and which enable a sensible comparison for the purposes of allocating resources and effort.

The assessments help to integrate hitherto isolated national education systems into a global network, thus facilitating wider exchange of experiences that spur policy discussions (UNESCO 2006) based on benchmarking against best practices or best-performing education systems. Additionally, compared with national examinations, only cross-national assessments can provide information on whether school organizations in other countries have fewer disparities in quality of instruction, socio-economic, gender and ethnicity and if these factors have a positive or negative influence on learners (UNESCO 2006). Postlethwaite (2004) posits that cross-national studies encourage investments in education, curriculum reforms, the establishment of revised standards and the review of teacher training programs. The assessments also present the opportunity to reconsider disparities in school resources, school organization and teaching practices. However, Wagner et al. (2012) caution that concrete evidence that cross-national assessments have had any policy impact in developing countries is relatively sparse and largely based on non-scientific surveys and interviews. This current study attempts to address this gap of information in three African countries.

1.1 The research statement

One of the main reasons for initiating and implementing cross-national learning assessments is to inform curriculum reforms and policy decisions (Barrett and Sørensen 2015; Kellaghan et al. 2009; UNESCO 2006; World Bank 2004). However, even after years of implementation, there are little empirical data and the literature on the optimal use of the assessment findings and the effects that they have had on policy and curricula decisions, especially in developing countries (Kellaghan et al. 2009).

The relationship between research, policy formulation and practice in education has been high on the agenda of countries for a number of years (Ion and Iucu 2014). The debate and scrutiny have been in terms of the quality, relevance and impact of research on policy formulation and the overall relationship with improvement in education systems (Witty 2006). The focus on the nexus between research and policy formulation implies that education stakeholders are not satisfied with research for the sake of research, but are becoming even keener to see that education research leads to tangible results in education. This study is premised on this thinking, with the following research question: is there evidence that developments in educational policy and curricula reforms in East and Southern Africa are related to results and recommendations emanating from the cross-national learning assessments? The study adopts the theories of change (ToC) to draw inferences. There is limited literature on methodology for assessing the relationship between cross-national learning assessments and policy formulation. There being no common agreed upon matric, even with its limitations, the current study could potentially shed more light on the approach that could be adopted to fill this gap.

2 Literature review

There are mixed views on the impact and significance of cross-national learning assessments to participating countries. Those that support these assessments cite the ability to document the poor performance of a country relative to others at similar levels of economic development and help such countries to reform their policy and inform investments for the sake of human capital development (Greaney and Kellaghan 2008; Wagner et al. 2012). Other writers observe that cross-national studies provide hands-on training and equip national staff of participating countries with skills to design and use data collection instruments and other related skills such as computer-based management and analysis of data, policy analysis and development and report preparation (UNESCO 2006).

Those that argue against the cross-national learning assessments indicate that by adopting these quantitative targets; learners, teachers and countries are encouraged to adapt their behaviour to maximize perceived rewards even when there are dysfunctional education systems. The near impossible challenges of creating achievement tests that are culturally or educationally specific, hegemonic control of individual systems by multinational and donor institutions as well as demoralization of poor performing countries are other drawbacks postulated by the opposing arguments (Barret 2009; Goldstein 2004).

Education and development experts unanimously agree that learning outcome assessments are not an end in themselves and that these endeavours should inform policy reforms in education (Wagner et al. 2012; World Bank 2004). This is an issue that is explored in this study.

2.1 Impact in developed countries

Various studies have been conducted to systematically analyze and document the impact of cross-national learning assessments on policy reform and education practice (see Baird et al. 2011; Breakspear 2012; Figazzolo 2009). The findings are varied and sometimes controversial. Baird et al. (2011) argue that the cross-national learning assessments have been greeted only with ‘political rhetoric’ in France and England. Knight et al. (2012) assert that the influence of PISA on policy formulation is increasing over time at both the national and local levels. Breakspear (2012) concludes that there is credible evidence that indeed PISA methodologies have been embedded within national/federal systems for assessment and evaluation and that several countries have provided large-scale evaluation for systems that did not previously conduct national/federal assessments and that PISA has stimulated increased scope of national assessment systems, has complemented national data and helped to validate national results against an international benchmark.

Breakspear (2012) goes on to indicate that some policy makers have used the PISA assessment frameworks and instrument as a best-practice ‘model’ or ‘guide’ in formulating improved national/federal assessment policies and practices and that some countries have explicitly sought to incorporate and emphasize PISA-like competencies in revised national standards and curricula. More importantly, Breakspear (2012) cites specific examples of the influence of PISA: for instance, both Switzerland and Germany initiated significant education policy reforms in response to lower-than-expected results. New Zealand inspired by the high performance on PISA reinforced existing policies while the results did not surprise the USA, whose domestic evaluation programme that had already raised public awareness of its poor educational outcomes.

In specific terms, Germany used the PISA results to justify a new focus on ‘testing testing testing’, in other words, ‘result-based evaluation’ (Figazzolo 2009, p. 14). Other initiatives in response to these results include the launching by the education ministers of the so-called ‘seven action areas’ meant to improve the quality of schooling and learning. These seven priority areas of the Kultusministerkonferenz with regard to the results of PISA are: (1) improvement in language competence as early as preschool education, (2) strengthening of the link between the preschool sector and primary school with the aim of an early school entry, (3) improvement in primary education and the continuous improvement in reading literacy and basic understanding of mathematical and scientific concepts, (4) efficient support of educationally disadvantaged children, (5) thorough development and assurance of the quality of teaching and schools on the basis of binding educational standards and result-oriented evaluation, (6) improvement in professionalism in teaching with particular regard to diagnostic and methodical competence as an element of systematic school development and (7) expansion of the provision of school and non-school activities for the whole day with the aim of increasing opportunities for education and support with particular regard to pupils with educational deficits, and especially gifted pupils (Figazzolo 2009, p. 14).

The author further states that Mexico (as an example of a developing country), using PISA as a point of reference, decided to directly involve the Convention on the Organization for Economic Co-operation and Development (OECD) in the formulation and implementation of her policy The Alliance for Educational Quality was launched in 2008. This was lauded by experts as “a very peculiar reform, as it is based, for the first time, on an agreement between the government, the education union and the OECD itself” (Figazzolo 2009, p. 12). In the concluding remarks, the author asserts that PISA results greatly affected the education policy debate both within the global and national contexts. Figazzolo (2009) opines that PISA has continuously constituted the point of reference for governments in their educational reforms, especially in testing pupils and evaluating teachers, resulting in reforms inspired by efficiency considerations.

Breakspear (2012) asserts that although specific policy initiatives were not immediately available, Hong Kong, China, Spain, Canada, Norway and Poland increasingly valued the skills assessed in PISA and that these countries adopted the PISA instrument to monitor their own performance and equity. Baird et al.’s (2011) exploratory case study concluded that there was some form of national pressure for extra congregated structures or data collection in Canada, but that there was little information about the policy response in Shanghai, China, as response to PISA.

2.2 Impact in Africa

A paper by the World Bank titled Assessing Student Learning in Africa (2004) provides examples of outcomes of cross-national assessments in the context of Africa. For instance, the paper citing SACMEQ indicates that results from these assessments have been utilized in areas of policy debate and featured in presidential and national commissions. Zimbabwe and Namibia, prime ministerial and cabinet reviews of educational policy in Zanzibar, national education sector studies in Zambia, and reviews of national education master plans in Mauritius are cited as examples. Moreover, in Mauritius, SACMEQ data were used in a debate on strategies to mitigate the harmful role of the Certificate of Primary Education and to support improvement in the pupil-to-book ratio. In Seychelles, SACMEQ results initiated a debate on streaming and caused education managers to reappraise the adequacy of the resources provided for education. Kenya, for example, introduced benchmarks—such as desks per pupil and books per pupil—for the provision of classroom facilities.

The World Bank (2004) goes on to indicate that the results from Monitoring Learning Achievement (MLA) were used in the preparation of education reform programmes in Malawi and Uganda, and that PASEC results were used in country analysis reports in Burkina Faso and Niger. Further, in Kenya, education indicators adapted from SACMEQ were used to supplement the educational statistics collected by the Ministry of Education (UNESCO 2006), and in 2002, World Bank’s Educational Sector Analysis made extensive use of SACMEQ’s findings to develop proposals aimed at addressing deficiencies in the Kenyan education system. This led to the establishment of the Education Sector Strategic Plan (Goverment of Kenya 2005). Moreover, the same results were useful in informing the development of the Public Expenditure Framework (UNESCO 2006). Participation in these assessments also helped to build the capacities of Madagascar, Mali, Mauritius and Morocco countries that later carried out their own assessments. The World Bank (2004) further argues that in Kenya and Zimbabwe, data cleaning methods used in SACMEQ were adapted for school census data and that participant countries in the PASEC programme improved their capacity in test construction and in the design and execution learning assessments.

Wagner et al. (2012) also offer an analysis of the impact of the international learning assessments on education development in developing countries. They postulate that these assessments have motivated regulatory and behavioural policy reforms, have helped create a learning environment in which assessment specialists have improved their technical skills and related performance, and have helped increase transparency regarding education system outcomes and human capital development in the participating countries. Moreover, as far as policy is concerned, Ross and Genevois (2006) indicate that results from cross-national assessments (especially SACMEQ) featured in presidential and national commissions, in cabinet policy plans and informed planning in education. Prime ministerial and cabinet reviews of educational policy in Zanzibar and reviews of national education master plans in Mauritius are cited as examples. Furthermore, in Namibia, the results from SACMEQ inspired the introduction of interventions such as the English Language Teacher Development Programme (ELTDP), an expansion of the Basic Teachers Diploma Programme (BETD), introduction of the Efficiency Programme Management Policy Coordinating Committee and the 5-year (2001–2006) Ministry of Basic Education, Sport and Culture strategic plan (UNESCO 2006). In addition to these contributions, the results from these assessments are alleged to have justified the granting of donor support. In Guinea, for example, the results of the PASEC prompted the government and the World Bank to develop a programme to promote instruction in local languages in the early grades.

As far as capacity building is concerned, the World Bank (2004) asserts that following the MLA project, several countries, including Madagascar, Mali, Mauritius and Morocco, carried out their own assessments. Citing practical cases in Kenya and Zimbabwe, the paper argues that data cleaning methods used in SACMEQ were adapted for school census data and that participating countries in PASEC improved their capacity in test construction and in the design and execution of learning assessments. Commenting on SACMEQ, Best et al. (2013) observe that these assessments have been hugely successful in influencing policy and the authors recommend further investigation to gain deeper understanding of what contributed to the success of this assessment programme.

This discussion has shown the positive outcomes of participating in the cross-national learning assessments. However, information from certain countries may not be as positive; a World Bank study (2004) in Ethiopia, Malawi, Niger, Nigeria, South Africa and Uganda concluded that while the findings of national assessments were sometimes covered in the media, in none of these countries did they feature in parliamentary debate. In only one country were the findings used to justify the granting of additional resources to schools. In four countries, the results were shared with curriculum authorities, but only in two countries was feedback provided to teachers or schools, and in only one country was feedback provided to textbook publishers.

Best et al. (2013) aver that the large-scale assessments have impacted on the ways in which analytical political decisions, strategies and policies are constructed. Citing the example of Morocco, Gilmore (2005 cited in Best et al. 2013) indicates that the country changed its orientation and perception of the cross-national learning assessments as a mechanism of sanction and instead focused on improving the quality of education in the country.

However, looking at these discussions as pertains the impact of the cross-national learning assessments, there seems to be a conceptual misunderstanding of the term ‘impact’: for instance, phrases such as ‘results were used in national debates on quality of education’, ‘the minister gave a speech citing results of the assessment’, ‘results featured in presidential and national commissions’ have been used commonly to describe the impact of the assessments in the participating countries (see World Bank 2004, 2007). The higher-level results that may have resulted from this participation are not presented. This issue is discussed under Sect. 5.

3 Theoretical framework

The theory of change (ToC) was used as the theoretical framework in the present study. Retolaza (2011, p. 4) defines ToC as ‘a thinking-action approach that helps us to identify milestones and conditions that have to occur on the path towards the change that we want to see happen’. According to Leeuw (in press), this theory expresses an intervention logic between inputs, outputs, outcomes and impacts. The ToC is an important management, monitoring and evaluation tool (O ́Flynn 2015). It is especially useful in analyzing critical assumption, both long-term and short-term that underpin social and community development initiatives (Barnett and Gregorowski 2013). It is an important approach in the evaluation of complex and dynamic social programs and interventions (Westhuizen and Abrahams 2002). This paper has contextualized the theory of change to explain how the participation in cross-national learning assessments could have contributed to policy and curriculum outcomes in the three research countries.

4 Methods

A descriptive qualitative multi-case study research design was adopted. Case study is an in-depth analysis of a particular research problem rather than a sweeping statistical survey and is often employed to narrow down a broad field of research into one or a few easily researchable examples (Curtis et al. 2014; Zainal 2007). Two countries from the East African region, namely Kenya and Tanzania as well as South Africa, were nominated as the cases. Three inclusion criteria were used to select the three countries, namely (a) that the country had participated in various cross-national learning assessments, (b) there was no evidence of recent relevant data on that particular country as far as the research topic is concerned, and (c) that the country was convenient to access.

In terms of cross-national learning assessments, since 2000, Wasanga and Kyalo (2007) indicate that Kenya has participated in not less than five cross-national learning assessments, namely Uwezo, Southern and Eastern African Consortium for Monitoring Education Quality (SACMEQ), Monitoring of Learning Achievements in Lower Primary (MALP), Monitoring Learning Achievement (MLA) and Female Education in Mathematics and Science in Africa (FEMSA). The United Republic of Tanzania has participated in various cross-national learning assessments that include the annual learning assessments by Uwezo East Africa. Others include SACMEQ II and SACMEQ III. South Africa participated in at least six cross-national learning assessments, namely SACMEQ I and II, TIMSS (1995, 1999, 2003) and PIRLS of 2006 (Kanjee 2013).

4.1 Data collection and analysis

Data collection took place between August 2015 and March 2016. The study adopted a hybrid of two techniques to collect data to determine the contribution of cross-national learning assessments on policy and curricula formulation. Firstly, the study adopted Elks’ (2016) technique which involved desk review of documents and in-depth interviews with key informants to analyze the impact of learning assessment results on education policy and practice in East Africa. Secondly, to strengthen the methodology, the current study adapted summative content analysis that was utilized by Anil et al. (2014) to assess the resonance and use of the Uwezo data and publications in Kenya, Uganda and Tanzania. In terms of interviews, the Ministries of Education, curriculum development centres, national examinations bodies and civil society organizations implementing cross-national learning assessment programmes were nominated to participate in this study. The premise of purposively selecting these institutions is the central role they play in formulating education policies in any country. While inviting these institutions to participate in the study, care was taken to request for senior official conversant with the topic of concern, namely learning assessments and policy making/formulation. The key informants were asked to identify the policy and curriculum reforms that had taken place between the years 2000 and 2015 whose contents or formulation could be attributed to cross-national learning assessments. The period 2000–2015 is important because it is the period that was designated for the achievement of Education for All (EFA), goals (UNESCO 2006), with the cross-national learning assessments implemented under this auspice acting as a mechanism to provide data for monitoring these EFA goals.

Content review was conducted on the policy or strategic documents that were identified by the key informant interviewees. Content review of the policy and strategic documents was conducted to extract compelling text related to the research question, particularly the number of times specific learning assessment programmes was cited. The policy documents for summative content review were sourced from the Ministries of Basic Education or through searching from online databases and official government websites of the participating countries. The researchers developed a tool/checklist for keywords such as ‘learning’, ‘literacy’, ‘reading’, ‘numeracy’, ‘learning outcomes’ and ‘learning assessments’ were used to search the documents for the themes. Once the keywords were found, the researcher categorized the context in which they appeared into the following thematic areas/questions.

  1. (a)

    When was the policy document developed?

  2. (b)

    What does the policy document say concerning learning assessments, learning outcomes, literacy?

  3. (c)

    Does the document make any specific reference to cross-national learning assessment frameworks/programmes (such as SACMEQ, MLA, Uwezo and so forth?)

  4. (d)

    How many times are the cross-national learning assessment frameworks/programmes cited in the document?

5 Findings

The results presented in Table 1 show the key policy documents that were cited by the research participants as having been formulated and that specifically cited in content, the cross-national learning assessment programmes.

Table 1 Policy and curriculum reforms
Full size table

The study established that between the years 2000 and 2015, a total of 11 policy/strategic documents were formulated across the three countries, anchored on their participation in cross-national learning assessments.

5.1 Kenya

Kenya had the highest number of policy/strategic documents that were formulated most likely due to participation and recommendations emanating from learning assessments. The first document is the Sessional Paper No. 1 of 2005 titled ‘A Policy Framework for Education, Training and Research’. The aim of this Sessional Paper was to address the challenges of equity, access, relevance and quality. It was also meant to provide a policy framework for the education and training sector in order to meet the challenges of the twenty-first century. As stated by the Ministry of Education, Science and Technology (Government of Kenya 2005), the Sessional Paper resulted in the adoption of a Sector Wide Approach Programme Planning (SWAP) to the provision of education. To show the contribution of cross-national learning assessments in the formulation of this policy, a senior official of policy in the Ministry of Education (MoE) observed the following:

results from research [learning assessments] continuously revealed that we had challenges particularly in early grade reading and numeracy. The Sessional Paper No. 1 of 2005 was formulated partly to address this challenge and it focuses particularly on literacy and numeracy…. But the best studies that we could actually say have made a lot of difference are the Uwezo studies. Research Triangle International (RTI) also started their studies- that’s when we had inside information and realized that it’s possible our children were not getting the competencies required. Based on these, we have put in place a lot of interventions (Key informant, MoE).

The second policy document attributable to large-scale learning assessments is the Sessional Paper (2012), which is a Science, Technology and Innovation (ST&I) policy document meant to support country’s Vision 2030. The premise is that the foundations for socio-economic transformation is anchored on a progressive ST&I platform. According to the document, the goal of the ST&I policy is to build critical capacity and capability in ST&I that will create change and transform Kenya into a newly industrialized country through the utilization of knowledge as the driving force. Moreover, to underscore the contribution of large-scale learning assessments in the formulation of this policy document, a participant asserted that ‘in addition to Sessional No. 1 of 2005, findings and recommendations from the large-scale learning assessments further informed the writing of Sessional Paper of 2012’ (Key informant Interview, MoE Official).

The third document is the Task Force Report on the Re-Alignment of the Education Sector to the Constitution of Kenya 2010: this document provides recommendations on how to realign the Education sector to the Vision 2030 blue print, the Constitution (2010) and the East African Community treaty. Mainly, this Education Task Force Report (Government of Kenya 2012) addresses the issues of access, relevance, quality, retention, transition, equity, governance and management, in the light of the devolved governments to the County. The document cites the Uwezo (2011) learning assessment findings as the basis for drawing this conclusion. This indicates that the findings of this assessment informed the deliberations of the task force, which was constituted at the highest policy-making levels in education. Supporting this observation, a participant postulated that ‘recent task force reports such as the report on Realignment of the Education Sector to the Constitution of Kenya, 2010 borrow their thinking from findings and recommendations from large-scale learning assessments’ (Key informant, MoE Official, August 2015).

The fourth is the Kenya Education Sector Support Programme—(KESSP)—2005–2010. The KESSP was intended to ‘operationalize’ the budget for prioritized programmes. When accomplished, it was meant to ensure that the goals and objectives spelt out in the preceding education-related policy documents were attained. This is therefore not a policy document as such but a strategic programme. The KESSP document proposes the establishment of the Teacher Management Investment Programme that will help facilitate efficient management and utilization of teachers as critical resources necessary for better learning outcomes in schools. Other strategies outlined in the document include the establishment of the Learning Achievement Monitoring Unit for quality assurance purposes and institutionalization of the culture of monitoring of learning achievements, especially at the primary level. The document further proposes that the various learning assessment programmes such as SACMEQ, MLA and MALP should be analyzed with a view to harmonizing them into a coherent national assessment system. Finally, the document cites initiatives such as SACMEQ (6 times), MLA (6 times), MALP (13 times) and Female Education in Mathematics and Science in Africa (FEMSA; 4 times). This suggests that these cross-national learning assessment initiatives contributed to the shaping of ideas behind the KESSP programme.

The fifth strategic document is the National Education Sector Plan (NESP). The NESP is a 5-year plan that outlines the education sector reform implementation agenda in six priority areas. The areas are based on challenges affecting the sector and lessons learnt in the Kenya Education Sector Support Programme. The focus areas include sector governance and accountability; access; equity; quality; relevance as well as social competencies and values. Noteworthy features of the NESP include an emphasis on the quality of education to address the challenges arising from the introduction of the Free Primary Education initiative in 2003 and the subsequent Free Day Secondary Education in 2008. In terms of the contribution of cross-national learning assessments, the document relies on findings and recommendations from these assessments to arrive at inferences and the outlined policy guidelines. Moreover, NESP recommends a harmonization of the different approaches currently implemented in Kenya (such as Early Grade Reading Assessments—EGRA, Early Grade Mathematics Assessments—EGMA and Uwezo). The contribution of large-scale learning assessments in the writing of this document was elaborated upon by a senior education ministry official who posited the following:

..in it (NESP), we factored in the results and recommendations from the learning assessments, the results that indicated that we had challenges in particular, early grade reading and numeracy…..It was a useful coincidence that while we were discussing this [document], results from Early Grade Mathematics Assessment (EGMA) and Early Grade Reading Assessments(EGRA) by Research Triangle International came out…..it borrows its thinking from findings and recommendations from large scale learning assessments” (Key informant, MoE).

The sixth document is the Basic Education Act No. 14. The purpose of this Act was to give effect to Article 53 of the Constitution and other enabling provisions; to promote and regulate free and compulsory basic education; to provide for accreditation, registration, governance and management of institutions of basic education; to provide for the establishment of the National Education Board, the Education Standards and Quality Assurance Commission and the County Education Board. The date of commencement of the Act was 25th January 2013. Commenting on the extent to which large-scale learning assessment influenced this Act, a participant observed the following:

The Basic Education Act and Vision 2030 clearly state that the curriculum should be competence-based. While designing the revised curriculum, the Kenya Institute of Curriculum Development should consider this recommendation. But most importantly, the genesis of these recommendations on the need for a competence-based curriculum are the learning assessments such as the English Language Literacy in Kenya project, SACMEQ, Uwezo and Monitoring Evaluation Learning (Key Informant Interview RTI).

This participant highlights an important link between the recommendations from the large-scale learning assessments, the Basic Education Act and the curriculum review debate and efforts in Kenya, as discussed below.

In terms of curriculum review, comparatively, between the three countries covered by this study, Kenya has had the least curriculum reform initiatives in the last 15 years. The ‘reforms’ have been alterations to the current 8-4-4 education curriculum or pronouncements of intent to review the curriculum. Even with minimum curriculum reforms recorded in Kenya, it is apparent that the cross-national and national learning assessments have created awareness amongst educationists in this country and informed debates on the curriculum. A key informant from the Ministry of Education, Science and Technology observed the following:

Through Primary Math and Reading Initiative (PRIMR), we got hands-on lessons on why children were not learning, getting the skills that are required; we realized there could be a problem with the syllabus. So we reorganized the syllabus in the way children learn. Thereafter, we developed curriculum support materials in form of books…

A participant from African Population, Health and Research Centre (APHRC) underscored similar views;

…..that is exactly what we are doing at the moment, developing a competence-based curriculum ….for example, in reading, we should be able to do this and this and that, described and well defined so that when for instance I am in grade 6, nobody should come and tell me that I have not learned. There should be a target. At the moment, the target is illusive, mostly focused on chasing for the mean score [in summative, high-stake national examinations.

Corroborating these argument about the impact of large-scale learning assessments and curriculum reforms in Kenya, a participant from the Kenya National Examinations Council indicated that ‘as a result of the SACMEQ and Monitoring Education Learning recommendations, holiday tuition in Kenya was abolished, HIV/AIDS integrated in the draft curriculum and the mean grade for entry into teacher training raised’ (Key informant, KNEC Interview, August 2015).

The government has acknowledged the need to review the curriculum. To affirm the government’s commitment to reforms based on learning assessments, a key informant from the Kenya Institute of Curriculum Development (KICD) reported that the Institute has developed a curriculum policy which by the time of this study, was at advanced stage of being released. He asserted that competence-based learning is incorporated into the policy, as well as issues of Science Technology Innovation and Mathematics (STIM). This assertion seems to signify a link between research, policy and curriculum development.

5.2 Tanzania

Within the period covered by this study (2000–2015), there have been at least five education-related policy and strategic documents written in Tanzania, namely the Secondary Education Development Programme II (SEDP II) document, the Higher Education Development Programme Strategic (2010–2015) document, the Education Sector Development Programme (2008–2017) document (Revised Edition), the Big Results Now in Education Programme and the Education and Training Policy of 2014. However, between them, only the Education Sector Development Programme (2008–2017) Revised Edition and the Big Results Now in Education Programme documents cite specific cross-national learning assessments.

The Education Sector Development Programme was revised and launched in August 2008 in order to improve educational provision to achieve poverty alleviation. It is a key part of the systemic public-sector reform that was to support the National Strategy for Growth and Reduction of Poverty (MKUKUTA). This document is also a macro-level plan that sets out four major pillars clustered outcomes that tie all elements of the education sector (basic, higher, technical, vocational and folk education) together, in a coherent and synergic manner. It envisages the creation of a well-educated nation with high quality of life for all by facilitating improvement in learning outcomes, especially in rural and marginalized areas. It is built upon the recognition that a well-functioning educational system is a necessary condition and prerequisite for achieving improvements in educational outcomes and for growth through enhancing human capabilities. The document acknowledges that this will only be realized by instituting and strengthening performance and outcome-oriented monitoring and evaluation of education provision in the country.

The document notes the significant role that cross-national learning assessments such as Southern and Eastern Africa Consortium for Monitoring Education Quality (SACMEQ) and International Testing Services play in shaping education quality and discourse in the country, and it acknowledges the role of impact studies to monitor the quality of education.

Initiated in 2012, the Big Results Now (BRN) aimed at transitioning Tanzania from low- to middle-income economy. The programme which covered the period 2013–2014 focused on six key priority areas necessary to spur improved economic area, namely (i) energy and natural gas, (ii) agriculture, (iii) water, (iv) transport, (v) mobilization of resources and (vi) education. As far as education is concerned, the Big Results Now in Education Programme (BRNeD) aimed at improving the quality of education quality in primary and secondary schools as measured through a set of key indices, namely national average performance of grade two students in reading and numeracy; percentage of teachers found in classrooms during spot checks; and percentage of teachers with minimum knowledge in mathematics and languages in primary schools (Bank 2014).

The National Key Result Area Roadmap for Education (2013/14–2015/16) (United Republic of Tanzania 2013), is a document that operationalizes the BRNeD. This document promotes the adoption of some of the assessment models used by several cross-national learning assessment programs. The models include Uwezo (cited twice) Early Grade Reading Assignment (cited twice) and an Early Grade Mathematics Assessment (cited twice as well). As a further indication of the significance of cross-national learning assessments in shaping this programme, the World Bank appraisal document for funding of the BRNEd programme (World Bank 2014) acknowledges cross-national learning assessment programmes such as Research Triangle International (RTI) and Uwezo. The document avers that national large-scale assessments of literacy should be cross-checked with reports by these two cross-national learning assessments. Moreover, World Bank (2014) acknowledges the role of Uwezo’s assessments in shaping education discourse and debate on quality of education in Tanzania.

Although the BRN initiative held the promise of accountability and result-based management (OECD 2013), the program faced several challenges such as misalignment with government budgetary cycle and underfunding (Republic of Tanzania 2013) as well as a lack of political goodwill because of regime change. A participant observed the following:

To me, BRN was a short-term solution. Today, BRN is no more. It has been integrated in the government departments, especially in priority Ministries such as Education. BRN was a program that intended to instil result-based thinking in the civil service, that is, if you worked hard, you got rewarded. The programme achieved some good results, and some of the ideas have steered the course of LANES (Literacy and Numeracy Education Support) programme. However, the BRN ended prematurely. I think it ended because it was regime based. The program was pioneered by the fourth regime and was not well entrenched by the time the fifth regime took over. It started late in the fourth regime and was difficult to sustain. If it is not your baby, it is difficult to nature! (Key Informant B, LANES).

In terms of curriculum reforms, the present study established that three curriculum reforms took place in Tanzania between 2000 and 2014. These include the Curriculum for Advanced Secondary Education in Tanzania, the Curriculum for Ordinary Level Secondary Education in Tanzania and the Curriculum for Certificate and Diploma Teacher Education Programmes in Tanzania. These curricula are robust in terms of learning outcomes, and competency-based teaching. In addition, the curricula demonstrate innovativeness by proposing competency-based assessment methods instead of methods that rely on the memorization of facts.

The study established from an informant from the LANES Program that the government through the Tanzania Institute of Education, (TIE), changed the curriculum to reflect the 3Rs (reading, writing and arithmetic) in standard 1 and 2 and more time (70%) allocated to the 3Rs. The informant further observed that the 3Rs concept demonstrates how Tanzania is committed at the national level to prepare curricula that responds adequately to the need for better learning outcomes and assessment/monitoring. Corroborating this assertion, a Ministry of Education respondent observed that ‘since 2012/2013, there was a lot of discussion about improving literacy and numeracy skills of the children in the lower grades, and this was a big agenda in the Big Result Now (BRN) initiative. The BRN in the end of 2012, had already decided amongst the 9 initiatives, one of those would be improving the 3Rs’ (Key Informant, Ministry of Education and Vocational Training). Despite this, none of the curriculum documents acknowledge/or cite cross-national learning assessments programmes.

5.3 South Africa

Amongst the key education-related policy/strategic documents formulated in South Africa during the 2000–2014, are the Action Plan to 2014: Towards the Realization of Schooling 2025; Action Plan to 2019: Towards the realization of Schooling 2030; Handbook—Introduction to Whole School Evaluation; White Paper for Post-School Education and Training; and The National Development Plan 2030. These documents are strong on learning outcomes and serve as proponents of literacy and learning assessments. However, only three amongst the five cite cross-national learning assessment programmes, perhaps an indication of the role they have played in influencing the contents. The three are: The National Development Plan 2030, Action Plan to 2014: Towards the Realization of Schooling 2025 (Department of Basic Education, 2014) and the Action Plan to 2019: Towards the realization of Schooling 2030. Below, we discuss these three in brief:

The National Development Plan 2030 focuses on the critical capabilities needed to transform the South African economy and society by eliminating poverty and all forms of inequality by 2030. As far as the education sector is concerned, the Plan identifies the improvement in education outcomes as one of the priorities. The Plan is vocal on the issue of accountability in education and asserts that education outcomes cannot improve if accountability is not enforced throughout the education system. The document further elaborates on issues concerning literacy and numeracy. It is one of the documents in South Africa that presents a good awareness as far as technical matters about literacy is concerned. The document is comprehensive in that it clearly details the grade level competencies required of learners in the country. It is also a strong proponent of large-scale learning assessments—it challenges the country to improve her education until the performance of learners in the country is comparable to other learners in the more developed countries. Perhaps, as demonstration of the central role played by the cross-national learning assessment in the writing of this Plan, the document observes the following:

The Department of Basic Education aims to improve South Africa’s average Southern and East African Consortium for Monitoring Education Quality results for grade 6 languages and maths from 495 to 600 points by 2022. It also aims to improve average grade 8 scores in the Trends in Mathematics and Science Study (TIMSS) from 264 to 420 points by 2023. The Commission proposes that grade 8 scores in the round of Trends in International Mathematics and Science closest to 2030 should reach 500 points (Republic of South Africa 2010, p. 305).

Two important cross-national programmes are cited in this quotation, namely SACMEQ and TIMSS. They are both applied here as important yardsticks at baseline upon which future scoring on learning assessments can be gauged.

The Action Plan 2014 provides a high-level commitment of how the government plans to attain its goal on improving the quality of basic education. The Plan also promotes actions that were tested and proven to work in improving learning outcomes. Through its 36 indicators, it specifies the pace with which the country would take to achieve these outcomes. The targets for each province as elaborated in the document seem ambitious but attainable. For instance, in Sect. 6, goal number 6 states that ‘Improving all aspects of the quality of education and ensuring excellence of all so that recognized and measurable learning outcomes are achieved by all, especially in literacy, numeracy and essential life skills’ (Republic of South Africa 2011, p. 56).

As far as a focus on learning outcomes is concerned, goals 1 to 6 in the document concentrate on the proportion of learners achieving certain standards. In fact, except goal 4, all the other goals focus on the achievement of learning outcomes. In addition, goals 14 to 27 expound on the different interventions and types of support that schools and teachers require to improve learning outcomes. The document further acknowledges that the country’s schooling system performs well below its potential and that improving basic education outcomes is a prerequisite for the country’s long-term development goals.

The document is also a strong proponent of the Annual National Assessments (ANAs). As its name implies, the ANA programme is run annually for grades 1 to 6 and for grade 9. In 2011, it was run in February, with the focus being on what learners should have learnt in the previous year. Upon this backdrop of assessment, the Action Plan to 2014 Action Plan (Department of Basic Education, 2011) document notes that regular monitoring of education quality through learning assessments instils and strengthens accountability. While supporting learning assessments, it emphatically asserts that testing by itself does not guarantee improvement in learning outcomes and explains that the mechanisms by which ANA will impact on learning must be made very clear, partly because these influences the way in which ANA is carried out. The document cites cross-national learning programmes at least 73 times. The programmes cited include TIMSS (cited 19 times), PIRLS (cited 14 times) and the SACMEQ (cited 40 times).

The Action Plan 2019 covers the years from 2015 to 2019. Its focus is towards the realization of schooling by 2030. The document takes stock of key developments in the basic education sector. It reiterates many of the priorities outlined in the earlier plan (Action Plan to 2014), and thus, to a large extent, the priorities have remained the same as those in the earlier plan. However, there are shifts in emphasis, especially in the wake of lessons learnt, and the adoption of the priorities put forward by the National Development Plan 2030.

In light of learning achievement, this document identifies a sequence of events that influenced the roll out of interventions that impacted on learning outcomes in South Africa. The document further synthesizes the ANAs testing system and underscores that while it will take some years for ANAs to evolve into the world class testing system, it can still play a crucial role in heightening an awareness of the standards and the core learning competencies amongst teachers, parents and society as a whole. The document further acknowledges the central role played by standardized assessments and observes that the implementation of these assessments will lead to improvements in learning outcomes.

The document also indicates that the delivery of quality learning materials to the classroom and its functions is relatively well laid out in South Africa, but that it is marred with wastage and uneven service delivery across the country. It also documents the government’s admission that educational quality is a serious national challenge in South Africa. Thus, on these flaws, it recommends a more concrete national policy framework for monitoring service delivery.

Finally, the Action Plan 2019 references cross-national learning assessments. It underscores that in order to move forward, information from internationally standardized testing programmes should continue to be a key basis for gauging progress with respect to learning outcomes in South Africa. Like its predecessor, the Action Plan to 2019 is a strong proponent for both national and cross-national learning assessments to monitor learning outcomes.

As far as curriculum reforms are concerned, the study established that the cross-national learning assessments programmes and the Annual National Assessment (ANAS) could have influenced the contents of the revised National Curriculum Statement Grades R-9 (Schools) and the Curriculum and Assessment Policy (CAP) Statement in South Africa. A participant observed that ‘it is more of the outcomes from Annual National Assessment and Southern and Eastern African Consortium for Monitoring Education Quality (SACMEQ) that led to the formulation of the CAPS’ (Education Expert, University of Cape Town). Furthermore, ‘this underscores the fact that So Africa’s involvement in both internal (ANA) and cross- national (SACMEQ) has had a positive effect on policy formulation’ (Participant, Grace Community Development and Education). Another respondent observed the following:

The policy changes influenced by learning assessments have focused on trying to improve literacy. Some of those relate to the curriculum, the new curriculum statement, the introduction of the annual national assessments and introduction of the workbooks, many of the attempts to try to deal with teacher training. All of these have been part consequences of finding out how badly we have done in these assessments (Key informant)

As a further illustration of the nexus between participating in large-scale learning assessments and policy formulation, Table 2 shows the number of times specific cross-national learning programs were cited across the 11 policy instruments formulated in the three countries.

Table 2 Number of times the policy documents cited specific cross-national learning programmes
Full size table

The distribution of these citations by country and specific learning assessment programmes is shown in Table 3.

Table 3 Number of times specific cross-national learning programmes are cited in each country
Full size table

6 Discussion: the theory of change; the causality dichotomy of the study findings

At the centre of theories of change is the issue of causality/attribution (Leeuw and Vaessen 2009). On one side of the dichotomy are those who argue that results or outcomes in social development programme can never be isolated accurately for a specific program/project (attribution) and that many intricate intervening variables often interplay to produce socio-economic results, and therefore, a project or programme can only claim contribution (Almquist 2011). Mayne (1999) observed that measuring contribution and attribution are the most difficult elements in evaluating development programmes. Mayne (1999, pp. 2–3) defines contribution as being able to determine the value addition a programme has made to the outcome while attribution is making scientific determinization of the extent the success—or failures can be credited to the programme. Attribution claims a direct link between a cause and an effect (Department for International Development 2012, p. 38) while ‘contribution’, only makes a ‘donation’ causal claim about the results of an intervention which ‘usually involves verifying a theory of change in a way that takes account of other influencing factors and thus reduces uncertainty about the contribution the intervention is making’. The current study leans towards ‘contribution’. Focusing on attribution would have required sophistically econometric tools and resources. This is a major weakness in the current study.

Stathers et al. (2013) observe that a theory of change presents a logical explanation of the ways by which change is anticipated as a result of an appropriate mix of inputs through the execution of activities/interventions, which in turn produce outputs, then outcomes and impact along an impact pathway. Stathers et al. (2013) term the activities and outputs as ‘lower-level results’ while the outcomes and impacts are referred to as ‘higher-level results’. This is illustrated in Fig. 1. According to these writers, ‘the theory of change questions the assumptions about causality underlying the relationships between outputs, outcomes and impact’ (Stathers et al. 2013, p. 320).

Fig. 1
figure 1

(Source: Stathers et al. 2013)

Different levels of results that encapsulate the theory of change.

Full size image

Education experts aver that participation in the cross-national learning assessments is not an end in itself, but that a country participates in these programmes in order to learn about the quality of her education relative to other countries and so as to formulate policies to improve education (Hanushek and Kimko 2000; Hanushek and Wossmann 2007; OECD 2004; UNESCO 2005). In this research, this participation has been regarded as the intervention. For the intervention to occur, inputs such as financial resources, technical expertise, tests, software and so forth are required (Mbabu et al. 2014). The outputs emerging from these assessments include policy and curricula-related recommendations (see the conceptual framework in Mulongo and Amod 2017). The outcomes that can be attributed to or have been contributed by participating in the cross-national learning assessments by the three countries include the formulation of policies and curricula (Hanushek and Wossmann 2007; OECD 2004). The indicators of success in the theory of change (ToC) include the number cross-national learning assessment citations that have been made in the policy/strategic documents. Additionally, for attribution purposes, the study considered the views of education experts (the key informants) to ascertain which policies were influenced by the cross-national learning assessments. If these indicators were evident, based on the ToC, the study was to conclude that that there is sufficient evidence indicating that participation in cross-national learning assessments impacts policy and curriculum reforms and therefore worthwhile.

This paper has therefore contextualized the theory of change simply as a logical illustration of how social development occurs, that is, by how early changes and outputs arising from the participation in cross-national learning assessments relate to more intermediate changes in policy and curriculum reforms (Fig. 2).

Fig. 2
figure 2

Impact pathway adopted in the study

Full size image

From the literature review, one notes that the higher-level results (outcomes) of participating in cross-national learning assessments are missing in most of the reports. In the instances where the ‘the so what’ is highlighted, the results presented are mostly lower-level results (see Fig. 1). For instance, increased national debate or results presented to a commission of education as a result of participating in the cross-national assessment are not impacts. In this example, an outcome would be the action taken/implemented after the said debate or presentation to the commission was made—for example, were there any changes in education practices as a result of the heated national debate? Alternatively, did the commission of enquiry results in any policy recommendation or formulation? The current study therefore may have gone beyond these low-level results by analyzing the contribution of cross-national learning assessments on educational policy and curricula formulation in three countries Therefore, adopting a ToC enforces the discipline to focus on higher-level results and not just outputs (Mbabu et al. 2014). Relying on the ToC for higher-level outcome indicators, this paper has demonstrated to some extent the nexus between participating in cross-national learning assessments and contribution to education policy development and curricula reforms in education Kenya, Tanzania and South Africa. However, while adopting the ToC, the researchers were cautious not to get entangled with the politics of attribution versus contribution.

7 Limitations of the study

This paper provides an exploratory overview of a complex topic requiring sophisticated methodologies. First, the study was limited to a sample of only three countries; however, many countries, especially in Africa, have participated in one form of cross-national learning assessment or the other. Resources permitting, it would have been interesting to analyze the situation in all these countries. Moreover, the researchers only conducted content analysis on the 11 policy instruments that were cited as attributed to large-scale learning assessments. However, neither the analysis of the cons and pros of these policy documents nor investigation of their impact on learning outcomes was conducted. An evaluation of the extent to which these policies have been implemented in the sampled countries was also not conducted. The findings and inferences thereof are also based on a limited number of interviews, desk review and summative content review as the data collection techniques. Perhaps, more sophisticated methodologies in addition to these could have generated better attribution results.

The research has not considered the challenges and debates surrounding the education policy environment in the three countries. Furthermore, an ongoing critical examination of these policy frameworks and initiatives and their efficacy and impact is recommended. There is also a gap of information on the use learning assessment data in different phases of the planning cycle and factors that explain use of learning data. Kanjee (2013) further indicates that limited information is available on the effect of large-scale assessments on the actual learning and teaching that takes place in the classroom. These are potential areas for further empirical investigation.

8 Conclusion

For a long time now, cross-national learning assessments have been conducted not only to analyze the quality of education of one country relative to others, but to guide education policy, curriculum reforms and resourcing decisions (Benavot and Tanner 2007; Braun and Clarke 2006; Forster 2001 cited in Best et al. 2013). However, evidence of the nexus between cross-national learning assessments, policy and curriculum reforms is seldom referenced. The current study has attempted to demonstrate this connection with Kenya, Tanzania and South Africa as case examples. In conclusion then, firstly, the findings seem to indicate greater impact of the cross-national learning assessments in influencing policy in Kenya and South Africa than Tanzania. This study did not, however, establish the reason for the differing rates of diffusion between these three countries. This could be a potential area for further research. Secondly, despite its methodological deficiencies, the study reveals the impact of international practice (in this case, the implementation of cross-national learning assessments) on local policy environment. Specifically, the findings suggest that the objective of conducting the cross-national learning assessments, that is, to influence policy formulation, may have been achieved somewhat, in these three countries.