Introduction

With the increasing need to scale up immunization programs in diverse contexts, there is a rise in the demand for evidence on how interventions work and why [1,2,3]. The literature contains reviews that assess whether strategies such as reminders/recalls [4], education of mothers [5, 6], m-health interventions [7,8,9], or community mobilization interventions [10] are effective in improving routine childhood immunization outcomes. While there is extensive evidence on whether programs created an impact, there are gaps in our knowledge of how that impact was achieved. Consequently, there is a renewed focus at the international level on exploring pathways of change using an iterative, learning approach to monitoring and evaluation, with context-specific actionable data [11, 12]. Guidelines for evaluating complex interventions underline the need for understanding the causal mechanisms underpinning the intervention and for basing the evaluation on it to build an evidence base that informs policy [13].

Several tools such as the Theory of Change (TOC) have been used to depict the causal mechanisms through which programs are expected to create impact. Developing TOC is an exercise of drawing a diagram that links impact and the relevant inputs and intermediate outcomes and has gained prominence with the rise of complex, multilevel interventions in public health [14, 15]. With increasing complexity, TOCs were introduced to address the limitations of other tools such as the logic models and results frameworks. The logic models served more as a descriptive tool mandated by funding agencies and donors than as an explanatory tool to understand why the activities led to change. In a review, logic models were said to have a ‘missing middle’ that does not explain how outputs and intermediate outcomes translate to long-term outcomes [14]. Additionally, they did not facilitate critical thinking and learning as context, and the inherent beliefs and assumptions underpinning the program were not majorly accounted for [3, 16]. Since its initial use in the early 1990s, TOCs have been used as an implementation tool to aid program development, internal organizational communication, external dissemination and advocacy, program adaptation and learning, and more importantly, to determine how we evaluate program implementation and its effectiveness [14, 17,18,19].

In this manuscript, we present results from the analysis of the use of TOCs by reviewing published impact evaluations and relevant supplementary papers on immunization interventions in low- and middle-income countries (LMICs). We also explored how TOCs and causal pathways were translated into evaluation approaches for these immunization programs to understand the gaps in evidence on how interventions improve immunization uptake. We aimed to answer the following questions on the use of TOC for immunization interventions: a)Where and how were TOCs used b) How have TOCs and causal pathways translated into monitoring and evaluation strategies, and what are the implications for evidence on how programs work?

Methods

Criteria for selecting impact evaluations in this review

Evidence bases used for the search

We conducted this review of impact evaluations (IEs) from a representative sample of countries from LMICs. We searched for studies from the Evidence Gap Map (EGM) on routine childhood immunization [20] and the review conducted by the Yale Institute for Global Health [21] (Refer to Fig. 1 for details). The last search of EGM and Yale review were conducted in May and March of 2020, respectively.

Fig. 1
figure 1

Methods – search and inclusion of studies

Full size image

Inclusion criteria

We identified impact evaluations from select WHO (2022) and GAVI (2022) priority countries from both the EGM and Yale reviews based on three main criteria: (i) countries that have stagnant or low vaccination rates and (ii) those that have a sizeable evidence base (at least 3–4 studies) for a given country for synthesis of evidence. If three studies were not found from one of the three regions (Sub-Saharan Africa, South and South East Asia, or Latin America), countries with fewer studies were included. iii) Of the countries in each of these three regions, we selected one country that has a high routine child immunization coverage rate. Impact evaluations were included from the following 11 countries: India, Indonesia, Bangladesh, and Pakistan from the South and South East Asia regions; Ethiopia, Rwanda, and Nigeria from the African region; and Brazil, Nicaragua, Columbia, and Guatemala from the Latin American region.

We selected all interventions that impacted the routine immunization coverage of children aged 0–5 years or any intermediate outcome as causal factors. We only included experimental and quasi-experimental studies with counterfactuals published on or after 2010 to determine the causal impact of an intervention in comparison to standard or usual care.

Identification of secondary /qualitative studies

We identified feasibility studies, qualitative studies, formative or process evaluations, cross-sectional or observational studies, project documents or funder reports, and policy briefs related to the IEs from clinical trial registries and Google Scholar. In addition, we also carried out targeted searches of the websites of implementing agencies and funders. We also identified studies about the study area of the IE, discussing the intervention in question, if it was conducted in or referring to the year in which the intervention was implemented.

Review and analysis of the use of theory of change

We reviewed the above IEs to identify explicit usage of TOCs, program theories, logic frameworks, conceptual diagrams, and explanations of causal pathways of the intervention written in the form of intended impact or rationale for the program. We analyzed and extracted data from the identified IEs and their supporting documents using a list of questions (Refer to Table 1 for details). This list was populated based on the report on TOCs commissioned by Comic Relief and UKAID in 2011 [14, 17]. The questions also draw from the core principles identified by Coryn and his colleagues in their review on the use of program theory for evaluation and Medical Research Council guidance on evaluating complex interventions [22,23,24]. The studies were coded using NVIVO software to identify any text on TOCs, causal pathways, or the mechanisms of action of the programs. Attributes such as the year of implementation, scale of implementation, and use of program theory in designing programs were also captured. We defined large-scale interventions/programs implemented at the state or national level including studies implemented in many districts. All the other interventions were categorized as small- to medium-scale interventions.

Table 1 List of questions guiding the data extraction for the review
Full size table

The TOCs were classified in terms of scope and detail as Broad/Narrow or Deep/Shallow based on the work of ActKnowledge on TOCs [16]. In the context of immunization programs, pathways of broad/deep nature identify all the outputs and intermediate outcomes that can cause improvements in immunization coverage and identify all the pathways that can explain what, how, and why change has occurred. Narrow and shallow TOCs focus on none or a few intermediate outcomes affecting the outcome of the intervention. TOCs are classified as shallow TOCs if they do not provide any detail.

The data extracted for this review were analyzed to identify the extent of the usage of TOCs and how TOCs or causal pathways were translated into monitoring & evaluation. We present the results of the analysis using tables and matrices.

Results

The study reviewed 92 IEs along with 93 related articles to analyze the usage of TOCs (Refer to the Supplementary File 1). The studies evaluated different types of intervention strategies that were used either singly or in combination with other strategies. Of the 92 IEs, 47 were large-scale interventions implemented at the national or state level, and 45 were small- to medium-scale interventions (Refer to Table 2 for details). Forty-six interventions were implemented before the year 2010, while the rest were implemented on or after 2010. Most of the included studies did not have a TOC or described causal pathways in text with varying degrees of detail; hence, most of the studies were classified as narrow or shallow. A total of 19.56% of the studies used a theory of change, 56.52% described a causal pathway or used a conceptual diagram with varying degrees of detail, and 25% of the studies did not provide any information on how their intervention was expected to affect change. Only 19 of the 92 IEs explicitly outlined any assumptions associated with the implementation of the interventions. Of the interventions included in these 92 studies, 37 were single-component interventions, while 55 were multicomponent interventions. Forty interventions measured one or more of the outputs or intermediate outcomes leading to improved immunization coverage.

Table 2 TOC usage vis-a-vis characteristics of impact evaluations
Full size table

For this manuscript, we divided the results section into findings about large-scale interventions and small- to medium-scale programs and analyzed them considering the type of TOC, year of implementation, type of intervention, and TOC content (Refer to supplementary material 2 for details of data extracted from the included impact evaluations).

Large-scale programs

Most of the large-scale interventions were started before 2010 and involved an equal number of single and multicomponent programs (Refer to Table 3 for details). The large-scale interventions were implemented by governments or in partnership with governments using multiple strategies.

Table 3 Description of the intervention included in the review
Full size table

Theory of Change: Most of the evaluations of the large-scale interventions did not provide any TOC diagram. Only three of the large-scale interventions had a detailed TOC explaining the various inputs, outputs, and intermediate outcomes [32, 34, 67]. Thirteen of the forty-seven large-scale programs had no details on how the intervention was expected to have an impact (Refer to Table 2 for details).

The authors of 33 of the 47 IEs of large-scale programs provided details of the causal pathways in the text in varying degrees of detail, by referring to the literature in the area or through program theories (refer to Table 4 for details). The descriptions of the pathways of change can be described as shallow, as they provided little information on how these macro-level changes create impact. The authors of evaluations of these nationally implemented programs relied on the literature to explain how these programs could improve immunization outcomes.

Table 4 Examples of interventions providing causal pathways
Full size table

For example, Aggarwal cited studies that showed how access to all-weather roads could improve access to healthcare services by reducing transport costs, improving employment opportunities, and thereby improving incomes and increasing awareness through improved social interaction [113]. Kusuma et al. cited theories exploring how Conditional Cash Tranfers (CCT) could incentivize behaviors. The authors cited the human capital theory, which posits that consumers will invest in health if the expected private benefit exceeds the cost. In terms of vaccination, the cash element of the CCT was thought to help with financial barriers, and the conditionality element was shown to transfer health information on the benefit of vaccination and signal the importance of vaccination for both households and health workers [52]. The authors also used conceptual diagrams depicting the program elements and their links with outcomes [57].

Assumptions: Notably, in only 12 of the studies, the underlying assumptions were outlined in varying degrees of detail. For example, Okeke et al. noted that implementers assumed that the program would be effectively implemented without any roadblocks. Some authors have discussed the conditions that have been assumed to exist, in the causal pathways. Cristia et al. emphasize that there is a basic assumption about the number of potential providers in a market that would allow for competitive bidding [93]. Talukdar et al. assumed that the providers of health services would be aware of the voucher program [99]. In their evaluation of the performance-based financing (PBF), Sato and Belel assumed that stockouts and absenteeism were barriers to vaccination uptake [97]. However, largely, the interventions provided little to no information on the assumptions underlying the program.

Context: Many of these large-scale programs were implemented by the government and were rolled out at the national or state level. Understandably, the context considered for these programs has been the broader national healthcare context, such as the burden of disease, the lack of coverage of services, or the critical bottlenecks in the health system in general. The local context was not considered in many of these evaluations, which used national surveys to assess these programs.

Use of program theory or theoretical frameworks: Seven of the large-scale interventions were based on an established program theory. Examples include the PROCEED-PRECEED framework for designing interventions, human capital theory, and the social justice theory for conditional cash transfers. Many of the large-scale interventions did not cite any program theory.

Integration of TOC/Causal pathways in the evaluation of Large-Scale Interventions: In 14 of the evaluations of the large-scale programs, one or more of the outputs and intermediate outcomes on the causal pathway were assessed that led to improvements in vaccination outcomes. Our review showed that of the three evaluations with a TOC, two measured intermediate outcomes and outputs, leading to improved vaccination uptake. For example, in the evaluation of the Midwife Service Scheme in Nigeria, Okeke et al. explored how the intervention affects mothers’ knowledge levels, access to quality care, and perceptions of quality [32]. Saggurti et al., in evaluating health intervention integration with women’s groups, measured the link between the intervention and the extent of collectivization [34].

Of the 33 studies that described only the causal pathway, 12 measured intermediate outcomes. For example, in their evaluation of the voucher scheme in Bangladesh, Talukdar et al. measured improvements in the quality of health services, reductions in out -of-pocket expenditures, and increases in beneficiary awareness of health issues [99]. In an evaluation of the impact of the rural electrification program, Jyotigram Yojana in Gujarat, Chen et al. assessed the impact of the availability of continuous electricity on cold chain maintenance which could in turn affect vaccination outcomes [115]. However, most of the evaluations did not assess intervention’s impact on intermediate outcomes (refer to Fig. 2 for details).

Fig. 2
figure 2

Use of TOC in monitoring and evaluation

Full size image

Small- to medium-scale intervention in immunization

There were 45 small- to medium-scale interventions that were included in this review. Unlike large-scale interventions, which started before 2010, most small- to medium-scale interventions started after 2010 and included twice the number of multicomponent interventions as single-component interventions (Refer to Table 2 for details).

Theory of Change Diagram: Compared to large-scale interventions, the use of TOC is substantially greater in this category. Of the 45 interventions, 15 studies used a TOC diagram to explain how the intervention was likely to cause impact, while 18 of them described the causal pathway in the text. There are 13 TOCs in multicomponent programs compared to two single-component programs. Among the programs that used a detailed TOC were complex multicomponent community mobilization interventions [117, 118].

Eighteen of the evaluations of the interventions described a causal pathway in the text (Refer to Table 4 for details). Many of the interventions involved reminders/recalls or education interventions, and the cited literature that links education and reminders with improved knowledge and awareness among mothers [37, 45]. For community mobilization interventions, the authors hypothesized the pathway of impact. Sengupta et al. hypothesized that if outreach immunization clinics and community guardians were made available, then access to immunization would increase, thereby increasing childhood vaccination uptake [104]. Twelve interventions did not report any causal pathway in any form.

Assumptions: Only seven studies explicitly stated the assumptions underlying the interventions. Of these, six of the impact evaluations had a detailed TOC. For example, Adamu et al. stated that the intervention assumed that missed opportunities for vaccination rates would remain unchanged in the absence of the intervention [88]. Banerjee et al., in the evaluation of their intervention on incentives for immunization, reported being based on the assumptions that ASHAs (community health workers) were aware that the caregivers received the incentives, mobile recharge incentives were valued, and caregivers trusted and valued ASHAs [61]. Most of the studies that listed the underlying assumptions of the program had a detailed TOC.

Context: In many of the small- to medium-scale programs, the social, geographical, demographic, cultural, and health system context in which the program was being implemented was described. The context was also cited to provide a rationale for implementing the program in the region. For example, IEs provided a detailed profile of immunization rates in a region along with the socioeconomic profile to show the lack of resources and poor health in a certain context. In some cases, such as the team-based goals and incentives program in Bihar, India, a feasibility study was performed to assess the acceptability of the program in a certain region [107, 119].

The context was also discussed to explain the evaluation results or any unanticipated changes that affected the program or evaluation’s implementation. For example, Domek et al., in the evaluation of the text message reminder intervention in Guatemala, wrote that ‘political instability that led to unusually high levels of vaccine shortages affected the study and the team’s ability to track specific vaccine completion rates’ [70]. Wallace et al. found that ‘healthcare providers in the Indonesian Ministry of Health did not have standard protocols in place on how to remind parents about future vaccination visits, and that providers followed a variety of localized practices, including creating their own written reminders’ [80]. The context was discussed in interventions that had a feasibility study or a process evaluation as a part of the impact evaluation of the intervention.

Program theory and theoretical framework: Nine interventions cited a program theory or model underlying the interventions. The theories include diffusion theory, social cognitive theory, motivation theory, the framing of health theory, the health belief model, and the COM-B model.

Integration of TOC/Causal pathways in the evaluation of small- to medium-scale interventions: Evaluations of twenty-six of the forty-five small- to medium-scale interventions measured the outputs or intermediate outcomes leading to improved immunization outcomes (Refer to Fig. 2 for details). Of the programs with a TOC, 11 impact evaluations measured the outputs and intermediate outcomes that could affect immunization uptake. For example, Oyo-Ita et al., in their evaluation of the program on engaging with traditional religious leaders, measured the degree of community engagement; qualitative assessment of knowledge, attitudes, and beliefs; and satisfaction with TRLs [117].

Of those without a TOC but with a causal pathway, 13 had one or more intermediate outcomes. For example, interventions that focused on training or educating mothers about child health practices using face-to-face interactions or m-health interventions measured the knowledge levels among mothers [30, 36, 37, 43, 45, 75, 78]. Manyazewal et al. measured process indicators such as human resources, documentation, and supply for quality improvement initiatives implemented in Ethiopia [90].

Discussion

Our review showed that TOCs have not been used in the evaluation of large-scale programs, particularly health system strengthening interventions. TOC usage is more common in small- to medium-scale programs and multicomponent interventions and has increased since 2010. We also find that there is a weak integration of TOCs/causal pathways into many evaluations, which adds to the evidence gap on what we know about how and why interventions work. Additionally, aspects such as the use of program theory and the articulation of assumptions underpinning program implementation have been sparse, with only programs with a detailed TOC including these aspects in the evaluation. Our review also revealed that context has been mostly used to provide a background for the program—except in instances where a detailed TOC is provided—or as a part of the process evaluation performed alongside the impact evaluation.

Strengths and limitations

Our review is one of the few research publications on the use of TOCs in the context of routine immunization interventions. While there can be disagreements about the extent of the evidence included in this review, our analysis provides an initial understanding of the use of TOCs in the evaluation of immunization programs. With the increase in the use of theory-based evaluation techniques and the acknowledgment of newer approaches to evaluating complex interventions, much of the literature on TOCs is dominated by evidence on the process of developing TOCs [1, 120,121,122,124], the synthesis of multiple theories of change [125], or the validation of TOCs by checking whether they are consistent with the literature [126, 127]. Our review adds to the small body of evidence examining the extent of the use of TOC in a specific area. Other such reviews include a scoping review on the use of TOC for child health interventions [128], a systematic review examining the use of TOCs in public health in general [129], and a review to understand the use of TOCs in the development sector [14]. Unlike these reviews that focused mainly on high-income contexts, ours is the first to cover a large evidence base from 11 low- and middle-income countries in South and South East Asia, Latin America, and Africa in routine immunization.

It should be noted that our review is based on impact evaluations and relevant program documents identified for another review on immunization and its drivers. The analysis is limited by what is reported in impact evaluations, specifically for large-scale programs that do not have a detailed TOC. Additionally, the review included evaluations conducted at the national level and did not include subnational process evaluations or qualitative studies related to nationally implemented large-scale programs. We also acknowledge that we conducted this analysis on studies from a representative sample of countries included from the EGM and Yale review which were conducted in the year 2020.

Our review has the following policy and research implications for immunization programs:

  1. 1.

    Evidence on how large-scale health system strengthening programs affect immunization has limited use for scalability without a detailed TOC

Many of the evaluations of large-scale interventions, specifically the Health System Strengthening (HSS) interventions included in this review, did not include a broad TOC to explain how the intervention strategies could affect immunization. The review showed that in many large-scale HSS programs, evaluators had taken a linear and siloed view of how the various health system strategies work. Recently, there has been an increasing focus on taking a systems approach to evaluate HSS interventions [130]. The literature in the HSS has shown a gradual move from a view where theories of change exist for individual health system building blocks to a view where health system changes are viewed as interventions influencing the whole system [130, 131].

Our results agree with the review by the ReBuild and ReSYST Consortia for FCDO in 2021, where authors reviewed 96 HSS studies to understand the state of evidence [132]. The FCDO review showed that for HSS interventions focusing on service delivery integration, health financing, and logistics, evidence needs to go ‘beyond the effectiveness of an HSS intervention on service delivery, toward gaining an understanding of the processes and expected mechanisms of change’. For health financing interventions that facilitate functions within various health system elements, the review revealed that studies need to explore the underlying pathways of change, as several different mechanisms of action can result in the same health system effects.

In the context of immunization, using TOCs to understand how health system changes influence the wider system has greater importance, as routine immunization services are integrated with other child and development services, and can be influenced by other supplementary vaccination programs for emerging infections such as COVID-19. They can also depend equally on the demand side of the intervention.

  1. 2.

    Integration of TOC into evaluation needs to use multiple methods and approaches

Our review showed that the lack of a detailed TOC meant that there was a lack of or weak integration of TOC in the approaches used to evaluate the programs. Many of the evaluations did not measure any outputs or intermediate outcomes leading to improvements in immunization uptake. Evaluators took an impact evaluation approach for many of these large-scale interventions, especially HSS. They used quasi-experimental designs with secondary data from nationally administered surveys or routine administrative data to evaluate these programs. Few of them supplemented the findings of the results with concurrent process evaluations or used mixed methods. Consequently, the HSS intervention outputs—human resources, supervision, training, or infrastructure—could not be assessed to determine how the intervention influenced these aspects. The spillover effects of HSS interventions on other health system components were also not measured. Finally, given that nationally administered survey data were used to assess impact, the subnational context could not be accounted for when interpreting the results of the evaluation. Multicomponent interventions that operate at the familial, community, and health system levels did not measure many causal links between the intervention and immunization outcomes.

There is a need to use mixed methods and novel approaches to better understand how interventions cause change. In the context of HSS interventions, the FCDO review showed that for programs improving human resources and skills, the challenge is to understand which combination of strategies works and how, pushing research towards exploring other fields, such as organizational development, to answer these questions [132]. Process evaluations have gained prominence for evaluating complex programs with a specific focus on implementation aspects such as fidelity, adaptation, and dose, as well as mechanisms of impact [133]. Process and impact evaluation approaches are being integrated to better understand if an intervention works and how [23]. Future research needs to leverage multiple approaches and use both qualitative and quantitative methods for evaluation.

  1. 3.

    Evaluations need to be based on program theories

Program theories and frameworks are used to create a shared understanding of how a program is expected to have an impact in a certain context [134]. Our review showed that in most of the interventions, any theory or framework underpinning the intervention was not cited. The lack of a TOC and any relevant theory informing the intervention strategy adds to the gap in our understanding of how programs are expected to make an impact on routine immunization. It also prevents the synthesis of evidence on what worked and how in the context of existing theories.

  1. 4.

    Context needs to be accounted in the design and evaluation of programs

The review has also shown that context is mostly used to provide only a background or rationale for the intervention and to explain the challenges in the implementation or evaluation of the programs in a few studies. Overall, the descriptions of the context were cursory and did not adequately specify which features of the context were relevant or were most significant to the intervention or its delivery.

In the case of HSS interventions, there are stark differences in health system responsiveness and readiness across different contexts. Understanding why and how an intervention performs in a certain context as opposed to another requires constant adaptation and learning from evaluations. TOC helps identify and elaborate on the critical components that could be tested using implementation research techniques to adapt or drop across contexts. Furthermore, the availability of a TOC can help address the persistent challenge of balancing program fidelity and adaptation in newer contexts. This approach might enable some flexibility in intervention implementation and thereby its transferability to different contexts [134, 135]. In 2018, UNICEF, in partnership with GAVI, funded its implementation research for immunization programs in LMICs to understand what contextual and programmatic factors influence program implementation and why [136]. Context has also gained importance in learning systems to adapt programs according to the context and lessons learned [137].

There is a need to move away from looking at interventions as separate packages of components that are introduced in a certain context. In developing and justifying a theory of change to inform an intervention and its evaluation, researchers should show a clear understanding of how the context influences the program and vice-versa [13]. An intervention that is effective in some settings could be ineffective or even harmful elsewhere [13, 24, 138]. Thus, it will be beneficial to prioritize the analysis of heterogeneous and unintended effects in the evaluation depending on the usefulness of information for decision-making.

Conclusion

Our review highlights the strong need to strengthen theory-based evaluation approaches for the routine immunization of children in LMICs. Future implementers and evaluators need to develop clear TOCs that are based on established theory and have clearly articulated the underlying assumptions. Importantly, for the evidence to be usable for policy or practice, the lack of evidence on causal mechanisms can be a major roadblock. An increase in complex programs also require novel approaches and the use of multiple evaluation methods. Large-scale programs, specifically health system strengthening initiatives implemented by the governments, also need to focus on building TOCs and integrating them into their results frameworks. There is a need to combine both impact and process evaluations and to use implementation research techniques to understand how the intervention has affected the outputs and outcomes that can impact immunization uptake.