Introduction

Through substantial investment and evaluation, evidence-based programs (EBPs) are available to improve population health and prevent adverse experiences (Council 2014). Funders and policymakers have recognized the potential of EBPs, and now often require that only EBPs be funded. For example, the Family First Prevention Services Act of 2018 (FFPSA) offers an unprecedented opportunity to improve family and child well-being (Stoltzfus 2018). The FFPSA only allows diversion of Title IV-E funds for child maltreatment prevention programs with sufficient evidence listed in a FFPSA clearinghouse of eligible EBPs. These include substance use and mental health treatment for parents and parenting programs (ABT Associates 2020).

The decision about which EBP to adopt, however, requires consideration of many dimensions beyond evidence strength and completeness. Decision-makers (DMs), such as social service and public health department directors, and community members they involve, care about EBP fit with local context (e.g., available resources, staff acceptability), (Palinkas et al. 2018) as well as broader cultural and political contexts (Buffett et al. 2007). However, EBPs may not have been tested in settings similar to the community in question, requiring DMs to make assumptions about whether EBPs will have desirable effects in their context.

Cognitive biases can impact how DMs consider characteristics of different EBPs, and these biases are exacerbated by the complexity of simultaneously comparing multiple alternatives (Thokala et al. 2016). DM’s assessment of EBP fit may be impacted by their preferences and values, such as their tolerance for change (risk aversion), overconfidence when connecting limited evidence to a “likely” outcome (illusion of validity), and certainty of the representativeness of their previous experiences (availability bias) (Tversky and Kahneman 1974). Cognitive biases lead to systematic errors in how evidence is interpreted and how risk is associated with a particular EBP (Tversky and Kahneman 1974). These errors lead to inconsistent preferences for which EBP to adopt (Tversky and Kahneman 1974). Considering the ubiquity of these biases, efforts to improve EBP implementation will be limited if decision-making processes fail to account for DM’s preferences, values, and expertise in addition to the evidence base (Buffett et al. 2007; Palinkas et al. 2018; Sheldrick et al. 2019).

Decision-support tools can mitigate challenges when considering multiple alternatives and the resulting cognitive biases (Thokala et al. 2016). Multi-criteria decision analysis (MCDA) tools are one promising approach. MCDA tools use quantitative or mixed methods to estimate and compare the value of various alternatives (such as various EBPs) by asking DMs to consider and rate the importance of decision criteria (e.g., cost, feasibility of implementation) (Belton and Pictet 1997; Mühlbacher and Kaczynski 2016; Thokala et al. 2016). The MCDA process limits the extent to which DMs employ heuristics, or cognitive biases, that artificially simplify the problem’s determinants and solutions by transparently identifying and structuring how the problem, criteria, and values of various DM’s are defined and incorporated in the decision process (Thokala et al. 2016). Furthermore, by making these factors explicit, MCDA is reported to have many benefits, including (1) support of DM learning about the problem, alternatives, and evaluation criteria, which improves their ability to process complex information (Marsh et al. 2014); (2) reduced decision uncertainty and improved decision-making quality (Thokala et al. 2016); and (3) improved incorporation of diverse stakeholders’ values, preferences, and expertise (Marsh et al. 2014). MCDA methods offer guidance for combining decision-makers’ responses. Thus, MCDA facilitates consensus building as opposed to prescribing the “best” solutions (Belton and Pictet 1997). In short, MCDA tools can minimize cognitive biases and lead to improved, transparent decision-making.

Despite the potential benefits, MCDA tools have been underutilized in healthcare. Although MCDA was introduced in 1990, the majority of healthcare applications are after 2012 (Mühlbacher and Kaczynski 2016). MCDA has supported four types of healthcare decisions: prioritizing interventions for insurance/financial coverage, selecting interventions or priorities for implementation, authorizing interventions for licensing, and allocating research funds (Mühlbacher and Kaczynski 2016; Thokala et al. 2016). To our knowledge, no MCDA tools have been developed for use across child welfare or multi-organization efforts such as FFPSA EBP implementations. Existing MCDA tools are highly specialized and context-specific (e.g., singular insurance companies or hospitals), leaving a crucial gap in the field (Thokala et al. 2016). To address this gap and support decision-making around EBP implementation choices such as those prompted by the FFPSA, the purpose of this study was to (1) develop and pilot test a MCDA tool using a collaborative approach to help DMs compare and select prevention EBPs for community implementation and (2) evaluate whether EBP preferences, or value rankings, differ when guided by the MCDA tool compared to no decision support.

Methods

First, we iteratively developed the basic components of a MCDA tool with child and family serving stakeholders in North Carolina (n = 8). These components included the criteria, scoring alternatives, and weights. Second, we completed a pilot study with a separate set of stakeholders responsible for adopting and implementing EBPs (n = 11) to test the impact of the initial MCDA tool on EBP value rankings. The pilot study was exempt by the UNC-Chapel Hill IRB.

Phase I: Development of Initial MCDA Tool

A component checklist of the MCDA development process is in Online Appendix Table 1. To develop the initial MCDA tool, we worked with eight stakeholders between July 2017 and February 2018 who implemented EBPs to improve child and parent well-being or to prevent maltreatment (Cruden 2019). Stakeholders were mid- to late-career child and family therapists or educators, child and family nonprofit agency administrators, and parents.

The stakeholder engagement process is described in detail elsewhere (Cruden 2019). Briefly, following best practice for MCDA processes, we first identified the problem to be solved. This ensures a shared vision of what the solution alternatives are intended to address and shapes criteria prioritization (Thokala et al. 2016). Our problem had two components: (1) identify EBPs responsive to FFPSA or similar policies/funding restrictions and (2) a definition of child maltreatment. We first identified each stakeholder’s definition of child maltreatment and prevention, then synthesized these definitions, resulting in a broad definition of child maltreatment included “anything that harms the well-being of a child.” Prevention included efforts to strengthen protective factors and reduce risk factors that might lead to harm. We delineated maltreatment to include physical, sexual, and emotional abuse and neglect. Stakeholders prioritized neglect because it is the most common type of maltreatment and has shared risk factors with other types of maltreatment (Council 2014).

We began identifying criteria while stakeholders prioritized which EBPs to compare with the tool. We first created a list of 7 universal and selective child neglect prevention EBPs along with short summaries of each intervention’s characteristics (Online Appendix Table 2). We then asked stakeholders to discretely rank the EBPs and complete a brief survey, which prompted them to identify the reasons (i.e., criteria) underlying their rankings. Survey responses informed an initial list of MCDA criteria. This transparent process that involved stakeholders in criteria identification is preferable but underutilized in MCDA (Thokala et al. 2016).

Next, we mapped the initial criteria onto the RE-AIM (reach, efficacy, acceptability, implementation, maintenance) implementation framework to explore the comprehensiveness of the proposed criteria across factors known to affect intervention selection and implementation (Online Appendix Fig. 2; Glasgow et al. 1999). For example, when stakeholders were drawn to an EBP because it provided support to families “prior to baby’s birth” and to a “specific vulnerable population,” we generalized these comments to become the Program Targets criteria within the Reach domain, because the criteria reflected whom the EBP would “reach” or include. The stakeholders identified criteria across all RE-AIM domains except for one: maintenance. Thus, we proposed one maintenance-related criterion (sustainability). The full list of potential criteria was then presented to the stakeholders during a virtual meeting. They refined criteria phrasing and proposed four additional criteria, resulting in a total of thirteen criteria (Online Appendix Table 3). The criteria were all positively framed so that endorsement would translate to a higher EBP score, or rank. Our criteria validation with stakeholders follows best practice guidelines for developing MCDA tools (Thokala et al. 2016).

To develop the scoring alternatives, which indicate how much DMs agree with each criterion for the EBP being scored, we proposed a Likert-type scoring alternative from 1 to 5 for each criterion (i.e., 1 “Strongly Disagree,” 5 “Strongly Agree”). This scoring range has been used for similar tools and deemed acceptable for differentiating between alternatives without being too granular (Buffett et al. 2007). Stakeholders agreed with the proposed range.

We developed weights for each criterion based on the frequency each was indicated in the EBP prioritization survey, with higher weights assigned to frequently cited criteria. We further differentiated and balanced weights by ensuring that higher weights were assigned to intervention characteristics known to affect implementation and patient outcomes (Glasgow et al. 1999). After ensuring all weights summed to one, we asked stakeholders to review and discuss the weights, which resulted in some minor changes.

Part II: Decision-Maker Pilot

We conducted a pilot study with North Carolina DMs (n = 11) to evaluate: (1) the acceptability of the MCDA tool’s criteria, scoring alternatives, and weights and (2) whether the tool resulted in an EBP different ranking compared with an initial ranking without the tool. Participating DMs were either responsible for directly implementing family-based interventions or for selecting interventions for implementation as part of their professional duties. They included local health department directors, executive directors of county partnerships for children, non-profit administrators, current and previous North Carolina state legislators, and North Carolina Department of Health and Human Services employees. DMs were mid to late career, primarily female (n = 7), and non-Hispanic White. We recruited via snowball sampling and direct contact. MCDA does not require a minimum number of stakeholders to participate; instead, the focus is on establishing the reasoning for criteria and weights and the consistency of DM’s preferences (Thokala et al. 2016). Reasoning was iteratively checked with our stakeholders and again with the pilot study DMs. Consistency was explored via sensitivity analyses in the pilot study, described below.

DMs completed a four-step process as part of this pilot study, which lasted 60–90 min (Online Appendix Fig. 2). First, they read about three child neglect prevention EBPs in the California Evidence-Based Clearinghouse for Child Welfare, (CEBC; Nurse Family Partnership, Incredible Years, and SafeCare), which were prioritized by stakeholders in the prior development phase. The FFPSA registry was not live at this time. Second, we provided DMs with the hypothetical situation that they had a $3 million block grant for implementing a child neglect prevention EBP in their community over the next 3 years and asked them to record their initial discrete ranking for the EBPs without the MCDA tool (e.g., 1, 2, 3). Third, we asked DMs to complete the MCDA tool in an Excel spreadsheet (Online Appendix Fig. 3a,b). To check robustness and reasoning of the criteria and weights, the spreadsheet provided space for DMs to add a criterion if desired. DMs could also alter the default weights. Finally, we conducted semi-structured interviews to assess the tool’s acceptability and identify potentially missing criteria.

MCDA Analyses

We used a weighted sum approach to derive the MCDA score for each EBP, multiplying the criteria score (1–5) by the proposed respective weight (0–1) at each timepoint. Thus, the lowest possible overall score for each intervention was 1 and the highest was 5. While criteria scores could vary by EBP, the degree to which each criterion contributed to the overall EBP score was consistent across EBPs due to the weights. In base-case analyses, the DMs’ proposed weights, should they differ from the suggested weights, were used to calculate EBP scores and ranks. In sensitivity analyses, we used the mean weight across all the proposed weight sets (n = 6) to calculate EBP scores and ranks (Belton and Pictet 1997).

Results

Six of the eleven DMs (55%) ranked the EBPs differently with the MCDA tool compared with their initial ranking. Four of these DMs ranked all three EBPs differently. Two of these DM’s MCDA responses resulted in a tied score between two EBPs. Nurse Family Partnership had the highest average initial ranking and the second highest average MCDA ranking (Online Appendix Table 4). SafeCare’s average score increased, while Incredible Year’s average score decreased. Appendix Fig. 4 depicts the degree to which each criterion contributed to the MCDA score.

Five DMs altered the proposed weights, resulting in 6 unique weight sets. The three highest weighted criteria, strong evidence base, cost-benefit, and program targets, were further upweighted when changed (Online Appendix Fig. 5). In sensitivity analyses using mean weights across the 6 unique weight sets, EBP scores and rankings were relatively consistent, with three DM’s rankings changing: this approach broke EBP ties (n = 2) and reversed the 2nd and 3rd choice for another DM. MCDA computed rankings still varied from unassisted initial rankings.

Discussion

We used a replicable, stakeholder-driven development process to develop and pilot test a MCDA tool for comparing evidence-based child neglect prevention programs. Results demonstrated that decision processes supported by the MCDA tool will potentially result in different preferred EBPs compared with an unassisted decision process. In semi-structured interviews, nine DMs explicitly noted that the tool would be helpful for facilitating transparent conversations and “forcing” funders and local leaders to think about EBP fit with local context.

Our pilot study tested the comprehensiveness and acceptability of criteria with both implementers and DMs, resulting in 16 criteria, which is within the range in most health care MCDA tools (Marsh et al. 2014). Based on DM feedback, we adapted the piloted MCDA tool to incorporate six additional criteria and removed three original criteria (described in Online Appendix Tables 5 and 3, respectively). Previous studies have not focused on developing criteria across implementation evaluation domains or emphasized stakeholder engagement during both MCDA development and refinement (Buffett et al. 2007; Marsh et al. 2014). In contrast, our process yielded criteria across the RE-AIM domains with consistent stakeholder involvement. The criteria in our MCDA tool provoke evidence-informed consideration of key factors known to affect successful EBP implementation in child welfare such as the EBP’s acceptability by providers and families, appropriateness of the EBP for local context, and potential organizational support for the EBP. (Aarons and Palinkas 2007).

We also improved the tool’s usability across six features, outlined in Online Appendix Panel 1 and demonstrated in Fig. 6a. Our tool is primed for integration into decision-making processes. The tool can be shared as a spreadsheet or integrated into EBP registries such as the Title IV-E Clearinghouse. A “results” tab automatically displays the score and rank for each evaluated EBP (Online Appendix Fig. 6b). EBPs can be added or changed by replicating the spreadsheet tabs and re-labeling which EBP is being evaluated. Learning about an EBP could be as straightforward as consulting an EBP registry, although more systematic evidence reviews could also be adopted—such reviews are independent of the MCDA tool’s design. The tool also allows DMs to provide a preferred weight, allowing for additional incorporation of DM preferences and ownership over the MCDA process. We refer the interested reader to Belton and Pictet (1997) for additional suggestions and principles to guide local decision-making and quantitative approaches to handle conflicting MCDA rankings between DMs.

This tool and pilot study have several limitations. The tool was developed with a relatively small (though diverse) group of stakeholders from North Carolina. While the criteria align with considerations that have been highlighted across states and organizational settings, there may be considerations unique to other state contexts. Additionally, our sample size limits our ability to make inferences about the causal strength of EBP ranking changes or differences in proposed criteria or weight changes by DM characteristics. We observed similar changes across stakeholders and sensitivity analyses, however, suggesting our sample was sufficient for initial development and testing. Third, we developed the tool with the targeted outcome of child neglect prevention. However, our methodology resulted in a generalizable tool that can be easily adapted to include EBPs for a variety of population health outcomes. None of the criteria or weights are specific to child neglect. Further, the criteria include assessments of EBP fit with generalizable characteristics of the organizational and broader community context (Aarons and Palinkas 2007).

Future research is needed to extend evaluation of our MCDA tool. For one, research should further explore the consistency of the proposed weights across heterogeneous community and organizational contexts. Our tool is responsive to individual DM’s weight changes or priorities, and EBP scores can be computed and compared regardless of weight changes. Second, the MCDA tool should be tested with EBPs targeting additional population health outcomes that can promote child and family well-being. Finally, future research should assess whether EBPs adopted with the MCDA tool have differential effectiveness and implementation outcomes compared with EBPs adopted without the MCDA tool. We envision two potential lines of inquiry to this end. First, research could evaluate whether EBPs selected with MCDA tools undergo fewer adaptations. While EBPs will ostensibly have better population health outcomes compared with programs with less evidence, implementation science has shown that certain adaptations can affect EBP fidelity and population health impact (Baumann et al. 2017). Adaptation often occurs during implementation due to lack of fit with local population characteristics, culture, resources, and implementation capacity (Baumann et al. 2017). Unfortunately, these factors are often explored after adoption (Aarons and Palinkas 2007). We hypothesize that the MCDA tool will support more careful consideration of various EBP and contextual characteristics prior to adoption, thus aiding decision-makers in identifying characteristics that may impact EBP fit earlier in the implementation process. Second, implementation and effectiveness outcomes need to guide future iterations of the tool’s criteria and weights. The “optimal” solution, or top-ranked EBP, may vary based on the included criteria. Thus, criteria and their weights should be updated as research clarifies what community, patient, and organizational characteristics most impact EBP fit. However, we stress that MCDA tools are not meant to be prescriptive, but instead serve as supporting tools that can add transparency and structure to engaging conversations (Marsh et al. 2014). The malleability of our tool also signifies an important MCDA principle: reflecting stakeholder values is often more important than intervention evidence (Mühlbacher and Kaczynski 2016).

Stakeholder-developed MCDA tools with flexible, accessible formats (i.e., Microsoft Excel spreadsheets) allow DMs to update, tailor, and utilize such tools for collaborative decision-making. MCDA tools could be particularly valuable for facilitating transparent, effective responses to evidence-based policies such as the FFPSA that stand to substantially improve the lives of children and their families. Failure to attend to the errors of cognitive bias and opaque decision processes could result in public dollars being allocated to interventions that never realize their potential impact on public health. Our results add to literature that warns against “trusting your gut” and support the need for tools to guide EBP adoption (Bonabeau 2003).