Background and Introduction

SciMetrika supported the evaluation of Essentials for Childhood by collecting, tracking, analyzing and reporting data on Essentials for Childhood (EfC) grantees’ efforts to implement the four primary goals of CDC’s Essentials for Childhood Framework (Centers for Disease Control and Prevention 2014) using the collective impact framework (Kania and Kramer 2011). The collective impact approach brings together organizations from different sectors to solve a social problem by aligning their efforts and using five key elements. The key elements are: (a) a common agenda for change, including a shared understanding of the problem, and joint approach to solving it through agreed-upon actions; (b) collecting data and measuring results consistently to ensure shared measurement for alignment and accountability; (c) a plan of action that outlines and coordinates mutually reinforcing activities for all participants; (d) open and continuous communication across the many players to build trust, ensure the mutual objectives, and create common motivation; and (e) a backbone organization with staff and specific sets of skills to serve the entire initiative and coordinate participating organizations and agencies (Kania and Kramer 2011). CDC anticipated that using a collective impact approach would help break down organizational silos and align shared goals, objectives and strategies, and thus contribute to the prevention of child maltreatment.

As we conceptualized our plan for evaluating this complex initiative, we recognized that CDC’s ambitious goal, primary prevention of child maltreatment, is based on the premise that large scale social change is complex and emergent (Patton 2017). Through regular conference calls and a shared website, we established a community of practice, providing evaluative information and feedback to states and their partners to inform adaptive development of their collective impact initiatives.

A single overarching evaluation question guided this initiative: what progress will be made by state grantees implementing each of the four goal areas of the Essentials Framework? We also asked several other important process evaluation questions: (a) How and how well is collective impact implemented? (b) What strategies emerged? (c) What circumstances or contextual conditions will lead to new strategies or adaptation, implementation or rejection of strategies proposed? (d) What strategies and partnerships enhance child maltreatment prevention awareness, commitment, programs, norms and policies? (e) How does the larger system or environment respond to the initiative or strategies? (f) What are the challenges, facilitators and opportunities encountered, successes observed, and lessons learned? In sum, these evaluation questions assessed the degree to which Essentials for Childhood grantees implemented the five conditions from the collective impact framework (Kania and Kramer 2011) and the four goals within the Essentials for Childhood guidance document (Centers for Disease Control and Prevention 2014). Meaningful results for this project were defined as the degree to which strategies were proposed and successfully adopted for each grantees’ EfC goals. Additional questions assessed context, barriers, facilitators, successes, lessons learned and promising practices for EfC content, and implementation and measurement approaches.

This article describes our methods, cross state synthesis of findings, limitations, lessons learned, and next steps based on the experience from the five state grantees (California, Colorado, Massachusetts, North Carolina and Washington) who participated in this five-year initiative.

Methods

Study Design

We used a mixed methods design combining quantitative and qualitative data (Greene and Hall 2010).

Data Sources

We used multiple data sources for the evaluation: (a) a landscape assessment conducted by states at baseline to identify available data and existing efforts; (b) baseline state grant applications and continuation applications throughout the grant period (inclusive of states’ annual progress reports); (c) transcripts from bi-monthly evaluation calls with state grantees; (d) transcripts and notes from in-person meetings at CDC that took place twice during the first year and once thereafter for each of the remaining four grant years; (e) notes and documents shared by state grantees; (f) annual collective impact self-assessments conducted by each state grantee; (g) a policy tracking document updated weekly based on Lexis Nexis StateNet Capitol Journal (Lexis Nexis® State Net® Capitol Journal, n.d.); and (h) state profiles of their use of Community-Based Child Abuse Prevention dollars (CBCAP, FRIENDS, National Resource Center for Community-Based Child Abuse Prevention, 2013, 2014, 2015, 2017). The last two data sources provided information that showed states are moving to evidence-informed practice.

Quantitative Analysis

We conducted quantitative analysis of three data sources listed above (i.e., the policy tracking document, CBCAP dollars, and the annual collective impact assessments. For the policies and CBCAP dollars, we compared the five funded states to seven self-supported (identified as “success stories” in year 2), and six non-engaged states–states that had never participated in any Essentials webinars or meetings and were not implementing similar efforts such as Mobilizing Action for Resilient Communities (Health Federation of Philadelphia 2018) or Building Community Resilience (Milken Institute School of Public Health 2018)–to assess if states had policies that have shown evidence of preventing child maltreatment or its risk factors, as summarized in CDC’s Technical Package (Fortson et al. 2016). We used data collected between 2016 and 2018 for this analysis as this time coincided with the 2016 launch of The Raising of America documentary film and early implementation of state grantees’ EfC strategies to raise awareness and commitment (goal 1) and create the context for healthy children and families through policy (goal 4). Additionally, we tracked CBCAP dollars used for programs that were considered “supported” or “well supported” by evidence according to the National Resource Center for Community-Based Child Abuse Prevention (CBCAP, FRIENDS, National Resource Center for Community-Based Child Abuse Prevention, 2013, 2014, 2015, 2017). CDC required funded state health departments to partner with their CBCAP leads as part of their EfC work. Therefore, we tracked pre- (before 2014) and post- (after 2014) intervention CBCAP data.

Finally, we used the collective impact assessments conducted annually by state grantees to provide a cross-state snapshot of trends in both collective impact and Essentials for Childhood goals between 2014 and 2018 (Preskill and Russ-Eft 2016). To assess collective impact and progress achieving Essentials for Childhood goals, state grantees assessed yearly progress on 9 indicators and 34 related sub-indicators on a scale of 0 to 10, where 0 equaled non-existent or weak progress and 10 equaled strong progress. Assessment occurred around the end of each grant year. Each state grantee used the same method to assess progress yearly to increase reliability (see Appendix A Collective Impact Assessment Tool).

Qualitative Analysis

We conducted qualitative analysis using the data sources described above on a semi-annual basis during all five years of the grant. Our objectives were to assess each state’s progress toward the implementation of the Essentials for Childhood guidance document using the collective impact framework, facilitate data sharing of promising practices among states, and identify training and technical assistance needs.

To analyze the data, the SciMetrika performed a thematic analysis to generate themes from bi-monthly state grantee evaluation calls. Deductive codes were developed based on evaluation questions and concepts derived from the collective impact framework and the four goals of Essentials. The codes were organized into a codebook and applied to all transcripts. Then, a systematic analysis of coded reports generated by ATLAS.ti was conducted, reading for emergent themes common across data sources. The resulting semi-annual data reports described cross-state grantee progress in their program implementation of the four goals of Essentials using collective impact. We synthesized the data from these semi-annual data reports and transcripts to create this paper.

SciMetrika synthesized process evaluation data, short-term outcome data (1–2 years), and intermediate outcome data (3–5 years) to include in data reports produced semi-annually. Cross-state implementation findings were generated from cross-site qualitative analysis of common themes and trends associated with implementing the collective impact approach. Themes included common implementation strategies across sites by EfC goal, short-term achievements, barriers and facilitators to implementation, and lessons learned. Cross-state achievements were generated from cross-site qualitative analysis of short- and medium-term outcomes reported by each state during monthly conference calls and through cross-site analysis of data from annual reports. We generated findings by EfC goal and descriptions of the degree to which states achieved goals over the course of the project highlighting aspects of the collective impact process that were particularly significant or useful for each state. Tools and promising practices were identified and captured during bi-monthly evaluation calls, training and technical assistance webinars, and in-person meetings with CDC attended by funded states.

Findings

This cross-state synthesis of findings presents selected quantitative findings and common themes of five state grantees using the collective impact framework to achieve the four goals of the Essentials for Childhood Framework.

Comparison of Policies Supportive of Families in Funded, Self-Supported, and Not-Engaged States

Table 1 depicts policies enacted during the 2016–2018 data collection period in (a) funded EfC grantees (n = 5), (b) self-supported states (n = 7), and (c) not-engaged states (n = 6). These policies are from CDC’s child abuse and neglect technical package (Fortson et al. 2016). In funded states, there were 26 policies supportive of families passed from 2016 to 2018, for an average of 5.2 policies per state. In self-supported states implementing the Essentials for Childhood Framework, there were 5 policies passed, for an average of 0.7 policies per state. In states that were not engaged in implementing Essentials or other similar efforts, there was one policy passed for an average of 0.2 policies per state. Thus, EfC states implemented over 7 and 26 times more policies informed by evidence-based prevention practices than the self-supported states and not-engaged states, respectively.

Table 1 Policies in funded, self-supported, and not engaged states, 2016-August of 2018 (‘16-‘18)
Full size table

Changes in CBCAP Dollars

Table 2 shows the percent increase in CBCAP dollars from 2011 to 12 to 2015–16 among CDC-funded EfC grantees, self-supported, and not-engaged states. The percent of dollars dedicated to “supported” or “well supported” evidence-based programs almost doubled in CDC-funded states while it remained low in self-supported states and fluctuated in not-engaged states.

Table 2 Average percent of CBCAP Dollars invested in “supported” or “well supported” evidence-based programs among CDC-Funded, self-supported, and not-engaged states, 2011–2016
Full size table

Cross-State Trends in Collective Impact and the Four Goals of Essentials

Table 3 summarizes cross-year (2014–2018) trends for indicators of collective impact and the four goals of the Essentials for Childhood guidance document for the five state grantees. State grantees used findings from their yearly assessments to reflect on progress and achievements over the prior year and to help focus plans for the following year. While most scores increased steadily, some in the area of engagement decreased in years two and three before increasing in later years.

Table 3 Cross-year trends on selected collective impact indicators (2014–2018)
Full size table

Cross-State Themes by Essentials for Childhood Goal

Goal 1: Raising Awareness and Commitment to Promote SSNREs and Prevention Child Maltreatment

All states reported making the most progress for activities focused on Goal 1: Raising awareness and commitment to promote safe, stable, nurturing relationships and environments. Key to raising awareness has been the screening of The Raising of America documentary film and post-screening educational sessions. California’s Epidemic Intelligence Services (Cal EIS) Officer describes one experience as follows:

I presented at the Cal EIS and Preventive Medicine Resident Seminar at CDPH [California Department of Public Health]. We screened the Raising of America and then we discussed the social determinants of trauma and adverse childhood experiences, and the broader frame of how we view child and family wellbeing in the U.S. It was very well received; I got good evaluations. And the CAL EIS program has now taken on the Raising of America as something that they want to continue to screen in the future. So, we did provide them with DVDs and they said they plan to ask me back in the future to facilitate that conversation.

Additionally, newsletters sent to key partners and stakeholders describing events and progress related to Essentials for Childhood, as well as social media outreach inclusive of websites, Facebook, and Twitter accounts were viewed as helpful to successfully raise awareness about the initiative. Further, all states actively presented their EfC initiatives at state and local conferences. Leadership Action Team/Steering Committee (LAT/SC) members in some states have obtained strategic appointments on task forces, panels, or councils, which also helped promote work within the safe, stable, nurturing relationships and environments umbrella.

Goal 2: Using Data to Inform Actions

Four out of five states had active data work groups throughout most of the five-year funding period. The data work groups helped to identify shared measures and indicators, which facilitated the prioritization of the initiatives’ activities. All states used the Social Determinants of Health indicators provided by CDC as a reference point for identifying available data sources that represent a mix of short-, medium-, and long-term outcomes.

To align the activities of the work groups and the LAT/SC, members of the LAT/SC were often also work group members or co-chairs. Data work group members attended work group meetings to expedite the identification and decision-making process. Grantees reported additional data activities were borne by leveraging the work of the data groups. For example, California worked with Kidsdata.org to add Adverse Childhood Experiences (ACEs) data to the database which is searchable by topic, region, county, and demographic data.

Each state’s activities differed according to their landscape. For example, Colorado focused on developing a Child Fatality Prevention toolkit and implementation of prevention training across the states; Massachusetts developed a paid family leave infographic for partners to educate people about the research and impact of paid family leave on child maltreatment outcomes; and Washington leveraged one of its work groups as an incubator to implement systems change (Help Me Grow) throughout the state of Washington.

Goal 3: Creating the Context for Healthy Children and Families through Norms Change and Programs

All states prioritized norms change as an important part of their initiative, as demonstrated by its inclusion in every state’s common agenda. Many states have begun to turn their attention to ways to address norms change by identifying existing programs and initiatives with whom they can collaborate. Using the results of YouGov’s Awareness, Commitment, and Norms Survey, Colorado and Massachusetts have identified help seeking and social connectedness as social norms that need to improve. They developed work products aimed at positively influencing these norms in their states. Additionally, YouGov may conduct a second wave of data collection for Awareness, Commitment, and Norms to measure norms change over time.

Changes in CBCAP funding dedicated to evidence-based programs were also observed and described in the quantitative section. In addition to these changes in CBCAP funding, Colorado and Washington also invested more in evidence-based home visiting programs and North Carolina recently obtained a waiver from Medicaid to fund evidence-based home visiting in three pilot communities.

Goal 4: Creating the Context for Healthy Children and Families through Policies

All states educated stakeholders and decision makers about evidence-based prevention programs and practices to prevent child maltreatment. Each state incorporated the framing and messaging training and technical assistance provided by an expert consultant into their EfC materials. Several leveraged their role with the Child Fatality Task Forces to facilitate the discussion of supportive policies. This is important because Child Fatality Task Forces have the ability to make recommendations to help prevent child deaths. Educating these groups about what works to prevent child maltreatment makes them an important audience so they can help inform others and make informed recommendations. Due to their engagement with high-level decision-makers, two states, Washington and California, had language and goals from Essentials incorporated into their state health plans.

According to California, “our biggest accomplishment is that the Essentials Initiative and its focus on preventing and addressing trauma are not only on the agenda of multiple policy/decision making groups but also to some degree integrated and aligned with their ongoing priorities.”

The number of evidence-based policies enacted in the CDC-funded states was previously described.

Cross State Facilitators and Challenges for Implementation of the Initiative Using the Collective Impact Framework and EfC Goals

Facilitators

Through the data analysis, there were a variety of themes related to facilitators which emerged. Grantees all shared that their state had a strong history and commitment to addressing child maltreatment, but efforts were, for the most part, focused on changing individual parenting behaviors or treating maltreated children. With public health at the table, there was an important shift from reaction to cases to primary prevention and changing the context. A statement from one backbone partner echoed by all funded state sums it up well: “EfC has been a gift that has widened the lens to public health prevention.” Another grantee reported,

I think beyond our Essentials work,a lot of our work is shifting to this sense of prevention. And, I think a lot of our stakeholders are really excited about that. So, we see a lot of synergy that way.

Further, most grantees benefitted from private foundation funding to implement their initiative. This funding often provided additional staffing and training as well as meeting space, meeting facilitators, and refreshments. For example, Washington received in-kind funding from the Department of Early Learning, Department of Health and direct funding from the Bill and Melinda Gates Foundation and Empire Health Foundation, as well as foundations for additional staff and training to support their initiative.

States also frequently shared their appreciation for the training and technical assistance provided on the project through CDC’s grant, indicating the support was “excellent and beneficial.” Training and technical assistance was designed in consideration of grantees’ requests during bimonthly conference calls and the collective impact phase in which they were operating. The topics for the training and technical assistance ranged over the years from conducting process evaluation data to framing and messaging of the Essentials for Childhood information by the expert consultant.

All states’ initiatives were supported by the growing awareness across the country of ACEs and that a social determinants of health approach to child well-being increases awareness of safe, stable, nurturing relationships and environments as indicated, for example, by the increased inclusion of ACEs questions into the Behavioral Risk Factor Surveillance System (BRFSS) survey.

Finally, new and strengthened partnerships between the state health department and strategic partners increased reach and funding for all states’ initiatives. One state grantee put it this way,

Overall, there are many factors that make Essentials for Childhood a strong partnership. The robust process it has undergone to select specific activities and a unifying theme has both built member loyalty and forged relationships across sectors. While challenges remain and it will be difficult to move the needle on child maltreatment in the next year, the unique contribution this work has already made is that it has broken down silos, and caused leaders in child welfare to look beyond their particular slice of the pie, and consider other policies or angles of a problem that they may be tempted to believe they already fully understand.

Challenges

While there were many successes and facilitators, there were also challenges related to the initiative. Grantees frequently shared that participant fatigue, limited staff and monetary resources, as well as staff turnover were significant challenges to the forward momentum of their initiative. One grantee reported,

More than 35 percent of them are fatigued by the process so far. There’s a real need for people to see some action. A lot of people came to the table because they cared passionately about improving outcomes for kids and this has been a long planning process for them and they’re ready for action.

Further, many states reported identifying shared metrics was difficult, and the necessary datasets to incorporate social determinants of health were scarce.

Finally, while states maximized the use of the CDC funding, most states reported limited funding left gaps in their initiative. As described in the section on facilitators, to fill these gaps states sought out grants, in-kind resources, or aligned their activities with other initiatives.

CDC required grantees to engage the business sector, which proved to be challenging for most of the grantees except Colorado. Most states had difficulty engaging the business sector with a request that was doable and meaningful. However, once Colorado had established their Family-Friendly Workplace Toolkit, states began dialoging and sharing ideas based on their experience with this publication.

Lessons Learned

Acknowledge Early on that the Collective Impact Approach Is Dynamic and Iterative

Moving between the phases of collective impact (i.e., ideas to action to impact to sustainability) takes time and activities are ever-changing. There are many contributors involved and many moving parts, which require alignment and coordination. The ability to recognize that the process is dynamic and iterative can allow members to feel a sense of ease during a sometimes-chaotic process:

I think that one challenge with collective impact is just that it is iterative. And so, people think linearly often. And they feel like we have to get this completely perfect. And then we can do that part. And then we can do this piece. But that’s not how it really works. Everything kind of has to happen almost simultaneously. But you have to have a method where people can revise things as they go along. And so, we’re working through that.

Frame the Essentials for Childhood Framework as a Supportive Concept with Elements that Can Be Incorporated into Existing Work

This approach tends to help people identify how they can contribute, even in some small way, to promoting safe, stable, nurturing relationships and environments. Referring to Essentials as a concept or framework has helped remove competition and break down silos as well as allowed folks to identify a piece of the initiative they want to own. One grantee said,

Where we've really gained some ground is talking about it more like a concept, and that the strategies that we picked, the priorities that we picked, are a way to achieve Essentials for Childhood versus Essentials for Childhood as a project in and of itself.

Conduct an Ongoing Assessment of Other Potentially Related Initiatives and Constantly Seek Alignment Possibilities

It is important to conduct research and document organizations with similar missions or strategies and related programs. The energy spent conducting the landscape assessment to identify partners with whom to align was critical but time-consuming. Strategic alignment with the right organizations will increase effectiveness.

Identify and Recruit Partners and Work Group Members Based on Common Goals, Objectives, and Strategies

Choose strong, organized, and engaged leaders as co-chairs to help drive the activities of the workgroups. States were very strategic in their choice of workgroup co-chairs. Some chose leaders in their field who carry weight with their messages. Others deliberately chose high-level individuals who have a seat at the decision-making table regarding prevention of child maltreatment. Still others identified and recruited co-chairs using existing personal relationships. One noted, “We have been trying to be strategic. We’ve built on existing relationships. We have done an excellent job of identifying chairs that are engaged in the process.”

Clarify the Structure of the Initiative by Clearly Defining and Distinguishing the Role(s) of the Backbone Organization, LAT/SC, and Work Groups

States indicated there was often overlap between members of the co-backbone organizations, leadership action team, and workgroup co-chairs, with some members wearing multiple hats. This was often due to commitment of some members to other existing projects. Clarifying roles of each group and revisiting as needed will keep the work on track. Cross-training of existing members to fill positions that become empty will also help maintain momentum.

Establish a Communication and Decision-Making Process Early on

Many states struggled to find ways to communicate effectively with partners. They used a range of methods (e.g., quarterly newsletters, weekly emails, regular face-to-face meetings), but often the issue was not only how to communicate, but with whom, how often, and why. Establishing effective communication and decision-making protocols early-on will help keep partners engaged and avoid unnecessary delays.

Discussion

Our findings indicate that progress was made by state grantees in each of the four goal areas of the Essentials Framework. Albeit challenging, collective impact was well implemented and was successful in providing states with a method to engage and work with state-level partners to promote primary prevention of child maltreatment. With CDC funding, states established strong backbone organizations, built multi-sector partnerships, and developed a common agenda with shared goals, strategies, and activities aimed at accomplishing the four goals of the Essentials for Childhood Framework. Several cross-cutting accomplishments deserve highlighting.

All states recognized the value of using a public health lens to address child maltreatment with its focus on prevention and changing the context to achieve population-level change. Using a social determinants of health framework to focus upstream on primary prevention led to new and innovative partnerships and helped create a shift in partners’ thinking, planning, and alignment of resources. True to the nature of complex social change, each state’s activities differed according to their landscape.

Using language from the Essentials for Childhood Framework to find common ground among partners led to new and increased collaboration across multiple sectors in all states. Language from the Framework was widely disseminated and included in many state health plans and organizational strategic plans.

Making the connection between social determinants of health and policies that prevent child maltreatment upstream led to educating partners and broader stakeholders about the impact of childhood adversity on children and families. Grantees educated stakeholders about the research showing associations between policies and programs, such as paid family leave, Earned Income Tax Credit, family friendly work policies, early child care, housing, and increases in safe, stable, nurturing relationships and environments. Similarly, Essentials for Childhood may have contributed in some way to assuring that CBCAP dollars are invested in evidence-based programs between 2014 and 2016.

Strengths and Limitations

There were several strengths associated with this evaluation. We recognized that each state was a “case” implementing a complex intervention. Using the case study method to understand each state’s adaptation of the collective impact approach to implement the four goals of the EfC initiative permitted us to assess how change emerges. Rather than focusing on establishing causality, we sought to document efforts we observed that contributed to the goals, objectives, strategies, and activities established by each state. Through systematic comparison and exploration of common themes within and across states, we began to understand what works where (in what context), when (in what temporal context), and in what order (Byrne 2013).

There were also several limitations associated with this evaluation. The collective impact framework is based on the premise that outcomes are dependent upon many interactions and feedback loops so that prediction within the planning process is fraught with difficulties and unintended consequences (Rydin et al. 2012). As states began implementing the five elements of collective impact and recognized that the process was dynamic and iterative, not linear, they spent considerable time redefining or reprioritizing their objectives to account for this dynamic process. Further, the evaluation focus of EfC was on how EfC facilitates change rather than on how it causes change and looked to assess contribution rather than attribution. Thus, although we gathered data on the actions of state grantees and their partners, it was not possible to imply a causal relationship between the two. Concomitant with this challenge was the difficulty states had in identifying indicators or metrics which measured broad-scale social change. Trying to understand the entire system and all its connections made it difficult for states to identify metrics which were both available and measurable. Finally, measuring complex social change in an emergent environment also meant taking into consideration the unique geographic, social, cultural, economic, and political context in which states were implementing Essentials. Some states were operating in environments or contexts that were open and supportive to prevention efforts, while others encountered more resistance.

Conclusion and Next Steps

The Essentials for Childhood Initiative is one of the first federally-funded initiatives to address child maltreatment prevention with a public health approach at the state level. It required state public health departments (focusing on primary prevention and population-level impacts) and child welfare agencies (that typically take a harm reduction approach) to collaborate. This contributed to stakeholders moving beyond solutions fixing parents to changing the context and preventing child maltreatment from happening in the first place. This is an emerging shift in the field of child maltreatment prevention with considerable positive momentum. Each state grantee made progress in their efforts to prevent child maltreatment in the first five years of CDC’s Essentials for Childhood grant. In August 2018, all five original grantees (California, Colorado, Massachusetts, North Carolina, Washington) were awarded funding for an additional five years, along with two new grantees (Kansas and Utah, two self-supported states). They will use this funding to launch the next phase of their initiatives, which will focus on strengthening economic supports (e.g., promoting family-friendly work environments) for children and families and changing social norms that promote safe, stable, nurturing relationships and environments - strategies that research shows are effective at preventing child maltreatment. Sustained by a strong foundation and new resources, they will continue their work with partners to identify and prioritize strategies that will contribute to the primary prevention of child maltreatment.