Introduction and Background

As in other states, child maltreatment is an important public health problem. According to the Kids Count Data Center (2017), Colorado ranked 22nd among states in terms of child wellbeing in 2017. In 2016, 11,226 children were victims of abuse or neglect in Colorado with substantiated reports, a rate of 8.9 per 1000 children (U.S. Department of Health and Human Services 2018). Of these children, 80.5% were neglected, 11.5% were physically abused, and 9.5% were sexually abused (U.S. Department of Health and Human Services 2018). Black and Latino children are disproportionately represented in the child welfare system in Colorado at rates higher than the national average, with substantiated case rates of 18.5 per 1000 African American children (13.9 nationally) and 10.6 per 1000 Hispanic children (8.0 nationally), while White children are confirmed as victims at a rate of 7.3 per 1000 in Colorado (8.1 nationally) (U.S. Department of Health and Human Services 2018). These disparities likely reflect not only service biases in the child welfare system but also underlying structural factors such as poverty and challenges in accessing supportive services (Allan and Howard 2013).

The Essentials for Childhood project in Colorado was undertaken within this context, with a focus on fostering a primary prevention approach that engaged multiple partners. In addition to Essentials for Childhood funding, the Colorado state health department is one of 23 recipients of funds for the Core State Violence and Injury Prevention Program (Core SVIPP). This funding from the Centers for Disease Control and Prevention (CDC) is designed to help state health departments implement, evaluate, and disseminate strategies to address injury and violence issues, including child abuse and neglect. Because the Core SVIPP funding was housed in the same branch as the Essentials for Childhood project, there was an opportunity to align the goals and activities.

This case study examines the development of the Colorado project over a five-year period and the lessons learned about implementation. We couch our analyses within the context of the literature on collective impact for community change. The collective impact approach prioritizes five conditions for collective success: a common agenda, shared measurement systems, mutually reinforcing activities, continuous communication, and backbone support organizations (Kania and Kramer 2011). This evaluation summarizes the results of the case study in the context of these conditions for collective impact success with the intent of informing other similar collective impact efforts.

Social and Political Setting

The social and political contexts in which the Essentials for Childhood project was executed were critical in shaping the project’s directions. Although Colorado has a growing population and thriving economy, it faces particular social challenges due to being a large Western state with substantial rural and frontier areas, as well as a diverse urban population mostly located along the Front Range (the most populous area just east of the Rocky Mountains).

Financial insecurity is a common cause of parental stress, a risk factor for child maltreatment. Although Colorado does have a strong economy by many metrics, growth is uneven. Wages in Colorado have plateaued since 2000, while housing costs have risen sharply. In 2015, 31% of the state’s children lived in cost-burdened households (i.e., spending more than 30% of income on housing) (Early Childhood Colorado Partnership 2018). Low income is another risk factor for child maltreatment. A 2017 national study found that a $1 increase in minimum wage is associated with nearly a 10% decline in reports of child neglect (Raissian and Bullinger 2017). When the project started in 2013, the minimum wage in the state was $7.78, and has increased to $9.30 in 2017. In 2016, Colorado passed legislation to annually increase the state minimum wage until it reaches $12 per hour; this will not take effect until January 2020 (Colorado Department of Labor and Employment 2018).

Colorado is a local-control state, meaning many policy decisions related to education, health, and other sectors are made at the local level rather than through a centralized, state-administered system. While this can present challenges for passing sweeping policy to enhance public health across the state, it also means Colorado jurisdictions have the opportunity to enact policies locally. For example, local-control enabled Boulder County to adopt paid leave for county employees, despite no state mandate.

Other risk factors for child maltreatment include unintended pregnancy, lack of parenting skills, and domestic violence (Child Welfare Information Gateway 2004). Colorado experienced a significant decrease in unintended teen pregnancy between 2009 and 2014 as a result of expanded access to long-acting reversible contraceptives (LARCs) statewide (Colorado Department of Public Health and Environment 2017).

Poor parental mental health also increases the risk of child maltreatment. The percentage of Colorado parents reporting poor mental health days in the past month increased slightly from 38% to 41% from 2012 to 2016 (Early Childhood Colorado Partnership 2018). While these data suggest an increase in mental health concerns among parents, this trend may also reflect a greater degree of mental health awareness and a general reduction in stigma related to mental health and parenting (Early Childhood Colorado Partnership 2018). In the wake of a shooting at an Aurora movie theater in 2012, Governor Hickenlooper prioritized improving mental health in Colorado and charged the state with developing a comprehensive plan to expand mental health services, which includes a state wide crisis hotline and improved access to mental health and substance abuse care, including the expansion of crisis centers throughout the state. A new child welfare plan for the state was also announced in 2012, including a statewide hotline reporting system, a public awareness campaign, and more standardization of the system for responding to reports (Colorado Department of Human Services 2018).

Difficulty accessing affordable child care is an additional stressor for parents. The most recent data available show that the number of currently licensed child care slots in Colorado only have the capacity to support 64% of 0 to 5-year-olds in the state (Early Childhood Colorado Partnership 2018). In 2016, Colorado ranked second in the country for the least affordable child care, with the average cost comprising 16.7% for a two-parent family income (Child Care Aware® of America 2017). The relative shortage of care and the high cost are burdensome for families; in 2016 11% of Colorado families reported quitting a job, not taking a job, or greatly changing a job because of problems with child care (Early Childhood Colorado Partnership 2018). These data suggest that more than 1 in 10 Colorado families experience child care challenges influencing employment.

Early childhood programs, such as high-quality preschool, enhance protective factors associated with improved developmental outcomes for children. High-quality early childhood programs have also been associated with lower rates of child maltreatment. From 2012 to 2017, the estimated percentage of 4-year-olds eligible for the Colorado Preschool Program or Head Start who were actually enrolled in either program increased from 55% to 62% (Early Childhood Colorado Partnership 2018). During the same time period, the percentage of Colorado children accessing full-day Kindergarten increased from 70% to 77% (Early Childhood Colorado Partnership 2018). From 2012 to 2017, there was also a 60% increase in total state dollars funding the Colorado Preschool Program, from $67 M to nearly $108 M (Early Childhood Colorado Partnership 2018).

The state has unique fiscal challenges. The Taxpayer Bill of Rights (TABOR) is an amendment to the state constitution that was passed in 1992. TABOR restricts the growth of state revenue and requires surplus revenue to be refunded to voters, effectively limiting any expansion of government programs. Due to TABOR restrictions, Colorado school funding remains below the national average during a time of state economic growth, and overall investment in public programs from the state general fund (3.7%) is at one of the lowest points in state history (The Bell Policy Center 2018). With low taxes and TABOR’s strain on public financing, advocates for early childhood programs in the state struggle to secure additional support.

Early Childhood Landscape

In 2012, one year before the Essentials for Childhood project began, the Governor’s Office announced consolidation of the state’s multiple early childhood programs into one new office: the Office of Early Childhood. This change was intended to enhance the sharing of information and generally improve the quality of Colorado’s early childhood initiatives. Prior to the creation of the Office of Early Childhood, seven early childhood programs existed in four divisions of the Colorado Department of Human Services, and additional programs were managed by the state health department (“Colorado Office of Early Childhood,” 2017).

A number of other grants and initiatives addressing early childhood and social-emotional health were underway in Colorado when the Colorado Department of Public Health and Environment (CDPHE) received funding for Essentials for Childhood in 2013. These include Project LAUNCH through the Substance Abuse and Mental Health Services Administration (SAMHSA), Launch Together under Early Milestones Colorado, Communities That Care (a program to promote youth wellbeing and improve behavioral health outcomes), and funding for the State Innovation Model (SIM) to enhance access to behavioral health care including a small pediatric focus. The Mile High United Way also coordinated (and continues to coordinate) with the City of Denver to facilitate the Family, Friend and Neighbor (FFN) Care Learning Community, which strives to expand access to informal child care.

An important partner for Colorado’s Essentials for Childhood (COEfC) work is the Early Childhood Colorado Partnership (ECCP), which formed in 2012. This partnership is a collective impact effort with similar grant deliverables as the COEfC project, and aims to bring diverse partners together to comprehensively address the early childhood system. In 2013, the ECCP formed a Data Action Team so state partners in early childhood could better organize for data-informed decision-making. The ECCP also developed Colorado’s Early Childhood Shared Message Bank in 2015, serving as a resource for early childhood partners across the state. The databank provides messages to enable stakeholders to share a single, collective voice to effectively engage audiences in strong early childhood development.

The Colorado Children’s Trust Fund serves as another key entity in Colorado’s early childhood landscape. Established in 1989, the Trust Fund works to enhance community capacity to prevent child maltreatment in the state. The Colorado Department of Human Services’ Child Maltreatment Prevention Director serves as the COEfC co-backbone lead and manages the Trust Fund’s Board, facilitating connections between COEfC and the Trust Fund.

Organizational Structure of the Project

In 2013, Colorado received a five-year grant from the CDC of $174,600 annually for the first three years and $200,000 for each of the final two years to develop the COEfC Initiative. The initiative aimed at promoting safe, stable, nurturing relationships and environments to enhance healthy child development and prevent child maltreatment. CDPHE housed the project in the Violence and Injury Prevention – Mental Health Promotion branch, serving as the primary backbone organization. Staffing included one Principal Investigator (PI), one Project Coordinator, and temporary research assistants hired at various points in the grant cycle. The original PI for this project was the Director of the branch. As a result of staff transitions and reorganization, the PI overseeing the final two years of the project was the Supervisor of the Interpersonal Violence Prevention Unit.

The CDC required that the project employ a collective impact design. The funding announcement called for the creation and convening of a Collective Impact Team and for the project to be facilitated by the state health department, along with another co-backbone organization. In Colorado, the co-backbone organization was the Office of Early Childhood in the Colorado Department of Human Services (CDHS). This newly created office in CDHS was selected as the project’s co-backbone because its mission focuses on child maltreatment prevention and aligns with the Essentials for Childhood project. The backbone organizations were required to organize a Leadership Action Team (LAT), with specified types of participants. The LAT began by developing a shared vision and later served primarily as an advisory body to guide project activities and priorities. To evaluate the Essentials for Childhood project, the CDC prescribed a developmental evaluation approach. In Colorado, project evaluation was contracted to a team from a local university.

Project Development

Figure 1 depicts a timeline of key events that occurred throughout the Colorado Essentials for Childhood project. Based on direction from the CDC, the Colorado project staff initially focused on developing its LAT and the collective impact process by convening a Collective Impact Team consisting of a broad set of stakeholders to inform priorities. While CDPHE’s original funding application listed six partners (e.g., Director of Prevent Child Abuse Colorado, Director of the Children’s Health Advocacy Institute at Children’s Hospital Colorado, etc.) as key members of the LAT, additional members were recruited throughout the grant to align with the project’s common agenda. Five meetings between February 2014 and August 2015 focused on engaging varied organizations and community members in developing priorities and strategies for addressing change. At a two-day meeting (August 2014) in the Denver metro area, approximately 50–60 people engaged in a planning discussion. From this, the LAT selected four primary project priorities: a) access to preschool and full-day Kindergarten; b) access to quality child care and after-school programs; c) family-friendly employment practices; and d) social and emotional wellness. During the third year of the project, the LAT transitioned from convening a Collective Impact Team to forming work groups for each of the four project priorities.

Fig. 1
figure 1

Colorado Essentials for Childhood project timeline

Full size image

Of these initial four work groups, two were sustained throughout the five-year period while two were eventually absorbed (in year 4) into ongoing work led by other organizations. However, the COEfC project coordinator remained in touch with entities working in these areas (e.g., concurrent efforts to improve access to child care) to ensure that project staff and the LAT could remain informed of related work.

Once family-friendly employment practices were identified as a priority, project staff worked to strengthen ties to the business community. A business representative was originally on the LAT, but left in year 1 after which project staff forged a relationship with a Colorado coalition of business leaders called Executives Partnering to Invest in Children (EPIC). The project coordinator recruited a staff member from EPIC to the Leadership Action Team and as the chair of the Family-Friendly Employment work group, along with an administrator from Children’s Hospital Colorado. The family-friendly employment practices work group remained active throughout the project.

The chairs of the Family-Friendly Employment work group focused their efforts on (a) developing a toolkit to assist businesses in identifying ways to enhance family-friendly business practices and (b) holding business forums to raise awareness among business leaders about family-friendly employment strategies. Throughout the course of the project, a total of five business forums were held in four different Colorado communities. At these forums, attendees received a Family-Friendly Workplace Toolkit, developed by the project and available publicly online. The toolkit provides evidence-informed examples of policies and approaches that can enhance the family-friendly nature of a workplace (“Family-Friendly Workplace Toolkit,” n.d.). The work group also partnered with Health Links, a worker health program at the Colorado School of Public Health, to develop an online self-assessment for business executives and human resource professionals designed to measure the degree to which a workplace is family-friendly. The system generates an individualized impact card for users, scoring results for four unique sections and providing users with suggestions for additional resources (“Family-Friendly Workplace Assessment,” n.d.). To enhance the sustainability of this work, a ‘train the trainer’ model has been developed and will be managed by Health Links. This will provide businesses and human resource professionals with technical assistance in utilizing the Toolkit.

The other project priority that started slowly but that was sustained through the lifespan of the project focused on social and emotional wellness. The work group addressing this priority evolved throughout the project and was ultimately renamed “Community Norms.” At the time of this work group’s development, a non-profit called the Colorado Alliance for Drug Endangered Children focused on the intersection of substance abuse and child maltreatment. This non-profit eventually merged with the state’s chapter of Prevent Child Abuse America to form a statewide non-profit called Illuminate Colorado. The director of Illuminate Colorado joined the Leadership Action Team and served as chair to the work group.

The work group’s efforts have been informed by the CDC’s purchase of the Awareness, Commitment, and Norms Survey (ACNS) data, as well as Colorado-specific data collected by the state’s Office of Early Childhood. These data guided the work group to focus on altering social norms related to help-seeking behaviors of guardians and caregivers in Colorado. Seeking guidance in how to approach this goal, the Community Norms work group consulted Davey Strategies to assist with developing a logic model and an initial communications approach. The group plans to focus its messaging to promote informal caregiver supports shared between neighbors and local organizations (e.g., public libraries).

Study Design

To understand the project’s development and how its priorities evolved, along with the challenges it enountered and the success it achieved, we conducted a series of semi-structured interviews with 26 key stakeholders including members of the LAT, CDC representatives, and staff employed by the project during its entire history. Three of the authors were included among the interviewees based on their roles in the project.

Measures/Instrument

We developed an interview guide in collaboration with project staff from CDPHE. The instrument included 16 open-ended questions, beginning with interviewees’ involvement in the project. Most interview questions focused on critical elements, and the final question asked interviewees if there was anything the interview did not address that they felt would be important to capture. The overarching research questions guiding data collection were:

  1. 1.

    How did the Colorado Essentials for Childhood Initiative develop over the past 5 years?

  2. 2.

    What are defined as the major successes of the Essentials for Childhood Initiative?

  3. 3.

    What challenges did the project face in implementing and achieving the goals of the Essentials for Childhood Initiative?

  4. 4.

    What are the major lessons learned through the efforts of the Essentials for Childhood Initiative over the past five years that could help inform future efforts of a similar nature?

Process of Conducting Interviews

A member of the evaluation team conducted all key informant interviews in person or via video conference. At the beginning of each interview, she asked for permission to record and explained that interview transcripts would be de-identified to protect confidentiality before sharing with project staff at CDPHE.

Data Management and Analysis

A transcription service transcribed each recorded interview enabling the interviewer to use NVivo 12 to organize and code interview transcripts beginning with deductive coding from project document review and adding additional, emerging codes throughout analysis of the interview transcripts. Two other members of the evaluation team validated coding by reading transcripts as coded by the interviewer. The final codebook contains a total of 17 codes, including three sub-codes.

Ethical Considerations

Because this study constitutes a quality improvement evaluation and we were collecting information from professionals about their work and organizations, it did not require Institutional Review Board approval.

Results

All 26 individuals invited for interviews participated between February and May 2018. We begin by highlighting the reported accomplishments and successes of the project. Following brief discussion of these successes, we present additional results organized by themes that emerged from the data, consistent with principles of the collective impact literature.

Successes

The principal success discussed by nearly all interviewees is the project’s efforts to increase family-friendly employment practices. Interviewees noted that addressing this priority helped to fill a gap that had previously lacked attention and resources among early childhood partners in Colorado. The family-friendly work group also had the most concrete success throughout the project, as it produced and distributed the Family-Friendly Workplace Toolkit and held family-friendly employment forums. Interviewees from the state health department also noted increased adoption of family-friendly employment policies and practices at CDPHE. As a result of this progress, CDPHE has initiated discussions with other state agencies to infuse family-friendly employment values into their workplaces, hopefully facilitating future progress in this area.

Another key accomplishment is that CDPHE, the project’s lead backbone organization, has worked strategically to tie the Essentials for Childhood Framework to efforts outside of child maltreatment, emphasizing risk and protective factors shared by other public health issues. This has helped integrate the EfC Framework and project priorities into other upstream approaches to primary prevention of injury and violence and leverage additional funding for work on project priorities. For example, project staff worked closely with members of the state’s Child Fatality Prevention System to effectively engage local CFPS teams and learn about efforts occurring at the local level that support the EfC Framework. The EfC Framework itself has also influenced the health department’s strategic plan for the Violence and Injury Prevention – Mental Health Promotion branch. These strategic integrations were noted by health department staff as project successes in that they will enhance the sustainability of COEfC work beyond the current funding cycle and they helped stimulate other branch work to address community and societal levels of the socio-ecological model.

The COEfC project also resulted in new and strengthened partnerships between the state health department and early childhood partners, as well as other non-traditional stakeholders (e.g., EPIC and its position in the business sector). Interviewees identified long-term commitment to the family-friendly employment and community norms priorities as a success in itself, as pre-existing organizations have committed to sustaining the efforts initiated by project work groups.

Finally, several interviewees identified a ‘shifting frame’ as a mark of project success, indicating that the Colorado Essentials for Childhood project had helped to ‘change the conversation’ about child maltreatment. To this point, stakeholders shared the following statements:

  • “To me, the success lies in the shifting frame with how we’re talking about the work and the ability to take what we learned and incorporate it and embed it in a lot more sustainable funding sources than EfC is ever gonna be.”

  • “I think that it helped folks think about change at a policy and systems level.”

While noting these achievements, there was consensus among nearly all interviewees that the project would have been more successful if more resources were available, and if the project’s scope had been more narrowly defined.

Resources

When examining the challenges experienced by Colorado’s Essentials for Childhood project, a recurring theme discussed by interviewees was a recognition of the limited funding, staffing, and time to accomplish ambitious project goals. At least five interviewees commented on the challenge of having only one primary staff overseeing project activities, and a couple also noted that the project coordinator position was not a very senior-level role. In addition, the project encountered staff turnover – with two principal investigators and three project coordinators within the five-year period. Interviewees noted this turnover was especially challenging given the project’s collective impact and collaborative nature.

Interviewees identified the length of the project as a distinct challenge, with five years not being long enough to address goals such as the creation of norms change. Related to the project’s ability to effect substantial change in either of the chosen priority areas given the length of the grant, one interviewee shared:

“Having learned a lot more about that kind of public policy change and social norm change ...I don't think it's realistic that that kind of thing happens in five years, particularly when a state government entity is running the project.”

Vision

The CDC’s grant guidelines mirrored their Essentials for Childhood Framework and asked states to promote safe, stable, nurturing relationships and environments by addressing four large goals: (a) raise awareness and commitment to support safe, stable, nurturing relationships and environments and prevent child maltreatment; (b) use data to inform action; (c) create the context for healthy children and families through norms change and programs; and (d) create the context for healthy children and families through policies. In their funded proposal, CDPHE purposely outlined strategies to pursue these goals, but, in conformance with grant requirements using a collective impact approach, did not, at the outset, specify activities to achieve the goals.

Interviewees indicated that the COEfC project failed to define a clear purpose throughout the grant. Many shared that early stages of the project spent a great deal of time trying to clarify what added value Essentials for Childhood could bring to other early childhood initiatives. A number suggested that early stages of the project should have focused on stronger strategic planning informed by assessment of the pre-existing landscape.

Although the intent of this project was to convene actors from different sectors to align on a shared vision and common agenda, interviewees shared comments such as:

  • “I feel like there was a while where our ultimate goal was too fuzzy to be able to really direct our efforts.“

  • “It didn’t feel like there was clarity about what EfC ultimately was or was trying to do.”

  • “EfC [was] a little squishy and hard to define and hard to evaluate and hard to communicate.”

Concurrent and Related Efforts

Some respondents shared that a degree of the project’s ‘squishiness’ was a function of project personnel not initially being fully aware of what gaps existed in Colorado’s early childhood landscape. The ultimate narrowing of project priorities from four to two is evidence of this. While family-friendly employment and community norms now serve as the common agenda for stakeholders, interviewees noted that it would have been productive to narrow the project’s focus earlier in the funding cycle.

At the start of the project period in 2013, Colorado had a number of active initiatives addressing child maltreatment and promoting protective factors in early childhood. Two other federal grants had also recently been awarded to fund early childhood efforts: Colorado Project LAUNCH and the Colorado State Innovation Model (SIM). Each interfaced with the COEfC Initiative in different ways. Regarding these particular grants, one interviewee noted:

“We knew they were both related, but we didn’t really know what either one looked like, and so I think there was just this feeling of like, until they – until they figure out what those things are and can tell us, it’s hard for us to identify how we can either support that work and/or fill the gap they’re not gonna fill.”

Such concurrent efforts added a layer of complexity to strategic planning for the COEfC project, as some related initiatives were newly developing themselves. This crowded landscape of related projects coupled with the diffuse nature of COEfC project goals led project stakeholders to report difficulties in distinguishing what made EfC unique from other early childhood initiatives in the state. One interviewee expressed the following:

“For a while, EfC was only a seat at everybody else’s table. And so we were doing some moving of the issue at other people’s tables, but we weren't – we didn’t set our own table to allow us to inform it even on a greater level. We were doing some infiltrating of other’s work, but we weren't doing enough behind the scenes thinking and organizing of our own stuff to be able to take it out effectively to other places.”

Communication

Despite efforts made by the project team at several points to create a project communication plan, both internal and external communication challenges were frequently noted by interviewees. This finding is also consistent with feedback we collected throughout the project via annual stakeholder surveys, in which stakeholders consistently noted communication as a challenge for the project. It is likely that limited overall staff resources, particularly communications expertise, contributed to this challenge. However, project staff did assist in creating a shared monthly newsletter with partners; this newsletter was developed in year 3 of the project and is called “Our Voice.”

Some interviewees said they found it difficult to articulate, internally, what COEfC ultimately set out to do, even near the end of the project. They noted experiencing inconsistent communication throughout the grant. Several LAT members, for example, commented on receiving unclear communication from project staff, while project partners noted inconsistent communication related to general project updates (e.g., work group members not understanding why particular work groups had dissolved, or why strategic priorities had been narrowed). One interviewee remarked on the lack of clarity related to the narrowing of four priorities to two, stating: “it seemed like the rest kind of dropped by the wayside.” Another who was a member of one of the dissolved work groups shared the following sentiment: “honestly, in my perspective, it felt like it just stopped meeting […] I wasn’t really aware of why it stopped meeting.”

Several respondents also noted challenges in externally framing the project as being directed at child maltreatment prevention versus childhood wellness promotion:

“[…] there was, for a while, this hesitancy to really be child abuse prevention or really talk about child abuse prevention, even behind the scenes. I understand that the messaging has always been around safe, stable, nurturing relationships and environments, and I love that messaging. And if, at the end of the day, we’re doing child abuse prevention or child maltreatment prevention, I don’t think we should be afraid in our Leadership Action Team or in our work groups or in our planning to really connect what we’re doing to child maltreatment prevention.”

A few interviewees discussed the distinct need to tailor project messaging to be more relevant to different stakeholders (e.g., business professionals may require different messaging than traditional early childhood partners). Regarding engagement of Colorado’s business community, one interviewee remarked: “I would say engaging multidisciplinary stakeholders around the idea that we want to create safe, stable, nurturing relationships and environments for families to thrive as opposed to engaging stakeholders around child abuse prevention. That’s just off-putting.”

The distinct contrast in the two previous quotes demonstrates that those involved in COEfC lacked a shared approach to communicating the project’s aims. While stakeholders participated in the project with different priorities of their own, a communication strategy may have helped partners project a more unified voice and better understand when/how messaging should be nuanced to aid potential partners in understanding their stake in preventing child maltreatment.

Stakeholder Roles

The stakeholders we interviewed worked in various agencies and organizations doing work related to child health and wellbeing, but many found it challenging to see how they fit into the work of Essentials for Childhood. A number reported feeling unclear about the role they were expected to play in the project, and what purpose they served in the broader initiative. This confusion about roles and expectations was expressed by interviewees representing an array of stakeholders, including individuals from partnering organizations, work group members, work group chairs, and members of the LAT. One member of the leadership team shared:

“I think [roles and expectations] needed to be much clearer throughout the whole project, in terms of are you coming on this to advise, and what we want is basically your brainpower once a month? Are there going to be some requests of your time in between meetings? Are there going to be requests of your taking up components of this work and driving that? That is very different than what my initial understanding was of what I was being asked to do.”

Other interviewees identified lack of clarity regarding their roles as a source of frustration. However, all of the interviewed stakeholders did remain engaged in the project throughout the funding cycle despite feeling frustrated.

Several interviewees cited ambiguity of roles and expectations as a challenge which also hampered engagement of new stakeholders throughout the project. In the absence of clearly defined stakeholder roles, some interviewees suggested that the project struggled to effectively recruit additional partners. Though the project cast a wide net to invite stakeholders to the initial strategic planning events as part of the Collective Impact Team, stakeholder engagement eventually narrowed to focus on intentional outreach to partners related to the sustained priorities (i.e., family-friendly employment and community norms). Although this narrowing of stakeholders was strategic to some degree, one interviewee shared the following sentiment:

“I think about two and a half years in, we stopped thinking that we should convene large groups of people in a collective impact way because people were confused about what they were coming to the table for.”

Project Structure

Similar to the ambiguity felt by interviewees related to their specific roles in the project, some indicated there was also a general lack of clarity about the project’s overall structure. Many interviewees cited a lack of distinction between the roles of the backbone organizations, the LAT, work groups, and project staff. Interviewees also noted the need for an established process related to decision-making, accountability of work groups and project staff, and when or how to appropriately engage stakeholders extending beyond the LAT or work group members. One stakeholder shared the following statement:

“I think that there needs to be so much more clarity on who is doing what, who’s responsible for what, the time commitment that you’re asking of people, what the backbone organization is responsible for, because I know that part of collective impact is that […] you are supporting those that are there. And sometimes I think that ends up in this gray area that means you just kind of go round and round, and nobody takes it and runs with it, and so people are either like, ‘I’m either going to not continue to be on this, ‘cause we had the same conversation over and over,’ or, ‘I’m going to stop complaining about it and actually do it.’ And I think that that’s what happened.”

Backbone Capacity

Another consistent challenge cited by interviewees was the limited capacity of the project’s backbone organizations, as state agencies, to effectively address legislative policy. COEfC funding could not be used for advocacy or lobbying, because Congress prohibits the use of federal funds (e.g., CDC grants) for such activities. Although work on organizational policy was under the purview of the Family-Friendly Employment work group, interviewees struggled to identify the project’s strategies in addressing ‘big P’ policy work. This suggests that the project, as a whole, lacked a strategy to directly address legislative policy. Those who had served as project staff noted that technical assistance from the CDC largely focused on the collective impact nature of the project rather than practical guidance on how to navigate policy work as a state agency with limited autonomy in the political sphere. To this point, interviewees shared the below statements:

  • “I think some people might feel like we should’ve done more for policy but, understanding the kind of constraints, it was more challenging than anybody could’ve ever imagined […] The CDC as an agency doesn’t even allow you to advocate. And the Department of Public Health as an agency doesn’t allow you to advocate. So how do you have a goal around policy without having the ability to advocate?”

  • “I mean I think as a state entity, most people are not given training on affecting social policies and, if anything, it’s kind of like we have to be careful because we’re not allowed to lobby, and sometimes it’s a fine line when you’re doing education and awareness, but you’re not lobbying. […] [CDC] provided us with a list of public policies that were showing evidence toward preventing child maltreatment, which that too was helpful, but we didn’t have any TA to say like, ‘Here is how you actually get there.”

An additional challenge cited by interviewees related to backbone support available at the state health department is the limited authority of health department staff leading the project. Given the many complexities inherent in aspects of the project, including addressing legislative policy, a couple of interviewees noted that it would have been helpful to have increased project engagement from high-level leadership at the health department (e.g., the department’s Executive Director). These interviewees suggested that such high-level leadership from the backbone organizations may have strengthened the project’s ability to impact state policy by more effectively drawing attention from the Governor’s Office and other leaders.

Collective Impact

A related theme emerging from the key informant interviews concerned CDPHE’s facilitation as a collective impact backbone organization. When discussing the project’s structure and facilitation, numerous respondents specifically noted that the COEfC project lacked a feeling of inclusion. Some interviewees suggested the work felt like a ‘CDPHE-only project,’ while others expressed that the work simply failed to feel like a collective impact project. A few interviewees noted that the majority of project meetings were held at the state health department, with a substantial presence of state health department staff in attendance. One interviewee shared the following statement:

“I don’t know that it felt like collective impact. I know it’s collective impact. I hear that this is a collective impact project. I don’t know that it felt that way. It felt like it was a CDPHE project that we were all participating in and I don’t know if that’s just because all the meetings were held at CDPHE and half the people in the room were CDPHE, and CDPHE was facilitating all of the meetings.”

Beyond the physical location of meetings or the number of state health department staff participating in discussions related to the project, several interviewees noted a general sense that the project was facilitated in a manner incongruent with collective impact principles. For example, some stated that they felt the representatives of the lead backbone organization were not always receptive to suggestions or feedback from the leadership team. One interviewee expressed feeling as though “there was a lot of talking at us but not with us, and I don’t think that they did a very good job of collectively soliciting input from the people that were at the table.”

Geographic Scope

In addition to interviewees expressing a desire for the project to feel more inclusive amongst those who were engaged, many remarked on stakeholders who were missing from the project. Some shared that COEfC should have been more intentional in engaging rural communities and counties outside of Denver. One interviewee from the state health department stated:

“Everything we do is typically very Denver metro centric. This held true here. So that's always a way things could be done better, to include a broader voice from outside of the Denver metro area, or outside of the Front Range.”

Although COEfC was led by state agencies and was intended to be a state wide initiative, interviewees noted the project lacked accurate representation of the full state and Colorado stakeholders reaching beyond the Denver metro area.

“I don’t think we could say that we had a representative group of Coloradans or a lot of local agencies or input who are really outside the Denver Metro. I mean, it was pretty Denver Metro and state agency centric, which is partly just who shows up at meetings.”

Noting that some participants did engage via conference calls or webinars, one interviewee suggested that these media could have been used to diversify stakeholder engagement and better include those living and working outside Denver.

“We do webinars with local people here in Denver, but not in rural or anything. This wasn’t a statewide effort. […] I really would have liked to hear from the rural [voices] and what they’re experiencing out there in their communities, but we don’t have it.”

Evaluation

Because the funding agency required the project to employ a developmental evaluation approach, the evaluation team largely focused its efforts on capturing the evolving relationships between stakeholders/organizations and reporting project events or activities during the grant. This was consistent with the goal of developmental evaluation, which, according to Kania and Kramer (2013), “is to provide an on-going feedback loop for decision making by uncovering newly changing relationships and conditions that affect potential solutions and resources” (p. 4). Despite this approach being in line with the funding agency’s prescribed evaluation method, interviewees noted the difficulty of “building an evaluation plan for a project that didn’t have strategy yet.” One stated that, because the project lacked clear vision for the majority of the funding cycle, a great deal of the project’s evaluation “was really just about evaluating the process, [which] was challenging.” In addition, very limited resources were available for project evaluation.

The developmental evaluation approach also presented challenges for COEfC. The lead evaluator was assigned a role on the LAT because she has extensive public health experience in injury prevention. One interviewee noted a lack of clarity in understanding why an evaluator would serve on a project’s leadership team, suggesting this dual-role was inappropriate and confusing for other stakeholders.

Another challenge was measuring early childhood progress that was clearly attributed to COEfC work. One interviewee noted that the funding agency seemed to prioritize examination of the collective impact process rather than evaluating the strategies employed to prevent child maltreatment:

“It seems like the emphasis that CDC put on the project was around the collective impact piece more than evaluating what the actual work was or really helping states coming up with those actual strategies.”

Emphasis on the backbone organization’s facilitation of the project as a collective impact effort, combined with the project’s lack of clear strategy and goals in the first half of the funding cycle, made it difficult for some interviewees to articulate just how much the project accomplished. One interviewee shared the following statement:

“I think there were gains made, although they’re hard to measure and define and hard to know if they would’ve happened without the initiative. But there’s not a lot to point to to say ‘aha,’ that’s the product of this initiative.”

Lessons Learned

Our interviews sought to identify and examine lessons learned from Colorado’s Essentials for Childhood project. This effort represented a new approach to child maltreatment prevention both for CDC as well as CDPHE and, as with any novel undertaking, there were unanticipated challenges. Other organizations seeking to undertake similar efforts may benefit from considering the following recommendations in the planning process.

Key findings from stakeholder interviews reveal that the strategic planning process is critical. Clear project goals and objectives are essential for alignment on a common agenda. Defining stakeholder roles early and clearly might have helped people know what was expected of them and from the project and enabled them to be clearer about their contributions.

The following section of this paper suggests recommendations for approaching collective impact efforts and implementing the Essentials for Childhood Framework to prevent child maltreatment based on what we learned.

Leverage the Landscape and Infuse Work into Other Initiatives for Enhanced Sustainability

According to Edmondson and Hecht (2014) collective impact efforts should make use of existing assets by applying a new focus to them. This can help initiatives more efficiently allocate limited resources, prevent duplicative work already addressed by partners, and may help the project avoid infringing on other partners’ ‘turf’ in the work (Edmondson and Hecht 2014). This is closely related to building intentional relationships with partners already engaged in the work, as it can be helpful to infuse a project’s objectives and priorities into other organizational missions, strategic plans, and funding streams. Colorado’s EfC effort was successful in leveraging pre-existing organizations to adopt long-term commitments to its two sustained project priorities (i.e., family-friendly employment and community norms). The lead backbone organization was also successful in tying the Essentials for Childhood Framework to other shared risk and protective factors addressed in work at the state health department, integrating child maltreatment prevention into upstream approaches and primary prevention for other public health issues. Both of these examples are important to the project’s long-term sustainability beyond the grant’s lifecycle.

Engage in Effective Strategic Planning to Cast Clear Vision

Important elements of a shared common agenda are: an “identifiable overarching goal & vision for [the] initiative within [a] clearly defined, bounded/actionable problem space,” and “a common understanding of the problem among partners” (Lynn et al. 2018, p. 46). Lynn et al., argue that in order to clearly understand the problem and identify project goals, stakeholders should understand the context in which they will be working. A thorough landscape assessment can help identify potential partners, competing initiatives, and gaps in pre-existing or concurrent efforts. These strategic planning steps are critical to understanding the foundation on which an initiative is selecting priorities to pursue. Investing sufficient time to engage in strategic planning before selecting project priorities can help to clarify the project’s vision and enhance communication about its agenda.

“I feel like we may have overpromised in those early meetings around what this was gonna be like, and, ‘Oh, there’s all these opportunities.’ Then we realize, ‘Oh, everyone’s actually really confused. Do we really wanna have these giant meetings for people who don’t know what they’re there for and why would they come?’ So, I think spending more time planning on how to leverage the current landscape, I guess, is a sure way to say what I would do.”

While considerable effort was put into developing a collective impact structure with various partners, the Colorado Essentials for Childhood project might have benefitted from more time spent in strategic planning, especially considering other early childhood initiatives already working to address early education and access to child care. A more thorough landscape assessment during strategic planning may have identified such complementary activities earlier. A focus on data-informed decision-making could also enable clearer selection of a feasible set of priorities in the scope of available resources. Interviewees noted that time may have been saved during the project if early efforts had focused on developing a clearer vision and adopting a common agenda among project stakeholders. They also cautioned against committing to too many project priorities.

  • “maybe it needs to be a little bit of a narrower scope”

  • “at least initially, trying to do fewer things and do them really well and gain some momentum”

Build Intentional Relationships with Those Already Doing the Work

To truly understand the existing power dynamics among partners working to address a social issue, collective impact leaders must “understand the context within which they work and […] stay vigilant because context shifts frequently” (Ryan 2014, p. 10). The COEfC project forged partnerships with many other initiatives doing similar work, which is critical in a crowded field of organizations tackling related goals. Further capitalizing on these partnerships to develop mutually reinforcing activities, while identifying a unique identity for COEfC, might have helped to avoid duplicative efforts earlier in the grant cycle. This may have also helped project staff and leadership determine how COEfC’s work could be both distinct from and supportive of the other initiatives (Lynn et al. 2018).

Intentional relationship-building is key to ensuring related projects are reinforcing each other rather than reinventing the wheel or competing. Conducting a thorough landscape assessment to identify key players already engaged in the work is a necessary prerequisite for this. A broad landscape assessment was conducted early in the project period in Colorado before the priority areas were defined, but as the project focus narrowed, it is possible that a more focused landscape assessment could have helped identify key partners in those areas. Once relevant key players are identified, forging relationships with them can be helpful to ensure clear understanding of their work. Formalizing partnerships, for example with roles on work groups/committees or Memoranda of Understanding documents, can help to ensure that partnerships are not reliant on individual relationships and can survive staff turnover. Ideally, these relationships can be leveraged to strategize about shared measurement systems that are mutually beneficial to partners engaged in the work, and to create a mutually reinforcing plan of action. This action plan would help to ensure that each partner or related effort remains consistent with the overarching common agenda (i.e., preventing child maltreatment) and is informed by shared measurement systems, while also maintaining the autonomy to pursue its own mission effectively.

Develop and Maintain a Project Communication Plan

According to Lynn et al. (2018), elements of continuous communication include: a) internal communications that support effective functioning of the initative work, and b) external communications that inform and engage the public about the initiative, facilitate knowledge and understanding, increase buy-in to the initative, and provide opportunities for feedback and input.

COEfC experienced communication challenges both internally and externally. Because of communication difficulties, the project experienced challenges in meeting two conditions of collective success: continuous communication and a common agenda. Clearer and more consistent and effective communication at the outset could have included more discussion of stakeholder roles and expectations for the project and potentially alleviated some of the ambiguity experienced throughout the project. Continuous communication regarding the project’s goals and alignment on a common agenda also could have aided in clarifying what kind of partners might be missing from the project. These elements could have assisted project staff and leadership in better engaging new partners to ensure they were invited to join the project’s work with a clear objective.

Clearly Define (and Communicate) Stakeholder Roles for Effective Engagement

Without clear understanding of one’s role or what is expected, stakeholders are less likely to remain engaged. Thus, it helps for the initiative to be very clear about what it is asking of stakeholders when they are invited to the table; for example, defining roles, estimating time commitment and length of anticipated involvement, and describing benefits to partner organizations. To create a successful mutually reinforcing plan of action, each stakeholder and partner must understand how they fit into the collective action plan (Lynn et al. 2018). One mature element of mutually reinforcing activities is a collective action plan which specifies strategies and actions that different partners commit to implement. Such clarity related to defined roles and expectations in the collective action plan can also aid in stakeholders’ ability to hold partners accountable for implementing activities as planned (Lynn et al. 2018).

Explicitly Discuss Project Structure and Processes among Leadership

One element related to the collective success condition of continuous communication includes structures and processes in place to inform, engage, and seek feedback from internal partners (Lynn et al. 2018). Additionally, a key element of strong backbone support includes well-functioning, established leadership that is responsible for governance and decision-making. Stakeholders we interviewed noted that COEfC lacked sufficient structures and processes, and suggested that those which did exist were not communicated clearly. As one Leadership Action Team member suggested in the following statement, project staff and leadership should explicitly discuss the structures and processes by which the project will operate.

“I think we needed to have very clear job descriptions for what the backbone was to do, what the LAT was to do, how we were going to decide on things […] I think a little bit more clarity on how decisions are gonna be made, how funds are going to be spent, how you’re going to be structured […] I feel like we just lacked clarity on that.”

Acknowledge Backbone Constraints and Ensure Technical Assistance is Practical

The most critical precondition for collective impact success is leadership from an ‘influential champion’ who can command the necessary attention from “CEO-level cross-sector leaders” and keep their active engagement sustained (Hanleybrown et al. 2012, p. 3). Although COEfC did engage with leadership, at various points in time, from the Colorado Department of Human Services (co-backbone organization) and the state health department, the “face” of the project was the project coordinator position who was the champion for the work, implementing the day-to-day tasks for the project.

Because policy is a core goal of the EfC Framework, at least one of the selected backbone organizations should have had the capacity to move in the political sphere without the restrictions that limited the state agency backbones in COEfC. In addition, the backbone organizations could work to recruit partners who have more autonomy in working with legislative policy and make it clear that that is their role on behalf of the project. Five interviewees suggested that additional backbone autonomy in legislative policy might have aided COEfC:

  • “I would definitely have liked to see [the project], especially with the goals for policy change, housed somewhere outside of a government entity.”

  • “I would say government agencies are important folks to have at the table definitely, but maybe the initiative should be housed somewhere that has a little bit more power to leverage some of those policy decision-making pieces or at least to advocate for those things because that’s part of the challenge.”

Given the limited capacity of COEfC backbones to address legislative policy, future technical assistance to provide project staff with practical guidance on engaging in the political process could help in building this element. This technical assistance should consider the limitations and constraints experienced by state entities in their environmental context and strategies to engage other stakeholders who do have the ability to move a legislative agenda forward.

Assure that Collective Impact Feels Collective

According to Hanleybrown et al. (2012, p. 6), “backbone organizations must maintain a delicate balance between the strong leadership needed to keep all parties together and the invisible ‘behind the scenes’ role that lets the other stakeholders own the initiative’s success.” While efforts were made to engage community members in various aspects of the COEfC project, interviewees reported feeling a lack of inclusion as stakeholders not affiliated with the state health department. One interviewee indicated that stakeholders were in “sort of [a] fog of this is a CDPHE only project, which is what it’s felt like.” This kind of statement, echoed by others in our study, suggests that stakeholders lacked a sense of ownership in the project.

COEfC likely would have benefited from diversifying meeting locations to be hosted by different members of the leadership team or in other community spaces rather than consistently meeting at the health department. This diversification of meeting venues might have also helped with stakeholder engagement. Other project leadership teams striving to have balanced and diverse representation so external partners do not feel the project is overtly dominated by any one organization might find this useful. Equally important as diverse representation, however, is equitable and inclusive engagement of external partners. Without feeling as though their voices are heard and their participation is valued, a leadership team can easily fall short of realizing its full potential in driving a project toward success.

Align on Clear Project Objectives and Success Metrics for Evaluation

From the outset of a project, collective impact partners should identify and monitor “early performance indicators that focus on the quality of the initiative’s design and implementation” (Parkhurst and Preskill 2014, p.19). To effectively do this, project stakeholders need to agree on a common agenda. Once project goals and objectives are established, the project can then identify specific measures by which to evaluate its success. According to Lynn et al. (2018, pg. 46) a critical element of shared measurement in collective success is “agreed-upon common indicator(s) established to consistently track progress over time.” A project’s ability to define clear objectives and success metrics can enhance its ability to identify and communicate small victories and incremental progress toward achieving long-term goals. Such communication highlighting periodic benchmarks can enhance the quality of continuous communication with project stakeholders, keeping them abreast of progress in achieving strategic goals while helping to sustain momentum and engagement in the work.

Strengths and Limitations

This case study had a number of strengths. Interviews were conducted by a graduate student with little prior exposure to the project or its stakeholders, allowing for objective data collection. Of 28 key informants who were invited to contribute to the evaluation, 26 did agree to participate in an interview, including all instrumental project staff (past and present). Key informants were told that interview transcripts would be de-identified before being shared with the state health department, likely facilitating honesty and forthrightness among interviewees, minimizing potential social desirability bias.

This work is not without limitations, however. By conducting key informant interviews, the evaluation team relied on individual stakeholders’ memories and perspectives, making the evaluation susceptible to recall bias. The interview questions also focused more on eliciting critical comments from interviewees rather than focusing on elaboration of project successes. Although we examined historical project documents to triangulate some points, the case study evaluation relies heavily on qualitative interviews in the absence of objective, quantitative measures of project performance up to, but not including, the final 5–6 months of the project. It should also be noted that there are other stakeholders who engaged with the Essentials for Childhood project at various time points who were not interviewed for this evaluation, and it is possible those stakeholders would have introduced additional or differing perspectives. Lastly, the derived results reflect one project and may not be generalizable to collective impact efforts focused on other topics or in other locales.

Project Sustainability and Next Steps

In the final year of COEfC funding, CDPHE project staff, the Leadership Action Team, and work group members worked carefully to ensure family-friendly employment and community norms efforts would sustain beyond the life of the funding. Organizations currently engaged in the COEfC work groups for these priorities have committed to carrying the work forward. Additionally, backbone leadership of the project has effectively leveraged partnerships with the Colorado Children’s Trust Fund to provide advisory support and funding. In May 2018, the Trust Fund’s Board voted to formally serve in an advisory role for future COEfC work. The Trust Fund has also committed to matching up to $50,000 of future funding to support community norms work. Additionally, CDPHE’s Child Fatality Prevention System has committed to matching up to $50,000 of future funding to support strengthening economic supports such as family-friendly employment.

As of June 2018, the Community Norms work group was beginning to develop a toolkit for guiding community leaders and influencers in the creation and enhancement of connectable community spaces and supports. The Colorado Children’s Trust Fund and the Colorado Department of Human Services have helped actualize this work by investing time and resources. Illuminate Colorado plans to sustain this initiative beyond the Essentials for Childhood funding cycle.

Building upon the project’s family-friendly employment work, CDPHE has committed to fund a ‘train the trainer’ program that will be maintained by Health Links. This program will staff and equip trainers to provide businesses with technical assistance in applying the Family-Friendly Workplace Toolkit. The ‘train the trainer’ model aims to increase the utility of the toolkit and enhance long-term sustainability of positive family-friendly employment changes.

As of May 2018, CDPHE has partnered with Good Business Colorado, a business membership organization that focuses on family-friendly work policy advocacy, to align its policy agenda with best practices from the CDC child abuse and neglect technical package. CDPHE will educate Good Business Colorado members on the link between best practice policies and child abuse and neglect prevention and work to increase the coalition of members engaged in family-friendly work policies.

To better understand the current political landscape across Colorado counties, the state health department conducted a state wide policy scan from February to June of 2018. This policy scan highlights Colorado communities, representing urban and rural, that have effectively advocated for policy changes which support the creation of safe, stable, nurturing relationships and environments at the local level. This scan reflects the partnerships between COEfC and the Colorado Child Fatality Prevention System that have spent the past year and a half engaging local partners to learn about community-driven work to support safe, stable, nurturing relationships and environments. The scan will also be made publicly available so multi-sector partners can access it as a resource in their child maltreatment prevention work.

In addition to efforts directly tied to COEfC, the state health department is developing a new maternal and child health surveillance system called Health eMoms to measure the social and emotional wellness of parents in Colorado. This surveillance system will track maternal attitudes, beliefs, and behaviors of mothers shortly before, during, and three years following a child’s birth (Early Childhood Colorado Partnership 2018). Health eMoms will ask mothers about how they feel physically and emotionally after having a baby, their opinions on taking leave from work, experiences with health care providers during and after pregnancy, and their access to health care, health services, and other resources.

In August 2018, Executive Leadership from CDPHE and CDHS will host a funder’s meeting comprised of Colorado foundations to showcase the work of COEfC. The purpose of this meeting is to educate funders on the work accomplished to date and to seek future partnership opportunities.