Introduction

Although evidence-based interventions (EBI) are standard in fields such as medicine and public health, with increasing utilization in many areas of social work, EBI implementation in a public child welfare setting is relatively new. In recent years, the policy-driven push to adopt EBIs across social service sectors, including child welfare, has been strong. Federally-funded demonstration projects over the last decade have released hundreds of millions of dollars to public and private agencies to implement and evaluate EBIs. State, local, and private foundation funding streams have followed suit as EBIs promise to improve child welfare outcomes. As a result, “evidence-based” has become a buzzword. As with most buzzwords, the term is not always used precisely. For instance, it is not exclusively attached to interventions with strong empirical research supporting their efficacy. In addition to programs lacking rigorous prior evidence, various issues related to implementing an intervention interfere with accurate use of the term EBI. While an innovative approach may have proved superior to the status quo in one study, a lack of resources, training, or qualified staff may diminish its effectiveness when attempted in another setting. Particularly with child welfare populations, various programs for children and families are designated as “evidence-based” with varying levels of empirical support. There is also a dearth of knowledge about how to effectively put them into practice, creating a situation where both the intervention selection and the implementation require sustained commitment on the part of the intervention site (Aarons and Palinkas 2007; Proctor et al. 2007).

Policy makers, administrators, and practitioners experience frustration and disappointment when a significant investment into an innovative evidence-based approach does not result in intended outcomes. Responding to this, social scientists have broadened the scope of inquiry surrounding intervention research in order to better understand the facilitators and barriers to replicability. Implementation science is the study of factors and strategies that influence how to effectively put a program into practice (Fixsen et al. 2005). The expanding literature in this field suggests that successful innovation and full realization of the benefits of EBIs requires systematic implementation (Aarons et al. 2011; Fixsen et al. 2009).

The theoretical and empirical scholarship on the topic of EBI implementation has produced multiple models and frameworks aimed at identifying the key supports and barriers to successful program delivery. However, researchers have paid limited attention to the compatibility between these frameworks and EBI implementation in the child welfare system. This setting is distinct from many other social service arenas due to family- and individual-level factors, including the involvement of the multiple caregivers; distinct and complex needs of parents and children (Aarons and Palinkas 2007); and system-level factors such as federal laws, varying and changing state statutes and regulations, workforce issues, and shifting funding streams. Literature documenting EBI implementation in this context is sorely needed.

Implementation research arose in order to improve dissemination of innovations in fields such as business, health care, and human services. Although scientists were developing and testing improved approaches to service delivery, the translation to actual practice was often fraught with difficulties and disappointments. Similarly, in the human services, results of an experimental trial in one context often failed to replicate in another. Early research into this problem focused on identifying factors key to successful EBI implementation across many different fields and disciplines (Fixsen et al. 2005; Greenhalgh et al. 2004). For example, Durlak and DuPre (2008) reviewed over 500 quantitative studies and identified 23 contextual factors that influence implementation, categorized as community factors, provider characteristics, innovation characteristics, organizational capacity, and factors related to the support system. Their review revealed that implementation factors heavily influenced program effectiveness.

Recent advances in implementation science have focused less on specific factors, and more on broad, multi-level frameworks for understanding the ecological mechanisms of effective and replicable EBI delivery (e.g., Century et al. 2012; Chaudoir et al. 2013; Damschroder et al. 2009). Aarons et al. (2011) note that these frameworks share several commonalities. Most view the process of implementation in stages and recognize that the transition through stages is iterative and non-linear. Additionally, although different frameworks often share similar features, different models emphasize specific factors above others. Finally, current frameworks acknowledge a lack of research to support their hypothesized key factors. There is consensus that implementation is aided by a range of components, over time, at multiple ecological levels. Based on our review of the recent literature on implementation frameworks, and consistent with an earlier study by one of these authors (Akin et al. 2014), we identified six relevant implementation factors: (1) process, (2) provider, (3) innovation, (4) client, (5) organizational, and (6) structural. The following describes each and reviews the applicable literature.

Process factors affect fidelity to the EBI and are related to successful implementation. The National Implementation Research Network (NIRN) uses the term competency drivers to refer to process factors, which include staff recruitment and selection, training, coaching, and performance assessment (Fixsen et al. 2009). Appropriate and adequate training is key in a child welfare setting where qualified staff shortages and burnout are common issues (Kaye et al. 2012). Earlier research indicates that training should not be solely didactic; rather, it should utilize applied learning techniques delivered by competent staff (Aarons and Palinkas 2007) and should be paired with clinical supervision and high-quality coaching (Kaye et al. 2012). In addition to quality, coaching must be sufficiently intense (Barth 2008), ongoing, supportive, and provide individualized performance feedback (Aarons and Palinkas 2007; Aarons et al. 2009).

Research suggests that certain provider characteristics influence EBI implementation and effectiveness. Because implementing EBIs requires that workers both buy into the suggestion that evidence-based practices and programs are more effective than the status quo, as well as learn and adopt new skills and techniques, earlier research has focused on practitioners’ attitudes and openness to EBIs (Aarons 2004; Aarons et al. 2012a, b, 2007; Gray et al. 2013). The Evidence-Based Practice Attitudes Scale (EBPAS) identified four dimensions of attitudes toward adopting EBIs: intuitive appeal of EBI, likelihood of adopting EBI given requirements to do so, openness to new practices, and perceived divergence of usual practice with EBIs (Aarons 2004). The EBPAS was later expanded to the EBPAS-50, which included eight new factors: limitations, fit, monitoring, balance, burden, job security, organizational support, and feedback (Aarons et al. 2012a). Demographic and individual provider characteristics that were associated with more positive attitudes toward adopting EBIs included employment in a private versus public setting, African American compared to Caucasian race, and clinicians with smaller caseloads compared to higher caseloads (Aarons et al. 2012a).

Other research identified provider factors that acted as barriers to EBI implementation. Gray et al. (2013) reviewed 11 studies on EBI implementation and found that inadequate skills and knowledge and negative attitudes were barriers. Related to skills and knowledge, one study specifically identified the lack of research-based education, including resistance to scientific claims of objective superiority, particularly among social workers. This review also revealed that inadequate agency resources dedicated to EBI, poor organizational culture, and inadequate supervision further obstructed EBI implementation. Other studies suggest that organizational factors influence practitioner attitudes toward EBIs. Aarons et al. (2012b) found that more proficient organizational cultures and less stressful, more engaged organizational climates were associated with more positive EBI attitudes among providers.

Innovation factors are influential, and barriers to EBI implementation occur if a program is too complex, not adaptable, or unsustainably expensive (Damschroder et al. 2009). Related to EBI complexity or misfit, Proctor et al. (2007) found that agency directors were often concerned about whether the EBI was applicable to their client population. Regarding adaptability, research finds that innovations must also be compatible with the approach and values of the agency or system (Maher et al. 2009; Michalopoulos et al. 2012). Issues of cost include logistical problems such as transportation and staff scheduling, which have been cited as barriers to effective implementation (Maher et al. 2009; Wharton and Bolland 2012). Additionally, Wharton and Bolland’s (2012) survey of licensed social workers found that financial cost was statistically significantly associated with problems utilizing EBI for the profession as a whole, and problems for individual practitioners.

Limited research has evaluated client factors associated with EBI implementation in a child welfare setting, including challenges associated with delivering EBIs to a largely non-voluntary client population. Michalopoulos et al. (2012) conducted nine focus groups with child welfare workers across multiple regions in a state during a specific EBI implementation and found that engaging families was a significant challenge. Workers reported that families were resistant, felt a sense of distrust, assumed change would occur without active participation, and included extended or informal family members who were ineligible for cash assistance or services, thus further severing the therapeutic relationship and willingness to participate in the EBI.

Aarons and Palinkas (2007) also captured information on how client characteristics were perceived to influence EBI implementation in a child welfare setting. Of the six key implementation factors these authors identified through caseworker interviews, two incorporated family issues: acceptability of the EBI to the caseworker and to the family, and suitability of the EBI to the needs of the family. These factors included client characteristics such as cognitive abilities, child welfare case status (e.g., in-home/placement prevention, or post-foster care/reunification), age of parents and children, complexity of family problems, and level of family functioning. Caseworkers reported that EBI delivery was more difficult for involuntary clients who are less likely to engage in services.

Organizational factors include leadership capacities, which have been shown to contribute to implementation success through communicating and reinforcing a strong vision, priorities, and goals, and requiring that practice change to reflect the newly adopted EBI (Aarons and Palinkas 2007; Crea et al. 2008). Research suggests that organizational climate and culture, leadership, and providers’ openness to and interest in EBI interact to influence implementation effectiveness (Aarons 2006; Aarons and Palinkas 2007; Aarons and Sawitzky 2006; Gray et al. 2013; Kimber et al. 2012; Proctor et al. 2007). Kimber et al. (2012) reported on an in-depth case study of an agency during EBI implementation and identified several key organizational factors including the importance of a well thought-out and strategized clinical transformation period, effective management and working groups with clear goals and objectives, and inclusion of all levels of staff in the culture-change process.

Finally, the setting outside an individual agency or group of agencies can impact EBI implementation success. In child welfare, key structural factors include shortages of well-trained workforce (Kaye et al. 2012), challenges associated with policies and the courts, and difficulties building interagency and inter-system collaborative capacities (Maher et al. 2009). The case study by Maher et al. (2009) observed that coordinating the child welfare courts with mental health services and the education system, as was required for their EBI implementation, became difficult when questions of roles, responsibilities, and trust arose. Additionally, tracking outcomes across systems for children in foster care is a barrier to EBIs in child welfare. Solutions to these issues included key stakeholder meetings to problem-solve and develop procedures to follow the law while also facilitating EBI implementation.

Taken together, this growing body of implementation science literature reveals many critical aspects to implementation at multiple levels that will aide researchers, administrators, and agencies toward improved EBI delivery. Despite the fact that evidence is growing, a broad gap still exists, especially in knowledge about implementation in the child welfare system. This gap is an important one, as these families comprise some of the most vulnerable adults and children in the public service sector. Few earlier studies documented the real-world process of EBI implementation in child welfare. As noted, the child welfare population and setting present many unique challenges to service delivery in general, and particularly when that service is a structured EBI. Notably lacking in the literature are studies documenting this experience from the perspectives of child welfare staff. Given that these practitioners operationalize and enliven EBIs, getting their input on the ups and downs of EBI implementation is most relevant and useful.

This article aims to expand the existing literature by examining implementation of two EBIs in two child welfare jurisdictions and by including perceptions of frontline, administrative, and support staff from these different service settings. Consistent with the selected methodology, we began the study with the preliminary hypothesis that a number of implementation factors—including process, provider, innovation, client, organizational, and structural factors—present supports and challenges that contribute to the successful implementation of an EBI. Through content analysis of qualitative interviews, we sought to determine whether or not this preliminary hypothesis was tenable, and if so, in what specific ways these factors impacted implementation that were consistent with or divergent from prior research.

Method

Participants

Study participants included service providers, agency leadership, and administrative support staff from two programs in two Midwestern states. To qualify for participation, study informants had to be involved with implementation of SFP/CF! as part of the federal grant project in their respective agencies. This involvement included frontline staff who directly provided the EBI to families, as well as agency leadership responsible for program oversight, and support staff who provided other support functions as part of implementation, including assistance with the family meal and other support tasks during program activities. This sampling strategy was utilized to maximize opportunity to extract a wide range of perspectives on the challenges and necessary supports involved in implementation of EBIs in order to improve the complexity and richness of the data thus increasing transferability of the findings.

All participants (N = 15) involved with implementation of projects at these two sites were invited to participate in the study via email. Ten of the fifteen staff agreed to participate in the study, including three administrators, five frontline service providers, and two administrative support staff who served as coordinators for the EBIs. Although these three groups of agency personnel are not evenly represented, due to the size of both agencies and the staffing requirements of the EBIs, staff members at both agencies often served in more than one role. Administrators and agency leadership sometimes provided direct or support services, and clinical staff were sometimes involved in program oversight. This overlap allowed for many participants to provide perspective on more than one aspect of implementation. All participants were women and ranged in experience in child welfare from approximately 6 months to 30 years with a mean of approximately 9.25 years. Experience in their current positions within the agencies ranged from 6 months to 9 years with a mean of approximately 3.23 years. In terms of experience with the EBI, one participating site had been actively involved in delivering the EBI for 4 years, and the other site for 1.5 years.

Procedures

This study was a sub-study of two federally-funded projects: (1) the Tulsa County (Oklahoma) Children Affected by Methamphetamine project, funded by the U.S. Department of Health and Human Services (DHHS) Substance Abuse and Mental Health Services Administration; and, (2) the Iowa Regional Partnership Grant project, funded by the U.S. DHHS Administration for Children, Youth, and Families, Children’s Bureau. Both projects were implementing EBIs in child welfare settings to improve outcomes among children and families who were child welfare-involved due to caregiver substance use. The EBIs implemented in these projects were the Strengthening Families Program (SFP) and Celebrating Families! (CF!). SFP is a 14-week, group-based, family intervention that targets the promotion of resilience against behavioral, emotional, academic, and social problems for families with children ages 3–16 (Kumpfer and Alvarado 2003). CF! is a 15-week group-based family intervention for substance affected families that addresses social and emotional health and issues of addiction (Quittan 2004). Both participating sites implemented these two EBIs consecutively to enrolled families.

One member of the research team, who was not otherwise involved in training or implementation of the project, recruited agency staff for participation via email and then conducted all of the interviews by phone. Interviews, on average, lasted approximately 40 min with length dependent on provider responses. All interviews were digitally audio recorded with participant permission and were transcribed verbatim by a professional transcriptionist. Transcription was then followed by a review of the completed transcripts by the interviewer while listening to the audio recordings to check for accuracy and make corrections.

Measures

A semi-structured interview guide was developed based on the implementation literature and a previous EBI implementation study (Akin et al. 2014) and used to conduct all interviews. This guide, informed by NIRN’s Implementation Drivers framework (Fixsen et al. 2009, 2005) included open-ended questions about: (1) practitioner background; (2) EBI training; (3) EBI coaching; (4) EBI practice with families; (5) families’ response to the EBI; and, (6) administrative and organizational supports. Adaptations were made to this guide as appropriate to apply to the participating sites and varying provider roles; however, substantively, the core content of the guide was unchanged. Clarifying questions and probes were incorporated as needed and participants were encouraged to diverge from the guide if necessary to discuss their experiences.

Data Analyses

This study implemented a Modified Analytic Induction (Bogdan and Biklen 1998) methodology to conduct a comparative analysis of workers’ experiences with EBI implementation in two child welfare service settings. Modified Analytic Induction is an emergent methodological design that allows for the examination of preconceived hypotheses identifying patterns of behavior, such as implementation of an EBI in a practice setting. Application of this methodological approach includes examination of hypotheses that are derived from the literature in order to confirm or iteratively revise hypotheses. Rather than seeking a universal causal hypothesis, the purpose of this method is to adequately describe patterns of behavior that are anchored in the literature while also allowing for the emergence of new ideas from the data (Gilgun 1992, 1995). This analytic method relies on the a priori identification of a theoretical framework which was identified, in this case, through the use of the established interview guide (Braun and Clarke 2006).

Analyses were conducted in four stages. First, the interviewer conducted the initial analysis using open coding to classify the data into individual codes. Initial coding was conducted inductively, with no predetermined coding structure. Sixty-three codes were identified and iteratively refined as they emerged from the data. Then, the interviewer conducted a second analysis to classify the individual codes into the a priori themes established from prior research while also actively seeking to identify any negative cases that may dictate the need to iteratively revise the initial hypothesis. At this point, a summary of the findings was presented to study informants to conduct member checking. All respondents who participated in member checking reported agreement that the findings adequately represented their perspective. The findings were also reviewed by members of the research team—some of whom participated in the previous EBI implementation study and developed the initial themes—for the purposes of analytic triangulation. Primary coding and analysis was conducted using Atlas.ti (Version 6.2).

In order to strengthen trustworthiness, a second coder, who was not involved in the interviews, analyzed the data according to the six a priori themes and completed additional negative case analysis, seeking examples of data that did not fit within the final themes. In terms of procedure, the second coder began by organizing the first coder’s inductive coding guide according to the six themes. This organization was reviewed and confirmed by the first coder. Next, the second coder reviewed the contents of each of the inductive codes within its assigned theme in order to confirm the quote’s appropriate placement within each theme. Finally, additional illustrative quotes and examples for the negative case analysis were highlighted for further consideration.

Results

As guided by the study’s methods and analytic strategy, the results are organized around the six implementation factors as outlined in the introduction and grounded in existing literature: process, provider, innovation, client, organizational, and structural factors. Although these factors are reported separately for clarity and differentiation of the various constructs, the results show that these factors are often interrelated.

Process Factors

Staff Selection

Staff selection to identify who would receive training to deliver the interventions was conducted differently at the two implementation sites. This differential selection process resulted in differing benefits and challenges, and directly influenced organizational, structural, and provider factors including issues of staffing for implementation and staff buy-in.

At site A, due to its small size, every single agency staff member received training. Workers received some autonomy in terms of choosing their individual roles. While this meant the workload was higher for everyone, the workers reported that educating all workers increased agency capacities. For example, the all-inclusive training resulted in some personnel receiving important information who might otherwise not have been trained, such as front office staff. As a result, clients were immersed in an agency trained in the EBI. One participant described:

Even though [certain staff] may not be running the program, they might be seeing clients who are in the program for individual therapy, so it allows them to be able to talk with their clients about it and answer questions, or direct them to the right person for questions.

Site B approached staff selection differently. At this site, the training was made available to all providers, as well as a variety of “key players” from outside agencies who were involved as community partners in the grant funding supporting this implementation. These stakeholders were trained “for their benefit” and “didn’t really expect to ever participate.” While this inclusion increased understanding of the program among external stakeholders, it also introduced difficulty in staffing. One participant reported:

Because some people came to trainings that really didn’t intend to [implement the program], …it looked better in the room than it really was when I started putting a team together…I wish that we would have asked for more commitment from those stakeholders. You know, we are training one-on-one and they’re getting such a great thing and now most of them aren’t doing anything.

Once providers were trained at this site, trained providers were contacted to recruit willing group facilitators. Participation was voluntary at this site but roles for those who chose to be a part of the program were assigned by a program coordinator, with some input from providers on role preference when possible. Provider willingness to participate varied among staff because participation was voluntary and “there really isn’t any incentive for anybody to jump in and help.” Therefore, consistent staffing of groups at this site was difficult and resulted in additional strain on participating providers.

Training

Providers across both sites consistently reported positive experiences with SFP training. Providers described training as “fun,” “interactive,” “hands-on,” “helpful,” and “interesting.”

Well, the actual training in and of itself, I loved the role-play and I loved the very specific techniques. I mean, we got to practice. Some of us got to practice being the therapist, some of us got to practice being the kids, some of us got to practice being um…those real world experiences, where we say “yeah, this what my client would have said.”…I get a lot out of a training when you get to apply things and do things, as opposed to just read a book and take a test.

Beyond the content of the training, the logistics of the SFP training were also well-received by the site participants.

The training process, I mean it was really nice, because actually the trainers came to our facility, and I believe it might have been a day or two day process, where they came to our agency. I want to say they were so flexible with our schedule. I think they came like on a Friday and stayed on a Saturday, and so it gave our staff the ability to continue. We didn’t have to miss work or cancel appointments.

When discussing the differences and comparing SFP to CF! training, providers clearly preferred SFP, describing CF! as “mind-numbing.” These differences seemed to be related to the length of the training, the didactic approach of the instructor, and the different organization of the manual that providers found more difficult to use.

Coaching

Coaching consisted of site visits and phone consultation as a follow-up to the training. Not all trained providers were active participants in this aspect of the training and implementation. Several participants reported routinely missing scheduled meetings or calls due to other duties and responsibilities. Others reported a division of labor in which a supervisor or other staff would attend consultation and debriefing meetings on their behalf. This lax consultation participation requirement permitted agencies with limited resources (including time and personnel) to meet all of the obligations of the training requirement. However, this divide-and-conquer strategy meant that not all trained providers received the full benefit of the consultation which was intended to support best practice and high-fidelity delivery of the interventions.

Despite the fact that not all providers were active participants in this aspect of training and implementation, interview participants gave an overall positive report about the necessity, the length, and the utility of the coaching. With the exception of one participant who reported receiving limited coaching, the remaining participants were consistent in their reports on the helpfulness of feedback from the EBI purveyor. One stated:

[The SFP coach] is fabulous, and she gives the most amazing feedback. She will actually go into the group and then she would meet with us afterwards and just give us some tips. I mean, we never felt like we were alone, and it wasn’t as if she was coming to grade us or telling us that we were horrible.

Although there were mixed reports regarding receipt of consultation and coaching, those who were exposed to purveyor feedback expressed strong satisfaction with the assistance.

Provider Factors

The participating providers had a range of experience both in child welfare settings, as well as in their current positions. They also came from a variety of disciplines and educational backgrounds, including associates through master’s degrees in psychology, social work, education, criminal justice, and sociology. With this diversity of backgrounds came a range of previous experiences and attitudes about EBIs. Some participants reported little or no prior experience with EBIs, while several reported being at least familiar with EBIs, and a few reported having more experience learning and implementing EBIs.

Although some providers with extensive experience reported on some of the shortcomings of EBI training, no opinions were exclusively negative and most reported positive opinions about the utility and the importance of using interventions with established evidence. Participants reported that EBIs are “common,” “effective,” “necessary,” “useful,” and provide curricula that enhance provider skills and confidence to deliver the intervention. One veteran provider reported appreciating renewal of basic skills through training in EBIs.

Evidence-based…sometimes they go into really…um, remedial types of techniques, especially somebody who’s been providing therapy for a long time. But I mean, I think they’re necessary, and a lot of times going through trainings that kind of help renew a counselor to go back to the basics, or just provide a few, you know, specific techniques to use with specific clients.

One less experienced provider reported appreciation for EBIs as a guiding framework with evidence that enhances credibility to the families.

I guess I was always a little bit nervous about doing the parents group because I don’t have kids of my own…and I absolutely loved it, and I think that a lot of it is because it is evidence-based curriculum and it’s stuff that—it’s not me sitting up there, saying, “Well, you know, I tried this.” It’s from the book and it’s real, so you know what I mean? You’re able to feel confident about it…it’s easy to teach it and to feel confident about it.

In addition to issues surrounding background and experience, providers also discussed their “fit” with these particular interventions. All participants reported feeling that they were a good fit. Rationale included their own experiences and expertise personally and professionally. Others attributed their fit with these interventions to a shared philosophy in the importance of family systems interventions over individual approaches.

Innovation Factors

Three aspects of innovation emerged from the data: strengths and challenges associated with using a manualized curriculum; the bidirectional, relationship-based effect of EBI satisfaction between provider and client; and the usefulness of program content and style.

Multiple providers expressed hesitation and nervousness upon initial training and introduction to the intervention manuals. In general, providers felt there was too much material to cover and feared getting behind in the curriculum or leaving something out. Consultation, along with experience actually delivering the intervention effectively, reduced these concerns and the providers all reported feeling that the curriculum was flexible to the needs of the families and to the logistical issues that can arise in providing community-based services.

A provider discussed how she was able to make adaptations to the children’s group, with consultation from the coach, by breaking a large group with a “broad range” of ages into two smaller groups to meet the needs of the children more effectively. Another provider reported that while the curriculum fell short by not including enough information for families with special needs such as “ADHD,” “brain injury,” “trauma,” or “behavior problems” like “intermittent explosive disorder,” she was able to meet family needs by incorporating her own knowledge of these issues into the curriculum. Also reported as important was having the flexibility to make accommodations for children with special needs by instituting a “buddy system” to facilitate participation for several children with autism.

Participants also frequently discussed the ways that client and provider relationships and satisfaction with the EBI were linked. For example, one participant stated that the parents “enjoy and look forward to coming to the groups… they are just enthusiastic and they can’t wait till the next group and the kids are so excited. It’s wonderful to see.” Another commented:

[The] majority of my clients through the program are in Strengthening Families so that helped me, you know, build even a stronger relationship with them because I saw them for…fourteen weeks plus graduation.

The client’s enthusiasm about the programming and structure of the EBI that supported relationship-building positively influenced the provider’s program buy-in.

Finally, it was apparent that providers and clients enjoyed and benefitted from the program content and style of delivery. Multiple providers mentioned using the tools from the EBI in parenting their own children. Additionally, most providers appreciated the “interactive” and “reality-based” aspects to how the program content was delivered. One participant stated, “I think [the] most beneficial [aspect] is just the hands-on part of it, getting to actually practice the skills with a counselor’s help.”

Client Factors

Multiple client factors impacted implementation of EBIs in this study, both positively and negatively. First, as this implementation occurred in partnership with a family drug court, caregiver substance use—and the consequences of that use on the children—often influenced delivery. Providers noted that parents are not necessarily substance abstinent during participation. Additionally, the children were noted as facing various challenges including “significant developmental issues” and “behavioral problems” due to exposure to parental substance abuse.

At the same time, though, providers reported that despite these barriers to service engagement, parents were often more committed to SFP/CF! participation than other aspects of their child welfare system mandates. One provider commented:

Even if they’re failing UAs, even if they’re failing to do other things, they almost always come to SFP. They almost always participate. I mean, like 99 percent of the time—even if they’re AWOL from everything else, they’ll show up. And, you know, I think that speaks to the program. And that’s been a little bit hard, because initially, the courts are designed to be punitive often, and so “well they can’t come if they’re not doing anything else.” So we’ve had to work through that.

A couple of client factors found in the data were unique to EBI service delivery in the child welfare system. Many of the children in the participating families were in out-of-home placements, which presented barriers related to the child and parents, as well as with the foster families. One major benefit and incentive of the program was that it often served as a visit—or additional visit—for the parents and children in out-of-home placement. But this dynamic also introduced challenges. One provider commented on the difficulties associated with separation that occur during group after the family meal, when participants would experience “huge meltdowns with mommies and babies.” The providers reported that this separation reaction diminished over time as parents and children learned that the structure of the program was predictable and that separation was not long-term (the parents and children reunited after a period of separate activities).

Additionally, in some cases, “foster parents have been the most difficult to get on board” with this program. One participant reported that in her state foster parents are also seen as potential adoptive placements from the start, which introduces challenges with foster parent willingness to participate as “that might be their child and they don’t want to drop it off with someone who has this long history of substance abuse, child neglect…it’s a lot of trust issues with the foster parents.”

Despite these challenges, family response to this program was overwhelmingly positive and though providers reported that participation “can be court ordered,” this is seldom necessary to ensure family participation. Tying back into the finding related to parental substance abuse, one participant noted that “the only time that maybe we’ve had some people be discharged from Strengthening Families is because they’ve relapsed or, you know… they’re using and they’ve just cut everybody off.” In sum, participants reported that despite client challenges related to ongoing substance abuse, implementation of the EBI was strengthened by client engagement and participation.

Organizational Factors

As implementation at both sites was championed by leadership, organizational cultures and climates as a whole were hospitable for implementing the interventions. In discussing the impact and importance of supervisors on implementation, participants were overwhelmingly positive. Providers noted that stakeholder and supervisor enthusiasm about the program was contagious and influenced staff motivation for implementation. This positive impression was also reflected in the discussion of the impact of agency administrators on EBI implementation.

Well, I think they’re the only reason that we can still do it even when there’s no money to do it. But I have a great agency that says, “you know what? We’re just gonna do this. We’re gonna do a good job for our families.” Um, you know the CEO here is up for anything that will help our families. And, so, we have a wonderful supportive agency.

Each of the implementation sites also worked with implementation teams known as steering committees which comprised key stakeholders, project staff, and grant funding representatives. As part of this research, the role of the steering committee was also assessed. However, responses to the steering committee support for implementation were inconsistent. While some participants praised the steering committee and gave specific instances of their paving the way for policy changes that smoothed implementation, several reported not knowing who the steering committee was, and many others felt the steering committee would be more useful if they took a more hands-on approach.

As previously discussed, the buy-in of frontline staff at Site B was viewed as inadequate and a challenge to implementation. This persisted despite the consistently high praise of agency leadership and organizational support. Choices made during implementation at this site, such as voluntary staff participation, seemed to exacerbate the problem. One participant reported the ability to become a SFP/CF! group leader existed because Site B reorganized her position to include implementation duties rather than asking her to voluntarily add them on to her regular duties. She suggested accommodations such as this, or otherwise incentivizing participation, may be necessary to rectify this implementation challenge as “they have a hard time getting anyone wanting to do it…everybody works full-time and, so, it’s really hard to convince somebody it’s a good idea to give up some of your free time.” Another participant shared the view that “you almost have to have that as your only job to really do a good job.”

Staff at both sites overwhelmingly reported that having enough staff and having appropriate facilities in which to hold groups were the most significant challenges to implementation.

This is probably the most rewarding thing that our counselors said that they had ever done, but it’s a really intense program and when you do it for three years, two nights a week, it can lead to burnout of some of our staff. And because it’s so staff-intensive, where you have to have six/eight staff members there, plus volunteers, as far as keeping staff motivated to want to do it, that’s a little bit difficult. And then if the old staff don’t want to do it anymore and you bring on new people, then you have to get them trained. And so I think the intensity of the staff that this requires, this program could be seen as sometimes a barrier.

Space was a substantial issue. Intervention trainers recommend holding groups off-site. However, holding groups offsite presented challenges of disruption to their operations. Providers noted that going off-site required travel time, added additional burden to staff who remained on-site, and interfered with the typical flow of the organization.

However, community-based sites were not always ideal either due to the demands of the program that requires multiple group rooms and kitchen facilities. One facilitator held a preschool aged group in a church hallway with an adult-sized table and no climate control, as a last resort, and reported “that’s hard to keep their attention.” She shared:

My biggest challenge was…You don’t have any place to hang anything. You didn’t have an easel. You didn’t have anything. We’re just in the hallway.

In sum, organizational factors both challenged and supported implementation. Ambivalent, detached stakeholders and a lack of resources were primary barriers, while organizational commitment to implementation was viewed as a key strength.

Structural Factors

The impact of structural factors on implementation was also mixed. One structural factor that impacted implementation was involvement of the court and the states’ Department of Children and Families (DCF). Providers reported some difficulty dealing with the rigid court structure and turnover with DCF, but also reported that overall, collaboration with these systems brought about positive results for the system and for the families.

It was just kind of a lot of bureaucratic red tape, essentially, and making sure that we got—that what we were doing was going to satisfy the court in terms of what the parents needed to do, to fulfill their treatment plan with the court, and even just you know kind of getting the courts to see that this is treatment, and that just because somebody maybe made a minor mistake, it doesn’t necessarily mean they need to be thrown out of the program altogether.

Another major barrier to implementation had to do with the interaction of caregiver substance use with the involvement of the drug court. This interaction often influenced how and when services were delivered. Providers reported that the drug court sanctions would, at times, interfere with parent participation if, for example, a parent received a jail sanction (which was used at one site). Additionally, the drug court would occasionally mandate inpatient treatment after the parent started SFP/CF! This was cited as disruptive to the parent, the child, and the group.

Despite the difficulties of cross-system collaboration, multiple participants reported positive outcomes in relation to community partners and agencies.

I think it’s maybe even gotten stronger, and it’s really the outside agencies and the outside partners really have buy-into that philosophy now, where some of them didn’t before. But the majority of them do now. I mean, definitely even all the way to our judges and everything.

Sometimes it was hard, but there were also even more times when those community stakeholders would come and help…we had community stakeholders, including judges and family members of judges come and volunteer on this program…it was really a community team in implementing this, even from the family drug courts. Workers provided transportation. I mean it was a community effort to implement this program.

Discussion

In this era of revenue-driven mandates to simultaneously cut costs and improve services, policy makers are increasingly focused on the need for evidence that demonstrates the effectiveness of publicly-funded services. Therefore, understanding the multiple supports and challenges related to EBI delivery is needed across social service areas. This study represents an effort to build knowledge on how child welfare practitioners experience EBI implementation from which several key findings emerged.

As suggested by the NIRN active implementation framework (Fixsen et al. 2005), the competency drivers of staff selection, training, coaching, and fidelity monitoring were salient themes among practitioners. Staff selection protocols that simply added the EBI as an additional responsibility were perceived as more problematic and stressful for frontline practitioners. Given an overburdened workforce, child welfare implementers should carefully consider staff selection issues and ensure adequate allocation of worker time and availability. Underestimating the importance of this implementation driver may result in more stumbling blocks or inadequate implementation.

Regarding training, practitioners frequently complained about one of the two EBI trainings they received. Similar to another recent child welfare implementation study (Akin et al. 2014), this study indicated that selecting EBIs that utilize interactive, hands-on, and engaging training approaches may be an important consideration during implementation planning. Additionally, practitioners in this study noted that flexibility was critical to successful EBI implementation. Importantly, coaching is key to determining appropriate adaptations that solve problems and maintain adequate fidelity. The finding that practitioners were often unable to participate in coaching and fidelity monitoring indicates that additional resources were needed. Together with the growing implementation literature that stresses the importance of coaching (Barth 2008; Beidas et al. 2013; Fixsen et al. 2009; Kaye et al. 2012; Nadeem et al. 2013), these findings suggests that this aspect of implementation may be the wrong corner to cut.

This study’s findings contrast with earlier research on provider factors, which indicated that child welfare workers were generally resistant to EBIs (Aarons et al. 2012c; Gray et al. 2013). Our participants expressed significant interest in and satisfaction with EBIs. They articulated a clear rationale for using EBIs and described that they could integrate it with their prior education, training, and practice experience. This shift in the workforce’s attitudes toward EBIs was likely influenced by the increasing buy-in and commonness to EBI at all levels of the child welfare system. Perhaps the presence of parental substance abuse also contributed to worker satisfaction with the EBIs. In an earlier work by one of the study’s authors, workforce members expressed difficulty in working with substance abuse, and verbalized needing intervention tools to help improve parenting. In the case of both of these EBIs, the intervention was designed and targeted to this specific population, thus giving the workforce a set of “working tools” that they lacked prior to implementation (Akin et al. 2014).

These findings build upon prior implementation literature on innovation factors, which suggests that practitioners have concerns about EBI’s complexity and fit in their setting and practice context (Proctor et al. 2007). Despite initial hesitation, after receiving consultation and gaining experience with the EBIs, our study participants viewed the EBIs as having a good fit with child welfare. An important aspect of adopting the EBI was receiving consultation, which assisted practitioners in knowing what they could adapt for their site while maintaining fidelity to the EBI. Given the multiple, intense, and complex needs of child welfare clients, EBI flexibility within a framework of fidelity may be even more pertinent to child welfare settings.

Consistent with prior literature, our findings indicated that EBI implementation in child welfare is affected by client factors, particularly the amount and nature of client problems as well as their interaction with the service context (Aarons and Palinkas 2007; Akin et al. 2014; Maher et al. 2009; Michalopoulos et al. 2012; Wharton and Bolland 2012). Practitioners described overcoming implementation obstacles related to children’s complex behavioral health needs, uneven parent participation primarily due to inpatient treatment, and foster parent buy-in. Despite these challenges, practitioners viewed parents’ responses to the EBI as largely positive and engaged.

Another client factor finding was related to the need for attention to issues of substance abuse at the levels of courts, agency administration, and day-to-day EBI delivery. Although client substance abuse was likely more prevalent in our study because the EBIs were delivered in conjunction with a family drug court, parental substance abuse is highly common among families involved in foster care (Testa and Smith 2009). Therefore, these findings may have implications for EBI implementation in more general foster care settings. Practitioners made the interesting observation that parents affected by substance abuse, who may be disengaged in many other aspects of their case plans, were committed to participating in these EBIs. Practitioners indicated that EBI participation offered these parents an additional opportunity to see their child who was in foster care. Given these findings, courts and child welfare agencies may need to consider policies and protocols that support parent involvement in EBIs at all points in the life of the case, even prior to sobriety or substance abuse treatment. We submit that because the parent–child interaction is inherently supervised during the EBI session, thus ensuring child safety, this presents a unique and compelling opportunity to reengage parents struggling to move toward reunification. Future research should further investigate this phenomenon, including parent recovery and family reunification outcomes.

Among all the factor categories, the most prominent was organizational factors. The themes related to leadership, EBI implementation planning, and access to adequate resources were prevalent across many findings. In short, organizations must install and maintain the supports, resources, and procedures that create a hospitable environment for successful implementation of the EBI (Fixsen et al. 2005). Regarding leadership, for example, the practice of relying on practitioners to volunteer to facilitate an EBI was seen as a significant barrier. Agency leaders must assign and support roles for EBI delivery. Earlier literature suggests that openness and overall attitudes toward EBI should be considered when selecting staff (Aarons 2004; Aarons and Palinkas 2007). This study also suggests that successful EBI delivery requires adequate staff resources and facilities. Although some portion of this is a structural resource issue, leadership may play a significant role. For example, agency administrators could facilitate successful implementation by building relationships with other agencies in the community in order to maximize the use of space resources.

Access to adequate resources emerged as relevant throughout multiple factors within our analysis framework, including organizational factors. The common themes of space and staffing also related to EBI implementation teams, planning, problem-solving, and communication. Earlier literature suggests that a “thoughtful and intentional” organizational change process enables effective EBI implementation (Kimber et al. 2012, p. 323). In this study, the steering committee’s efforts to problem solve the day-to-day obstacles of using an EBI were viewed by practitioners as a critical factor to successful implementation. Likewise, practitioners noted the relevance of buy-in and support they received from various organizational levels. In contrast, some practitioners knew little about the steering committee and others had complaints. Low levels of vertical communication and/or perceived ambivalence from the steering committee were viewed negatively. Frontline staff wanted the project leaders to acknowledge how difficult it was to implement the EBIs. These findings suggest that implementation teams must perform critical functions across the stages of implementation, such as increasing readiness, developing and maintaining enthusiasm, communicating support, installing and sustaining key components of the infrastructure, problem-solving, and building linkages with external systems (Metz and Bartley 2012, p. 15).

The findings regarding structural factors suggest that EBI implementation in child welfare settings invariably requires the involvement of external stakeholders (Akin et al. 2014; Kaye et al. 2012; Kimber et al. 2012; Maher et al. 2009). These practitioners recommended informing and involving the courts and the public child welfare agency, describing the implementation as being taxed by rigid court structures and high worker turnover. Accordingly, implementers should consider the extra time and effort that may be needed to inform stakeholders of the program and obtain their buy-in. On the positive side, cross-system collaboration was identified as a both a key facilitator and a payoff. That is, the time and staff resources that they invested in educating external stakeholders resulted in gaining their support and building champions for the EBIs.

Limitations

A primary limitation of this study was the length of engagement with study participants and the number of interviews conducted. Traditionally, qualitative inquiry—as well as the guiding methodology of this study—both dictate prolonged engagement with informants that continues until saturation is reached in an effort to fully understand the phenomena under examination. However, it has been suggested that this task may be too large for the scope of most studies and that by tightly defining the population to whom the findings apply we may reduce the scope of the study to a more manageable sample (Bogdan and Biklen 1998). Despite the limitations of the sample size and engagement strategy, participant responses were largely consistent and met the standard of saturation. We, therefore, suggest that the sample size and engagement strategy were sufficient for the purposes of this study. Though interpretation of the findings should not be extended beyond this study’s sample, when taken together with the findings of previous implementation research, these findings serve to broaden our understanding of implementation of EBIs in child welfare practice settings.

A second limitation has to do with the inclusion of client related factors examined only through the standpoint of agency staff. Though these providers, administrators, and support staff offer a valuable perspective worthy of examination, this standpoint is not a replacement for family perspectives. Because the purpose of this study is to examine provider—rather than family—perspectives, and the general consensus among these providers suggests validity to the perspectives reported, we propose that this was not a major limitation to the current study. However, as client perspective is an important consideration, we suggest that future exploration of EBI implementation investigate client views.

This study explored the supports and barriers to EBI implementation in a child welfare setting by seeking the input of the local implementers. The results contribute to the implementation literature by confirming that diverse but connected factors at multiple levels affect the success of EBI implementation (Aarons and Palinkas 2007). While selecting and adopting an EBI is important, it is only the first step in a systematic process of using an EBI to affect change in child welfare settings. Administrators and practitioners should carefully consider the various factors that may impede or facilitate the full and adequate implementation of an EBI throughout different stages. Underestimating the relevance of these key facilitators can result in greater barriers and inadequate implementation. While there is no one right way to implement an EBI, informing the process of these multiple levels of factors may be key to supporting the path to improved outcomes.