Evidence-based practices (EBPs) are not widely available to youth and families in community mental health (CMH) settings in the United States (Hogan 2003). EBPs are intervention approaches supported by the best available empirical evidence (Chambless and Hollon 1998), allows for clinician judgment and expertise, and consumer choice, preference and cultural background (Institute of Medicine 2001; APA Presidential Task Force on Evidence-Based Practice 2006). Large, national surveys conducted in the United States have revealed estimates of roughly 1 in 5 children meeting criteria for a mental health disorder (Costello et al. 1996), yet nearly 80 % did not receive mental health care in the previous year, with significant disparities for ethnic minority youths and those who are uninsured (Kataoka et al. 2002). The gross underutilization of community-based mental health services (McKay et al. 2005) indicates that engagement in an EBP embedded in CMH should also be considered. Delivery of culturally relevant EBPs explicitly designed to engage caregivers and youths in community settings could improve the quality of care that families receive and increase service utilization. The Family Check-Up (FCU; Dishion and Stormshak 2007) is one such promising model. The FCU is grounded in the principles of parent management training programs (Forgatch and Patterson 2010) and has demonstrated efficacy through randomized, controlled trials conducted in public middle schools (e.g., Stormshak et al. 2011) and as a home visiting program (e.g., Dishion et al. 2008). This article presents the findings of a pragmatic trial testing the effectiveness and implementation outcomes of the FCU delivered in low-resource CMH agencies.

The Family Check-Up

The FCU was designed to be implemented in community service settings and to have a public health impact on reducing antisocial behaviors and associated problems (Dishion and Stormshak 2007). The intervention aims to improve children’s adjustment across settings (home, school, neighborhood) by motivating effective and positive parenting practices exemplified by parental involvement in the child’s activities, positive reinforcement and support of the child’s behaviors, positive structuring of interactions, and proactive anticipation of potential problems. The FCU is a three-step intervention that is delivered to caregivers in 2–3 sessions. It begins with an initial interview with the caregivers to gather background information about the youth and caregivers and establish rapport and a trusting therapeutic relationship. Next, an ecological assessment is conducted that comprises brief (about 20 min total), developmentally appropriate interaction tasks that assess salient family functioning and caregiving domains implicated in the etiology of problem behaviors (Patterson et al. 1992; Shaw et al. 2003) and identified as mechanisms of change for youth psychopathology (Smith and Dishion 2013). A multi-informant questionnaire-based assessment is also collected to complement the observational assessment. Beginning at around age 10, youths provide self-report data, and data are also collected from caregivers and from teachers when the child is school age. Third, the therapist engages the family in a collaborative feedback session based on the techniques and goals of motivational interviewing (Miller and Rollnick 2002). Feedback emphasizes parenting and family strengths, yet draws attention to possible areas of change. The ecological approach of the FCU ensures that feedback sessions are tailored to focus on parenting strengths and challenges within the family’s cultural context (Smith et al. 2014b).

The assessment results provide norm-based data that can be used to shape an individually tailored “menu” of subsequent intervention options presented to the family. These intervention options are based on an ecological parent management training perspective. As such, intervention options often include referrals to services outside the scope of the current service setting the FCU is embedded in. In the context of CMH, the menu of options would include potential targets for family-based intervention, which were indicated by the assessment data as areas in need of attention. Therapists practicing the FCU are encouraged to use modules from the Everyday Parenting curriculum (Dishion et al. 2011) to tailor a treatment plan to meet the unique needs of each family, as opposed to delivering a predetermined course of treatment.

The FCU has been previously tested in multiple randomized trials for families with youths ages 2–17. The results indicate that participation in the FCU is predictive of improvements in antisocial behaviors in early childhood (e.g., Dishion et al. 2008; Shaw et al. 2006), at school age (Dishion et al. 2014; Smith et al. 2014), and during the transition from middle school to high school (Connell et al. 2007; Dishion et al. 2002; Stormshak et al. 2005; Van Ryzin et al. 2012b). Improvements in youth outcomes have been frequently found to be either fully or partially mediated by intervention effects on family management variables, such as positive behavior support (Dishion et al. 2008), and family conflict (Smith et al. 2014b). Previous and ongoing studies also demonstrate successful implementation of the FCU in the public middle school context (Stormshak et al. 2005, 2010, 2011). Delivery in multiple service delivery systems contributes to the potential of significant reach of the FCU model. Reach is an important concept in implementation as it concerns the number of individuals who receive a program and engagement of individuals who are most in need of a service. Reach is essential for a program or policy to have a significant public health or population-level impact (Glasgow et al. 1999).

This Study

This trial was the first to implement the FCU in CMH under typical practice conditions and evaluate its effectiveness. Initial implementation trials of EBPs in community comprise two broad aims: (a) evaluate the success of embedding the EBP in a particular system of care, and (b) demonstrate that the EBP produces improved outcomes for families and agencies. To address both aims simultaneously, we conducted a hybrid effectiveness–implementation trial. This hybrid approach has been described and endorsed in studies of intervention models in community settings where external validity and factors related to implementation are the primary focus (Wells 1999). Curran et al. (2012) defined a hybrid trial as “one that takes a dual focus a priori in assessing clinical effectiveness and implementation” (p. 217). We designed this trial to serve this purpose with a primary focus on intervention effectiveness and secondary aim of gathering empirical information and observing factors related to implementation of the FCU—a Type 1 hybrid trial (Curran et al. 2012). Families with youths age 5–17 years seeking care for a variety of mental and behavioral health concerns were treated by therapists randomly assigned to deliver the FCU or provide treatment as usual (TAU). Our hypotheses and results related to implementation are organized in accordance with Proctor et al. (2009) taxonomy of outcomes salient to the early stages of implementation research, including adoption, acceptability, feasibility, and fidelity. Training providers to competently deliver EBPs is perhaps the greatest challenge to the field of implementation science (McHugh and Barlow 2010). Fidelity meeting or exceed minimal standards is a primary outcome indicating successful implementation (Proctor et al. 2009).

Interpretation of the results of the clinical effects in this study should occur within the context of comparative effectiveness research. Three meta-analytic studies are useful for interpreting our results: (a) Miller et al. (2008) reviewed 23 studies with 1,060 participants that compared two EBPs for youth disorders and found an overall Cohen’s d of .22 across all disorders; (b) Weisz et al. (2006) meta-analyzed 32 studies that compared an EBP to usual care for youth psychopathology (d = .30). Further, differences were negligible when the youths were seeking treatment or were referred and not recruited (Weisz et al. 2013); (c) Wampold et al. (2011) meta-analyzed the effects of 14 studies that compared EBPs for anxiety and depression in adults with TAU. EBPs were generally more effective when the comparison treatment was not a psychotherapeutic intervention (d = .50), but the effect size was a modest .33 and was not statistically significant when the comparison was psychotherapeutic. These reviews provide a range of expected effect sizes from trials comparing an EBP with CMH services for treatment-seeking families.

To provide the reader with an overview of the study’s design and aid in the interpretations of the findings, we applied the pragmatic–explanatory continuum indicator summary (PRECIS; Thorpe et al. 2009). The PRECIS tool was conceived to help study designers consider the characteristics of trials that are explanatory (efficacy) or pragmatic (effectiveness) in nature. We applied this tool post hoc to elucidate the standing of our trial on this continuum (see Fig. 1) in a fashion similar to that of Selby et al. (2012). Our description of the trial recounts how we arrived at the ratings presented in the figure and these characteristics are further expounded upon in our discussion of the findings.

Fig. 1
figure 1

Characteristics of the study on the explanatory–pragmatic continuum (PRECIS)

Full size image

We hypothesized that evidence of intervention fidelity and therapist acceptability would be evident, therapists would adopt the FCU, delivery would be feasible, and evidence of meaningful reach would be found. Second, we hypothesized that families in the FCU condition would indicate significant improvements in youth conduct problems and positive parenting, compared to families in the TAU condition, while reporting high levels of satisfaction with the model. Further, analysis of outcomes for intervention engagers would result in larger clinical benefits.

Methods

Recruitment and Randomization

Therapists and Agencies

Three CMH agencies (one with two locations) primarily serving children and families in Multnomah County, Oregon, were the implementation sites. These particular agencies were carefully selected based on the populations served, the catchment areas of each agency, and the source of funding. These agencies serve a high volume of ethnically, culturally, and economically diverse children and families from a large geographic area. Further, each agency enlists a philosophy of ecological support and parent involvement in services for families, which provides a proper comparison to the FCU. Last, these agencies are predominantly funded through clients in the CareOregon network, the state’s safety net managed care organization, of which nearly 93,000 (59 %) are under the age of 19 (based on 2011 figures retrieved from www.careoregon.org on January 8, 2014).

All therapists employed at the agencies and hired during 3-year period of active therapist enrollment period were approached and offered the opportunity to participate. Participation in the study was supported by the agencies’ administrations but not mandated. Among the 30 therapists employed in the agencies at the start of the study, 29 enrolled, one elected not to participate, and two consented to participate but left the agency before being randomized and provided no data. Fifteen more therapists were approached when they were hired; 13 consented, 2 elected not to participate. In total, forty independently licensed therapists with master’s degrees in counseling or marriage and family therapy consented to participate and were randomly assigned to either the FCU or TAU conditions (20 in each group). The participation rate was 93 %. Training in the FCU was offered to the therapists randomized to both conditions—at the end of the study for therapists assigned to the TAU condition. Randomization was conducted within agency to maintain balance and reduce agency-level variability in the outcomes. Fifteen, fourteen, and eleven therapists participated at the three sites respectively. Therapist training in the FCU was conducted periodically as new therapists were enrolled in the study. The initial training comprised 13 therapists and three additional trainings were conducted with 3, 2, and 2 therapists over a 3-year period. Therapists had an average of 2.85 years (SD = 2.45, range 0–10) of clinical experience and an average of 1.20 years (SD = 1.38, range 0–5) of employment at their current agency.

Participants

The flow of families through the study is detailed in Fig. 2. Therapists in the FCU and TAU conditions approached families with children age 5–17 years for participation in the study when families sought services at the agency. The study was not advertised nor was recruitment of families conducted outside of the agencies. Beyond age, youths with severe developmental disabilities were to be excluded from participation. Therefore, families sought services for a wide range of behavioral and emotional issues. At the intake appointment, eligible families received a brochure describing the study procedures and the families interested in participating received contact information for the study personnel. Study representatives then met with interested families to complete consent/assent procedures and deliver the pretreatment questionnaires. Randomization at the therapist level resulted in 51 families assigned to the FCU and 31 to TAU. Figure 1 shows that participation and treatment completion rates were similar across conditions. Therapists recorded the reasons enrolled families did not complete the study: did not return to the agency for the first session (FCU = 10, TAU = 7); uncomfortable with videotaping (2, 1); relocated (2, 2); other (5, 2). Reasons given for not completing the study did not differ by condition.

Fig. 2
figure 2

Recruitment, randomization, and flow of participants

Full size image

The average age of the child was 11.6 years (SD = 2.6), and 49 % were female. The average age of the primary caregiver was 40.1 (SD = 9.8) years. Primary caregivers were predominantly the biological mothers (78 %) or fathers (12 %), with adoptive mothers (4 %), foster mothers (4 %), and grandmothers (1 %) represented. Caregivers were single (37 %), divorced (22 %), separated (11 %), married (15 %) or living together (13 %). The majority of primary caregivers reported that the youth’s other biological parent did not live in the same household as the child (87 %). The ethnic background of the child, as reported by the primary caregiver, was representative of the Pacific Northwest: European American (65 %), African American (16 %), Hispanic-Latino (3 %), Native American/American Indian/Alaska Native (3 %), Asian/Asian American (1 %), and multiple ethnicities (11 %). The average annual income before taxes was $16,884 US (SD = 942), which is below the federal poverty line for families of two or more, according to 2010 guidelines—the middle year of this study (United States Department of Health and Human Services 2010).

Procedures

Adaptation

The FCU was designed for dissemination and implementation in community service settings with high rates of contact with youths and families, such as schools, primary care settings, and CMH. The FCU is individually tailored and brief compared with other evidence-supported treatment models, which makes it more easily transportable. The assessment process is streamlined and can be completed and effectively used by providers with a master’s or bachelor’s degree in human services (e.g., psychology, social work). We used a collaborative approach between FCU developers, agency leadership, and therapists when adapting the intervention for implementation in these particular agencies. The local adaptations described in the remainder of this section were based on pilot testing of the model in these agencies, collaboration with the providers and agency administration to identify approaches that would ensure successful delivery of the core components, and consultation with the intervention developers to guarantee that the active ingredients of the FCU remained intact so as to support optimal fidelity and reduce voltage drop.

In our original efficacy trials, a large, comprehensive assessment battery was included in the FCU model. For community implementation, we substantially reduced this assessment battery by empirically refining our measurement scales from previous research and relying on brief, published versions of measures that would provide reliable assessments of clinically relevant constructs that could also be used to test for intervention effects. The result was an assessment battery for parents and youths that could be completed in a waiting room setting prior to a first appointment. The videotaped ecological assessment that is an intrinsic element of the FCU was retained for the CMH setting and delivered solely in the clinic.

Therapist Training

Therapists assigned to the FCU condition attended a 2-day training workshop before enrolling families in the study. The same training was provided to the TAU therapists at the end of the study period. Although the therapists had no prior training in the FCU model, nearly all indicated during training that they had experience and training in parent management interventions. Following the 2-day training, therapists were supervised by a licensed psychologist with expertise in the FCU with the families initially enrolled in the study. The supervisor was available to the study therapists throughout the project period to answer questions specific to delivery of the model. As new therapists were hired at the agencies, those consenting to participate in the trial and randomized to the FCU were trained in the model.

Assessment Procedures

Before randomization, therapists employed at the agencies were assessed on a number of questionnaires. They were assessed again at the end of the study period or when they withdrew from the study (e.g., change of employment). At pre- and posttreatment, youths and caregivers completed questionnaires about various domains of youth adjustment and family management practices. Pretreatment assessment occurred at the recruitment appointment, and posttreatment assessment was completed 6 months after the pretreatment assessment, whether or not the family was currently receiving services or had terminated treatment (all but two of the families had terminated at 6 months). After treatment commenced, members of the research team contacted caregivers via phone to complete a follow-up assessment, which occurred an average of 7.5 months after postassessment. Families with children between ages 5 and 8 years (n = 4) completed caregiver questionnaires only. Treatment sessions in both conditions were videotaped. Parents were paid for participation in the assessment portion of the study but were responsible for the costs of the services in the agencies.

Intervention

Families participating in this study received either TAU or the FCU prior to additional services. The types of services delivered to families in the TAU condition were not prescribed so as to reflect typical services provided in CMH agencies. However, the agencies had been specifically selected for their family-based approach to youth mental health, which provides a more appropriate comparison condition to test effectiveness of the FCU. In the FCU condition, after completion of the model, therapists were encouraged to select interventions on the basis of the ecological assessment data and the family’s preferences on the menu of intervention options presented in the feedback session. Family intervention content and parent management strategies are described in the Everyday Parenting curriculum (Dishion et al. 2011).

Measures

Intervention Fidelity

Two raters, one a graduate psychology trainee and one an advanced undergraduate psychology student were trained in the fidelity of implementation rating system for the FCU, called the COACH (Dishion et al. 2014). The COACH assesses five dimensions of fidelity to the FCU: Conceptual accuracy and adherence; Observant and responsive to client needs; Actively structures sessions; Careful and appropriate teaching; Hope and motivation are generated. Scores derived from the COACH have been found to be reliable and predictive of change in parenting practices and child outcomes (Smith et al. 2013b). Raters viewed a videotape of the entire FCU assessment feedback session after the trial was completed. Each of the five dimensions of the COACH is rated on a 1–9 Likert-type scale and then a composite score (mean) is computed with higher scores indicating better fidelity to the FCU. Twenty percent of the sessions were double-coded and a one-way random effects model intraclass correlation coefficient (ICC; Shrout and Fleiss 1979) was calculated: According to conventional interpretative guidelines, the reliability of the COACH composite score was in the good range (ICC = .73).

Acceptability

Therapists’ global acceptability of the FCU was assessed using the Evidence-Based Practice Attitudes Scale (EBPAS; Aarons 2004). The EBPAS comprises four subscales pertaining to different aspects of EBP implementation: Appeal, Requirements, Openness, Divergence. A total score can be computed to assess global attitudes toward adoption of EBPs with higher scores indicating greater acceptability and more favorable attitudes toward adoption. This measure had adequate internal consistencies (pre, α = .72; post, α = .79) that are similar to those obtained for this scale in the original study (α = .77; from Aarons 2004).

Conduct Problems

We used the parent and youth self-report versions of the conduct problems subscale of the Strengths and Difficulties Questionnaire (Goodman 1997; Goodman et al. 1998) comprising 5 items pertaining to antisocial behaviors such as fighting, lying, stealing, noncompliance, and losing one’s temper. Items were rated on a 3-point scale referring to how well the statement applied to the youth (e.g., 0 = not true, 1 = somewhat true, 2 = certainly true) with higher scores indicative of greater conduct problems. Interrater reliabilities (Cronbach’s α) were adequate at each wave for youth (pre: .65, post: .59, follow-up: .66) and caregiver reports (.80, .79, .78).

Effective and Positive Parenting

A composite measure of effective and positive parenting behaviors was created using three caregiver-reported constructs: positive behavior support (PBS; 7 items), positive proactive parenting (PPP; 7 items), and negative parenting behaviors (NPB; reverse scored; 6 items). Items for these subscales were drawn from validated parenting self-report questionnaires, such as the Parenting Young Children survey (McEachern et al. 2012), which has demonstrated convergent and predictive validity. Internal consistencies were acceptable for each scale ranging from .54 to .81 at pre, and from .58 to .86 at post. Each of these subscales was assessed on a 5-point scale ranging from 0 (never) to 4 (very often). Higher scores indicate endorsement of using more effective and positive parenting skills. The three subscales were significantly intercorrelated at each wave of assessment and were combined using a principal axis factor analysis, which resulted in a one-factor solution with loadings ranging from .62 to .86 at pre- and .66 to .90 at posttreatmeant. The resulting factor scores were then used in subsequent analyses. PBS was not assessed in the follow-up assessments; thus, a mean score of the PPP and NPB subscales was examined from pretreatment to follow-up.

Family Satisfaction with Services

A nine-item client satisfaction survey was created to evaluate caregivers’ satisfaction with the services they received (included as an Appendix). This scale was developed by adapting items from well-validated questionnaires, such as the Client Satisfaction Questionnaire (Nguyen et al. 1983), to be specific to parent training interventions. Items such as My therapist helped me identify my strengths as a parent and My therapist respected me were rated on a Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). A total satisfaction score was computed by averaging the 9 items. Higher scores indicated higher satisfaction. The scale had excellent internal consistency (α = .95).

Data Analysis

Prior to evaluating group differences on the implementation and clinical outcomes we conducted design effect analyses on all outcome variables to determine the amount of variance corresponding to the nesting within agencies, which can affect estimates of the standard error and require a multilevel analytic approach when significant. Muthén and Satorra (1995) specify that design effects less than or equal to 2.0 are indicative of nonsignificant variation. Scores from the COACH rating system were evaluated based on a predetermined level of fidelity established by the intervention developers and validated in a previous study (Smith et al. 2013b). The EBPAS was used to assess therapist acceptability of the FCU. We used a one-way ANOVA to evaluate differences between the FCU and TAU conditions at pre and postassessment. The number of FCUs completed by study therapists was used to determine adoption and reach of the intervention. Therapist reports during semi-structured exit interviews are used to interpret the results; although, rigorous qualitative analyses were not conducted.

We used two complementary analytic approaches to examine clinical effectiveness of the FCU. First, we used an intention to treat (ITT) approach to examine group differences, including all participants who provided at least pretreatment data at the time of enrollment (FCU: 43; TAU: 28). Second, we conducted the same analyses on a subset of families, termed engagers, to determine whether intervention effects differed between conditions for those families who received the entire FCU and those families who received a comparable three-session TAU. We controlled for a number of covariates in our intervention outcome analyses, including child age and ethnicity, the total number of treatment sessions, and children’s initial conduct problems in analyses of parenting outcomes. Given noted differences in the developmental trajectories and levels of conduct problems between boys and girls (McFadyen-Ketchum et al. 1996), gender was examined as a moderator of intervention effects.

Results

Implementation Outcomes

Intervention Fidelity

Of the 33 FCUs completed, 32 videotapes from 13 therapists were available for coding. The average fidelity score was 4.46. This score is just below the a priori benchmark (5.00) for satisfactory fidelity. A closer examination of the ratings revealed a within-therapist composite score average range from 2.67 to 5.50 with 10 of the 13 (77 %) therapists achieving an average score greater than 5.00.

Acceptability, Adoption, and Feasibility

Therapists completed the EBPAS at the beginning and at the end of the study. There were no statistically significant differences (one-way ANOVA) between therapists assigned to the FCU or TAU conditions on the total EBPAS score at pre- (FCU = 3.09 [.36]; TAU = 2.99 [.46]) or postassessment (FCU = 2.83 [.58]; TAU = 2.75 [.48]). FCU completion and family engagement rates provide evidence of adoption and feasibility because provision of the service was at the discretion of the therapists and completion rates were comparable to the TAU condition (3 sessions FCU compared to 3 sessions of TAU). Thirteen of the 20 therapists completed at least one FCU and 33 were completed in total with a modal number of 2 and a range of 1–4. The 65 % completion rate is identical across conditions.

Intervention Effects

Preliminary Analyses

Intercorrelations and descriptive statistics are presented in Tables 1, 2, 3 and 4. Tables 1 and 2 contain caregiver and youth report data, respectively, for the full sample. Tables 3 and 4 contain caregiver and youth report data, respectively, for the subsample of engager families. Design effect analyses of quantitative data resulted in an average effect across implementation and intervention outcomes of .58 and 1.62, respectively. These effects are within the acceptable range (≤2.0; Muthén and Satorra 1995) and indicate that the amount of variance corresponding to the nesting within agencies did not significantly affect estimates of the standard error. Preliminary analyses revealed two significant differences between the families assigned to the two groups. A one-way ANOVA revealed that children assigned to the FCU condition were somewhat older, F(1) = 4.430, p = .039; FCU: M = 12.1, SD = 2.6, TAU: M = 10.9, SD = 2.5, and had somewhat higher levels of pretreatment youth-reported conduct problems, F(1) = 3.715, p = .06; FCU: M = 4.25, SD = 2.25, TAU: M = 3.25, SD = 2.22.

Table 1 Intercorrelations between study variables and descriptive statistics of caregiver reports—full sample
Full size table
Table 2 Intercorrelations between study variables and descriptive statistics of youth reports—full sample
Full size table
Table 3 Intercorrelations between study variables and descriptive statistics of caregiver reports—engagers
Full size table
Table 4 Intercorrelations between study variables and descriptive statistics of youth reports—engagers
Full size table

To include the entire randomized sample in the analyses, we imputed data using the expectation–maximization algorithm (Dempster et al. 1977), a maximum likelihood estimation method that has been shown to provide unbiased estimates when data are missing completely at random (MCAR). There was some degree of missing data in our sample (see Tables 1 and 2 for valid Ns of each variable), but the data were found to be MCAR: Little’s (1988) MCAR test, χ2(79) = 70.04, ns. Thus, the missing data did not introduce bias into the analyses.

Engagement

Engagement rates in the current study were consistent with recent estimates of engagement in CMH by urban youths, which tend to be low (McKay et al. 2005). Specifically, 18 families enrolled in the study, provided pretreatment assessment data, yet failed to complete the FCU or the equivalent three sessions in the TAU condition. We refer to these families as nonengagers. Engagers were considered those families that completed the FCU or the comparable three sessions of TAU, resulting in a sample of 53 families (FCU = 33; TAU = 20). Characteristics of the nonengager and engager families were compared using a one-way ANOVA. The only significant difference found was in the total number of sessions received among the engager families (inclusive of the FCU), F(1) = 6.422, p = .01; FCU: M = 7.64, SD = 6.51, TAU: M = 5.59, SD = 4.91. Seventeen therapists from each condition were associated with the engager family subsample.

Intervention Effects

Intention to Treat

A repeated measures ANOVA analysis revealed that the FCU significantly outperformed TAU in terms of reducing conduct problems from pre- to posttreatment reported by the youth, F(1) = 7.134, p = .01, d = .33 (Means: FCU = 1.95; TAU = 2.92), but not by the caregiver, F(1) = 2.055, p = .16, d = .21 (2.70; 3.26). Analyses of follow-up effects revealed no significant differences: youth report, F(1) = .001, p = .97, d = .02 (1.12; 1.60); caregiver report, F(1) = .1.424, p = .24, d = .29 (2.06; 2.28). Analyses of parenting practices from pre- to posttreatment, F(1) = 1.641, p = .21, d = .25 (3.06; 2.81) and pre to follow-up F(1) = .676, p = .41, d = .14 (3.15; 3.02) revealed no statistically significant group differences.

Intervention Engagers

Pre- to posttreatment change in youth-reported conduct problems was large and statistically significant in favor of the FCU, F(1) = 7.662, p = .01, d = .50 (2.00; 2.59). The caregiver report effect size was medium and also significant, F(1) = 5.603, p = .02, d = .36 (2.90; 2.99). Analyses of follow-up effects were not significant: youth, F(1) = 1.825, p = .18, d = .51 (1.01; 1.72); caregiver, F(1) = .622, p = .43, d = .21 (1.81; 2.55). Group differences in the parenting construct were not statistically significant: pre- to posttreatment, F(1) = 1.280, p = .26, d = .28 (3.04; 2.79) or pre to follow-up F(1) = .177, p = .41, d = .08 (3.15; 3.07).

Satisfaction with Services

No significant differences emerged between the intervention conditions regarding family satisfaction with services. Both groups were highly satisfied with the services they received: FCU, 4.29 (.87); TAU, 4.52 (.52).

Discussion

The transfer of efficacious interventions to community-based service delivery systems is a significant challenge that is imperative to improving mental health outcomes. This is particularly germane to services for children and families due to the high rates of children with impairing mental health conditions and the low quality of services often provided by community agencies (Department of Health and Human Services 2000). This study reports the results of an effectiveness–implementation hybrid trial of the FCU in three community mental health agencies serving youth and families in the Pacific Northwest. This study is the first to evaluate the effectiveness of this intervention model while also providing evidence of successful implementation in a community service delivery setting.

Consistent with our hypotheses, the results indicated feasible delivery of the FCU with fidelity, acceptability of the model, and successful adoption. Concerning fidelity, therapists trained to deliver the FCU were able to achieve adequate levels. The fidelity scores obtained through observational coding of videotaped FCU sessions are meaningful indicators of successful delivery, particularly since they have been found to be predictive of improvements in parenting and child behavior problems (Smith et al. 2013b). Given the level of training and supervision provided to these therapists during the trial, the scores also suggest that it is feasible to train providers to deliver the FCU in busy CMH agencies and still achieve fidelity. Therapists also indicated high acceptability of the model on the EBPAS. The number of therapists completing FCUs provides limited evidence of adoption and reach of the intervention as delivery of the model was at the discretion of the therapist. Further, results indicate that most therapists completed more than one FCU. On the other hand, only 13 of the 20 therapists completed a FCU, which evidence suggests can be mainly credited to client attrition and not the intervention approach itself. Therapist turnover might have also been a factor in rates of FCU completion and it presented a challenge for FCU trainers, as it necessitated ongoing training. However, this is likely to be a common situation in CMH agencies serving youth and families where annual turnover rates can exceed 50 % (Aarons and Sawitzky 2006). Unfortunately, therapist turnover was not rigorously tracked in this study.

We attribute our promising findings regarding adoption, acceptability, feasibility, and fidelity to our collaborative approach to adapting the model for these settings. This is a critical step in the implementation of an EBP into a new setting or system of care. For example, we successfully adapted the model to fit within the 50-min session constraints of the agencies, allowing time for billing to managed care, while retaining core intervention components. This is an important aspect of the cost of delivery, because the FCU reaps the same reimbursement as TAU, and of adoption, reach, and potential sustainability, which needs to be studied further.

The PRECIS tool (Fig. 1) provides a framework to evaluate the trial design and interpret the results of this study. In particular to practitioner adherence to and flexibility of the experimental intervention (delivery of the FCU), we valued practitioner adherence (fidelity) to the FCU but we did not assess it during the trial, nor did we provide remediation when it failed to reach minimum levels. Thus, we rated this category in the middle of the continuum. In terms of the flexibility of the intervention and of its delivery, adaptations were systematically devised prior to participant enrollment in the study and local adaptations were permitted to some extent. Thus, we rated delivery of the FCU as somewhat flexible. Because this was the first effectiveness trial of the FCU, we sought to have confidence that intervention effects would be attributed to the model and not to other factors.

Intervention effects found in this study are promising and generally consistent with our hypothesis that families in the FCU condition would report greater improvements in child conduct problems and parenting practices. In the full sample, youth-reported conduct problems were significantly reduced in the FCU condition compared to TAU. Although it did not reach statistical significance, caregiver-reported conduct problems were also larger for families in the FCU condition. Analyses of these outcomes for intervention engagers indicated larger effects in favor of the FCU with both youth and caregiver-reported conduct problems found to be statistically significant. These findings are consistent with those of previous trials of the FCU with youths in early childhood and mid adolescence that found significant reductions in conduct problems and oppositional behaviors compared to a control condition (e.g., Dishion et al. 2008; Smith et al. 2014b). However, in contrast to previous studies that found trajectories of significant divergence between the FCU and control groups over time (e.g., Van Ryzin et al. 2012a), analyses of follow-up effects in this trial revealed no significant differences for the full sample of the engager subsample analyses.

Research on the FCU has consistently found intervention effects on age-appropriate parenting variables. In this study, caregivers in the FCU condition did not report significantly improved parenting from pre- to posttreatment or pre to follow-up. However, the magnitude of the effects suggests that caregivers receiving the FCU were doing better in this domain. Thus, we view the modest effects in favor of the FCU as promising evidence in need of further investigation.

The effect sizes found in this study are consistent with Wampold et al. (2011) benchmarks for EBPs compared with TAU and surpass the average effect when comparing the EBP with a psychotherapeutic intervention. They are also generally better than the average effect found when comparing two EBPs (Miller et al. 2008) or an EBP with usual care (Weisz et al. 2006, 2013). Additionally, completion of the FCU was predictive of attending more total sessions of treatment (inclusive of the FCU) in the agency. This is a positive indicator in support of using the FCU to engage families in this service setting, given the extreme underutilization of community services by families. In part because the FCU was developed as a prevention model in which the intervention is offered to families who may or may not need services, therapists explicitly target motivation to change behavior and engage in additional services. The high satisfaction with the FCU reported by families likely also contributed to higher rates of participation in services.

One additional consideration emanating from this study is the issue of engaging families in CMH services. Attrition rates are traditionally high in this setting and intervention models that specifically target engagement in services have the potential to address this issue. The FCU was specifically designed to increase engagement by addressing families’ differing levels of motivation to change (Dishion and Stormshak 2007). In this study, families had similar levels of overall attrition in both conditions. However, those families that completed the FCU attended significantly more intervention sessions (inclusive of the FCU) compared to families who completed a comparable three sessions of TAU. Elucidating the precise reasons for this difference are beyond the scope of this study but are nonetheless a promising indicator of the FCU’s potential to increase engagement in CMH.

Again considering the trial’s clinical effectiveness characteristics described with the PRECIS tool, the assessment of clinical outcomes was somewhat less pragmatic in that primary indicators of effectiveness were known consequences of the FCU. The intensity of the follow-up assessment strategy (self-report) was more pragmatic than our team’s typical use of micro and macro-level observational assessment of caregiver–child interactions. The primary analysis of intervention outcomes is a mix of methods akin to a pragmatic trial (ITT) and an explanatory trial (engagers). The comparison intervention, expertise of the providers, follow-up assessment intensity, inclusive participant eligibility criteria, and participant compliance, are all pragmatic characteristics of the trial and add to the external validity of the findings.

Limitations and Caveats

Although the results of this randomized effectiveness–implementation hybrid trial are promising, a few limitations and caveats must be mentioned. First, the costs associated with training providers and then delivering the FCU in CMH agencies were not assessed. However, we successfully adapted the model to fit within the agencies’ 50-min session timeframe, and we found that participation in the FCU increased service utilization; both factors have an impact on cost to the system. This is a crucial area for future research. As is the case with many grant-funded implementation trials, indicators of sustainability were also lacking. Next, randomization occurred at the level of the therapist. Even though participating families in each condition were provided with the same compensation for completing the assessment, therapists in the FCU condition may have been more motivated to recruit families because of a personal desire to deliver a newly learned intervention. These factors might account for the difference in the number of families between the conditions and potentially some different family characteristics that were not measured (i.e., motivation). The unequal group sizes are also likely a product of the naturally occurring assignment of families to a therapist in the agency. Lack of a design effect suggests that the results of the analyses are statistically sound, but the considerations mentioned are still relevant. Reliance on caregiver and youth report of the primary clinical outcomes is a third limitation of this study as they could be prone to demand characteristics. However, our inclusion of both caregiver and youth report is a strength. Fourth, the size of the sample limits power to detect statistically significant group differences in terms of the moderate effect sizes expected in this type of study. Yet, the sample is an ecologically valid representation of families seeking community services and encompasses a wide age range of youths. The inclusivity of our criteria for participation in the study strengthens the generalizability of the findings but could have also affected our ability to detect significant effects. Specificity regarding the FCU model, compared to EBPs in general, could be achieved by adapting the EBPAS in future implementation studies. We did not rigorously track therapist turnover, which has been associated with implementation of EBPs (Aarons et al. 2009) and could affect the sustainment of the FCU in similar settings. Last, additional evaluation of the relative contributions of the components and techniques of the FCU could be explored to assist in adaptation. These elements include motivational interviewing and video feedback procedures (see Smith et al. 2013a).

Conclusions

This study provides promising evidence that the FCU can be effectively implemented in CMH agencies and doing so improves the clinical outcomes experienced by families. Therapists delivering the FCU delivered the FCU with acceptable levels of fidelity, reported enthusiasm about the model, and showed that it could be feasibly implemented with minimal alteration to typical service delivery procedures. The magnitude of the intervention effects on youth conduct problems were commensurate with previous benchmarks obtained in comparative effectiveness research. Comparison of the FCU to a community treatment that is also family-based renders the observed effects even more noteworthy. Families that completed a FCU attended more total sessions of treatment, which was likely a product of the motivational and collaborative aspects of the model that promote caregiver engagement. The FCU has also been found to be culturally relevant (Smith et al. 2014b), which is an important feature of the model for scale-up efforts in community settings that serve diverse families. In conclusion, multiple indicators of successful implementation and evidence that the intervention was effective suggest that the FCU is a viable model for scale-up in CMH service delivery systems. The costs of implementation and long-term sustainability ought to be considered in future studies.