Introduction

Wraparound is a family-driven, youth guided, team-based process for planning and implementing services and supports (Miles et al. 2006). The National Wraparound Initiative (NWI) has identified ten elements of wraparound (i.e., family voice and choice, team based, natural supports, collaborative, community based, culturally competent, individualized, strengths based, persistence and outcomes based) and four phases through which teams consisting of the identified youth, his/her parents or caregivers, family members, community members, mental health professionals, and others are expected to move as they develop and implement a single plan of care. The plan of care includes the services and supports necessary to build on the strengths of the youth and his/her family and addresses the complex needs of the youth involved in the wraparound process.

Emerging evidence supports the effectiveness of wraparound for youth who have needs in multiple life domains (e.g., home, school and community). Nine controlled studies of wraparound (see Bruns and Suter 2010; Suter and Bruns 2009) found improved outcomes for youth in wraparound compared to similar youth in other programs, with effect sizes similar to those found in studies of other evidence based interventions implemented in real world practice (Suter and Bruns 2009). However, only one of the nine studies considered the relationship between wraparound fidelity and outcomes (Bruns et al. 2006). Research on other evidence-based practices has repeatedly found that fidelity to the practice model is vital to outcomes (e.g., Henggeler et al. 1997; McGrew et al. 1994; Walton 2006). Additional research on wraparound that includes a measure of fidelity as well as further research on the factors that predict successful outcomes for youth involved in the wraparound process are needed (Bruns et al. 2010; Cox et al. 2010).

Successful implementation of any practice model depends on several key community and organizational factors, such as organizational support, staff selection, training, coaching, and system-level partnerships (Glisson and Schoenwald 2005; Lehman et al. 2002; Metz et al. 2007). A few authors have discussed similar factors that are associated with effective wraparound implementation (e.g., Bertram et al. 2010; Walker 2008; Walker et al. 2003). For example, Walker et al. (2003) identified several necessary conditions to support the implementation of high-quality individualized planning, such as wraparound, at the system, organization and team levels. Studies examining the relationship between these necessary conditions and wraparound fidelity and/or outcomes are only just emerging (Bruns et al. 2006; Stephens et al. 2004; Walker and Sanders 2010). Understanding these relationships is essential to reforming child-serving systems to effectively address the needs of youth and families (Bruns et al. 2006).

The purpose of our study was to examine the relationships among the implementation of necessary support conditions for wraparound, wraparound fidelity and youth outcomes. The local evaluation of a grant authorized by the Centers for Medicare and Medicaid Services to demonstrate that intensive community based services could be a viable alternative to psychiatric residential treatment facilities (PRTF; George 2010) provided the data necessary to explore these relationships. A site assessment was used to measure the stage of development (Rogers 2003; Fixsen et al. 2005; Glisson and Schoenwald 2005) that each community in Indiana achieved in terms of implementing the necessary conditions for wraparound (Walker et al. 2003). Wraparound fidelity was measured and examined in relationship to a community’s stage of development and to outcomes observed for youth and families. Outcomes for youth were measured with the Child and Adolescent Needs and Strengths (CANS; Lyons 2009). We hypothesized that communities in later stages would have higher fidelity to wraparound than communities at earlier stages and that youth receiving high fidelity wraparound would have better outcomes than youth not receiving wraparound or experiencing wraparound with lower levels of fidelity.

Method

Participants

Participants in this study included youth who received intensive community-based services through the Community Alternatives to Psychiatric Residential Treatment Facilities (CA-PRTF) Medicaid Demonstration grant in Indiana between January 2008 and June 2010. Youth served have severe emotional and behavioral health needs as well as functional impairments that might otherwise be treated in a PRTF. Study participants were limited to youth for whom site assessment, wraparound fidelity, and outcome data were available.

Dependent Variables and Procedures

Stage of Development

The Strengths-Based Site Assessment (Effland 2009) measures the extent to which the following organizational and system-level support conditions for effective wraparound are in place within each community (Walker et al. 2003): collaboration and partnerships (representation by child-serving agencies and families); capacity building and staffing (project staff, strength-based supervision); acquiring services and supports (funding sources, Medicaid billing, flexible funds); accountability (outcome measurement and reporting); and family involvement.

During regular visits to Indiana communities, coaches employed by a technical assistance and training center dedicated to supporting the implementation of wraparound (Choices 2010) rate the level of development in each of five support conditions. The ratings are based on change theories that identify five stages through which individuals (Prochaska et al. 1994) and organizations (Rogers 2003) move as they contemplate and implement change. The five stages were consolidated into four stages for this study because of the small number of sites rated in the first two stages (i.e., precontemplation and contemplation). An overall stage of development was determined for each community by averaging the ratings assigned across support conditions and identifying the closest stage.

Wraparound Fidelity

The Wraparound Fidelity Index 4.0 (WFI; Bruns et al. 2007) measures the extent to which communities apply the ten wraparound principles identified adopted by the NWI (Miles et al. 2006) in services to youth and families The ten principles of wraparound became the ten subscales of the WFI with four questions reflect each principle. Table 1 describes each wraparound principle, WFI domain.

Table 1 Wraparound Fidelity Index subscales: principles of wraparound (Bruns et al. 2008)
Full size table

The WFI 4.0 and earlier versions of the tool are both reliable and valid (Bruns et al. 2004, 2007). WFI interviewers complete a comprehensive training protocol that includes certification to insure that WFI ratings are made reliably. The WFI was administered through telephone interviews with wraparound facilitators at multiple times (i.e., approximately 3 months after grant services began, 1 year after the beginning of grant services and within 30 days of discharge) during a youth’s involvement in the grant. In order to attain sufficient sample sizes, the most recent interview completed for each youth was included in the analysis. As a result, some youth had been discharged from grant services, while others had been involved for 12 months or less.

Interviewers assigned a score to each question on a scale of 0 (low fidelity) to 2 (high fidelity). WFI scores were calculated by summing across items rated on the WFI and dividing by the highest possible fidelity score. For descriptive analyses, total fidelity scores from facilitators were assigned to one of four categories based on established fidelity benchmarks (Bruns et al. 2005): low wraparound fidelity (scores below 65%); borderline (scores between 65 and 74%); adequate (scores between 75 and 84%); and high (scores of 85% and above). For predictive analyses, raw fidelity scores were used.

Outcomes

In 2007, Indiana adopted the CANS as the outcome management tool for the behavioral health service system. The CANS has been found to be reliable and valid (Lyons 2009). An online training and certification process allows individuals throughout Indiana to become certified users of the CANS. The average certification reliability for Indiana’s 4300 certified CANS users is .79, consistent with Anderson et al. (2003). Local support for rating and using the CANS in the treatment planning process is provided by SuperUsers who receive advanced training on the CANS. All wraparound facilitators have become SuperUsers with a minimal certification reliability of .75.

Indiana’s comprehensive multisystem CANS tool includes the following dimensions: child strengths, functioning, behavioral/emotional needs, risk behaviors, and caregiver strengths and needs. CANS items are rated using a four-point scale (0–3) to indicate whether identified needs require action and whether strengths can be used in treatment planning. CANS assessments are completed several times during a specific episode of service: baseline, every 6 months, and at discharge or when a change in the level of care is being considered. The CANS tool includes six dimensions (functioning needs, behavioral/emotional needs, risk behavioral needs, youth strengths, caregivers’ strengths and needs and acculturation). Ratings from three CANS dimensions (i.e., functioning, behavioral/emotional needs and risk behaviors) were averaged to create a youth needs domain. This scoring strategy allows researchers to use a single score to represent the functional status of a youth (Lyons 2009). The strategy is supported by Doucette’s (2007) scaling of the comprehensive version of the CANS. Functional improvement was then defined as any decrease in youth needs from the baseline assessment to the most recent CANS assessment. For this study, only change in youth needs was measured.

Relationship Between Stage of Development and Wraparound Fidelity

To better understand the relationship between necessary organizational and system-level support conditions and fidelity to the wraparound, the data was categorized into four stages of development (i.e., precontemplation/contemplation, preparation, action and maintenance) and four levels of wraparound fidelity (not wraparound, borderline, adequate and high fidelity wraparound). An ANOVA was used to determine if significant associations existed.

Predicting Improvement

To determine whether variables that predict positive outcomes for youth could be identified, two hierarchical regression models were run using a forced entry method in SPSS (2010). In the first model, the following independent variables were included as possible predictors of change in youth needs: total facilitator WFI fidelity score, stage of development, baseline functioning from CANS domains (behavioral health, functioning, risks, strengths and caregiver strengths and needs), age, gender, race (Black/African American, Native American, multi-racial) and ethnicity (Hispanic). In the second regression model (Model 2), the high level, aggregated predictors were deconstructed to better understand what components of the service delivery model and which behavioral health needs are related to outcomes. Baseline behavioral health CANS items ratings were substituted for baseline behavioral health domain scores and facilitators’ ten WFI subscales replaced the facilitator Total WFI scores. Specifically, Step 1 began with the ten elements of wraparound. Step 2 included baseline behavioral health needs from the CANS (psychosis, impulsivity, depression, anxiety, oppositional, conduct, adjustment to trauma, anger control, substance use and eating disorder). Step 3 retained other baseline CANS domains (functioning, risks, strengths and caregiver) which were subsequently deconstructed. From youth demographic characteristics, Step 4 retained the predictive racial item. For each model, non-significant factors were removed.

Results

Participants

This study includes 515 youth who received services through the CA-PRTF grant between Jan 2008 and June 2010. Most of these youth were male (72.6%); 77.3% were Caucasian, 16.1% were African American, 1.2% were Native American, 5.4% identified themselves as being of two or more races; and 3.7% were Hispanic. Youth ranged from 6 to 20 years old (M = 13.67, SD = 3.1).

Stage of Development

TA Center coaches assessed 65 communities between December 2008 and May 2010 using the Strengths-Based Site Assessment (Effland 2009). The majority of communities were evaluated as being at Stages 2 (28.8%) and 3 (34.8%) at their most recent assessment; 15.2% of sites were evaluated as being at Stage 1 and 19.7% in Stage 4.

Wraparound Fidelity

The average overall facilitator WFI score was 80.7 (SD = .10). Of the 515 participants, 41.3% received high fidelity wraparound, 32.6% were receiving an adequate level of wraparound, 15.5% were receiving wraparound at a borderline level of fidelity, and 7.1% did not receive wraparound.

Stage of Development and Wraparound Fidelity

WFI interviews were completed for at least one youth in 46 of the 65 communities that also had a completed site assessment. Specifically, 56 youth were interviewed in 7 communities in Stage 1, 63 youth were interviewed in 13 communities in Stage 2, 309 youth were interviewed in 17 communities in Stage 3, and 79 youth were interviewed in 15 communities in Stage 4.

To examine the relationship between a community’s stage of development and the level of wraparound fidelity, the percent of youth that received wraparound at each of the four fidelity levels was computed for each of the four stages of implementation. The results indicate that a higher percent of respondents (46.4%) reported receiving high fidelity wraparound in communities in Stage 1 than in communities at the remaining three stages (i.e., 25.4% at Stage 2, 49.7% at Stage 3 and 38.1% at Stage 4). A higher percent of respondents reported an adequate level of wraparound fidelity in Stage 4 (34.0%) than in Stage 1 (32.1%), Stage 2 (30.2%) or Stage 3 (32.4%). Approximately 23.8% of respondents in Stage 2 communities reported a borderline level of fidelity, compared to 16.1% of respondents in Stage 1 communities, 12.1% in Stage 3 communities and 15.3% in Stage 4 communities. The percent of respondents that indicated that they did not receive wraparound was highest in Stage 2 communities (14.3%), followed by Stage 4 (9.3%), Stage 1 (3.6%) and Stage 3 (2.9%).

There was a significant association between stage of development and the level of wraparound fidelity as reported by facilitators. Figure 1 graphically describes the frequency of different levels of wraparound fidelity across stages of site development. A significant association was found between the with community stage of development and levels of fidelity, F(3, 5.72), p < .01, w = .2. There is a significant cubic trend, F(3, 506) = 21.449, p < .001, w = .20. Figure 2 depicts the directional change in mean levels of fidelity among sites. Specifically, planned contrasts revealed a significant relationship between earlier (contemplation and preparation) and later (action and maintenance) stages of site development, t(156.57) = 2.28, p < .05, r = .168.

Fig. 1
figure 1

Relationship between site stage of development and level of wraparound fidelity

Full size image
Fig. 2
figure 2

Mean wraparound fidelity score for sites at each stage of development

Full size image

Outcomes

Change in youth needs is operationally defined as the difference between youth needs (i.e., ratings across the behavioral/emotional, functioning and risks domains) at the most recent assessment minus baseline youth needs. Given the item scoring, a negative change score indicates improvement. Of the 515 youth included in the analysis, 60.7% experienced reduced needs.

Predicting Improvement

Table 2 reports means, standard deviations, and inter-correlations for dependent variable and the seven significant predictor variables in the first regression model (Model 1) for 515 youth. Since reduction in youth needs is indicated by a negative change score (i.e., decreasing scores indicate decreased needs), negative standardized beta scores, β, indicate a positive relationship between higher predictive values for fidelity and baseline CANS ratings and improved youth’s needs. For example, the mean youth change (β = −1.59, SD = 4.16), reflects an overall improvement on the 0–3 CANS scale.

Table 2 Means, standard deviations, and intercorrelations for dependent and predictor variables, Model 1
Full size table

In Model 1, significant correlations exist between change in youth needs and each remaining predictive variable. The strongest correlations between dependent and predictive variables were between change in youth needs and baseline behavioral health needs (r = .38, p < .001) and change in youth needs and baseline functioning (r = .38, p < .001). The strongest inter-correlation between predictors was between baseline behavioral health needs and risks (r = .49, p < .001) and functioning (r = .44, p < .001), respectively. The correlation between the fidelity score and change is significant, but relatively small (r = −.13, p < .01).

Total wraparound fidelity has a significant, but small impact on improvement (β = −.14, p < .001) when baseline functioning, gender, age, race and ethnicity are held constant (see Table 3). The strongest predictors of improvement are baseline needs: behavioral health (β = −.18, p < .001), functioning (β = −.18, p < .001) and risks (β = −.16, p < .001). Youth who identify themselves as being of two or more racial groups are less likely to benefit (β = .10, p < .001) from wraparound. Overall, the model’s predictors account for 26.7% of the variability in change in youth needs.

Table 3 Hierarchical regression model predicting change in youth needs, Model 1 (N = 515)
Full size table

The results of the second regression model, reported in Tables 4 and 5 (n = 498), indicate specific significant elements of the wraparound services model and baseline needs. Some youth were excluded from the analysis when individual fidelity and CANS items were predictors. The final model included 15 predictive factors, 14 of which were components of wraparound fidelity or baseline needs and strengths plus youth with mixed racial identity.

Table 4 Inter-correlations for dependent and predictor variables in Model 2
Full size table
Table 5 Regression model predicting change in youth needs, Model 2 (n = 498)
Full size table

The strongest correlation between the dependent variable and the predictor variables was found between change in youth needs and baseline conduct behavioral needs (r = −.276, p < .001), followed closely by the outcomes based element of wraparound (r = −.244, p < .001), bullying (r = −.242, p < .001), functioning at home (r = −.217, p < .001) and functioning at school (r = −.207, p < .001). Among predictors, the strongest correlation was found between baseline conduct and delinquency (r = .314, p < .001).

Among the strongest predictors of improvement for youth were the outcomes based (β = −5.87, p < .001) and the community based (β = −2.47, p < .001) elements of wraparound fidelity. Unexpectedly, the cultural competency element (β = 5.09, p < .01) predicted increased needs, with 95% of wraparound facilitators reporting that the wraparound teams met cultural fidelity requirements. Specific baseline needs and strengths made relatively small contributions to positive change. Youth who identified themselves as being multi-racial were less likely than Caucasian youth to improve (β = 1.76, p < .05).

Discussion

The descriptive analysis conducted in this study suggests that there is a significant relationship between the implementation of several system and organizational components and wraparound fidelity. Specifically, high fidelity was reported more often by facilitators in communities at Stages 3 and 4 of implementation than in Stage 2 communities. Unexpectedly, nearly 46.4% of facilitators in Stage 1 communities also reported a high level of fidelity. When interpreting this result, it is important to remember that only seven communities were included in Stage 1 compared to over 13 communities in each of the remaining three stages. Additionally, the absence of necessary system and organizational conditions to support wraparound at the local level resulted in grant representatives spending extra time in these communities and infusing a higher level of support and coaching of wraparound facilitators and supervisors than was provided elsewhere. The grant’s policies, procedures and quality improvement initiatives may be compensating for missing strong stable local organizational and system support necessary for high quality collaborative, individualized service planning, support and effective services (Walker et al. 2003). Thus, the level of wraparound fidelity observed in Stage 1 is likely the function of state-level support for wraparound rather than the presence of necessary support conditions in the local community. Measuring the level of state team monitoring and support given to local providers is indicated for future studies.

The results of this study indicate that although stage of development is significantly associated with the level of wraparound fidelity, it is not predictive of improvement in youth needs. This result can be partially explained by the different methods used in this study. The descriptive analysis compared the level of association between the stage of development and the level of wraparound fidelity. The regression analysis used the average implementation score across the necessary system and organizational support conditions assessed and the total WFI score from interviews with facilitators. By doing so, the relationship between stage of development and both wraparound fidelity and youth outcomes diminishes. Further research is needed to determine if lack of inter-rater reliability of the site assessment used in this study contributed to this pattern of results.

An association between high fidelity wraparound and improvement in youth needs was supported by the regression correlations. The initial regression model includes composite dimensions, which support the relevance of fidelity to wraparound, but are difficult to interpret. Consistent with other studies (e.g., Cox et al. 2010; McGrew et al. 1994), the components of total fidelity scores, baseline needs and strengths not only explain more of the variability in the change in youth needs, but add meaning to the findings as well. While all of the elements of the wraparound services model are important to the integrity of the model, not all components may be related to outcomes for youth.

Although the specific predictors in this study differ from other findings (Cox et al. 2010), the importance of involvement in community based activities, is reinforced by the predictive relationship between community-based services and improvement in youth functioning. The current study also suggests that adherence to the outcomes-based element of wraparound contributes to decreased needs for youth, when holding baseline functioning and demographic variables (i.e., age, gender, race and ethnicity) constant. In other words, evaluating and linking needs and strengths to the intervention plan, actively monitoring progress and making changes is important to improvement.

The unanticipated and incongruent finding that cultural competency predicts worsening of needs may reflect rating issues for this fidelity element. Specifically, about 95% of facilitators reported that wraparound was implemented with fidelity to the cultural competence element. Future research that includes WFI interviews with caregivers and youth is needed to determine whether wraparound was actually implemented with this high level of fidelity to this element or if the ratings of the wraparound facilitators were somewhat biased.

The importance of specific baseline needs and strengths and race, help answer the important question of who benefits from wraparound. Youth with high levels of anxiety conduct disorders, delinquent and bullying behaviors and who demonstrate functional problems at home, at school and in the community were more likely to benefit from wraparound than youth who did not have needs in these areas. These findings reinforce previous assumptions that the model is better suited for youth with some internalized behavioral health needs, behavioral issues, functional impairments and disruptions in family relationships. Absent are findings related to high levels of depression, impulse control or psychosis.

Limitations

Several limitations of the current study should be noted. First, the data for this study came from the evaluation of Indiana’s CA-PRTF grant. The communities and youth represented in this study may not be comparable to youth participating in this demonstration grant in other states or to youth who receive wraparound that is supported by other funding mechanisms. Additional research is needed to determine the extent to which these findings will generalize to other populations of youth.

Second, the distribution of youth served in communities across the four stages of implementation was not equivalent, with nearly 60.9% of the youth represented in this study served in Stage 3 (i.e., action). Similarly, the overall WFI score for the youth included in this study was 80.7% and the majority (73.9%) of youth received wraparound at adequate or high levels of fidelity. These results suggest that wraparound has been successfully implemented in Indiana, but may have limited our ability to fully understand the relationships that exist among stage of development, wraparound fidelity and outcomes. Future research should include one or more comparison groups of similar youth receiving usual public behavioral health services and/or youth receiving wraparound at lower levels of fidelity.

Third, although WFI interviewers and CANS raters were required to achieve minimum reliability requirements in order to use the tools, similar requirements do not exist for the stage of development ratings made using the Strengths-Based Site Assessment. A method to assess inter-rater reliability for the site assessment needs to be developed and included in future studies. Fourth, a measure of the level of state oversight provided to communities at each stage of development is needed in order to help explain the unexpected finding that youth in Stage 1 communities experienced high level of wraparound fidelity. Fifth, the complexity of the issues examined limited the conclusions which could be reached from the data used in this study. For example, the outcome measure used (i.e., change in youth needs) is a composite rating of behavioral health needs, functioning and risk behaviors. This composite score lacks details about specific areas of change that would help clarify how the identified predictor variables contribute to overall improvements in functioning.

Sixth, measurement of wraparound fidelity in this study was limited to WFI interviews conducted with facilitators and not caregivers or youth. This is a significant limitation of the study because the families’ experiences of wraparound could have differed substantially from that of the facilitator. Finally, the wraparound interviews included in this study were the most recent interviews available for each youth, rather than interviews conducted at a specified time (e.g., 6 months after starting wraparound or at the completion of wraparound). As future interviews are completed, especially for youth who have completed the wraparound process, the results of this study should be re-examined.

Conclusions

The results of our study support the hypothesized relationship between (1) stage of development of necessary system level support conditions for wraparound and wraparound fidelity and (2) wraparound fidelity and youth outcomes. Additionally, this study identifies several variables that predict improvement in youth functioning within the wraparound process. Specifically, the results suggest that when wraparound is implemented with high fidelity to the outcomes-based and community-based elements, youth are more likely to improve. Wraparound is most beneficial to youth with high levels of anxiety, behavioral disorders (conduct disorders, delinquency or bullying) and low functioning at home, school and in the community. Interestingly, community stage of development was not directly associated with youth outcomes in this study. The fact that stage of development was significantly associated with high fidelity to wraparound, however, suggests that researchers should continue to refine tools to measure system level conditions supporting wraparound.

The relationship between wraparound fidelity, youth strengths, and functional outcomes also deserves further investigation. The continuing expansion of wraparound in Indiana will provide an opportunity to examine some of these issues. Not only will the CA-PRTF grant continue to provide information on wraparound implementation, the expansion of wraparound into other child-serving systems (e.g., child welfare) will provide additional sources of information and opportunities for comparison. The lessons learned from this study should help leaders in the behavioral health system keep focused on the importance of effectively implementing the necessary support conditions for wraparound and supporting high fidelity wraparound for youth and their families.