Introduction

Research suggests that immigrant status increases a child’s risk for a host of psychological and behavioral problems including anxiety disorders, depression, posttraumatic stress disorder (PTSD), substance abuse, conduct and eating disorders (Pumariega et al. 2005). These children face challenges at school and in the low-income areas where many resettle after migration as they undergo a process of acculturation and adjustment to the new society. PTSD and depression are of particular concern in immigrant groups given that many children are exposed to violence before, during, or after migration (Guarnaccia and Lopez 1998). A study of immigrant school children in Los Angeles found high levels of exposure to violence with 80% of the sample witnessing a violent event and 49% experiencing violent victimization in the past year (Jaycox et al. 2002). In addition, a subset of the immigrant population, refugees fleeing persecution and war, have experienced multiple traumas prior to resettlement (Birman et al. 2005). As a result, many immigrant children are thought to have serious mental health needs that can be addressed by trauma-informed mental health services.

However, current research indicates that immigrant children have difficulties accessing effective mental health services (Hernandez 2004; Huang et al. 2005; USDHHS 2001). Specifically, Latinos and uninsured children who need mental health services are less likely to receive them than their insured, White, or African American counterparts (Kataoka et al. 2002). Citizenship status has also been found to affect health care utilization among immigrants, as non-citizens were less likely than citizens to have insurance, have visited a doctor, dentist, or mental health provider in the past year, and have a usual source of health care (Huang et al. 2006). Other barriers to seeking treatment include lack of familiarity with mental health services and stigma associated with mental illness and services (Ellis et al. 2006). Culturally sensitive outreach conducted by multicultural staff has been shown to be effective at engaging immigrant families in treatment (Dillman Carpentier et al. 2007), as can comprehensive service interventions (Birman et al. 2008).

In the US, there has been a push by experts and funders to develop, test and disseminate evidence-based mental health practices though few treatment effectiveness studies with immigrant children exist. Even outside the immigrant context, there are many concerns in the field about translating evidence-based treatments into clinical practice (Kazdin, 2008). Evidence-based treatments are developed in controlled clinical trials, where researchers administer a manualized treatment protocol to carefully selected clinical samples that meet specific diagnostic criteria; those deemed inappropriate for the intervention are referred elsewhere. In contrast, community-based mental health providers serve diverse individuals with a variety of mental health issues, necessitating the use of multiple types of treatments in creating individualized service packages (Kazdin 2008). Thus, it has been argued that community-based providers benefit from having a variety of tools at their disposal and the flexibility to use their clinical judgment to employ a specific “practice element” (Chorpita et al. 2005) or distinct intervention technique as part of an overall intervention strategy.

Comprehensive and flexible community-based services are important for immigrant youth because they can address the complexity of mental health needs arising within the larger context of acculturation and resettlement to the new country (Birman et al. 2005, 2008). Immigrant families experience acculturative stress (Westermeyer and Wahmanholm 1996) as they struggle to meet their basic needs of housing, employment, and health care in a new language, within the norms and laws of a new culture. In this context, families may not consider mental health as a high priority to address, and may not seek mental health care for their children (Westermeyer and Wahmanholm 1996). Further, while traditional clinic-based services may be sufficient to meet the mental health needs of many children, for immigrant children and families specific psychological difficulties are embedded within ongoing stressors related to adjusting to living in a new country. As a result providers need to be flexible and creative in engaging and delivering services to this population (Davies and Webb 2000). Comprehensive mental health programs for immigrants provide not only therapeutic services to address symptoms of disorders such as depression and PTSD, but also case management and tangible adjustment support to address the multiple needs of refugee and immigrant families in resettlement (Birman et al. 2005, 2008). For example, providing psychoeducation and assistance with resettlement to immigrant families can create conditions under which immigrant children maximize benefits from psychological treatment aimed at symptom reduction. Such comprehensive mental health programs may offer a multiple treatment options including manualized treatment components to address specific needs in dynamic contexts of ongoing adjustment.

The purpose of this study was to describe a comprehensive school-based mental health program for immigrant youth and assess its outcomes. We extend previous descriptive, practice-based research on “usual” mental health care for youth (Garland et al. 2010a) by linking intervention elements to outcomes (Chorpita et al. 2005), and extend this literature by focusing on school-based mental health services for immigrant children.

Schools as Settings for Mental Health Interventions

While schools provide an excellent setting for intervening in the broader ecology of immigrant youth, to date there have been few studies of comprehensive school-based mental health service models for immigrant children. Schools have emerged as key settings for the provision of mental health interventions where a full range of services including prevention, early identification, and treatment of mental illness are possible (Adelman and Taylor 1999; Huang et al. 2005; Masia-Warner et al. 2006; Owens and Murphy 2004). Mandatory attendance at school ensures access to a broad range of children and child behavior can be observed in schools by many individuals across a range of settings, creating opportunities for screening and early intervention. Stigma surrounding mental health services can be reduced when provided as an educationally connected intervention in a safe, familiar setting, and schools have natural access to families who may be reluctant to seek mental health services for their children in more traditional settings.

For immigrant children, school-based interventions have been suggested as a particularly effective way to provide access to mental health care (Ehntholt et al. 2005; Fazel et al. 2009; Kataoka et al. 2003; O’Shea et al. 2000; Rousseau et al. 2005, 2007; Stein et al. 2003) because schools are where these children’s acculturative and adjustment struggles unfold (Birman et al. 2007). School-based mental health interventions represent a way of treating a child’s problems within the context of the larger systems that support or inhibit the child’s adaptation to these new surroundings. Such comprehensive, ecologically minded approaches to mental health services (Birman et al. 2005; Davies and Webb 2000; Pumariega and Vance 1999) may be more effective in creating sustainable change when they target aspects of the surrounding school environment in addition to individual behavior (Trickett et al. 1972; Trickett and Birman 1989).

However, only a few studies have reported on the effectiveness of school-based interventions with immigrant children. Group cognitive behavioral interventions designed to target symptoms of PTSD have been shown to be effective in controlled studies. Significant declines in symptoms of PTSD (Ehntholt et al. 2005) and of both PTSD and depression (Kataoka et al. 2003) were reported for refugee and Latino immigrant children, respectively. In addition, intervention models designed to help immigrant children with general adjustment issues have also been studied. Expressive arts workshops have been shown to be effective in increasing feelings of popularity and integration (Rousseau et al. 2005) and decreasing levels of emotional and behavioral symptoms (Rousseau et al. 2007) for newly arrived immigrant students relative to non-intervention controls. In addition to these group interventions, several papers report on interventions more comprehensive in scope, providing assistance with general adjustment as well as specific symptoms when needed, individualized for particular immigrant students. In one study school staff identified and referred refugee students for treatment to a mental health worker who conducted outreach to families to engage them in treatment and provided a range of customized treatment options including individual and family therapy on site at the school and referrals to other services to address the broader adaptation and resettlement issues faced by the families (O’Shea et al. 2000). Though no comparison group was available, data from seven students showed a non-significant trend of improvement on emotional and behavioral symptoms.

Fazel et al. (2009) reported on a service where mental health professionals provided extensive consultation to teachers on strategies to use in the classroom to address the needs of students identified as having problems. In situations where problems continued, mental health professionals conducted outreach to the children’s families and provided direct individualized therapeutic services. The 47 refugee children who received services were compared to two non-intervention control groups of ethnic minority and white UK-born students. Though the groups were not equivalent on measures of emotional adjustment at baseline, the refugee students were reported to show relatively greater improvement on symptoms of hyperactivity. Further, within the refugee group, the 11 children who received direct therapeutic services improved significantly more with respect to peer problems than the 36 children who received consultation services alone. These findings suggest that intervening with the school environment through consultation with teachers may be an effective strategy, and that more comprehensive services (consultation and direct services) were more beneficial than consultation alone.

As evidenced in the review above, conducting intervention studies with immigrants and refugees poses unique methodological and ethical challenges (Birman 2005). For example, in studies of diverse samples of immigrant and refugee students it is unclear what might represent a “matched” control or how unequal a “nonequivalent” control group can be. Further, providers often feel it is unethical to deny immediate services to traumatized immigrant and refugee students particularly when they are newly arrived and in need of assistance. As a result, conducting controlled studies with these populations is often not feasible.

However, alternatives to randomization with small samples are emerging in the literature (Allen et al. 2009; Henry 2009, 2011). One alternative way to test effectiveness without withholding treatment is to examine the impact of service “dosage” or amounts of services received (Shadish et al. 2002). Effectiveness studies are strengthened when quantities of distinct service elements can be linked to mental health outcomes (Garland et al. 2010). Further, longitudinal analyses of outcomes that include time-varying service quantities representing dosage before a given measurement point help establish the direction of causality (Singer and Willett 2003). Finally, by analyzing the effect of time separately from service dosage, intervention effects can be isolated from maturation effects, strengthening causal inferences (Allen et al. 2009; Singer and Willett 2003).

The purpose of our study was to describe and evaluate a comprehensive school-based intervention that provided individualized services for diverse immigrant students. Data are presented on a sample of immigrant students served across two large school districts. While no control group was available, we analyzed the link between practice elements and outcomes (Chorpita et al. 2005) using statistical techniques recommended for longitudinal data (Singer and Willett 2003) and with small, diverse samples (Allen et al. 2009). To this end we report extensive information on an intervention model that included both evidence-based manualized treatment components, as well as individualized services that employed a range of intervention techniques to match the child’s needs.

CATS—A Comprehensive Mental Health Program for Immigrant Children

Cultural Adjustment and Trauma Services (CATS) was a comprehensive school-based mental health service program of the International Institute of New Jersey (IINJ) operating in two school districts. CATS targeted first and second-generation immigrant children with significant trauma exposure and/or cultural adjustment needs. CATS services were funded by a grant from the Substance Abuse and Mental Health Services Administration (SAMHSA), through the National Child Traumatic Stress Network (NCTSN). The purpose of the grant was to implement promising and evidence-based practices in a mental health program for immigrant children. The overall CATS service model was based on the Family, Adult, and Child Engagement Services model (Birman et al. 2008), designated as a promising practice for traumatized refugee children by NCTSN. In addition, CATS received training on several evidence-based treatments disseminated through the NCTSN, and incorporated them into this overall service model.

The CATS staff was comprised of bicultural and/or bilingual licensed clinicians, who provided a range of clinical services, and “culture brokers” who were ethnic paraprofessionals primarily responsible for the outreach and case management activities of the program. Clinicians were master’s level psychologists and social workers, including students placed to complete their practica in the program. Several clinicians were also bilingual/bicultural and themselves of immigrant backgrounds. Culture brokers were members of immigrant groups in the community. All of them had experience within the larger agency as refugee resettlement staff and received additional training from the agency on mental health issues and treatment options. To be hired as culture brokers staff had to have a history of serving as “connectors” in their own communities and a willingness to translate their “connector” skills to other immigrant communities and schools, in addition to being flexible, patient, adaptable, outgoing and open to others, able to handle ambiguity in their role, and comfortable advocating for students and families in multiple settings. CATS staff spoke 8 different languages and offered services in students’ primary language or English; where program staff could not meet the language needs of the clients, interpreters or translators were used. Clinicians and culture brokers worked closely as a team and with schools to conduct outreach and identify immigrant children at risk for or currently experiencing adjustment difficulties and decide on appropriate treatment (e.g., which referrals to make and/or which services to provide).

In addition, CATS adapted their comprehensive service model over time to fit with local resources and respond to the needs of the two school districts, Jersey City and Clifton, where they were placed. Jersey City is a large, urban area that has been an entry point for immigrants since the early 1900 s where today fifty percent of the total population speaks a language other than English at home (US Census 2000). Program staff was based in a high school and a K-8 grade school with particularly large immigrant student populations, and immigrant students from the other 37 schools in the district had access to off-site services. Clifton is a middle-class, suburban area in New Jersey with 1 high school, 2 middle schools and 14 elementary schools. Previously a majority white district (City of Clifton 2010), the ESL population in the district has increased by 250% in the past 3 years, and currently fifty-seven percent of the student body comes from families that speak a language other than English (Clifton Public Schools 2009). Due to the large size of the high school (4,000 students), CATS staff was based there but provided services to K-8 students as needs arose.

CATS Service Components

Model components included: (1) relationship-building; (2) outreach services; and (3) comprehensive clinical and case management services.

Relationship-Building

The philosophy of the CATS model was to focus on building relationships with school personnel and immigrant students within those schools. By being present and available to address a range of student issues, CATS staff sought to increase their chances of identifying and engaging immigrant students with mental health issues. Sixteen different program staff were placed part-time in nine different schools across both districts. Each culture broker spent the majority of her/his time in one school, while clinicians divided time between schools and the agency offices.

Culture brokers positioned themselves at the school in locations particularly relevant to immigrant students (e.g. the ESL office), so that teachers and students could turn to them for consultation and assistance. Teachers regularly sent immigrant students for consultation to CATS staff if they were visibly upset or seemed uncharacteristically distracted. CATS staff provided teachers with consultation on classroom practices, assistance in negotiating decisions of academic placement, and interpretation during phone calls with parents. At the request of school administrators, CATS staff provided training on cultural and mental health issues to teachers, and consultation on translation services. In Jersey City, for example, teachers and support staff raised concerns about high numbers of newly arriving immigrant students who were socially isolated and struggling academically. In response, school and program staff created a system to ensure these students were introduced to CATS program staff within the first week of arriving at school, and created a special acculturation group to help orient them to the new culture and school.

Outreach Services

Outreach was a defining feature of the CATS model, in that culture brokers provided a link to mental health services without themselves having a predetermined mental health agenda. Outreach services were aimed at preventing acute/minor adjustment and educational issues from escalating into chronic/major problems by providing immediate tangible assistance to students; and at identifying students with enduring mental health issues to engage them in treatment. This model of outreach created the possibility for early detection and treatment of mental health issues. The intent was to intervene before relatively simple adjustment problems turned into more serious psychological difficulties; to this end staff intervened in the schools on behalf of students to create a more hospitable and supportive environment. Specifically, CATS staff provided informal tangible and supportive services including concrete resources such as food pantries, job placement, advice about college, after-school support groups, and advocacy and guidance regarding class schedules. Further, clinicians also provided some outreach services such as psychoeducation, relaxation training, and other brief clinical interventions to identify immigrant children with serious mental health issues and engage them in clinical services.

Clinical Services

Decisions to offer clinical services were made based on formal clinical assessments conducted by clinicians in consultation with culture brokers, and with school staff in some cases. The treatment philosophy was client-centered, designed to address a range of issues including symptoms, resettlement concerns, and other stressors that students discussed in regular sessions. While services provided were trauma-informed, trauma was seen as one of many factors impacting on the students’ lives.

The overall service model was designed to impact multiple levels of the child’s ecology. At the broader school and community level, coordination services were provided by the clinicians to link students and their families to community resources, and to provide consultation to and work with school staff in addressing student issues related to class placement, scheduling, and transportation. Here, the intent of these services was to reduce stress in the surrounding context (in classrooms and schools) or provide supportive resources for families, rather than address intrapsychic issues such as feelings of hopelessness on the part of the child. Many of these services included tangible support that is not typically conceptualized as part of clinical treatment.

Family services included parent training, family therapy, and techniques to strengthen families. Parent training involved meeting with parents to discuss the treatment process, provide psychoeducation about the impact of trauma on children and help with parent–child difficulties. Family therapy was used when parents and students felt the need and were open to exploring family dynamics in the therapeutic context. When family therapy was not feasible, clinicians used techniques to strengthen the family, such as giving suggestions to parents about how to improve communication with their children.

At the individual level, clinicians used a range of therapeutic techniques, selected in response to the students’ presenting problems and ongoing concerns brought into treatment. Supportive therapeutic techniques consisted of clinical interventions that involved reflective and empathic listening, engaging clients in processing complex issues and identifying their feelings, validating and normalizing these feelings and generating options for addressing problems. Psychoeducation involved raising awareness of mental health issues among students, providing basic information about effects of trauma, depression, and setting expectations for treatment. Cognitive behavioral therapy (CBT) techniques were used to address presenting problems associated with thought distortions or catastrophic thinking, helping students learn relaxation techniques and understand the cognitive triangle. CBT techniques were used eclectically, as needed in response to clinical presentation, rather than as part of a structured, clinician-driven CBT protocol.

Trauma-focused cognitive-behavioral therapy (TF-CBT; Cohen et al. 2004 ) was the only manualized intervention used. TF-CBT is an intensive, short-term approach to treating children with PTSD or other issues associated with traumatic life experiences. Individual and group sessions for children and parents are provided with the goal of reducing child PTSD symptoms. TF-CBT was initially developed to treat sequelae of child sexual abuse, but has been extended to treatment of other types of trauma (Cohen et al. 2004; Cohen et al. 2010). The treatment protocol involves selecting a particular traumatic experience and processing it using cognitive behavioral techniques. The Kauffman Best Practices Project considered TF-CBT a best practice (Chadwick Center for Children and Families 2004), and the US Department of Justice rated it in the highest class of evidence-based interventions (Saunders et al. 2004). CATS clinicians received training from the intervention developers to administer the clinician-directed, 8–20 session treatment protocol, with minor modifications as needed to address cultural concerns.

Students were engaged in TF-CBT only if they identified a specific traumatic event or experience as the source of their current problems during the clinical assessment and intake process. They also had to demonstrate ability and willingness to process a narrative about a single traumatic experience. TF-CBT was discontinued when a student was no longer able to focus on a particular trauma narrative, or life stressors such as school, parent, or relationship problems increased to the extent that exclusive focus on the selected trauma was counterproductive. For example, recurrence of traumatic events such as domestic or community violence interfered with several students’ abilities to complete a trauma narrative. Importantly, for all of these students regardless of the number of sessions they received, TF-CBT was conceptualized as part of the overall treatment model, and other services such as treatment coordination, family services, and other therapeutic techniques were used as needed, along with TF-CBT.

Given that such diverse and individualized treatment elements comprised CATS’ comprehensive approach to mental health services for immigrant youth, our goals in this paper are to (a) describe the comprehensive model and its client base and (b) test the relationships between distinct treatment elements and outcomes. In assessing the effectiveness of eclectic, individualized services and a manualized intervention like TF-CBT within one comprehensive service model, we contribute to emerging research on the effects of integrating evidence-based interventions within community-based service models (Brookman-Frazee et al. 2009; Garland et al. 2010a).

Methods

Sample

A total of 1,043 students received services from CATS staff during the SAMHSA funding period. Of these, 894 received outreach services only, and 149 enrolled in clinical services. Demographic and clinical information was collected only on these 149 students; other service-related data were used to describe service patterns in each school district, and not to report on individual clients.

Measures

Services Data

For each service encounter CATS staff tracked the location, type, language of service, clients, duration of service contact, and primary intervention technique used during each session. There were 18 different types of services tracked in all and we selected 7 that represented the most frequently provided service elements. For example, we excluded assessment and post treatment meetings from our service counts, because they did not constitute treatment. We excluded other services because they were provided to only 2–3 students (e.g. expressive therapy, medication consult) or were not tracked reliably, with different clinicians tracking diverse activities under a single service category (narrative therapy). The variables we chose represented quantities of seven separate services. CATS tracked service quantities in 15 min units such that a session lasting 1 h in length was counted as four service units. We present the services in terms of service units because we felt this was a more accurate reflection of the total service ‘dosage’; however, the number of sessions and session lengths cannot be inferred from the total number of service units received. Outreach services were provided prior to enrollment in clinical services primarily by culture brokers. Clinical services included cognitive-behavioral therapy and relaxation techniques, trauma-focused cognitive-behavioral therapy, supportive therapy, psychoeducation, coordinating services, and family services. While TF-CBT included use of psychoeducation, family services, and CBT techniques, when these were done as part of the process of working through a trauma narrative consistent with the TF-CBT protocol they were coded as “TF-CBT” rather than the individual techniques. Similarly, psychoeducation was coded as “family services” when conducted with parents.

Demographic and Background Data

Variables of interest for students receiving clinical services included school district, birth country, ethnicity, and primary language. In addition, information about the types of traumatic events experienced was collected during intake by the clinician. The NCTSN General Trauma Information Form (NCTSN 2004) assesses whether a child had experienced one of 19 types of traumas, or a possible “other” type of traumatic event.

Clinical Data

Clinicians reported on reasons for referral, referral source and primary problem treated. Two clinical assessment measures were used to track client progress: the CAFAS and the PTSD-RI. These were administered at intake into clinical services, and approximately every 3 months during the course of treatment.

The Child and Adolescent Functional Assessment Scale (CAFAS; Hodges 2000) was used by clinicians to rate a child’s functioning over the past 3 months across eight life domains: school, home life, community living, behavior toward others, behavior toward self, moods/emotions, substance abuse, and thinking. In addition, an optional caregiver subscale assesses caregiver ability to meet emotional and material needs. The measure provides specific behavioral examples of functioning at different levels of impairment. Subscale scores are summed to create a total score reflecting overall level of dysfunction (range 0–240), with higher scores reflecting greater impairment. The CAFAS was selected because it is a clinician-report scale, making it possible to use with all clients regardless of their age or English language skills. CATS staff were trained to reliability on the measure. Prior studies report good inter-rater, test–retest, and Cronbach alpha reliability within clinical samples (e.g., Hodges et al. 2000), including in a study with a diverse immigrant and refugee sample (Birman et al. 2008). In this sample CAFAS reliabilities ranged from Cronbach’s α = 0.71–0.78 for the first three administrations.

Posttraumatic stress symptoms were measured with Part III of the adolescent version of the UCLA PTSD Reaction Index (PTSD-RI; Pynoos et al. 1998). The PTSD-RI is a self-report measure that has been used across a variety of situations, settings, and cultures since its development. Many studies have found sufficient discriminant validity of the PTSD-RI, high internal consistency and test–retest reliability (Steinberg et al. 2004), and adequate convergent validity. In addition, the PTSD-RI was found to be a reliable and valid screening measure for Somali adolescent refugees (Ellis et al. 2006). Part III of the measure contains 22 items that assess the frequency of trauma symptoms during the previous month in reaction to a specific traumatic event. Items are rated on a 5-point Likert scale ranging from 0 (none of the time) to 4 (most of the time); 17 of the items are summed to create a total score. PTSD-RI reliabilities ranged from Cronbach’s α = 0.90–91 for the first three administrations. The CAFAS and the PTSD-RI were positively correlated at intake (r = 0.32, p < 0.001) and at last administration (r = 0.33, p < 0.001). These correlations provide evidence of concurrent validity of the PTSD-RI, a self-report measure, with the CAFAS, a well-validated clinician-report measure.

Results

Client Characteristics

A total of 149 students (94 female, 55 male) from 29 different countries (see Table 1) received comprehensive clinical services from CATS over the three-year period. These clients spoke 19 different languages (see Table 1) and attended nine different schools within the two school districts. Fifty percent were born outside of the US, 45% were born in the US to immigrant parents, and 5% were born in the US to US-born parents. The average age at intake was 14.4 (range 6.4–21; SD = 2.9), with Clifton students older on average (M = 16.3, SD = 1.7) than those in Jersey City (M = 14, SD = 3).

Table 1 Demographic information on children and adolescents receiving services (n = 149)
Full size table

Clients had experienced an average of 4 types of traumatic events before the start of treatment (range 1–9; SD = 2.1) and all but 15 students experienced two traumas or more. The most commonly experienced types of traumatic events (see Table 2) were community violence, traumatic loss or bereavement, physical maltreatment/abuse/assault, and domestic violence. The most frequent primary presenting problems as reported by clinicians were PTSD (n = 24), traumatic/complicated grief (n = 23), depression (n = 22), generalized anxiety (n = 19), and general behavior problems (n = 16; see Table 2). Initial student CAFAS scores (M = 48.71, SD = 33.34) represented moderate levels of dysfunction (Hodges, 2000). Initial PTSD-RI scores (M = 24.22, SD = 14.91) represented moderate PTSD symptoms; 32 students were above the clinical cutoff for PTSD.

Table 2 Clinical information on children and adolescents receiving services (n = 149)
Full size table

Service Patterns

Pathways to clinical services differed substantially between the school districts, particularly in the role of outreach in engaging students in individualized clinical services. A total of 1043 students (Clifton = 413, Jersey City = 630) received services. In Clifton, a culture broker was dedicated to the high school for 3–4 days per week, and provided outreach services to a total of 410 students. Of these, 26 went on to engage in individualized comprehensive clinical services, and only 3 (10%) came to clinical treatment directly without culture broker contact. In contrast, Jersey City culture brokers were spread across more schools and provided significantly fewer hours of outreach service. There, CATS staff implemented outreach group interventions to handle the students’ needs with few resources. In all, 586 students received individualized (n = 506) and group (n = 80) outreach services, with 76 students continuing on to receive individualized clinical services; 44 (37%) students came to individualized clinical services directly without having been assisted/screened by a culture broker. In both school districts students who came to clinical services directly were referred to the program by nurses, guidance counselors, teachers, and parents.

Across the two districts 54 (36%) students left treatment before accomplishing their treatment goals. Of these, 25 stopped attending school due to moving, the end of the school year, or because they dropped out of school. Comparative analyses were conducted to examine differences between those who left and those who completed treatment on first and last outcome scores, the covariates included in the analyses, primary presenting problems, types of trauma, and types of services. The only significant difference was that most of the students with conduct disorder as their primary presenting problem (5 out of 7) dropped out of treatment, χ 2(1, n = 149) = 3.94, p < 0.05.

Service Components

Students received different combinations of the seven service components. Of the 149 students who received clinical services, 90 received outreach services prior to enrollment (M = 12.94 units, SD = 21.69). While in services, 113 received CBT (M = 22.32 units, SD = 20.85), 107 received coordination services (M = 17.81 units, SD = 19.65), 36 received family services (M = 8.67 units, SD = 16.23), 82 received psychoeducation services (M = 6.5 units, SD = 5.77), 95 received supportive therapy (mean 8.24 units, SD = 7.58), and 50 received TF-CBT (M = 16.4 units, SD = 16.23). On average the 50 students completed 7 TF-CBT sessions (the range was 1–21, with one outlier who completed 46 sessions), but only 15 completed 10 or more sessions.

Correlations were conducted to explore whether the presenting problem or number of traumatic experiences was associated with the total quantity of each service component. Quantity of supportive services was positively associated with having suicidality as the primary presenting problem (r = 0.31, p < 0.01). More family services were provided to students with PTSD as their primary presenting problem (r = 0.16, p < 0.05) and those who experienced more traumatic events prior to treatment (r = 0.17, p < 0.05). In addition, TF-CBT was positively related to having PTSD (r = 0.25, p < 0.01), acute stress disorder (r = 0.16, p < 0.05), and conduct disorder (r = 0.16, p < 0.05) as primary presenting problems. Quantities of CBT, psychoeducation, coordinating services, or outreach were not associated with primary presenting problems or number of traumas.

Intervention Effectiveness

Random-effects regression analyses were used to assess overall change on the CAFAS and PTSD-RI as a function of seven time-varying predictor service variables: psychoeducation services, supportive therapy, TF-CBT, CBT, family services, coordinating services, and outreach case management services. Because time intervals between measurements varied within and across students, one time variable was computed for each student on each measure to reflect actual time to measurement since intake. Time-varying time variables were included to reduce potential error associated with fixing varied time intervals, to help distinguish intervention effects from time effects (Allen et al. 2009) and to provide more precise information on each student at each point of measurement (Singer and Willett, 2003). Service quantities were calculated to represent the cumulative service units provided up to each of their CAFAS and PTSD-RI measurement points. Because time intervals between measurements could vary across the two different measures for a given student, two sets of service quantities (one for each measure) were calculated for each student. Computing service quantities this way minimizes problems of reciprocal causation that can arise when interpreting links between time-varying predictors and outcomes (Singer and Willett, 2003). With each service variable representing cumulative services received before each measurement point, any changes in outcomes can be interpreted as the result of cumulative services provided before (rather than during or after) the point when the student was measured.

Covariates

Covariates included gender, whether student was born in US/Puerto Rico or abroad, school district, age at intake, number of traumatic events experienced before treatment, and caregiver impairment. In addition, attrition was included as a covariate because it was correlated with the outcome variables. Attrition was defined conservatively as leaving treatment before treatment goals were met, even when the reason was moving away or switching schools.

Random-effects regression analysis was chosen for its ability to handle varying numbers of measurements across individuals, allowing estimation of a slope for each subject regardless of how many time points they have in the longitudinal course (Hedeker and Gibbons 2006). We ran separate analyses for each measure, and went through the same steps for each analysis. First, base models were analyzed to determine whether intercepts and slopes varied across individuals; a random intercept and slope model was suitable for the CAFAS, and non-significant variability in slopes resulted in a random intercept model for the PTSD-RI. Second, all covariates were included to allow them to account for maximum variance in the outcomes. Third, service variables were entered into models with the significant covariates retained from step two. Finally, interactions of the significant service variables in each analysis were included. There were a maximum of 8 CAFAS and 7 PTSD-RI administrations, with each student having completed at least one, and on average between 2 and 3 administrations of each measure (CAFAS, M = 2.73, SD = 1.23; PTSD-RI, M = 2.53, SD = 1.26).

Functional Impairment

The first random effects regression showed that functional impairment decreased as a result of greater cumulative totals of supportive therapy, F (1, 150) = 24.25, p < 0.001, TF-CBT, F (1, 150) = 7.86, p < 0.01 and CBT services, F (1, 150) = 4.68, p < 0.03 (see Table 3). Further, the interaction of these 3 services was significant, suggesting that students who received more of this combination of services had steeper slopes than those who did not, F (1, 150) = 15.86, p < 0.001. Finally, these results were not due to time alone because CAFAS scores did not significantly increase or decrease over time, F (1, 141) = 2.45, ns.

Table 3 Results of the random-effects regressions conducted on the CAFAS (n = 145) and PTSD-RI (n = 142)
Full size table

PTSD Symptoms

The second random-effects regression showed that PTSD symptoms decreased as a result of greater cumulative totals of TF-CBT, F (1, 259) = 4.06, p < 0.05, and coordinating services, F (1, 259) = 4.11, p < 0.04; CBT services resulted in marginally significant improvements in PTSD symptoms, F (1, 259) = 3.30, p < 0.07 (see Table 3). The interaction of these services was not significant, such that providing this particular combination of services did not produce benefits over providing them separately. Finally, these results were not due to time alone because PTSD-RI scores did not significantly change over time, F (1, 259) = 1.33, ns.

Discussion

This study provides “practice-based evidence” of effectiveness of a comprehensive school-based mental health service model for immigrant students in two school districts. CATS services resulted in improved functioning and fewer PTSD symptoms for their clients. Data on implementation of the overall service model in two different school districts provide examples of how comprehensive services can be structured and implemented in schools, and complements the emerging literature on evidence-based mental health practice with immigrants and other underrepresented groups.

True to its intent, CATS provided trauma-focused services to a diverse group of first- and second-generation immigrant students. Regardless of their place of birth, all students were dealing with sequelae of trauma in the context of the family facing challenges with immigration, acculturation, and resettlement issues. Trauma exposure among this community sample of students was high, with 4 types of events experienced on average, and the vast majority experienced multiple traumas. Exposure to multiple traumas, particularly within the context of ongoing environmental disruptions experienced by immigrant children and families, is consistent with the construct of “complex trauma” (Cook et al. 2003). Most treatments of PTSD have not been designed for or studied with samples of people with complex trauma, and experts recommend caution in using “classic PTSD” techniques with such individuals (Courtois 2004). Without a strong evidence base on effective treatments for immigrant students with complex trauma, CATS staff had to proceed carefully and creatively in selecting service components.

Service pattern data demonstrated that clinical judgment resulted in a variety of services titrated to address different client problems. For example, supportive therapeutic services were more likely to be provided to students with problems with suicidality as their primary presenting problem. These services were designed to provide extensive support to students engaging in self harm behaviors. In future research, tracking the clinical decision making behind allocating services would provide a rich understanding of community-based service models.

Program Effectiveness

Our two outcome measures, the CAFAS and PTSD-RI, captured different aspects of the students’ mental health. The correlation between the two was moderate and suggests that the two measures assess distinct areas of adjustment. The PTSD-RI focuses on self-report symptoms and focuses on specific aspects of a student’s emotional experiences related to PTSD symptoms. In contrast, the CAFAS reflects functioning across a broad area of life domains as rated by clinicians. Both measures have limitations, but together provide a broader and more differentiated view of the child’s experience. Having different outcomes predicted by different service components highlights the value of assessing program impact both psychically and behaviorally, since one indicator cannot serve as a proxy for the other.

It is also possible that CATS’ immigrant students underreported their symptoms because they were uncomfortable disclosing them at the beginning of treatment prior to forming a relationship with the clinician, or because the PTSD-RI assumes cultural knowledge, understanding, and awareness of particular symptoms. Clinicians, on the other hand, may have overestimated the level of dysfunction on the CAFAS because of their empathy for the difficult circumstances of the students they served, and may have been optimistic and positive about student improvement, since they were invested in it.

Supportive therapy predicted improvement on the CAFAS but not on the PTSD-RI. Supportive therapy may have been particularly relevant for functioning because it involved helping students problem solve important life choices and crises as they arose, thus improving active coping and functioning. Supportive services were the least directive and the most reflective/empathic of services CATS provided, and were provided to the majority of clients. These findings highlight the need for treatment to address ongoing adjustment issues over the course of trauma informed treatment.

Service coordination, on the other hand, predicted improvement on the PTSD-RI, but not the CAFAS. It is unclear why providing tangible assistance to students and families and making changes in the students’ school environment would reduce PTSD symptoms and not improve functioning. It is possible that by alleviating stressors in the students’ environment CATS staff were able to reduce anxiety, a major aspect of PTSD symptoms as measured on the PTSD-RI. Importantly, service coordination and case management are generally not considered to be an active part of mental health treatment. Many programs such as Medicaid do not allow providers to bill for travelling to multiple intervention sites, providing assistance with social services, giving advice to teachers, or organizing field trips for groups of students. However, our findings suggest that service coordination/case management is an important component of the overall treatment model and contributes to student improvement.

CBT resulted in improved functioning, and was marginally significant in predicting symptom reduction. This is in contrast to one prior study that found CBT delivered in a group intervention context helped immigrant students with symptom reduction but not behavioral adjustment as assessed by teachers (Stein et al. 2003). In this comprehensive individualized services program, it appears that CBT techniques, which help students make connections between their feelings, thoughts, and actions, improved students’ abilities to cope with demands of the various settings in their lives, such as home and school. However, reducing symptoms of PTSD was more effectively achieved with TF-CBT, which employs cognitive behavioral principles to process traumatic experience.

TF-CBT was associated with both improvements in functioning and reduced PTSD symptoms. In interpreting these findings it is important to highlight the form that TF-CBT took as it was integrated within the CATS service provision model. While TF-CBT was organized around processing a trauma narrative and addressing child and family dynamics, CATS did not pre-determine the number of TF-CBT sessions they provided, and the majority of students who received TF-CBT benefited from fewer than the 8–20 sessions recommended.

TF-CBT contributed to improved functioning separately and in interaction with both supportive therapy and CBT. Supportive therapy is often considered to be less effective than structured and directive evidence-based practices, and is often used as a control to test the effectiveness of more structured interventions. For example, in intervention studies with sexually abused children (Cohen et al. 2004; Cohen et al. 2005) TF-CBT was found to yield significant improvement over supportive psychotherapy. The effects of supportive therapy we found in our study suggest that, at least for an immigrant sample struggling with complex trauma, supportive therapy should not be dismissed as a treatment option, and should optimally be combined with CBT and TF-CBT to improve functioning.

Psychoeducation, family services, and pre-enrollment outreach had no effect on functioning or PTSD symptoms. It is possible that psychoeducation and family services were not adequately captured by these variables because both were components of TF-CBT, and subsumed within that category when conducted as part of the TF-CBT model. It is possible that had we captured the services provided across these different types of sessions we would have been able to isolate their impacts on outcomes. Future research on community-based services needs to attend to precise measurement of service variables, particularly ones common across several types of interventions, to better capture their impact on outcomes.

With respect to outreach, service patterns suggest that it is an important pathway to services and clinician experience suggests that it is essential in developing positive relationships with school staff, teachers and students that allowed the program to operate smoothly. Outreach represents the “invisible” work that CATS providers felt must be done prior to providing school-based mental health services to immigrant youth. They saw outreach services as essential to understanding the contexts of the schools, building relationships with school staff, preventing minor issues from escalating, and engaging students in treatment. Though outreach activities are extensive in many successful school and community-based programs, they are largely unmeasured or unreported for a number of reasons, including funder preferences for evaluating formal, clinical treatment rather than less formal outreach services, there are ethical challenges inherent in collecting information on outreach-only (i.e., non-consented) clients, and limited time and resources to gather detailed service and outcome information on such a large number of clients. CATS staff decided against devoting time to collecting detailed information on the outreach-only clients. It is possible that the majority of students benefited from outreach, as evidenced by their not continuing to engage in clinical treatment. However, we were unable with this dataset to assess the impact of outreach on students who did not engage in clinical treatment. Future research is needed to understand the extent to which outreach and engagement may be important components of community mental health work.

Implications for Implementation of Evidence-Based Practices

Given the current push to disseminate evidence-based mental health interventions, our findings suggest a number of considerations when introducing externally developed programs into ongoing community service programs. This agency serving diverse immigrant clients approached implementation of TF-CBT with a sincere commitment to implementing it as designed. However, over time clinicians began to take the pieces they found most useful and integrate them into their own overall service model. This implementation process illustrates the challenges in negotiating the demands of evidence-based practices and the accumulated wisdom and practices of the community organization as described elsewhere (Garland et al. 2010a; Kazdin 2008).

Clinicians reported several challenges to administering TF-CBT according to the protocol. Students with complex trauma found it difficult to focus on a particular traumatic experience in developing a trauma narrative. Many students experienced a number of substantial stressors and disruptions related to ongoing issues of acculturation and adjustment, in addition to recurring trauma that made it difficult to focus on processing events in the past due to coping demands in the present. As a result, in many cases the TF-CBT treatment protocol was abandoned or put aside temporarily in favor of addressing more pressing concerns in the present. At other times CATS clinicians used TF-CBT treatment components in a particular treatment session, but did not work toward completing the entire protocol. Our data suggest that the components of TF-CBT were helpful to students served and appeared to fit well as an additional clinical offering within the CATS comprehensive service model. Importantly, from the perspective of program staff TF-CBT was integrated successfully into the CATS service model.

As pressures to disseminate evidence-based practices are increasing, so is inquiry into implementation challenges associated with them. However, much of the literature on this topic locates the source of challenges in community-based practitioners, who have variously been portrayed as having negative attitudes toward EBPs, or having insufficient time, training, access, skills to review scientific literature, or other resources related to implementing these practices with fidelity. Yet as experienced by CATS program staff, TF-CBT implementation challenges stemmed from the ways the intervention fit with client needs. The way program staff used TF-CBT components is consistent with APA’s current definition of evidence-based psychological practice (EBPP) as “the integration of the best available research with clinical expertise in the context of patient characteristics, culture, and preferences” (APA 2006, p. 273). This definition mirrors the Institute of Medicine conceptualization of evidence-based practice in medicine where individual practitioners are made aware of the evidence and use their own judgment in how to combine it with the individual characteristics, symptoms, and situations of their clients (IOM 2001). Interestingly, both are practice-centered rather than intervention-centered in their conceptualization of evidence-based health care, and highlight the relational nature of practice by emphasizing practitioner responsiveness to patient values and preferences.

Therefore, the adaptations made to TF-CBT were consistent with CATS own model of services and prevailing notions of quality health care practice, though perhaps not prevailing conceptions of evidence-based interventions as discrete structured protocols demanding adherence. Tensions between adapting structured interventions and implementing them as designed may be of particular concern to practitioners serving immigrant and ethnic minority groups since few evidence-based interventions have been developed for them, and generalizing evidence collected in large efficacy trials where these populations are underrepresented is problematic (Bernal and Scharron-del-Rio 2001; Kataoka et al. 2010). CATS staff handled these tensions by prioritizing consistency with their service model over rigid implementation of TF-CBT, and were able to help students in the process. It is unclear, however, how alternate ways of integrating TF-CBT with the CATS service model would have affected outcomes and more research is needed to understand the ways in which complex issues related to fidelity and adaptation are negotiated by community-based mental health providers.

Implications for Generating Practice-Based Evidence

Consistent with previous research, our findings suggest that it is both productive and feasible to collect practice-based evidence on techniques or strategies that include evidence-based interventions as part of an array of possible services (Chorpita et al. 2007; Garland et al. 2008). This approach to studying the effectiveness of community-based mental health services has the potential to generate evidence on how evidence-based interventions are being integrated with other types of services, to what effect, in which combinations, and for whom (Brookman-Frazee et al. 2009; Garland et al. 2010). Since we know that conceptualizing mental health services as treatment elements reflects actual practice, developing and evaluating mental health interventions as constellations of treatment elements has the potential to link research and practice in new ways.

An important problem faced by community agencies that try to add evidence-based interventions to their practice is that their existing service models are complex, individualized, and rarely explicit (Birman et al. 2008). Though “usual care” is often used as a control condition in treatment effectiveness studies, its service components are rarely assessed or described. In this study it is unclear to what degree elements of the overall CATS model (flexibility, responsivity, clinical decision making) contributed to the effectiveness and successful integration of new treatment components, such as TF-CBT. Our data suggest that positive outcomes are enhanced when service components are used in combination, as evidenced by the finding that students experienced greater functional improvement when TF-CBT was provided in combination with CBT and supportive therapy. Describing and studying the effectiveness of practice as usual increases opportunities to learn from clinical wisdom, disseminate relevant information on best practices, and understand how to integrate evidence-based treatments with effective local service models.