Introduction

Identifying processes associated with positive outcomes for youths placed in residential treatment for emotional and behavioral problems is of paramount interest to researchers and practitioners. Despite the persistent and even increasing need for out-of-home placements for hundreds of thousands of children (Connor et al. 2004), controversy remains regarding residential treatment given issues of separating the child from his or her family, the considerable costs (Hoagwood and Cunningham 1992), and questions about long-term effectiveness (Barth et al. 2007; Hair 2005).

In a recent meta-analysis including studies conducted between 1990 and 2005 on the outcomes of residential treatment, Knorth et al. (2008) found that, on average, youths who received residential care showed improved psychosocial functioning and reduced problem behavior. The authors aptly noted that although investigating whether or not residential treatment “works” is important, it is even more critical to identify how it might work. Researchers and community partners must identify collaboratively the theory of change within residential treatment to assess how positive outcomes are achieved (Hair 2005). This involves moving beyond describing a series of outcomes to empirically testing relationships between interventions and outcome variables.

Over the past several decades, residential treatment has shifted from a model of providing custodial care to a more ecological focus on teaching skills (Guterman and Blythe 1986; McCurdy and McIntyre 2004). There is some evidence to suggest that behavior modification and specific training of social-cognitive and social-emotional skills in residential treatment achieve the most positive results (Armelius and Andreassen 2007; Knorth et al. 2008; Lipsey 1995), but research is needed to assess the specific processes (e.g., staff behavior) that relate to the development and maintenance of important skills for youths (Knorth et al. 2008). It is also important to identify individual variables that may mediate the relationships between staff actions and youth outcomes to better understand the process by which interventions exert their effects. The purpose of this study was to test the residential treatment center program theory, which explicates a complex set of client characteristics and staff actions expected to have positive effects on youths’ outcomes. Specifically, the study was designed to assess the extent to which residential treatment staff’s cognitive and behavioral disputations at the mid-point of the youths’ stay predicted the youths’ problem-solving skills at discharge from residential treatment after controlling for violent behavior, academic achievement, and psychological symptoms. A second study goal was to assess the direct and indirect effects of engagement in treatment on problem-solving skills.

Problem-Solving and Aggression

Effective problem-solving requires individuals to identify and define problems, generate alternative solutions, select and implement a solution, and evaluate the outcome (D’Zurilla and Nezu 1990). A number of researchers have demonstrated the importance of problem-solving for positive life outcomes, such as reduced stress (D’Zurilla and Sheedy 1991), increased adaptation (D’Zurilla and Nezu 1990), and acquiring and maintaining relationships with peers (Richard and Dodge 1982). In addition, problem-solving skills have been found to predict academic performance (Rodriguez-Fornells and Maydeu-Olivares 1999).

Problem-solving deficits are common among children and adolescents who display aggressive behavior and other conduct problems (D’Zurilla and Nezu 1990; Jaffee and D’Zurilla 2003; Rabiner et al. 1990; Richard and Dodge 1982). Youths with aggressive behavior generate fewer alternative solutions to problems (Lochman and Dodge 1994; Rabiner et al. 1990; Richard and Dodge 1982), evaluate aggressive alternatives less negatively (Orobio de Castro et al. 2005; Quiggle et al. 1992), and enact more aggressive behavioral responses than nonaggressive comparison groups (Dodge and Frame 1982). Richard and Dodge (1982) found that aggressive boys were able to generate effective initial solutions, however, when asked to generate subsequent solutions, these boys offered aggressive and ineffective solutions characteristic of their behavior. These findings suggest that problem-solving skill deficits and aggressive behavior are interrelated, which has important implications for residential treatment centers that serve youth who often display clinical levels of aggression (Baker et al. 2007).

Cognitive and Behavioral Disputations

Cognitive–behavioral interventions are the approach most commonly used to target the problem-solving deficiencies of aggressive and antisocial youth (Ronan and Kendall 1990). These interventions, which use modeling, behavioral rehearsal, coaching, cognitive restructuring, relaxation, anger management, or social skills training have been shown to be effective in reducing anger and aggression among clinical populations such as youths with conduct disorder (Beck and Fernandez 1998; Bennett and Gibbons 2000; Kazdin et al. 1989; Williams et al. 2004). Cognitive–behavioral interventions appear to produce generalized improvements for antisocial youth (Lochman et al. 1984), and such gains seem to be maintained over time (Bennett and Gibbons 2000). Cognitive functioning and severity of clinical dysfunction, however, have been found to moderate the relationship between cognitive–behavioral interventions and outcomes (Kazdin and Crowley 1997).

The aforementioned studies assess the effectiveness of cognitive–behavioral treatments implemented by a trained therapist over a specified number of treatment sessions. In residential treatment, the multidisciplinary team is responsible for implementing the treatment plan, and it has been suggested that the direct care professional may be the team member who has the greatest impact on the youth (Leichtman et al. 2001; McCurdy et al. 2000). Indeed, the direct care staff who work with individuals on a daily basis are increasingly using cognitive–behavioral and skills training interventions as part of their work (Corrigan et al. 2001; McCurdy and McIntyre 2004). Therefore, it is important to study the relationship between the cognitive–behavioral interventions as implemented by all staff in residential treatment and youth behaviors. The specific cognitive–behavioral intervention of interest in this study was staff’s use of cognitive and behavioral disputations, which involves helping youths challenge their beliefs, examine costs and behaviors of actions, and think about different ways of doing things.

Engagement

Engagement in treatment involves many attitudinal, behavioral and affective components. These components include (a) client attitude about, motivation for, and expectations regarding treatment; (b) the relationship between treatment staff and client; and (c) client behaviors, such as cooperation, participation, and effort (Cunningham et al. 2009). Engagement encompasses other complex concepts, such as readiness to change, which includes attitudinal and behavioral components (see Prochaska et al. 1992) and the therapeutic alliance, which includes affect as well as behavior, such as collaboration in the treatment process (see Martin et al. 2000). Different aspects of engagement have been found to relate to treatment outcome. For example, individuals in Prochaska et al. (1992) precontemplation stage are more likely to prematurely terminate treatment than individuals in the preparation and action stages (Smith et al. 1995). In addition, individuals who show significant change in target behaviors are more likely than those who do not show behavior change to be contemplators (Dozois et al. 2004; Prochaska et al. 1992).

Smith et al. (2008) found that youth who were engaged early in a residential treatment program were more likely to receive interventions (e.g., exposure to new experiences, cognitive–behavioral interventions) and to experience more positive outcomes at discharge (e.g., family trust, self-efficacy, school attachment) compared to youths with lower levels of early engagement. These same researchers also found, however, that less engaged youth showed greater positive changes in family trust and self-esteem over treatment than more engaged youth. A meta-analysis conducted by Shirk and Karver (2003) revealed that the therapeutic relationship played a consistent, modest role in child and adolescent treatment outcomes across various treatment modalities. The therapeutic alliance had its strongest effects on global functioning (as opposed to specific problems) and when the treatment provider (as opposed to the child) rated the alliance.

Present Study

The goal of this study was to examine the association between a specific cognitive–behavioral intervention, namely cognitive and behavioral disputations, and problem-solving skills of youth in residential treatment. Because past research has demonstrated a relationship between problem-solving and aggressive behavior (Rabiner et al. 1990; Richard and Dodge 1982), academic achievement (D’Zurilla and Nezu 1990), and social and behavioral adjustment (Shure and Spivack 1972; Shure et al. 1971), violent behavior, academic achievement, and psychological adjustment were also included in the model. In addition, engagement in treatment was explored to assess the extent to which it influenced problem-solving skills directly and indirectly (through its impact on cognitive and behavioral disputations).

This study is part of a larger research initiative called Service Outcome Action Research (SOAR). SOAR is a research partnership of the University at Albany School of Criminal Justice, School of Social Welfare, and School of Education, and two residential treatment centers (RTCs), La Salle School and St. Anne Institute. About 50% of the youths were adjudicated as persons-in-need-of-supervision and another 30% have been adjudicated delinquent. The other 20% were admitted through either child protective services or school district committees on special education. Treatment provided at the RTCs includes educational services, individual therapy focused on client strengths, group therapy, behavior modification in residence, substance abuse treatment, and discharge planning. This research partnership was formed so that university researchers could assist agency staff members in identifying and measuring client outcomes and implementing evidence-based practices in their programs.

SOAR uses a theory of change approach to evaluation (Patton 1989; Rossi et al. 2004) that involves a collaborative process beginning with the construction of the residential program logic model. A theory of change approach allows researchers to construct a program model that captures stakeholders’ ideas about how their program works as opposed to constructing an academically driven, standardized model that attempts to fit a program to it. A theory of change approach identifies what is happening in a specific program, shifting the emphasis from generalizability to utility (Patton 1989). Researchers can then generate information about the specific program to identify discrepancies between this intended model and actual practice to provide stakeholders with a more complete understanding of the central components of their program. This approach provides agencies with information about their program performance but also addresses important questions about program processes and the structure of service delivery (McClintock 1984).

The SOAR residential program logic model was developed by first conducting 44 semi-structured interviews with RTC staff members, which were inductively coded for themes regarding client characteristics, staff actions, and treatment outcomes (Strauss and Corbin 1998). The project committee then organized these themes and articulated the practitioners’ program theory through a logic model that identified expected relationships between youth and family intake characteristics, staff processes and actions, and outcomes. University faculty then developed a means of collecting information about the types of clients being served, the specific interventions being provided and relevant outcomes, which all reflected components of the logic model. It should be noted that the logic model contained very detailed relationships between different staff actions, client characteristics, and outcomes to allow for specific analyses to be conducted to attempt to specify which actions were associated with which outcomes. Therefore, although a variety of interventions were implemented at the RTCs, the focus of this study was on the specific intervention of cognitive and behavioral disputations.

The logic model indicated that, similar to the literature, practitioners expected that the specific intervention, cognitive and behavioral disputations, would have a positive effect on youths’ problem-solving. However, agency staff also asserted that engagement is a necessary condition for treatment progress. Client engagement, the staff expected, would affect the types of treatment services and the level of services the youth received. More engaged clients would receive more effective treatment, which would result in better treatment outcomes, such as improved problem- solving. The hypotheses included: (a) youths exposed to more cognitive and behavioral disputations by staff will have improved problem-solving skills at discharge; (b) the relationship between cognitive and behavioral disputations and problem-solving will remain after key intake and early treatment variables are controlled; and (c) client engagement will be directly related to problem-solving skills at discharge (with this relationship mediated by staff cognitive and behavioral disputations).

The theoretical rationale for these hypotheses is that youths with emotional and behavioral problems have cognitive distortions that exacerbate their aggressive behavior. By intentionally disputing these cognitions and the associated behaviors, it was expected that youths would recognize their cognitive distortions and ineffective solutions to problems, think them through, give them up, and replace them with less aggressive alternatives (Ellis and Grieger 1977). Thus, it was expected that cognitive and behavioral disputations would lead to a change in the ability to problem-solve. However, it is likely that a youth’s engagement in treatment will make a difference in terms of the effectiveness of this intervention. Engaged youth are likely to get more from the intervention because they will have established a working relationship with residential treatment staff, will acknowledge their problem solving deficits, accept responsibility for these deficits, and express a commitment to improving their problem-solving skills.

In order to specify clearly and comprehensively how staff actions relate to client characteristics and outcomes, we used youth, childcare staff, and clinician measures of cognitive and behavioral disputations and engagement. A growing body of literature has suggested that using multiple measurement sources is a best practice and generally provides more complete information (Capaldi et al. 1996).

Method

Data from the current study were obtained from the residential treatment pilot study, which used a multi-informant, multi-source measurement strategy to collect data to measure each aspect of the logic model for 130 youths whose caregivers gave consent. The study and interview schedule were reviewed and fully approved by the Institutional Review Board of the university. Interviews, administered by 10 graduate students trained by the project co-investigator, were conducted at four points in time: 4 weeks after admission (Wave 1), the projected midpoint of the youth’s stay (Wave 2),Footnote 1 immediately prior to discharge (Wave 3), and 4 months after discharge (Wave 4). Data were also gathered from the youths’ (a) primary caregivers at Waves 1 and 4; (b) education staff, clinician, and childcare staff members at Waves 2 and 3; and (c) case files and school records. For the current study, constructs of interest were measured using youth reports at Wave 1, 2, and 3, and clinician and childcare staff reports from Wave 2 and 3.

Participants

Participants were approached by the intake staff at the two agencies to determine their interest in the study and to obtain the necessary consent. Only participants who had complete data for all measures utilized in the current study were included.Footnote 2 Of the original 130 youth participants, 105 remained at Wave 2, and 94 remained for interviews at Wave 3. Although every attempt was made to keep youths in the study, five refused to participate after the first interview, 20 were Away without Leave (i.e., leaving the RTC without authorization from the treatment team), four were in jail, five were discharged early by the referral county, and two could not be located for unknown reasons.

For each set of respondents, the resulting sample size after missing data were imputed varied, but the overall sample study included 59 youths for whom complete data were available for every measure in the study by all informants (i.e., youths, as well as childcare staff and clinicians for cognitive and behavioral disputations and problem-solving).Footnote 3 In order to determine if there were any differences between the youths that remained in the study and those who were excluded, independent sample t-tests were estimated on youth Wave 1 academic achievement, problem-solving, violent behavior, psychological symptoms, and engagement in treatment. The only significant difference was in engagement (p = .05), where those that were excluded from the analyses scored an average of 7 points lower than those who remained. Because the excluded sample consisted of many youths who were AWOL, it is likely that these youths were not highly engaged in treatment.

Of the youth respondents, 54.7% were male, and 46.7% were White/Caucasian. The average age of youths responding to the interviews was 15.51 years and the average length of stay in treatment was 9.37 months. No demographic information was available for the staff, as they completed their questionnaires under anonymity with the exception of their job type.

Measures

Items for each measure used in this study, as well as internal consistency coefficients for each, are provided in Appendix 1. Although a number of standard measures with established reliability and validity exist for some of the scales used in the SOAR analysis, the nature of the study made it difficult to use most of them. The agencies were restrictive as to how much time researchers had available to them for the interviews and the need to ask questions about a number of subjects necessitated the editing of some measures and the creation of others.

All SOAR scales were determined to be acceptable through a three stage process: confirmatory factor analysis using the maximum likelihood estimation function in SPSS, reliability determined by Cronbach’s alpha, and an examination of the standard deviations of items in each scale to determine if the differences in variances between items were too large (the scale development information is described more fully in Cunningham et al. 2009). If the proposed scale fulfilled all the measurement criteria, the items were summed to create the scale. Because the youths often have interaction with a number of staff members, youths were instructed to answer questions based on the staff in general.

Youth Measures

Cognitive and Behavioral Disputations

The key index of treatment fidelity in this study was measured with an 8-item cognitive and behavioral disputations scale developed by SOAR staff based on the logic model interviews in which staff described how they delivered services and what effects they expected as a result. This scale measured the youth’s perceptions about how the staff members recognize and challenge negative thoughts and behaviors during Wave 2 (projected midpoint of stay). All items were rated on a 7-point scale ranging from Not at all to Always or from Strongly Disagree to Strongly Agree. Higher scores corresponded to greater provision of cognitive and behavioral disputations.

Youth Engagement in Treatment

This 17-item youth scale measured the attitudinal (i.e., “readiness to change”), relational (affective relationship with treatment provider), and behavioral (e.g., participation, cooperation, effort) aspects of engagement as reported by the youth on a 7-point scale from Strongly Disagree to Strongly Agree. The attitudinal aspect was adapted from the University of Rhode Island Change Assessment Scale (URICA, n.d.; McConnaughy et al. 1983) and the relational and behavioral items were adapted from the Working Alliance Inventory (Horvath and Greenberg 1986, 1989). The scale had strong internal consistency and content validity, modest construct and criterion validity, and confirmatory factor analysis revealed a single underlying factor of engagement (Cunningham et al. 2009). This variable was measured at the youth’s projected midpoint of stay (Wave 2). A higher score corresponded to a higher level of engagement in treatment.

Problem-Solving

Due to issues involved in completing a study over time, there was no scale measure of youth’s problem-solving at intake (Wave 1). In order to control for the youth’s ability to problem-solve prior to treatment, a single item measure asking how much they agreed or disagreed with the statement “When I do not reach an answer to a problem the first time, I persist in seeking solutions” was used. The dependent variable, a problem-solving scale prior to discharge (Wave 3), required the youth to answer three questions about their own problem-solving (e.g., comparing alternatives, evaluating decision) on a 7-point scale from Strongly Disagree to Strongly Agree. A higher score on both of these measures corresponded to greater problem-solving ability.

Psychological Symptoms

The 8-item psychological symptoms scale asked youths to report, on a 7-point scale from Not at all to Always how often in the past month they experienced internalizing symptoms (e.g., overtired, depressed, lonely, difficulty eating or sleeping). This scale was administered at Wave 1. Higher scores represented more reports of symptoms and greater distress.

Violent Behavior

This measure, adapted from the Rochester Youth Development Study (see Pogarsky et al. 2003), included a count of the violent actions the youth reported committing in the 3 months prior to entering residential treatment (Wave 1). It consisted of five Yes or No questions asking whether or not the youth had been involved in the acts such as hitting someone and attacking someone with a weapon. A higher score represented more violent behavior.

Academic Achievement

Academic achievement was assessed via the youth’s report of academic achievement in the school term prior to admission at the agencies (Wave 1). This was a single item ranging from A student (1) to F student (5).

Staff Measures

To avoid single source bias, we used staff measures of cognitive and behavioral disputations and staff perception of youth engagement in treatment at Wave 2 to provide a validity check. These responses were gathered from two of the staff members: child care and clinical.Footnote 4 These scores were averaged to create the staff members’ perception of these two measures. The measure of cognitive and behavioral disputations used the same eight questions as those posed to youth. The measure of youth engagement in treatment was a 15-item measure that overlapped with the youth measure. As with the youth scales, higher scores on these measures represented greater amounts of each variable.

Analysis Plan

The first step of the analysis included a calculation of descriptive statistics for all variables. Next, bivariate correlations between measures were calculated to assess relationships between all variables and informants. Finally, a limited information estimate approach in a traditional path analysis using SPSS was conducted. Two models were estimated, one without engagement in treatment from all responders (youth and staff average) and one with engagement from all responders (youth and staff average).

Results

Descriptive statistics for all variables for youths included in the analysis are displayed in Table 1. The mean number of violent behaviors reported by the youths in the 3 months prior to admission in residential treatment was 1.56 (SD = 1.25) and the mean score on the psychological symptoms scale was 16.41 (SD = 11.57) out of a possible high score of 45. For academic achievement in the term preceding residential treatment, youths reported a mean of 2.77 (SD = 1.12), which equated to the average youth in treatment reporting receiving Bs or Cs in school.Footnote 5 The means for engagement in treatment, cognitive and behavioral disputations, and problem-solving all fell within the approximate midpoint for each scale (see Table 1). Although there is a large difference between the means of problem-solving at Wave 1 and Wave 3, much of this is due to the fact that the Wave 1 measure consisted of only one item while the Wave 3 measure had three items.

Table 1 Descriptive statistics for measures
Full size table

Bivariate correlations were conducted for all variables (see Table 2). For youth reports, the Wave 1 measure of problem-solving was only significantly related to psychological symptoms (r = .29) and the youth report of cognitive and behavioral disputations at Wave 2 (r = .28). Violent behavior and academic achievement did not correlate significantly with any other variable. Cognitive and behavioral disputations as reported by youth at Wave 2 were positively correlated with problem-solving at Wave 3 (r = .32). In contrast, the measure of cognitive and behavioral disputations from the staff was negatively correlated with youth’s problem-solving at Wave 3 (r = −.26). In addition, there was no significant correlation between the staff average and youth report of cognitive and behavioral disputations.

Table 2 Bivariate correlations-youth and staff average reports
Full size table

Engagement at Wave 2, as reported by the youth, was significantly positively correlated with cognitive and behavioral disputations at Wave 2 (r = .74) and problem-solving at Wave 3 (r = .52). It was also significantly related to the staff average rating of the youth’s engagement in treatment (r = .49), suggesting a high level of agreement between the two sources. The measure of staff rating of youth engagement was also significantly positively related to problem-solving at Wave 3 (r = .31).

Results for the multivariate analyses are provided in Figs. 1 and 2. For all models, the standardized slope is provided, with the unstandardized scores provided in parentheses. Error residuals were provided in the circles. The first model (see Fig. 1) predicting problem-solving with cognitive and behavioral disputations was not significant (F = 2.02, p = .080). Figure 2 presents the model that predicted problem-solving with cognitive and behavioral disputations and engagement in treatment. The overall model was significant (F = 3.15, p < .01). While measures of engagement from both the youth and the staff average significantly predicted the youth’s report of cognitive and behavioral disputations, they did not predict the provision of cognitive and behavioral disputations as reported by the staff. The youth’s report of his or her own engagement in treatment was the only significant predictor of problem-solving at Wave 3.

Fig. 1
figure 1

Effects of cognitive and behavioral disputations on problem-solving, youth and staff average report

Full size image
Fig. 2
figure 2

Effects of cognitive and behavioral disputations on problem-solving controlling for the effects of engagement in treatment, youth and staff average report

Full size image

Discussion

The study tested the residential treatment center program theory that predicted a relationship between cognitive and behavioral disputations, client engagement, and youth problem- solving at discharge. Cognitive and behavioral disputations, as reported by the youth and the staff, did not predict problem-solving at Wave 3 when youth intake characteristics, including intake measures of violent behavior, academic achievement, and psychological symptoms, were considered. When engagement in treatment was added to the model, both youth and staff measures of engagement predicted the youth’s report of cognitive behavioral disputations, however, they did not predict the provision of cognitive and behavioral disputation as reported by the agency staff. Engagement in treatment, as reported by the youth, was the only significant predictor of problem-solving at Wave 3.

Cognitive–behavioral interventions are often used with youth in residential treatment to target problem-solving deficiencies (Ronan and Kendall 1990). Findings from this study, however, did not reveal a direct or indirect relationship between one specific cognitive–behavioral intervention, namely cognitive and behavioral disputations, and youth problem-solving skills. There may be several reasons for this. First, although the model was not significant, there were significant bivariate correlations between youth report of cognitive and behavioral disputations and later problem-solving and significant inverse bivariate correlations between staff report of cognitive and behavioral disputations and youth’s report of problem-solving. These findings may suggest that this intervention is perceived differently. Perhaps youth perceive having their thinking challenged as helping them to problem-solve, whereas staff who report doing this more with youth is associated with less independent problem-solving. This may leave open the possibility that the more staff dispute youth cognitions and behavior, the more youth will perceive that they do not have this skill. Another issue relates to the fact that cognitive–behavioral intervention involves an entire repertoire of interventions such as anger management, behavioral rehearsal, coaching, cognitive restructuring, modeling, relaxation, and social skills training (Beck and Fernandez 1998; Bennett and Gibbons 2000; Kazdin et al. 1989; Williams et al. 2004). In this study, one specific component (cognitive and behavioral disputations) was investigated; this aspect alone may not be associated with improved problem-solving. Perhaps including other aspects of CBT, such as skill teaching and reinforcement of newly learned behaviors would yield different findings. Finally, it is possible that the small sample size in the study contributed to the lack of statistically significant findings.

This study’s results also indicate that more engaged youths reported that they received more of the intervention. It may be that the youth reports of cognitive and behavioral disputations is a proxy measure of engagement in that clients see their relationship with the treatment provider and collaboration on goals and tasks as an indication of the level of intervention that they are receiving. Both youth and staff measures of treatment engagement, however, did not predict the provision of cognitive and behavioral disputations as reported by the agency staff. This is consistent with current agency policy that staff members provide interventions to all clients, and not only those that demonstrate high levels of engagement.

Youth report of engagement predicted problem-solving, as a great deal of treatment theory hypothesizes (Dozois et al. 2004; Prochaska et al. 1992; Smith et al. 1995). However, staff report of engagement did not predict problem-solving after controlling for relevant intake characteristics including problem-solving at wave 1. This different pattern of findings for staff and youth informants may be attributable to characteristics of engagement. Perhaps staff members are able to report on the youth’s observable participation in treatment, collaboration, and effort, but they may find it more challenging to report on clients’ attitudes about treatment and affective relationship with providers. These attitudinal and relational aspects of engagement (Cunningham et al. 2009), which are not necessarily observable, may be more accessible to youth.

Implications for Practice

Findings from this study underscore the importance, from the youth perspective, of engagement in treatment and suggest that practitioners should use techniques that have been found to increase client engagement (e.g., motivational interviewing; Miller and Rollnick 2002). In particular, creating a bond with youths and collaborating on goals and tasks appear to be related to youths’ problem-solving skills (Martin et al. 2000). The present study supports Shirk and Karver’s (2006) contention that a critical part of transporting evidence-based interventions to real-world treatment settings requires complementing the intervention with proven strategies to engage youths because in this study, it appears that engagement results in improved problem-solving, while the intervention actually does not.

In addition, agencies may profit from identifying and selecting youth who are more likely to engage in treatment because knowing the youth’s stage of change may allow the agency to better predict client outcomes (Dozois et al. 2004; Prochaska et al. 1992; Smith et al. 1995). Similarly, specific problem-solving treatment outcomes might vary with the proportion of admitted youth who are more likely to engage in the process (Shirk and Karver 2003).

Findings from this study highlight the importance of using multiple measurement sources (Capaldi et al. 1996). Staff demonstrated different findings from youth that may not have been identified if only youth reports were used. Because it was not clear or even true that one source was more accurate that others, utilizing a multi-informant approach provided the most complete information. Of course, the differing perspectives of youths and staff members also have implications for practice. These differing perspectives are to be expected, especially since youth reported on staff behavior in general whereas each staff member reported on his or her own behavior. However, it is important that youth see a relationship between their engagement in treatment and problem-solving, whereas staff members’ report of youth engagement is not related to youth report of their own problem-solving. Discussion about change and the possible reasons for change should be a part of treatment planning. Also important is the conceptualization of how cognitive and behavioral disputations are used in treatment and whether they may be linked to improved problem-solving or decreased problem-solving. Perhaps it is not the level of intervention that is important, but the way this intervention is implemented.

Limitations

As with all research, this study had some limitations, including measurement. Although we controlled for youths’ problem-solving at the beginning of treatment, this analysis is hindered by a problem with measurement equivalence. Youth report of problem-solving at Wave 1 was assessed by only a single item due to problems with the full measure originally used. Because it was important to find a good assessment of problem-solving for the agencies to use in their permanent data collection, a different three item problem-solving indicator was completed by the youth in Wave 3. Therefore, the items assessing problem-solving for youth are not the same across waves. Although youth report of Wave 1 problem-solving was used as a control, this may be an inadequate baseline measure because of its scale quality and unevenness across waves. Nevertheless, the spirit of the questions assessing problem-solving by youth is similar at Wave 1 and Wave 3, and the response options are identical. In addition, the measure used to assess academic achievement was not as good as a standardized measure of academic achievement. None of the intake variables predicted problem-solving and this may be due to the inadequacy of these two measures.

Another limitation is that youths were receiving several other interventions while in residential treatment (e.g., family therapy, school interventions) that likely had an impact on outcomes. Also, because this was not an experimental study, we cannot state conclusively that engagement in treatment or the intervention led to problem-solving skills. Additionally, the sample size was modest, and there was some attrition from the study. Attrition is problematic because the association between cognitive–behavioral interventions and problem-solving behavior may have been overemphasized with the exclusion of these clients who may have continued to have severe problem-solving deficits after terminating treatment.

Directions for Future Research

There are several directions for future research. Further specifying how clients become engaged and what staff actions facilitate this process is an important area of study. In addition, further study of the specific aspects of cognitive–behavioral interventions that may affect problem-solving is warranted. As mentioned previously, perhaps this one specific component of cognitive–behavioral interventions operates differently when not studied in conjunction with other aspects of the intervention. Although we controlled for several intake variables, there is another variable that has been related to problem-solving that we were unable to assess in this study, specifically locus of control (D’Zurilla and Nezu 1990). Future research should examine the role of locus of control in problem-solving skills.