Introduction

The focus of much psychotherapy research over the past few decades has been on defining, evaluating, and cataloguing evidence-based treatments (EBTs; e.g., Chambless and Hollon 1998; Ollendick et al. 2006). EBTs have been shown to be effective in treating a wide range of childhood and adolescent disorders, including disruptive behavior disorders (DBDs), relative to no treatment (Weisz et al. 1987, 1995), and even relative to usual care (Weisz et al. 2006). A recent review article identified no less than 16 different psychosocial EBTs for children and adolescents with disruptive behaviors (Eyberg et al. 2008). Examples include Parent Management Training (Patterson et al. 1975), Problem-Solving Skills Training (Kazdin 2003), and the Incredible Years program (Webster-Stratton and Reid 2003).

Skill Acquisition and Utilization

The acquisition and utilization of skills is central to many behavior therapy (BT) and cognitive-behavioral therapy (CBT) treatments, including treatments for disruptive behavior disorders. For example, in Parent–Child Interaction Therapy (PCIT; Funderburk and Eyberg 2011; McNeil and Hembree-Kigin 2011), parents must demonstrate mastery of a set of skills before moving on to the next phase of treatment and graduating from treatment. Therapists are trained to assess mastery of skills by coding parent–child interactions using a behavior observation coding system during each treatment session. Parents are also given homework sheets to report on their daily skill-use between treatment sessions. The purpose of this review and meta-analysis is to summarize the recent literature (since 2000) on skill acquisition and utilization, constructs that are central to psychosocial treatments for disruptive behavior.

Definitions

There are no standard definitions of skill acquisition or skill utilization, and other terms have been used to describe similar constructs. For example, the term treatment receipt is similar to skill acquisition. Bellg and colleagues define treatment receipt as, “the degree to which the participant understands and demonstrates knowledge of and ability to use treatment skills” (Bellg et al. 2004, p. 444). Likewise, the term treatment enactment is similar to skill utilization. Treatment enactment is defined by Bellg and colleagues as, “the degree to which the participant applies the skills learned in treatment in his or her daily life” (p. 444). For the purposes of this review and meta-analysis, we use the terms skill acquisition and skill utilization, as these terms are more commonly used in the DBD literature (e.g., Lyon and Budd 2010; Nash et al. 2003; O’Callaghan et al. 2003; Shanley and Niec 2010; Thornberry and Brestan-Knight 2011; Timmer et al. 2011; Ware et al. 2008).

Skill Acquisition

We defined skill acquisition as the degree to which a parent/child can perform a skill learned during the course of treatment. This definition includes only the formal assessment of a skill by an “expert” rater (e.g., therapist, clinician, and assessor) using a rubric, checklist, behavior observation, or rating scale. It does not include a parent’s self-report on a parenting measure (e.g., Parenting Practices Inventory or Alabama Parenting Questionnaire). An example of a measure that meets our definition of skill acquisition is the Dyadic Parent–Child Interaction Coding System (DPICS; Eyberg et al. 2005). The DPICS is an observational coding system used to record the frequency of encouraged behaviors and discouraged behaviors that are taught in PCIT. For example, Bagner et al. (2010) used the DPICS to measure skill acquisition in both treatment and control groups. In their study, trained coders observed 5-min parent–child interactions and recorded “do skills” (behaviors encouraged in PCIT: behavior descriptions, reflections, and praises) and “don’t skills” (behaviors discouraged in PCIT: questions, commands, and negative talk).

Skill Utilization

We defined skill utilization as the degree to which a parent/child uses a skill in his/her daily life outside of treatment sessions. This might include the practice of a skill assigned as “homework” or “home practice” by the therapist, but this is only a very small piece of the definition. Rather, the term refers more generally to skill-use that is initiated by the parent/child on a day-to-day basis. An example of a study that meets our definition of skill utilization is Leathers and colleagues’ KEEP study (Leathers et al. 2011). Parent skill utilization was measured using a telephone interview. Foster parents were asked to report on the number of times they had used intervention skills in the past 3 days. The KEEP intervention skills that were assessed included praise and specific discipline techniques (e.g., time-out, privilege loss, and work chores).

Rationale for a Review and Meta-analysis

Whereas there is a growing research literature on predictors and moderators of treatment outcome (e.g., symptom severity, comorbid conditions, and parent psychopathology), less is known about specific mechanisms through which these variables exert their influence on the course of treatment. From behavior therapy (BT) and cognitive-behavioral therapy (CBT) perspectives, these variables may influence the course of treatment via several mechanisms, including (a) participation/engagement, (b) skill acquisition, (c) skill utilization, and (d) maintenance of cognitive and behavioral changes. In order to improve treatment outcome, therefore, it is critical to measure these constructs in order to pinpoint barriers to treatment progress. Traditional randomized clinical trials have focused on assessing outcome in terms of symptoms, diagnoses, and functioning, with less attention to measuring these theoretically central intermediate processes (Nock and Ferriter 2005). A review and meta-analysis of skill acquisition and utilization is needed to guide future research and refine existing treatments for childhood behavior problems. Although numerous reviews and meta-analyses have focused on DBD treatments (e.g., Eyberg et al. 2008; Kaminski et al. 2008; Reyno and McGrath 2006), this is the first to focus exclusively on the constructs of skill acquisition and utilization. The most closely related prior reviews are perhaps studies by Kaminski et al. (2008) and Boyle and Lutzker (2011). The Kaminski et al. (2008) study is a component analysis, with a focus on treatment content and delivery method, but includes skill acquisition as one of several outcomes for a subset of the studies. Boyle and Lutzker (2011) examined the generalization (i.e., skill utilization) of parent training skills in 16 studies. However, their review did not estimate effect sizes or include studies on the generalization of child skills. In contrast to these prior reviews, the purpose of the current review and meta-analysis was to focus explicitly on skill acquisition and skill utilization for both parent skills and child skills and to estimate effect sizes.

Current Study

The aim of the current study was to summarize the recent literature (since 2000) on skill acquisition and skill utilization for psychosocial DBD treatments. We chose 2000 as our cutoff to focus on the most recent decade (approximately) and thereby reflect the current state of the field. The aim of the review was to summarize (1) measurement approaches; (2) predictors, correlates, and outcomes examined in relation to these constructs; and (3) the frequency with which these constructs are specific targets of intervention. This information will be useful for identifying knowledge gaps and guiding future areas of research. The goal of the meta-analysis was to estimate effect sizes for skill acquisition and skill utilization. This information has the potential to inform treatment refinement (e.g., which skills are not being learned well by parents and children?) and generate hypotheses about specific mechanisms of change (e.g., which skills do parents and children actually use in their day-to-day lives?).

Method

Procedure

Articles were included in the review and meta-analysis based on the following criteria: (1) the target of treatment fell within the group of symptoms related to disruptive behavior disorders (e.g., externalizing behavior, behavior problems, conduct problems, and oppositional behavior); (2) the treatment was evidence-based (i.e., efficacy or effectiveness supported by at least one published study in a peer-reviewed journal); (3) the treatment was not primarily school-based; (4) there was a measure of skill acquisition and/or skill utilization; and (5) the measures were consistent with our definitions described above. As this article focuses on the recent literature, only articles published from January 2000 through January 2012 were included. A total of 85 articles were found and included in the review using these methods. Of these 85 articles, 68 were also included in the meta-analysis (see Fig. 1).

Fig. 1
figure 1

Study selection flow chart

Full size image

Study Selection

Studies included in the review and meta-analysis were identified through a systematic online database search of PsycINFO and Medline. The search was comprised of terms related to disruptive behavior disorders (DBDs) and skill acquisition or skill utilization. We began with an initial list of terms and added to this list as more terms were identified through the literature. DBD-related terms included: Disruptive Behavior Disorders, DBD; Conduct Disorder, CD; Oppositional Defiant Disorder, ODD; disruptive behavior, behavior problems, conduct problems, externalizing behavior, aggressive behavior, antisocial behavior, oppositional behavior, and noncompliance. Skill acquisition terms included: skill acquisition, skill mastery, and skill competence. Skill utilization terms included: skill utilization, skill use, skill practice, compliance, adherence, generalization, and skill implementation. DBD terms were crossed once with each skill acquisition term and once with each skill utilization term for a total of 154 permutations. The search yielded 9,172 possible articles. Our literature search also yielded a brief quantitative review on the generalization of parent skills learned in behavioral parent training programs (Boyle and Lutzker 2011), which gave us an additional 16 possible articles. Of the articles identified through our literature search and the quantitative review, 15 met our inclusion criteria and were included in the review.

For a more thorough search of the literature, we also manually searched through articles published in relevant journals. A list of possible journals to search was compiled prior to the literature search. The publication titles of the articles found through the database search were added to this list. We added only the journals in which at least one relevant article was found, as these journals would be likely to have published relevant articles. The following journal titles were searched: Behavior Therapy, Behavioral Disorders, Behavioral Interventions, Child and Family Behavior Therapy, Child Psychiatry and Human Development, Clinical Psychology Review, Clinical Psychology: Science and Practice, Education and Treatment of Children, International Journal of Child & Adolescent Health, Journal of Abnormal Child Psychology, Journal of Applied Behavior Analysis, Journal of Child Psychology and Psychiatry, Journal of Clinical Child and Adolescent Psychology, Journal of Consulting and Clinical Psychology, Journal of Emotional and Behavioral Disorders, Journal of Marital and Family Therapy, Journal of Pediatric Health Care, and Psychology in the Schools. Eleven articles meeting our inclusion criteria were found using this method.

A second online database search using PsycINFO and Medline was conducted using the names of DBD treatments and models found in our first 26 articles. Keyword searches of behavioral parent training, Fast Track, Incredible Years. Keeping Foster Parents Trained and Supported, KEEP, Multidimensional Treatment Foster Care, MTFC, Parent–Child Interaction Therapy, PCIT, Parent Management Training, PMT, Parent Management Training Oregon Model, PMTO, Positive Parenting Program, Triple P, Problem Solving Skills Training, and PSST yielded 1,358 articles. Fifty-nine of these articles met our inclusion criteria and were included in our review. Thus, our literature searches yielded a total of 85 articles on skill acquisition and/or skill utilization that met our inclusion criteria were included in the review.

Seventeen of the initial 85 articles were excluded from the meta-analysis because we were unable to calculate an effect size for skill acquisition and/or skill utilization based on the information provided in the article. For seven articles, skill acquisition and/or skill utilization was measured only once (e.g., at post-treatment) and the study did not have a control group, so no effect size could be calculated. For two articles, skill acquisition and/or skill utilization was measured during treatment sessions but not at post-treatment, so no effect size could be calculated. Five articles that measured skill utilization by tracking homework completion were also excluded. For these articles, homework completion (skill utilization) was measured for the treatment group throughout the intervention. No control group data and no pre- and post-data were collected, so no effect size could be calculated. For nine articles, statistics required for calculating an effect size (e.g., means and standard deviations) were not reported in the published article and the authors of these articles were contacted for additional data. The authors of six articles responded and were included in the analysis; the authors from three articles did not respond and were excluded. A total of 68 articles were included in the final meta-analysis.

Study Coding

Study coding methods were based on Lipsey and Wilson’s (2001) recommendations. The studies were coded using the coding procedures described in Appendices 1 and 2. The second author was trained by the first author in all coding procedures. Any questions or discrepancies were resolved through discussion between the first and second author. Individual skills within each study were coded by the first and second authors to assess reliability (88.4 % inter-rater agreement). Consistent with meta-analytic methods, multiple articles describing the same dataset were coded as a single study. We therefore had 68 articles but only 59 separate studies included in the meta-analysis. Study variables included in the meta-analysis were as follows (see Appendices 1 and 2 for a complete list of codes and operationalized definitions):

  1. 1.

    Research design descriptors: Total sample size, treatment group sample size, and control group sample size.

  2. 2.

    Nature of intervention descriptors: Intervention name, whether it was prevention or treatment, and type of control group (no treatment, waitlist, treatment as usual, alternative treatment, or no control group).

  3. 3.

    Skill descriptors: Whether the skill measured was acquisition or utilization, whether it was a parent skill or a child skill, and the specific category of skill (Positive Parenting, Negative Parenting, Praise, Verbal/Nonverbal Communication, Positive/Negative Consequences, Other Parent Skill, Emotion Labeling, Problem-solving, Anger management, Social Skills, or Other Child Skill).

  4. 4.

    Effect size descriptor: Whether the effect size was calculated for pre–post change or treatment–control difference (analyzed separately).

The studies were analyzed at multiple levels of specificity according to the above variables. In the first set of analyses, we examined effect sizes for overall skill acquisition and overall skill utilization. In the second set of analyses, we divided the effect sizes further and examined parent skill acquisition, parent skill utilization, child skill acquisition, and child skill utilization separately. In the final set of analyses, we examined effect sizes for parent/child skill acquisition and skill utilization for each of our 11 skill categories (Positive Parenting, Negative Parenting, Praise, Communication, Consequences, Other Parent Skill, Emotion Labeling, Problem-solving, Anger management, Social Skills, or Other Child Skill; see Appendix 2 for category definitions).

Analyses

Meta-analyses were conducted with Lipsey and Wilson’s (2001) SPSS macros. First, a meta-analytically derived mean effect size (d) was obtained for each outcome. Pre–post effects were analyzed separately from treatment–effect sizes. We relied on random effects models because this more conservative approach allows generalization beyond included samples (Rosenthal and DiMatteo 2001). A homogeneity statistic (Q) was derived based on these effect sizes to examine the amount of heterogeneity in the effect sizes. The Q statistic has a chi-square distribution based on k − 1 degrees of freedom, with k representing the number of effect sizes in the analysis. A statistically significant homogeneity Q statistic supports a heterogeneous distribution.

Results

Overview

The 85 articles that were included in the final review are summarized in Tables 1 and 2. Seventy-four (87 %) of the articles included in this review measured skill acquisition. A variety of skill acquisition measures were used, but the majority were observational (n = 60; 81.1 %). The rest included clinician report (n = 3; 4.1 %), self-report (n = 1; 1.4 %), or a test of knowledge/performance assessment to measure skill acquisition (n = 12; 16.2 %). A large number of these articles measured only the acquisition of parent skills (n = 64; 86.5 %) and did not examine child skills. Twenty-four (32 %) of the articles included in this review measured skill utilization. Types of measures used in these articles included observational (n = 7; 29.2 %), clinician report (n = 1; 4.2 %), self-report (n = 5; 20.8 %), parent report (n = 1; 4.2 %), test of knowledge/performance assessment (n = 1; 4.2 %), and homework (n = 10; 41.7 %). Articles measuring skill utilization also tended to assess parent skills (n = 21; 87.5 %) rather than child skills.

Table 1 Characteristics of articles measuring skill acquisition
Full size table
Table 2 Characteristics of articles measuring skill utilization
Full size table

Many of these articles examined skill acquisition and skill utilization as a primary measure of treatment outcome. While these articles reported parent and child competency in performance of target skills and/or frequency of use of target skills in their daily lives, these constructs were not studied as specific mechanisms of change. In other words, a large number of the articles we reviewed did not examine any antecedents/predictors of, outcomes associated with, or ways to enhance skill acquisition and utilization. Thirty-five (47.3 %) skill acquisition articles and 11 (50 %) skill utilization articles measured skill acquisition and utilization but did not further examine antecedents, correlates, or outcomes.

Methods of Assessment

Articles varied greatly in methods and procedures of assessment of skill acquisition and utilization (see Tables 1, 2). There was no discernable pattern in the frequency, duration, and sampling of assessments. For example, although many articles used observations to assess skill acquisition and utilization, the specific methods for observation were different across most articles. Observation periods ranged from 5 to 75 min long and they included both structured tasks and play interactions. Frequency of observations ranged from once or twice during pretreatment and/or post-treatment to every treatment session. Coding methods also varied and included coding of every utterance/parent–child exchange, selected nonverbal behaviors, a set number of parent instructions/demands, and behaviors occurring at specific time intervals (e.g., 10 s). Other types of assessments such as clinician report and self-report used several different factors (e.g., homework completion, session attendance, and general impressions of skill competency) to come up with composite ratings of skill acquisition and utilization. Homework and tests of knowledge/performance assessments were generally tailored to the specific treatment or intervention used in the study, so their assessment procedures were very different from each other.

Antecedents/Predictors of Skill Acquisition and Utilization

Only 13 skill acquisition articles and five skill utilization articles looked at antecedents and predictors of skill acquisition and utilization (see Tables 1, 2). A study by Nock and Kazdin (2005) looked at parent motivation and its relation to skill acquisition and utilization in PMT. Before treatment, parents were given a self-report measure of motivation (Parent Motivation Inventory [PMI; Nock and Photos 2006]) to assess their Desire for Child Change, Readiness to Change, and Perceived Ability to Change. Skill acquisition and skill utilization were measured at three sessions throughout treatment via clinician and parent self-report. This study found that a few parent motivation variables (readiness to participate, perceived ability to participate) were related to clinician reports of parent skill utilization (Nock and Kazdin 2005). Another study by Forgatch et al. (2005) looked at clinician treatment adherence and competence and its effects on skill acquisition. Clinicians conducting PMT were rated on competent adherence to PMT according to the Fidelity of Implementation Rating System (FIMP; Knutson et al. 2003). Parents were rated on use of PMT skills during parent–child interactions by observers. According to the results of this study, clinician treatment adherence and competence had a positive effect on parents’ acquisition of PMT skills (Forgatch et al. 2005).

Outcomes Associated with Skill Acquisition and Utilization

Outcomes were most frequently investigated with 21 skill acquisition articles and seven skill utilization articles in this category (see Tables 1, 2). The majority of these articles looked at the relationship between skill acquisition and utilization and improvement rates in child behavior problems, finding a positive correlation between the variables (Delaney and Kaiser 2001; Kling et al. 2010; Marchant and Young 2001; Marchant et al. 2004; Marcus et al. 2001; Martinez and Forgatch 2001). For example, Martinez and Forgatch (2001) conducted a study relating child noncompliance levels to parents’ acquisition of PMT skills. Skill acquisition was measured via laboratory observations of structured parent–child interactions. This study showed that both coercive discipline and positive parenting skills mediated treatment outcome. Rates of child noncompliance improved as use of coercive discipline decreased and positive PMT skills increased (Martinez and Forgatch 2001).

Enhancing Skill Acquisition and Utilization

Enhancement was studied in a small number of these articles. Eleven skill acquisition articles and six skill utilization articles investigated specific methods for improving enhancement of skill acquisition and utilization (see Tables 1, 2). Shanley and Niec (2010) studied the effects of in vivo feedback on skill acquisition among parents receiving PCIT. For parent–child pairs in the coaching group, parents received feedback and coaching during play interactions with their children. Parents were coached using modeling and praise via a bug-in-the-ear device. Parents who were coached during play interactions had significantly higher rates of PCIT skill acquisition than parents who were not coached (Shanley and Niec 2010). In Nock and Kazdin’s (2005) study with families in PMT, they developed a participant enhancement intervention (PEI) designed to improve both skill acquisition and utilization among parents. Parents in the PEI group received an added 5- to 15-min discussion during sessions one, five, and seven. These discussions focused on parent plans for change and potential barriers to treatment participation. Parents also completed a worksheet on plans to overcome these barriers as they occurred during treatment. Parents who received this added discussion scored significantly higher on clinician and self-report measures of PMT skill acquisition and utilization (Nock and Kazdin 2005). Kling et al. (2010) compared practitioner-assisted group training and self-administered training in terms of their effects on skill utilization among parents receiving PMT. Parents in the self-administered training group received a single training session and then implemented PMT on their own. Those in the group training met weekly with other parents for instruction, discussion, and practice of PMT skills. Homework completion (a component of skill utilization) was significantly higher for parents receiving group training versus those receiving self-administered training (Kling et al. 2010).

Meta-Analytic Results

Of the 85 articles included in the review, 68 were included in the final meta-analysis (see Fig. 1). These 68 articles represented 59 separate studies. Of these 68 articles, 62 measured skill acquisition and eight articles measured skill utilization. (Two articles measured both skill acquisition and utilization). Effect sizes are summarized in Table 3 (skill acquisition) and Table 4 (skill utilization). Separate sub-category (e.g., positive parenting) effect sizes were only calculated when three or more studies met inclusion criteria. Because there were so few studies on skill utilization, separate effect sizes could not be estimated for the utilization of child CBT skills or for the utilization of separate skill categories. Figure 2 depicts a histogram of the mean treatment–control effect sizes for individual studies.

Table 3 Effect size estimates for skill acquisition (random effects models)
Full size table
Table 4 Effect size estimates for skill utilization (random effects models)
Full size table
Fig. 2
figure 2

Mean treatment–control effect sizes for individual studies

Full size image

Pre–post Versus Treatment–Control Effect Sizes

In every instance, pre–post effects sizes were substantially larger than treatment–control effect sizes, by a magnitude of 300–1400 %. Across all analyses, average pre–post effect sizes ranged from 1.15 to 2.02 compared to treatment–control effect sizes which ranged from .04 to .66. We focus the remainder of this section on treatment–control effects sizes as the more conservative estimate of effect sizes.

Skill Acquisition

The overall treatment–control effect size for skill acquisition was .31 (p < .01; see Table 3). Orwin’s (1983) fail-safe N calculation indicated that an additional 21 studies with null findings would be necessary to reduce the mean effect size below d = .20 (Cohen’s minimum d for a small effect size). All fail-safe N estimates listed below are for the same effect size standard of d = .20. The effect size was .40 (p < .01; fail-safe N = 7) for child skills and .30 (p < .01; fail-safe N = 17) for parent skills. Separate effect size estimates were calculated for positive parenting strategies (ES = .41, p < .01; fail-safe N = 25), negative parenting strategies (ES = .26, p < .01; fail-safe N = 9), praise (ES = .66, p < .05; fail-safe N = 14), communication skills (ES = .13, p = ns), and consequences (ES = .04, p = ns). For child skills, a separate effect size was calculated for problem-solving (ES = .17, p = ns).

Skill Utilization

The overall treatment–control effect size for skill utilization was .20 (p = ns; see Table 4). All three effect sizes included in this effect size estimate were for parent skills. A separate effect size could not be calculated for child skills or for separate skill categories.

Discussion

Based on the results of our review and meta-analysis, we recommend that future research focus on three areas in order to fill gaps in the empirical literature. First, we recommend that researchers seek to develop standardized measures and methods for assessing the acquisition and utilization of cognitive-behavioral skills that can be used across the various EBTs for childhood behavior problems. Second, we recommend additional research on the determinants, correlates, and sequelae of skill acquisition/utilization. Third, we recommend that research focus on innovative ways to enhance the acquisition and utilization of BT and CBT skills.

Need for Standardized Measures of Skill Acquisition and Utilization

The absence of standardized measures and methods for assessing skill acquisition and skill utilization makes it difficult to compare the extent to which patients and families are using the skills they are learning in treatment in their day-to-day lives across treatments. This makes it difficult to compare treatment results with one another across the various psychosocial EBTs for childhood disruptive behavior problems. Measures of skill acquisition were generally specific to a particular treatment protocol and tended to assess either broad categories, such as positive or negative parenting, or a very narrow set of skills such as praise and problem-solving. To this end, we recommend the development of a measure from a “common elements” perspective (e.g., Chorpita and Weisz 2009) that can be used to assess the acquisition of a wide range of BT and CBT skills that are common across several psychosocial EBTs for childhood disruptive behavior problems. Researchers (McLeod and Weisz 2005) have developed an observational measure of therapist behavior from a “common elements” approach (Therapy Procedures Observational Coding System) to understand the content and extensiveness of skills reviewed by therapists in treatment sessions. A similar approach may be helpful to measure parent and/or child skill acquisition and utilization.

Measures of skill utilization were largely retrospective self-reports or infrequent (often just one time) home observations. Both modalities appear to have their place as self-reports are more convenient and less costly, whereas behavioral observations in the home are typically more accurate indicators of day-to-day functioning. Therefore, efforts should be taken to evaluate the utility of self-report versus observational methods of skill utilization as it unclear whether the more costly and labor-intensive behavioral observation approach is necessary. In terms of specific assessment methods, we recommend the development of an ecologically valid time-sampling or event-sampling approach to gather data on skill utilization between treatment sessions. There are several approaches to collecting valid data outside of the laboratory (or clinic) including daily diary methods (e.g., Bolger et al. 2003), ecological momentary assessment (EMA; Stone and Shiffman 1994), experience sampling methods (ESM; Reis and Gable 2000), and ambulatory assessment (AA; Ebner-Priemer and Trull 2009). Such methods have numerous advantages over traditional retrospective self-reports including ecological validity, real-time assessment, and the opportunity for multiple assessments (e.g., Bolger et al. 2003; Ebner-Priemer and Trull 2009).

Determinants, Correlates, and Sequelae of Skill Acquisition/Utilization

Although BT and CBT skills are conceptualized as specific mechanisms of change, they are rarely examined as such in the empirical literature. To this end, it will be important in future research to explore the determinants, correlates, and sequelae of skill acquisition and utilization. For example, it will be important to assess the association between skill utilization and the reduction in child behavior problems. A good exemplar of such a study was conducted by Chamberlain and colleagues (Chamberlain, et al. 2008). In the study, specific parenting skills (e.g., positive reinforcement) were found to mediate the reduction of behavior problems in a group of foster children. More studies like this one will have important implications for understanding and addressing nonresponse and partial response to treatment. If poor or infrequent skill utilization is strongly associated with treatment response and outcome, then targeting skill utilization may enhance treatment response and outcome.

Enhancing the Utilization of Skills

Finally, we recommend developing interventions designed specifically to increase skill utilization outside of treatment sessions beyond traditional “homework” or “home practice” assignments. These interventions would not be stand-alone treatments, but rather designed to supplement or augment existing skill-based treatments. Although there are numerous possibilities for such interventions, procedures would likely involve increased between-session contact between families and clinician through additional phone contact with the primary clinician, a third-party skills coach, or virtual contact and feedback through the use of digital technologies (e.g., PDA, iTouch, or smart phone). We recommend that any protocol receive ongoing input from families and clinicians to ensure acceptability, feasibility, and clinical utility. This input could be in the form of formal consultation or focus groups. Once a protocol has been developed, its efficacy could be examined through a randomized clinical trial comparing those who receive an augmented skills-enhancement protocol versus those who receive a standard skills-based treatment alone. It will be especially important to examine group differences in terms of: (1) skill utilization, (2) symptom reduction, and (3) treatment outcome.

Effect Sizes

Overall, the treatment–control effect size for skill acquisition was significantly different from zero, but in the “small to moderate” range by traditional standards. However, there was some variability noted within specific skill categories with praise having the largest effect size, falling in the “moderate to large” range by traditional standards. This suggests that some skills might be easier to acquire than others. There were far fewer studies on skill utilization compared to skill acquisition. This is not surprising, given that skill utilization is more of a distal outcome and requires assessing children and parents outside of the clinic setting. Overall, the treatment–control effect size for skill utilization was in the “small” range by traditional standards and not significantly different from zero. Pre–post effect sizes appeared to be substantially inflated in comparison with effect sizes from studies using more rigorous treatment–control designs. In light of this, we recommend that studies using pre–post designs be interpreted very cautiously.

Summary

This review and meta-analysis of the empirical literature on psychosocial EBTs for DBDs showed an overall treatment–control ES = .31 for skill acquisition and ES = .20 for skill utilization. Our review of the literature also highlights the need for additional research on the acquisition and utilization of BT and CBT skills. Filling in this gap is especially important as BT and CBT skills are conceptualized as being specific factors of change. We recommend the development of standardized assessment measures and methods for assessing skill acquisition and utilization from a “common elements” perspective, followed by the investigations of determinants, correlates, and sequelae of skill acquisition and utilization. This research will likely have implications for enhancing treatment delivery through interventions targeting skill utilization which can be used to supplement or augment existing skills-based treatments. Such an approach has the potential to reduce rates of nonresponse and partial response to treatment.