Over the past 50 years, more than 500 psychosocial treatments for youth have been developed1,2,3 and tested through randomized controlled trials (RCTs). Meta-analytic studies have been useful in consolidating effects of psychosocial treatments and have demonstrated the growth in the sheer number of the studies within the literature over time.4,5,6 For example, Weisz, Weiss, Alicke, and Klotz5 identified 108 studies in their meta-analysis, while Kazdin, Bass, Ayers, and Rodgers4 identified 223 studies, and most recently, Weisz and colleagues3 identified 447 studies in their updated review. Findings from meta-analyses have helped to identify treatments that are effective for a given problem area. Given the breadth and depth of the literature base, it is unclear if the development of more treatments will incrementally benefit youth and their families.6,7

The current landscape calls into question a need for a paradigm shift8 in psychosocial treatment literature to examine the rate at which individual treatment techniques (e.g., cognitive, exposure) within RCTs develop to augment existing methods to test treatment innovation (e.g., hybrid trials). Furthermore, an emphasis on empirically supported treatments has been touted from psychology to medicine, and methods to increase the rigor, transparency, and replication of studies have been continually refined.9 Yet, despite the deep literature base, there are still youth who do not benefit from psychosocial intervention,10 and continue to suffer from mental illness. Empirically examining the historical development of treatment approaches across and within youth problem areas is warranted to understand both the growth and changes within problem areas to move treatment innovation forward. This is particularly important given limited resources and the need for funding agencies to make decisions about how to apportion funds across the continuum of scientific discovery.11

One empirical approach to evaluate growth in the development of new treatments can be found within the distillation component of the distillation and matching model.12 Distillation, or isolating individual elements of treatment protocols (i.e., treatment description, instructions, or manual), is one methodology that allows for the specification of discrete treatment techniques across treatment protocols (see “Method” for a more detailed explanation), or the content of interventions. Distillation involves coding for the presence or absence of discrete treatment techniques, referred to as “practice elements” (PEs), 12 within and across treatment protocols. For example, within treatments targeting youth anxiety, PEs common to a wide variety of treatment protocols include exposure, cognitive restructuring, psychoeducation for youth, and relaxation.13 To date, distillation-based literature reviews have been used to provide succinct summaries across hundreds of RCTs and their associated treatment protocols for various youth problem areas.1,14 This approach has also informed the development of modular treatment approaches (e.g., flexibly guided application of numerous PEs spanning a variety of youth problem areas),15 and state-level practice monitoring and feedback initiatives.14 Distillation methodology can further extend an understanding of the youth psychosocial treatment literature beyond typical meta-analytic strategies2,3 to allow us to understand the rate at which new PEs continue to be developed as new treatment protocols are created and tested within RCTs.

The current study leveraged the treatment protocol distillation methodology to examine innovation patterns within the existing youth psychosocial treatment literature. First, information on the number of studies, treatment protocols, and PEs across 11 major youth problem areas (e.g., anxiety, suicidal behavior) is presented. Second, the relationship between new treatment protocols and the identification of new PEs within each problem area over time was compared. There were hypothesized differences in treatment protocol growth and new PE generation across problem area, potentially due to changes in understanding of developmental psychopathology, diagnostic and classification criteria, and health care policy initiatives (e.g., a focus on trauma treatment emerged after 2000 when Congress passed the Children’s Health Act which created the National Child Traumatic Stress Network). The goal of the study was to use novel methodology to extend existing meta-analytic findings to unpack the rate in which PEs are developed to advance treatment innovation in conjunction with other pioneering methods.

Method

Data Source

The PracticeWise Evidence-Based Services (PWEBS)13 database is a private-source dataset of coded RCTs of youth mental health treatments and served as the basis for the original distillation and matching model.12 The dataset was initially derived in 2000 through (a) electronic database searches for youth psychosocial RCTs in PSYCINFO,16 (b) evaluation of studies in the APA Task Force on Empirically Support Psychosocial Interventions for Children,17 (c) communication with Task Force members, and (d) nominations by Task Force members. At this initial stage, over 26,000 articles were screened, with over 230 then subsequently read in full detail. Since then, the PWEBS database has been updated through (a) ongoing electronic database searches, (b) reference checking of seminal youth treatment reviews,18 (c) personal communication with treatment developers and researchers, and (d) nominations from the Hawaiʻi Department of Health Task Force for Empirical Basis to Services.14,16 To be included (i.e., inclusion criteria), (a) the study must have been published in a peer-reviewed journal, (b) participants must have been randomized to study groups, and (c) the majority of participants must have been under the age of 19 years. Since its inception in August 2000, the PWEBS database has been continually updated through ongoing literature searches and nominations from team members and community partners.

Studies within PWEBS are coded for domains of interest including (a) youth demographic characteristics; (b) treatment design; (c) sample size, measurement, and improvement; and (d) PEs within treatment protocol. Chorpita, Daleiden, and Weisz12 note three basic assumptions of PEs including “(a) practice elements can be explicitly defined (e.g., using a definition or coding manual), (b) their presence within various interventions can be reliably coded, and (c) different treatments may share practice elements in common” (p. 11). Every treatment protocol was coded by two raters for convergence and reconciled by an expert rater for the presence of 58 common PEs (see Chorpita, Daleiden, & Weisz12 and Chorpita & Daleiden14 for a list of PEs by problem area, and Chorpita and colleages16 for more specific methodology on PE derivation and coding). When possible and available, treatment manuals were coded over descriptions in articles given the brevity of details within RCTs.

Study Dataset

From this ongoing and regularly updated database, a sample was comprised of 689 RCTs (as of Fall 2016) that represented studies published between 1966 and 2016, across 12 youth problem areas: anxiety (n = 150, 22%), disruptive behavior (n = 149, 22%), attention and hyperactivity (n = 59, 9%), depression (n = 51, 7%), autism (n = 49, 7%), substance use (n = 43, 6%), elimination disorders (n = 34, 5%), trauma (n = 29, 4%), eating disorders (n = 11, 2%), suicide (n = 9, 1%), mania (n = 2, < 1%), and other (n = 103, 16%). The other category included studies related to engagement, community-based prevention, and early intervention services.

After removing the Other category (due to its focus away from a specific problem area), the final sample included 586 RCTs representing 531 unique treatment protocols (i.e., treatment manuals) across the 11 youth problem areas from anxiety to mania (see Table 1). Within RCTs, treatment protocols were defined as a “win” if they were statistically significantly superior to another treatment, or waitlist or no treatment on a group by time interaction or between group tests at post treatment. Significant groups needed to demonstrate statistically significant change in at least one measure identified by coders as the “best measure” of the primary intervention target (e.g., Children’s Depression Inventory for a study treating depression).19 This definition is distinct from other classifications (e.g., APA Division 12 Task Force for Promotion and Dissemination of Psychological Procedures)17 in that it focused on individual treatment protocols rather than broader classifications of treatments (e.g., cognitive behavioral therapy (CBT)) or conditional probabilities.1

Table 1 Treatment study and protocol frequencies within problem areas (1966–2016)
Full size table

It was possible for a treatment protocol to be tested in numerous RCTs in the sample; therefore, to eliminate redundancies, the first (i.e., earliest study) treatment protocol within each problem area was used.Footnote 1 Treatment protocols were considered unique the first time it was being evaluated within a problem area. Only PEs that occurred in 10% or more of treatment protocols were used, consistent with previous distillation methodological studies.20,21 This cutoff was chosen to eliminate less frequently occurring PEs which may have inflated the total number of PEs within a given problem area over time. A PE was defined as “new” based on the first time it was tested within a winning treatment protocol. An example list of PE codes used for similar analyses is available online (http://helpyourkeiki.com/wp-content/uploads/2013/08/2009-Biennial-Report.pdf) and a list of studies is available upon request from the first author.

Data Analytic Strategy

Descriptive statistics were used to provide information on the number of RCTs, treatment protocols, and PEs. Specifically, information on the (a) total number of RCTs, (b) total number of treatment protocols, (c) total number of PEs, (d) first year studied, and (e) mean number of studies per year by the 11 problem areas are presented in Table 1. Second, individual graphs for each of the 10 problem areas (excluding mania) were created to visually display growth in new treatment protocols and new PEs over time. Year was depicted along the x-axis and the y-axis displayed the cumulative number of protocols and PEs. For each graph, the last data point was labeled with a call out indicating the last year and cumulative sum of protocols and PEs.

Results

Number of RCTs, Treatment Protocols, and Practice Elements over Time

Descriptive analyses revealed several patterns in youth treatment studies between years 1966 and 2016 (see Table 1). First, there was a wide range in the number of studies conducted per problem area. The anxiety and disruptive behavior problems were studied most frequently, with 150 and 149 studies respectively, whereas eating disorders, suicide, and mania were studied least frequently. Second, problem areas have been under study for differing lengths of time. The disruptive behavior, anxiety, and attention/hyperactivity areas are the earliest problem areas for which there are published RCTs, with the first treatment studies emerging in the 1960s. Eating disorders, trauma, suicide, and mania have been studied only within the past 25 years. Third, despite being examined for equivalent lengths of time, some areas have more published RCTs than others. Although studies of trauma and eating disorders began roughly around the same time (1996 and 1994 respectively), trauma was in the focus of 29 separate treatment studies with 29 unique protocols, whereas eating disorders were the target of only 11 treatment studies with 10 unique protocols. A similar pattern emerged in disruptive behavior, anxiety, and attention/hyperactivity problem areas. Studies targeting attention/hyperactivity began a few years after studies of disruptive behavior and anxiety, yet disruptive behavior and anxiety were addressed in almost three times as many RCTs (150 and 149, respectively) and protocols (136 and 131, respectively) relative to the attention/hyperactivity problem area (59 treatment studies, 57 protocols). The number of new PEs identified within each problem area did not appear to be directly related to the length of time the problem area was under study. Specifically, the two problem areas with the fewest number of studies and protocols—suicide and eating disorders—evidenced more new PEs (30 and 22 respectively) within protocols than the frequently studied attention/hyperactivity (57 protocols, 25 PEs) or depression areas (40 protocols, 20 PEs).

Comparison of Treatment Protocol and Practice Element History

Figures 1, 2, 3, 4, 5, 6, 7, 8, 9, and 10 depict the number of treatment protocols and new PEs generated within treatment studies between years 1966 and 2016 for all problem areas except mania. The graphs suggest a pattern such that the cumulative sum of treatment protocols appears to be steadily increasing over time, whereas the cumulative sum of new PEs across those protocols tended to increase at first and then level off. Across six of the 11 problem areas (disruptive behavior, attention/hyperactivity, depression, substance use, autism, and elimination), this translates to 50% or more of treatment studies conducted after the identification of the most recent new PE. This was particularly salient for the elimination area, which continued to be studied, as evidenced by published RCTs until 2009, even though there were no new PEs identified in over three decades. However, there are several exceptions to this finding. Treatment protocols for anxiety, trauma, suicide, and eating disorders have identified and tested new PEs more recently, with up to 34% of treatment studies conducted after the identification of the newest PE.

Figure 1
figure 1

Anxiety protocols and practice element frequencies over time

Full size image
Figure 2
figure 2

Disruptive behavior protocols and practice element frequencies over time

Full size image
Figure 3
figure 3

Attention/hyperactivity protocols and practice element frequencies over time

Full size image
Figure 4
figure 4

Depression protocols and practice element frequencies over time

Full size image
Figure 5
figure 5

Autism protocols and practice element frequencies over time

Full size image
Figure 6
figure 6

Substance use protocols and practice element frequencies over time

Full size image
Figure 7
figure 7

Traumatic stress protocols and practice element frequencies over time

Full size image
Figure 8
figure 8

Elimination protocols and practice element frequencies over time

Full size image
Figure 9
figure 9

Eating disorder protocols and practice element frequencies over time

Full size image
Figure 10
figure 10

Suicide protocols and practice element frequencies over time

Full size image

Discussion

The goals of the current study were to examine historical trends within the youth psychosocial treatment outcome research to enrich understanding of the extent to which treatment studies continue to generate new PEs and to inform future investigations of treatment innovation.6,7 This was evaluated in two ways, by identifying the frequency of RCTs, treatment protocols, and PEs across problem areas over time and comparing treatment protocol and PE growth within youth problem areas. The results suggest that practice element growth across problem areas have varied over time, with some problem areas evidencing large growth while others have leveled off over time. Through distillation methodology within problem area, there appears to be a period where development of new practices recedes. This study offers new methodology for historically examining treatment development and an empirical basis for expanding treatment research to target mechanisms of change such as essential practices, practice coordination, implementation, and identification of populations for which mental health services are most effective.

Problem Area Innovation

Anxiety and disruptive behavior were identified as the problem areas in which the most RCTs were published, whereas eating disorders, suicide, and mania have had fewer RCTs published. However, despite the robust proliferation of treatment protocols in disruptive behavior, the identification of new PEs has slowed over the past two decades. This is similar within the depression and attention/hyperactivity problem areas, with no new PE additions in a decade. For anxiety and trauma, new PEs have recently been added to treatment protocols, suggesting that PE generation continues in these two areas. It is important to note here that the focus of this study was on psychosocial treatment innovation only, which does not account for advances in other types of treatment such as psychopharmacology and holistic interventions. As an example, for attention/hyperactivity, recent advances have been made in understanding the sequencing of medication related to psychosocial treatments.22 Although this type of innovation was not the focus of this review, findings begin to expand upon existing meta-analytic findings nuanced toward innovation within psychosocial PEs.

Regarding eating disorders and suicide, there were more PEs than treatment protocols produced in a short period of time. This finding might reflect the complexity of the suicide and eating disorder problem areas, such that treatment protocols incorporated a greater diversity of PEs than other problem areas or may be related to the newness of these areas within youth psychopathology. Specifically, PEs that were present in 10% or more of treatment protocol meeting the study criteria were used, which may have been inadvertently influenced by the duration and amount of studies within a problem area. For example, for a PE to be included in the anxiety problem area, it needed to be present in 14 treatment protocols, whereas in the eating disorder problem area, a PE would need to be present in one treatment protocol. While this may have conflated the number of PEs for relatively new problem areas, the implications of the findings remain. Especially for life-threatening disorders such as eating and suicide, researchers must balance the need for careful and controlled research with the lethality of youth if disorders are not treated.23 Therefore, creative methods to studying problem areas may be warranted particularly in areas that may require further treatment efficacy studies.

Practice Element Innovation

Practice element development is likely influenced by several factors such as changes in theories and mechanisms of change, diagnostic classification system, funding priorities, or a movement toward effectiveness and implementation research. Ad hoc analyses examining PE frequencies within the anxiety, disruptive behavior, attention/hyperactivity, and depression problem areas demonstrated a pattern signifying a shift away from focusing on only youth in-session to a youth’s ecology which emerged when examining the areas of anxiety, disruptive behavior, attention/hyperactivity, and depression. The first practices introduced in the 1960s focused one-on-one with youth, whereas later emerging practices addressed caregivers and systems with which youth interact (e.g., school). Historical perspectives of behavior therapy by pioneers in the field like Linehan,24 Kazdin,25 and Barlow26 recount a similar pattern with a shift away from psychodynamic and person-centered therapies in the late 1970s.

It is important to note here that aside from identification of new practices (i.e., the what of treatment and the focus of the current study), treatment innovation may also occur within arrangements, that is, understanding technique sequencing and coordination within treatment protocols, and within contexts, that is, testing treatment protocols with new populations, settings, and modalities (i.e., the how of treatment). For example, CBT for youth anxiety has been found to be efficacious in a number of RCTs.27,28,29,30,31 Kendall27 initially studied individual CBT with youth aged nine to 13, and Barrett, Dadds, and Rapee28 extended CBT to include families in treatment for youth aged seven to 14 (i.e., context innovation). Subsequently, Silverman and colleagues30 introduced group CBT for youth aged 6 to 16 years (i.e., context innovation). Later, Compton and colleagues29 studied the additive effect of psychopharmacological intervention with CBT in youth aged 7 to 17 (i.e., context and arrangement innovation). This was followed by Spence and colleagues31 advancing CBT through online delivery for youth aged 12 to 18 years (i.e., context innovation).

Additionally, PE innovation may have been too subtle or nuanced to be detected by the current coding system. For example, within the past few years, learning theory has re-conceptualized the mechanisms associated with change when clients receive exposure and as a result, the application and delivery of exposure have changed,32 representing another form of treatment innovation. Conceptualizing the multifaceted nature of treatment innovation allows researchers to carefully refine research questions and empirically examine innovation over time. Careful attention should be paid to studies examining the sequence and coordination of PEs.33,34 For example, Weisz and colleagues34 tested two arrangements of CBT and found that a modular approach produced steeper favorable outcomes for youth and greater therapist satisfaction. Additional research on testing treatments within varying communities (i.e., dissemination and implementation research) and rearranging packaged treatments to provide more personalized approaches for youth and their families may also be warranted.

Implications for Behavioral Health

One important area of treatment innovation is advances in research that tackles the question of “what works for whom?”.25,35,36 A recent special issue in the Journal of Clinical Child and Adolescent Psychology offered a variety of innovative strategies and methods37 to address questions related to what works for whom. Research could begin to tailor treatments by identifying groups of responders or predictors of early outcome38 or pre-intervention characteristics that are successful to symptom improvement.39 Furthermore, adaptive research designs could be used to target specific treatment responders.40 For example, Almirall and Chronis-Tuscano40 noted that sequential, multiple assignment, randomized trials (SMART) enable researchers to identify treatment responders and tailor interventions to the individual rather than traditional groups or clusters. These strategies in conjunction with well-established interventions would allow for more idiographic treatment and personalized medicine.

From a systems perspective, these findings may have implications for selecting treatment and training therapists. For example, it appears that treatment techniques for many problem areas have stabilized. Chorpita and Daleiden41 have suggested that for therapists in a general outpatient setting, creating treatment systems that incorporate and arrange PEs based on population characteristics, guided by conditional logic models, theory, and feedback systems, offers parsimony for learning treatments. From a training perspective, despite the known burden of learning PE coordination (e.g., how to sequence, which to use when), it may be more efficient and cost-effective for trainers and less overwhelming for therapists42 to be trained in nine to 30 discrete techniques than an ever-increasing number of treatment protocols.1,12,43 Understanding therapist competencies across youth mental health is an important endeavor for the field,44 especially given the limited workforce to meet the current need.25 Furthermore, many PEs are not unique to each problem area. For example, “cognitive” spans disruptive behavior, anxiety, and depression. This overlap would further decrease burden on therapists from learning the fundamental steps needed to provide cognitive skills, while allowing time to focus on the nuances specific to each problem area.

In light of the current study’s findings, there are important limitations to note. First, PE definitions and mechanisms of change have evolved over time.32 For example, Craske and colleagues32 have examined inhibitory learning as a mechanism for change in exposure trials, which has impacted the application of exposure in therapy. When coding PEs in the current study, nuances in the changes in application and delivery are not captured. Indeed, the focus of the current study was on the composition of treatment protocols and not the steps related to delivery. Future research may wish to examine innovation within PEs that appears to be most frequent within larger treatment protocols.

Furthermore, Weisz and colleagues’3 recent multilevel meta-analysis suggests that problem area is a significant moderator in treatment efficacy, with the largest effect sizes for anxiety (ES = .61) and disruptive behavior (ES = .46) problems and lower, yet significant, effect sizes for attention/hyperactivity (ES = .34) and depression (ES = .29). In light of these results, it is possible that effect sizes are related to the duration and number of studies within their meta-analysis. Indeed, one might question as to why effect sizes were not calculated in the current study. The high utility of meta-analyses depends in part on a categorical and exclusive grouping of practices to calculate average effects, for example, with such units of analyses as brand-named protocols (e.g., “Coping Cat”).27 It is difficult or even impossible to calculate effect sizes for individual PEs because of the cross-classified nature of their existence within and across protocols (i.e., a PE can simultaneously be in both a winning and losing protocol, both within and across studies). It is hoped that advances in analytics will eventually intersect with distillation methodologies for examining effect sizes associated with individual PEs. Until then, the “winning” count methodology utilized in this study should perhaps be viewed as a complimentary, yet powerful, approach for examining the literature alongside traditional meta-analytic approaches.

Relatedly, the sample was comprised of mostly CBT treatments that were amenable to discrete practice coding, as compared to interventions from other theoretical orientations. Researchers have noted the need for plurality among treatment efficacy research to examine the various modalities (e.g., psychodynamic therapy) of treatments across the continuum.45 In this respect, funding agencies may play an important role in treatment development and implementation. A clear next step is to examine the extent to which funding mechanisms have moderated the development, marketing, and implementation of treatment protocols, starting with the current sample and summarizing the funding acknowledgements within these trials.

Furthermore, psychopharmacological intervention trials were not included in the sample, which may be a focus of innovation for problem areas (e.g., attention/hyperactivity). Additionally, for the suicide problem area, the smaller number of trials may have been due to high acuity of the problem area23 and/or a secondary symptom to depressed mood or anxiety. Moreover, the inclusion criteria, while modeled after the definitions of empirically supported treatments18, were less conservative than typical meta-analytic strategies. Future research may wish to examine treatment innovation in select treatment protocols that have garnered rankings as empirically supported treatments. Finally, as with any type of meta-analysis or large synthesis of literature, the study was limited to the sample collected. As noted in the “Methods” section, the PWEBS database is continually updated through electronic searches, nominations by team members and the community, and communication with treatment developers, and while arguably comprehensive, it may not be exhaustive in encompassing all youth psychosocial literature. Therefore, there may be studies missing from the sample that would affect the outcomes of this study. Relatedly, studying PE innovation within subgroups of problem areas, age ranges, treatment modalities, and settings as well as the history of individual PEs may also be beneficial.

The current study provides a historical perspective on treatment approaches within the youth psychosocial treatment literature using the distillation methodology. The results offer a novel method for studying intervention development and suggest that there is variability in practice development across problem areas. Directly emerging from this line of questioning is the need for research that answers the question “What works for whom,” including innovative research design, implementation and dissemination trials, and refining predictive analytics that can identify mediators or moderators of treatment outcome.46 At this critical juncture in the mental health services field, it may be more helpful to identify mediators of youth outcomes including the sequence of practices and conditions needed for a treatment to be effective.22 This notion is consistent with the National Institute of Mental Health’s Strategic Plan’s9 Strategy 3.2 to tailor existing treatments to better serve the populations for whom they are intended.47 It is hoped that innovation within treatment design and categorization of existing treatments will ensure that youth receive effective treatments for mental health disorders that improve their lives.