Introduction

Assessing and improving the provision of mental healthcare for children is a fairly recent undertaking and presents a myriad of challenges (Garland et al., 2013; IOM, 2013). Of particular importance is successful intervention for early childhood disruptive behaviors, the most common mental health problem among American youth 7 years old and younger (Lavigne et al., 2014; Lochman et al., 2017). If not properly managed, early childhood disruptive behaviors can progress into more complex and serious problems in adolescence and adulthood that can cost over $100,000 per person in healthcare services, education, and criminal justice (APA, 2013; Brookman-Frazee et al., 2009; Foster et al., 2005; Scott et al., 2001; Sheldrick et al., 2011; Shevlin et al., 2017; Wertz et al., 2018).

Parent-directed behavioral therapy (also known as parent management training), where parent(s) are “co-therapists” with their child’s clinician and learn alternative ways to conceptualize and act towards their child’s behavior problems, is one of the most common evidence-based interventions for early childhood disruptive behaviors and, in the US, typically provided in private outpatient clinics affiliated with larger hospital systems or academic institutions (Acri et al., 2018; Forehand et al., 2013; Kazdin, 1997; Lochman et al., 2017; Michelson et al., 2013; Reyno & McGrath, 2006; Steiner & Remsing, 2007; Van Aar et al, 2017). While there is a robust evidence base for the efficacy of parent-based therapy models in treating early childhood disruptive behaviors (Acri et al., 2018; Boardman, 1962; Chorpita et al., 2011; Dishion et al., 2011; Eyberg et al., 2008; Forehand et al., 2013; Johnson & Katz, 1973; Lochman et al., 2017; Michelson et al., 2013; Weisz & Gray, 2008), information on the ‘real world’ effectiveness of these interventions in clinical programs has been limited and variable (Comer et al., 2013; Garland et al., 2013; Herschell et al., 2004; Van Aar et al., 2017; Whipple & Lambert, 2011).

The dearth of this type of practice-based research may be attributed in part to the lack of infrastructure to routinely monitor early childhood behavioral outcomes in clinical practice. Outcomes accountability systems can help close this knowledge gap. Also known as routine outcomes monitoring (ROM) or measurement feedback systems (MFS), outcomes accountability systems specialize in evaluating and improving treatment effectiveness in routine practice. Even a ‘basic’ outcomes accountability system is beneficial to patient, provider, and funders alike (Bickman, 2008a, 2012). Routine monitoring of outcome metrics is the basis for quality assessment to determine whether interventions are meeting treatment targets, a foundational principle of a learning health system and critical for the development of evidence-based psychotherapy (Bickman, 2008a; Eyberg, 2013; Scott & Lewis, 2015). The basic components of an outcomes accountability system include mechanisms for the initial assessment of problems, frequent routine monitoring of clinical status, characterization of outcomes when treatment ends, and a feedback process for timely sharing of information with clinicians, supervisors and administrators (Bickman, 2008a). Systems that characterize social and demographic factors, child clinical complexity, as well as parental health and functioning can provide further guidance as to what is needed to achieve optimal outcomes. When clinical progress is provided at consistent intervals during treatment, clinicians are able to make timely adjustments to therapy (Bickman, 2008a; Lambert et al., 2005). These aspects of consistent monitoring and feedback can be particularly beneficial to clinical programs that involve parents in helping to direct or manage the child’s behavior, as the parents’ ability to consistently engage in behavioral data tracking and trouble-shoot treatment strategies with their child’s clinician are key factors to treatment success (Post et al., 2017; Yu-Lefler et al., 2022). Thus, if integrated into a clinical program’s practice, outcomes accountability systems and quality improvement initiatives using its data have the potential to increase a clinical program’s success in managing disruptive behavior problems among early childhood patients.

When deployed at a population level across multiple clinical programs, data from outcomes accountability systems can measure the effectiveness of different treatment services in addressing early childhood disruptive behaviors. There is even potential to characterize how much (intensity) and how long (duration) each type of treatment requires to decrease specific problems, as well as to identify factors associated with different outcomes (Bickman, 2012). Such population-level data can allow providers, funders, and policy-makers to better gauge the value and fit of different treatment practices in different patient contexts (Bickman, 2008a, 2012), thereby improving mental healthcare’s ability as a whole to successfully manage early childhood disruptive behaviors. For most US clinical programs in routine practice, the limitation on financial resources makes it imperative that the program is deploying the most cost-efficient interventions (Garland et al., 2013). Also, with the passing of the Affordable Care Act (ACA) and Medicare Access and CHIP Reauthorization Act (MACRA), US policymakers, insurers, and the healthcare industry are emphasizing a payment model based more on service performance rather than volume. There are more than 24 “pay-for-performance” Medicaid and privately-insured reimbursement initiatives specifically targeting mental healthcare providers, with initiatives continuing to increase and incorporate more child mental health service systems (Bremer et al, 2008; Stewart et al., 2017). Initial investments and efforts for quality monitoring have focused on high-cost systems serving high-risk children with serious mental illness or neurodevelopmental disabilities in residential or wrap-around care (Armstrong et al., 2014, 2016; Backer et al., 2015; Lyons et al., 2001, 2004). Now, for reasons of efficiency, ethical care, financial reimbursement, and public health, it is imperative for other clinical programs, particularly those that focus on early intervention for common mental health problems, to develop structures of care that support systematic measurement and improvement of treatment effects.

Despite the various rationales for integrating outcomes accountability systems into clinical treatment for common childhood mental health problems, it is unknown how much this has been put into practice in clinical programs treating early childhood disruptive behaviors. Bickman’s commentaries (2008a, 2012) refer to outcome accountability efforts across mental healthcare populations in general. Whipple and Lambert’s (2011) review of outcome measurement feedback in the treatment of common mental health problems states that “there have been no studies on the impact of providing feedback to therapists about the treatment progress of their child and adolescent patients” (p. 105). This manuscript seeks to ascertain whether outcomes accountability systems in the clinical treatment of common mental health problems in children, specifically that for early childhood disruptive behaviors, have appreciably expanded in the decade since these publications. While the review will include examples from across the world, the authors will particularly analyze and discuss whether outcomes accountability systems for this target population have appreciably expanded in US clinical programs due to the prevalence of early childhood disruptive behaviors among American youth and the public health implications if these behaviors are not successfully managed.

Objective

A scoping review was conducted to summarize the research on stages of development and usage of outcomes accountability systems or assessments in clinical programs that may treat early childhood disruptive behaviors. Our research questions were: (1) What are the current stages of development and usage of outcomes accountability systems in clinical programs for children with common mental health problems; has it progressed since Whipple and Lambert’s (2011) review? (2) How many of these systems specifically address clinical treatment for common mental health problems in early childhood, particularly early childhood disruptive behaviors? (3) To what extent have these systems expanded in American clinical practices? (4) What challenges do clinical programs specializing in early childhood disruptive behaviors face when implementing outcomes accountability systems within their practice? Our intent is to inform US policy-makers, practitioners, and researchers on possible best practices and next steps to improve successful early treatment of common mental health problems among American children.

Methods

Search Strategy

The review consisted of peer-reviewed book chapters and journal articles (editorial, commentary, reviews, and empirical studies) published from January 1, 2005–January 17, 2021 and listed in PSYCINFO using the following search strategy: Subject terms consisted of primary keywords in past literature to describe outcomes accountability systems: “measurement feedback system” OR “measurement feedback” OR “routine outcome monitoring” OR “accountability system” OR “learning health system.” Searches were conducted with one subject term at a time to yield as many publications as possible. Additional PSYCINFO limiters were “Peer Reviewed” only, Publication Year from 2005 to the present (January 17, 2021 when last conducted), and “English” language. The timeframe starts a few years prior to Whipple and Lambert (2011)’s review in order to capture earlier studies that could provide historical context on the development of outcomes accountability systems in different countries and settings. Age limiters were not restricted to pediatric age groups only (i.e., < 18 years old) in order to minimize exclusion of outcomes accountability literature that included early childhood clinical populations along with older childhood, adolescent, and adult clinical populations. A medical librarian experienced in performing literature reviews assisted in developing and reviewing the search strategy.

Study Selection

To answer our first research question (“what are the current stages of development and usage of outcomes accountably systems in clinical programs for children with common mental health problems; has it progressed since Whipple and Lambert’s (2011) review?”), the authors thoroughly reviewed all PSYCINFO search results to identify publications regarding outcomes accountability in clinical treatment of common mental health problems in patient populations with early childhood patients. HYL screened 100% of identified PSYCINFO titles and abstracts for eligibility by first evaluating each result for publication type (i.e., book chapter/editorial/commentary, review article, empirical article), focus area, setting, and journal name (if article). Outcome accountability focus areas were categorized in terms of stage of development using Taylor et al. (2013)’s application of the Plan-Do-Study-Adjust framework; “plan” publications focused on system design, “do” publications focused on implementation, “study” publications focused on assessment of or impact on clinical outcomes, and “adjust” publications focused on outcomes improvement. Duplications were eliminated. Publications eligible for full-text review were those involving outcomes accountability systems in clinical settings that monitored mental health symptomology and behavioral improvement. Exclusions were publications on outcomes accountability in non-clinical settings (i.e., schools) or non-mental health settings, assessment of non-clinical outcomes (i.e., scholastic performance, professional competence, or organizational performance), standardized scale psychometrics for diagnostic purposes, factors impacting mental disorders unrelated to outcomes accountability, comparative effectiveness of different treatment models, or quality-of-care issues tangential to clinical outcomes (i.e., clinician fidelity to practice guidelines and/or standards of care, therapeutic alliance, shared decision making between clinician and the patient and/or patient’s family in the treatment process). Additionally, we excluded publications that generally discussed ethical concerns and implications of data-driven care. Exceptions were publications on developing a scale to measure implementation fidelity or impact of an accountability system on clinical outcomes, or using accountability data to assess predicting factors on clinical outcomes. We also did not exclude publications on outcomes accountability for youth mental health case management at state health departments or within youth service delivery systems that included public mental health facilities, as we considered these to be part of clinical services within the US. HYL then conducted a full-text review on the remaining publications to identify those including or targeting early childhood populations (i.e., ≤ 7 years old) with common mental health illnesses. Exclusions were publications that 1) addressed the general (i.e., non-age specific) patient population or exclusively focused on adolescent and/or adult patients, or 2) specifically focused on high-risk children with severe mental illnesses or neurodevelopmental disabilities. We did not exclude, however, publications that discussed clinical programs addressing both severe and common mental health problems, or programs addressing early childhood patients who had language or developmental delays and common behavioral problems. The two co-authors independently reviewed all selection results.

Data Extraction

The authors jointly developed a data abstraction form (Appendix 1) to extract relevant information to answer the remaining research questions from all publications meeting the inclusion criteria: The title, author(s), publication date and year, and publication type were obtained from the PsycINFO database and/or publication abstract. The country of origin, clinical setting, and description of the publication’s focus areas were derived from reviewing the abstract and full text. A full-text review of each publication provided further information to categorize the system’s stage of development, as described above. The full-text review also provided information on target patient population(s), review type (if the publication was a review article), study design (if the publication was an empirical article), key findings, and how the publication strengthens or highlights gaps in knowledge regarding outcomes accountability systems in clinical programs treating early childhood disruptive behaviors. Data extraction was first conducted by a single reviewer [HYL], with additional review by co-authors. Publications underwent data extraction in ascending order by publication date (most distant to most recent). Changes made to the data abstraction form were iterative. If a measure was encountered that had previously been identified, its presence was noted in a corresponding column. When previously unidentified measures were detected, the data was extracted and reviewed by authors for redundancy, and either incorporated into existing items or added as new items in the data abstraction form.

Data Analysis

Data analysis was conducted differently based on publication type. Key insights from book chapters, editorials, and commentaries were summarized descriptively. Review and empirical articles were summarized in a tabular format. To answer the second research question (“How many of these systems specifically address clinical treatment for common mental health problems in early childhood, particularly early childhood disruptive behaviors?”), all included publications were grouped into four “tiers” of specificity with regards to patient population and treatment model: “T1” includes but does not focus on early childhood cases, “T2” specifically addresses early childhood patients, “T3” addresses treatment for early childhood disruptive behaviors, and “T4” addresses parental management of early childhood disruptive behaviors as part of clinical care. To answer the last two research questions (“To what extent have these systems expanded in American clinical practices? What challenges do clinical programs specializing in early childhood disruptive behaviors face when implementing outcomes accountability systems within their practice?”), the authors analyzed the level of development (i.e., Plan, Do, Study, Adjust) and usage of outcomes accountability systems to address common mental health problems in children at each “tier,” with particular emphasis on US clinical populations as compared to that of other countries. This analysis also helped identify key barriers to development and utilization.

Results

Search Findings

The search yielded 389 records, of which 338 were unique. After title and abstract review, 205 of the 338 records were excluded due to focusing on outcomes accountability in school (n = 96) or non-mental health settings (n = 15), assessment of non-clinical outcomes (n = 20), standardized scale psychometrics (n = 14), factors impacting mental disorders unrelated to outcomes accountability (n = 8), comparative effectiveness of different treatment models (n = 7), quality of care issues tangential to clinical outcomes (n = 42), or ethics of data-driven care (n = 3). The remaining 133 records (39% of the unique records) underwent a full-text review, of which 110 records were further excluded due to targeting outcomes accountability systems for the general mental health population (n = 36), adults (n = 57), adolescents (n = 12), or special populations with severe mental illnesses or neurodevelopmental disabilities (n = 5). Twenty-three (23) records (7% of the unique records), consisting of 3 editorials, 2 commentaries, 4 reviews, and 14 empirical studies, were identified as focusing on outcomes accountably systems in clinical programs for children with common mental health problems (Fig. 1).

Fig. 1
figure 1

PRISMA flow chart of literature selection

Full size image

Figure 1 also indicates how many of the included publications and respective publication types fall into each “tier” of specificity with regards to patient population and treatment model. 20 of the 23 publications (3 editorials, 1 commentary, 4 reviews, 12 empirical studies) described outcomes accountability in the clinical treatment of common mental health problems for a patient population that included but did not focus on early childhood cases (i.e., “T1”). 2 publications (1 commentary and 1 empirical study) specifically addressed outcome accountability in the clinical treatment of common mental health problems in early childhood patients, but did not target treatment of early childhood disruptive behaviors (i.e., “T2”). Only 1 publication (an empirical study) addressed outcomes accountability in the clinical treatment of childhood disruptive behaviors (i.e., “T3”). This same study also touched upon how outcomes accountability practices can enhance parental management of early childhood disruptive behaviors as part of clinical care (i.e., “T4”).

Figure 2 illustrates the frequency and “tier” of publication from 2005 to 2020. Across all 23 included publications, there was less than 1 publication per year from 2005 to 2012. After 2012, the number of publications per year doubled before peaking at 4 in 2016 and decreasing back to 1 per year in 2017–2018. Publications have increased slightly in 2019 (3 publications) and 2020 (2 publications), but it is unknown if this indicates another increasing trajectory like that in 2013. The 3 publications specific to early childhood patients and/or early childhood disruptive behaviors all took place after 2012; the “T2” commentary and empirical study were published in 2016 and 2019 respectively, while the “T3”/“T4” empirical study was published in 2014.

Fig. 2
figure 2

Frequency and tier of publications (N = 23) by year

Full size image

Table 1 illustrates the development stages of outcomes accountability systems within the 23 included publications by country and target patient population or “tier.” Most of the publications (17 out of 23) described outcomes accountability efforts in early stages of development, focusing on system design logistics (“Plan”), implementation feasibility (“Plan/Do”), and implementation fidelity (“Do”). The early development accountability efforts were located in the US or UK, with publications targeting US-based clinical programs predominantly represented in the “Plan” stage, while publications discussing UK-based clinical programs were more prevalent in the “Do” stage. 4 of the remaining 6 publications that focused on later development accountability efforts, such as assessing an outcomes accountability system’s impact on clinical outcomes (“Study”) or lessons learned to improve outcomes (“Adjust”), were in non-US settings, i.e., in the Netherlands or UK. For the 3 publications specifically addressing early childhood patients (“T2”) and/or early childhood disruptive behaviors (“T3”/“T4”), the commentary targeting US-based practices (De Jong, 2016) was in the earliest stage of development (“Plan”), discussing the logistics to introduce outcomes accountability systems into clinical programs for early childhood populations. The UK-based empirical study targeting accountable practices to treat early childhood disruptive behaviors was also in early development (“Plan/Do”), namely discussing how to develop a meaningful outcome measure for behavioral data tracking (Emanuel et al., 2014). The only publication that discussed later development accountability efforts (“Adjust”) for an early childhood clinical population, namely illustrating how data collected from an outcomes accountability system could be used to assess factors influencing outcomes, was from The Netherlands (Vermeij et al., 2019) and specifically targeted clinical improvement of language skills rather than behavioral problems.

Table 1 Publications by focus areas [authors (year) in chronological order]
Full size table

Additional Details

Summaries of the empirical and review articles (Appendix 2), as well as key insights from editorials and commentaries, provide further insight into challenges integrating outcomes accountability systems within clinical programs treating common childhood mental health problems, and potentially, early childhood disruptive behaviors.

Two key themes seem to affect usage and integration of outcomes accountability systems in clinical programs treating children with common mental health problems: (1) clinicians’ and patient families’ perceived value in data-driven care and the outcome assessment to meaningfully measure progress, and (2) top-down leadership support and multi-level changes to meaningfully integrate outcomes accountability systems into practice. These themes are commonly cited within qualitative studies, reviews, commentaries, and editorials detailing early development accountability efforts, both within the US (Bickman, 2008b; De Jong, 2016; Garland et al., 2013; Kelley & Bickman, 2009; Kotte et al., 2016; Liu et al., 2019; Lyon et al., 2015, 2017; Seidman et al., 2010) and the UK (Batty et al., 2013; Ford et al., 2006; James et al., 2015; Norman et al., 2014; Waldron et al., 2018; Wolpert et al., 2012).

In the US, barriers limiting adoption of outcomes accountability practices stem from the valuation of clinical care, namely (1) clinical service is currently a commodity as it cannot be differentiated on the basis of performance or value and thus continues to be reimbursed based on a fee for a unit of service rather than on any value or performance basis; (2) without effectiveness data, states and funders can maintain that they are meeting the needs of clients (or would do so, if they just had more funding); (3) reliance on licensing and accreditation to demonstrate quality of care although evidence is lacking that they are associated with patient outcomes (Bickman, 2008b). Following from these barriers are a lack of financial incentives from policy-makers or payors and provider leadership support to integrate usage of outcome measurement and feedback to optimize clinical care into clinician training, accreditation, supervision, and practice time (Bickman, 2008b; Kotte et al., 2016; Lyon et al., 2015, 2017). For providers that did implement outcomes accountability practices, consistent data tracking and usage of the data during sessions seemed to be associated with better clinical outcomes (Milette-Winfree et al, 2019; Sale et al., 2020; Tsai et al., 2016). However, lack of confidence from clinicians and patient families in the measures, particularly standardized assessments, to meaningfully capture progress limited the usage and sustainability of outcomes accountability systems in practice (Kotte et al., 2016).

Reviews by Kelley and Bickman (2009), and Seidman et al. (2010) provided initial suggestions to address some of these challenges implementing outcomes accountability systems in US clinical practice. Outcomes accountability systems could incorporate more multi-dimensional assessments to provide meaningful data on patient progress and treatment success at the patient, family, clinician, organizational, and policy-maker levels. To support multi-dimensional assessments, outcomes accountability systems ought to have a data infrastructure that is user-friendly and flexible enough to incorporate feedback from each level. Garland et al. (2013)’s review presents a more comprehensive framework to guide outcomes accountability at each level, starting with how policymakers could provide more pay-for-performance incentives to support outcomes accountability in US children’s mental healthcare. The incentives would then facilitate provider organizations to build accountability infrastructure that meaningfully informs treatment progress, outcomes, and cost, as well as training programs for clinicians to meaningfully use outcomes data to enhance their therapy. Patient families could then use accountability data to compare quality of services and better understand the necessity of accountable care. However, as late as 2019, lack of top-down policy initiatives and inadequate clinician support continued to hinder implementation fidelity and sustainable outcomes monitoring throughout clinical programs for common childhood mental health problems (De Jong, 2016; Liu et al., 2019). A related finding was that clinicians, patients, and patient families within these clinical programs have generally pessimistic or ill-informed views towards measurement feedback and outcomes accountability practices, resulting in most outcomes accountability systems being implemented as special research projects (De Jong, 2016). The limited support for outcomes accountability systems within these practice settings also introduced gaps in knowledge for different age groups. Thus far, the challenges to and benefits derived from implementing outcomes accountability systems are from US-based programs treating older youth with more complex and multiple mental health conditions (Kotte et al., 2016; Lyon et al., 2015, 2017; Milette-Winfree et al., 2019; Sale et al., 2020; Tsai et al., 2016). It is unknown how much these findings generalize to clinical treatment of common mental health problems in early childhood populations (De Jong, 2016). Among private outpatient clinical programs specializing in behavioral and/or parent-based therapy, a common clinical setting to treat early childhood disruptive behaviors in the US, De Jong’s (2016) commentary identified only two studies that targeted adolescent populations (Bickman et al., 2011, 2014, as cited in De Jong, 2016), with none to date in younger populations.

In contrast to the US, since the early 2000s, the UK has had a consortium of service providers, funders, service user groups, and researchers (namely the Child Outcomes Research Consortium) to systematically develop best practice guidelines on outcome measures across all Child and Adolescent Mental Health Services (CAMHS) clinical settings and clinical populations, as well as optimal infrastructure design and clinician support to enhance outcomes monitoring and feedback (Ford et al., 2006; Wolpert et al., 2012). Resultingly, much of the outcomes accountability work in the UK focused on implementation fidelity disparities across CAMHS clinical settings and pediatric populations, with very sparse research targeting specific clinical settings or patient groups. However, similar implementation challenges as those found in the US included low buy-in or usage by clinicians and families, concerns regarding outcome measure validity and ability to meaningfully encapsulate patient’s problems, and suboptimal system design and integration into clinician training, supervision, and practice (Batty et al., 2013; Fleming et al., 2016; James et al., 2015; Norman et al., 2014; Waldron et al., 2018). These challenges seem to stem in part from leadership implementing systems into practice before there was adequate infrastructure in place or stakeholder buy-in. Two UK-based studies specifically highlighted clinicians’ dissatisfaction with CAMHS leaders “forcing” clinicians to adopt certain outcome accountability measures and practices, and had ethical concerns regarding how the data collected would be used by CAMHS leadership or policymakers to determine care effectiveness and value (Norman et al., 2014; Waldron et al., 2018). Emanuel et al. (2014) sought to address the issue surrounding meaningful outcomes measurement by introducing a goal-based outcome measure that clinicians generate with parents to use for ROM. While this measure could be used for older children with more complex clinical profiles, the authors also demonstrated how this measure could enhance treatment for early childhood patients with disruptive behaviors by encouraging parents to participate in their child’s treatment and help track data on their child’s behavioral progress. Emanuel et al. (2014) provides valuable insight on outcomes accountability assessments and practices that may be most meaningful for treating early childhood disruptive behaviors, particularly for therapy models with parental involvement. The caveat is that these findings are based on select case studies, and have unknown generalizability to larger clinical populations.

Outside of the US and UK, the Netherlands have also implemented outcomes accountability systems and practices into clinical programs treating common childhood mental health problems. Development of outcomes accountability systems and practices in the Netherlands are the most advanced, as clinical programs routinely administer assessments for outcomes accountability, leading to studies that assessed influential factors for outcomes improvement (Van Sonsbeek et al., 2014, 2020; Vermeij et al., 2019). Even so, the studies found did not address treatment for early childhood disruptive behaviors, as either the representative patient sample was older with more complex clinical profiles (Van Sonsbeek et al., 2014, 2020), or treatment focused on language skill acquisition in early childhood patients with language disorders (Vermeij et al., 2019).

Discussion

This review highlights the paucity of outcomes accountability systems in clinical programs treating early childhood disruptive behaviors, especially in the US. While the development and usage of outcomes accountability systems in clinical programs treating common mental health problems in children have appreciatively expanded since Whipple and Lambert’s (2011) review, only 3 out of 23 peer-reviewed publications found specifically discussed outcomes accountability efforts for early childhood clinical populations. Out of these 3 publications, the only outcomes accountability effort addressing treatment for early childhood disruptive behaviors has been found in the UK (Emanuel et al., 2014). Our scoping review did not find any studies on the development and usage of outcomes accountability systems to assess or improve treatment of early childhood disruptive behaviors in US-based clinical programs.

Unique challenges exist to implement and sustain outcomes accountability systems in US-based clinical programs for common childhood mental health problems. Compared to the UK and Netherlands (the two other settings in the 23 included publications), the US lacks top-down incentives from policy-makers and payors as well as national-level collaborative efforts among providers to develop, implement, and meaningfully use outcomes accountability systems within clinical care for common childhood mental health problems. Consequently, outcomes accountability efforts are hard to implement and sustain in practice. This is reflected in how most US-based publications found during this review are still concentrating on early development accountability efforts of design and implementation feasibility rather than later development efforts of assessing how data collected from outcomes accountability systems inform areas for outcomes improvement.

Nonetheless, the present review also highlights opportunities for improvement. The reported literature on US-based accountability efforts for common childhood mental health problems are predominantly in state youth services or public and community mental health clinical settings (Kotte et al., 2016; Lyon et al., 2015, 2017; Milette-Winfree et al., 2019; Sale et al., 2020; Tsai et al., 2016), which typically see older youth or adolescents with multiple mental health conditions and utilize a variety of treatment methods (Brookman-Frazee et al., 2009, 2010). While it is possible that this finding may be affected by the scope of the search results, this observation does draw two potential implications.

The first implication is that part of this finding may be attributable to reimbursement and grant incentives provided by the Health Resources and Services Administration (HRSA), the Substance Abuse and Mental Health Services Administration (SAMSHA), and several state governments to support behavioral health workforce integration, education, and dissemination of evidence-based and accountable practices in public and community-based healthcare practices (Bonilla et al., 2021; Brookman-Frazee et al, 2016; Garland et al., 2008). Additionally, the Centers of Medicare & Medicaid Services (CMS) and several states have required state managed care clinical programs to put into place a quality assessment and improvement system for children’s healthcare services (MACPAC, 2020, 2021). As of the late 2010s, public mental health organizations have increased adoption of evidence-based treatments and quality monitoring practices to increase treatment accountability (Brookman-Frazee et al, 2016). The current incentives may not have been applicable to private practices, and similar accountability requirements may not have been adopted or implemented by private, commercial insurers, whose dependents represent a greater proportion of the patient population at private vs. public clinics (Acri et al., 2018; Michelson et al., 2013). Similar financial incentives for and implementation science studies on accountable practices in private clinics, particularly those providing early intervention for common childhood mental health problems, have the potential to increase and highlight quality monitoring practices in clinical programs treating early childhood disruptive behaviors (Garland et al., 2008, 2013).

The second implication is assuming public clinical programs do have more incentives and infrastructure in place for accountable practices, an opportunity exists to further disseminate and assess the effectiveness of PDT or similar evidence-based interventions to address early childhood disruptive behaviors in these settings. Research has suggested that families with public insurance and/or lower socioeconomic means tend to access mental healthcare through federally-supported community health centers with integrated behavioral health staff or public mental health programs (Bonilla et al., 2021; Cook et al., 2007; Pourat et al., 2021; Southam-Gerow et al., 2012). However, public clinics have resource constraints that may make it particularly challenging to adapt PDT best practices (Beidas et al., 2017). A key feature of evidence-based PDT is to have multiple behavioral specialists or trained observers repeatedly collect observational data on behavior targets (known as direct observation), which is the gold standard for determining treatment effect (Johnston & Pennypacker, 2009; Vollmer et al., 2008). These practices can cost up to $10,000 to furnish an appropriate treatment room, and $40,000 to over $100,000 in salary (depending on geographic location and training level) and an additional $3000-$4000 in training costs for each behavioral health staff member (Goldfine et al., 2008). The extensive staffing and financial resources to reliably implement such practices has restricted the dissemination of PDT to more diverse clinical settings outside of hospital- or university-based private clinics. With that said, there is some evidence of PDT or evidence-based treatment for childhood disruptive behaviors being adapted to community health center or public mental health settings in the US. A pilot study assessed the efficacy of a PDT program for families of high-risk children with disruptive behaviors seeking care at community health centers (Lau et al., 2011). While not specific to PDT, some studies do indicate that clinicians in public mental health settings are more apt to apply evidence-based treatment models to patients fitting the early-onset disruptive behaviors profile (younger, male, clear aggressive-disruptive symptoms, and low multi-morbidity of other mental health issues) (Brookman-Frazee et al., 2010, 2016; Kim et al., 2020; Orimoto et al., 2014). More dissemination science research focusing on how to best adapt evidence-based practices such as PDT to address early childhood disruptive behaviors in public clinical settings could help increase early intervention access and care equity for lower income children with behavioral problems.

An additional area for improvement is expanding investigations into outcome measures that may be most applicable to behavioral problems encountered in early childhood populations. One possibility is a top-down approach by expanding on federal or state-level accountability initiatives for children’s healthcare to include outcomes for common mental health problems in early childhood. While both federal and state policies for Medicaid have required state managed care clinical programs to put into place a quality assessment and improvement system for children’s healthcare services (MACPAC, 2021), the current measures implemented in practice are not specific to the management of behavioral problems in early childhood (MACPAC, 2020). A related issue is that current formulated outcome measures are more focused on rating provider performance (i.e., meeting clinical quality benchmarks or service satisfaction) rather than evaluating the patient’s behavioral and functional improvement from treatment (MACPAC, 2020). It would be useful to survey Medicaid authorities to learn more about what they may be doing to make quality measurement more targeted towards common mental health problems in younger patient populations, as well as how they might frame the measures to better align with improving patient care rather than assessing provider performance. At the state level, two studies from this review indicated state-wide accountability efforts using the Ohio Scales to measure the progress of common childhood mental health problems (Kotte et al., 2016; Milette-Winfree et al., 2019). Findings imply that the Ohio Scales may be applicable for early childhood disruptive behaviors due to the ease of administration for younger youth with less complex clinical profiles (Milette-Winfree et al., 2019). A follow-up study assessing the utility of the Ohio Scales to monitor progress of early childhood behavioral problems would be informative.

Another possibility is a bottom-up approach wherein clinical programs that specifically treat early childhood disruptive behaviors start developing a meaningful outcomes measure, and then iteratively expand to an accountability system built around this measure. This approach may be particularly appropriate for private clinics specializing in PDT, as a key feature of these therapy models is that intervention effectiveness is individually determined, and clinicians make decisions on whether to continue, stop, or change interventions based on repeated observation of individualized behavior targets and goals (Baer et al., 1987; Bourret & Pietras, 2013; Vanselow et al., 2011). Clinicians from these practices may be resistant to top-down accountability and performance evaluation-style assessments that feel threatening; however, they do want to use data to determine if their patients are moving towards optimal behavioral health and functioning. As such, improvement systems centered around a bottom-up, iterative approach may get the most buy-in from clinicians in private practice. This approach has been demonstrated in the UK, whereby goal-based outcome measures were jointly generated by the clinicians and patient families, and then utilized for ROM for early childhood cases with disruptive behaviors (Emanuel et al., 2014). The value within this approach is the ability to increase parental and clinician trust in the initial outcome measure’s meaningfulness, and to increase administration and usage of the measure as part of the care process.

While not published in peer-reviewed literature, three private outpatient behavioral clinics of a large pediatric hospital in the US mid-Atlantic region have also developed an outcomes accountability/improvement system using a bottom-up, iterative manner (Yu-Lefler et al., 2019). The system started with clinicians at the clinics developing a simple 0–10 rating scale to measure the level of behavior problem at each appointment for all patients. Subsequent developments consisted of a more sophisticated data infrastructure to incorporate this measure, known as the “Behavior Rating Scale” (BRS), into the patient’s electronic health record; automatic graphing of ratings across time for clinicians and patient families to review at each appointment; and aggregate analyses of BRS change across patients and the proportion of patients reaching optimal BRS ratings at different points of treatment. The latest expansions entail updating the data infrastructure and clinical processes to allow more timely feedback of patient-specific or aggregate BRS data to clinicians and clinical leadership for decision-making; incorporating routinely-collected administrative data (demographic, appointment information) and baseline clinical assessments to study factors influencing BRS outcomes; and adding standardized assessment measures to capture multi-dimensional patient and family outcomes. The sum total of these efforts is a system across a large, multi-site pediatric behavioral health care system with ROM and feedback capabilities. Of particular note is that the system built upon existing clinical procedures and data informatic sources, and as such, cost was kept relatively low and it was easier for clinicians to learn how to utilize the system’s features and incorporate the information gathered into their care processes. This bottom-up approach can help expand the implementation of systems for outcomes accountability that are meaningful to US-based providers treating early childhood disruptive behaviors.

Relatedly, it may be more effective for policymakers to uniquely incentivize private outpatient clinics to build accountability infrastructure and practices from the bottom-up. One possibility is for incentives to include start-up capital for private practices to first develop a metric that can be collected on a routine basis to capture behavioral progress, and to integrate that metric as part of routinely-collected data in the patient’s medical record. Further bonuses can incentivize providers to expand this data infrastructure to include individualized and aggregate data reports and a feedback loop to clinicians and clinical leadership for outcomes improvement. This type of financial incentive structure may be quicker and more cost-effective to implement than those requiring a more sophisticated data infrastructure with multiple, well-developed outcome assessments in place. After such infrastructure is developed, future endeavors can include expanded usage of multi-dimensional assessments that can provide more nuanced data on treatment success and be generalizable across similar treatment programs and patient populations (Seidman et al., 2010).

Limitations

There were several limitations to the breadth and verification of data within this scoping review. This review was only limited to one search engine (PSYCINFO). While a wider search of the same criteria on other search engines (e.g., Cochrane Reviews, SCOPUS, Web of Science, EMBASE, PubMed) may yield a greater breadth of eligible publications, the authors judged that PSYCINFO would provide the most relevant information regarding outcomes accountability work specific to mental health populations. Eligible literature was restricted to peer-reviewed publications. However, since the review findings indicate much of the work in the UK had national-level government support, and there is at least one US-based outcomes accountability/improvement system that has not been published via peer-review, there may be relevant government publications, regulatory or unpublished trial data, white papers, and conference presentations that could potentially further fill the knowledge gaps on the breadth and scope of outcomes accountability system development for early childhood disruptive behaviors. Likewise, eligible publications were limited to only those in English. As one of the studies on early childhood outpatient systems was conducted in the Netherlands, there may be relevant non-English publications that discuss outcomes accountability work for early childhood behavioral healthcare. A follow-up systematic review effort would better address the limitations in this study and yield more comprehensive information on the international development of outcomes accountability systems addressing early childhood disruptive behaviors.

Conclusion

This scoping review illustrates the dramatic lack of scientific information regarding outcomes accountability systems in clinical programs treating early childhood disruptive behaviors. In particular, the present review did not find any peer-reviewed publications on such systems to monitor, assess, and improve behavioral outcomes for American children aged 7 years old or younger with disruptive behavior disorders. While it is challenging to implement outcomes accountability systems and practices, there are powerful voices calling for behavioral healthcare to systematically enhance treatment outcomes in order to mitigate long-term societal and economic burdens from poorly-managed early childhood disruptive behavior disorders (e.g., APA, 2013; Brookman-Frazee et al., 2009; Foster et al., 2005; Scott et al., 2001; Sheldrick et al., 2011; Shevlin et al., 2017; Wertz et al., 2018). As such, an opportunity exists for US mental healthcare service systems, clinical practitioners, and researchers to improve accountability for early intervention of significant disruptive behavior problems in American children.