Introduction

Engagement is widely conceptualized as an individual’s multidimensional (e.g., social, cognitive, affective, and behavioral) involvement in treatment (e.g., Becker et al. 2018; Pullmann et al. 2013). Moreover, engagement is a dynamic process involving interactions among individual, familial, professional (i.e., provider-client), service organizational, and ecological (e.g., availability of services in the community, sources of help typically sought by members of a group) factors (Becker et al. 2018).

Rates of treatment engagement are alarmingly low. National survey data suggest that although approximately 20–40% of youth have a psychiatric disorder (Costello et al. 2011), as many as 50% of youth in need do not enroll in treatment (Merikangas et al. 2010), and more than 50% of those who do enroll terminate treatment early (Pellerin et al. 2010). Given that poor engagement is associated with worse treatment outcomes (e.g., Danko et al. 2016; Kazdin and Wassell 1999), it is imperative that the field continue to find ways to improve engagement.

Research highlights the many obstacles (e.g., competing priorities, stigma, practical challenges such as transportation or scheduling) that make it difficult for youth and families to access and consistently participate in treatment (e.g., Buckingham et al. 2016; Kazdin et al. 1997; Lindsey et al. 2013). Fortunately, there also exists a sizable literature demonstrating the effectiveness of interventions designed to improve engagement (e.g., Becker et al. 2015; Gopalan et al. 2010; Lindsey et al. 2014; Moore 2018).

In the largest review to date, Becker et al. (2018) examined 50 randomized controlled trials testing engagement interventions. Using a “distillation” method (cf. Chorpita and Daleiden 2009), they identified the discrete clinical procedures (i.e., “practice elements”) within effective interventions and examined the empirical associations (i.e., fit) between these engagement procedures and five categories of engagement challenges (i.e., Relationship, Expectancy, Attendance, Clarity, and Homework; REACH; Becker and Chorpita 2016). Of note, certain practices (i.e., assessment and psychoeducation) were associated with effective interventions targeting each of the five REACH domains, whereas other practices were unique to a specific engagement domain (Becker et al. 2018). For example, cultural acknowledgement (e.g., exploring the client’s culture) was commonly associated with interventions that improved Relationship outcomes. Other practices that were associated with specific REACH domains included positive expectation setting (Expectancy), appointment reminders (Attendance), modeling (e.g., demonstrating what a treatment session involves; Clarity), and rehearsal (e.g., providing opportunities for skills practice; Homework).

Scholars have long considered questions about what works for what purpose (cf. Paul 1967); in this context, that means considering how to fit engagement procedures with the focal challenges they are best suited to address. In practice, this means knowing more than just which engagement procedures work (e.g., appointment reminders), but also which practices are best suited to achieve which engagement goal (e.g., appointment reminders are effective for improving attendance, but do not yet have empirical evidence for improving the therapeutic relationship).

Although fitting the right solution to the right problem might sound obvious, it is not a foregone conclusion in clinical practice. Consider an example from mental health services more broadly (rather than engagement in particular). In one study of 60 youth served by 21 providers in a service context that emphasizes and supports high-quality evidence-informed clinical services, of the 28% of youth receiving a high-integrity evidence-based treatment (EBT), nearly half of them (47%) received an EBT that did not fit any of their top three clinical concerns (e.g., receiving trauma-focused cognitive behavioral therapy for depression; Park et al. 2018). In mental health services research, Chorpita and Daleiden (2018) have argued that the fit between problems and evidence-based solutions has been understudied and underspecified, relative to the overall use of evidence-based solutions and the proper delivery of their codified procedures (e.g., fidelity to a specified set of steps for a given procedure). As implementation research continues to find ways to improve procedural integrity, there continues to be a need to emphasize structured guidance for complex decisions involving multiple problems in real-world settings. Such guidance could ensure that providers know which procedures to use, for what purposes, and in what contexts.

Despite the high prevalence of engagement challenges in routine care, there is a paucity of research investigating providers’ typical solutions to these challenges. A nuanced use of the evidence base while making clinical decisions is particularly important, given findings that engagement is multidimensional in nature and that different types of engagement concerns appear to benefit from different focal solutions, derived from different research evidence (Becker et al. 2018). Because of the multivariate complexities routinely encountered in this domain (e.g., a youth and caregiver experience transportation barriers to attending treatment but also express concerns about the fit of treatment for their problem), research findings, if applied at all, have a high potential to be misapplied, given how many different procedures exist corresponding to different presentations of low engagement.

Treatment engagement is ideally suited to the study of the use and fit of engagement procedures from the evidence base. Although low engagement is a ubiquitous problem in clinical services, there exist few “packaged” engagement interventions. Those that exist are often focal in nature (e.g., to improve relationship only; or attendance only), without a coordination model outlining which focal solutions to consider for which challenge. Further, most EBTs do not provide explicit instructions for how to handle engagement issues that emerge unexpectedly during treatment (cf. “run time” design challenges; Chorpita and Daleiden 2014). Finally, research showing what works to improve any engagement problem has not been widely disseminated. Providers are therefore left to address client engagement issues with almost no guidance from the literature; thus, this context is ideal for studying the extent of challenges to using research evidence in clinical care under natural circumstances.

Proper application of research findings to service delivery requires a complex set of behaviors on the part of mental health professionals. Graham et al. (2006) proposed a knowledge-to-action model highlighting behaviors related to the use of research evidence to inform health-related decision-making. Specifically, Graham et al. asserted that, among other key behaviors, the effective application of evidence includes the ability to (a) identify the clinical problem and (b) select a solution from the evidence base that is a good fit for addressing that problem. These two behaviors, identification of problems and their corresponding solutions, were of interest in the current study.

Our primary aim was to assess the reported use and fit of engagement procedures from the evidence base by mental health professionals. To this end, we surveyed school mental health (SMH) professionals about the challenges they encounter when trying to engage youth and families in mental health services and the solutions they apply to try to overcome these issues. We used a mixed-method approach that leveraged qualitative methods to acquire deep understanding of the perspectives of SMH professionals and quantitative methods to test and confirm hypotheses (Palinkas et al. 2011).

We had four aims and related hypotheses. First, what challenges do professionals encounter and do these challenges have solutions in the evidence base? We expected that professionals would report a variety of challenges, most of which would have solutions in the evidence base. Second, what proportion of professionals report using a solution from the evidence base? We expected a moderate proportion of professionals would report using a solution from the evidence base. Third, to what extent do reported solutions fit the challenges? Given other findings related to coordinated application of research to practice, we expected the overall fit between challenges and solutions would be low. Fourth, what are the patterns of use of solutions? This last question lent itself to exploratory analyses, for which we generally expected that certain solutions, such as psychoeducation, would be reported with great frequency, whereas other solutions, such as addressing barriers to treatment, would be reported with low frequency. We also expected to find evidence of overapplication (i.e., use of a solution for many problems, some of which the solution did not fit) and restricted range (i.e., not leveraging a solution for a problem it fit).

Method

Participants

Individuals were recruited at a mandatory, district-wide training provided to all SMH professionals. Although attendance was required, individuals were informed that their participation in the survey was completely voluntary.

Survey respondents were 244 SMH professionals in a large, urban school district serving an ethnically diverse student population in which nearly 80% of students qualify for free or reduced-price lunches (California Department of Education 2016). This district is situated within a county whose Department of Mental Health (DMH) launched in 2009 an ambitious initiative to prepare its workforce to implement more than 50 EBTs (Southam-Gerow et al. 2014), with six evidence-based treatments or evidence-informed systems specifically for children and adolescents: Cognitive Behavioral Interventions for Trauma in Schools (CBITS), Child–Parent Psychotherapy, Managing and Adapting Practices (MAP), Seeking Safety, Trauma-Focused Cognitive Behavior Therapy (TF-CBT), and Positive Parenting Program (Triple P). Therefore, the SMH professionals in this study were trained in one or more of these six interventions and worked within an organizational culture characterized as reflective, evidence-driven, and evaluation-focused. Annually, this SMH program receives approximately 8000 referrals and provides mental health services to roughly 1700 students with varied and complex clinical concerns.

The SMH professionals self-identified as psychiatric social workers (i.e., individuals with a master’s degree in social work, 92.2%), program facilitators (3.3%), clinical administrators (3.3%), psychologists (0.4%), and psychiatrists (0.4%). One individual did not report a professional role. All professionals provided services in health centers strategically located near or within school campuses around the county, to facilitate access to mental health services. Specifically, professionals worked in integrated health and mental health clinics (66.4%), standalone mental health clinics (20.9%), other related settings (e.g., special education or early childhood setting; 7.4%) and in district offices (4.1%). Three individuals did not report their professional setting. No additional information about the SMH professionals or the youth whom they served was collected.

Procedure

Surveys were administered following a 30-min presentation about engaging youth and families in SMH services. This presentation was designed to normalize challenges related to treatment engagement and orient mental health professionals to a strategic focus of the SMH program on increasing treatment engagement. During this presentation, the speaker guided audience members to reflect on engagement challenges they had encountered in their work with youth and families, which were then formally surveyed on a written questionnaire. Individuals were given 20 min to complete the survey; however, most took approximately 15 min to respond to the items. We achieved a 100% response rate (note that we did not recruit the director or associate director of the SMH program, both of whom helped to facilitate our presentation and survey administration).

Measure

Individuals responded to two open-ended survey questions that were the basis for this study: (a) “What is the biggest challenge that you could not solve when trying to engage a youth and/or caregiver in services?” (b) “What did you do to try to solve it?” Individuals also responded to a third question (i.e., “Why do you think it did not work?”) that was not coded for this study due to lack of resources. However, this item served as a validity check such that the study team reviewed responses to confirm that individuals had reported about engagement challenges they could not solve (as opposed to describing solutions they perceived as effective). Professionals who served in roles whereby they no longer provided direct treatment services (e.g., clinical supervision) were instructed to provide their impressions of challenges, based on their observations of direct service providers.

Data Preparation

The research team consisted of four people: one postbaccalaureate student, one postdoctoral scholar, and two doctoral level senior researchers with expertise in treatment engagement and youth mental health services. To capitalize on the synergy of qualitative and quantitative data in this mixed-methods study, data were prepared according to established approaches for each. The mixed-method design included sequential collection of qualitative data, followed by quantitative data (i.e., qual → QUAN) with an emphasis on achieving data integration through the transformation of qualitative to quantitative data (Palinkas et al. 2011).

Coding Engagement Challenges and Solutions

Data were analyzed through a consensual qualitative research method (CQR; see Hill et al. 1997 for full analytic protocol details). CQR involves: (a) researchers conducting inductive analyses of the data, (b) researchers independently coding the data, comparing coding, and making final coding decisions by consensus, (c) auditors checking the analytic process, and (d) researchers verifying results by systematically checking them against the data. Through these methods, CQR balances objectivity with the inherently constructive nature of qualitative data analysis.

Core ideas (i.e., codes) were generated by incorporating existing literature on treatment engagement challenges (e.g., Buckingham et al. 2016) and interventions (e.g., Becker et al. 2018) with initial impressions from independent reviews of the data by two members of the research team (i.e., postbaccalaureate student and postdoctoral scholar). The full research team participated in consensus meetings to review the data and the proposed codes. A final codebook was developed based on consensus of the full research team and included 23 challenges codes and 21 solutions codes (see Tables 1 and 2 for codes and definitions).

Table 1 Challenges, abridged definitions, examples, and kappas
Full size table
Table 2 Solutions, abridged definitions, examples, and kappas
Full size table

To enhance credibility and confirmability, two members of the research team with relatively less exposure to existing scholarship and evidence related to treatment engagement (i.e., the postbaccalaureate student and postdoctoral scholar) used the final coding structure to code all of the data independently (Thomas et al. 2000).

Interrater reliability (i.e., kappas) for the “challenge” codes ranged from 0.43 to 0.93 and were above published standards (at least 0.40; Fleiss 1981). The kappa for other challenges was low (0.06). Interrater reliability for the “solution” codes ranged from 0.40 to 1.00 (Fleiss 1981). Kappas were not calculated for three solution codes (i.e., assessment, goal setting, rehearsal) due to their low occurrence.

Coders met regularly to review nuances between codes, prevent coder drift, ensure overall coding consistency, and resolve discrepancies (Palinkas 2014). Questions for which consensus could not be reached were posed to the full research team for resolution. Additionally, consistent with CQR, a third member of the research team (i.e., one of the senior scholars) served as an auditor by monitoring the coding process and ensuring validity of all final codes.

Selecting an Evidence Base for Engagement Solutions

We used the Becker et al. (2018) review, which identified 30 discrete engagement procedures, as our benchmark for solutions from the evidence base. This review represented the most comprehensive synthesis of the literature available at the time of the current study. At the same time, it is important to note that of the 50 RCTs included in the Becker et al. review, only 1 was conducted in a school setting. Therefore, we used the broadest available literature on engagement in children’s mental health services because the RCT evidence for effective engagement interventions in SMH is so small.

Fitting Challenges to Solutions

The clinical reasoning task of fitting each challenge code with at least one solution code was performed by the two senior members of the research team who possess expert content knowledge about the breadth and depth of engagement challenges and their manifestations. This knowledge was informed by empirical associations established in the review by Becker et al. (2018) well as primary sources summarized in that review that provided more detail about the intended engagement targets for specific procedures (e.g., cultural acknowledgement targets cultural challenges as described in Breland-Noble (2012), and McCabe and Yeh (2009)). The fitting process was done independently by each researcher and was guided by abductive reasoning that is common in clinical decision-making (i.e., beginning with a set of observations and generating the most likely explanation). Specifically, each researcher began with one specific challenge code and, using the professionals’ responses as references, identified the set of possible underlying concerns. For example, the challenge “low hope/efficacy” might result from the youth or caregiver (a) feeling overwhelmed by the number or magnitude of problems, (b) having low clarity regarding how treatment relates to the problem and the typical course of treatment, (c) having low clarity regarding their treatment goals or progress, or (d) having pessimistic thoughts about their own ability to succeed in treatment, the provider’s expertise, etc. From here, each researcher generated a solution set that consisted of engagement procedures that could address these explanations and reduce the manifesting engagement challenge. Continuing the example of low hope/efficacy, engagement procedures would include assessment, psychoeducation goal setting, and positive expectation setting. Upon completion of independent solution-fitting, the two researchers arrived at consensus for the final set of fitting solutions for each challenge.

In this way, the fitting process followed principles of parsimonious covering theory, a model of diagnostic reasoning (Peng and Reggia 1987a, 1987b, 2012; Reggia and Peng 1987) in which the goals of “coverage” are balanced with the goals of “parsimony.” Thus, in the fitting process, the researchers attempted to explain all reported challenges by hypothesizing underlying concerns while at the same time attempting to minimize the complexity of the explanations (Reggia and Peng 1987). The process permitted “many-to-many” associations such that it was possible for a challenge to have multiple fitting solutions and for a solution to fit to multiple challenges (see Appendix for matrix of challenges and expert-fitted solutions). Although it would be possible to have numerous solutions in a set for any given challenge, parsimonious covering theory guided the identification of just those solutions that had an empirical basis and could plausibly lead to resolution of the challenge.

The fitting process was successful except in instances whereby challenges or solutions could not be fitted due to ambiguous or limited descriptions by professionals or due to constraints in the empirical literature (cf. Larsen et al. 2017). For example, the challenges engagement NOS and other could not be fitted to solutions due to their lack of specificity. Additionally, given their absence from the empirical literature on engagement, the solutions empathetic listening, empowerment, family therapy, and solution finding were not fitted to any specific engagement challenges, although they might be considered part of good clinical care in general or for specific situations.

Professionals’ coded data of challenges and solutions were then classified as fitting or not based upon this matrix of challenges and matching solutions. Given that any single challenge or solution was part of a participant’s entire response set, it was not possible to determine a definitive one-to-one fit in the instances in which a participant reported multiple challenges and solutions. We were therefore liberal in our assignment of matches; as such, our results have the potential to overestimate the extent of professionals’ ability to match challenges to solutions. Consider an example whereby a participant reported three challenges (e.g., caregiver wellbeing, crises, and low hope) and two solutions (e.g., case management and positive expectation setting). This participant would be awarded two matches for case management (i.e., caregiver wellbeing and crises) and one for positive expectation setting (i.e., low hope), even though these specific associations were not explicit by the participant.

Data Analysis

Aim 1: What challenges do professionals encounter and do these challenges have solutions in the evidence base? Toward this aim, we calculated the percentage of professionals reporting each of the 23 challenges coded in this study as well as the percentage of challenges that had a corresponding solution in the evidence base.

Aim 2: What proportion of professionals report using a solution from the evidence base? To this end, we calculated the percentage of reported solutions that represented the evidence base as well as the percentage of professionals who reported each of the 21 solutions coded in this study.

Aim 3: To what extent do reported solutions fit the reported challenges? We conducted descriptive analyses to determine the percentage of professionals who reported at least one challenge for which they also reported at least one fitting solution. Additionally, for each challenge category, we calculated the percentage of professionals who reported the challenge and a fitting solution. We conducted chi-square analyses for the sample of professionals reporting each challenge to examine the significance of the likelihood of reporting at least one fitting solution. We also calculated the percentage of professionals who reported each fitting solution for each challenge.

Aim 4: What are the patterns of reported use of solutions? To explore this aim, we calculated the number of professionals who reported each solution and, of those, the number of professionals who reported at least one fitting challenge. Finally, we calculated the percentage of professionals who reported a fitting challenge for each solution.

Results

Aim 1: What challenges do professionals encounter and do these challenges have solutions in the evidence base?

Nearly all (n = 236; 96.7%) of the 244 professionals reported at least one engagement challenge. Collectively, the sample of 244 professionals reported 373 engagement challenges (M = 1.53 challenges, SD = 0.82, range 0–4). As shown in Table 3, the challenge category reported the most by professionals was engagement NOS (18.9% of professionals), followed by motivation (17.6%), systemic factors (12.7%), and caregiver contact (11.5%). Most challenges (n = 311; 83.3%) had a corresponding solution in the evidence base, except for engagement NOS and other.

Table 3 Association of challenges and expert-fitted solutions
Full size table

Aim 2: What proportion of professionals report using a solution from the evidence base?

Most professionals (n = 232; 95.1%) also reported a solution. Collectively, the sample of 244 professionals reported 356 solutions (M = 1.46; SD = 0.79; range 0–4). Most responses (n = 308; 86.5%) represented practices found in the existing empirical literature on engagement. As shown in Table 4, only six solutions were reported by more than 10% of professionals: outreach (22.2% of professionals), psychoeducation (21.7%), accessibility promotion (20.5%), instrumental and professional support (20.1%), rapport building (13.1%), and case management (12.3%).

Table 4 Summary of expert-fitted solutions and challenges
Full size table

Aim 3: To what extent do reported solutions fit the reported challenges?

Of the 244 professionals, only 94 (38.5%) reported at least one challenge for which they also reported at least one fitting solution. As shown in Table 3 (“Professionals reporting at least one fitting solution”), there was variability across challenges for the percentage of professionals reporting a fitting solution. Chi-square tests (see Table 3) indicated that the likelihood of reporting at least one fitting solution was significant for the following challenges: location, (100.0% of professionals reported fitting solution), competing priorities (70.0%), caregiver contact (67.9%), stigma (63.6%), caregiver well-being (61.5%), crises (52.9%), and relationship: family-school (41.7%).

Table 3 (“Professionals reporting fitting solution”) also reveals significant variability in the extent to which specific fitting solutions were reported for each challenge. For example, of the 43 professionals who reported the challenge of motivation, a larger percentage (83.3%) reported motivational enhancement as their solution, relative to the other fitting solutions of barriers to treatment (16.7%) and positive expectation setting (0%).

Although there were no fitting solutions for engagement NOS, outreach was reported most by professionals (34.8% of professionals), followed by accessibility promotion (23.9%), and psychoeducation (17.4%).

Aim 4: What are the patterns of reported use of solutions?

Table 4 presents results for matches using solutions as the point of reference. This perspective demonstrates how often a professional reported using a solution for at least one fitting challenge compared to no fitting challenges. There was significant variability across solutions regarding the frequency of professionals who reported at least one fitting challenge. Reporting at least one fitting challenge was highly likely when professionals reported the following solutions: motivational enhancement (66.7%), positive expectation setting (66.7%), and accessibility promotion (64.0%). In contrast, fewer professionals reported fitting challenges when they reported using the psychoeducation (37.7%), outreach (31.5%), instrumental/professional support (14.3%), rapport building (6.3%), and other practices (see Table 4).

Table 4 (“Professionals reporting fitting challenge”) also reveals significant variability in the extent to which specific fitting challenges were reported for each solution. As an example, 40.0% of professionals who reported psychoeducation also reported trust/fear as a challenge, followed by stigma (30.0%). The remaining fitting challenges for psychoeducation (i.e., literacy, expectations, problem awareness, prior experiences, hope/efficacy) were reported by few or no professionals. As a contrasting example, most professionals who reported motivational enhancement also reported motivation as a challenge (83.3%), compared with competing priorities (16.7%) and problem awareness (0.0%).

Discussion

We endeavored to assess the reported use and fit of engagement procedures from the evidence base by SMH professionals. We achieved this by gathering information about engagement problems they encounter and solutions they apply and analyzing these data within the context of a mixed-methods design. The main results of this study were four-fold: (1) professionals encountered a wide variety of engagement challenges, most of which have solutions in the evidence base, (2) most professionals reported using a solution from the evidence base, (3) there was generally low fit between those solutions and the reported challenges, and (4) some solutions were overapplied, whereas others were underapplied in various ways.

Reported Challenges

Collectively, professionals reported 23 different challenge types in response to the question of their biggest engagement challenge. The patterns of reported challenges suggest that the treatment engagement decisions professionals face when providing clinical care to youth and families are complex in several ways. First, although some type of engagement challenge is likely, no single type of challenge (of the 23 coded) was endorsed by a majority, which suggests that professionals will commonly encounter a problem whose specific nature is itself relatively uncommon. Second, most challenges (83.3%), other than vague ones, had a solution from the evidence base. On the one hand, this is good news, in that these problems are not a limitation of the developing knowledge base itself (a responsibility of researchers). On the other hand, however, it means the challenge lies in the application of that knowledge base (whose responsibility currently falls on the professionals, and often in-the-moment).

Third, the top two challenge categories (i.e., engagement NOS and motivation), each reported by approximately 18% of professionals, hinted at difficulties in distinguishing among types of engagement challenges. The individual responses coded as engagement NOS often were synonyms for low engagement (e.g., “buy-in,” “parent engagement”), suggesting that professionals struggled to disambiguate the nuances of low engagement. Individual responses coded as motivation (e.g., “apathy,” “resistance,” “hard to engage”) appeared to reflect internal, stable attributions of youth and caregivers. However, it is possible that professionals misconstrue as low motivation other challenges that remain undisclosed by youth and families (e.g., stigma, distrust of government agencies, poor prior experiences with mental health services; McKay et al. 1996), and this may be an issue of sensitivity to those more granular and focal indicators of engagement (Becker and Chorpita 2016). It is also possible that behaviors that professionals interpret as “resistance” (e.g., expressing concerns about treatment, asserting oneself strongly about treatment preferences) could instead reflect high engagement as consumers advocate for their own needs (Buckingham et al. 2016).

Professionals who have difficulty differentiating nuance or who attribute low engagement to a youth’s or family’s personal trait might also then struggle during the process of selecting effective solutions for these problems. In the current study, professionals whose engagement challenges were coded as engagement NOS were most likely to report using strategies empirically associated with youth or caregivers attendance (i.e., outreach, accessibility promotion); suggesting that their default operationalization of low engagement is attendance, which then also influences the solutions that professionals employ.

Reported Solutions

Most reported solutions (86.5%) were consistent with practices in the empirical literature related to treatment engagement, suggesting that professionals have knowledge of engagement procedures from the evidence base. It is noteworthy that although the evidence base on treatment engagement included only one RCT conducted in a school setting, professionals are knowledgeable about and report the application of engagement strategies that have been developed and tested in other settings.

Six solutions were reported by a greater proportion of professionals (i.e., at least 10%) than were other solutions: outreach, psychoeducation, accessibility promotion, instrumental/professional support, rapport building, and case management. It is not surprising that these solutions were offered, given their fit with challenges related to youth/caregiver contact, practical barriers, and the fundamentals of engagement (e.g., relationship and understanding of therapy) during the early phase of treatment. Many solutions (e.g., barriers to treatment, cultural acknowledgement, motivational enhancement), however, were reported by few professionals. We cannot know from our data if these response patterns reflect professionals’ knowledge of the evidence base on engagement. It is possible that professionals know a variety of engagement procedures but did not record them as a response. However, it is also possible that certain procedures are not well-known by most professionals because with few exceptions, engagement interventions are not widely disseminated on their own, independent of EBTs.

Fit of Solutions to Challenges

Despite high reported use of at least one engagement practice from the evidence base, 61.5% of professionals reported no practices that fit any of their reported challenges. Patterns of fit varied across challenges, such that there was a greater likelihood of reporting a fitting solution for only 7 of 21 challenges for which fitting was possible. Our impression is that it generally appeared that nominating a fitting solution was more common when there was a clear signal for action (e.g., contact caregiver, crisis, competing priority) relative to other barriers that were more internal to the youth or caregiver (e.g., motivation, trust/fear, hope/efficacy). Future research would be necessary to further explore this idea.

Patterns of Fit

Non-optimal fit of solutions to challenges were one of two types: overapplication or underapplication. Overapplication involved using a strategy from the evidence base and applying it to a challenge for which the evidence base provides no support. For example, although psychoeducation was reported by 21.7% of professionals, it was reported in conjunction with at least one fitting challenge by approximately one-third of professionals and with no fitting challenges by two-thirds. Psychoeducation also provided an example of underapplication, which is the use of a known solution with only a subset of the engagement challenges for which it is supported in the evidence base (e.g., psychoeducation was never reported in conjunction with the fitting challenge of hope/efficacy, a challenge to which only one professional reported a fitting solution). Using the right clinical procedure for the wrong problem or not leveraging all of a procedure’s potential uses can take up valuable clinical time, might reduce a professional’s sense of efficacy for addressing engagement challenges, and could make the problem worse if the youth or family perceives that they are misunderstood by the professional.

Implications

This study demonstrates that the existing evidence base is sufficient to address engagement challenges and that professionals report using engagement strategies from the literature. Thus, the problem we are trying to solve relates neither to a limited evidence base nor to unfamiliarity with that evidence base. Instead, the problems are (a) how to expose professionals to a continually growing and evolving evidence base and, more importantly, (b) how to support the strategic application of the various engagement procedures from the literature, so that they are best matched to the challenge at hand.

In the absence of structured supports, it is unrealistic to expect professionals to optimally apply the evidence base to address engagement challenges. Although there are many possible options, we propose that these problems could be addressed by a system that maps the associations between engagement challenges and their empirically indicated clinical procedures. One resource in this system might be a tool for assessing engagement, with a taxonomy granular enough to identify specific engagement challenges. Another resource in this system might be a synthesis of the entire evidence base of engagement practices, thereby increasing the availability of underutilized options, reducing overapplication, and expanding the range of solutions to more and better-suited engagement challenges. Additionally, this system would require a common language or ontology across the resources to enhance the mapping of challenges to their solutions (Chorpita et al. 2005). We have found preliminary evidence that such a system increases the likelihood that a provider will identify an engagement challenge and select from the evidence base a matching engagement solution (Becker et al. 2019) and would encourage others to consider novel design solutions to solve these problems.

Limitations

Our primary survey included two open-ended prompts in a paper-and-pencil format. Survey responses might have been biased due to time, space, or professionals’ preferences about how to respond. Some professionals wrote brief, unclear, or indiscernible responses, which could not be coded and were therefore excluded from analyses. Although the sample size was large, caution is warranted when interpreting the fitting results for coding categories that had few endorsements.

Survey items were designed to elicit responses to unsolved engagement challenges and intended to serve as a needs assessment prior to a subsequent workshop training for supervisors and providers in a system of resources to support youth and caregiver engagement in SMH services (Becker et al. 2019). We did not survey professionals about successful resolution of engagement challenges; it is possible that there would be a higher rate of fit between solutions and resolved challenges. However, we contend that unsuccessful attempts at engaging youth and families is exactly what the field needs to attend to. The onus is on researchers to find ways to better leverage the existing evidence base to solve problems that matter to providers and that, left unsolved, have a detrimental impact on treatment success for youth and families. This study represents one small demonstration of how the evidence base can be applied to illuminate potential opportunities for improvement.

A different study design might have more precisely illuminated patterns of coordination between challenges and solutions. For example, providing a list of REACH engagement problems and a possible set of practices could offer some benefits over the current design by reducing error variability related to coding and allowing a clearer one-to-one mapping of challenges and solutions. By providing an ontology for engagement challenges and solutions, such a study design would likely have reduced the frequency of engagement NOS responses and might also have increased the reporting of solutions from the evidence base.

Certainly, any survey has the potential for underreporting of actual behaviors. The purpose of this study was to examine the natural ontology and solution mapping used by professionals so that we could understand their conceptualization of engagement and engagement practices. Providing a structured set of choices at the outset would have introduced the possibility of professionals biasing themselves more positively in response to questions about engagement challenges and solutions. In the absence of a method for one-to-one mapping of challenges and solutions, we were generous in our credit and our results likely overestimate the fit between challenges and solutions.

Our expert-fitted solutions represent one attempt to apply broad empirical associations from the engagement literature to granular and nuanced challenges reported by mental health professionals. Although informed by empirical evidence, expert matches are subject to critique for possible errors of omission that would end up underrepresenting the extent of fitting solutions reported by professionals. Additionally, the analyses of challenge-solution fit in this study have limitations, given that they were based upon interpretations of written responses. It is possible that our analyses of fit were a valid representation of what the professionals intended with their written accounts of how the engagement procedures were applied. It is also possible that they overestimate the concordance between challenges and solutions because we could not determine a definitive match in the instances in which professionals reported multiple challenges and solutions.

Another possible critique involves the use of the Becker et al. (2018) review as a benchmark for professionals’ solutions due to its overrepresentation of clinic- and community-tested engagement interventions relative to those tested in schools. It could be argued that the basis for comparison of solutions should be those studies in which engagement interventions were tested in schools. We relied on the broadest available literature because filtering down to the sole school-based RCT in the evidence base would have been methodologically inadequate. Moreover, in addition to setting, one could also make the case that there are many other potential ways to filter the evidence base (e.g., clinical problem, age, gender) to further approximate the expected values for any particular sample (e.g., for this study, one could look for RCTs involving Latinx youth). The evidence base does not have hard boundaries; instead, there are gradations that are only possible if the research literature is sizable. At this time, there exists evidence to demonstrate only that the target engagement problem is a moderator of intervention selection; that is, the Becker et al. study demonstrated that engagement solutions differ according to the REACH domain. Thus, we acknowledge that setting might likely be an important moderator of engagement solutions, but there would be significant tradeoff to increasing granularity by relying on a small literature.

Although only one code (i.e., other challenge) yielded a kappa below published standards (i.e., < 0.40; Fleiss 1981), several codes (i.e., caregiver coping, case management, empathetic listening, institutional support) were close to the minimum threshold. Possible explanations include high or low prevalence of the codes, which would result in lower kappas (Landis and Kock 1977), or ambiguity of the code definitions that led to inconsistent application. Additionally, three codes (i.e., assessment, goal setting, rehearsal) did not occur frequently enough in this dataset enough to calculate kappas. It is possible that the low occurrence reflects the true prevalence, but it is also possible that the definitions or application of these codes could be improved. It follows that the codebook would benefit from closer examination of the definitions for certain codes prior to its application to another dataset. Given that our primary purpose was to examine overall patterns of challenges and solutions from the evidence base, we chose to retain these codes and to use an auditor to ensure validity of the final codes (Hill et al. 1997).

We did not ask professionals about many factors that could aid the interpretation of these data, such as experience with specific engagement strategies more generally and self-efficacy addressing engagement challenges. It is unlikely that the specific challenge and solution findings from this study of SMH professionals employed by an urban school district with a student population described as primarily Latinx and meeting the threshold for poverty generalize to other SMH professional populations. Moreover, it is unlikely that the specific challenge and solution findings from this sample generalize to professionals who work in other service settings and likely encounter different challenges and have different training backgrounds and skill sets related to solutions. At the same time, this school district and county represent a context of high availability of evidence-based practice resources and fiscal incentives encouraging their use. Thus, we contend that the findings representing high reported use of the collective set of solutions from the evidence base and concomitant low fit of solutions to challenges should give pause in such a context and that less advantageous contexts for SMH services might yield less favorable results.

Conclusion

This study attempted to assess the use and fit of the evidence base to engagement challenges. Our discoveries show there may be a need for specific decision supports that leverage client-level evidence and research evidence and provide an explicit framework to coordinate decisions around problem identification and solution selection. Findings from this study have the potential to highlight opportunities for decision-centered design that supports the application of the scientific literature to complexities related to treatment engagement over the course of clinical care.