Introduction

As policymakers have turned their focus to closing the gap between mental health research and practice, states are increasingly seeking ways to increase the use of evidence-based programs for youth and families into public service systems. These initiatives aim to improve the quality of mental health services in the public sector, and to more effectively and efficiently address the needs of their population. Policy leaders, administrators, and expert trainers are faced with the challenge of finding effective means for training and retooling an already practicing workforce. Studies have consistently shown that single-incident trainings do not lead to lasting changes in clinician behavior, and subsequently, do not translate to the long-term use of new treatments, unless coupled with additional educational strategies (Beidas and Kendall 2010; Bero et al. 1998; Bickman 1999; Cauffman et al. 2002; Davis 1998; Forsetlund et al. 2009). However, there is little evidence on the most effective models for ongoing clinician training in evidence-based practices (EBPs) within these large-scale rollouts.

To support implementation of EBPs on a large scale, group training and consultation models that incorporate interactive in-person training workshops and ongoing expert consultation have gained traction as an alternative approach to traditional training and supervision models used in clinical trials (Ebert et al. 2011; McHugh and Barlow 2010; Mittman 2004; ØVretveit et al. 2002). Although findings are still preliminary, the addition of ongoing phone consultation to in-person training appears to be a promising strategy for improving uptake and the conversion of training techniques to actual practice (Beidas and Kendall 2010; Herschell et al. 2004; Miller et al. 2004; Morgenstern et al. 2001; Sholomskas et al. 2005). Existing studies point to the importance of consultant adherence to a consultation protocol (Schoenwald et al. 2004), and a sufficient number of consultation hours (Beidas et al. 2012) in order to support therapist adherence (Beidas et al. 2012; Schoenwald et al. 2004), and post-treatment youth outcomes (Schoenwald et al. 2004). This emerging evidence also suggests that consultation should be conducted with organizational, contextual, and provider-level issues in mind (Beidas and Kendall 2010).

Conceptual models for innovation implementation highlight critical organizational factors (e.g., tangible supports, leadership, provider buy-in), community factors (e.g., perceived relevance by community members, family engagement), policy-level factors (e.g., financing, legislation), provider characteristics (e.g., attitudes, prior experience), and innovation characteristics (e.g., ease of use, relevance) that impact the uptake and use of new practices (e.g., Aarons et al. 2011; Damschroder et al. 2009; Feldstein and Glasgow 2008; Fixsen et al. 2005; Greenhalgh et al. 2004; Wandersman et al. 2008). One of these models, the Interactive Systems Framework (ISF) is particularly useful for contextualizing the role of consultation within a larger implementation framework (Wandersman et al. 2008). The ISF includes three main interrelated systems relevant to dissemination and implementation: (1) the synthesis and translation system, (2) delivery system, and (3) the support system. The purpose of the synthesis and translation system is to take research and to make it usable by providers and consumers in real-world settings. The delivery system refers to service delivery entities at the organizational, community, state or national level. The third system, the support system, has two primary functions: (1) to provide innovation-specific support (i.e., information provided to decision-makers, training, consultation), and (2) to provide general support and capacity-building intended to enhance general infrastructure, skills, and motivation of an organization (Wandersman et al. 2008). Expanding on this model, (Wandersman et al. 2012) have proposed an approach to developing and testing evidence-based systems for innovation support (EBSIS) that highlights the importance of research on technical assistance, consultation, and quality assurance.

The current study explores the content of expert consultation provided to clinic supervisors in clinics implementing an evidence-based practice within the context of a statewide EBP training and implementation support program. In 2005, the New York State Office of Mental Health (NYS OMH) established the Evidence-Based Treatment Dissemination Center (EBTDC) to train clinicians to implement EBPs in outpatient community clinics and other service settings across New York State. EBTDC uses a translation-based training and consultation model that includes a 3-day in-person training workshop in conjunction with ongoing biweekly phone consultation for participating clinicians for 1 year (Gleacher et al. 2011). EBTDC was developed to support state system implementation of EBPs via ongoing consultation. In accord with current thinking on implementation strategies, EBTDC has supported agencies at multiple levels (supervisors, clinicians, agency leadership) and has been continuously funded for 8 years because of strong support from OMH and from the clinic leadership that has participated actively in each of the rollouts. Through the ISF lens, EBTDC represents a prime example of an interconnected statewide service delivery system and innovation support system. This study provides a window into one aspect of this broader model of service delivery and innovation support, specifically the state’s role in providing innovation-specific technical assistance (Wandersman et al. 2008, 2012).

In the year of this study, EBTDC focused on parent and child treatment for disruptive behavior disorders (DBD). Specifically, participants were trained in Parent Training for Disruptive Behavior Disorders (Wells et al. 2008) and the Coping Power Individual Child Program (Lochman and Wells 2004, 2008). The supervisor program was initiated because in prior years of EBTDC trainings, staff observed challenges in uptake and adherence to EBP protocols that appeared to be related to clinician-reported discrepancies between messages from EBTDC consultants and their clinic supervisors. As many community-based clinicians are simultaneously receiving both outside consultation and clinic-based supervision, consistency is important. Outside consultation is typically temporary, making ongoing clinic-based supervision critical in supporting clinicians’ use of new skills and agency-level sustainability of techniques over time.

In response to this concern, in its second year, EBTDC staff initiated a supervisor-specific consultation program designed to complement the ongoing consultation provided to EBTDC-trained clinicians. These supervisor-specific consultation calls were intended to support the clinic supervisors in their work with EBTDC-trained clinicians. During these calls, EBTDC expert consultants sought to increase supervisors’ knowledge of fundamental cognitive behavioral therapy (CBT) concepts and specific treatment protocols, so that they could more effectively support their clinicians to bridge this knowledge into practice. Calls also served as a forum in which supervisors could problem-solve around implementation barriers that arose, address administrative and programmatic issues, and discuss issues related to treatment fidelity and adaptation.

Clinic-based supervisory support for EBP implementation is critical, as supervisors are uniquely positioned to provide EBP support within the context of community service settings (Beidas and Kendall 2010; Hoagwood and Kolko 2009). However, participating EBTDC clinic directors reported that it was not feasible to reserve supervision time solely for clinical guidance because other issues (e.g., risk management, administrative tasks) also required attention. These reports accord with findings from research that suggests that clinic supervision can have a different focus from supervision provided in research and training settings. For instance, as part of larger study of treatment for disruptive behavior disorders in usual care, Accurso (2011) found that clinic supervision focused on administrative tracking, supportive assistance (e.g., self-care for clinicians), as well as clinical content. Even though half the supervision time examined in this study was spent discussing therapeutic interventions and case conceptualization, there was minimal coverage of elements of evidence-based treatments for children with disruptive behaviors. Thorough review of EBP techniques for these commonly treated disorders occurred in fewer than 5 % of the sessions (Accurso et al. 2011).

Although it seems intuitive that supervision would influence the quality of care or content of sessions, systematic reviews have found few methodologically sound studies that demonstrate a significant relationship between clinical supervision and clinician adherence to treatment protocols, or between clinical supervision and improved youth outcomes (Bickman, 1999; Callahan et al. 2009; Ellis and Ladany 1997; Kilminster and Jolly 2000; Wheeler and Richards 2007). Yet, there is a burgeoning interest in investigating the associations between ongoing supervision, quality of care, and treatment outcomes (Beidas and Kendall 2010; Heaven et al. 2006; Schoenwald et al. 2009). A study of multi-systemic therapy (MST) found that adherence to MST principles during supervision predicted therapist adherence to the model in session, as well as positive youth treatment outcomes (Schoenwald et al. 2009). Additionally, two studies using random assignment showed promising effects of supervision on different measures of treatment outcome and process, such as working alliance, symptom reduction, treatment retention (Bambling et al. 2006), clinician adherence and skills related to EBPs (Sholomskas et al. 2005). Feasible methods of providing expert consultation, (e.g., phone-based small group consultation), have been explored as a way to scale up EBPs. Phone consultation offered by EBTDC and similar programs to frontline clinicians may fulfill a similar role to supervision; however, little is currently known about the capacity and skills needed among real world, community-based supervisors to support quality implementation of EBPs.

Insight into the content and process of consultation calls within current practice settings will help identify core elements to optimize the scaling-up of EBPs in community settings. The current study was designed to increase understanding of consultation content and process among clinic supervisors charged with supporting EBTDC-trained clinicians. Minute-to-minute live coding of a subset of consultation calls with clinic supervisors was conducted during the fourth year of the EBTDC’s large-scale training effort. This pilot study builds directly on a previous microanalysis of clinician consultation calls to identify effective distance training and consultation strategies for clinicians (Pimentel et al. 2009). In this study, approximately one-third of the consultation call minutes were spent addressing specific CBT techniques and one-third of the call minutes on assessment and broader case formulation, with 20 % of call time spent on nonclinical administrative content. Given the absence of prior attention to supervisor-specific consultation, our primary interests in this exploratory study were descriptive (i.e., to identify topics discussed on these consultation calls with clinic supervisors, the time spent speaking by consultants and clinic supervisors, and the distribution of topics on the calls).

Methods

EBTDC Consultation Call Structure

In the year of this study, EBTDC trained 344 clinicians and supervisors in parent and child treatment for DBDs. Supervisors and clinicians attended one of nine 3-day workshops offered from June to November. The first day covered an overview of EBTDC requirements (i.e., consultation call assignments, completion criteria, structure), the fundamentals of cognitive-behavioral conceptualization, diagnostic issues, and assessment procedures. Days 2 and 3 covered the Parent Training for Disruptive Behavior Disorders (Wells et al. 2008) and the Coping Power Individual Child Program (Lochman and Wells 2004, 2008) presented by the respective treatment developers or their expert trainers. Following the EBDTC clinical training, which all EBTDC supervisors attended, supervisors were offered 1 year of monthly telephone consultation focused on implementing DBD treatments in real-world clinic settings.

Both EBDTC consultants had several years of experience training and consulting with community clinicians on different evidence-based treatments. The EBTDC consultants, in collaboration with OMH partners, developed the structure for the supervisor calls. The consultants and OMH intentionally did not adhere to a set curriculum for the calls, believing that call content should be shaped, in part, by emerging needs. Rather, the team developed core goals for the calls. These included a focus on implementation barriers (e.g., optimizing supervision time to support EBTDC treatments, selecting appropriate cases), EBTDC-related program issues (e.g., assessment of further training needs, program completion issues), and the use of a CBT conceptualization and methods in supervision (e.g., using supervision time for practice). In addition, supervisors were also given the opportunity to take part in “Special Topic” calls led by the treatment developers. Consultants met weekly to discuss their calls, development of content, and consultation strategies. Discussions with OMH about EBTDC needs were conducted weekly.

Consultation calls were scheduled to last 45–50 min. Participation in the supervisor calls was voluntary; however, supervisors were expected to present a case from a supervisory perspective one time over the course of the year and to attend 75 % of the calls. The EBTDC program served 80 supervisors assigned to calls with a maximum of 8 supervisors in each call group. Clinicians—not the subject of this current investigation—took part in biweekly calls for the entire year. Additional information about New York State OMH’s EBTDC training model has been reported elsewhere (Gleacher et al. 2011).

Sample

Consultation calls and individual minutes within these calls served as the primary unit of analysis. Over the course of approximately 5 months, all supervisor consultation calls (n = 33) totaling 1547 min were coded, representing 45.2 % of the 73 supervisor calls for the year. All consultation calls conducted during the 4 months study period were included. Calls were on average 47.75 min long (SD = 10.48; range 17–60). Twenty-nine of the calls were conducted by one of the two EBTDC supervisor consultants (12 and 17 calls, respectively), and four of the calls were “Special Topic” calls led by the treatment developers. The content of these special calls was guided by interest and need among clinic supervisors. The four Special Topic calls addressed: (1) family engagement related to the parent training protocol, (2) a demonstration of how to implement “time out” as part of parent training, (3) treatment adaptations for younger and older children within Coping Power, and (4) provider questions about the adaptation of specific manual techniques. Due to the requirement by OMH for anonymous data collection in this study, no supervisor demographics or attendance data were collected.

Procedures

This project received approval from the New York State Psychiatric Institute Institutional Review Board (IRB) and the NYS OMH. Prior to any call coding, all EBTDC supervisors were sent a letter via email describing the scope of the project, (i.e., to code the content of the EBTDC consultation calls). They were informed that if they did not wish to participate, they could opt out and their call group would be excluded. They were also informed that no individual-level information would be tracked (call attendance, names). No potential subjects opted out.

Measures

Coder Training and Reliability

As audiotaping consultation calls was not permitted, coders conducted live minute-to-minute coding verbalizations of call participants (i.e., the consultant, clinic supervisors). Utilizing a computer timer, they coded verbalizations at 60-second intervals by topic discussed and speaker (consultant or clinic supervisor). Coders completed a Minute-to-Minute Coding Sheet and summarized the data from each call on a Consultation Call Summary Coding Sheet. The two coders, both of whom had expertise in CBT, participated in 6 weeks of reliability training, which included the development of a codebook and decision rules. The coders then attended ten consultation calls and coded these calls in vivo. Codes were considered a match if the total minutes coded for the topic for each coder were within a 5 min range. Inter-coder reliability was assessed with intraclass correlations (ICCs), and reliability was adequate (all ICCs >0.75). Table 1 provides an overview of the codes.

Table 1 Description of codes
Full size table

Results

Supervisor Consultation Call Content

Topics

Figure 1 depicts the overall breakdown of topics discussed on the calls. Calls addressed program/administrative issues (26 % of the time), CBT model and techniques (14.2 %), case conceptualization and assessment (13.6 %), parent and child engagement (12.8 %), CBT supervision and clinician engagement (11.8 %), fidelity/flexibility (10.2 %), other clinical issues (5.4 %), and “off task” discussion (4.6 %). The percentages remained similar when we excluded the four Special Topic calls. The calls for the two primary EBTDC consultants differed significantly in the total number of minutes on their respective phone calls spent on CBT techniques (14.2 vs. 6.1 %; X 2(1) = 21.95, p < 0.001), conceptualization and assessment (19 vs. 10.9 %; X 2(1) = 17.35, p < 0.001), fidelity and flexibility (6.7 vs. 9.8 %; X 2(1) = 4.13, p < 0.05), parent and child engagement (8.4 vs. 17.2 %; X 2(1) = 23.56, p < 0.001), program administration (30.5 vs. 25 %; X 2(1) = 15.00, p = 0.001) and off-task discussion (6.5 vs. 2.6 %; X 2(1) = 10.60, p = 0.001). There were no differences in time spent on CBT supervision and other clinical issues.

Fig. 1
figure 1

Distribution of topics discussed across all consultation calls

Full size image

Topic Discussed by Speaker

In terms of total minutes spent speaking, the consultants (EBTDC consultants and treatment developers) spoke more on the calls than the clinic supervisors (1,052 vs. 522 min). Figure 2 depicts the percentage of total speaking time spent on each topic by the different speakers (n = 837 min for EBTDC consultants on ongoing calls; n = 215 min for treatment developers on Special Topic calls; n = 522 min for clinic supervisors across all calls). Chi square analyses revealed significant differences in the proportion of time each speaker spent speaking about each of the eight different topics (X 2(3) = 19.33–118.20; p-values range from 0.005 to <0.001). Of the total time speaking, EBTDC consultants leading ongoing consultation calls had an approximately even distribution across the clinically relevant topic categories with the exception of program administration, which was discussed 33 % of the time. Among the clinic supervisors, the most heavily discussed topics were conceptualization and assessment (24.1 %), program administration (22 %), and parent and child engagement (14.9 %). In contrast, of the total time spent speaking on the calls, the treatment developers leading Special Topic calls spent 36.7 % of their time speaking about CBT techniques, 25.6 % of their time speaking about fidelity and flexibility, and 19.5 % of their time talking about parent and child engagement.

Fig. 2
figure 2

Percentage time on supervisor consultation topics by speaker (n = 837 total minutes spoken by EBTDC consultants; n = 215 for treatment developer Special Topic calls; n = 522 for clinic supervisors)

Full size image

Distribution of Topics during Calls

As shown in Fig. 1, with the exception of program administration, which was discussed for one-fourth of all minutes coded across calls, the majority of the clinical topics were discussed for similar amounts of time. In order to explore how this time was distributed within the calls, we examined the proportion of call time spent on each topic for each of the 33 supervisor consultation calls, separating the Special Topic calls from the ongoing monthly consultation calls. Each call had 1–3 topics that were addressed for 25 % of the time or more, with the other topics discussed minimally or not at all.

Table 2 depicts the number of calls in which a given topic was discussed for at least 25 % of the total call time (major discussion topic), 15–24.9 % of the time (moderate discussion topic), or less (minor discussion topic) (see Table 1). CBT techniques and conceptualization and assessment were discussed were major topics on seven calls. Fidelity/flexibility and parent engagement were major topics on 6 calls. CBT supervision was a major topic on four calls. Only on two calls was ‘off-task discussion’ or discussion of other clinical issues a major topic. When the four Special Topic calls with the treatment developers were omitted, the number of calls addressing CBT techniques as a major topic dropped from 7 to 3, the number of calls addressing parent and child engagement as a major topic dropped from 6 to 5, and the number of calls for fidelity and flexibility dropped from 4 to 2. There was no change for the other topics.

Table 2 Proportion of time on spent on major, moderate, and minor topics across supervisor consultation calls
Full size table

Discussion

Community-based clinic supervisors are an important, but often-overlooked part of the workforce in the development of implementation and dissemination strategies for evidence-based practice. This exploratory study represents an effort to understand consultation content among supervisors taking part in a large-scale state dissemination program, EBTDC. We examined the topics discussed in consultation calls and the distribution of these topics through minute-to-minute live coding. Our findings point to a range of issues consistently covered by clinic supervisors and consultants, with a primary focus on the clinical application of parent and child disruptive behavior disorder (DBD) treatments, and on programmatic issues. There were also interesting patterns of usage within calls and by consultants, which point to the need to identify an optimal structure and format for consultation.

Broadly, our results suggest that supervisors and consultants spent approximately 51 % of their total speaking time covering a range of clinically relevant topics (e.g., CBT techniques, fidelity and adaptation issues, case conceptualization, parent and child engagement in the treatment for DBDs). An additional 12 % of the time was spent discussing supervisory issues, such as supervising from a CBT perspective and engaging clinicians in the implementation of the treatment protocols. The other major topic was administrative/programmatic issues, which comprised roughly one-fourth of the discussion across all the minutes coded. Not surprisingly, as shown in Fig. 2, the consultants spoke for more time than the clinic supervisors and spent 33 % of their speaking time covering program administration, with the rest of their time distributed across a range of clinical topics (CBT techniques, fidelity/flexibility, engagement, conceptualization). Of note, they spoke more frequently about specific CBT techniques than did the clinic supervisors (14 vs. 6 %). Clinic supervisors, on the other hand, spent the largest proportion of their speaking time discussing case conceptualization (more so than the consultants, 24 vs. 11 %). Although we do not know the sequencing of the discussion and there is need for further research, the greater focus by clinic supervisors on conceptualization may signify discussion of case details and a need for assistance in developing CBT-informed treatment plans. Consultants, in turn, may have been responding to this by providing information about specific CBT techniques related to the treatment plan and conceptualization. Consultants also spoke more about program administration than clinic supervisors (33 vs. 22 %). However, it was clearly a primary topic across both groups and consultants likely had more information to provide. The topics identified, and speaking patterns observed, are generally consistent with those found for clinician consultation calls in Pimentel et al.’s (2009) study of a similar EBTDC rollout of depression and trauma treatments. The coverage of a broad range of clinical and administrative issues in consultation seems to represent a parallel process to clinic-based supervision, as reported by EBTDC supervisors on calls and in the literature (Accurso et al. 2011).

Examining the treatment-related codes more closely, 24 % of the consultation call time was focused on CBT concepts/techniques and fidelity/flexibility. This is encouraging in that it suggests that there was substantial discussion of the direct application of the treatments and their core components. On a related note, a substantial proportion of the calls were spent on case conceptualization (14 %) and parent and child engagement issues and strategies (13 %). This is again an indicator that the calls were covering salient clinical issues ranging from case selection, assessment, core treatment components, to flexible application of the treatments. Parent engagement was particularly important given the nature of the Parent Management Training (PMT) component of the disruptive behavior disorder treatments. In PMT, parents must come in regularly for parent sessions. Anecdotal evidence from EBTDC participants suggests that while the critical role of parents in the treatment of DBDs was understood in theory, clinicians often needed consultation around how to communicate this critical role to parents and help problem-solve the logistical barriers, i.e., how to bill for collateral sessions with parents and address common barriers to treatment (e.g., McKay and Bannon 2004). It is difficult to interpret the larger proportion of time spent discussing the treatment protocols and their application versus the clinical supervisory issues (i.e., clinician engagement, using a CBT perspective in supervision). However, this finding may suggest that the supervisors themselves may be in need of further training in the specific treatment protocols before they can effectively address the supervisory-level issues.

What is perhaps most noteworthy is the large proportion of call minutes spent covering administrative and programmatic issues related to the EBTDC program. When we examined how time was spent on the topic within each call, we found that 61 % of the calls (20 calls) spent a quarter or more of their call time on program administration. This is in contrast other clinical topics (e.g., CBT techniques, fidelity and flexibility), which were discussed for this duration of time on far fewer calls. The “program administration” topic included reviewing progress, completion criteria, data reporting systems, and clinics’ consultation and support needs. In particular, supervisors and consultants spent time on calls reviewing the status of the individual clinicians on each supervisor’s roster, so that supervisors could help ensure that these clinicians were assigned the appropriate cases and were not encountering significant barriers. It would be useful to further explore the reason for this intensive focus on administrative issues, given that a similar pattern was found in earlier work on clinician consultation calls (Pimentel et al. 2009). Although it was not an explicit goal of the consultation to focus largely on program issues or the data reporting systems (which were introduced directly by OMH to the clinics), the EBTDC consultants essentially served as liaisons between OMH and the participating clinics. Assuming that both programmatic and clinical-training functions are important, one strategy may be to move administrative and program discussion to another forum and/or communicate about the program through other means (e.g., email blasts, web portals, special administrative calls).

A primary issue is how to maximize time focused on clinical and implementation issues so that the supervisors are able to support their clinicians clinically once expert consultation is no longer available. Our analysis of the distribution of topics on individual calls revealed that the call groups tended to spend time on two or three key topics per call. This suggests that each call did have some focused discussion. However, given the myriad consultation needs identified through these calls, it may be that the monthly consultation calls were simply not sufficient. While some clinic supervisors also participated in the EBTDC clinician program, many of them were not seeing EBTDC cases and their clinical backgrounds and expertise varied tremendously. This point is further underscored by data from an OMH-administered web survey of a subsample of EBTDC-trained clinic supervisors. The survey found that these supervisors supervised an average of 2.7 EBTDC-trained clinicians, and felt more effective as a supervisor in general than they did in supervising the use of EBPs. In fact, supervisors felt less effective in their supervisory role as the number of EBTDC-trained clinicians they worked with increased (Nadeem et al. 2011). Future research should continue to examine supervisor factors more closely so that appropriate supports can be developed and tested.

Interestingly, as EBTDC evolved in real-time, the consultants and EBTDC directors noted a need for more in-depth focus on particular issues and instituted Special Topic calls for supervisors that were led by the treatment developers. Our analysis included four of these calls, which, by design, minimized the potential for discussion of program administration (11.6 % across the calls) and no calls where program administration comprised a quarter or more of the call time. In fact, these Special Topic calls were most heavily focused on specific CBT techniques, fidelity and adaptation issues, and family engagement. When we examined within-call content for all the calls coded, we found that in the absence of these Special Topic calls, only 4 ongoing EBTDC consultation calls focused on CBT techniques and 4 calls focused on fidelity/flexibility for a quarter of the call time. These data suggest that a more structured consultation program, with clearly identified core components, and sufficient intensity may be important for effecting change in supervisory behaviors. The challenge is how best to offer this kind of intensive support while balancing the demands on participants and aligning implementation supports in the current fiscal climate with existing clinic structures.

The two primary EBTDC consultants leading ongoing calls differed from each other in their emphasis on the calls. One consultant spent more time on CBT techniques and conceptualization/assessment, while the other spent more time on fidelity/flexibility and family engagement. Since all of these topics have clear clinical relevance to the specific treatments being disseminated through EBTDC, it would be helpful in future studies to examine the ways in which different consultants are communicating similar concepts. Due to the relatively informal structure of the calls, we also do not know whether specific supervisor or clinic needs precipitated focus on certain areas. Nonetheless, the presence of these consultant differences points again to the need for standardization of the consultation structure, and for identification of core consultation components that can be flexibly applied to address issues at multiple levels of an organization.

In developing the ideal consultation approach, it is important to delineate the goals of the distance consultation provided by programs like EBTDC within the larger implementation support and delivery system, which includes local organizations and state-funded technical assistance centers (e.g., Wandersman et al. 2012). Nadeem et al. (2013), delineate several core functions of consultation, including skill-building, case application, accountability, engagement of stakeholders, problem-solving implementation barriers, and planning for sustainability. The primary goal of the supervisor consultation calls was to support clinic supervisors to use a CBT lens in their supervision with EBTDC consultants and to minimize the potential contradictory messages from EBTDC consultants and clinic supervisors. In reality, while calls focused on CBT-specific skill-building and case application much of the time, there was a clear need for discussions of programmatic issues and problem-solving implementation barriers. As similar policymaker-driven efforts are developed, it is critical that we are thoughtful about the purpose of consultation provided to different stakeholders. It remains an open question how to best achieve EBTDC’s goals of helping supervisors develop their own EBP-related clinical and supervisory skills, address implementation barriers, and provide support around project-related issues. Future studies can empirically test specific strategies, focus, structures, and intensity of consultation, as well as how consultation from state-led training centers may interplay with local organizations’ local resources, and their implementation climate and culture.

There are several limitations to the current study. Our analysis was limited in that we could not identify the individual clinic supervisors on the calls, precluding us from examining patterns within specific call groups or other relevant factors such as the attendance on the calls. Also, because coding occured live, we were not able to reexamine the calls for further information (thereby leaving such issues as whether the supervisors or consultations initiated the calls to future research). Our sample size was small as were the number of consultants; a larger sample or coding time frame would allow for greater understanding of variation across consultants and patterns of topics discussed over time. In addition, we were unable to relate our findings to supervisory survey data on satisfaction with the program, and it was beyond the scope of the project to examine features of the supervision provided at the clinics. However, given the minimal research to-date that has directly examined, coded, and analyzed the content of consultation calls targeted at supervisors in large scale implementation projects like EBTDC, this project provided an opportunity to pilot research methods that can be used in real-world, ongoing dissemination efforts by states and other large organizations. The study also provides insights into the consultation process itself for a unique portion of the clinical workforce. This helps to build our knowledge around the feasibility and effectiveness of low cost implementation and dissemination strategies (phone consultation in groups) that have the potential to be used widely across populations, states, and even countries.

Consultation strategies for middle-management supervisors in real-world community practice are necessary for improving uptake of new practices by clinicians. Analyses of the content of consultation provided via phone support to supervisory staff highlight the need to attend to the optimal distribution and emphasis of clinically relevant, relative to time spent on administrative topics, as well as and the variations in strategies to reinforce use of EBPs. Involvement of leadership and other relevant staff are important in effecting quality improvement in clinic practice (e.g., Beidas and Kendall 2010). Clinic supervisors have an important role in supporting clinicians’ learning of new skills and helping to overcome implementation challenges.