Introduction

Evidence-based programs (EBPs) focused on prevention are increasingly used in schools to promote students’ academic, social, emotional, and behavioral functioning. Although the efficacy of these EBPs has been established in the literature, issues with adoption and implementation persist (Domitrovich et al. 2008; Domitrovich and Greenberg 2000; Elias et al. 2003; Spoth et al. 2013). Several questions remain regarding the best way to optimize implementation of EBPs. Implementation supports, often in the form of coaching, have been identified as a promising approach for increasing the implementation fidelity of EBPs, and thus promote stronger program effects (Bradshaw et al. 2012; Kretlow and Bartholomew 2010; Pas et al. 2014). Coaching is, however, a complex and dynamic process, which includes coach engagement in multiple activities as well as a social process between the coach and implementer (i.e., working relationship) that may prompt teacher change in the behavior or skill targeted by coaching. A coach and teacher have a social relationship through which social persuasion can occur (Taylor 2007). Through this process, the coach empowers the teacher by indicating their own confidence in the teacher’s ability to use a strategy as well as the value of the performance for achieving the desired outcome. The coach also provides social and emotional support to the teacher during the teacher’s use of the new strategies (Taylor 2007). Little is currently known, however, about how these discrete activities or the interpersonal nature of the working relationship relate to implementation fidelity, which can be measured in a variety of ways. Some research has demonstrated the association between specific coaching activities and teacher changes in implementation (Coles et al. 2015; Reinke et al. 2007; Sanetti et al. 2014; Stormont and Reinke 2014).

Coaching includes many different activities such as needs assessment, modeling, technical assistance, and check-ins; coaches may vary their use of these activities in relation to a variety of implementer factors (e.g., teacher beliefs and perceptions regarding efficacy, burnout, and organizational factors; Pas et al. 2015). In turn, coaching activities may also influence implementation, these perceptions, and the teacher’s perceived working relationship with the coach. The teachers’ perception of their working relationship with a coach reflects collaboration, feelings of being supported by the coach, viewing the coaching process as competent, and overall satisfaction with the coaching (Johnson et al. 2016). Relatively few studies have documented the variety of coaching supports or the specific activities coaches engage in to optimize program implementation. Moreover, there has been limited consideration of the extent to which these activities relate to the coach–teacher working relationship (Johnson et al. 2016). The current paper examined how specific coaching activities and teacher’s report of the coach–teacher working relationship related directly to implementation dosage and quality of a widely-used preventive EBP called the PAX Good Behavior Game (GBG; Embry et al. 2003). The formation of a working relationship with the coach was also explored as a mediating pathway between coaching activities and implementation fidelity. Having a better understanding of how the specific coaching supports provided to implementers (i.e., teachers) promotes different elements of implementation fidelity of EBPs will inform researchers and practitioners on how to optimize fidelity.

Implementation of EBPs in School Settings

Schools present an ideal context for EBPs to address a range of student challenges due to easy access to students and availability of resources (Domitrovich et al. 2008; Kaftarian et al. 2004). In efficacy trials and highly controlled research studies, EBPs have reduced problematic behaviors and promoted adaptive development (Bradshaw et al. 2010; Durlak and DuPre 2008); however, once outside of these more controlled studies, implementation fidelity is often variable (Dusenbury et al. 2005; Gottfredson and Gottfredson 2002). In fact, less than half of EBPs are implemented with fidelity in school settings (Ringwalt et al. 2009; U.S. Department of Education 2012).

Implementation fidelity has been defined as “the degree to which teachers and other program providers implement programs as intended by the program developers” (Dusenbury et al. 2003, p. 240); it has been recognized to consist of five core elements, including: adherence (i.e., extent to which intervention was implemented as intended), exposure or dosage (i.e., amount or quantity of the intervention delivered), quality of delivery (i.e., how well the intervention was implemented), participant responsiveness (i.e., level of participant engagement with an intervention), and program differentiation (i.e., extent to which an intervention is different from others being implemented; for more information, see Carroll et al. 2007; Domitrovich and Greenberg 2000; Dusenbury et al. 2003, 2005). The inclusion of these elements in research studies varies, with dosage and adherence measured most frequently; less is known about quality (Domitrovich and Greenberg 2000; Dusenbury et al. 2003). Another framework of implementation fidelity proposes structural critical (e.g., procedural, dosage) and instructional critical (i.e., quality, responsiveness) components, viewing adherence as equivalent to the larger construct of fidelity (Century et al. 2010).

Although there are different elements of implementation fidelity, it is likely that they are not all equally important in their relevance to and need for coaching supports. The level of fidelity for each individual element may be variable (i.e., some more difficult to achieve than others) and each may be differentially responsive to support provided by a coach. Thus, many questions exist regarding which implementation elements to focus on and how to promote specific elements of implementation fidelity. Dosage and quality tap the structural and instructional components of implementation fidelity, respectively (Century et al. 2010), and can be directly targeted by coaching support. Previous research from this trial suggests that coaches tailored their time and activities to the specific intervention being implemented and to baseline levels of implementation quality (Becker et al. 2013a). Additional research demonstrated that teachers who received a greater number of contacts with a coach had higher levels of dosage (Pas et al. 2015).

Coaching as an Implementation Support

Coaching is one form of implementation support aimed at helping teachers to implement an EBP with fidelity (Stormont et al. 2015). Although coaching is increasingly used, it is a complex process and there is a lack of consensus on its definition and on which of the coaching activities are essential (Becker et al. 2013a; Pas et al. 2014). Furthermore, there is significant variability in how coaching is delivered as an implementation support. Little is known about various aspects of the coaching process (e.g., the role of different coaching activities), the optimal format or structure of coaching, the amount of coaching needed, and the interpersonal processes between the coach and teacher. Furthermore, the impact of these processes may vary as a function of teacher and contextual characteristics. For instance, coaches have been found to tailor coaching dosage and the way in which they do so is related to teachers’ beliefs and perceptions regarding personal resources and school organizational factors (Pas et al. 2015).

The activities that comprise coaching are varied and often depend on the coaching model used. Typically, coaching includes activities such as relationship building, assessment and identification of problems, feedback, and implementation support (Kratochwill et al. 1995; Reinke et al. 2011). More specifically, coaching activities can take the form of providing information about and training in an intervention, collecting data on classroom processes and teacher skills through observations and questionnaires, providing performance feedback regarding teacher skill or strategy use, engaging in goal setting/action planning, and modeling skills and intervention components (Denton and Hasbrouck 2009; Pas et al. 2014, 2016; Reinke et al. 2011). Although empirical research regarding the effectiveness of individual coaching activities is limited, research on professional development programs highlights the likelihood that the type of coaching activity used may contribute to different degrees of outcomes (Garet et al. 2016). Performance feedback has been identified as a valuable coaching tool to support teacher change (Reinke et al. 2007; Stormont and Reinke 2014). Case studies of coaching also lend preliminary support for the particular importance of some activities; implementation planning activities and discrete activities that target knowledge, skills, or attitudes are related to different types of outcomes (Coles et al. 2015; Sanetti et al. 2014). Further research that disaggregates the effects of discrete coaching activities on implementation fidelity can inform improvements in effectiveness and efficiency of coaching models.

Given the social nature of coaching and the expectation that this relationship, in part, may lead to teacher behavior change in the targeted skill (Joyce and Showers 1980), the working relationship or alliance between the coach and the teacher is likely another important part of the coaching process to consider. Teachers’ ratings of the working relationship reflect the quality of the relationship with the coach as well as perceptions of the coach’s skills and the usefulness of coaching (Johnson et al. 2016). Teacher beliefs and perceptions, such as level of burnout or views about the intervention, have been shown to relate to teachers’ implementation of EBPs (Cook et al. 2015; Domitrovich et al. 2015). Similarly, teachers’ perceptions of the coach may also impact implementation (Owens et al. 2017). In fact, working relationship has been associated with implementation adherence, suggesting that teachers may be more willing and able to implement an EBP when they perceive a positive coaching relationship (Wehby et al. 2012). A strong working relationship with a coach may reflect increased teacher investment and engagement as well as acceptability of the intervention (Johnson et al. 2016; Reinke et al. 2013), thereby promoting some elements of implementation, notably the more procedural and structural elements.

In summary, both the coaching activities and the teachers’ perceptions of the working relationship with the coach are important coaching processes that may each promote elements of implementation fidelity. It is also possible that there may be an interplay between coaching activities and working relationship. It may be that coaching activities that reflect relationship building, understanding the teacher, and coach credibility will contribute to a stronger working relationship (Frank and Kratochwill 2014; Johnson et al. 2016). Additional research is needed to better understand these complex associations.

Current Study

Prior research by our team suggests that coaches tailored their approach based on the intervention they were supporting and the level of teacher baseline implementation quality; greater overall contact with coaches was associated with higher implementation dosage (Becker et al. 2013a; Pas et al. 2015); however, the extent to which each discrete coaching activity is associated with subsequent implementation fidelity has not been previously examined. The coaching model employed utilized a collaborative approach consisting of several coaching activities to support teachers’ implementation of PAX GBG. The current study aimed to identify specific coaching activities that were associated with teacher report of the coaching working relationship; we were particularly interested in the extent to which these factors related directly to implementation dosage and quality of the PAX Good Behavior Game (GBG; Embry et al. 2003), as well as the mediating effect of working relationship on the pathway from coaching activities to implementation. Specifically, in the current study, we hypothesized that needs assessment, technical assistance, and modeling (which assess teachers’ needs, provide specific input on ways teachers can improve both their dosage and quality, and demonstrate needed skills to the teacher, respectively) would be directly related to both dosage and quality, whereas check-ins and working relationship, which both provide accountability but not necessarily specific skills, would only relate directly to dosage. Further, we hypothesized that needs assessment, modeling, and technical assistance would also directly relate to working relationship as well as indirectly to dosage through working relationship.

Method

Design Overview

The data for this study come from a randomized controlled trial testing the efficacy of 1 year of the PAX GBG, as implemented alone and integrated with the PATHS curriculum (Greenberg et al. 2011; Kusché et al. 2011). The trial included 27 elementary schools where nine schools each were randomized to one of three conditions: the integrated (PATHS/GBG) condition where teachers implemented PAX GBG with the PATHS program (Greenberg et al. 2011; Kusché et al. 2011), PAX GBG only, and a control condition where teachers conducted their usual practice. The study was conducted in a large urban, east coast public school district. Participating schools included a student population where the majority of students were African American (M = 89.06%) and received free and reduced meals (M = 88.09%). Recruitment occurred at the school level such that all principals agreed to participate in the year-long project and allow their teachers to receive training and coaching in the interventions; however, teacher participation in the training and data collection activities was voluntary. The Institutional Review Board at the researchers’ institution approved this study.

Participants

The current study included 12 out of the 18 intervention schools (i.e., all schools in the 2nd and 3rd cohorts), where coaching activity data were collected (i.e., control schools did not implement; detailed coaching activity data were not collected from the 1st cohort of intervention schools). The eligible teachers of 148 classrooms in grades K-5 across the 12 schools served as program implementers; data are included for all but 10 teachers who did not receive coaching either because they changed schools, went on leave during the study, or opted not to be coached.

The majority of the participating teachers were female and just under half of the teachers were 30 years of age or younger. The proportion of teachers who taught each grade was distributed relatively equally with 16% teaching kindergarten, 21% teaching 1st grade, 19% teaching 2nd grade, 15% teaching 3rd grade, 14% teaching 4th grade, and 15% teaching 5th grade. Roughly half of the teachers were assigned to each intervention condition. See Table 1 for a detailed description of teacher demographic characteristics.

Table 1 Descriptive information and statistics on teacher participants and study variables
Full size table

The research team employed three coaches to work with the teachers in these schools. All three coaches were former teachers; all were Caucasian, all had a master’s degree, two were female, and all had previous experience implementing the PAX GBG intervention. Coaches received intensive training from the intervention developers regarding the theory of the intervention, common challenges faced by teachers, and the coaching process for the specific intervention. Coaches received weekly supervision meetings with the research team, which included doctoral-level university faculty with expertise in behavioral interventions. Supervision focused on the review of coaching activities, discussion of individual teachers, development of plans to help teachers maintain implementation or reduce barriers to implementation, and review of implementation data. Although coaches were external providers to the schools, the coaches functioned as other support staff and traveled freely across classrooms and schools. They had school assignments (i.e., 2–3 schools per year) and regular access to teachers. They scheduled coaching sessions with the teachers based on mutual availability.

Interventions

Classroom Interventions

All teachers in the current study received training to implement the PAX Good Behavior Game (GBG) along with ongoing coaching support. The GBG uses a team-based, game-like context to reduce aggressive, disruptive, and off-task behavior and to foster self-regulation, thus facilitating academic instruction. The integrated condition involved implementing both PAX GBG and the PATHS curriculum, a universal social–emotional intervention (see Domitrovich et al. 2010 for description). All teachers received 1.5 days of training in PAX GBG, and teachers in the integrated condition also received an additional 2 days of training in the PATHS curriculum. After attending the initial training workshop, all participating teachers received face-to-face coaching for the entire school year.

Coaching Supports

The coaches’ primary role was direct coaching of teachers to promote implementation of PAX GBG; coaches also engaged in activities to foster administrative support and promote school-wide adoption of the interventions. A collaborative coaching approach was used that started with coaches building rapport with teachers and preparing teachers for implementation (i.e., setting up program materials), then cultivating skill development to promote implementation success (i.e., modeling, reflection, observation, data collection, performance feedback), and continued tailored coaching based on teacher needs. For a more detailed overview of the coaching model, see Becker et al. 2013b. Coaches followed consistent timelines and manualized guidelines regarding coaching activities (see Marchese et al. 2017). The frequency, intensity, and nature of the activities were intended to vary based on teachers’ level of skill and use of the GBG. In practice, the number of coach contacts with teachers also varied based on teacher receptivity to coaching and idiosyncratic factors, such as severe weather that disrupted the school calendar and teacher absence (e.g., leave time and illness).

Coaches were expected to meet with each teacher approximately once a week and engaged in a variety of activities, which were divided into six pre-defined categories and an “other” category on the logs coaches used to track their contacts with teachers (Becker et al. 2013a). The four most frequently-used categories were included in the current study (i.e., needs assessment, modeling, technical assistance/performance feedback, and check-ins). The remaining coaching activity categories had fewer than two contacts on average (i.e., implementation tracking, delivery), so they were excluded from the current study. Specifically, needs assessment included classroom observations by the coach of program elements, general teaching behaviors, and student behaviors, as well as walk-through observations by project staff and implementation observations by independent research staff, both accompanied by the coach. Program dosage data were also reviewed as part of the needs assessment activity and then used to guide further coaching activities. Modeling included the demonstration by the coach of how program elements should be implemented as well as modeling of general teaching or behavior management practices. Modeling was structured such that coaches prepared teachers to observe the modeling by describing the target skill, providing a modeling checklist, and discussing the teacher’s observations of the modeling session. Technical assistance included the coach providing specific information and feedback about the rationale, execution, or the teachers’ implementation regarding specific intervention elements and teacher and behavior management practices. Such sharing of information and feedback helped teachers and coaches identify problems and create an action plan for addressing the problem(s). Both modeling and technical assistance had subcategories to differentiate the focus of the activity (i.e., the classroom intervention or general teaching and behavior management practices) but only the collapsed categories were used. Lastly, check-ins included brief contacts by the coach with the primary goal of verifying that the teacher was implementing program components through actions such as collecting implementation tracking forms, asking about recent implementation, and scheduling observations and coaching visits. These check-ins also helped to encourage teachers to implement and helped coaches maintain regular contact, and thus a relationship, with the teacher.

Measures

Coaching Activities

The school year was broken into four, roughly quarterly, waves (i.e., fall, winter, early spring, late spring) and included approximately 31 weeks of coaching. Coaches completed an electronic log to record details about the services provided after each coaching contact made with a teacher throughout the school year, which was used to calculate the number of contacts for each coaching activity. Only contacts that were considered substantive (i.e., lasted at least 5 min) were recorded. The coach log reflected activities conducted one-on-one with a teacher and included check-ins, modeling, technical assistance, and needs assessment, which were the four coaching activities that the coaches in the trial most frequently engaged in (Becker et al. 2013a). Coach logs of specific activities were totaled for the time period following the baseline, wave 1, data collection (i.e., starting in October) through wave 2 data collection (i.e., December). These totals were included in the current study so that we could examine the coaching data in relation to subsequent data points regarding working relationship and implementation, and test for mediation (MacKinnon 2008).

Working Relationship

Teachers provided ratings of their perceptions about the coaching and the intervention at the second (i.e., winter), third (i.e., spring), and fourth (i.e., end of year) waves of data collection. Of specific interest in the current study were teachers’ perceptions of the working relationship with the coach during the second wave (i.e., winter), which was assessed with the Teacher–Coach Alliance Scale (Domitrovich et al. 2008; see; Johnson et al. 2016). This measure was an adaptation of the Wehby et al. (2012) measure and included 23 items reflecting several dimensions including perceptions on the relationship, coaching process, and overall satisfaction with the coaching (e.g., “I feel confident in my coach’s ability to help me implement PATHS to PAX”, “The time I spend working with my coach is effective and productive”) that are rated on a five-point scale (never to always). A total score was created by averaging all of the items (Cronbach’s alpha or α = 0.97).

Implementation Fidelity

Two elements of implementation fidelity were assessed using two different methods: dosage (i.e., how much of the intervention was delivered) and quality of intervention delivery (i.e., how well and how comprehensively the intervention was delivered). To assess dosage of the PAX GBG, teachers completed a weekly log of the number of PAX GBG games played and submitted it to their coach across the school year. The total number of games played was summed across the school year and used as the dosage outcome measure. Second, to assess the quality of the PAX GBG implementation, rubric ratings of teachers’ PAX GBG game quality were completed by research staff, which was comprised of coaches and research staff who were randomly assigned to complete observations. These observation ratings were done after each of four waves throughout the school year; only the fourth wave, and final, rubric ratings from the end of the school year were used in the current study. During this observation, teachers were asked to conduct a 5- to 10-min game so that the observer could determine whether elements were properly executed and how well. The Game Observation Scale of the PAX GBG rubric (Schaffer et al. 2006) included seven items assessing the quality of teacher preparation for and execution of the game (α = 0.93). This included: (1) preparing students, (2) the activity during which the game is conducted, (3) timer usage, (4) team structure, (5) teacher response to behavior, (6) game review at the end, and (7) the prize given. Ratings were made on a scale of 0–4, with higher scores indicating better quality implementation. Inter-rater reliability was assessed using pairs of staff members for the first 15% of teachers at each data collection wave. The intraclass correlation coefficient was 0.90 and 0.94 for cohorts 2 and 3, respectively, both representing excellent inter-rater reliability. After establishing reliability at or above intra-class correlations of 0.80 for each item, the remaining observations of quality were collected independently.

Teacher Demographics

Participating teachers responded to a series of questions regarding their basic demographic characteristics, including gender, grade taught, and age.

Overview of Analyses

After reviewing basic descriptive analyses, we conducted mediation modeling in Mplus 7.3 (Muthén and Muthén 1998–2014) to test our primary hypotheses. Specifically, coaching activity and working relationship were modeled to have a direct effect on both implementation dosage and quality, using a structural equation modeling approach. The indirect effect of each coaching activity on implementation through the working relationship was estimated as the product of the path from coaching activity to working relationship and the path from working relationship to each implementation fidelity outcome (see MacKinnon 2008). See Fig. 1 for a depiction of the model. Both outcomes were modeled as continuous (i.e., z-score of both games played and of the rubric). Each of the four coaching activities of interest (i.e., needs assessment, modeling, technical assistance, and check-ins) were modeled separately (i.e., in four models).

Fig. 1
figure 1

Conceptual path diagram of the direct and indirect effects on implementation. The a pathway represents the direct effect of the coaching activity on working relationship. The b 1 and b 2 pathways represent the direct effect of working relationship on games played/dosage and quality rubric rating, respectively. The c 1 and c 2 pathways represent the direct effect of coaching activity on games played/dosage and quality rubric rating, respectively. The c’ 1 and c’ 2 pathways represent the indirect effect of coaching activity on games played/dosage and quality rubric rating via working relationship, respectively. There were four statistical models run; one for each coaching activity (i.e., needs assessment, modeling, technical assistance, and check-ins)

Full size image

All analyses accounted for the clustering of teachers within schools by applying the complex approach, which estimates the model for the whole sample, correcting for possible inflation of estimated standard errors caused by the clustering of teachers within schools. The models included teacher demographic variables (i.e., gender, grade taught, age, and graduate degree) and intervention condition as covariates influencing both implementation fidelity outcomes. Gender was coded as male = 0 and female = 1. Grade, age, graduate degree status, and intervention condition were dummy-coded with grades 3–5, 30 years old or younger, attained a graduate degree, and PAX GBG only serving as the reference groups. Unstandardized coefficients are reported. Model fit indices included the root mean square error of approximation (RMSEA), Comparative Fit Index (CFI), and Tucker–Lewis Index (TLI). For RMSEA, a value of < 0.06 indicate a good fit (Hu and Bentler 1999). For the CFI and TLI, a value of 0.90 or higher is considered acceptable fit (Bentler and Bonett 1980).

Missing Data

A Little’s test of missing completely at random (MCAR) was conducted in SPSS, including all study participants and variables of interest; the non-significant result of this test indicated that it was likely that data are MCAR. Missing data for this study were imputed to ensure a complete dataset on teachers’ self-reported perceptual and implementation data, as the missingness for all data collected in the broader study (i.e., all variables of interest) ranged from 0 to 22%. The multivariate imputation by chained equation (MICE) method of multiple imputation was conducted in STATA (Azur et al. 2011; White et al. 2011). MICE imputes each variable conditional on all of the other variables in the imputation procedure and iterates that process until convergence. Additionally, three interaction terms with teacher-related variables were included to account for condition (grade taught, age, and graduate degree; for which we had complete data). School-level predictors such as enrollment, free and reduced meals, and mobility were also included to inform the imputation, as data on these variables were complete.

Results

Descriptive Analyses

Descriptive data on the study variables are presented in Table 1. Totaled coach log data from the fall through winter (i.e., waves 1–2) demonstrated that coaches on average engaged in 1.76 needs assessment contacts (range 0–7), 2.30 modeling contacts (range 0–11), 6.00 technical assistance contacts (range 0–16), and 8.18 check-in contacts (range 1–17) with teachers. Teachers rated the working relationship at the second wave as 4.48 (SD = 0.55), which corresponded to a rating on positively-worded items between “often” and “always” (i.e., the most positive possible response). On average, teachers implemented 172.33 games across the school year and received a rating of 3.42 out of 4 on the quality rubric at the final wave of data collection (i.e., at the end of the school year).

Mediation Models

Table 2 includes the full results of the mediation models for each coaching activity. The effects presented here control for condition and demographic variables on the outcomes. The average model fit indices for the imputed data files varied for the four coaching activities: needs assessment (RMSEA: 0.023, CFI = 0.986, TLI = 1.004), modeling (RMSEA = 0.039, CFI = 0.968, TLI = 0.889), technical assistance (RMSEA = 0.040, CFI = 0.933, TLI = 0.777), and check-ins (RMSEA = 0.047, CFI = 0.924, TLI = 0.698). Figure 1 illustrates the paths for direct and indirect effects.

Table 2 Mediation model path coefficients for implementation outcomes
Full size table

Predicting Working Relationship

In the overall model, the direct effects of coaching activities on working relationship were first examined. Needs assessment had a significant direct effect on working relationship (coeff. = 0.11, p < 0.01; i.e., path a in Fig. 1), such that engaging in a greater number of needs assessment contacts was related to more positive teacher perceptions of the working relationship. The other three coaching activities did not have a direct effect on teacher-reported working relationship.

Direct Effects on Implementation Fidelity

The direct effects of coaching activities on implementation fidelity were examined next. Results indicated that modeling had a direct effect on number of games played (coeff. = 0.17, p = 0.01; i.e., path c1 in Fig. 1). Teachers who were exposed to more modeling by their coach completed more games in their classroom. None of the other coaching activities were related to games played, although needs assessment had a trending effect (p = 0.08). There were no direct effects on implementation quality. Results from all four coaching activity models indicated a direct effect of working relationship on the number of games played (coeff.’s range 0.34–0.39, p’s < 0.01; i.e., path b1 in Fig. 1), but not on quality. More positive perceptions of the working relationship were related to a higher number of games played.

Indirect Effects on Implementation Fidelity

One significant indirect effect emerged such that needs assessment was related to a more positive working relationship and, in turn, to higher dosage (indirect effect coeff. = 0.04, p = 0.05; i.e., path c’1 in Fig. 1).

Discussion

Although prior research has assessed implementation fidelity of EBPs in school settings, the focus has typically been on just one element of implementation fidelity. Moreover, evolving research suggests that implementation supports like coaching improve implementation fidelity, yet there has only been a preliminary examination of the complex processes that comprise coaching (Noell and Gansle 2014; Pas et al. 2014). In order to better promote the translation of EBPs into real-world practice in school settings, an explicit study of effective implementation support systems is needed, as this may provide guidance to coaches regarding the specific activities to engage in for an improvement in fidelity. Coaching teachers as an implementation support has shown some effectiveness at promoting implementation (Coles et al. 2015; Pas et al. 2014; Reinke et al. 2007; Sanetti et al. 2014; Stormont et al. 2015); however, coaching supports reflect a complex process including the types of activities that the coaches engage in as well as the working relationship between the coach and teacher. The purpose of this study was to begin to address this gap by examining how specific coaching activities and teacher report of the coaching working relationship relate directly and indirectly to implementation dosage and quality of the PAX GBG.

Our results indicated that both coaching activities and the working relationship were associated with implementation dosage. More specifically, of the four coaching activities examined in the current study, two specific activities, needs assessment and modeling, emerged as important to working relationship and implementation dosage. These two activities also had better fitting models. The number of needs assessment contacts (i.e., when the coach conducted observations of teacher implementation and collected data from the teacher on the number of games played in the classroom) was marginally related directly to implementation dosage and was indirectly related to dosage by a positive association with working relationship. It is possible that visits during which needs assessment was conducted improved both working relationship and dosage. For dosage, it may have helped to hold teachers accountable, since coaches specifically reviewed program dosage documentation and progress during this activity. Needs assessment also may have allowed the coach to develop a more thorough understanding of a classroom and this activity may enhance a coach’s credibility and effectiveness in coaching. This, in turn, may have promoted a stronger working relationship and the teachers’ desire to implement a program with more frequency. This possibility is evidenced by the significant indirect relationship between needs assessment and dosage via working relationship, such that the extent to which needs assessment was related to a strong working relationship, teachers also implemented with a higher dosage. Other research similarly demonstrates that teachers show more growth when they view consultants as positive experts and providing direct information (Owens et al. 2017). Needs assessment activities may foster this view.

Modeling was another coaching activity that was directly related to the number of games played by a teacher; however, modeling was different from needs assessment in that it did not also relate to working relationship, nor was there an indirect effect on implementation. Modeling involved the coach demonstrating the core activities of the intervention as a tool for a teacher to observe, completing a checklist, and discussing the modeling and its effects on the students. Modeling was previously found to be used more frequently with low implementing teachers and was believed to be used to help demonstrate the effectiveness of the PAX GBG (Becker et al. 2013b), likely in an effort to increase buy-in and implementation. Thus, modeling may have been used more selectively than other coaching activities and therefore related only directly to dosage as compared to needs assessment.

Neither technical assistance nor check-ins were related to implementation or to working relationship. These null findings could be the result of these categories being comprised of multiple discrete activities, whereas modeling and needs assessments were defined more narrowly. On the other hand, prior research demonstrated that modeling, but not discussion, activities predicted implementation (Bearman et al. 2013). It is also possible that the four activities overlapped with one another in their underlying mechanism for promoting teacher change. For example, check-ins, like needs assessment, may have promoted greater teacher accountability. Needs assessment, with its greater structure for what was accomplished as compared to check-ins, just may have been more potent in its ability to change teacher behavior. Another important consideration is the amount of time spent in each activity; although needs assessment and modeling occurred less frequently than check-ins and technical assistance, these contacts were usually longer in duration (Becker et al. 2013a). Perhaps the length of the sessions is the mechanism for change. Additional research on specific coaching activities and with a more nuanced consideration for contact characteristics is warranted.

More positive teacher perceptions of the working relationship with the coach were also associated with higher dosage. The working relationship reflects several aspects including the teachers’ feeling of being supported and understood by the coach, perceiving the coach as useful, and working well with the coach. Thus, teachers who perceived a more positive relationship with the coach implemented the intervention with greater frequency. It may be that working relationship promotes teachers’ willingness to engage in an intervention. This pattern is similar to the therapeutic alliance literature demonstrating that a positive alliance is significantly related to the client’s behavior change (Ackerman and Hilsenroth 2003; Horvath and Greenberg 1989).

Interestingly, both coaching activities and working relationship were only significantly related to dosage and not to implementation quality, implying a differential relationship for these two important implementation fidelity outcomes. Understanding how different elements of implementation fidelity respond to implementation supports is necessary. Dosage is the amount of a program that is delivered and is a concrete count of the number of games played. It is easy to interpret and to address (i.e., focus on an increased frequency of game playing). Alternatively, quality of implementation fidelity is more nuanced, reflecting the manner of delivery and thus may be more ambiguous to action plan around and to improve. Furthermore, quality likely requires more support and time to achieve than dosage. The broader consultation and implementation empirical research in settings beyond schools (e.g., mental health clinicians, physicians) is similarly beginning to examine different consultation activities and functions and their role in promoting elements of implementation fidelity (Nadeem et al. 2013). In fact, consultation serves myriad functions such as training, engagement, accountability, skill building, and problem solving, and implementation research is unpacking the effects of these functions on implementation fidelity outcomes (Nadeem et al. 2013).

Limitations and Future Directions

It is important to consider some limitations when interpreting the results of the current study. Attuning to the sensitive measurement of quality and other implementation fidelity elements, as well as working relationship, is important and an area for further research. The greater variability in the dosage element (i.e., number of games played across the school year) than for the 5-point Likert scale observer rating of quality may, in part, explain the significant versus null findings for these two elements. The restriction in range for the quality element of implementation fidelity is one that needs to be addressed in future research. Similarly, the limited variability in working relationship may have masked additional relations between coaching activities and implementation. Related to coaching activities, additional research is needed to better quantify coaching activities (i.e., number of contacts, length of contacts, patterns over the working relationship) and to characterize and/or define the types of coaching activities. The current study examined coaching activity categories and it is likely that these categories could be further parsed out. Additional activities may also need to be considered. For instance, technical assistance reflects several distinct activities such as performance feedback and collaborative problem solving. Future research could also investigate the differential strength of coaching activities with regard to their impact on fidelity. This would help determine which activities should be used more frequently by coaches. Furthermore, this study was conducted specifically with the PAX GBG intervention and a manualized coaching model. It is important to extend this research to other interventions and coaching models to investigate whether these patterns generalize. Further, we only examined the coaching log data during the earliest parts of the coaching implementation (i.e., October through December), so that we could ensure temporal order in relation to the mediator and outcome variables. It is possible that different associations emerge over the course of the year-long coaching cycle. Lastly, the sample size of 138 teachers and three coaches is relatively small and may reduce the statistical power to detect significant effects. A larger number of coaches would also allow for the examination of any potential impacts of coach characteristics on different aspects of the coaching process (i.e., coaching activities, working relationship). Research in the field of coaching to promote implementation would benefit from more empirical research generally, as well as the inclusion of a larger sample of both teachers and coaches. This would allow for exploring potential moderating effects of personal characteristics, perceptions, and skills.

Conclusions and Implications

Schools serve an important role in identifying and often providing interventions to prevent and address public mental health. Understanding how coaching processes may improve the implementation fidelity of preventive EBPs targeting behavior in school settings has implications for the translation and effectiveness of such programs in these settings. The current findings highlight the need for research to include an examination of implementation supports as well as multiple implementation fidelity elements when studying EBPs. Some of the specific coaching activities examined as part of the implementation supports in the current study promoted implementation dosage, but not quality. For example, teachers who were exposed to more needs assessment and modeling implemented the intervention with greater frequency. Further, the working relationship within the coaching process was also a relevant factor for implementation and is an area for further research.

The current findings have implications both for school administrators and support staff as well as for coaches. School administrators have been identified as facilitators of successful EBP implementation (Langley et al. 2010; Saldana et al. 2016); facilitation could come in the form of supporting the coaching process and the implementers of the EBPs as well as engaging in coaching and supervision activities. Prior research has demonstrated that perceived administrator support for the implementation of an EBP in schools is associated with improved implementation (Aarons et al. 2014; Langley et al. 2010). The importance of particular coaching activities (i.e., needs assessment and modeling) and of the working relationship between the coach and teacher provide important feedback to coaches. Coaches may consider selecting the use of specific coaching activities to improve the working relationship and optimize their impact on implementation fidelity. Similarly, training and supervision of coaches should attune to the distinct purposes of coaching activities and their interplay with the working relationship and the desired implementation fidelity and more distal (i.e., teacher and student) outcomes.