Abstract
Previous research has demonstrated the importance of a variety of factors on the implementation of school-based prevention programs, specifically characteristics of program providers, program structure, school climate, and school and community structure. The current study expands this research by examining the potential relationships between all of these factors and implementation quality in a series of multilevel models. Using data from a nationally representative sample of 3,730 program providers surveyed in 544 schools, it was found that program structure characteristics were of greater importance in the prediction of high quality implementation than were characteristics of the program providers, school climate, and school and community structure. Implications of these findings are discussed.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
The focus of school-based prevention research in recent decades has been directed at identifying and replicating evidence-based programs that promote the general welfare of the student population. While many studies ultimately conclude that problem behavior, delinquency, and drug use can be reduced by school-based interventions (Botvin 1990; Botvin et al. 1995a, b; Catalano et al. 1998; Ennett et al. 1994; Gottfredson 2001; Gottfredson et al. 2002; Hawkins et al. 1995; Lipsey 1992; Lipsey and Derzon 1998; Tobler 1992; Wilson et al. 2003), considerable research has also documented the difficulties of achieving high quality implementation of these effective programs (Gottfredson et al. 2000; Gottfredson and Gottfredson 2002; Mihalic et al. 2008; Payne et al. 2006; Payne 2009; Rohrbach et al. 1993, 2006). Implementation quality, or how well an innovation is carried out in practice, has been an increasingly important topic to researchers since the 1970s (Fullan and Pomfret 1977). Real world application of interventions proven successful in research settings has been met with numerous obstacles, leading to varying degrees of implementation quality among these programs. Armed with the knowledge that poor implementation leads to lower program effectiveness (Battistich et al. 1996; Botvin et al. 1989a, b, 1990, 1995; Gottfredson et al. 1993, 1996; Nunnery et al. 1997; Silivia and Thorne 1997; Smith et al. 1997), scientists have been stressing the importance of evidence based implementation practices within the prevention community (Botvin 2003; National Institute on Drug Abuse 2003; Zins et al. 2000).
Although program implementation is an incredibly complex issue (Rohrbach et al. 2006), research has begun to identify factors that influence implementation. Despite the empirical literature documenting a number of influential factors at both the individual and school level, no known study has examined these factors in a multilevel model. This means, therefore, that predictors’ effects on implementation quality have yet to be fully and accurately estimated. In this study, four categories of such predictors are explored in multilevel models, with each predictor measured at the appropriate level: Program provider and program structure characteristics are measured at the program level while school climate and school and community structural characteristics are measured at the school level.
Program Level Predictors of Implementation Quality
Program Provider Characteristics
At the foundation of school-based interventions are the personnel who administer such programs. It is logical, therefore, to think that implementation is highly dependent upon program providers and that characteristics of these providers may influence implementation in a number of ways. First, implementation quality is said to increase when providers are motivated to implement the program (Rohrbach et al. 2006), have a positive attitude toward the program (Rohrbach et al. 2006), are comfortable with the content and delivery method (Brink et al. 1995; Rohrbach et al. 1996), and have a strong self-efficacy for implementation (Gingiss et al. 2006; Hunter et al. 2001; McCormick et al. 1995; Parcel et al. 1995; Rohrbach et al. 1993). In addition, program providers displaying a nonauthoritarian style, strong group leadership skills, good overall teaching skills, and a sense of conscientiousness may achieve higher quality implementation (Gingiss et al. 2006; Tobler 2000; Young et al. 1990). Finally, the background of the provider, such as their experience in implementation and leadership, may also play a role (Glasgow et al. 2003; Rohrbach et al. 2006). Taken together, higher quality implementation intuitively follows these characteristics, as an innovation would most aptly be carried out by someone who is positive, knowledgeable, experienced, and feels as if he or she could make a difference.
Program Structure Characteristics
The structure of the intervention itself, such as program training and planning, has also been found to influence its implementation. For instance, programs chosen as a result of a local planning process are likely to be better implemented (Payne et al. 2006; Payne 2009). This provides for participation in decision making and includes having school insiders initiate program selection rather than having a program forced on the school by district personnel or other outsiders, and having these insiders examine many information sources before choosing a specific program. These steps make it more likely that school insiders feel a personal connection to the program and a sense of “buy-in” (Bernd 1992; Boyd 1992; Gottfredson and Gottfredson 2002; Stoll and Fink 1996). Related to this, programs that are integrated into normal school operations display higher implementation quality (Bosworth et al. 1999; Gottfredson and Gottfredson 2002; Jaycox et al. 2006; Mihalic et al. 2008; Payne et al. 2006; Payne 2009), by affecting how enthusiastic and widespread the adoption of the program is in the school.
In addition, programs that include a good amount of high quality training, such as the use of behavioral modeling delivery methods and follow-up training, coaching, or ongoing program consultation (Goldstein and Sorcher 1973; Joyce and Showers 2002), are more likely to be effectively implemented. These elements reduce the amount of content deviation by program personnel, thus ensuring greater fidelity to program content (Ennett et al. 2003; Gottfredson and Gottfredson 2002; Payne et al. 2006; Payne 2009). Supervision of implementation is also an important component of program structure, such that it provides direction, coaching, and corrective feedback for provider behavior and can encourage striving for superior performance when it is linked with social or other rewards (Gottfredson and Gottfredson 2002).
Finally, one of the most important program characteristics is standardization. Materials such as manuals and handouts can provide structure and make deviation from intended content less likely (Gottfredson and Gottfredson 2002). Research confirms that standardized programs experience higher quality implementation (Gottfredson and Gottfredson 2002; Mihalic et al. 2008; Payne et al. 2006; Payne 2009; Rohrbach et al. 1993, 2006), thus demonstrating the link between program fidelity and overall effectiveness. It is also likely that many of these characteristics are related to each other. For instance, schools which engage in a local planning process are more likely to use standardized programs (Payne et al. 2006).
School Level Predictors of Implementation Quality
While programs and providers may exhibit characteristics conducive to high quality implementation, the absence of certain elements in the overall school environment has been shown to lower program quality. When factors such as principal support and organizational capacity are lacking at the school level, overall program effectiveness suffers (Kam et al. 2003).
School Climate
Aspects of the school climate, such as organizational capacity and principal support, also influence program implementation. Organizational capacity, or the ability of a school to implement a given program, is indicated by factors such as staff morale and positive communication between administrators and staff (Gottfredson and Gottfredson 2002). Not surprisingly, those suffering from low levels of these indicators have difficulty implementing programs, garnering community and administrative support, and projecting a competent image to the community (Berman and McLaughlin 1978; Ennett et al. 2003; Gottfredson and Gottfredson 2002; Gottfredson et al. 1996). Research has established that schools with greater organizational capacity are able to implement programs at a higher quality (Bosworth et al. 1999; Elliot and Mihalic 2004; Ennett et al. 2003; Fullan 1992; Gottfredson and Gottfredson 2002; Jaycox et al. 2006; McLaughlin 1990; Mihalic et al. 2008; Olds 2002; Payne et al. 2006; Payne 2009; Rohrbach et al. 2006; Schoenwald and Hoagwood 2001; Sobeck et al. 2006; Stoll and Fink 1996). This is not surprising: Given an increase in good administrative communication and staff morale, one would expect more positive results stemming from a positive environment.
Due to the administrative power and gatekeeper function of the position, the principal provides local legitimacy to the innovation (Berman and McLaughlin 1978) and can significantly impact implementation (Botvin et al. 1990, 1995a, b; Rohrbach et al. 1993). Of utmost importance is the presence of principal support that aids school personnel in creating an environment rich for program success (Berman and McLaughlin 1978; Gottfredson and Gottfredson 2002). Supportive principals have a general positive feeling toward the program and help the program run smoothly in the school, rather than acting indifferently or even creating obstacles to implementation. Schools with stronger principal support for interventions are far more likely to implement these programs with higher quality (Berends et al. 2002; Farrell et al. 2001; Fullan 1991, 1992; Gager and Elias 1997; Kam et al. 2003; Kegler et al. 1998; Payne et al. 2006; Payne 2009; Petersilia 1990; Rohrbach et al. 1993; Smith et al. 1993).
School and Community Structural Factors
Although less research has focused on structural factors, certain characteristics of the school and the surrounding community have also been found to influence implementation quality. Stability in terms of resources and personnel is positively associated with the quality of implementation (Fullan 1992; Gottfredson and Gottfredson 2002; Payne et al. 2006; Payne 2009), as is school size and urbanicity, such that larger urban schools demonstrate greater levels of program use (Payne et al. 2006; Payne 2009). Other structural characteristics, such as grade level and community poverty, have an indirect effect through certain school climate factors, such as organizational capacity and principal support (Payne et al. 2006; Payne 2009). Even district and state policies have been shown to affect programs, serving to both insulate and isolate school reform in various ways (Boerm et al. 2007; Bosworth et al. 1999; Walker 2004).
The Current Study
While previous research has laid a firm foundation on which to stand, it does suffer from some limitations. Numerous researchers have documented the impact of program provider factors on implementation quality but fail to take school factors into account (Hunter et al. 2001; McCormick et al. 1995; Parcel et al. 1995; Tobler 2000; Young et al. 1990). Conversely, many studies have examined school characteristics without including characteristics of the provider (Bosworth et al. 1999; Botvin et al. 1990, 1995a, b; Gottfredson and Gottfredson 2002; Jaycox et al. 2006; Kam et al. 2003; Kegler et al. 1998; Mihalic et al. 2008; Payne et al. 2006; Payne 2009; Petersilia 1990; Smith et al. 1993). Finally, those studies that do examine several factors from each of the above categories do so without multilevel modeling techniques (Berman and McLaughlin 1978; Brink et al. 1995; Gingiss et al. 2006; McCormick et al. 1995; Rohrbach et al. 1993, 2006), thereby not estimating the predictors’ effects on implementation quality as accurately as possible (Raudenbush et al. 2004). In the current study, we seek to improve this situation by examining multilevel models that include factors from all four of the categories discussed above; thus, the impact of program provider characteristics, program structure characteristics, school climate factors, and school and community structure factors on implementation quality will be studied with each predictor measured at the appropriate level.
Methods
The National Study of Delinquency Prevention in Schools
The National Study of Delinquency Prevention in Schools (Gottfredson et al. 2000) examined factors related to the successful implementation of school-based prevention programs, practices, and policies that sought to improve school safety and/or prevent or manage a variety of problem behaviors, including criminal activity, substance use, dropout, truancy, tardiness, classroom or school misbehavior, and risky sexual behavior. As a preliminary step in the original study, examples of prevention and intervention models used in schools were collected and classified to develop a comprehensive list of prevention activities and strategies.Footnote 1 The original researchers started with a probability sample of 1,287 public, private, and Catholic schools, and attempted to survey principals, teachers, students, and program providers in each school. Of the 1,287 schools included in the original study, 66% (N = 848) of the principals responded to the Principal Questionnaire Phase One survey. Of the 848 schools who responded in Phase One, 75% (N = 635) returned the Principal Questionnaire Phase Two and 65% (N = 555) returned at least one Activity Coordinator Questionnaire. Within these schools, the average program provider response rate was 83% (N = 3,730 providers). In addition, 310 schools (37%) participated in the Student Questionnaire and 403 schools (48%) participated in the Teacher Questionnaire; within these schools, the average student and teacher response rates were both 75% (N = 17,260 students and 14,183 teachers).Footnote 2
When correlations between school characteristics and survey participation were run, it was found that schools located in small towns or rural areas were significantly more likely to have participated, while those located in communities with more female-headed households with children, a greater proportion of urban population, and more households that received public assistance were less likely. In addition, high schools participated less often than other schools. Implications of this non-random attrition will be addressed in the Discussion section.
The Current Study Design
Drawing from the list developed by the original researchers, we focus on categories of prevention practices that have a clearly defined strategy towards addressing deviance. These include prevention curriculum or instruction; behavioral programming or behavior modification; counseling, psychological, or therapeutic activity; mentoring, tutoring, or job apprenticeship; improvements to instructional practices or methods; improvements to classroom organization and management practices; activities to change or maintain culture, climate, or norms; intergroup relations and school–community interaction; interventions involving a school planning structure or process to manage change; and youth participation in school discipline.
In this study, we use data from four surveys: Principal Questionnaire Phase One, administered in 1997; and Principal Questionnaire Phase Two (which asks different questions from Phase One), Activity Coordinator Questionnaire, and Teacher Questionnaire, administered in 1998. The Activity Coordinator Questionnaire was completed by the program providers (the personnel responsible for implementing and maintaining interventions) and differed depending on the type of program for which the provider was surveyed. More detail on these surveys can be seen in Gottfredson et al. (2000). Of the 848 schools who responded in Phase One, the 293 schools that did not return the Activity Coordinator Questionnaire were excluded from our sample. In addition, nine schools were excluded because the Activity Coordinator Questionnaire indicated they no longer had the sampled program(s) and two schools were excluded because they did not have data on the indicators of implementation quality. Therefore, the final sample for the current study is 544 schools and the 3,730 program providers surveyed in those schools.Footnote 3
Measures Footnote 4
Implementation Quality
Several indicators of implementation quality have been discussed in the prevention literature. Dane and Schneider (1998) identified exposure or dose, adherence to the program as described, delivery, participant responsiveness, and program differentiation as important implementation quality dimensions. Because exposure or intensity can be assessed in a more straightforward fashion than the other dimensions mentioned above, this study focuses on measures of implementation intensity as dependent variables. Indicators of implementation intensity include Level of Use, Frequency of Operation, Duration, Number of Lessons or Sessions, and Frequency of Student Participation. These indicators are measured at the program level and are taken from the Activity Coordinator Questionnaire, which differed depending on the type of program for which the coordinator was surveyed. As often as possible, the same survey items were asked about each program type; however, many questions that work for a certain activity are not appropriate for another activity (Gottfredson et al. 2000). Consequently, there were different measures of implementation intensity within these questionnaires, depending on what was most appropriate for that particular type of prevention activity; thus, not all program types have data on all five implementation intensity indicators. Each program type, however, did have data on at least three of the indicators, and regression imputation was used to provide values for the missing measures of implementation intensity.Footnote 5 Following the procedure used in Gottfredson and Gottfredson (2002) and Payne et al. (2006), three measures available for most program types examined in this article (duration, number of lessons or sessions, and frequency of student participation) were combined into a composite Intensity scale by averaging the z-scores of the three measures. More detailed information and descriptive statistics for these indicators can been seen in Table 1 and correlations among the indicators can be seen in the Appendix.
Provider, Program, School, and Community Characteristics
Measures from the Activity Coordinator, Principal, and Teacher Questionnaires were used to assess the impact of program provider, program structure, school climate, and school and community structural characteristics on implementation quality. Program provider and program structure factors were measured at the program level, while school climate and school and community structural factors were measured at the school level. Descriptive statistics for all predictors can be seen in Table 2 and correlations between predictors within each level of measurement (i.e., provider or school) can be seen in the Appendix.
Two indicators from the Activity Coordinator Questionnaire were used to measure program provider characteristics. Background is a 12-item scale asking the provider to describe their previous experience, such as whether they have ever conducted a formal training workshop for other educators or developed an instructional method or plan adopted by other educators; responses were “yes” or “no” and a program provider’s score is the proportion of items endorsed. Conscientiousness is a scale of provider responses to a list of 20 adjectives used to describe themselves and their leadership style, such as careful, negligent, and organized; possible responses ranged from “yes, I am very much like this” to “no, I am not at all like this” and a program provider’s score is the mean of the responses to each item.Footnote 6
Seven indicators from the Activity Coordinator Questionnaire were used to measure program structure.Footnote 7 Standardization is a five-item scale asking about the availability of prepared materials such as an instructor’s manual or a written description of activities; a program’s score is the mean of z-scores for all items. Integration is a factor score of two items asking how large a part of the program provider’s job was his/her work related to the program and if the program was a regular required activity in the school. Local Program Process is a factor score of three items asking the number of different information sources used in program selection, whether the program was developed by people outside of the school, and how much responsibility school insiders had in program initiation. Training is a factor score of two scales asking about the amount and quality of the training received by the program provider, such as whether on-going coaching or support was provided and whether participants practiced applying program principles during training. Supervision is a three-item scale gauging the level of monitoring over program activity, asking whether a supervisor directly observed the program, whether the provider was required to keep records, and whether the provider’s personnel appraisal depended on performance in this program; a program’s score is the mean of the three items. Finally, Full-Time is a single item asking whether the program provider is considered a full-time employer and Extended Day is a single item asking whether the program occurs outside regular school hours; possible responses for these two questions were “yes” and “no.”
Two school-level indicators were used to measure school climate factors. Organizational Capacity is a factor score of four scales from the Principal Questionnaire (school capacity for program development, teacher–principal communication, amenability to implementation, and obstacles to implementation)7; a school’s score for each scale is the mean of t-scores for all items. Principal Support is a single item from the Activity Coordinator Questionnaire with responses ranging from “the principal works hard to make sure the program runs smoothly” to “the principal often creates problems for the program.”
Finally, four indicators were used to measure school and community structural factors.7 Community Poverty is a factor score based on measures from the 1990 Census.Footnote 8 Size and Urbanicity is a factor score of three measures: Student enrollment (natural log), number of students taught (calculated from a Teacher Questionnaire item asking how many different students are taught within an average week), and Urbanicity (a factor score based on measures from the 1990 CensusFootnote 9). Grade Level (elementary versus middle versus high school) and Teacher Turnover are single items based on data from the Principal Questionnaire.
Analytical Strategy
We first examined the distributional characteristics of the study measures. Two school level variables, Community Poverty and Urbanicity, were trimmed by capping three outlier schools’ values to three standard deviations above the mean. In addition, we used regression imputation to supply a small amount of missing data for some of the level two predictors; census variables not included in the model were used to predict scores for the missing variables.Footnote 10
We then used Hierarchical Linear Modeling (HLM 6.0; Raudenbush et al. 2004) to examine relationships among program provider and structure characteristics, school climate and structural factors, and implementation quality. Since schools are naturally nested environments, HLM offers the clear advantage of allowing us to examine the ability of school-level variables to explain variation in program-level outcomes, while still examining the effect of program-level predictors. Thus, the effects school-level climate and structural factors have on program-level implementation quality can be seen along with the effects of program-level provider and structure characteristics. This allows for greater accuracy in the estimation of program-level implementation quality (Raudenbush et al. 2004).
For this study, we estimated two-level HLM models; level one is the individual level (i.e., programs) and level two is the group level (i.e., schools).Footnote 11 Three one-way ANOVA models were estimated first, one for each of the implementation indicators (Level of Use, Frequency of Operation, and Intensity), showing whether the school-level variance significantly differs from zero and the percentage of the outcome variance that lies between schools. Three sets of models were then estimated, one set for each of the implementation indicators (Level of Use, Frequency of Operation, and Intensity) as the program-level outcome variable. For these models, only level one predictors were included. The first model contained only the program provider factors (Conscientiousness and Background) while the second model contained only the program structure factors (Standardization, Integration, Local Program Process, Training, Supervision, Full-Time, and Extended Day); the third model contained all nine level one predictors. Next, three new sets of models were estimated, again one set for each of the implementation indicators (Level of Use, Frequency of Operation, and Intensity) as the program-level outcome variable. This time, only level two predictors were included. The first model contained only the school climate factors (Organizational Capacity and Principal Support), while the second model contained only the school and community structural factors (Community Poverty, Size and Urbanicity, Grade level, and teacher turnover); the third model contained all six level two predictors. Three final models were then estimated, one for each of the implementation indicators (Level of Use, Frequency of Operation, and Intensity). Each model contained all nine level one predictors and all six level two predictors. For all models, interval-level variables were mean-centered while binary variables were left uncentered.Footnote 12
Results
One-Way ANOVAs
Table 3 shows the results of the one-way ANOVA models estimated in HLM. The intraclass correlation (ρ) was first calculated to find the percentage of the outcome variance that is between schools (ρ = τ00/(τ00 + σ2) where τ00 is the school-level variance and σ2 is the individual-level variance). Calculating the intraclass correlation (ρ) for all equations shows that 2.6% of the variation in level of use is between schools, while 1.6% of the variation in frequency of operation and 1.9% of the variation in the intensity scale is between schools (Table 3).
Program Level Predictors
The next set of models that were estimated contained only the level one predictors of implementation quality (Table 4). When program provider characteristics were the only predictors included in the models, Conscientiousness and Background displayed relationships only with Level of Use (0.194, p < 0.05, and 0.587, p < 0.001, respectively), such that schools make more extensive use of programs run by more conscientious and experienced providers; these factors had no impact on Frequency of Operation or Intensity. When program structure factors were the only predictors included in the models, however, more significant relationships were seen. Supervision and Standardization were related to Level of Use (0.122, p < .05, and 0.222, p < 0.01, respectively) and Integration was related to Intensity (0.114, p < 0.001), demonstrating that standardized programs that are more supervised during use and more integrated into normal school operations are likely to be better implemented. Interestingly, Training was positively related to Frequency of Operation (0.132, p < 0.05) but negatively related to Intensity (−0.080, p < 0.05), showing that programs with more high quality training are likely to be used more often but implemented with less intensity. When both categories of level one predictors were included in the models, all but one of the factors that were significant in the previous models remained significant: Notably, Conscientiousness no longer displayed a significant relationship with Level of Use; this relationship was rendered spurious once program structure characteristics were included in the model. Thus, it appears that the conscientiousness of the provider does not affect implementation if the chosen program is standardized and well supervised.
School Level Predictors
Table 5 shows the results of the models containing only level two predictors. When school climate factors were the only predictors included, Principal Support was related to Level of Use (0.191, p < 0.05) and Frequency of Operation (0.182, p < 0.001), such that schools with more supportive principals are likely to engage in higher quality implementation. Among the school and community structural factors, Urbanicity and Size impacted Level of Use and Frequency of Operation (0.091, p < 0.01, and 0.048, p < 0.05, respectively) and Community Poverty impacted Frequency of Operation (0.064, p < 0.01), such that programs are better implemented in larger schools located in poorer urban communities with a greater percentage of minorities. When both categories of level two predictors were included in the models, all factors that were significant in the separate models remained significant.
Full Model Results
Finally, three models were estimated that included all predictors of implementation quality (Table 6). Similar to the above models, Standardization and Supervision were positively related to Level of Use (0.269, p < 0.01, and 0.206, p < 0.05, respectively), demonstrating that standardized programs that are run by providers who are better supervised are likely to experience more extensive use. In addition, Level of Use was negatively related to Teacher Turnover (−0.011, p < 0.001); thus, schools with lower rates of teacher turnover are likely to make more extensive use of their programs. Notably, Background no longer displayed a significant relationship with Level of Use; this relationship was rendered spurious once school level factors were included in the model. Thus, it appears that the experience of the provider does not affect implementation if the chosen program is standardized, well supervised, and implemented in schools with lower teacher turnover.
As before, Training was negatively related to Intensity (−0.180, p < .01), illustrating that programs with frequent high quality training are likely to be implemented with less intensity. Interestingly, Organizational Capacity also had negative relationships with Intensity and Frequency of Operation (−0.087, p < .01 and −0.074, p > .05 respectively), suggesting the counterintuitive finding that schools with greater ability to organize for program implementation are actually less likely to achieve high quality implementation. Intensity was also related to Integration, Supervision, and Extended Day (0.119, p < .001, 0.175, p < .001, and −0.160, p < 0.05, respectively), such that programs are more likely to be effectively implemented if they are more integrated into normal school operations, better supervised, and run during regular school hours. Finally, Urbanicity and Size impacted Frequency of Operation (0.079, p < 0.05) such that programs are better implemented in larger schools located in urban communities.
Discussion
In this study, we sought to fill a gap in the prevention literature by investigating the impact of program provider, program structure, school climate, and school and community structural characteristics on program implementation. Using hierarchical linear modeling, we built upon previous research by measuring each of these four categories of factors at the appropriate level in order to more accurately assess their influence on implementation quality.
Despite prior research indicating the importance of personal provider characteristics (Gingiss et al. 2006; Glasgow et al. 2003; Hunter et al. 2001; Tobler 2000; Young et al. 1990), we found little support for this relationship. Both the conscientiousness and previous experience of the provider were significantly related to implementation quality when they were the only factors examined; however, when program structure was also considered, the relationship between provider characteristics weakened, so much so that conscientiousness was no longer statistically significant. The full model, in which all predictors of implementation quality were examined, yielded similar results: Neither the experience nor conscientiousness of the provider influenced program implementation. These findings provide a notable twist to other research, since support had previously been found highlighting the importance of provider leadership style and experience to implementation quality (Gingiss et al. 2006; Glasgow et al. 2003; Hunter et al. 2001; Tobler 2000; Young et al. 1990). Our initial analyses did support this notion. However, it appears as if these provider characteristics matter less when other characteristics of the program and school are present, thus rendering the relationship between program provider characteristics and implementation quality spurious.
When all program level factors were considered, program structural characteristics, such as the use of standardized materials, supervision over the program, integration into normal school operations, high quality training, and the use of the intervention during normal school hours, were related to the implementation quality of the program. In fact, these indicators of the structure of the intervention were the most consistent predictors of implementation quality throughout all analyses. These findings are in accord with previous research (Bosworth et al. 1999; Ennett et al. 2003; Gottfredson and Gottfredson 2002; Jaycox et al. 2006; Mihalic et al. 2008; Payne et al. 2006; Payne 2009; Rohrbach et al. 1993, 2006).
Additionally, when examining all school-level factors, a number of school climate and structural components were significantly related to implementation quality, including support of the school principal, school size, the urbanicity and poverty of the surrounding community, and teacher turnover. Again, our results are consistent with previous literature (Berends et al. 2002; Berman and McLaughlin 1978; Bosworth et al. 1999; Botvin et al. 1990, 1995a, b; Elliot and Mihalic 2004; Ennett et al. 2003; Farrell et al. 2001; Fullan 1991, 1992; Gager and Elias 1997; Gottfredson and Gottfredson 2002; Jaycox et al. 2006; Mihalic et al. 2008; Kam et al. 2003; Kegler et al. 1998; Payne et al. 2006; Payne 2009; Petersilia 1990; Rohrbach et al. 1993).
Interestingly, when all program and school characteristics were considered together, both similar and contrary results were found. As before, standardized programs that are better supervised are more likely to experience extensive use; in addition, schools with lower rates of teacher turnover are also more likely to extensively use these programs. The quality of training was also found to be significant, such that programs with higher quality training tend to be used more, but implemented with less intensity; perhaps this occurs because better trained providers can convey effective material in less time, thus leading to fewer sessions and shorter program duration. Additionally, programs that are well integrated into normal school operations, better supervised, and held during normal school hours are more likely to be effectively implemented. These relationships are consistent with previous research (Bosworth et al. 1999; Ennett et al. 2003; Gottfredson and Gottfredson 2002; Jaycox et al. 2006; Mihalic et al. 2008; Payne et al. 2006; Payne 2009; Rohrbach et al. 1993, 2006).
One finding not seen in previous models, however, was the negative relationship between organizational capacity and implementation quality, which suggests that schools with a greater ability to organize for implementation are actually less likely to achieve high implementation quality. Despite previous research showing that schools with greater organizational capacity are able to implement programs at a higher quality (Bosworth et al. 1999; Elliot and Mihalic 2004; Ennett et al. 2003; Fullan 1992; Gottfredson and Gottfredson 2002; Jaycox et al. 2006; McLaughlin 1990; Mihalic et al. 2008; Olds 2002; Payne et al. 2006; Payne 2009; Rohrbach et al. 2006; Schoenwald and Hoagwood 2001; Sobeck et al. 2006; Stoll and Fink 1996), we found the opposite effect. Perhaps these schools are aware of their ability to organize and, as such, feel as though high quality implementation will automatically follow; future research should investigate this possibility.
Another finding inconsistent with previous literature was the nonsignificant relationship between high levels of principal support and implementation quality (Berends et al. 2002; Farrell et al. 2001; Fullan 1991, 1992; Gager and Elias 1997; Kam et al. 2003; Kegler et al. 1998; Payne et al. 2006; Payne 2009; Petersilia 1990; Rohrbach et al. 1993; Smith et al. 1993). Our analyses suggest a spurious relationship once other factors were considered. Thus, it appears that once a school chooses a structured program that is well supervised and integrated into normal school activities, support from the principal is less important; again, future research should investigate this situation.
Limitations
One limitation of this study is the cross-sectional nature of the data. Because all data were collected at the same time, it is impossible to truly determine the causal direction of the associations found within this study. To assess the proper temporal ordering, future studies should be longitudinal in nature, collecting data on various predictors and outcomes at several points in time. Another limitation is the low school response rate overall and the relationship between survey participation and community characteristics. Schools in areas with more female-headed households with children, a greater proportion of urban population, and more households that received public assistance were significantly less likely to have participated in the study. Therefore, the study results may not generalize well to schools located in such communities.Footnote 13 Future research should replicate this study with samples that are more representative of schools in urban, disadvantaged communities. Other limitations include the fact that data were collected in the late 1990s, prior to the age of No Child Left Behind and the emphasis on standardized testing, and the fact that certain measures, such as organizational capacity and the implementation quality indicators, are based on only one person’s responses. A final limitation is the use of only three indicators of implementation quality, all of which measure intensity of implementation. As discussed earlier, Dane and Schneider (1998) identified several aspects of program quality, including exposure or dose, adherence to the program as described, delivery, participant responsiveness, and program differentiation. These other aspects of implementation are important to consider, as they are just as likely to affect program effectiveness as intensity; future research should explore other indicators of implementation quality.
Conclusion
The results of our study confirm the association between the structure of a program and the quality of its implementation. Significant relationships exist between implementation and standardization, supervision, training, and integration into normal school hours and operations. The findings, however, do not lend as much support to the importance of program provider, school climate, and school and community structural characteristics, once program structural characteristics are taken into account. Previous studies have failed to assess the effect of these factors in a multilevel fashion; our study provides a more accurate representation of their impact and thus suggests the spurious nature of some of these relationships.
Overall, our findings propose some guidelines to help schools increase the implementation quality of their prevention programs. While our results do not completely diminish the importance of other factors, they do emphasize the empirically supported importance of program structure factors. In general, a school that chooses a standardized program, supervises the prevention effort, provides frequent high quality training to team members, and integrates the program into normal school operations can increase the implementation quality of the intervention, which can then increase its intended effectiveness.
Notes
The original researchers used materials from government agencies, technical assistance providers, professional organizations, program marketers, and scientific literature to develop a list of 20 categories of activities and strategies to prevent and/or reduce delinquency (Gottfredson and Gottfredson 2002). These categories were intended to describe each important aspect of any prevention program or activity. Principals were then asked to name up to five different specific programs or activities for each category; thus, how each program was classified was a principal decision. One program/activity could be listed under multiple categories; in the original study, such multi-component programs made up 17% of the activities identified by principals (Gottfredson et al. 2000). The original researchers then randomly sampled 1 program or activity in each of 14 categories per school and sent the Activity Coordinator survey, which asks for more detailed information about the program, to the individual identified as responsible for the activity. If the same individual was named by the principal multiple times and this person was identified multiple times in the random selection process (as could be the case with multi-component programs), the principal was asked to identify a different person as responsible for one of the activities. If this was not possible, the original researchers re-sampled the activities (Gottfredson et al. 2000). In the current sample, multi-component programs were found in 63 schools (12%). To ensure that the inclusion of these programs did not alter the results, the models were rerun without these schools in the sample. Results were very similar to the results reported here, displaying significant relationships in the same direction and of the same strength.
Generally, all teachers in participating schools were sampled, and a sufficient number of students were sampled to produce an estimated 50 respondents per school. When a student roster containing student gender was available, students were systematically sampled within gender. Otherwise, students were stratified by grade level and sampled.
The final sample of 544 schools is 42.27% of the original 1,287 schools and 64.15% of the 848 schools that responded in Phase One. Student enrollment in these 544 schools ranged from 97 to 2,912, with a mean of 790.31 and a standard deviation of 478.40.
Measurement scales are based on scales developed and copyrighted by Gary Gottfredson (see Gottfredson et al. 2000).
Regression imputation is a standard process for dealing with missing data (Raghunathan et al. 2001). Given the significant correlations among the various indicators of implementation quality (ranging from 0.110 to 0.518, p < 0.01), using three or four indicators to predict a program’s score on one or two indicators fall within the acceptable guidelines of this practice.
As can be seen in Table 2, Conscientiousness appears to have limited variability and a positive skew; this can potentially affect this study’s findings. Self-reported questionnaires designed to collect responses aimed at personal performance can threaten the reliability and validity of outcome measures due to social desirability bias. While these threats have been found to be limited in research, there nevertheless exists the possibility that social desirability can affect outcome measures by producing spurious results, suppressing real results, or moderating the relationships between the independent and dependent variables (Ganster et al. 1983).
Results of exploratory factor analyses from which these factors were created can be seen in Payne et al. (2006).
The following Census variables are markers for the Community Poverty factor: welfare (average household public assistance income), female-headed households (ratio of single females with children under 18 to married couples with children under 18), median income (proportion of households with income below $27,499), poverty (ratio of persons below the 1.24 poverty level to persons above), divorce rate (ratio of persons over 15 years who are married to those who are separated, divorced, or have a spouse absent), and unemployment (proportion of unemployed males/females in the labor force) (Simonsen 1998).
The following variables are markers for the Urbanicity factor: population size (total population), urban level (city level type), and urbanicity (the proportion of people living within an urban area) (Simonsen 1998).
Regression imputation is a standard process for dealing with missing data (Raghunathan et al. 2001). Ten different census variables were used for imputation. For each imputed variable, those census variables with the largest correlations with the variable to be imputed were used. Between 1 and 128 schools required imputation for exogenous variables taken from sources other than the teacher surveys. The two exogenous variables that are taken from the teacher survey (Percentage Teachers African–American and Number of Different Students Taught) were missing data for 221 and 220 schools, respectively. Because imputation was required for a large number of schools, results were examined for possible changes when these two variables were not included in the model; no significant changes were seen.
The generic level one model is Yij = β0j + β1jX1ij + … βkjXkij + rij where Yij is the outcome for ith individual in jth school, β0j is the intercept or the mean level of the outcome for each j school, βkj is the regression coefficient for the effect on the outcome variable of Xij, which is a level one predictor of the outcome, and rij is the level one error term. This is also known as the within-school equation. The generic level two model is β0j = γ00 + γ01W1j + γ02W2j + … γ0qWqj + u0j, β1j = γ10 + γ11W1j + γ12W2j + … γ1qWqj + u1j, βkj = γk0 + γk1W1j + γk2W2j + … γkqWqj + ukj where β0j is the school level mean of the outcome from level one equation, γ00 is the level two intercept or the mean level of the level one outcome for the entire sample, γ0q is the regression coefficient for the effect on the school level mean of Wqj, which is a level two predictor, and u0j is the level two error term. As in the level one model, βkj is the coefficient for a level one predictor of Yij, the individual-level outcome. However, in level two, βkj becomes an outcome itself of a level two predictor Wqj. This is also known as the between-school equation.
Centering adjusts the outcome (either level one or level two) for the particular control variable and the outcome is the predicted value for the program or school with the mean value of that interval variable. Binary variables were left uncentered. Therefore, in the level one equation, the outcome is the predicted value for a program whose value is zero on the binary variable, while the outcome in the level two equation is the predicted value for a school whose value is zero on the binary variable.
It is unlikely, however, that the basic results of our study would change had more of these schools been included. Exploratory analyses of potential biases introduced by the limited response rates examined participating schools that were located in similar communities as the majority of non-participating schools and found results similar to those seen here. Therefore, it seems likely that the inclusion of the non-participating schools would have resulted in actually intensifying the relationships reported in this study. Of course, it is possible that the relationships of interest are not linear in the region of the distribution in which non-participating schools fall, or that some characteristic might alter the relationships established. However, the linear relationships among this study’s measures seem to indicate that, if anything, the results presented here provide conservative estimates of the relationships.
References
Battistich, V., Schaps, E., Watson, M., & Solomon, D. (1996). Prevention effects of the Child Development Project: Early findings from an ongoing multi-site demonstration trial. Journal of Adolescent Research, 11, 12–35.
Berends, M., Bodilly, S.J., & Kirby, S.N. (2002). Facing the challenges of whole-school reform. New American schools after a decade. Santa Monica, CA: RAND.
Berman, P., & McLaughlin, M.W. (1978). Federal programs supporting educational change, Vol VIII: Implementing and sustaining innovations. Santa Monica, CA: RAND.
Bernd, M. (1992). Shared decision making requires effective instructional leadership. NASSP Bulletin, 76, 64–69.
Boerm, M., Gingiss, P., & Roberts-Gray, C. (2007). Association of the presence of state and district health education policies with school tobacco prevention program practices. Journal of School Health, 77, 207–214.
Bosworth, K., Gingiss, P., Potthoff, S., & Roberts-Gray, C. (1999). A Bayesian model to predict the success of the implementation of a health education innovation. Journal of Evaluation and Research, Evaluation and Program Planning, 22, 1–11.
Botvin, G.J. (1990). Substance abuse prevention: Theory, practice, and effectiveness. In M. Tonry & J.Q. Wilson (Eds.), Drugs and crime (pp. 461–520). Chicago: Chicago University Press.
Botvin, G.J. (2003). From research to policy: Advancing prevention science and practice. Presidential Address presented at the 11th Annual Meeting of the Society of Prevention Research. Washington, D.C.
Botvin, G.J., Batson, H.W., Witts-Vitale, S., Bess, V., Baker, E., & Dusenbury, L. (1989a). A psychological approach to smoking prevention for urban Black youth. Public Health Reports, 12, 279–296.
Botvin, G.J., Dusenbury, L., James-Oritz, S., & Kerner, J. (1989b). A skills training approach to smoking prevention among Hispanic youth. Journal of Behavioral Medicine, 12, 179–296.
Botvin, G.J., Baker, E., Dusenbury, L., Tortu, S., & Botvin, E.M. (1990). Preventing adolescent drug abuse through a multi-modal cognitive-behavioral approach: Results of a 3-year study. Journal of Consulting and Clinical Psychology, 58, 437–446.
Botvin, G.J., Baker, E., Dusenbury, L., Botvin, E.M., & Diaz, T. (1995a). Long-term Follow-up results of a randomized drug abuse prevention trial in a white middle-class population. Journal of the American Medical Association, 273, 1106–1112.
Botvin, G.J., Schinke, S., & Orlandi, M.A. (1995b). School-based health promotion: Substance abuse and sexual behavior. Applied and Preventive Psychology, 4, 167–184.
Boyd, V. (1992). School context: Bridge or barrier to change? Office of Educational Research and Improvement Contract No. RP 1002003. Austin, TX: Southwest Educational Development Library.
Brink, S.G., Based-Enquist, K.M., O’Hara-Tompkins, N.M., Parcel, G.S., Gottlieb, N.H., & Lovato, C.Y. (1995). Diffusion of an effective tobacco prevention program. Part I: Evaluation of the dissemination phase. Health Education Research, 10, 283–295.
Catalano, R.F., Arthur, M.W., Hawkins, J.D., Berglund, M.L., & Olson, J.J. (1998). Comprehensive community and school-based interventions to prevent antisocial behavior. In R. Loeber & D.F. Farrington (Eds.), Serious and violent juvenile offenders (pp. 248–283). Thousand Oaks, CA: Sage.
Dane, A.V., & Schneider, B.H. (1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control? Clinical Psychology Review, 18, 23–45.
Elliot, D.S., & Mihalic, S. (2004). Issues in disseminating and replicating effective prevention programs. Prevention Science, 5, 47–53.
Ennett, S.T., Tobler, N.S., Ringwalt, C.L., & Flewelling, R.L. (1994). How effective is drug abuse resistance education? A meta-analyses of Project DARE outcome evaluations. American Journal of Public Health, 84, 1394–1404.
Ennett, S.T., Ringwalt, C.L., Thorne, J., Rohrbach, L.A., Vincus, A., & Simons-Rudolph, A. (2003). A comparison of current practice in school-based substance use prevention programs with meta-analysis findings. Prevention Science, 4, 1–14.
Farrell, A.D., Meyer, A.L., Kung, E.M., & Sullivan, T.N. (2001). Development and evaluation of school-based violence prevention programs. Journal of Clinical Child Psychology, 31, 235–253.
Fullan, M.G. (1991). The new meaning of educational change. New York: Teachers College Press.
Fullan, M.G. (1992). Successful school improvement: The implementation perspective and beyond. Philadelphia: Open University Press.
Fullan, M.G., & Pomfret, A. (1977). Research on curriculum and instruction implementation. Review of Educational Research, 47, 335–397.
Gager, P.J., & Elias, M.J. (1997). Implementing prevention programs in high-risk environments: Application of the Resiliency Paradigm. American Journal of Orthopsychiatry, 67, 363–373.
Ganster, D.C., Hennessey, H.W., & Luthans, F. (1983). Social desirability response effects: Three alternative models. Academy of Management Journal, 26, 321–331.
Gingiss, P., Roberts-Gray, C., & Boerm, M. (2006). Bridge-it: A system for predicting implementation fidelity for school-based tobacco prevention programs. Prevention Science, 7, 197–207.
Glasgow, R.E., Lichtenstein, E., & Marcus, A.C. (2003). Why don’t we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition. American Journal of Public Health, 93, 1261–1267.
Goldstein, A.P., & Sorcher, M. (1973). Changing managerial behavior by applied learning techniques. Training and Development Journal, 27, 36–39.
Gottfredson, D.C. (2001). Delinquency and schools. New York: Cambridge University Press.
Gottfredson, D.C., & Gottfredson, G.D. (2002). Quality of school-based prevention programs: Results from a national survey. Journal of Research in Crime and Delinquency, 39, 3–36.
Gottfredson, D.C., Gottfredson, G.D., & Hybl, L.G. (1993). Managing adolescent behavior: A multi-year, multi-school study. American Educational Research Journal, 30, 179–215.
Gottfredson, D.C., Gottfredson, G.D., & Skroban, S. (1996). A multimodal school-based prevention demonstration. Journal of Adolescent Research, 11, 97–116.
Gottfredson, G.D., Gottfredson, D.C., Czeh, E., Cantor, D., Crosse, S., & Hantman, I. (2000). A national study of delinquency prevention in school final report. Ellicott City, MD: Gottfredson Associates, Inc.
Gottfredson, D.C., Wilson, D.B., & Najaka, S.S. (2002). School-based crime prevention. In D.P. Farrington, L.W. Shernman, & B. Welsh (Eds.), Evidence based crime prevention (pp. 56–164). London: Harwood.
Hawkins, J.D., Arthur, M.W., & Catalano, R.F. (1995). Preventing substance abuse. In M. Tonry & D.P. Farrington (Eds.), Building a safer society: Strategic approaches to crime prevention (pp. 343–428). Chicago: University of Chicago Press.
Hunter, L., Elias, M.J., & Norris, J. (2001). School-based violence prevention: Challenges and lessons learned from an action research project. Journal of School Psychology, 39, 161–175.
Jaycox, L.H., McCaffrey, D.F., Weidmer, B., Shelley, G.A., Blake, S.M., & Peterson, D.J. (2006). Challenges in the evaluation and implementation of school-based prevention and intervention programs on sensitive topics. American Journal of Evaluation, 27, 320–336.
Joyce, B., & Showers, B. (2002). Student achievement through staff development (3rd ed.). Alexandria, VA: Association for Supervision and Curriculum Development.
Kam, C.M., Greenberg, M.T., & Walls, C.T. (2003). Examining the role of implementation quality in school-based prevention using the PATHS curriculum. Prevention Science, 4, 55–63.
Kegler, M.C., Steckler, A., Malek, S.H., & McLeroy, K. (1998). A multiple case study of implementation in 10 local Project ASSIST coalitions in North Carolina. Health Education Research, 13, 225–238.
Lipsey, M.W. (1992). Juvenile delinquency treatment: A meta-analytic inquiry into the variability of effects. In T.D. Cook, H. Cooper, D.S. Cordray, H. Hartmann, L.V. Hedges, R.J. Light, et al. (Eds.), Meta-analysis for explanation: A casebook (pp. 83–127). New York: Sage.
Lipsey, M.W., & Derzon, J.H. (1998). Predictors of violent or serious delinquency in adolescence and early adulthood: A synthesis of longitudinal research. In R. Loeber & D.P. Farrington (Eds.), Serious and violent juvenile offenders: Risk factors and successful interventions (pp. 86–105). Thousand Oaks, CA: Sage.
McCormick, L.K., Steckler, A., & McLeroy, K.R. (1995). Diffusion of innovations in schools: A study of adoption and implementation of school-based tobacco prevention curricula. American Journal of Health Promotion, 9, 210–219.
McLaughlin, M.W. (1990). The RAND change agent study revisited: Macro-perspective and micro realities. Educational Researcher, 19, 11–16.
Mihalic, S.F., Fagan, A.A., & Argamaso, S. (2008). Implementing the lifeskills training drug prevention programs: Factors related to implementation fidelity. Implementation Science, 3, 1–16.
National Institute on Drug Abuse. (2003). What do schools really think about prevention research? Blending research and reality. Bethesda, MD: Author.
Nunnery, J., Slavin, R.E., Madden, N.A., Ross, S., Smith, L., Hunter, P., et al. (1997). Effects of full and partial implementations of Success For All on student reading achievement in English and Spanish. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL.
Olds, D.L. (2002). Prenatal and infancy home visiting by nurses: From randomized trials to community replication. Prevention Science, 3, 153–172.
Parcel, G.S., O’Hara-Tompkins, N.M., Harrist, R.B., Basen-Engquist, K.M., McCormick, L.K., Gottlieb, N.H., et al. (1995). Diffusion of an effective tobacco prevention program. Part II: Evaluation of the adoption phase. Health Education Research, 10, 297–307.
Payne, A.A. (2009). Do predictors of implementation quality of school-based prevention programs differ by program type? Prevention Science, 10, 151–167.
Payne, A.A., Gottfredson, D.C., & Gottfredson, G.D. (2006). School predictors of the intensity of implementation of school-based prevention programs: Results from a national study. Prevention Science, 7, 225–237.
Petersilia, J. (1990). Conditions that permit intensive supervision programs to survive. Crime and Delinquency, 36, 126–156.
Raghunathan, T., Lepkowski, J., Van Hoewyk, J., & Solenberger, P. (2001). A multivariate technique for multiply imputing missing values using a sequence of regression models. Survey Methodology, 27, 85–95.
Raudenbush, S.W., Bryk, A.S., Cheong, Y.F., & Congdon, R.T. (2004). HLM 6: Hierarchical linear and nonlinear modeling. Lincolnwood, IL: Scientific Software International.
Rohrbach, L.A., Graham, J.W., & Hansen, W.B. (1993). Diffusion of a school-based substance abuse prevention program: Predictors of program implementation. Preventive Medicine, 22, 237–260.
Rohrbach, L.A., D’Onofrio, C.N., Backer, T.E., & Mongomery, S.B. (1996). Diffusion of substance abuse prevention programs. American Behavioral Scientist, 39, 919–934.
Rohrbach, L.A., Grana, R., Sussman, S., & Valente, T.W. (2006). Type II translation: Transporting prevention interventions from research to real-world settings. Evaluation and the Health Professions, 29, 302–333.
Schoenwald, S.K., & Hoagwood, K. (2001). Effectiveness, transportability, and dissemination of interventions: What matters when? Psychatric Services, 52, 1190–1196.
Silivia, E.S., & Thorne, J. (1997). School-based drug prevention programs: A longitudinal study in selected school district. Research Triangle, NC: Research Triangle Institute.
Simonsen, A.A. (1998). The effects of community disorganization on school administrative practices: Implications for delinquency prevention practice. Unpublished Masters Thesis. University of Maryland: College Park.
Smith, D.W., McCormick, L.K., Steckler, A.B., & McLeroy, K.R. (1993). Teachers’ use of health curricula: Implementation of growing health, project SMART, and the teenage health teaching modules. Journal of School Health, 63, 349–354.
Smith, L., Ross, S., & Nunnery, J. (1997). Increasing the chances of Success For All: The relationship between program implementation quality and student achievement at eight inner-city schools. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL.
Sobeck, J.L., Abbey, A., & Agius, E. (2006). Lessons learned from implementing school-based substance abuse prevention curriculums. Children and Schools, 28, 77–85.
Stoll, L., & Fink, D. (1996). Changing our schools: Linking school effectiveness and school improvement. Philadelphia: Open University Press.
Tobler, N.S. (1992). Drug prevention programs can work: Research findings. Journal of Addictive Diseases, 11, 1–28.
Tobler, N.S. (2000). Lessons learned. Journal of Primary Prevention, 20, 261–274.
Walker, E.M. (2004). The impacts of state policies and actions on local implementation efforts: A study of whole school reform in New Jersey. Educational Policy, 18, 338–363.
Wilson, S.J., Lipsey, M.W., & Derzon, J.H. (2003). The effects of school-based intervention programs on aggressive behavior: A meta-analysis. Journal of Consulting and Clinical Psychology, 71, 136–150.
Young, R.L., deMoor, C., Wilder, M.B., Gully, S., Hovell, M.F., & Elder, J.P. (1990). Correlates of health facilitator performance in a tobacco-use prevention program: Implications for recruitment. Journal of School Health, 60, 463–467.
Zins, J.E., Elias, M.J., Greenberg, M.T., & Pruett, M.K. (2000). Special issue: Implementation of prevention programs. Journal of Educational and Psychological Consultation, 11, 1–172.
Author information
Authors and Affiliations
Corresponding author
Additional information
The authors would like to extend a special thank you to Gary and Denise Gottfredson for the generous use of their National Study of Delinquency Prevention in Schools data.
Appendix
Appendix
Rights and permissions
About this article
Cite this article
Payne, A.A., Eckert, R. The Relative Importance of Provider, Program, School, and Community Predictors of the Implementation Quality of School-Based Prevention Programs. Prev Sci 11, 126–141 (2010). https://doi.org/10.1007/s11121-009-0157-6
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11121-009-0157-6