Introduction

Community coalitions serve an important role in the provision of social services in the United States. When implemented with high fidelity to an effective model that uses a science-based approach to prevention, coalition-based strategies can produce population-level outcomes. Two coalition-based strategies have been shown to effectively prevent youth mental, emotional, and behavioral problems: PROmoting School-community-university Partnerships to Enhance Resilience (PROSPER) (Spoth et al. 2004, 2011) and Communities That Care (CTC) (Hawkins and Catalano 1992; Hawkins et al. 2002). Although their specific implementation processes differ, both strategies rely on community stakeholders to mobilize a coalition that supports the community’s use of scientific principles to guide prevention planning (Hawkins et al. 2010). The current study uses data from a community-randomized trial of CTC to examine capacities of these coalitions.

In the CTC prevention system, coalitions facilitate community adoption of a scientific approach to prevention using a five-stage process (Shapiro et al. 2013b). First, community stakeholders make a commitment to collaboration and science-based prevention, addressing any initial barriers to these commitments. Second, a coalition is formalized that receives training and technical assistance to develop their general capacities (i.e., non-intervention specific capacities, e.g., coalition structure, team functioning, and member or organizational capacities) at the member and organizational levels (Wandersman et al. 2008; Quinby et al. 2008). Third, coalitions initiate the collection of epidemiological data from community youth to obtain reliable and valid population-level estimates of the prevalence of preventable problems and the risk and protective factors shown by research to predict undesirable outcomes (Arthur et al. 2007; Glaser et al. 2005). Fourth, coalition members use the youth survey data to prioritize the risk and protective factors that the coalition wishes to target, and they create an action plan to implement programs and policies that have been tested and demonstrated to be effective in changing the targeted risk/protective factors and problem behaviors (Substance Abuse and Mental Health Services Administration 2005; Center for the Study and Prevention of Violence 2011). Fifth, coalition members work with community organizations to promote high-quality program implementation by monitoring regularly collected data to ensure the newly installed programs and policies are well implemented and effective (Fagan et al. 2008). Thus, coalition members intend to facilitate the use of effective prevention practices community-wide.

Findings from the randomized trial of CTC indicate that 1.5 years after initial implementation of CTC, community leaders in CTC communities reported the adoption of a scientific approach to prevention in their communities to a greater extent than community leaders in control communities (Brown et al. 2007). In turn, greater adoption of a science-based approach to prevention in CTC communities explained the significantly lower level of youth problem behaviors in CTC compared to control communities observed in a cohort of 8th graders followed since 5th grade (Brown et al. 2014). These findings provide empirical support for CTC’s hypothesized link between science-based community coalition efforts and improving population health (Hawkins et al. 2008).

In the community-randomized trial of CTC, all 12 CTC coalitions implemented CTC with high fidelity (Quinby et al. 2008; Fagan et al. 2008). Yet, the degree to which communities adopted a science-based approach to prevention (as reported by community leaders) still varied across CTC communities (Shapiro et al. 2013b). This finding suggests that high-fidelity implementation of CTC does not necessarily lead to equal coalition capacity to achieve community-wide changes. While there are core components of CTC that each coalition is expected to implement, there are many choices that coalitions make during implementation. Some CTC coalitions may be more successful at science-based prevention as a result of ways in which their coalitions are structured, function as a team, or develop member or organizational capacities (Florin et al. 2000; Roussos and Fawcett 2000; Allen et al. 2008). To improve the effectiveness of community coalitions in their prevention work, we need to understand how coalitions differ in their capacities and how these differences may impact their ability to achieve community-wide change. To achieve this goal, this study (1) describes newly developed theoretically driven scales that measure coalition capacities; (2) reports the extent to which CTC coalitions differed in these capacities 1 year after CTC implementation began in the randomized trial; and (3) examines the extent to which coalition capacities moderate the impact of CTC on a community-wide adoption of a science-based approach to prevention as reported by community leaders. If specific coalition capacities are associated with greater community-wide adoption of a scientific approach to prevention, building these capacities could be a mechanism for making CTC coalitions more effective.

Coalition Capacities

Coalition capacities are the characteristics that affect a coalition’s ability to identify, mobilize, and address social and public health problems, described as “central to addressing the gap between research and practice” (Goodman et al. 1998; Wandersman et al. 2008, p. 173). Capacity is a term that is used to convey a dynamic, adjustable, and transferable nature of member and organizational characteristics (Butterfoss 2002; Foster-Fishman et al. 2001). Understanding the relationship between coalition capacities and outputs such as the adoption of a science-based approach to prevention is important for improving the work of coalitions (Florin et al. 2000; Roussos and Fawcett 2000; Allen et al. 2008). Building coalition capacities is an important step in a sequential process through which successful coalitions affect community change (Florin et al. 2000). Inputs (such as externally supplied training and technical assistance) are utilized by diverse coalition members who, if organized into a well-functioning team, will generate the coalition capacities that produce desired community outputs, such as the adoption of a science-based approach to prevention. This paper explores the extent to which CTC coalitions were able to build three theorized types of coalition capacities (member capacities, member attitudes, and organizational capacities), how much CTC coalitions differed in their ability to generate these capacities, and the extent to which CTC coalitions with more capacities were able to achieve greater adoption of a science-based approach to prevention in their communities (as reported by community leaders).

Member Capacity

CTC intends to build member capacity for effective prevention practices. Member capacity concerns the knowledge and skills of coalition members (Florin et al. 2000; Foster-Fishman et al. 2001). There is broad conceptual consensus that this is an important dimension of capacity building and an important outcome of coalition work in its own right (Goodman et al. 1996; Mitchell et al. 1996; Mizrahi and Rosenthal 2001; Roussos and Fawcett 2000), but little agreement as to how to measure it. Previous investigations of member knowledge and skill building informed the measurement of member capacities in this paper in five ways. First, some prior research relating coalition capacity to outcomes has focused on the knowledge and skills of staff members hired to coordinate the work of public health coalitions (Kegler et al. 1998) rather than of coalition members themselves. While staff skills are likely to be important to coalition success, the knowledge and skills of paid staff may be less important to the sustained effects of a coalition than the knowledge and skills developed within the membership of the coalition itself. This study will focus on the capacities of coalition members rather than the capacities of paid coalition staff.

Second, this study measures the acquisition of new skills that are built through coalition activity rather than the skills coalition members bring to the coalition. Coalition members report joining coalitions with pre-existing skills in the areas of priority setting and developing action plans, understanding different perspectives, facilitating groups, communicating effectively in groups, and conflict resolution (Kegler et al. 2007). This study focused on coalition member perceptions of their acquisition of new skills in the domains of organization and communication, changing local politics, and designing and carrying out prevention programs. Third, this study assesses knowledge attainment directly, rather than through a proxy or perception. Prior research indicates that a large majority of coalition members will retrospectively report increases in their knowledge (Florin et al. 1993), but these self-reports tend to be higher than external reports of members’ expertise (Rogers et al. 1993). Other studies have operationalized member knowledge by determining the percentage of coalition members with college degrees; yet members’ education level was not related to their perceptions of the coalition’s impact (Wells et al. 2009). A more accurate way of measuring coalition members’ knowledge attainment may be to test their knowledge of relevant prevention science principles by assessing the number of correct responses to factual questions (Crowley et al. 2012).

Fourth, member capacity is often studied as a single concept, since the component parts, knowledge and skills, are highly correlated. However, disaggregating knowledge from skills may reveal important distinctions. For example, coalition members have ranked the benefits of knowledge acquisition higher than skill building (Kegler et al. 2005), and coalition chairs and coordinators have judged knowledge acquisition as more strongly related to effectiveness and success (Allen et al. 2008; Mizrahi and Rosenthal 2001). Although both knowledge and skills are theorized to be important, knowledge seems to be differentiated from skill building in the eyes of practitioners, and their independent impacts on community change have not been as carefully studied. Furthermore, perceptual differences in member knowledge acquisition by informant (Rogers et al. 1993) suggest a need to determine whether knowledge about the targeted social issue, measured directly through the number of correct responses to substantive questions, predicts intermediate or long-term outcomes (Kumpfer et al. 1993).

Finally, previous research has found strong positive correlations between self-perceptions of growth in member capacities and perceptions of coalition impact on community objectives (Rogers et al. 1993; Florin et al. 2000). However, when coalition members were asked about the coalition’s impact, 80 % of members characterized their coalition as a total success (Mizrahi and Rosenthal 2001), suggesting that there is very limited variation in perceived coalition success that could be explained by member capacities. To determine if member capacities influence actual coalition impact on community-wide change, it is important to assess coalition impact independently of the coalition members’ perceptions. This study examines coalition impact using reports of community leaders on the adoption of a science-based approach to prevention in the community and examines its association with coalition member reports of new skill acquisition and directly assessed substantive knowledge.

Member Attitudes

CTC intends to build attitudes conducive to effective prevention practice. Granner and Sharpe (2004) reviewed instruments that have measured attitudes in a range of ways, including satisfaction with progress, enjoyment of membership, satisfaction with group functioning and achievement, satisfaction with the planning process, satisfaction with operations and accomplishments, and attitudes toward the partnership and desire to remain a member. Few studies have linked attitudes to outcomes. Some studies indicate that coalition member attitudes of satisfaction correlate directly with expectations and perceptions of coalition accomplishment (Rogers et al. 1993; Kegler et al. 2005), and inversely with member’s self-report of their own undesirable (e.g., drug and alcohol related) behavior (Kumpfer et al. 1993). Member attitudes have failed to predict the quality of objectively rated community action plans, lending support to the idea that favorable attitudes of coalition members may be insufficient for improving community prevention planning (Butterfoss et al. 1993; Kreuter et al. 2000). Although attitudes seem to predict expectations for change, the impact of attitudes on actual community-wide change is largely unknown.

Organizational Capacity

The organizational capacity of a coalition is conceptualized as having two dimensions: the extent to which the coalition has involved various constituencies (i.e., inter-organizational linkages) and the extent to which this involvement has influenced practices within the sectors that the members represent (i.e., the spread of new technologies). CTC intends to build organizational linkages by engaging diverse community sectors in the work of the coalition and to influence organizational practices by supporting staff development for the implementation of selected prevention practices. Previous studies have reported difficulty in forming linkages with non-health sectors, such as faith communities (Roussos and Fawcett 2000), but that broad involvement from community sectors predicts community leader’s perceptions of coalition impact (Florin et al. 2000) and program implementation success (Halgunseth et al. 2012). The depth of collaboration across constituencies, however, appears unrelated to perceptions of coalition impact on system change (Hays et al. 2000). The count of the sectors that the coalition has worked with closely, as perceived by paid coalition coordinators, also does not predict perceptions of effectiveness (Feinberg et al. 2004). Therefore, in the present study, organizational capacity is a measure of average levels of involvement in the coalition across all sectors rather than the deep involvement of any or all sectors.

Coalition impact on organizations has been presumed to occur in many ways, including an exchange of information across organizations, an increased awareness of resources within other organizations, improved contact and communication across organizations, and the change of policies and use of resources within organizations (Butterfoss et al. 1996; Florin et al. 2000; Hays et al. 2000; Kumpfer et al. 1993; McMillan et al. 1995). Previous studies have shown an impact of CTC training on perceptions of coalition influence on diverse community sectors (Feinberg et al. 2002). In this study we seek to determine if member perceptions of influence on sectors predict independently measured community-wide adoption of a science-based approach to prevention.

Research Opportunity

Studying the relationship between coalition capacities and coalition outcomes has proven difficult. Several methodological challenges exist (Darnell et al. 2013). These include an underspecified theory of community change, the lack of comparison groups to help rule out alternative hypotheses, lack of random assignment to comparison and experimental conditions, limited temporal ordering, the absence of outcome assessments by independent reporters that can be compared against predetermined criteria, inadequate sample sizes for detecting effects, and low response rates to surveys (often as low as 35–44 %; rarely over 67 %) that may render samples non-representative (Allen 2005; Florin et al. 1993, 2000; Merzel and D’Afflitti 2003; Kegler et al. 2005; Kumpfer et al. 1993; Rogers et al. 1993). The present study addresses many of the methodological challenges of prior research on coalitions. Data from the Community Youth Development Study (CYDS) are used to test a clearly specified theory of change. The study uses a community-randomized design that compares a sufficient number of CTC communities to control communities. It is well suited for exploring the extent to which member capacities, member attitudes, and organizational capacities of CTC coalitions, as reported by coalition members, differ among coalitions and predict a community’s adoption of a science-based approach to prevention, as reported by community leaders. Capacities were assessed 1 year after capacity-building efforts (e.g., training) began, but before the adoption of science-based prevention was measured, to reflect the theory of change. The outcome of “Adoption” is a well-defined outcome criterion, measured by unique informants participating at a high response rate.

Methods

Sample

The CYDS was approved by the Institutional Review Board. The 24 communities in 7 states selected for the CYDS were incorporated towns with clear boundaries and relative autonomy in the provision of educational and social services. Population sizes ranged from 1,578 to 40,787 people (average 14,646 people), and the percentage of White residents ranged from 64 to 98.2 % (average 89 %). Communities were matched into pairs, within state, based upon similar population sizes, race and ethnic compositions, economic indicators, and crime rates (Arthur et al. 2005). A community in each of the 12 matched pairs was randomly assigned to the CTC condition.

In the 5 years prior to the trial, communities within each pair did not differ in their use of a science-based approach to prevention (Arthur et al. 2003). During the 5-year intervention period of the trial (2004–2008), CTC was implemented with technical support and trainings intended to develop community capacities (Quinby et al. 2008). The control communities continued “prevention as usual,” which typically included coalition-based approaches, but did not necessarily reflect the CTC approach. Since “prevention as usual” did not always include a coalition-based approach, coalition information, including capacities, is only available in CTC communities.

Coalition capacities were measured in CTC communities 1 year after the introduction of CTC. The timing of this observation was designed to assess capacities after sufficient capacity-building efforts had been made, but before communities wrestled with issues of member turnover and initiative sustainability. The median number of CTC coalition members per community was 40, and a maximum of 20 coalition members per community (randomly selected from coalition rosters) were invited to participate in an interview about their individual and organizational capacities (Shapiro et al. 2013a). The total rate of interview completion was 96.6 % (n = 218), with every CTC coalition’s response rate ≥90 %. Race, class, and gender information was not collected from these respondents.

Community leaders served as informants regarding the community’s adoption of a science-based approach to prevention 1.5 years after the introduction of CTC in experimental communities (Arthur et al. 2005). In each of the 24 communities, across both the CTC and control conditions, individuals in formal leadership positions (e.g., mayors, city managers, police chiefs, school superintendents, business leaders, and heads of social service agencies) across 11 different community sectors were interviewed. At the end of this interview, each leader was asked to identify two individuals who were the most knowledgeable about current prevention efforts in the community. Of the people in each community most often identified, five additional leaders per community were invited to participate in an interview. This sampling strategy yielded 335 complete interviews (a 96 % response rate) across both CTC and comparison communities (Brown et al. 2007). The samples of leaders did not differ across intervention condition by age, gender, level of education, length of time living in the community, or community sector that the leaders represented (see Brown et al. 2007 for additional detail).

Measures

Coalition capacities were assessed using data from telephone interviews with CTC coalition members (Shapiro et al. 2013a). Five components of coalition capacity were assessed: members’ substantive knowledge of prevention, members’ acquisition of new skills, members’ attitudes toward CTC, organizational linkages to the coalition across community sectors, and members’ perceptions of their coalition’s influence on organizations in the community. Member substantive knowledge of prevention is a count of the number of correct responses to five questions that reflect information about prevention science presented in training sessions (e.g., “which factor would you say is more important for preventing adolescent problem behaviors: opportunities for active involvement in the classroom or information on the effects of drugs?”). Member acquisition of new skills is an average of three items that ask coalition members if they have benefited (a great deal to not at all) from learning new skills in the areas of organization and communication, changing local politics, and designing and carrying out prevention programs.

Member attitudes toward CTC is an average of seven items (strongly disagree to strongly agree or a lot better to a lot worse) that captures the member’s attitudes (e.g., “The CTC board is addressing important community concerns”; “I support the CTC prevention approach”; “How does the CTC prevention approach compare to what your community was doing before CTC to prevent adolescent problem behavior?”). To evaluate organizational linkages, responses to 13 questions about different sectors’ involvement (very involved to not at all involved) in the coalition’s work were averaged (including elected leaders, parents, school teachers/staff, school district administrators, social service providers, students, business leaders, faith leaders, law enforcement, media representatives, recreation officials, community volunteers, and unelected leaders). The influence on organizations in the community was ascertained by asking coalition members how much influence (1–4; none to a lot) they think CTC has had on seven different local sectors: government leaders, business leaders, law enforcement, media, parents/parent leaders, school administrators/staff, and youth/health service agencies.

The adoption of a science-based approach to prevention was assessed through telephone interviews with community leaders across all 24 communities (Arthur et al. 2005; Shapiro et al. 2013b). The 22 questions were not CTC-specific, but used generic language to assess the extent to which communities were using a science-based approach to prevention in their communities (e.g., “Did your community prioritize risk and protective factors that you wanted to address with prevention activities?”). An adoption score of 0 indicated little or no awareness of prevention science concepts. A score of 1 indicated awareness of risk and protection focused prevention, but no intention to use these concepts to guide prevention services. A score of 2 indicated that the community intended to use risk and protection focused prevention, but did not collect epidemiologic data to guide prevention activities. A score of 3 indicated that the community collected epidemiologic data but did not use effective preventive interventions. A score of 4 indicated that the community developed a plan to use effective preventive interventions to address prioritized risk and protective factors based on epidemiologic data collected in their community. Finally, a score of 5 indicated the implementation of effective preventive interventions and ongoing monitoring of the implementation and effects. Each stage of adoption required that the criteria for prior stages had been met.

Previous studies have determined that community leaders in CTC communities reported greater levels of adoption of a science-based approach to prevention 1.5 years after implementation of CTC relative to community leaders in control communities. Community adoption scores in CTC communities ranged from 1.87 to 3.73 (M = 2.80, SD = .55), whereas community adoption scores in control communities ranged from .62 to 3.29 (M = 1.69, SD = .79) at this point in CTC implementation. The standardized mean difference between CTC and control communities in adoption was d = .78, with 37 % of the total variation in adoption attributable to differences among community pairs (see Shapiro et al. 2013b for figure and detail).

Analysis Strategy

To explore the measurement properties of coalition capacities, means, standard deviations, and internal reliabilities of each scale were examined (Cronbach 1951). A 0.6 criterion was used to indicate acceptable internal reliability, with the understanding that constructs with low coefficients will need to be carefully evaluated for the potential of attenuated relationships with other variables (Schmitt 1996). To understand the extent to which coalition capacities varied across the 12 CTC coalitions, unconditional multilevel linear regression models (one for each capacity measure; accounting for the nesting of respondents within coalitions) were estimated to determine the proportion of variance due to differences between coalition members and differences between CTC coalitions.

Examining the extent to which the effect of CTC on the community adoption of a science-based approach to prevention depended on coalition capacities required linking community leader reports with those of coalition members. This was only possible by mean-aggregating individual responses of coalition members within each coalition and community leaders within each community. Using meta-regression, the average coalition capacity in each of the 12 CTC coalitions was then used to predict variation between community pairs in the size of the CTC effect on community adoption of a science-based approach to prevention. This effect size was calculated for each of the 12 matched community pairs by subtracting the adoption score in the control community from the adoption score in its matched CTC community (Shapiro et al. 2013b). A Hedges Correction was applied to generate unbiased effect size estimates for each pair (Hedges 1981). Positive effect size values indicate that leaders in intervention communities reported higher levels of adoption of a science-based approach to prevention relative to leaders in their matched control communities.

Because of the matched-pair randomization process, it was possible to examine the impact of coalition capacities, which were only observed in the 12 intervention communities, on the adoption of a science-based approach to prevention by treating the 12 effect sizes as 12 experiments and analyzing them using meta-regression procedures (Monahan et al. 2013; Shapiro et al. 2013b). This involves two steps. Fixed-effect regressions are modeled first, which assumes variation in the effect size between pairs to be completely explainable by variation in the coalition capacity under examination (Lipsey and Wilson 2001). In cases where coalition capacity does not explain all of the variance in the relationship between CTC and adoption, mixed-effect regressions are estimated to account for the residual variance between pairs. Specifically, mixed-effect regressions account for the systematic (modeled) between-pair differences, subject-level sampling error, and an additional random component (perhaps unmeasured or immeasurable components).

Analyses were conducted using the METAREG.SPS macro (Wilson 2005) in SPSS (v.19). In a meta-regression, the sum of squares for the regression model (QR) is analogous to an F-test in traditional regression analysis. A significant QR statistic suggested that coalition capacities were significantly associated with effect size differences in the adoption score. A significant value of the sum of squares of the residual (QE) suggested that coalition capacities did not fully explain all of the variability between community pairs in the impact of CTC on adoption of a science-based approach to prevention. In cases where QR and QE for the fixed-effect model were both significant, the model was re-specified as a mixed-effect model using a maximum likelihood estimate of the variance component to consider sources of variance in addition to the moderator (Raudenbush 1994; Lipsey and Wilson 2001).

Results

Results indicated that CTC coalition capacities were reliably measured and, on average, were highly developed at the member and organizational level (Table 1). The typical coalition member correctly answered four out of five (80 %) of the knowledge questions 1 year after CTC implementation began (substantive knowledge of prevention). Coalition members reported that their skills benefited “a little” to “some” by nature of their involvement with CTC (M = 2.53 on a 4-point scale). The new skill acquired to the greatest extent was “developing and carrying out prevention programs” (M = 2.82, SD = .86). The typical CTC coalition member reported very favorable attitudes toward CTC (M = 3.50 on a 4-point scale). The attitude statement rated least favorably was: “The personal benefits of CTC exceed the personal costs” (M = 3.13; SD = .70), and the item rated most favorably was: “The CTC board is addressing important community concerns” (M = 3.74; SD = .49).

Table 1 Coalition capacities in 12 CTC coalitions: scale descriptions and correlations
Full size table

Organizational linkages to the coalition had more missing data than the other scales based upon the relatively high number of items comprising the scale and the likelihood that all sectors did not exist in all communities. Items related to organizational linkages with the media, recreation offices, and unelected leaders were those most often left blank. The typical coalition member reported that diverse sectors were “somewhat involved” in the work of the coalition (M = 2.94 on a 4-point scale). Social service providers were mentioned most often, by 56 % of coalition members, as being “very involved,” followed by school district administrators (50 %), teachers and school staff (43 %), community volunteers/residents (37 %), law enforcement (34 %), recreation officials (31 %), religious leaders (27 %), unelected leaders (21 %), elected leaders (18 %), parents (15 %), youth (15 %), business leaders (13 %), and media representatives (10 %). This pattern is similar to the involvement trends of various sectors found in the coalition literature, with law enforcement, schools, and social service sectors being highly involved, and the business sector, parents, and youth being less involved (Florin et al. 1993). However, the CTC coalitions in the current study had more linkages with community residents and religious leaders than coalitions in other studies.

Typical coalition members reported that their coalition had “some influence” on diverse community organizations (M = 2.47 on a 4-point scale). Nineteen percent of coalition members reported having “a lot of influence” on school administrators, followed by law enforcement (16 %), youth/health services (15 %), elected leaders (12 %), parents (7 %), business leaders (5 %), and media representatives (4 %). The relationships between coalition capacities are reported in Table 1.

Examination of the variance components of the five coalition capacity measures indicated that variation in member capacities was largely due to differences between coalition members. As indicated by the ICCs (Table 1), 2 % or less of the variance in the knowledge, skills, and attitudes was attributable to differences between coalitions. More variation in organizational capacities (organizational linkages, influence on organizations) was attributed to differences between CTC coalitions. Fourteen percent of the variance in organizational linkages and 5 % of the variance in organizational influence was attributed to between-coalition differences.

Results from the meta-regressions (Table 2) indicated that CTC had a greater impact on increasing the adoption of a science-based prevention approach in community pairs where the CTC coalition had greater member and organizational capacities. Specifically, the difference in the adoption score between control and CTC communities was significantly greater for CTC coalitions whose members experienced greater acquisition of new skills (β = .32; SE = .73) and CTC coalitions whose members perceived greater organizational linkages between the coalition and other sectors in the community (β = .27; SE = .61). Coalition members’ substantive knowledge of prevention, their attitudes toward CTC, and their perceptions of how much the coalition influenced other organizations in the community were not associated with variation in the effect of CTC on the adoption of a science-based prevention approach as reported by community leaders. In the fixed-effect meta-regressions, member-reported new skill acquisition predicted 10 % of the heterogeneity in adoption scores across matched community pairs, and organizational linkages explained 22 % of that variation. Because significant residual variance remained to be explained, as indicated by the Q E statistics, mixed-effects regressions were estimated as well for regressions including new skills acquisition and organizational linkages as predictors. When estimating residual random variation between coalitions in the adoption score in the mixed-effect model, new skill acquisition by coalition members was not a significant predictor of variation in the effect of CTC on adoption (Table 2), but organizational linkages continued to significantly predict the relationship between CTC and leader reports of adoption of a science-based approach to prevention (β = .50; SE = .94). In the mixed-effect regression, organizational linkages explained 25 % of the variance in effect size differences across matched community pairs. The residual variation was not significant, suggesting that organizational linkages fully explained the heterogeneity across matched pairs in community leaders’ reports of adoption of a science-based approach to prevention in their communities. We discuss alternative interpretations of this finding below.

Table 2 Coalition capacities predicting effect size differences between CTC and control community leaders in the community-wide adoption of a science-based approach to prevention
Full size table

Discussion

CTC has been shown in a community-randomized trial of 12 matched community pairs to increase the community-wide adoption of a science-based prevention approach (as reported by community leaders; Brown et al. 2007). This paper examined the extent to which CTC had a greater impact on increasing the adoption of a science-based prevention approach in community pairs where the CTC coalition had greater member and organizational capacities. We showed that five types of coalition capacities (member substantive knowledge of prevention, member skill acquisition, member attitudes toward CTC, organizational linkages to the coalition, and influence on organizations) could be reliably measured using coalition member reports and that the 12 CTC coalitions participating in the trial varied only in some types of capacities. Member capacities were high and did not differ substantially among CTC coalitions in the context of this well-resourced efficacy trial. The strength of CTC coalition member capacities and attitudes may indicate the efficacy and acceptability of CTC training (Feinberg et al. 2002).

However, CTC coalitions varied somewhat in the degree to which their members perceived there to be linkages with other community organizations and their work having an influence on community organizations across diverse sectors. Evaluating whether this variation was due to a particular sector or set of sectors would be useful for recognizing accomplishments and facilitating improvement in linkages with, and influence on, diverse sectors. For example, previous studies have documented that the role of the business sector in coalitions is often different from that of other sectors; the business sector may contribute resources rather than integrate new ideas into their own work (Chervin et al. 2005).

The small between-coalition variance in most coalition capacities measured in this study makes it difficult to draw conclusions about the role of these coalition capacities in advancing the community-wide adoption of a science-based approach to prevention. However, CTC coalitions with greater organizational linkages, and to a lesser extent, coalitions whose members acquired more new skills were more successful in achieving community-wide adoption of a scientific prevention approach. A larger sample of coalitions with greater variability in coalition capacities would be needed to provide greater confidence about the role of different types of capacities in achieving community impact. It may also be useful to consider additional dimensions of member knowledge and attitudes, and the coalitions’ influence on community organizations. For example, process knowledge of how to complete tasks related to prevention science, such as knowledge of prevention program information sources, standards of evidence, methods for assuring program fidelity, and program evaluation techniques, may be an additional important capacity (Crowley et al. 2012). Technical knowledge may need to be complimented by local knowledge that can be used to make strategic decisions to achieve community change (Payne et al. 2009; Wells et al. 2009). Attitudes toward CTC specifically may be less important than beliefs about prevention in general for changing community prevention practices. Studies have found that coalition member beliefs about prevention affected the quality of prevention program delivery, one important tenant of science-based prevention (Spoth et al. 2007). Unfortunately, data are not available in the CYDS to assess coalition member beliefs about prevention generally. In addition, the theoretical concept of spreading technologies was operationalized in this study as member perceptions of influence on other organizations. Influence could alternatively be measured in a way that accounts for the different types of prevention practices that each CTC coalition chose to implement and the corresponding community sector in which influence was most expected. In other words, if a CTC coalition chose to implement a new school-based curriculum, coalition influence might be detected in schools, but might not be as readily observable in other sectors. It may be important to consider the CTC coalition’s influence on “targeted sectors,” which may be different in each community, but more closely reflects the CTC process and, thus, may do better in predicting community adoption of a science-based approach to prevention. Finally, although capacities were measured in this study for the first time after a year of capacity building, it could also be of interest in future studies to measure coalition capacity at baseline to determine what capacities existed in communities before external inputs were applied. This could help determine whether it is the absolute level or the growth in capacities that leads to the adoption of science-based prevention.

Replicating the findings of the present study in evaluations of coalitions with more between-coalition variance may also help facilitate interpretation of the different meta-regression results. Compared to the fixed-effect regression model, the mixed-effect regression did not detect member skills as a significant predictor of variation in the effect of CTC on adoption. This may be because mixed-effect models tend to produce regression coefficients with larger confidence intervals than fixed-effect models, thus providing a more conservative criterion for determining significance. However, there is some question as to whether a mixed-effect model has sufficient power to detect the moderation of an effect across 12 comparisons. The procedure is not standard for analyses with less than ten units (Berkey et al. 1995). Although the meta-regression results for the effect of new skill acquisition are inconclusive, it is plausible that coalition member skill acquisition in communication, political change, and prevention programs would moderate the adoption of a science-based approach to prevention, and the fixed-effect model provides some supporting evidence for this claim.

Although the findings of this study contribute to our understanding of the coalition-level moderators of CTC efficacy in important ways, there are also limitations. The data used for this study were drawn from a community-randomzied trial of CTC, which was designed to test the effect of the CTC strategy in small towns in order to implement and observe the impact of CTC without the risk of contamination across intervention conditions (Yin and Kaftarian 1997). There are, therefore, limits to the generalizability of these findings to other types of communities in the United States, to communities outside the United States, and to coalitions that do not use the CTC model. Furthermore, previous studies have found differences in the perceptions of coalition members based on gender and sectorial affiliation (Riggs et al. 2002), but these individual differences could not be accounted for in this study since such demographic data was not systematically collected from respondents. In addition, the measures developed to assess coalition capacity in this study should be subjected to future tests of construct and criterion validity, and whether unmeasured aspects of the concept, or other constructs, add clarity to the interpretation of these results.

Aside from these limitations, the CYDS provides a data set for examining coalition-based approaches that overcomes many common methodological limitations encountered in the study of coalitions. This study permitted the testing of hypotheses of coalition capacity building by asking CTC coalition members about their capacities, and by assessing the relationship between coalition members’ reports of their coalition’s capacities and CTC community leaders’ reports of the community’s adoption of a science-based approach to prevention, compared to the responses of community leaders in matched and randomly assigned control communities. The high response rates from 24 communities provided high-quality data to examine the association between CTC coalition capacities and community-wide adoption of a scientific approach to prevention. This study found some evidence for the importance of the acquisition of new skills by coalition members and considerable evidence for the engagement of diverse sectors with the coalition process for achieving community-wide adoption of a science-based prevention approach. CTC coalitions, and perhaps all community coalitions intending to lower community rates of youth mental, emotional, and behavioral problems, should focus on building broad community linkages as part of their work. Future research should attempt to understand why greater community linkages were achieved by some CTC community coalitions than were achieved by others. It could be, for example, that coalition structure and communication skills as well as coalition functioning and its leadership may play an important role (Foster-Fishman et al. 2001). Results of such studies could help build coalition capacities, thus increasing the efficacy of coalitions intended to influence rates of youth mental, emotional, and behavior problems through the adoption of a science-based approach to prevention.