Introduction

Over the past decade, a community systems model has garnered increasing attention in the substance abuse prevention field, arguing that alcohol and drug problems result from the complex interaction between the individual and a dynamic community system (Holder 2002). In this model, environmental strategies (ES), such as media, policy and enforcement, target the community context that encourages and sustains substance abuse by attempting to reduce risk factors such as easy access to substances and community norms that promote the social acceptability of use (Aguirre-Molina and Gorman 1996; Holder 2000; Pentz 2000, 2003). Evidence has been accumulating that community-based interventions utilizing a combination of ES can be effective in decreasing rates of substance use, access, and related problems (Dent et al. 2005; Friend and Levy 2002; Holder et al. 1997, 2000; Wagenaar et al. 2000).

The encouraging empirical evidence has led to substantial investments in initiatives that emphasize the use of ES. For example, the Strategic Prevention Framework (SPF) is a major prevention innovation launched by the Center for Substance Abuse Prevention (CSAP) of the Substance Abuse and Mental Health Services Administration (SAMHSA). The SPF offers a structured, sequential, data-driven approach that explicitly targets environmental conditions in the community and aims for change in substance use and problems at the population level. As of August 2011, a total of 49 states, 9 jurisdictions, and 20 tribes received SFP contracts, distributing funding to over five-hundred communities. In addition, the Office of National Drug Control Policy adopted the SPF planning model for its Drug Free Communities program and requires community coalitions, now numbering over 700, to use award money exclusively to fund ES.

In the case of community-based ES, the organizational platform through which they have been implemented has primarily been coalitions (Butterfoss et al. 1993; CADCA’s National Coalition Institute 2008; Wandersman and Florin 2003). Coalitions offer a mechanism through which to bring together diverse community stakeholders in order to implement comprehensive evidence-based prevention approaches aimed at shifting environmental contexts (Wandersman and Goodman 1993; Wandersman et al. 1997). Despite their promise, however, merely providing monies to coalitions to implement prevention activities is not necessarily sufficient to lead to substantial declines in consumption and associated negative outcomes associated with substance use (Berkowitz 2001; Hallfors et al. 2002; Stevenson and Mitchell 2003). While environmental strategy utilization has become widespread, and compelling evidence for their impact is accumulating, basic questions remain concerning the conceptualization and measurement of variables that represent the mediating mechanisms whereby strategies produce their impacts on public health outcomes (Roussos and Fawcett 2000). What factors, in what sequence, can influence effective implementation of ES?

Wandersman et al. (2008) suggest that capacity is a central construct associated with the effective implementation and dissemination of evidence-based prevention interventions by community coalitions. Two types of capacity, general and innovation-specific, may be relevant to consider when implementing prevention interventions (Flaspohler et al. 2008; Wandersman et al. 2008). General capacity is defined as both individual-level skills and characteristics, and overall organizational or community-level functioning required to implement any prevention intervention or strategy. Innovation-specific capacity refers to the specific individual skills or competencies, as well as other the human, technical, and financial resources needed to successfully implement a particular innovation (Flaspohler et al. 2008).

Previous literature has described key elements or constructs related to the successful organization and functioning of coalitions (Butterfoss et al. 1993; Florin et al. 1993). General coalition capacity associated with greater success influencing substance use outcomes include diverse community sector representation, defined and manageable goals, greater formalization and task structure, and utilization of appropriate data to assess community needs as well as monitor implementation of prevention strategies (Florin et al. 2000; Hallfors et al. 2002; Mitchell et al. 2004). In contrast, the empirical literature offers little guidance regarding what specific capacities may be related to successful implementation of ES. The measurement investigation we describe in this paper complements earlier work on ES effort and outputs (Nargiso 2007; Stein 2007), with the theoretical and practical questions associated with the best way to enhance environmental change outcomes, potentially distinguishing the roles of general coalition capacity and innovation-specific capacity associated with ES implementation.

For the current study, the following five general coalition capacity constructs were examined: (1) Mobilization: capacity to mobilize an active and general based membership; (2) Structure: capacity to establish an organizational structure with clear rules and procedure; (3) Leadership: capacity for leadership that promotes action and structures tasks; (4) Cohesion: capacity to create collaboration and shared responsibility within the coalition; (5) Planning and implementation: capacity to establish priorities and implement tasks and timelines. These infrastructure components, leadership skills and organizational processes represented by coalitional capacities are by definition, general, and do not focus on a specific innovation.

We measured general capacities in two ways, representing two key informant perspectives on coalition functioning. The SPF Cross-Site evaluation team, as part of the National Evaluation of the SPF communities, tapped the coalition leader perspective with a Community-Level Instrument (CLI) (Substance Abuse and Mental Health Services Administration 2006) which includes quantitative items regarding coalition capacity. The second measure of general capacity drew on the perspectives of expert observers of the coalitions using a rubric designed for the purpose.

Innovation-specific capacity in this context refers to the capacity required to implement media, policy, or enforcement strategies. No capacity measures specific to ES were included as part of the CLI, but the evaluation team created a second measurement rubric to be applied by the expert observers, in order to quantify the capacities thought to be directly related to ES implementation.

Training and Technical Assistance

A primary mechanism through which to build and sustain both general and innovation-specific capacities for implementing evidence-based prevention strategies has been training and technical assistance (TTA) (Florin et al. 1993; Mitchell et al. 2002). TTA has been defined as a variety of activities designed to improve the capacity of groups or organizations (e.g., community coalitions) to implement strategies and enhance outcomes. TTA can be provided through the provision of products or resources (e.g. workbooks, assessment tools), individual consultation and group trainings or workshops. Additionally, TTA can focus both on building organizational capacity and competencies, as well as on the dissemination of evidence-based practices and programs (Mitchell et al. 2002). Lack of adequate TTA has been suggested as a potential reason for the lack of findings from coalition-led health initiatives (Feinberg et al. 2008; Mitchell et al. 2002). Evidence suggests that sufficient dosage of high-quality proactive TTA is needed to build general coalition capacity (Chavis 1995; Feinberg et al. 2002; Spoth et al. 2004). Feinberg and colleagues (2002) found that technical assistance was required even for coalitions with a high-level of functioning in order for these organizations to accomplish complex tasks, such as conducting a needs assessment or selecting or implementing prevention strategies. A study which randomized 24 communities to receive either TTA, training alone, or a control condition that received neither training or technical assistance, found that both groups of intervention communities demonstrated better prevention plans, plan implementation and functioning at meetings compared to the control communities one and a half years later (Riggs et al. 2008). Mitchell et al. (2004) suggested that initial coalition capacity building may be needed for coalitions to understand their own strengths and weaknesses in order to utilize TTA effectively. Understanding the role of TTA in building general and innovation-specific capacities is necessary to facilitate the effective implementation of ES by community coalitions.

This research used multiple key informant perspectives to examine general as well as innovation-specific capacities and their associations with ES-related implementation efforts and outputs. Efforts refer to the amount of energy exerted to implement an ES while outputs refer to the materials produced through these efforts. For example, coalitions can devote extensive energy to local or state policy change work, but these efforts may or may not be translated into policy output (e.g., successful policy change). In addition, this research explored the role of TTA in building coalition capacity as well as its linkages with ES implementation efforts and outputs. Specifically, the current study uses newly-developed measures to answer three primary questions: (1) Are greater general and/or innovation-specific capacities associated with more ES implementation efforts and outputs produced by each community coalition? (2) Is general coalitional capacity at the start of the project related to utilization of TTA resources over the course of project? More specifically, do coalitions with poor initial capacity seek out appropriate TTA resources? and (3) Is a greater amount of TTA resources utilized during the grant associated with ES implementation efforts or outputs?

Methods

Procedure

Fourteen communities in Rhode Island with high rates of alcohol and other drug use received CSAP’s SPF SIG funding. These communities were tasked with implementing a 5-step strategic planning process, which includes data-based decision-making in order to select and implement a comprehensive set of ES to address the specific prevention needs of their communities (Florin et al. in press). A community coalition from each funded community was the “platform” through which the planning process and ES implementation was delivered in each community. In addition, each community had access to specialized TTA throughout the three-year grant to assist with each step of the SPF. Coalition capacity and TTA were measured throughout the project.

Measures

Two different methods were employed to assess general and innovation-specific capacity.

General Capacity Measures

Coalition-Rated General Capacity

General coalition capacities were assessed through two sets of key informants. First, a SPF designee from each of the 14 coalitions, typically the SPF coordinator responsible for the oversight and management of the grant within each community, completed the CLI, a two-part self-report evaluation measure developed by the SPF-SIG Cross-Site Evaluation Team (Substance Abuse and Mental Health Services Administration 2006). The CLI was completed every 6 months during the three-year project; Part 1 of the instrument included process items assessing each community’s progress through the five steps of the SPF, including items assessing general coalitional capacities. Ten CLI items were selected to create five general coalitional capacities. Table 1 summarizes the items and scoring methodology used for each general capacity construct. Some items were reverse scored (as indicated on Table 1) so that higher scores indicate greater capacity. A standardized composite score for each construct was then computed based on the average score of these two items and then converted into a z score. In addition to scores on each of the five capacity scales, a composite score was calculated to represent an overall Coalition-rated General Coalition Capacity for each community. This was done by standardizing each of the five scales and then averaging across the standard scores.

Table 1 Five general coalition capacities
Full size table
Expert-Rated General Capacity

A three point rating scale for general capacity was developed for this project (see appendix for complete rating rubric). The General Capacity Scale focused on the general capacities needed to function effectively as a coalition, including the following areas: coalition leadership, membership and staff turnover rates, quality of meetings, level of visibility within the community, as well as technological capacity for general coalition tasks. The 3-point scale categories were (1) Low functioning: Not likely to be effective or sustainable, (2) Moderate Functioning: More likely to be effective than not, and (3) High Functioning: Strong and sustainable. The scale was formatted as a holistic rubric for categorical classification of the 14 coalitions.

State-level prevention professionals involved with the SPF who had extensive contact with all of the coalitions provided this second source of ratings for general capacity. Four experts were identified who had reviewed the strategic plans of the 14 communities, attended coalition meetings, and provided both group and individual TTA sessions. Experts included the state’s program manager of the SPF-SIG as well as individuals who provided TTA to each of the communities. These individuals were asked to provide a general capacity rating for each coalition and to consider the overall capacity built over the course of the project. These experts, in two subgroups, generated independent retrospective ratings for each community. The bivariate Pearson correlation between the two subgroups across the 14 communities was .79, demonstrating respectable inter-rater reliability. The average of the two subgroup scores was used to generate an Expert-Rated General Coalition Capacity score for each community.

Innovation-Specific Capacity Measure

The CLI did not include a corresponding set of quantitative items appropriate to measure innovation-specific capacity, which in this case refers to ES-specific capacity, so we were not able to quantify coalition leaders’ perspectives on this dimension. We did, however, develop an expert rating scale to measure capacity directly related to the implementation of ES. Criteria for this rubric included: understanding and/or expertise in use of media or policy, development of linkages with key stakeholders (e.g., local media, legislative allies, retail vendors), knowledge of local decision-making processes regarding relevant policy, membership support of the use of ES, and quality of strategic plan and logic model for ES. Rating classifications included: (1) Low Capacity: Not likely to progress far toward implementing any ES beyond enforcement, (2) Moderate Capacity: More likely than not to implement at least some ES, and (3) High Capacity: Highly likely to successfully implement specific ES. Our two groups of expert raters used these categories to retrospectively rate each community on environmental-specific capacity. The correlation between the two sets of expert ratings was .67, indicating acceptable, though modest, inter-rater reliability. To calculate Expert-Rated Environmental-Specific Capacity for each community we averaged the two scores.

Training and Technical Assistance (TTA) Received

The TTA Center provided a series of formalized proactive TTA on the 5 steps of the SPF, as well as the implementation of specific ES. These TTA sessions were offered to all of the funded communities. Attendance logs from each of these sessions were used to calculate the “dose” of the TTA received by each community, which is the number of TTA sessions attended.

ES Implementation Efforts and Outputs

Three broad categories of environmental strategy include: media, policy change, and enforcement. A Monthly ES Tracking Interview—a semi-structured interview developed for this evaluation project (Florin et al. in press)—was utilized to assess the level of effort each community devoted to these three types of ES. As for the CLI, designated coalition informants were the source for these monthly data.

Effort was operationalized as number of hours devoted to media strategies (e.g., media advocacy, social marketing), hours of policy change work, and hours of enforcement devoted to enforcing laws or policies associated with restricting underage drinking or illicit drug use. The total number of hours reported for each of these areas was tallied and then a monthly average was computed to represent the average level of effort devoted to that strategy type over the 36 months of the grant leading to three separate effort ratings—media, policy and enforcement.

Outputs were specific to the area of environmental strategy implemented. The number of articles, advertisements (e.g., radio, TV, newspaper), letters to the editor, and other pieces of produced media all constituted outputs that were tallied during the 36 months of the grant. The average number of distinct pieces of media per month was utilized as the amount of media output for each community. The total number of policy changes reported during the entire project represented the policy output for each community. Finally, the average number of arrests reported per month throughout the course of the grant resulting from any enforcement strategy implemented was used to reflect enforcement output.

Data Analysis Plan

Bivariate correlations were conducted to examine relationships between coalition-rated and expert-rated capacities, ES efforts and outputs, as well as TTA utilization. Capacity ratings from the first CLI assessment time point (Time 1 covered the first 6 months of the project) were used to examine correlations between General Capacity and TTA utilization. Capacity ratings from the final CLI assessment time point (Time 6 covered the last 6 months of project) were used to examine correlations between General Capacity and ES efforts and outputs. With directional hypotheses for each of our tested relationships, we used one-tailed tests. To account for the small sample size, trends of p < .10 will be reported in addition to significant findings of p < .05.

Results

Table 2 provides a summary of the means and standard deviations of capacity measurement scores, total TTA dosage for each community, as well as the average monthly ES efforts and outputs. There was a wide range of coalition efforts devoted to ES implementation, as well as outputs produced in the areas of policy, enforcement and media across the 14 communities. The largest variability was in policy change efforts, where one community devoted an average of 1.33 h per month on policy change work while another community devoted a high of 92.90 h per month on this type of work.

Table 2 Summary of capacity measures, TTA and ES efforts and outputs
Full size table

Coalition-rated general capacity at the end of the project, as measured by a composite average of the 5 constructs, was highly correlated with the single item expert-rated general capacity (r = .71). There was a high correlation between expert-rated general and innovation-specific capacity (r = .80).

Associations Between Capacity and ES Efforts and Outputs

Table 3 displays correlations of coalition-rated general capacities with ES efforts and outputs. All 5 general capacity measures were positively correlated with a greater number of hours dedicated to local or state policy change efforts, with Task Focus and Implementation Planning capacities most strongly related (p < .05). Coalition-rated general capacities were not however, associated with media or enforcement efforts nor with any outputs.

Table 3 Correlations between CLI-based general capacity scores and ES efforts and outputs
Full size table

Table 4 shows the correlations between expert-rated capacity and ES implementation efforts and outputs. Expert ratings of general capacity were associated with enforcement output. That is, coalitions with greater expert-rated general capacity had a greater average number of arrests in their communities. Expert-rated innovation-specific capacity (e.g., specific capacity to implement ES) was positively correlated with both greater number of arrests produced by enforcement and greater number of media products produced monthly during the project.

Table 4 Correlations of expert capacity ratings (averaged scores) with ES efforts and outputs
Full size table

Antecedents of Training and Technical Assistance

Study results support a significant negative relationship between three types of general coalitional capacity (rated early in the life of the project) and TTA utilization. Table 3 includes the correlations between CLI-measured general capacity and dosage of TTA. Communities characterized by lower initial capacity to mobilize around key issues of alcohol or illicit drug use, as well as lower coalition structure and task leadership were associated with higher subsequent utilization of TTA. Neither expert-rated general nor expert-rated innovation-specific capacity was correlated with dosage of TTA utilized (See Table 4).

Training and Technical Assistance Dose as a Predictor of Subsequent Coalition Action

To address the second set of TTA-related questions, utilization of TTA was also examined in relation to the amount of ES implementation efforts and outputs produced over the course of the project. Results are summarized in Table 5. The only significant relationship that emerged was that communities that attended more TTA sessions reported significantly more policy change output than communities that reported less utilization of TTA resources.

Table 5 Correlation between TTA utilization and ES efforts and outputs
Full size table

Discussion

Coalitions have unique attributes that have contributed to their prominent role in environmental strategy implementation. The current study seeks to improve our understanding of how general and innovation-specific coalitional capacities might facilitate the implementation of environmental prevention strategies.

One contribution of the paper is to introduce into the literature on evaluating ES simple but useful measures of capacity, TTA and ES efforts and outputs. This paper utilizes quantitative measures of capacity derived from the Community Level Instrument (CLI) of the National Cross-Site Evaluation of the SPF, and introduces quantitative holistic rubrics for separately measuring general and innovation-specific capacities through expert-raters. The expert-rated general capacity score demonstrated a high degree of correlation (r = .71) with the composite score of the 5 general capacity measures, thereby supporting the construct-validity of both as measures of general organizational capacity. The strong correlation between expert-rated general and innovation-specific capacities (r = .80) may be partly due to shared method variance but is also consistent with our conceptualization of coalition capacity.

Additionally, the paper contributes to the sparse literature examining the necessary capacity needed to ensure appropriate implementation of environmental prevention strategies in the areas of enforcement, policy and media, as well as how TTA can facilitate the development of these capacities. First, capacity developed throughout the project was associated with some ES efforts and outputs produced by the 14 communities. Greater general capacity as rated by the coalition designee near the end of the project was associated with greater amount of policy efforts, meaning more hours dedicated to either local or state policy change. While post hoc expert ratings of general capacity did not support the link between greater capacity and level of effort dedicated to policy change, there were significant associations with other ES outputs, specifically enforcement output (number of arrests) and media outputs (number of media products produced). We were not able to test the relationship between outputs and reductions in community-level consumption or consequence patterns; however, theoretically it holds that greater ES outputs would be associated with declines in these areas. In addition, although policy outputs were not related to expert ratings of general or innovation-specific capacity, it is important to note that in order to effect policy change, multiple political and social factors outside the purview of coalitional capacity are also implicated. Therefore, while capacity is likely an important factor to consider for community coalitions working toward policy change, the circumstances leading to success in creating local or state policy change may be more capricious or variable than other types of ES. In addition, the length of time needed to get successful legislation passed may often be outside the scope of a typical grant period and therefore longer follow-up time may be needed to capture the full effect of coalition efforts to implement policy change.

This paper also sought to address whether weaker coalitions sought more TTA services in order to improve capacity to implement evidence-based prevention strategies, or whether coalitions that demonstrated higher initial capacity utilized more TTA. Consistent with earlier work (Mitchell et al. 2004), findings from this study are mixed regarding the role of the utilization of TTA to increase general capacity. More specifically, current results support the view that coalitions with weaker general capacity in the areas of mobilization, structure and task leadership, as rated in advance by coalition leader informants, subsequently sought more group TTA services. However, there was no difference in TTA utilization based on the initial level of coalition cohesion or capacity to implement and plan intervention strategies. Moreover, expert-rated general coalition capacity was not associated with utilization of more or less TTA. It is also important to note that the expert ratings were obtained at least a year after TTA had occurred, and therefore do not provide the same prospective frame that the Time 1 CLI general capacity ratings do. It is possible that the initial differences seen by coalition informants were cancelled out by the capacity-building effects of the larger doses of TTA. In one of the few studies analyzing the link between TTA and capacity outcomes quantitatively, Feinberg (2008) found limited support for a relationship between TTA and coalition functioning; however, this effect was stronger for coalitions that were less established than for those that had higher initial level of functioning, suggesting that higher functioning coalitions benefit less from the same amount of TA compared to lower functioning coalitions.

Finally, regarding the relationship between TTA and ES efforts and outputs, greater utilization of group TTA resources was associated with greater number of policy changes (r = .59, p = .02). Greater TTA may be needed for successful policy change since the tasks involved in understanding and implementing policy change work are varied and complex and may require greater amounts of capacity to implement effectively (Dievler 1998; Pentz et al. 1989). This may be particularly true when compared to media and enforcement strategies which require less collaboration or multisector effort than policy strategies. Policy change is also unique in that it is not easily sub-contracted to another community agency or individual, since it requires general support by members and relies more heavily on network and/or connections compared to other ES (e.g., enforcement strategies are contracted to local police departments). Therefore communities that wanted to do more policy change work may have sought out more TTA in order to build their capacity in this area.

This finding regarding TTA is particularly notable, in light of the significant association between general capacity and policy efforts, since coalitions offer a unique platform through which to work for policy changes. Moreover, once coalitions develop capacity around effecting policy changes, this type of ES is likely more sustainable compared to media or enforcement which require more significant financial resources to implement effectively. Stevenson and Mitchell (2003) suggest that the strongest evidence of success in substance abuse prevention through community collaborations, such as coalitions, is in the area of policy change. The study findings support the need to build both general and innovation-specific coalition capacity related to policy change efforts.

Limitations

The study provides results from a small sample of communities and therefore must be interpreted with caution. One-tailed correlations were conducted and a cut-off p value of .10 was utilized to report positive trends. While the study offers interesting preliminary findings, they need to be further examined with larger samples of communities. Additionally, since no control communities were funded, comparisons of differences in capacity, TTA utilization, or ES implementation efforts and outputs within non-intervention communities could not be made. Therefore it is not possible to rule out other intervening variables that may account for the correlations between these variables. Also, while TTA was measured equivalently across communities, we did not investigate whether communities obtained other sources of training or assistance which may have influenced their capacity to implement ES. Additionally, while significant correlations are notable considering the small sample, the risk of Type 1 error is potentially increased with a p value set at p < .10 and the use of one-tailed tests with multiple correlations.

Our use of experts as one source to measure coalition capacities has much to recommend it, but the retrospective nature of these ratings gives rise to a concern for a method artifact—the raters may have been aware of coalition efforts and outputs when they made their ratings. However, efforts were made in the expert-rating process to consider the totality of the general and innovation-specific capacity, including resources, personnel, community networks and relationships to implement ES regardless of what the degree of actual ES implementation. Two of the expert raters noted that for several communities the general and ES-specific capacities exceeded actual level of implementation efforts.

Community-level data on substance use outcomes (e.g. consumption patterns) were not available to explore the essential next link, the relationship between increased ES implementation efforts and outputs and reduction in actual risk of substance use (e.g., change in community-level risk factors, substance use prevalence rates). However, our work does fill in important steps leading up to that one.

Future Directions/Implications

The current study offers preliminary findings regarding the relationship between community coalition capacity and ES efforts and outputs, along with an exploration of the role of TTA in this context. In order to examine these relationships, the paper puts forward several methods of measuring general as well as innovation-specific coalition capacities that appear promising. The measure of innovation-specific capacities related to ES introduced in the paper crystallizes some of the key competencies related to the effective implementation of ES (e.g., leader and members have sophisticated understanding of the local decision-making process for particular policies, well developed relationship with local media). Future studies should extend and refine measures of innovation-specific capacity related to ES implementation in order to understand how best to assess key areas of capacity associated with ES.

The next step in linking TTA and coalition capacity to community health outcomes would be to examine these in a large enough sample to test a mediational model to determine whether specific types of capacity are needed to influence particular ES outputs that may in turn have an impact on community-level indicators of substance use consumption and consequence patterns. For example, if communities with low innovation-specific capacity are able to develop adequate capacity via focused TTA, will this in turn lead to more access-restricting policies and enforcement of underage drinking laws, and does that ultimately translate to less underage alcohol consumption in a community? These represent important questions related to improving the prevention systems (e.g., TTA provision system, coalitions) in order to see larger impact of prevention activities on community-level substance use rates. The present study does provide measures for important links in that chain, and some preliminary evidence regarding how improved understanding of coalition capacity can aid in making the desired health outcomes more likely in future coalition-based use of ES.