The Longitudinal Effect of Technical Assistance Dosage

Prevention scientists, having made significant progress in developing efficacious preventive programs capable of reducing the public health burden of depression, antisocial behavior, and substance use, are now challenged to achieve broad dissemination while maintaining fidelity of core components (Greenberg et al. 2000; O’Donnell et al. 2000). The provision of ongoing technical assistance is assumed to be an important means of facilitating program fidelity (e.g. Davis 1998; Durlak 1998; Kelly et al. 2000). However, little quantitative research to date has examined whether technical assistance (TA) does in fact improve local prevention systems and programs, and the most effective method of delivering TA (Florin et al. 1993; Mitchell et al. 2003; Stevenson and Mitchell 2003). This report focuses on the question of whether TA provided to Communities That Care community-based prevention coalitions has a positive influence on the functioning of those coalitions, and whether the following factors are important to consider in developing TA systems: on-site vs. off-site TA, developmental phase of the coalition, and level of coalition functioning.

Community coalition and partnership approaches to addressing public health problems have intuitive appeal (Backer 2003; Wandersman and Florin 2003). Coalitions can facilitate flexible, multi-sectorial planning and coordination to address problems that arise through complex interactions between multiple levels of communities (e.g. individual behaviors, family relations, neighborhood culture, school quality, economic stress; Bronfenbrenner 1986). The use of such models has grown over the last three decades since the initial cardio-vascular disease prevention trials in the 1970s (Mittelmark et al. 1986; Puska et al. 1983). The federal government and foundations have promoted the community coalition strategy to address problems such as substance use, obesity, crime and violence, teen pregnancy, cancer, etc. In the field of substance use alone, there are an estimated 5,000 community coalitions in the U.S.

However, much prior research on the outcomes of coalition or partnership approaches has generally produced null results (COMMIT Research Project 1995; Hallfors et al. 2002; Roussos and Fawcett 2000; Saxe et al. 1997). These results have led some to question the value of the partnership approach in promoting public health activities (Klerman et al. 2005).

One of several possible reasons for the frequent failure of the coalition strategy to demonstrate public health benefits has been a neglect of TA. To function well, local coalitions supporting the implementation of evidence-based prevention programs require an adequate dosage of high-quality and pro-active training and TA support (Chavis 1995; Feinberg et al. 2002; O’Donnell et al. 2000; Spoth et al. 2004).

Two recent studies have shown the potential effectiveness of specific models of community partnership; in both cases, ongoing TA has been provided to the sites: We have recently found evidence of positive family and youth outcomes as a result of preventive interventions implemented within a community partnership model which utilized evidence-based youth and family interventions and featured a structured and intensive TA support mechanism (Spoth et al. 2007). This university-community partnership model (PROSPER), employing the land-grant university Cooperative Extension service as the partnership catalyst and provider of TA, is one of the few partnership or coalition models to evidence positive outcome effects at the level of public health impact in a randomized trial.

The second study is an evaluation of the outcomes of CTC in Pennsylvania (Feinberg et al. 2005), which is also the context for the data presented in the current paper. In a quasi-experimental study, we compared youth-reported risk factors and outcomes in school districts associated with vs. not associated with CTC sites. We found a pattern of significant differences favoring the CTC-associated school districts; a stronger pattern of evidence was found when grade-cohorts that had not received CTC-sponsored, evidence-based programs were eliminated.

The need for TA within a coalition framework should not be underestimated (see Kreuter et al. 2000). Our field experience with local prevention boards in the Communities That Care model (Feinberg et al. 2002) suggests that even when local prevention boards were operating relatively effectively, TA was still needed in the management of complex tasks such as assessment, program selection, implementation, achieving sustainability, board development and cohesion. Moreover, providing TA on specific issues such as board recruitment, new board orientation, and running effective meetings is likely to affect attainment of these broader goals. A number of researchers and evaluators have come to similar conclusions in qualitative and case study investigations of other community-based initiatives (Davis 1998; Florin et al. 1993; Goodman et al. 1993; Roberts and Wasik 1996). For example, some evidence suggests that TA has helped improve outcomes in a Centers for Disease Control-sponsored community-based AIDS network (Yin et al. 1999). Kegler et al. 1998, b) found that high quality technical support from state-level staff was an important factor supporting implementation of community tobacco coalitions. More generally, TA seems to facilitate positive change in practices in medicine, education, and community services (Kelly et al. 2000; Reifler et al. 1999; van Houten 1996).

However, there has been little quantitative research on the effects of TA in either community coalition models or the dissemination of prevention programs generally. Although studies cited above demonstrate the potential of well-organized and supported coalition models that use evidence-based programs, without experimentally varying TA support, it is difficult to isolate the differential effects of varying amounts, types, and models of TA service delivery. Ideally, randomized or quasi-randomized controlled studies would assess implementation fidelity for different types and dosages of TA. Given the high cost and low priority placed on such investigation, longitudinal observational studies of naturally occurring variation in TA may provide some guidance. Few such studies have been reported in the literature. In the only other study we could locate assessing the effect of TA on local prevention coalitions, the investigators found that the dosage of TA was not related to intermediate coalition outcomes (Mitchell et al. 2004).

It may be that the effects of TA dosage on prevention system functioning are non-existent in one model but are substantial in another. TA impact may depend on a number of factors. These relevant factors may include characteristics of the coalition model (e.g. the degree of structure, the use of evidence-based interventions, flexibility, self-determination, or complexity in the coalition’s tasks and processes), characteristics of the TA providers (e.g. education, training, experience), or of the TA system (e.g. supervision of TA providers, method of determining the need and focus of TA).

Communities That Care

Communities That Care (Hawkins and Catalano 1992) engages local leaders and citizens in a prevention coalition whose goal is the reduction of adolescent problem behaviors (e.g. violence, alcohol/substance use, teenage pregnancy, school drop-out, and delinquency) and the promotion of positive youth development. Members of the coalition attend intensive trainings to learn how to bring relevant aspects of prevention science, such as risk and protective-factor focused prevention, to the community. The leaders carry out an assessment of risk factors in the community, and based on a prioritization of risk factors formulate and then implement a plan to address those factors. The CTC model employs evidence-based strategies and programs in a coordinated fashion to reduce community risk.

CTC has been disseminated in communities across the United States and is being utilized in several other countries as well. The Substance Use and Mental Health Service Administration (SAMHSA) now disseminates the CTC materials. Pennsylvania was an early adopter of CTC and over the past decade has developed the most extensive CTC network in the United States. The state has devoted significant funds to providing over 100 communities with training and technical assistance, start-up funds, and modest programmatic funding for an initial three-year implementation period. CTC evolved from one TA provider in 1995 to a regionalized TA system with five full-time TA providers and a supervisor by 2001.

The Current Paper

In this study, we examined whether dosage of TA−measured in terms of contact time between a state-funded technical assistant provider and a local prevention coalition−was related to the functioning of CTC coalitions. Although this was an observational study, we included multiple years of data and conducted longitudinal analyses in order to model the effects of TA over time (see Fig. 1). The model includes paths representing the stability of TA and the stability of coalition functioning across years.

Fig. 1
figure 1

Path diagram of longitudinal model of associations between technical assistance and CTC board functioning. (Note: “Err” = error variance)

Full size image

Because the amount of TA contact time is not proscribed in the CTC model, but instead was decided by both the TA provider and the coalition leadership, it is possible that TA dosage was responsive to the need−and thus functioning level—of the local CTC boards. In fact, in the middle of the time period covered by this study, TA providers completed a needs assessment and created a rudimentary TA plan for each site. To account for responsiveness of TA dosage to level of board functioning, we included cross-paths in our analytic models (Chatfield 2004), estimating the influence of coalition functioning on later TA dosage. Thus the direct paths estimate the effect of TA on coalition functioning controlling for the reciprocal relation (i.e., the cross-paths).

Prevention system developers require guidance in determining what type of TA to provide. The costs associated with travel to geographically dispersed sites can be a major obstacle to providing TA within real-world dissemination models. It is much less costly for a provider to service a larger number of sites through telephone and email contact. However, the benefit of these different delivery processes is unknown. Thus, we examined two types of TA: consultation delivered on-site, through in-person meetings vs. off-site, including phone and email contact. Our analyses aimed to determine whether on- and off-site TA had differential effects.

Moreover, we posit a partnership model in which local community teams proceed through a series of broad developmental phases (Livit and Wandersman 2004; Stevenson and Mitchell 2003; also see Chinman et al. 2004; Florin et al. 2000; Hawkins et al. 2002). It may be that TA is more potent in supporting coalitions during certain periods of coalition development. For example, it is possible that coalitions are most malleable early on in the coalition lifecycle; in one study of non-profit volunteer organizations, younger programs valued TA more than older programs, although the authors did not find any association between use and perceived value of TA and measures of program performance (Leviton et al. 2006). Thus, we examined whether the age of the CTC board moderated the impact of TA.

We also examined whether two characteristics of the local boards moderated TA impact. First, we examined whether the initial level of functioning of the CTC board influenced the impact of TA. It may be that TA is particularly effective where the recipients are functioning at a reasonably high level and thus can better take advantage of the assistance. Alternatively, it may be that there are greater potential gains to focus support on more poorly functioning coalitions as the same amount of TA time may have more impact. Second, we explored the influence of the members’ perceived need for TA. One might expect that coalitions that perceive a high level of need for TA would be willing to make greater use of assistance.

Finally, we recognized that our measure of TA—time spent by the provider with each board—did not capture differing levels of sophistication or skill exhibited by TA providers. Thus we conducted follow-up analyses to examine whether the sites handled by some providers improved to a greater extent than the sites handled by other providers. Similarly, we reasoned that some boards would be more skillful consumers of TA; thus we examined whether boards’ cooperativeness with TA moderated the impact of TA on board functioning.

Methods

Procedure

The data for this study are based on the evaluation of Communities That Care programs funded by the Pennsylvania Commission on Crime and Delinquency (PCCD). Pennsylvania initiated a comprehensive plan to implement CTC projects throughout the state in the mid-1990s, supplementing federal Title V funds. Communities whose applications for the CTC program were successful received a one-year planning grant of about $25,000. The CTC coalitions generally were initiated by one or more interested individuals or agencies involved in local youth development or addressing problem behavior. Activities during this planning year included attendance of key community leaders at three, multi-day, statewide training sessions. The state contracted with the CTC program developers and trainers to provide the CTC training. Developmental Research and Programs initially provided training; it was later provided by the Channing-Bete Company. Currently, support for CTC is provided by the Substance Abuse and Mental Health Administration.

CTC Prevention Board functioning was measured by the use of two methods. The state-funded TA provider for each coalition completed a written measure assessing each CTC board in three consecutive years (2003–2005). In addition, for the same three years, CTC board member and staff self-report data was collected via a web-based evaluation system involving the following process. First, each coalition sent the names and email addresses of coalition members to the administrator of the system. That information was then entered, and email messages were automatically sent to each coalition member with a link to their own personal survey webpage and password, facilitating participants’ sense of security in the confidentiality of the process. The online questionnaire took participants about 20 min (longer for staff who reported on several additional domains).

Since the mid-1990s, approximately 120 communities have joined and were trained in Pennsylvania’s Communities That Care initiative. TA staff reported on the amount of time they spent with 116 sites during the period covered by this study. However, about 20 of those local CTC sites had ceased operations before the period of this study began. The largest number of sites reported on by TA providers in any one year regarding board functioning was 95 (see Table 2).

For the self-report web-based questionnaire, 68 sites participated in 2003; 79 in 2004; and 75 in 2005. In 2003, 1081 board members and staff from the participating sites were invited to participate; 570 participants responded for a response rate of 53 percent. In 2004, 1502 individuals were invited; 867 participants responded for a response rate of 58 percent. In 2005, 1624 were invited; 819 responded yielding a response rate of 50 percent. Our impression from feedback from the sites is that the more active members tended to participate, while board members who are less active, who do not attend meetings, and who are not knowledgeable about their CTC site’s functioning and activities, tend not to participate. Although some sites participated in only one or two years of the web-based questionnaire, all data were included in the maximum-likelihood path analyses.

Measures

Board Member Self-Reported Board Functioning

Board member self-reported board functioning was calculated utilizing scales from the Coalition Web-based Self-report Questionnaire. The scales on this instrument were derived from previous measurement development work in our earlier evaluation of CTC (Feinberg et al. 2004), as well as items and scales adapted from other research projects (Arthur et al. 1998; Kegler et al. 1998a, b; Wandersman et al. 1987). The measures represent topic areas that research and theory have indicated are important features or correlates of coalition functioning (Crowley et al. 2000; Eng and Parker 1994; Oetting et al. 1995). Greater detail on psychometrics and validation of the scales can be found in Feinberg et al. (2008).

Board functioning is a multi-dimensional construct. We created a single self-reported board functioning score by taking the mean of four factor scores. (Factors were developed through exploratory factor analyses guided by theoretical considerations.) Each factor was comprised of one to four subscales, and operationalized as the mean of those subscale scores.

The first factor was Board Work, comprised of the following four scales: Board Directedness, assessing the quality of the board’s goals and community plan, as well as the leadership’s adherence to clear agreements on decision-making processes; Board Efficiency, assessing whether board members worked hard and efficiently made use of work and meeting time; Board Leadership Style, which measured whether leadership provided praise and recognition, sought out members’ views, and approached members to assist with specific tasks; and Board Leadership Competence, which assessed ability to mobilize resources, respect in the community, political knowledge and competence, skillfulness in resolving conflict, and ability to provide a strong leadership. The second factor was Organizational Resources, comprised of two subscales: Board Membership, which assessed turnover and ease of recruiting new members; and Barriers, which assessed the severity of a list of typical barriers to coalition functioning. The third factor was comprised of a single subscale: Staff-board Communication, which assessed frequency and productivity of such communication. The fourth factor was Board Relations, which included two subscales: Board Cohesion assessed social cohesion among board members, and Board Conflict assessed conflict among members.

The response format for 2003 items included 4-point Likert scales (e.g. 1 = strongly disagree, 4 = strongly agree); 5-point Likert scales (e.g. 1 = huge problem; 5 = not a problem); and yes/no response formats. Refinement of the questionnaire for 2004 and 2005 entailed converting all response formats to 7-point scales. In order to provide user-friendly results for CTC board members, all items were recoded to a 10-point Likert scale. Scales were then created as means across items. Descriptive information on the scales can be found in Table 1. The alpha for the composite board functioning score with the four factors considered as items was .93 for 2003 and for 2004, and .94 for 2005.

Table 1 Descriptive statistics for self-report CTC board member scales
Full size table

TA Provider Ratings of Board Functioning

Annual TA provider ratings of site functioning in written questionnaire format were utilized to create a single score for representing board functioning. This global score was composed of five component scales: Board Efficiency measures active and work-oriented participation on the board; Fidelity to CTC Process assessed the degree to which the board’s work reflects the specific components of the CTC model, such as the quality of risk factor assessment, prioritization of risk factors, selection of programs, and evaluation of program fidelity; Board Membership measured the ease of membership recruitment and the extent to which members serve on the board without dropping out prematurely; Board Cohesion assessed the extent to which board members feel a sense of unity and collaborative working style; and, Community Relations, assessing the reciprocal, cooperative relationship between the CTC board and other community actors and organizations.

The items for each scale used a seven-point Likert response format (1 = poor; 7 = excellent). Note that interrater reliability could not be assessed as each TA provider reported on his or her own set of communities, and there was no overlap among the five sets.

Board Functioning Composite Score

For each year, the TA provider report and board member report of board functioning were aggregated to yield a single score of each site’s board functioning. This procedure was supported by moderate correlation of TA report and board member report of board functioning were .47, .39, and .54 (all p-values <.01) for 2003 through 2005, respectively. Descriptive statistics of the composite score are presented in Table 2.

Table 2 Composite board functioning and TA dosage descriptive statistics
Full size table

Board Cooperativeness with TA

In addition to the scales described above, TA providers also assessed Board Cooperativeness with TA with one item, utilizing a 7-point Likert scale (mean = 4.93; SD = 1.87).

TA Dosage

As part of internal reporting requirements, TA providers completed monthly reports on amount of time spent on various categories of support for each site. This information was provided to us for 2003 and 2004. We aggregated the data into two broad domains: on-site TA (i.e. consultation in meetings in the local community) and off-site TA (time spent on the phone and in correspondence with staff and members for the site). Descriptive statistics regarding the average minutes per month of on-site and off-site TA dosage per site are presented in Table 2.

Need for TA

TA providers assessed the priority of sites for TA based on three levels: high need, moderate need, low need. We contrasted the sites with a high need for TA (n = 19) with the other sites who had a moderate or low need, based on the assumption (backed by preliminary analyses) that the sites with a moderate degree of need were more similar to sites with a low level of need.

Age of Site

At the time of the first web-based questionnaire data collection in March 2003, the sites ranged in longevity from 8 months to 71 months (i.e. 5 years, 11 months) based upon the start date of their initial program implementation grant. However, most prevention boards had been engaged in the training and planning process one year (and sometimes up to two years) before the start of the implementation grant. A median split was performed on the sites by age, resulting in a “younger group” whose duration since the implementation start date ranged from 8 to 23 months, and an “older group” whose duration ranged from 30 to 71 months.

Board Cooperation with TA

TA providers completed this four-item scale regarding the board’s seeking out of assistance and cooperation with the TA provider. Data for 2003 were used in this report. The four items had a 7-point Likert response scale; the alpha was .76.

Analyses

To test for moderator effects, analyses were conducted separately by type of TA: on-site dosage (e.g., traveling to board location to assist or providing formal training) and off-site dosage (i.e., e-mail and phone communications). Moderator hypotheses were tested using three nested models, which were variants of Fig. 1. A median split was used to create two groups, one high and the other low on the moderator. In the first model, all paths were constrained to be equal between the two groups. In the second model, paths modeling the directional and reciprocal influence of TA dosage on CTC board functioning were freed to vary between groups. In the third model, paths depicting stabilities in TA dosage and in CTC board functioning were also allowed to vary freely across groups. Model fit and path coefficients were estimated using AMOS 4.01 (Arbuckle 1999). Tests between model fits used variance–covariance matrices and unstandardized path coefficients. However, results of path models were interpreted in terms of standardized coefficients. Because the number of sites is on the lower end of what is recommended for path analyses, interpretation of results includes non-significant as well as significant paths in a few cases.

Results

Correlations among total TA dosage and board functioning for each of the years of data utilized here are presented in Table 3.

Table 3 Correlations between total technical assistance dosage and CTC board functioning
Full size table

Model Results for On-site and Off-site TA

Results for the analysis of on- and off-site TA models, conducted separately, are presented in Table 4. For most paths, coefficients are similar for on-and off-site TA. All of the significant paths in the model reflect stabilities. Stability in board functioning is substantial for consecutive years (coefficients for paths G and H are about .5), with an additional component of stability across two years (path I = .25). The stability of off-site TA dosage across two years (path F) is also about .5. However, the stability of on-site TA is smaller in comparison (.22, NS). The contemporaneous reciprocal paths between TA dosage and board functioning (paths A and D) are modest and not significant. The influence of both on- and off-site TA dosage in 2003 on 2004 board functioning is negligible (path B). However, the influence of board functioning in 2003 on 2004 TA dosage (path C) is larger, but still modest and non-significant (.13) for both types of TA. Finally, the influence of 2004 TA dosage on 2005 Board functioning is modest for on-site TA (path E = .12, NS), and negligible for off-site TA (.03, NS). Thus, the main differences across on- and off-site TA models are: (1) a lower level of stability for on-site TA compared to off-site TA; and (2) a somewhat larger influence of on-site TA dosage on board functioning for one path, although the coefficient is not significant.

Table 4 Standardized path coefficients for on site and off site technical assistance
Full size table

Moderators

Next, we examined whether TA dosage had a larger impact on board functioning for sites that were younger vs. older, had a lower vs. higher level of baseline board functioning, or had a lower vs. higher level of need for TA. These analyses were conducted as two-group models, based on a median split of sites for each moderator; the test of moderation involved examining whether the fit of the model improved when parameters were allowed to vary freely for the median-split groups vs. constrained as equal across the two groups.

Age of Prevention Board

Tables 5 and 6 present tests of model fit when allowing parameters to vary across younger and older CTC board groups. For on-site TA, significantly improved fit to the data was found with (a) freeing the paths between TA dosage and CTC board functioning (model 2) and (b) additionally freeing paths for stability in TA time and CTC board functioning (model 3). However, model 3 did not demonstrate an improved fit over model 2. There was no evidence that allowing paths to vary freely across older and younger groups for off-site TA improved model fit. Thus, we do not present results for the two-group (older and younger boards) off-site TA model and turn to the findings for model 2 for on-site TA. Table 7 presents the standardized coefficients for model 2 for on-site TA, i.e. a two-group (older vs. younger boards) model. The paths for the influence that on-site TA dosage has on board functioning (paths B and E) were modest and moderate for younger CTC boards (0.14, NS, and 0.21, p < 0.05) and about zero for older CTC boards (0.003 and −0.06).

Table 5 Model fit for analyses of moderators of on-site TA, based on median split of age of site, baseline board functioning, and need for TA
Full size table
Table 6 Model fit for moderator analyses of off-site TA, based on median-split of age of site, baseline board functioning, and need for TA
Full size table
Table 7 Standardized path coefficients, critical ratios, and p-values for median-split, two-group moderation of TA dosage-board functioning association: moderation based on age of site, baseline functioning, perceived need for TA
Full size table

Baseline Board Functioning

Tables 5 and 6 present tests of model fits for the two-group models in which the groups are formed by a median-split on baseline (2003) board functioning. For on-site TA, significantly improved fit to the data occurred after freeing the paths between TA dosage and CTC board functioning (model 2). In contrast, allowing paths to vary for the groups formed by baseline functioning of CTC did not improve model fit for off-site TA.

For the group with low baseline board functioning, the two longitudinal effects of on-site TA dosage on board functioning (paths B and E, respectively, in Fig. 1) were inconsistent in sign:−.15 and .11 (both NS; Table 7). For the high baseline board functioning group, the longitudinal paths were .11 (NS) and .29 (p < 0.05), indicating a modest to moderate positive effect of on-site TA for the high functioning group. In addition, the contemporaneous reciprocal association between TA dosage and board functioning is small for both groups for 2003, and for the low baseline board functioning group for 2004. However, the reciprocal association is large (standardized beta = .63) for the high baseline board functioning group for 2004.

Need for Technical Assistance

Tables 5 and 6 present the results of tests of model fit examining the two-group model based on a median-split of CTC boards’ rated Need for TA. No improved fit to the data was observed for on-site TA. However, for off-site TA, when path coefficients were freed to vary across groups (models 2 and 3), significantly improved fit to the data was observed. However, as above, model 3 did not yield a better fit compared to model 2.

For the group with a high level of perceived Need for TA, just as for the low baseline board functioning group in the previous set of analyses, the coefficient for the path from 2003 TA dosage to 2004 board functioning was negative (−.20, NS; i.e., greater off-site TA dosage was associated with lower levels of board functioning). For the low TA Need group, the estimate was modest and positive (.11, NS). For both groups, the estimate of the path from 2004 TA dosage to 2005 board functioning was about zero.

Examining Alternative Processes

TA Provider or Region Effects

The CTC sites were grouped administratively into five regions of the state. Each region had a dedicated TA provider who lived in the region and worked exclusively with the sites in the region. It is possible that some providers were more skilled than others, or that for some other reason the sites in a region showed greater response to TA than sites in other regions. To examine this possibility we conducted ANOVA analyses with region as a factor predicting change in board functioning. The result of this analysis did not support the hypothesis (F = 1.11, p = 0.36). More importantly, the interaction between region and on-site TA dosage was not significant in predicting change in CTC board functioning from 2003 to 2005 (F = 0.732, p = 0.60).

Board Cooperation with TA

Another potential source of complexity influencing how TA impacts CTC board functioning is the level of cooperativeness of CTC board with the providers. It is possible that boards that are more cooperative with TA would gain relatively more benefit from the same dosage of TA. An initial ANOVA demonstrated that Board Cooperativeness was associated with change in CTC board functioning from 2003 to 2005 (F = 7.89, p = 0.006). This association was fully accounted for by including the 2003 board functioning in an ANCOVA model (F = 0.002, p = 0.96). Most importantly, CTC Board Cooperativeness did not interact with TA dosage in predicting change in board functioning from 2003 to 2005 (F = 0.186, p = 0.67).

Discussion

This report provides one of the few quantitative examinations of the impact of TA on public health prevention programs or community-based health promotion coalitions. Assessing the impact of TA and understanding the conditions under which TA is most effective is an important step in supporting the dissemination of evidence-based programs. We investigated the effect of TA dosage on the functioning of Communities That Care prevention boards in Pennsylvania using longitudinal, path models incorporating three years of multiple-informant data.

Impact of TA

The results demonstrate evidence for limited impact of technical assistance on board functioning. In the models where all sites were analyzed together, off-site TA dosage—comprised primarily of telephone and email contact—was not related to improvement on board functioning the following year. For on-site TA, dosage predicted a small but not statistically significant improvement in board functioning the following year. Thus, our data for CTC in Pennsylvania show minimal evidence that TA dosage is linked to improved coalition functioning across all sites.

We also examined three potential moderators of the impact of TA: age of the CTC site, baseline board functioning in 2003, and boards’ perceived need for TA. Results of the longitudinal analyses indicated that TA dosage had a positive effect on board functioning the following year for younger sites and for sites that exhibited a higher level of board functioning in 2003. These results are important in helping to provide some understanding of the conditions under which on-site TA—which is relatively expensive—is currently most effective.

There are several possible reasons why younger sites may be more susceptible to TA influence. First, the older sites were beyond the stage of three-year implementation funding. Although these sites were eligible to apply for one-year “sustainability” grants from the state—and some have been awarded these grants—the older sites generally perceived fewer constraints and obligations to the state-funded CTC system and its goals. TA providers have informed us that some older sites are not as cooperative and open to engagement with TA providers as they once were. However, follow-up analyses failed to find evidence that TA-rated Board Cooperativeness moderated the impact of TA dosage.

A second possibility is that younger sites are in a developmentally more malleable period than older sites. As coalitions and programs mature, the relations among local organizations and individuals on the CTC board may take on more rigid patterns; staff-member relations may develop expected patterns; and coalition leaders may develop an institutionalized style that impedes change. It is also possible that over time, entrenched interests co-opt the CTC process or develop an interest in maintaining the status quo. Thus, older sites may be less open to the influence of a change agent.

It is possible that the direction of effects is reversed, despite our use of longitudinal data. That is, it is possible that TA providers are more comfortable spending more time at higher-functioning young sites—where there may be a high level of enthusiasm for CTC. Such sites may show more respect to or interest in the thoughts of the TA provider. In the absence of an experimental design, this possibility cannot be completely eliminated. Still, the longitudinal data and the null results for Board Cooperation as a moderator of TA dosage seem to argue against this interpretation.

The moderation analyses also indicated that those CTC sites that initially demonstrated a high level of functioning gained relatively more benefit from the same amount of TA than lower functioning sites. There may be a threshold of functioning where limited TA is unable to help poorly functioning sites overcome local barriers. Such sites may require more intensive, focused TA than provided for in the system studied here. It also may also be that higher-functioning sites have a greater ability to articulate TA needs (see Mitchell et al. 2004), and thus receive consultation that is focused on the underlying and salient problems at hand. This may be part of a broader feature of high functioning sites regarding the ability to learn from external sources and adapt accordingly.

Finally, we explored whether the impact of TA dosage was dependent on perceived need for TA or the TA provider. Level of perceived need for TA did not moderate the impact of on-site TA, but it did moderate the impact of off-site TA. However, because path coefficients were inconsistent across years, we do not offer an interpretation of these results. And we did not find evidence that certain regions or TA providers (the two are linked in this case) demonstrated stronger evidence of TA influence than others.

Stability in Board Functioning and TA Dosage

The results also include estimates of the stability in TA dosage and board functioning. Board functioning showed substantial stability when assessed by multiple, independent raters. Thus, there is some degree of inertia in levels of board functioning, which we view as a likely result both of the stability of personnel involved and their ongoing relations with each other, as well as the ongoing influence of community context. Although off-site TA dosage also showed a moderate degree of stability across the two years of data, on-site TA dosage showed little stability.

These differing levels of stability across on-site and off-site TA may be linked to the characteristics of visits vs. off-site (e.g. phone, email) contact. Given the travel time involved, on-site visits generally consist of several hours, while phone and email contacts frequently consist of a few minutes each. The relatively greater frequency of phone and email contact allows for a more continuous dispersion of TA time across sites. On-site visits, on the other hand, are more intensive, costly, and infrequent. It may be that TA providers balance visit time over a longer period of time than a calendar year. It may also be that urgent issues and crises emerge at different times for different sites, and then tend to recede as problems are handled. Thus, TA providers may be called for more intensive on-site work on a rolling basis across sites. In any case, it appears that sites that receive higher levels of on-site TA dosage one year do not tend to receive continued high levels of TA dosage the next year.

Given the stability of board functioning, the lack of stability in on-site TA may be problematic. Proactive and reliable TA should be deliberately responsive to the need of the site. Although strength vs. weakness in board functioning is fairly stable, TA dosage is not. Moreover, the data suggest that TA providers do not provide more assistance to sites that were functioning poorly the previous year, as one might expect. Together these findings may be diagnostic of a TA system that is not optimally responsive to level of board functioning.

The provision of variable amounts of TA depending upon the functioning of a community collaborative is an exemplar of a class of studies that have been characterized as “adaptive designs” in which variable amounts of service may be provided given the identified need. In such programs, the criteria identified and the measurement method (reliability and validity of clinical judgment) used to assess the needs of community coalitions and to recommend different dose levels become an integral part of the program itself, affecting its replicability and impact (Collins et al. 2004).

In a parallel study examining how prevention services were adaptively adjusted to meet the needs of high-risk families, findings indicated that specific clinical ratings of parental functioning were more valid and less vulnerable to bias than global assessments of family need for home visiting (Bierman et al. 2006). Similarly, there is a need to develop clear and valid criteria along a variety of dimensions to better target type, amount, and delivery format for TA to communities. It may also be the case that communities that may be avoidant or resistant of TA may lead TA providers to back off and thus provide the least TA to those in most need. Clinical findings on therapists with challenging patients indicate that therapists sometimes reduce efforts to alter maladaptive parenting practices when working with highly disorganized or resistant families (Patterson and Forgatch 1985). Although understandable, this kind of extinction of attempts to promote behavior change is unlikely to lead to improvement in coalition functioning.

The findings of Bierman and colleagues (2006) illustrate the vulnerability of certain kinds of judgments to biases that may reduce long-term coalition effectiveness: In contrast to standardized ratings that focus clinical judgments on clearly identified aspects of coalition functioning, global assessments may be more subject to other factors, such as the perceived feasibility or likely success of TA attempts, which may create a self-fulfilling prophesy.

Limitations and Conclusion

Assessing the impact of TA dosage on coalition functioning represents a recent approach to assessing the adequacy and effectiveness of TA. Most prior work has consisted of qualitative reports. However, there is a need for a more nuanced evaluation approach in which the dosage of TA is integrated with an assessment of the quality and focus of TA provided, which then can be used as a predictor of improvement in specific aspects of prevention coalition and program functioning and outcomes.

It is difficult to generalize the conclusions of this study regarding the impact of TA on coalition functioning to other coalition models and communities. We did not collect a random sample of prevention coalitions, but rather gathered data within a sample of convenience—that is, within an existing state-supported system of TA for a particular group of coalitions utilizing the Communities That Care model. It may be that the TA provided in this system was not of high quality or did not appropriately match the needs of the coalitions.

Nonetheless, the finding of relatively limited impact of TA on the entire sample of Pennsylvania CTC sites corresponds with the findings of a prior study examining another coalition model (Mitchell et al. 2004). The current report extends research in this area not just by studying another set of coalitions, but also by demonstrating that TA may have a greater influence on certain coalitions—in the present case those that are less mature or are already functioning well.

On a practical note, these findings highlight the need for a high-quality and evidence-based assessment of TA need (Feinberg et al. 2008) and systematic monitoring of TA services to ensure targeted sites (i.e. those not functioning well) receive the required support. When technical assistance decisions are made based on practitioner intuition and inclination, rather than based on an established, clear, and valid assessment protocol, positive effects may be reduced. In addition, our findings imply that there is a need to articulate more clearly TA models that might be used when working with different types of sites along such dimensions as maturity, malleability, and resistance. Finally, fostering the ability of poorly functioning sites to articulate needs and develop readiness to effectively use TA may be an area of study itself.