The Mpowerment Project (MP) is a community-level, evidence-based HIV prevention intervention for young gay/bisexual men that is a combination HIV prevention approach (small groups, community outreach, publicity, a drop-in center, and community mobilization). These components work together synergistically to target individual, interpersonal, social, and structural issues for change. Mathematical modeling has demonstrated that this program is one of the intervention strategies with the potential to prevent the greatest number of new HIV infections in the US in a cost effective manner (Cohen et al. 2005). The program has been implemented in over 150 US community-based organizations (CBOs) and is part of the Center for Disease Control and Prevention’s (CDC) Compendium of Effective HIV Prevention Interventions with Evidence of Effectiveness that they recommend health departments and CBOs consider when choosing evidence-based approaches to HIV prevention. Because the MP is the type of approach that the recently released National HIV/AIDS Strategy (www.whitehouse.gov) is promoting, and because it is being implemented in many locales, it is important to assure that it is implemented successfully.

Based on concepts of empowerment, diffusion of innovations, peer influence and outreach, and community organizing (Hays et al. 2003), the MP was shown to be effective in reducing rates of unprotected anal sex among young gay men, a group that remains at high risk for HIV (Kegeles et al. 1996, 1999). Research from other fields has shown that implementation of evidence-based interventions is challenging (Fixsen et al. 2005), and although there has been relatively little research regarding HIV/AIDS interventions, it is unlikely that they will be implemented more easily than programs in other fields (Norton et al. 2009). To our knowledge, no research has been conducted about how to help CBOs implement evidence-based, community-level HIV prevention interventions. This research gap is unfortunate given the cost-effectiveness and wide-spread positive impact of community-level interventions (Kelly 1999). In addition, given the current interest in implementing combination HIV prevention, including multilevel, multicomponent HIV prevention approaches that target individual, interpersonal, and social/environmental issues (Coates et al. 2008), similar to the MP, it is important to learn how to help CBOs implement such complex strategies.

The Interactive Systems Framework for Dissemination and Implementation (ISF) has been proposed as a heuristic to organize different aspects of dissemination and implementation processes, and within this framework, interventions to help CBOs implement new approaches are considered Prevention Support Systems (Emshoff 2008; Wandersman et al. 2008). Although it has been recognized that it is important to provide assistance to organizations seeking to implement evidence-based programs, there has been relatively little focus on examining what approaches are effective.

Translating Research into Practice (TRIP) Study

The TRIP Study is a longitudinal project in which a primary goal is to determine if an intervention for CBOs (i.e., a Prevention Support System) improves agencies’ capacity to implement an evidence-based program effectively (Rebchook et al. 2006). The intervention we developed and discuss here is the Mpowerment Project Technology Exchange System (MPTES), and the program we sought to help organizations implement was the MP. As the sole organization that provides information and trainings to CBOs seeking information about the MP, we had access to the entire population of CBOs seeking information on the MP. We were able to recruit most organizations that were implementing the MP or were about to implement it at the time of study initiation. The MPTES was provided to 72 CBOs that were implementing the MP for diverse populations of young gay/bisexual men around the US, and these organizations were followed for up to two years. This paper discusses the issues that were considered in the development of the MPTES; CBOs’ uptake, utilization and perceptions of the MPTES components; and the issues that arose during technical assistance (TA). A subsequent paper will describe the impact of the MPTES on CBOs’ implementation of the MP.

The recognition of the need for and the development of an intervention for CBOs to help implement the MP was an iterative process that occurred over several years, through the course of multiple informal (i.e., unfunded) and formal collaborations with CBOs implementing the MP. Our approach to assist CBOs implement the MP changed considerably over time. At the time of our original efficacy studies there was little discussion in the field about how to move research into practice in HIV/AIDS. The general approach at that time, and one that we originally adopted, has been characterized as a “top-down” or “Research-to-Practice” method (Flaspohler et al. 2008; Wandersman et al. 2008). We followed what Wandersman (2003) has called the dominant intervention science approach of the time: we conducted randomized, controlled trials of the MP, and assumed that the program would be picked up and implemented by practitioners, that is, AIDS organizations. Hence, publication of the outcome study was our initial way of disseminating information about the intervention (we were unsuccessful in our attempts to include substantial information about the intervention in the publications). Our initial “technology transfer” approach involved developing implementation materials for organizations by primarily focusing on describing the program’s core elements. However, as we obtained input from a community advisory board comprised of staff from diverse CBOs early in our efforts and continued to collaborate with organizations seeking to implement the MP over time, it become clear that an effective intervention for CBOs requires consideration of the realities that CBOs face. These include, for example, addressing funding complexities, political issues in the community, and organizational issues. Determining the issues that CBOs face in implementing a novel HIV prevention intervention has required learning from CBOs what they need. Hence, we shifted from a Research-to-Practice model and towards incorporating many aspects of a community-centered model (Flaspohler et al. 2008), in which we have extensively used community consultation and collaboration methods in the development of the MPTES. These are described later in this paper. Rather than problems to be fixed, we value CBOs as partners to be engaged and encouraged.

Overview of the Mpowerment Project

The MP seeks to propel community and individual empowerment and community mobilization processes, and incorporates community-based participatory methods (Israel et al. 1998); while such approaches are often called for in community psychology, they may be complex to scale up and translate into wide-spread practice. The MP was designed from the start to be adapted for and by each community. While specific methods are used to run the project, the intervention is not highly scripted. It requires that project staff, usually young gay/bisexual men, work with a group of volunteers who run the project and together critically analyze the community’s needs in order to develop activities to reach diverse social networks of young men. The intervention’s multiple components and emphasis on empowerment, community-building, and community mobilization and engagement processes make it more challenging to translate to practice than is an intervention that can be implemented by following a “cookbook” that details exact steps to follow in a particular sequence that are all directed by CBO personnel.

The MP’s activities are designed to be self-perpetuating and set in motion an ever-widening diffusion process through which young men communicate with and encourage each other about HIV risk-reduction. The MP is based on an empowerment model in which young men take charge of the project because when individuals are actively involved in finding and implementing solutions to their problems, the behavior change is more lasting. Intervention activities promote young men’s critical self-assessment of the reasons for their vulnerability to HIV, as well as analysis of the young men’s community, while also striving to build a community of young men that supports each other about HIV prevention. The project also teaches skills for safer sex negotiation and clarifies misunderstandings of which sexual activities are risky for HIV transmission. The MP draws on diffusion of innovations (Rogers 2003), which suggests that people are most likely to adopt new behaviors that have already been accepted by others who are similar to them and whom they respect. Since formative research indicated that HIV is not motivating or captivating for young gay men, the program relates HIV risk reduction to the satisfaction of other, more compelling needs, including social concerns. The goal of the program is to create a stronger and healthier young gay men’s community in which safer sex becomes the mutually accepted norm.

A CBO that runs multiple HIV prevention interventions is typically the home for the MP. Paid CBO staff (“coordinators”), young gay/bisexual men from the community, facilitate the project and coordinate the project activities. The CBO’s director of HIV education/prevention supervises the coordinators. Coordinators recruit a core group of 10-20 young men from the major subgroups in the community, and the coordinators and the core group, along with other volunteers, are the decision-makers for the project, and design and carry out all project activities. Coordinators, the core group, and volunteers are program core elements.

The MP has five other interrelated core elements that act together synergistically. Formal outreach involves teams of young gay men that go to locations frequented by other young men to promote safer sex in fun, engaging, and interactive ways; in each instance, they deliver appealing literature on HIV risk reduction that they develop, and distribute condoms and lubricants. Additionally, the teams create their own outreach events to attract young men from different subgroups and at which to promote safer sex. M-groups are peer-led, 3-hour meetings of 8-10 young men that use various skills-building exercises to address factors contributing to unsafe sex among the men (e.g., beliefs that safer sex is not enjoyable, poor sexual communication skills). Participants are also trained and motivated to conduct informal outreach with their peers. Informal outreach involves men encouraging their friends to be safe sexually, to attend project activities, and to join the project. An ongoing publicity campaign attracts men to the project by word of mouth, via social networking and websites, through the distribution of promotional materials at venues attracting young gay men, and through articles and advertisements in gay media. Ideally, the project and most activities are run out of a young men’s community/drop-in center, the focal point for community building.

Background Leading to the Development of the MPTES for CBOs

The Institute of Medicine analyzed the CDC’s HIV prevention efforts and noted that the CDC had paid too little attention to the dissemination and implementation of evidence-based interventions (Ruiz et al. 2001). This led to the CDC-supported DEBI (Diffusion of Effective Behavioral Interventions) Project, a strategy to disseminate evidence-based HIV/STD prevention interventions to health departments and CBOs nationwide (Collins et al. 2006). The REP (Replicating Effective Programs) Project is a CDC-funded mechanism by which evidence-based HIV prevention programs are developed into replication packages that CBOs and health departments can use as information and guidance to implement the interventions (Neumann and Sogolow 2000). The MP was one of the first DEBIs and through the REP project we developed an initial package of materials for CBOs to use when seeking to implement the MP. Recognizing the vital importance of obtaining community input in order to develop a package that would be usable by CBOs, we worked with a community advisory board comprised of individuals from diverse CBOs who reviewed all aspects of the preliminary package over the course of numerous meetings (Kegeles et al. 2000). The advisory board emphasized that the package had to be suitable for CBO staff with diverse educational backgrounds, show diversity among gay/bisexual men, include abundant examples of how to implement various components, and be visually appealing and easy to use.

Subsequently we conducted a one year pilot-test of this initial package with a local CBO (Kegeles et al. 2000). Since there was no funding to provide training and TA to CBOs beyond REP, and no other entity was providing it (this was prior to the CDC’s current work to provide capacity-building to AIDS organizations), we kept training and TA to a minimum in order to examine the extent to which our package would be effective in facilitating implementation. We found that many individual, organizational, and community factors affected implementation, including characteristics of the front-line staff who run the intervention, high staff turn-over at all agency levels, congruence of the intervention with the agency’s managerial systems and values, the organization’s HIV prevention philosophy, and community issues (Kegeles et al. 2000). Importantly, we recognized the need to supplement written materials and videos with proactive and on-going TA, a comprehensive training, and web-based resources.

We also learned that CBOs need help addressing evaluation of the program. A number of CBOs requested help from us on how to evaluate their MP implementation, because their funders required evaluation, but were not specific about what they wanted. This led us to conduct a substudy in which we interviewed CBOs around the US about how they evaluated community-level interventions and their capacity to conduct evaluation (Kegeles et al. 2005), so that we would provide advice that would fit with organizations’ capacity and needs.

In order to identify which issues and approaches were important to incorporate into an intervention for CBOs to help them implement the MP, our team held a series of meetings to examine our experiences to that point – participating in REP, the pilot collaboration with a CBO, the study of CBOs’ experiences with evaluation, and our informal experiences with other CBOs. Team members included the developers of the MP and individuals who had worked at CBOs implementing the MP. We systematically identified issues that had arisen. As an issue was identified, we discussed whether these were new issues that had not been discussed previously or if the issue might be subsumed under a previous topic, and if it was something that several CBOs had grappled with, or something that occurred only occurred infrequently. The issues we identified to address when developing an intervention for AIDS CBOs are identified below, together with brief discussions of them.

Lessons Learned about Developing an Intervention for AIDS Community-Based Organizations

General Issues

  • Written materials and limited training and TA are insufficient for effective translation of evidence-based interventions into practice (also discussed in Adams et al. 2000; Fixsen et al. 2005, 2009; Kelly et al. 2000; O’Donnell et al. 2000). Some CBO staff members dislike reading written materials, and staff who have relatively little education will often not use written materials extensively. Oftentimes frontline staff at AIDS organizations have relatively low education (high school degrees). Therefore TA and training need to be comprehensive and thorough.

  • While CBOs need information about the intervention’s core elements, they also need information about a myriad of other issues that affect implementation. Some issues relate directly to implementation of core elements (e.g., how to find an ideal space for the project), while others have to do with organization infrastructure (e.g., how to fit the intervention in among other projects the agency is conducting; finding the ideal staff). Some issues are broader (e.g., how organizations can evaluate the intervention). Flaspohler et al. (2008) distinguish between general capacity and innovation-specific capacity, but our experience indicates that it is not necessarily easy to establish a clear line between the two types of issues.

  • It is important to consider whom to intervene with at CBOs. It is crucial to focus on the staff who implement the intervention (in MP, these are the coordinators), and materials, training and TA must all be addressed to them. But people higher in the organization must understand the program as well (Fixsen et al. 2009), particularly supervisors of the frontline staff.

  • CBOs need advice about supervision, selection and retention of staff members, as well as preparation and planning for staff turnover. Consistent with our experiences, Fixsen et al. (2005, 2009) include proper staff selection as a core component for successful implementation when translating evidence-based programs into practice.

  • Encouraging diffusion and communication across organizations is helpful and desired by CBOs. Whereas hearing about the program from its developers is helpful, learning about how the program is being implemented by other CBOs may build even more credibility for the program and also facilitate independent problem-solving. Consistent with this approach, early adopters can become consultants for later adopters (Kelly et al. 2000).

  • CBO staff do not always think through the logic of their programs. Although agencies may be able to identify broad program goals, their objectives (the program activities) do not necessarily match the goals (Kegeles et al. 2005). As a result, organizations sometimes lose track of program components or have difficulty in thinking out how and when to start them. Therefore, the logic of the program needs to be clearly presented so CBO staff can understand why they are doing certain activities, when they should be done, and what outcomes should result. CBO staff need to be encouraged to evaluate and critically analyze the program’s functioning so that they can make changes as they implement it.

  • Organizations that request information about the program are at different stages of implementation. Some agencies are simply asking what the program entails, some are seriously considering implementing it and want to search for funding, while others are fully implementing it (Fixsen et al. 2009; Rebchook et al. 2006; Rogers 2003). Therefore, different materials and strategies to assist CBOs considering implementation of the program should be developed for organizations at different stages. For example, support for grant writing and sample budgets are important when helping a CBO that wants to adopt the program but do not have sufficient funding for it. Providing such information is also helpful in disseminating the program.

  • Funders often want CBOs to conduct evaluations of the program. As mentioned earlier, funders are frequently unclear about the type of evaluation they want, what is feasible for a CBO to do, or how to match evaluation requirements with the program (outcome evaluation of a community-level program is challenging). CBOs desire and need help in thinking out feasible evaluation approaches.

  • When an intervention is relatively unscripted, it is essential to provide abundant examples of how to conduct core elements. As previously described, the MP was designed to be adapted by and for each community, and uses community-based participatory methods. Although considerable guidance is provided about how to implement the core elements, precise step-by-step instructions are not possible. Given this, CBOs want many examples of how the core elements can be operationalized (e.g., what kinds of social activities to implement) in order to stimulate them to create their own community-specific program. Moreover, CBOs appreciate examples of approaches that have not worked before as well.

  • Community issues may have an enormous effect on program implementation. Community-specific idiosyncrasies (e.g., a conservative socio-political environment) can pose particular challenges to conducting programming for young gay men. Thus, these issues must be addressed in TA, materials and training.

Materials

  • All materials should help the project “come alive,” and convey the essence, excitement, and dynamism of the program. CBO staff want to be able to “see themselves” and their target community in the materials, as well as see the program as a whole in order to envision the nature of the program and how to implement it. This, in turn, increases their belief in the credibility of the program. This can be achieved through the abundant use of photos or videos of program participants and activities.

  • Written materials need to depict and fully describe how the program should be implemented. People want to easily find complete information about program implementation. Such written materials must be very user-friendly or they will not be used, and thus they should be visually attractive with a layout and graphics that are inviting and easy to follow.

  • Written materials should be durable and inexpensive. Staff turnover at agencies is very high and when they leave, they may take copies with them, or will have written on them. New staff will need their own materials. Hence it is important that materials be inexpensive and easy to obtain.

  • It is important to make provisions for visual learning styles. This can be achieved by liberally including graphics and photos, such as visuals of outreach team performances, outreach social events, and safer sex promotional materials. Describing these in words is far less effective, and as noted earlier, some CBO staff do not read manuals, or have relatively low reading abilities. Visuals make materials more interesting and compelling.

Technical Assistance (TA)

  • The most effective interactions between the TA and training providers and the CBOs is one of “exchange” rather than the “top-down” approach that is inferred by the term and process of “technology transfer.” Indeed, in the course of this work we have learned a great deal from CBO staff about the difficulties their organizations face in trying to survive, and in striving to implement an evidence-based program. Learning about implementation issues is a two-way street with CBOs teaching the TA provider about real world challenges, and the TA provider teaching CBO staff about implementing the program.

  • Organizations often delay in requesting TA. CBOs often delay seeking assistance with a problem until it has become a crisis (Kegeles et al. 2000). Sometimes organizations simply fail to identify a problem early on. On other occasions, they are reluctant to request assistance for some specific reason, such as a fear that their funder will learn of their difficulties. Thus, providing TA proactively is important.

  • Building and maintaining rapport between CBO staff and TA providers is imperative for developing trust. It is essential to build a positive relationship between the TA provider and CBO staff to facilitate the free exchange of ideas, and to ensure that the CBO staff do not feel “judged” (which causes them to reduce their engagement with the TA provider). TA providers can facilitate this by communicating by phone or in person, rather than through e-mails, which can be misinterpreted.

  • CBO staff do not want to be told what to do by someone outside their organization. Instead, they want to be encouraged to think out how to successfully implement the program and adapt it for their community, and to be supported to critically analyze the program themselves.

Training

  • Training should involve active-learning methods. Individuals should get an experience of how the program “feels” in the training, and various interactive approaches (e.g., problem-solving exercises, role plays) should be used to stimulate thinking about implementation in their own agencies.

  • Organizations want to learn from each other, not solely from trainers. There should be considerable opportunity for organizations to share experiences and problem-solve together during trainings, as well as encouragement to network with each other subsequently. Learning that occurs between peer CBOs may enhance the belief that the program can be implemented successfully, particularly when early adopters of an evidence-based program share information with those organizations seeking to newly implement the program (Rogers 2003).

These experiences taught us that CBOs face considerable barriers and challenges in implementing this evidence-based program. This brought us to the development of the MPTES, described below, and the TRIP Study to examine its use and impact on CBOs’ implementation of the MP.

The Mpowerment Project Technology Exchange System (MPTES)

The MPTES is based on social learning theory (Bandura 1977), diffusion of innovations (Rogers 2003), and theories and approaches to adult education (Knowles et al. 2008). We focus here on the MPTES components used with CBOs that are either about to or are currently implementing the program. In the terms of diffusion of innovation, these are organizations at “ready to implement” (i.e., beyond the decision stage but not fully into the implementation stage) and “implementation” stages.

The MPTES consists of seven components (see Table 1). Although each component discusses different aspects of the MP, there is overlap in content. Using mixed approaches to providing information on how to implement the program addresses the diverse educational backgrounds of individuals who use these materials and recognizes people’s unique learning styles (Hawk and Shah 2007). After piloting, written materials were redesigned with assistance from an editor specializing in adult-learning theories and a graphic designer to ensure that they are user-friendly and attractive. The manuals and videos were packaged together and the cost was set as low as possible to encourage CBOs to purchase them ($75). The manuals could also be downloaded from the website for free.

Table 1 The Mpowerment Project Technology Exchange System (MPTES)
Full size table

Program Manual

The program manual describes the MP’s philosophy and how to implement the program’s core elements. It was hoped that program coordinators and supervisors would read it fully and refer to it throughout implementation, and that Executive Directors would familiarize themselves with it. The manual contains many photographs of the different implementations of the MP that we conducted, (including photos of men of diverse ethnic/racial groups, ages, and physiques), and many examples of how the different core elements have been implemented, as well as descriptions of approaches that did not work well. It includes an overview of the intervention and the MP’s guiding principles (Module One); detailed information on conducting a community assessment, which is a practical ethnographic exercise to learn about diverse segments of young gay men in the community and how to reach them through publicity and outreach (Module Two); a comprehensive guide to the MP’s operating structure, including recruiting, hiring and supervising coordinators (Module Three); a description of establishing and maintaining the drop-in center (Module Four); information on conducting outreach (Module Five); background about M-Groups (Module Six); recommendations about program publicity (Module Seven); and a detailed evaluation guidance including a fold-out, detailed program logic model (Module Eight).

Overview Video

This video provides a visual depiction of all parts of the program, including what the core elements look like and how they interrelate. It was filmed during the research trial in Albuquerque, with actual program staff, core group (which showed white, Latino and Black core group members), and volunteers giving their own thoughts about the program. We hoped that all key project staff, volunteers, and other stakeholders (e.g., CBO board members) would watch the 22-minute video.

M-Group Facilitator Guide

This guide provides detailed instructions about how to conduct the small-group component. We also developed a smaller, pullout guide that provides facilitators a step-by-step script they can follow while running the groups. The M-group is the only highly scripted part of the program where the staff can follow directions about precisely what to do as they conduct this program component.

M-Group Training Video

This video demonstrates how to facilitate an M-Group, the important topics to cover, and tips on successful implementation. It was also filmed in Albuquerque with young men from the project acting the part of individuals attending an M-group. A professional actor served as the video narrator, pointing out tips for facilitating the groups.

Training Program

A 3-day, comprehensive, interactive training was developed to provide participants with an in-depth understanding of the MP. Coordinators and program supervisors are strongly urged to attend these trainings. Early in the development of the program, we obtained input from a consultant with extensive experience in developing trainings. Considerable effort was spent in analyzing who and what should be trained (Salas and Cannon-Bowers 2001). Trainers were former MP coordinators from the research trials who, subsequent to the studies, had experience in running the program in a CBO. In accordance with Salas and Cannon-Bowers’ advice about training, we focused on influencing knowledge, skills, and attitudes that are needed to conduct the MP. The trainers demonstrated these knowledge, skills and attitudes throughout the trainings, gave opportunities to the attendees to practice the skills during role-plays, and provided feedback to trainees during and after practice.

The training includes PowerPoint slide shows, group discussions, and presentations as well as interactive training exercises (role plays, games, and brainstorming exercises) to simulate thinking about the start up and implementation of the MP. The agenda includes time for both formal and informal question and answer sessions. Participants also have opportunities to share their personal and professional experiences with each other, exchange ideas, brainstorm new approaches, offer critical feedback and support to each other, and learn from each other’s experiences. Evaluations of the trainings (conducted after each training) have shown this to be particularly valued by participants.

The trainings cover a wide range of topics, including background of the MP’s research, discussion of what contributes to HIV risk-taking among young gay men, the guiding principles of the MP, the purpose and content of each core element and how to implement each, participant recruitment strategies, how to conduct a community assessment, and an overview of how to evaluate the program. Trainings are provided to individuals wanting to attend them regardless of whether they are part of the TRIP study.

Technical Assistance Program

Technical assistance (TA) involved frequent, regularly scheduled interactions with the TA providers to provide support and advice about implementation issues and problem-solving assistance. TA was provided proactively; the TA providers did not wait to be called for help, but instead contacted the CBOs and asked to discuss implementation issues. The two major goals of the TA program were to help CBOs learn how to overcome barriers to effective implementation so they could implement the program with fidelity, and to help CBOs critically analyze and reflect on their program’s functioning so that they could make program changes as needed. The TA providers were trained to use problem-posing dialogue (Freire 1973) to support CBO staff to critically analyze their implementation challenges and derive solutions. TA providers also assessed additional needs CBOs had (e.g., organizational/infrastructure development) and made recommendations about how to meet these needs.

There has been discussion in the literature about whether TA should be provided on-site versus off-site (Feinberg et al. 2008; Fixsen et al. 2005). While our preference would be to use a combination of on and off-site methods, site visits are not possible with the number of CBOs that we need to reach. Since the CBOs implementing the MP are located across the country (in this project, the CBOs were in 31 states plus the District of Columbia and Puerto Rico), and thus travel costs and TA provider time would be prohibitively expensive. Yet, CBOs often request that their TA providers visit their agency and community to gain a better sense of the issues and implementation challenges they face. Since that is not possible, the TA provider seeks information about the community via interactions with the CBO staff and online.

Our objective was for TA providers to have phone calls with program coordinators approximately every two weeks, and less often with supervisors. The TA providers were in frequent e-mail contact as well. The TA providers strived to build rapport with the CBO staff, and so occasionally sent birthday or holiday cards and, if it seemed appropriate, asked about individuals’ personal lives. The TA providers tried to meet with the CBO staff in person during trainings or when they were at one of the national HIV/AIDS conferences in order to increase rapport. During TA sessions, the TA providers focused on issues they anticipated might become problematic before becoming crises, discussed whatever topic the coordinators or supervisors mentioned, and asked probing questions to assess the extent to which each core element was understood by the coordinators, how each was being implemented, and to help CBO staff consider fidelity issues. The TA providers kept detailed notes and wrote summaries about each TA session, which were entered into a database. The major topics that arose during the conversations were coded in another database immediately after each session. TA providers refreshed their memories about program implementation and CBO issues by reading their notes before each TA session.

Web-Based Resources

The mpowerment.org website was designed to increase CBOs’ access to information about the program. The MP program manual is available in either HTML or Adobe Acrobat format to download or to read on-line, so that if an organization has no funding to purchase a hard copy of the program manual or needs additional copies, they can download them for free. The website also contains photos of the TA providers, PDFs of journal articles about the MP, an electronic version of the overview brochure, and examples of visual materials that were developed in our previous research for conducting formal outreach (e.g., safer sex promotional materials, photos from social outreach events). There was also a “Hot Topics” page to present new information and to make announcements about things such as up-coming trainings. For example, to respond to a number of requests for more detailed information on running core group meetings, a core group supplement was created and posted to the Hot Topics section.

The website also contained interactive features, including a chatroom for periodic online discussions concerning current issues that programs were facing (e.g., alcohol use, dating policies, fund-raising, outreach). The online chats were intended to create peer support for CBOs, and encouraged appropriate adaptation of the program while maintaining fidelity. There was also an e-mail forum for CBOs to post questions or share experiences, ideas, frustrations, resources, and materials. CBOs could communicate directly with each other via the forum. The site also had an “Ask a TA question” feature where users completed an online form to ask one of our TA providers a question about their project. It was hoped that all coordinators would access the website, and that many supervisors would do so as well.

Methods

Recruitment into Study

CBOs that requested information on the MP were sent an overview pamphlet about the program and were asked to recontact us if they wanted additional information. During the recruitment period, all 153 CBOs that requested more information (e.g., implementation materials, training, or TA) were invited to participate in the TRIP Study. The overall project studied CBOs that were currently implementing the MP, as well as organizations that were seeking information but had not moved into implementation. Of the 153 CBOs, 101 (66%) agreed to participate. Of the 52 organizations that did not join the study, 15 refused participation, 8 had decided not to implement the MP, and 27 failed to respond to repeated e-mails and phone calls about the project, but other information we had on them indicated that they were not implementing the MP. Two CBOs had discontinued implementing the MP.

Of the remaining 101 CBOs, 72 (71%) were getting ready to implement or were implementing the MP, and were of primary interest to the TRIP Study. This paper focuses on the 49 CBOs that were still implementing the MP two years after the baseline assessment for TRIP. The 23 CBOs that are not included in this paper discontinued implementing the MP by the year 2 follow-up, primarily because their funding had been eliminated.

Participating Community-Based Organizations

As can be seen in Table 2, the CBOs in this study were diverse in numerous ways. Three-quarters of the CBOs were AIDS organizations, but a variety of organizations implemented the MP for at least two years. There was substantial variation in the size of the organizations, ranging from CBOs that were comprised of only one half-time employee to 750 employees, and likewise, the size of the staff who focused on HIV prevention ranged from one half-time employee to 100 staff. The size of the agencies’ budgets varied tremendously and about half the CBOs’ primary purpose was HIV/AIDS whereas the other half’s primary focus was something else. The size of the communities where MP was implemented ranged in size, from small towns with 30,000 inhabitants to very large urban areas. Some of the small towns were in semi-rural areas. The CBOs were also located across the US.

Table 2 Characteristics of the 49 community-based organizations
Full size table

Some CBO staff had been in contact with us before the TRIP Study, and had received services from us before they were assessed at baseline. Some had sought technical assistance from us when they were deciding whether or not to adopt the program, and so we had had some, albeit limited contact with them before starting this study. Other organizations that were implementing the program had requested implementation materials (manuals, videos), or had attended a training.

Data Collection Procedures

Once CBOs consented to participate in the TRIP Study, we conducted semi-structured telephone interviews with 2-4 staff members (MP coordinators and their supervisors), as well as 1-2 core group members. Those individuals (or their replacements, when staff turnover occurred) were reinterviewed at 6-months, 12-months, and 24-months post-baseline to assess their use of the MPTES, implementation of core elements, and barriers and facilitators to implementation (N = 532 interviews with 329 individuals). Participants were compensated $25 for their time. Each interview lasted 1-2 h, and the interviewer typed detailed, near-verbatim notes of all open-ended responses and numerically coded responses to items with pre-defined response sets into a database. In the interviews, we asked how each core element was being implemented, their target population, the CBOs’ budgets, staff turnover, various attitudinal items, their utilization of and experience with the MPTES components, and their feedback about the components. The institutional review board at the University of California, San Francisco approved all protocols and procedures.

Staff Turnover

Staff turnover was very high. Given that new staff would presumably need to use the MPTES more than established staff, an issue of great relevance to the our understanding of the MPTES was staff turnover of project coordinators. By the 6-month assessment, 13 agencies (26.5%) had replaced a coordinator. By 12 months, an additional 19 agencies (38.8%) had lost a coordinator, 32 agencies (65.3%) cumulatively. At 24 months, another 13 organizations (26.5%) had seen a turnover in the staff who run the MP on a day-to-day basis, which meant 45 out of 49 agencies (91.8%) had reported a coordinator turnover since the baseline assessment. Only 4 agencies (8.2%) had the same coordinators at the 2-year assessment they had at baseline. Almost half the CBOs (24, or 49%) had replaced more than one coordinator during the two years.

Results

Utilization of the MPTES

It was a challenge to develop agency-level scores regarding MPTES utilization since CBOs varied in how many staff members were interviewed at each assessment. We opted to create an agency-level score for each MPTES component by taking the maximum number of times each component was used by any one individual a the organization. For example, to derive a score for showing the overview video, if three staff members were surveyed from a CBO, and one respondent reported that s/he had viewed it 10 times, one reported 8 times, and one reported 5 times, the agency score for that component was 10, the maximum of the 3 scores. This score can be interpreted as the minimum number of times the overview video was viewed in total across respondents at that agency. A simple summation across individuals was not used because agencies varied in numbers of staff they had, and because some materials might be used simultaneously (e.g., the videos). Website utilization was not included because of complexities of measuring it (e.g., should opening a website or downloading a manual be considered utilization?). Executive directors were not interviewed, but our sense, based on other respondents’ reports, was that they rarely looked at any of the MPTES except, occasionally, the overview video.

The program manual was used much more often than any other MPTES component, and its use did not significantly decline over the time of the TRIP assessment despite the apparent change in median utilization (see Table 3). It is important to note the large standard deviations in manual use, as this indicates tremendous variability among CBOs in utilization. Some CBOs rarely used the manual (three CBOs did not report using it even once at the 6-month assessment), whereas other organizations used it quite often (two CBOs reported using it 100 times or more at the 6-month assessment). Although it was used more than other MPTES components, the program manual was not used as frequently as we expected.

Table 3 Organizations’ utilization of various Mpowerment project technology exchange services
Full size table

In contrast to the program manual, the utilization of the other materials declined over time. The M-group facilitator guide was the second most used material, but was used far less often than the manual. Its use did not change at 6 and 12-months post-baseline, but declined significantly by the 2-year assessment, when it was not used very often. The M-group training video was used much less often than the M-group facilitator guide at each time point, which makes sense since the video is specifically meant for training how to conduct the groups, as compared with the M-group guide which can be used for every group to keep facilitators on track while running groups. Use of the M-group facilitator guide declined substantially by the 2-year assessment, at which point it was used infrequently. The overview video was used more often at baseline and at 6-months, and then declined in use by the 1 and 2-year assessments.

In contrast to the decline of the use of most materials (other than the program manual), TA increased at 6-months, which makes sense given that proactive TA only began post-baseline. It then remained at the higher level over the course of the project. CBOs also increased in their requests for TA after baseline and continued requesting it over time. Delivering TA to the CBOs was often challenging, as some CBO staff were very receptive to it, while others did not want these phone calls. Staff that were less interested in TA would often miss or reschedule phone calls, or would fail to return calls.

At every assessment an average of only 20% of staff implementing the MP had attended a training, and this did not change over time. Again, it is important to note the large standard deviations in this variable, indicating that some organizations had a far higher proportion of staff who had undergone training on the MP, while other organizations had substantially fewer staff who had attended a training.

Although we heard of substantial use of the website, the interactive elements were not very successful. Scheduled chats were poorly attended. It was challenging to implement the e-mail forum, since e-mails sent to it were supposed to be sent on to other CBOs for their responses, but there became too much e-mail traffic and many people opted out of forum participation. The “Ask a TA question” feature did not work well either, as CBO personnel simply called or e-mailed their TA provider questions instead.

Feedback on the MPTES

Coordinators and supervisors gave a wide range of positive and negative feedback about the MPTES which did not seem to vary substantially over time. Table 4 shows summaries of the qualitative feedback given to us during the interviews regarding each part of the MPTES. All responses were written down during the interviews and then subsequently coded into positive, negative, a mixture of positive and negative feedback, and neutral, when it was difficult to discern if the response was more in a positive or negative direction.

Table 4 Qualitative feedback about each MPTES component from coordinators and supervisors
Full size table

There was a great deal of positive feedback. Around half the overall feedback was positive, and no MPTES components were assessed more negatively than positively. Technical assistance was most highly rated of all components. The multiple examples of how core elements had been operationalized successfully, as well as what had not worked previously, were perceived as very helpful. Participants enjoyed how interactive the trainings were, and particularly appreciated having opportunities to learn from and network with staff from other CBOs. There was considerable enthusiasm for the comprehensiveness of the manual, with some people reporting using it “all the time” and that they regarded it to be “the purple bible.” Working with TA providers who thoroughly understood the program from personal experience was very important to many CBO staff members. Staff who worked in unsupportive organizational environments felt less lonely and more supported when being able to communicate with a TA provider who understood implementation and issues. Having a chance to brainstorm with someone else also was perceived as helpful.

Negative responses regarding MPTES components fell into three major categories. First, many respondents felt that insufficient diversity was depicted in the materials. Racial/ethnic minority respondents often felt that depicting that the program related to diverse groups by showing photos of “many diverse people together” failed to recognize that some ethnic/racial groups primarily socialize together, and that many MPs are implemented solely for one ethnic/racial group. Hence, instead of mainly using images of young gay men together of diverse racial/ethnic backgrounds, the preference is to show all-Black or all-Latino men together, and showing the diversity within these groups. The second area of criticism was the need to discuss and visually depict how the MP is implemented by CBOs in the “real world,” not just how we implemented it in our studies in different communities. Learning from staff at other CBOs that are perceived as similar to themselves, that are targeting the same ethnic/racial or age-group, or are similarly situated (e.g., in urban areas or in semi-rural areas) was desired. The third area of criticism was about CBO staff members’ desire to learn how to adapt the program for their own ethnic/racial group, geographic location, or funding level. Some participants wanted the TA provider or the materials to tell them specific ways to adapt the program for their own situation. Other participants wanted information about what to consider in making their own decisions about how to adapt the program.

Issues Discussed During Technical Assistance

Many different topics were discussed during TA sessions (see Table 5), including many conversations about core elements. At baseline, CBO staff most wanted help with the particularly innovative core elements (i.e., how to work with or develop a core group and how to create events and activities that draw young men to the program and then conduct safer sex promotion with them); these two core elements remained the topics that they most often discussed over time. The core element that was discussed least often was informal outreach, which makes sense since project participants conduct it without coordination by agency staff.

Table 5 Percent of CBOs that discussed particular topics once or more during technical assistance
Full size table

TA also addressed issues other than core elements. Organizational issues were often discussed, such as challenges with management that did not always buy into the project’s methods, or discord with other agency programs. Community issues were also often discussed, such as dealing with homophobic responses to the project in the community. The topic of how to conduct evaluation arose quite often, as help was sought about how to conduct evaluations of the program that would satisfy funders. An unanticipated topic that was discussed were problems with funders, as the latter did not always understand the intervention well, and CBO staff wanted to discuss ideas about how to deal with this. Finally, one topic that arose relatively frequently was “Program Review”—a federal requirement that a local board must review program activities and safer sex promotional materials used in programs that the CDC directly funds. Such review was often experienced as censorship by CBO staff and had a chilling impact on their creativity when developing safer sex promotional materials.

The topics that were discussed changed over time. Community issues were discussed less often over time, presumably because they had been dealt with earlier. M-groups were also discussed less often, which corresponds to less use of M-group facilitation materials. Several topics were increasingly discussed by one year post-baseline, but then decreased in frequency of discussion, possibly because the staff knew more about how to deal with them (e.g., how to publicize the project, how to work with the core group, issues regarding the coordinators, and funding). Yet other discussions about topics increased over time through to year-2, including how to encourage informal outreach and challenges with funders.

Discussion

This is one of the few studies that has focused on analyzing how a Prevention Support System is utilized over time. Much of what we learned about MPTES utilization is relevant for the development of other Prevention Support Systems, particularly in HIV/AIDS prevention, but also in supporting the movement of other evidence-based programs to practice. Except for the widely used program manual, written and video materials about the program are used early in implementing a program, and then their use declines. It may be that once multicomponent programs are being implemented, materials that only focus on one component are not perceived as being useful anymore. However, once TA is proactively provided and rapport is established, its usage remains fairly constant over time. This might, in turn, promote motivation to request assistance. TA may help keep CBO staff focused on program components and implementation strategies that they would otherwise forget over time, as well as help with new, idiosyncratic implementation issues. When a single coordinator was running a program alone, the TA sessions provided both informational and emotional support, as the TA provider pulled the individual into a network of people who understood the complex issues facing him. Such TA may be enormously helpful as such complex programs are scaled up. While ongoing TA that extends for a long period of time may be impractical, periodically providing it, even over a long time span may be successful.

The topics discussed in TA also changed over time, and went beyond solely focusing on core elements, mirroring the earlier feedback we had received from CBOs that the MPTES must address the spectrum of issues that CBOs face in implementation. This illustrates that Prevention Support Systems should follow a community-centered model, in which agencies tell the developers of such a system what they need.

Turnover of the frontline staff was exceedingly rapid. To have staff leave within a year of going through training and learning how to implement this multifaceted program makes it challenging for organizations to implement a complex program. Fixsen et al. (2005) showed that in implementation research across fields, full implementation of a novel program can take several years. This much staff turnover may make it difficult to move into full implementation efficiently or stay in full implementation (Fixsen et al. 2009). Increased use of technology exchange services would be a way of coping with such rapid turnover, such as getting new staff into trainings as quickly as possible. But at each time point, only 20% of staff had attended a training.

Given the enormous health disparities in the US, many programs need to be implemented by and for ethnic/racial minority populations. Many CBO staff felt that they neither saw themselves nor their target populations in the manuals and videos, which caused some CBO staff to doubt the program’s relevance for their work. For implementation materials to be credible in speaking to ethnic/racial minorities’ needs, visual depictions of and examples about implementation need to reflect how the US is organized socially. To a great extent, our society remains segregated along racial/ethnic and class lines, and attempting to convey that a program can be implemented for diverse groups is likely to be more successful when it shows people of particular ethnic/racial backgrounds together and then depicts diversity within those groups (e.g., in clothing styles, physiques, skin tone).

In translating a research-based program into practice, new users of the program want to know how it is implemented by “real” organizations that are similar to themselves – organizations that target the same populations (particularly if they are ethnic/racial minorities), and that conduct the intervention in similar settings. Lessons learned from “peer organizations” are seen as more credible than only learning from trainers or TA providers who primarily refer to research experiences. In addition, learning about real world implementation challenges and ways of overcoming such barriers is extremely helpful. Learning about what had occurred in the research communities was of interest, but was insufficient.

Adaptation issues continually arose in this project. While the implementation literature is rife with discussions about the tensions between “implementation with fidelity” versus “adaptation,” we did not envision this as a dichotomy. Instead we sought to create a program that could be modified by CBO staff while retaining fidelity to the original methods. We hoped that by providing many examples of different ways to operationalize core elements, CBO staff could use the program’s guiding principles to make the program their own by adapting it for their own ethnic/racial group, age group, or geographic location. While many CBO staff found the guiding principles and examples to be sufficient in order to develop their own programs, some staff members wanted more detailed instructions about the process to follow in adapting the program. Other staff members wanted to be told of specific program modifications for their populations and settings. CBO staff vary in their capacity to conduct adaptation work, and many desire more direction. Providing guidance about how to make adaptations that retain fidelity is likely important for most Prevention Support Systems.

Our experiences with the MPTES have led us to consider several specific changes to the system that would be of relevance to other Prevention Support Systems. First, we question the use of videos. When we first began showing the overview video, people would break into applause at the end because the young men’s testimonials in the film about the program’s impact on them were quite moving. Over the years, however, the film did not age well; fashions, hairstyles, and slang rapidly changed, so that the film quickly felt outdated. Young gay men who viewed the video were distracted by the changing styles, and therefore the video was no longer experienced as a powerful, positive statement about the program. Yet the intent of the overview video, as a tool to depict the HIV prevention program in its entirety so that viewers can get a feel for it, is still important. We suggest using audio-slideshows instead, a low-cost alternative, in which voiceover accompanies a slideshow. These are relatively easily produced and updated. In contrast to the overview video, the M-group video still seems needed, since it was designed only for use in training coordinators how to conduct M-groups and they, hopefully, can overcome the appearance of individuals in the video.

The second change to the MPTES we would make is to have the TA provider be somewhat more directive about topics to address in the sessions. In this exploratory study, we structured TA sessions to be largely client-centered because we were uncertain what CBO staff needed or wanted. However, some topics were not addressed enough. For example, there were few conversations about informal outreach, an essential core element. Unlike other program components, CBO staff do not carry out informal outreach themselves, although they are expected to try to mobilize men to have conversations with peers. Concerns about “deliverables” for contracts drove many TA sessions, and since informal outreach was not typically measured in evaluation required by funders, CBO staff did not discuss it much. Hence, we believe a TA program must be more directed and focus on topics CBO staff members may forget, while preserving the client-centered aspect.

Empowerment, community organizing, and community engagement methods have all been widely recommended in the developing programs that address a multitude of issues, including HIV prevention (e.g., Israel et al. 1998). While these approaches may be used relatively easily in individual programs, scaling them up so that they can be run in many communities is more challenging. To move such programs into practice requires teaching organizations how to implement a process and conduct ongoing critical analysis of how the process is functioning, as well as how to implement program elements that may look somewhat different with diverse populations. Although challenging, information about how to implement such an approach can be part of a Prevention Support System, as we have shown here.

The CDC has identified technical assistance, training, materials development, technology transfer, and information sharing as crucial components of capacity building for organizations implementing HIV/AIDS prevention programs (Collins et al. 2007). While the MPTES includes many of these conventional components of capacity building, our approaches to each of these components has been innovative. For example, a typical method of training is a “train the trainers” approach (Taveras et al. 2007), in which the individuals conducting trainings are health department staff who have little or no first-hand experience with implementing the program (and also do not necessarily have experience with the target populations). In the MPTES, all the trainers were program staff who had coordinated the MPs during the trials and had implemented the program at CBOs subsequently, or the researchers who developed and supervised the original intervention. Hence the trainers had substantial experience with the program and could easily discuss implementation challenges and solutions. This is particularly important with an intervention that is not very scripted. But it may also have been important with respect to affecting the perceived credibility of the program by implementers. Indeed, there often was some suspicion at the start of trainings about the extent to which the trainers “really knew” about the intervention and the target population. As those concerns were allayed, it seemed that trainees were more receptive to learning from the trainers, particularly as adaptation issues are discussed and familiarity with issues facing young ethnic/racial minority men who have sex with men is displayed.

Another way that our program makes a contribution to HIV/AIDS capacity building concerns how to obtain TA. In the CDC approach to capacity building, CBO staff are required to officially request TA via a web-based request system (Taveras et al. 2007). While this approach may facilitate accountability, it relies on CBO staff (or a CDC Project Officer, i.e., the funder) to request TA, thus requiring staff members to publicly acknowledge a problem with program implementation to their funder. In some sense, this requires the CBO staff to “air its dirty laundry.” As we noted earlier, waiting for CBO staff to recognize that they have implementation problems and then asking for help can mean that assistance is requested far later than is optimal. Our proactive approach to delivering TA 1-2 times/month for each agency helped overcome these barriers to building capacity, and resulted in CBO staff members contacting us to request help. It also meant that rapport was developed over time, which may have enabled more disclosure about implementation problem areas. A long-term study conducted in a different area—the prevention and reduction of youth violence—has also found that ongoing, proactive TA is very helpful in helping organizations achieve fidelity of implementation (Spoth and Greenberg 2011). Similar to our approach, the PROSPER delivery system is a researcher-community partnership that links capacity-building agents (at universities) with community agencies. However, PROSPER is a developmentally based approach, in which the collaboration of the scientists and practitioners begins from the start-up of the program (adoption) to implementation and through sustainability of the programs. We are in agreement that a developmental approach makes sense, since we observed that some organizations found it challenging to move into full implementation of the MP (data not shown), but the design of our study did not enable this approach. Spoth and Greenberg believe that it is essential that the TA is provided in an ongoing way and proactively, which is what we found as well.

Additionally, dissemination/implementation materials are often developed by national publishers of educational curricula (e.g., Fuller et al. 2007; Galbraith et al. 2009), but in the MPTES, the original implementers and program staff developed all the materials ourselves, with the assistance of a graphic designer and an expert editor in adult learning theories. Hence the many lessons that were learned about challenges and solutions to implementation were incorporated into the materials from the outset. Furthermore, all the materials were available free-of-charge on the website so that they could be available electronically to any staff at the agency, regardless of the location of the hard-copy. Full packages, complete with manuals and videos, and printed in color, were provided at very low cost. Thus these methods can overcome difficulties accessing implementation material that has been identified as a barrier in technology transfer research (Veniegas et al. 2009).

Based on the many findings from these findings, we have since completely revised and updated the MPTES including the various implementation materials. These materials now include abundant implementation examples from community-based implementations of MPs from diverse communities around the country, the materials are now current with the state of the HIV prevention field (King et al. 2008), and substantial diversity of the populations implementing the program is depicted. More specific information on how to adapt the MP for various populations (particularly for young Black men who have sex with men) is provided, as is guidance about the adaptation process for staff who want to create adaptations themselves. The website has dropped some of elements that used to be on it (e.g., the chatroom), and many improvements have been made, such as including audio slideshows and more information about MPs around the country.