The Participatory Evaluation of Healthy Municipalities, Cities and Communities Initiatives in the Americas

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Activities addressing the basic determinants of health have increased significantly, yet social and economic inequities continue to erode health conditions for many population groups. This is the reason why health promotion must continue to focus on bridging gaps among and within countries. Creating a healthy and supportive setting, also known as the settings approach, continues to be the most used health promotion strategy. Proven, cost-effective health promotion strategies can protect and improve the health of all persons. Determinants of poverty and equity, and their influence on health can be addressed through creating sustainable public policies and laws, developing supportive environments, building public-private partnerships, strengthening networks, mobilizing the media and other means of communication, and promoting an active role of municipal and local governments in health promotion and development. This article describes experiences and summarizes the main lessons learned from the application of a participatory evaluation methodology to three Healthy Municipalities, Cities and Communities initiatives in Brazil and Peru.

The Healthy Municipalities and Communities (HMC) Movement in the Americas

The Pan American Health Organization (PAHO) defines health promotion based on the Ottawa Charter (1986), which states that health is “the process of enabling and empowering people to take control over and improve the determinants of health.” Health is promoted in the social context of people’s daily lives, and is supported by public policies that affect social conditions and life styles, and these in turn give shape to healthy behaviors. PAHO developed and introduced the Healthy Municipalities, Cities and Communities (HMC) strategy in the 1990 s to improve and promote local health and development. This strategy is being actively implemented in 18 of the 35 countries and three territories of the Americas.

The orientation of the Healthy Municipality, City or Community strategy is to ensure continuous improvements in the underlying conditions that affect the health and well-being of its members. This is achieved by facilitating joint action among local authorities, community members and key stakeholders aimed at improving their living conditions and quality of life. Based on the notion of health as having a good quality of life, the actions of the HMC strategy focus more on the underlying determinants of health than on their consequences in terms of disease and illnesses (PAHO, 2002).

HMC is based on the premises that (1) various systems and structures governing social, economic, civil and political conditions, as well as the physical environment, can impact individuals’ and communities’ health; and that (2) health is inherently linked to an individual’s capacity to act in the community and society to which he/she belongs. HMC strives to create a synergy between these two premises: promoting individual actions and society’s response. The ultimate goal is to enable processes that allow people to take control over their own health while improving equity, social participation, accountability and responsive local governance.

The evaluation of health promotion strategies, such as HMC, has been recognized by the international community as critical to strengthening the capacity of institutions and communities to promote measures that are effective and coherent with the needs and priorities of the population. Due to the complexities of evaluating social and developmental interventions, there is a scarcity of information regarding evaluation and effectiveness in developing countries. In addition, existing evaluation tools and methodologies have not appropriately captured changes in central health promotion elements (social participation, community empowerment, intersectorial collaboration, equity, etc.), nor have they provided insights into the multiplying effect of working with various determinants of health in a coordinated manner.

PAHO’s Evaluation Initiative

In 1999, PAHO established a Healthy Municipalities Evaluation Working Group formed by evaluation experts from leading institutions in the Region working on issues related to health promotion and local development. The Working Group was comprised of governmental, non-governmental and academic sectors from countries throughout the Region, including Argentina, Brazil, Canada, Chile, Colombia, Ecuador, and the United States. The Group met periodically for several years to develop tools and resources to support investment in health promotion. Drawing upon resources developed mainly by, English-speaking countries, the Working Group selected elements most relevant for the settings in the Region, as well as those reflecting the most relevant principles. The Working Group agreed that specific evaluation tools, frameworks and evidence of effectiveness were needed to support health promotion and similar initiatives. Building upon these recommendations, the Evaluation Working Group developed a series of evaluation tools, among them, a Participatory Evaluation Guide for Healthy Municipalities, Cities and Communities.

A participatory evaluation is considered to be an appropriate methodology because it reflects the principles of the HMC strategy, recognizes the complexities of HMC as a local development initiative, and facilitates the development of capacities, learning and empowerment. The process of conducting a participatory evaluation stimulates autonomy and community self-determination as it allows communities to improve their ability to resolve their own problems (PAHO, 2006).

In a participatory evaluation, the key stakeholders are involved in all phases of the process, including the design, implementation, management, interpretation, and decision-making about the evaluation and its results. The methodology implies more than the application of participatory techniques to conventional approaches for monitoring and evaluation. It requires:

• Participation of key stakeholders in all phases of the process.

• Negotiation and consensus about what to evaluate and how results will be interpreted and utilized.

• Continuous learning that results in capacity building and incorporation of lessons learned in the decision-making process.

• Flexibility to adapt to a continuously changing environment (PAHO, 2006).

The Participatory Evaluation Guide for Healthy Municipalities, Cities and Communities provides guidance and tools to HMCs to evaluate their own efforts, and contribute to the evidence base of the strategy’s effectiveness. The Guide provides recommendations on evaluation processes and tools, as well as a mechanism to showcase and document the rich, extensive, and varied experiences and results related to the HMC strategy.

The Guide offers an evaluation framework that incorporates essential elements of HMC, and other health promotion initiatives, such as intersectoral collaboration, social participation, capacity building, individual physical and material conditions, health determinants, and community capacity, among others. The Guide orients users through a step-by-step process to design and implement continuous cycles of monitoring and evaluation throughout the life of an HMC initiative. The methodology supports the documentation and analysis of changes and accomplishments in terms of processes, outcomes and results, and guides users on how to communicate and act on the results to improve their HMC initiative. A Facilitator’s Guide and training modules have been developed to support training activities in the application of the Guide.

When conducted in a truly participatory manner, the methodology proposed in the Guide promotes accountability and motivates continuous and active participation from all stakeholders. Since the participatory evaluation process is based on the commitment and dedication of all stakeholders, it is expected that the process will create a sense of common interest among those involved and produce positive changes in their community.

Nevertheless, as with most collaborative and participatory efforts, the participatory evaluation methodology has some inherent challenges. Bringing together and building consensus among people from various backgrounds, sectors, institutions and groups that often have different, if not conflicting, needs, agendas and interests, can be complex and time-consuming. The flexibility and openness required in the participatory evaluation process can also be perceived as less effective and objective compared to traditional approaches by those who are used to working with such approaches.

The Guide is a unique resource for the Region, providing an alternative orientation for evaluation that reflects the underlying health promotion principles of many long-term initiatives. Built upon philosophies of health promotion operating throughout the Region for decades, and incorporating additional elements from traditional evaluation models, the Guide affords an opportunity for HMCs to provide the information necessary to improve their initiatives while continually building capacity through participation. This dual approach has not previously been available to HMCs in the Region of the Americas.

Selected Countries’ Experiences with the Participatory Evaluation Methodology

During 2004–2006, the Participatory Evaluation Guide for Healthy Municipalities, cities and Communities was introduced into several countries in the Americas through formal trainings conducted by PAHO in collaboration with country partners. Following the trainings, several communities in various countries of the Americas applied the participatory evaluation methodology to their HMC initiatives. This section describes three experiences that took place in Brazil and Peru.

The Application of the Participatory Evaluation Guide in the Campinas Region, State of São Paulo, Brazil

Since 2003, the Network of Potentially Healthy Municipalities (NPHM) has been working with municipalities of the Campinas region, in the state of São Paulo, in the southeast region of Brazil, to support their efforts to implement the HMC strategy. The Network, spearheaded by the University of Campinas (UNICAMP), comprises over 30 municipalities, accounting for about two million inhabitants of the Campinas Region. The NPHM’s objectives are to (1) support, promote and monitor HMC strategies through the construction of public policies, participation from all sectors, and the development of sustainable initiatives; (2) support local governments in the implementation of integrated initiatives; and (3) improve capacity among managers, technical staff, academia, organizations and society that have as a common goal to promote improvements in quality of life through health promotion.

During 2004, a series of workshops were conducted to introduce the Participatory Evaluation Guide for Healthy Municipalities, Cities and Communities to mayors of the Campinas region. The main goal of these workshops was to introduce participants to the core concepts and methodology, and to discuss the applicability of participatory evaluation to some of the initiatives taking place in the region. Ten municipalities participated in this effort. Participants also included municipal managers and experts of various technical areas.

During these workshops the different sections of the Guide were presented followed by an in-depth discussion on how the concepts and steps proposed could be applied to the context of their HMC experiences. As a result, the workshops provided participants with a very rich opportunity to discuss, exchange and reflect on their experiences with the HMC strategy. Participants reported that the Guide concretely helped them to consider the consequences of their actions more consistently, stimulating interest in the issue of evaluation and an in-depth reflection about health promotion activities being implemented in their communities.

An interesting characteristic of these initial discussions was the myriad of interpretations given by participants of health promotion-related concepts presented in the Guide, such as intersectoral collaboration, participation, empowerment, etc. Acknowledging that the diversity of opinions could have direct implications for the planning and implementation of the evaluation process, participants engaged in a productive and positive dialogue to reach consensus on the interpretations given to the material. At the conclusion of the workshops, participants assumed the commitment to introduce the Guide to their communities, and, in collaboration with other community stakeholders, to develop and apply a participatory evaluation plan adapted to their local realities. In the following months, these participants reported back on their experiences and lessons learned from the application of the Guide.

One of the first observations made by participants when applying the Guide to their HMC initiatives was that the methodology proposed emphasized various aspects of health promotion that had been overlooked in previous evaluation efforts. This brought up a “difficulty” for the evaluation as participants realized that their HMC initiatives had not appropriately taken into account those health promotion elements during their planning stages (for example, programs were not intersectoral), and, therefore, presented a real challenge for evaluation. As a result, communities reported the need to first engage in a process of discussion and reflection on how to revise the planning and implementation processes of health promotion initiatives in order to more appropriately incorporate some of the missing health promotion principles and, in a subsequent phase, conduct an appropriate evaluation.

The political timing of the application of the Guide was described as sensitive, since it occurred right before the local elections. This generated uncertainty about who would still be present to follow up on the initiatives, or even if initiatives would continue. The municipalities’ experiences demonstrated how the transitory nature of local political decisions can weaken programs and public policies, particularly when there is a change of political parties. Such situations emphasize the need to form strong coalitions among all sectors of society to strengthen and sustain the HMC initiative and its evaluation.

The Campinas experience demonstrated that moments of political and administrative transition can have considerable impact on work conducted at the community level. It can cause delays and losses (including financial) since the time necessary to explain and get agreement on the continuation of programs can be very long. This has consequences for the evaluation of programs being implemented, and especially for programs conducted by a previous administration that are not continuing under the new one.

Another challenge faced by those applying the Participatory Evaluation Guide to Healthy Municipalities, Cities and Communities in the Campinas Region was the establishment of an intersectoral Evaluation Subcommittee, whose role was to take charge of the planning and implementation of the evaluation process. Particularly challenging were the cases in which municipal managers were not totally on board with the program and/or its plan of action. Lack of support from those in charge of the budget and general management of health promotion initiatives can seriously impair the ability of the remaining stakeholders to undertake the evaluation initiative. It also decreases the probability that the results of the evaluation will be taken seriously and utilized in the planning and implementation of future initiatives. This underscores the need to guarantee buy-in from all relevant stakeholders in order to launch and sustain the initiative, as well as the need to continuously promote awareness among all stakeholders of the objectives and benefits of the participatory evaluation.

Those participating in the Campinas experience also highlighted that the methodology presented in the Guide was new to most of the health secretariats of the municipalities. As a result there was a need to generate an internal orientation throughout public institutions to discuss the new concepts and assess the degree of agreement among staff in order to incorporate the new methodology into existing programs. This was particularly true when it came to generating intersectoral collaboration and guaranteeing social participation in actions and decision-making processes. This was a slow process, as it needed to take place through meetings, forums and discussion groups, and required linking various levels of administration, questioning existing paradigms, and dealing with resistance at both individual and collective levels.

Respect for the time needed to achieve acceptance by those working in public institutions was crucial in the Campinas experience in order to put in place programs that were consistent with the communities’ expectations. However, participating municipalities reported that patience is paying off and resulting in more optimal use of resources, adoption of more consistent health promotion practices, and improvements in personal motivation among public staff in the administration.

Application of the Guide in the Community of Vila Paciencia, State of Rio de Janeiro, Brazil

The HMC initiative in Vila Paciencia was launched in 1999 in a poor urban community located in the state of Rio de Janeiro, Brazil. The community context is one of great poverty and social vulnerability. Created in the 1960 s to shelter victims of a large flooding, the Vila Paciencia community remained as a relocation site for populations displaced from nearby shantytowns. Today, the community comprises 8000 people living under very precarious conditions with high unemployment rates, and within a context of marked repression of basic human and citizenship rights. Community life is permeated by drug trafficking, which often represents the main lifestyle alternative for unskilled and unemployed youth. Organized crime strictly controls access in and out of the community and leaves great stigma on its residents. Community organizations and the neighborhood association are weak and have not been able to become established in a sustainable manner. There has been little public investment in the area and a marked disinterest from the public sector to get involved with the community. Actions toward improving the community have been limited to those implemented by religious groups and NGOs.

The Vila Paciencia HMC initiative focused on developing community empowerment based on the principles of health promotion. The strategic approach was to incorporate the community’s inhabitants in the process of developing and improving health and quality of life through (1) mobilizing and strengthening local actors and leadership, and (2) creating a network of social intervention projects aimed at solving the most pressing community problems. This was implemented through participatory workshops to identify priority issues and develop action plans based on available resources; the application of a community survey used to create a database of basic community data; concerted efforts to involve various sectors (public and private) and other stakeholders working in the community; and the organization of community participation and strengthening of community associations. Five thematic areas were defined as the focus of actions to be taken: health and education; community organization; housing and sanitation; cultural and recreational activities; and generation of employment and income. Based on these themes, 41 intervention projects were implemented from 2002 to 2004.

The next phase for the HMC initiative was to monitor and evaluate the activities undertaken during those years. During 2005–2006, the Participatory Evaluation methodology was introduced and applied to the Vila Paciencia experience. The initiative’s intersectoral committee was not functioning due to the abandonment of its members resulting from their loss of interest in continuing to work in a community that did not receive sufficient resources from the local government. There were also difficulties in establishing linkages with the public sector, due to the community’s “historical social isolation and lack of citizenship rights”, which is marked by “structural oppression and violence.” (Becker et al., 2006)

The fragile community organization did not guarantee voice to its inhabitants and there was constant tension between community desires and the interests of those regulating the community life. As a result, formal participation from the community association became impossible, since the association’s president was “ousted” due to his “involvement in drug trafficking activities”, which generated mistrust within the population about the participation of the local association. (Becker et al., 2006)

Due to the complexity of the situation, a decision was made to evaluate the experience by reviewing and reflecting on actions already taken since 1999 and analyzing points of convergence with the five HMC pillars proposed in the Participatory Evaluation Guide to Healthy Municipalities, Cities and Communities as evaluation domains (participation, intersectoral collaboration, healthy public policies, sustainability, healthy structures and good governance). The work was conducted by a team composed of three psychologists, a social worker and a medical doctor who had been working in the community and participated in the planning and implementation of the HMC initiative. Later, participants from a local community committee joined the group. It was not possible to include a representative from the public sector, given the lack of interest it demonstrated towards the community.

The group formed an Evaluation Subcommittee to lead the evaluation process. However, it was not possible to form one that was representative of the various segments of the community due to the complications of the community context described above. For the same reason, the group decided that it would not be feasible to implement all the steps proposed in the Guide. The Evaluation Subcommittee decided to focus its work on evaluating three phases of the initiative that were planned and implemented with input from the community: Community Development, Local Action, and Community Mobilization. Community participation in these phases included the development of a community assessment, defining indicators and collecting the data. The efforts of the Evaluation Subcommittee focused on determining and analyzing points of convergence between the methodology, concepts and pillars of the guide with the actions implemented by the initiative. This was done through meetings and discussions among those participating in the Evaluation Subcommittee in which they analyzed the products of each of the determined initiative phases and the process that took place.

In terms of community participation, the group concluded that the community assessment demonstrated two forms of community participation: one based on personal gains and another based on leadership and voluntarism to achieve collective goals. In relation to the healthy public policies pillar, since the initiative was spearheaded by a civil society organization, working within the context of extensive social exclusion, the conclusion was that the initiative did not result in any contribution to public policies. As for the sustainability pillar, the group concluded that community actions taken in the context of the HMC initiative had favored the incorporation of various projects in the community as well as the allocation of more financial resources. Projects that resulted from the community assessment included various trainings and community development activities, particularly activities focused on children; a community kitchen with the creation of a women’s group to generate revenues; the organization of recreational activities for children, youth; development of a community committee that met on a monthly basis to discuss community issues. When analyzing the intersectorial collaboration pillar, the group concluded that through the Community Committee it was possible to incorporate representatives from the community and the university. Partners included: the local school and day care center, the neighborhood association, the Municipal Social Action Secretariat, a STD/AIDS organization, among others. As for the healthy structures and good governance pillar, the group found that as a result of the initiative, community residents had started to increase their participation (for example, creating a community kitchen, participating in health fairs, etc.).

The Vila Paciencia experience points out some of the many challenges that a participatory evaluation initiative faces due to contexts of social exclusion, poverty and violence in which the initiative might be immersed. However, as the Evaluation Subcommittee reported, the Participatory Evaluation Guide to Healthy Municipalities, Cities and Communities was useful in highlighting aspects of the initiative that were lacking or weak, such as participation from the public sector, and bringing about discussions on how to address the issue. The transitory nature of local political decisions and an agenda based on electoral priorities were some of the main factors inhibiting participation by the public sector. The public sector was present at the beginning of the initiative, but its participation faded through the years. Absence of this public sector’s contribution led to emphasis being placed on the process of getting the community organized.

Similar to the implementation of a health promotion initiative such as HMC, the application of a participatory evaluation methodology requires joint effort from all relevant stakeholders. This was one of the main challenges in Vila Paciencia. However, the process of applying the methodology proposed in the Guide demonstrated the difficulties in gaining representation of key stakeholders, which, in turn, stimulated the group to bring about change in this arena. In this case, use of the Guide in the evaluation process actually stimulated participation in the HMC initiative.

The Guide’s emphasis on intersectoral collaboration contributed greatly to the understanding of local politics and the role of different actors (public, private, community, etc.), constituting a reference for discussions and decision-making. In this initiative, intersectoral collaboration was difficult, particularly in relation to developing and maintaining the intersectoral committee and evaluation subcommittee. Upon realizing the challenge, the group decided to create a new intersectoral group centered on the construction of a new community kitchen. The new group includes representatives from the local public sector (municipal education and social development, state social assistance), and community members.

In this new phase the group decided to discuss the Guide’s proposed pillars indirectly, relating them to community issues. This was aimed at facilitating comprehension of the concepts by putting them into the context of a local practical experience, using the various implementation phases of the community kitchen project. New actors have demonstrated interest in being involved with the evaluation process and have participated in the monthly meetings. It is expected that working on the evaluation will help to strengthen the work of the new intersectoral group and support the sustainability of the Vila Paciencia initiative.

The Application of the Participatory Evaluation Methodology in Peru

The HMC strategy in Peru dates back to 1996 with the launching of the “Healthy Municipalities and Communities Declaration”, which was coordinated by the Peruvian Ministry of Health and the Pan American Health Organization (PAHO), and signed by various institutions, community-based organizations, civil society organizations, universities and local authorities. In 2003 the Peruvian Network of Healthy Municipalities was established with 29 municipalities, catalyzing the HMC process throughout the entire country. Today (2007) the network comprises over 130 municipalities and seven regional networks.

In 2004, a workshop was conducted in Peru to introduce the Participatory Evaluation methodology. Participants included technical staff from the Ministry of Health, municipal authorities and staff, health professionals, students, and representatives from NGOs and universities involved with the HMC strategy in the country.

As a result of this workshop, a series of activities and actions took place, such as the inclusion of a participatory evaluation module into the curriculum of the Health Promotion Certificate Program offered by the San Marcos University and the launching of a series of participatory evaluation processes by Proyecto Amares (a program supported by the European Community) in rural communities in Peru. In addition, participatory evaluation was incorporated into the Ministry of Health’s Healthy Municipalities Program, which defines the technical norms for the HMC strategy at the national level. The municipality of Miraflores also applied the participatory evaluation methodology to their “Tai Chi in the Parks” Program. The remainder of this section describes the process and the results of the Miraflores experience.

The Miraflores’ Experience: Participatory Evaluation of the “Tai Chi in the Parks” Program

The “Tai Chi in the Parks” initiative was implemented in the 1990 s as a public health strategy in the municipality of Miraflores, in Lima, Peru. Today, the “Tai Chi in the Parks” Association is responsible for the maintenance, improvement and advancement of the initiative along with Tai Chi Clubs and more than 20,000 elderly people who practice Tai Chi in the municipality.

The mission of the “Tai Chi in the Parks” Program is to transform Miraflores into the municipality with the healthiest and most active elderly population in Peru, thereby, promoting healthy aging of the population. The initiative’s main objectives are to incorporate the practice of Tai Chi and its philosophy as a daily, voluntary and accessible habit in the life of Miraflores’ elderly population; and to achieve physical, psychological, social and spiritual development of Miraflores’ elderly population through the practice of Tai Chi.

To this end, the initiative offered free Tai Chi classes during weekdays in various parks in Miraflores; supported the formation of Tai Chi clubs, which are informal Tai Chi groups, and developed a “Tai Chi in the Parks” network; promoted various community activities such as festivals, Tai Chi championships, conferences, etc.; and trained community elderly to become Tai Chi instructors, thereby increasing human resources necessary to expand the program.

During 2004–2005, the participatory evaluation methodology was applied to evaluate this experience. An Evaluation Subcommittee was formed to plan and implement the evaluation process. This group was comprised of technical staff from the municipality, the program coordinator, program participants and elderly members of the community who did not participate in the program. Initially, one of the main concerns of the group was to engage in an independent, empowering process that would not be dominated by the municipality and program managers. To address this issue, the group changed its name to “Evaluation Group” and determined by consensus who would be part of the group.

A series of meetings took place to introduce the evaluation methodology to all participants and to reach consensus on all its core concepts. These meetings were guided by a trained facilitator. Through weekly workshops, all of the Evaluation Group members were trained in the participatory evaluation methodology. Working with the elderly population was described as a facilitating factor since most of the participants were retired and had flexibility to participate in these initial meetings. The fact that most participants of the Evaluation Group were not involved with the health sector or were not health professionals was also a facilitating factor in these initial discussions. This allowed the group to be more open to explore issues related to social and psychological benefits of the program, and not to be focused on evaluating the health benefits of the program in terms of disease prevention.

Based on the process and the steps proposed in the Participatory Evaluation Guide, the group developed an evaluation plan. This included developing evaluation questions, reaching consensus on key concepts, and defining indicators, data collection methods and a work plan. The group requested that a representative from the Universidad Mayor de San Marco join the process in order to provide support and guidance related to the data collection and analysis processes.

When planning for the evaluation, the group came across a major issue, which was that the “Tai Chi in the Parks” Program had not been planned and implemented in a participatory manner. It had also not fully taken into account core health promotion principles (such as intersectorial participation), which posed a challenge in applying the evaluation framework proposed by the Participatory Evaluation Guide. However, engaging in the participatory evaluation process had the positive impact of highlighting these deficiencies and mobilizing the group to search for solutions. In order to address these issues, the group approached its problems from different perspectives and took into account the factors that might facilitate or hinder the participation of other stakeholders.

The Evaluation Group reported facing many challenges in its work. Some of the group’s participants demonstrated great resistance to the idea of implementing a participatory methodology due to ingrained concerns and negative ideas related to actions taken with the input of the community. There were fears of excessive criticism and an increase in “demands” by the community if offered the opportunity to participate. The data collection and analysis phase of the process suffered delays due to difficulties in coordinating the work with the technical staff from the Ministry of Health and the university, who were providing technical guidance in these matters. The Evaluation Group also reported initial discrepancies related to the various interpretations given by the group to the concept of health promotion and other core concepts related to the evaluation. In addition, turnover of key personnel in the municipality caused major delays in the implementation process.

Other challenges faced by the group included difficulties with data analysis (due to low technical capacity among participants) and lack of flexibility and openness on the part of some group members in listening and engaging in a true dialogue. Having a good facilitator was reported as a key aspect of the process to guide the discussion and help the group reach conclusions.

During site visits for data collection the group identified various issues such as difficulties with sound systems, and the need to limit the access of dogs to the parks during the Tai Chi classes. The group quickly passed this information on to those responsible for the program coordination, and they in turn were able to work with the parks management to solve these problems. Seeing the results of their efforts highly motivated the participants of the Evaluation Group to be more involved in the process. Many manifested an interest in evaluating other aspects of the “Tai Chi in the Parks” Program and learning more about the participatory evaluation methodology. As a result, a series of workshops were conducted to identify other key aspects of the program and to define priorities for the next round of evaluation. These workshops were organized by the Evaluation Group itself, which included a skilled facilitator to help participants identify the main strategic lines to be addressed. This turned out to be an important opportunity to bring together program managers and program beneficiaries to participate in the process. Dialogue and participation was also strengthened and provided a broader vision for the program.

The Evaluation Group devised two strategies to broaden the evaluation process: (1) to incorporate the San Marcos National University, to provide technical support in the processes related to the evaluation, and (2) to engage the current Evaluation Group in the evaluation of other programs aimed at the elderly population in the municipality. The participatory evaluation brought about significant changes in the way programs are planned in the municipality, particularly with respect to involving various stakeholders and sectors, and requiring participatory planning and evaluation as part of how programs are devised and implemented.

Discussion

Health promotion has advanced significantly in the past few decades, accompanied by an increasing interest in evaluating its effectiveness. Participatory evaluation holds great promise for helping to generate this evidence and promote understanding of the factors that affect, positively and negatively, the advances of health promotion in the Region.

The experiences described above highlight some of the challenges posed by the complex and multidimensional local and national contexts into which participatory evaluation is introduced. Factors affecting the success of evaluation initiatives were identified at various levels (individual, institutional, political, community, etc.). These factors intertwine and impact each other in very complex ways, a fact that was reflected in the municipalities’ experiences in implementing participatory evaluation.

Political context and timing were two of the main factors affecting the evaluation process. Given the strong emphasis of initiatives like HMC on the active involvement of local authorities and the public sector, election periods and political transitions can cause major delays (if not termination) of initiatives, shortage and/or change of personnel and funds, and great uncertainty about the future of the initiatives. This highlights the importance of establishing strong coalitions among all sectors of society to strengthen and sustain the HMC initiatives and their evaluation. A stronger and broader base of support can provide continuity and sustainability to such initiatives during these transitional periods.

Being able to work in a truly intersectorial manner poses another challenge for most health promotion initiatives and their evaluation, yet it is an important factor affecting sustainability of these efforts. Lack of support from critical stakeholders, such as municipal program managers or key personnel at public institutions, can seriously deter or isolate the advancement of the initiative. It can also jeopardize the possibility that the evaluation results will be taken into consideration by all relevant stakeholders, hence threatening the likelihood that results will be used to improve the initiative.

All experiences reported that the participatory evaluation process was lengthy and time consuming. This is due to various factors, such as bringing together a variety of stakeholders from various backgrounds, sectors and interests; reaching consensus on core concepts and paradigms; and working through institutions and organizations with rigid and bureaucratic structures and work cultures. The various levels of knowledge and literacy among those involved also affected the time it took to complete the process.

Recognizing the time needed for institutions, organizations and individuals to adapt and accept a new methodology is crucial. The implementation of a participatory evaluation often requires great changes in how organizations and institutions function and work. However, given the appropriate consideration and time, people become motivated and apply dedicated efforts to implementing these new programs and methodologies. Achieving this acceptance, particularly from public institutions and their staff is critical in order to put in place programs that are consistent with the communities’ expectations, make optimal use of resources, more effectively incorporate health promotion practices, and improve personal motivation among public staff and other stakeholders.

The experiences described in this chapter also reflected a general lack of understanding about the concept of health promotion (often considered an approach to disease prevention) and the participatory evaluation methodology. This can have a direct impact on the planning of the evaluation since how people understand key concepts will shape the design, data collection, analysis and presentation of results of the evaluation. The introduction of the Participatory Evaluation Guide to Healthy Municipalities, Cities and Communities can play an important role in addressing these issues by serving as a catalyst to engage people in a joint reflection and learning process.

Also reported were concerns related to the benefits of conducting a participatory evaluation, particularly related to the time it takes to conduct the process and the usefulness of the data it will produce. Resistance by key institutions to applying a participatory evaluation methodology was also common. It is important to address these concerns and take into account the challenges faced by stakeholders coming from institutions with rigid and bureaucratic structures, that often do not have a policy that enables or facilitates coordination with other institutions or working in an intersectorial manner. It is also important to take into account that these stakeholders are often under great pressure to produce results in a short period of time.

Concerns arose about working with the community, often expressed as fear of receiving negative comments, prejudice against actions taken with “too much” input from community members, and the possibility that the process would generate “unrealistic demands” made by community members. This can be particularly true of communities in which, traditionally, programs and approaches were implemented from the top-down and truly representative and participatory mechanisms for community participation were scarce or non-existent. However, the experiences described above suggest that the process itself of engaging in a participatory evaluation and having the opportunity to engage with other community stakeholders resulted in positive changes in attitudes and perspectives related to the potential of community participation.

Working with institutions with rigid and bureaucratic structures can also pose a challenge for participatory evaluation. Barriers include lack of institutional support or excessive bureaucracy, lack of coordination among public sector institutions, strict guidelines regarding the use of funds, and conflicts among the different actors involved (federal, state, municipal). High turnover of personnel at all levels and institutions can be particularly disruptive. Public sector personnel are frequently transferred to another state or unit/program within their institutions and it is often the case that in their new posts they are no longer in a position to follow through with the initiatives for which they were previously responsible. On the positive side of working with institutions, the process of engaging in participatory evaluation can open communication channels with other levels and sectors providing valuable inputs for the evaluation process, and clearing a path for exploring new modes of intersectoral collaboration.

Working with institutions also offers the opportunity to routinize processes and methodologies within work plans, programs, curriculums, etc. For example, the inclusion of the participatory evaluation methodology into the curriculum of the San Marcos National University’s Health Promotion Certificate Program, in Peru, provides an ongoing opportunity to build capacities and increase technical knowledge among professionals working in the field. Institutions can often have a far-reaching impact with the potential to promote and support the implementation of health promotion activities, their evaluation and the allocation of resources for these priorities.

Having strong, sustained and dynamic leadership is central to the sustainability of an evaluation initiative. Active commitment and engagement from institutions both at the local and national levels is key to the success of such initiatives, as is collaborative work among them. National and regional HMC networks can be central in these efforts, given their potential far-reaching connections to municipalities throughout a country or region, as well as their connection with key stakeholders that can support the evaluation process.

Conducting a participatory evaluation can be an empowering process by itself, as the experiences described demonstrate. Merely by engaging in the planning and implementation of the participatory methodology, communities and stakeholders were more willing and interested in participating and maintaining this participation. The process itself also provided a very rich opportunity to discuss, exchange and reflect on countries’ experiences with the HMC strategy. These discussions brought to light the various interpretations that participants gave to health promotion concepts and principles. This often resulted in a productive and positive dialogue among participants in order to reach consensus on the various concepts and principles utilized in their evaluation processes.

The Participatory Evaluation Guide to Healthy Municipalities, Cities and Communities was developed to respond to a direct need expressed by those implementing the HMC Strategy and other health promotion programs. However, once the methodology was made available and applied, most stakeholders reported not being ready to implement such an innovative approach to evaluation. Primarily, stakeholders came to a realization that their HMC and health promotion programs and initiatives had not appropriately taken into account key health promotion principles (such as intersectorial collaboration or community participation).

The application of the participatory evaluation approach made an important contribution to these initiatives, as it shed light on the gaps in their efforts and mobilized those involved to confront the problems and reflect on how to address them. This is stimulating many municipalities to review their planning and implementation processes in order to more appropriately incorporate health promotion principles. As a result, the community groups and institutions involved in this initiative are engaged in re-examining and reorienting their planning and implementation processes in order to more effectively apply the participatory evaluation methodology in the future. Thus, engaging in the participatory evaluation process has served as a catalyst to generate intersectoral and participatory processes essential to the development of HMC initiatives.

Participants in the three experiences described above reported that engaging in a participatory evaluation was highly motivating and revitalizing, concretely stimulating those involved to look at their actions more consistently, and promoting interest in the issue of evaluation. The participatory evaluation experience strengthened capacities among those involved, generated commitment to follow health promotion principles, strengthened alliances among key stakeholders, and emphasized the potential of participatory evaluation as a decision-making tool. As such, participatory evaluation holds great promise for contributing to the advancement of health promotion in the countries of the Region.