The largest children’s mental health project ever conducted is the “System of Care Initiative” or the Comprehensive Community Services for Children and their Families Program (hereafter referred to as the Children’s Services Program or CSP). The CSP requires the participation of caregivers of children with mental health problems (hereafter referred to as “caregivers” or “family members”) in conducting research and evaluation (Jivanjee and Robinson 2007; US DHHS 2003). However, the few publications available on the success of this collaborative practice indicate that the inclusion of families in decision making can be less than fully authentic, as decision-making power may reside primarily with the professional researcher, family members may feel less involved than professionals believe them to be, and family members may feel intentionally or unintentionally exploited by researchers for their knowledge with little reward for themselves (Jivanjee and Robinson 2007; Osher et al. 2001; Slaton 2004). Research also suggests that few evaluators at CSP grantee communities have had experience in conducting participatory research or evaluation prior to their work in the CSP (Jivanjee and Robinson 2007; Koroloff et al. in press). The CSP evaluation may be missing an opportunity to harness family participation to create transformative and beneficial systemic change.

Theorists and researchers in the field argue for participatory methods in children’s mental health for epistemological and social justice reasons (Jivanjee and Robinson 2007; Koroloff and Friesen 1997; Osher and Osher 2002; Turnbull et al. 1998). They believe caregivers have different access to knowledge and forms of knowledge about children’s mental problems, and they try to correct what they see as the unjust historical exclusion of families from positions of power and decision making in research. Feminist activists have long been critical of practices that “add women and stir,” arguing that involving women in systems that are male constructed and male dominated is inauthentic and sometimes detrimental to the women who participate (Harding 1995). Similarly, I believe that an “add families and stir” approach to involving families in research on systems of care may also be detrimental. Authentic and effective caregiver participation requires deconstructing of the dynamics of power, ways of knowing, and types of knowledge that shape research, and then reconstructing the research process to include the strengths, experiences, knowledge, and power of families. This article presents a brief theoretical context for participatory research, presents a framework for power and its relationship to participatory approaches, critiques the CSP within that framework, and offers practical improvements suggested by the framework. This critique is offered to help the CSP more appropriately engage the strengths of families participating in research and better achieve the primary goal of the CSP, which is collaborative, transformative system change.

A Brief History of Family Participation in the CSP

Over the past 20 years, a dramatic paradigm shift occurred in federal mental health and educational service agencies’ attitudes towards caregivers and family members of children with serious emotional, behavioral, and mental health disorders. Moving away from blaming caregivers for their child’s mental and behavioral health problems, newer federal priorities reflect a commitment to the inclusion and participation of caregivers in all levels of child-serving agencies (Osher and Osher 2002). This includes family participation in roles like service provision, policy making, advocacy, mutual aid and support, and research and evaluation (McCammon et al. 2001; Osher et al.1999). Several documents reflect this priority shift, including the National Agenda on Children’s Mental Health (US Public Health Service 2000), the 1997 amendment to the Individuals with Disabilities Education Act (US Department of Education 1998), the Surgeon General’s Report on Mental Health (US Department of Health and Human Services 1999), and the President’s New Freedom Commission on Mental Health (2003).

Family involvement is also one of the guiding principles of the CSP (Stroul and Friedman 1986). The CSP requires family involvement in research and evaluation (Jivanjee and Robinson 2007). Grantee communities in states, territories, Indian tribes, counties, or regions receive funding from the US Substance Abuse and Mental Health Services Administration (SAMHSA) to restructure their service systems to match system of care values and principles. These values and principles have been described in detail elsewhere (e.g. Stroul and Friedman 1986), but they include the core values of being community-based, child-centered, family-focused, and culturally competent. The principles assert that children and families should have access to an array of services that are comprehensive, coordinated, individualized, flexible, and provided in the least restrictive environment possible. Services for youth who are transitioning into adulthood should be seamless. Most importantly for this paper, a principle states that families should be full partners at all levels of the system of care. Consequently, CSP grantee communities encourage a high amount of involvement by caregivers in steering committees, policy boards, service delivery, and evaluation activities. For instance, some communities require that policy boards are made up of 51% family members or more.

Since 1993, SAMHSA has funded 126 communities at a total cost of $1.1 billion (http://systemsofcare.samhsa.gov), making the CSP the largest children’s mental health project ever conducted. Each grantee site is required to commit a large part of their funding to research and evaluation. These activities are generally coordinated by a national evaluation team that has determined an extensive research protocol for grantee communities to follow. Since the late 1990’s, the CSP has encouraged grantee communities to involve family members fully in research and evaluation activities such as evaluation planning, data collection, interpretation of findings, and dissemination. Since 2003, family involvement in evaluation is required instead of just encouraged (Jivanjee and Robinson 2007; US Department of Health and Human Services 2003). Though the government’s motivation for making this a requirement is unclear, it may be in response to the application of pressure by advocacy groups or federal policy makers’ increased awareness of this issue. This evolution toward family participation in evaluation and research is an essential point to consider. The evaluation of the CSP was not begun as a community based participatory process, and family participation in the evaluation was added after its inception. The CSP has taken large steps to incorporate a participatory approach, and now it may be time to consider the opportunity to take the next step.

Grassroots family advocacy organizations have promoted family involvement in all aspects of the system of care, including research and evaluation. Specifically, the Federation of Families for Children’s Mental Health (2006, 2007) designed and implemented a training series on evaluation for family members in order to introduce them to the world of evaluation. Additionally, this organization has produced white papers interpreting SAMHSA’s requirements and providing a framework for involving families and youth in evaluation (2006). Advocacy groups are motivated by a desire for a more equitable distribution of power and resources to those who are served, a strong belief that they have knowledge and understandings of the issues that are not easily tapped by traditional approaches to research, and a desire to gain skills and build capacity within local organizations.

Behind these top-down and bottom-up pressures is the broader historical and theoretical context that shapes inquiry. There has been a movement toward more democratic modes of inquiry that involve communities and stakeholders at all levels (Jivanjee and Robinson 2007; Minkler and Wallerstein 2003). Participatory Action Research (PAR) is an approach to research consistent with these goals and has been endorsed as a model for conducting community based children’s mental health research. Turnbull et al. (1998) present PAR as a process of collaboration between researchers and stakeholders in all phases of research, including decision making, formulation of research questions, collection of data, interpretation of findings, and dissemination. They believe PAR links research findings with the community, increases the relevance of research in the community, increases the rigor of research, increases the longitudinal involvement of families in the research, decreases logistical problems with working in the community, increases utilization of the research, and empowers the community. In practice, PAR research represents shared power between the community and researchers. While PAR often emphasizes participation of the geographic ‘community,’ these theories and practices are also applicable to communities of shared interest, such as family members with children with mental health problems.

Some evidence suggests that many children’s mental health researchers at CSP sites may not have knowledge or training on how to conduct participatory research (Jivanjee and Robinson 2007; Koroloff et al. in press; Vander Stoep et al. 1999). In a study of CSP site evaluators with reputations as effective collaborators with families, over half reported having no academic training on family participation in research (Jivanjee and Robinson 2007). Over two-thirds reported they had learned on the job, sometimes through “bitter experience,” as one person commented (Jivanjee and Robinson 2007, p. 8). In a report on family and evaluator collaboration in one study, the authors confessed, “[the evaluation team] designed the evaluation process long before any of its members had read the literature on the family support or participatory research movements,” (Vander Stoep et al. 1999, p. 343). It is clear that attempts at collaboration in research in children’s mental health have sometimes been conducted without awareness or training in PAR.

While the CSP requires family participation in research, evaluators may be reluctant to involve families for a number of reasons. Foremost may be a lack of experience and knowledge on how to do it. Even for those who do have some knowledge, implementing an effective participatory approach may be frustrated by the context of the system being studied and the power relations within that system. Below I provide an overview of the theoretical and historical context for participatory research. I also discuss the relationships between power, knowledge, and participatory methods, and I use these relationships to shed light on evaluation practice in the CSP.

A Brief History of Participatory Action Research

The roots of PAR are inextricably tangled in various disciplines, including social psychology, community psychology, community organizing, community development, anthropology, adult education, social work, and program evaluation (Wallerstein and Duran 2003). There are two broad classes of participatory research approaches which have generally inconsistent labels in the literature (Brown and Tandon 1983; Starrin and Svensson 1991; Wallerstein and Duran 2003; Williams 1999). Both embrace values that make them “Participatory Action Research,” including (1) the democratic inclusion of stakeholders in decision making, including those traditionally excluded from research, (2) the generation of practical knowledge, (3) the importance of developmental changes in systems to make them more useful, (4) a continual process of communal reflection and action, and (5) the co-learning of professional researchers and participating community members. In addition, both of these types share the belief that objective research is a myth and that researchers should be open about the values that shape their work. However, these types differ in ideology (Brown and Tandon 1983). Table 1 outlines the major differences between these two traditions, and these are described in detail below.

Table 1 Differences between the utilitarian and liberatory traditions of participatory research
Full size table

The first broad tradition has been termed participant action research (Chein et al. 1948), action research (Brown and Tandon 1983), the utilitarian tradition (Starrin and Svensson 1991), practical participatory evaluation (Cousins and Whitmore 1998), collaborative participatory research (Williams 1999), and the Northern tradition (Wallerstein and Duran 2003). The utilitarian tradition, is focused on collaborative work with key stakeholders and aims for practical system improvement, as depicted in Table 1 (Brown and Tandon 1983; Wallerstein and Duran 2003). Researchers in the utilitarian tradition view human systems as rational and reliant on scientific knowledge to solve problems. They believe that no parties should be excluded from decision making (Wallerstein and Duran 2003), the research should serve the cooperating groups (Williams 1999), and the parties involved will work in good faith to achieve consensus. This tradition springs out of work in the United States on racism (Chicago Commission on Race Relations 1919; Lippitt and Radke 1946).

Research conducted through a utilitarian participatory lens often focuses on the participation of stakeholder groups in the collection and use of data for system improvement. It assumes the political system for which the research is being conducted is open to evidence-based decision making, and all parties—from the least to the most powerful—can work together on a relatively equal footing. It takes a moderate view of action, and the professional evaluator is seen as the expert (Cousins and Whitmore 1998).

The second broad approach argues that few systems are non-hierarchical, decision making is often based more on politics than on evidence, and “consensus” often means manipulated agreement to other’s interests (see Table 1). This approach has been termed participatory action research (Brown and Tandon 1983), the radical tradition (Starrin and Svensson 1991), transformative participatory evaluation (Cousins and Whitmore 1998), liberatory participatory research (Williams 1999), and the Southern tradition (Wallerstein and Duran 2003). I will use the term liberatory research.

Liberatory research springs out of work with oppressed peoples in impoverished countries in Latin America, Africa, and Asia, and the worldwide social upheavals in the 1960’s, which prompted intellectuals to search for new ideas and theories promoting equality and democracy (Hall 1975; Wallerstein and Duran 2003). Many postmodern and post-positivist thinkers believe that the world is impossibly complex, the search for intrinsic orderliness is futile, and objectivity is an unattainable goal (Wallerstein and Duran 2003). They endorse PAR for practical, epistemological, and political reasons, arguing that it can better describe the complexities of the world, balance a lopsided production of knowledge, and motivate the oppressed to act for change (Fals-Borda 2001; Gaventa and Horton 1981; Rahman 2001). They believe that a fundamental structural transformation of hierarchical power relationships in research would involve those that have been excluded from the production of knowledge, help liberate the oppressed, and motivate action (Brisolara 1998; Fals-Borda 2001; Freire 1970; Hall 1975, 1981). Because of this focus on power and hierarchy, liberatory approaches are more conflictual than utilitarian approaches, arguing that in some situations consensus is impossible (Gaventa and Horton 1981).

These two traditions are not necessarily exclusive. A utilitarian approach is appropriate when working with relatively flat systems with broad cooperation, a fairly equal distribution of resources, a great deal of mutual trust, and generally widespread consensus on goals (Brown and Tandon 1983). A liberatory approach is more appropriate in the converse—hierarchical systems with an unequal distribution of resources. Both emphasize that stakeholder participation is essential to generating accurate and useful knowledge, and that the knowledge production system should be reformed to be more just and power balanced. As participatory methods have become more common, however, researchers and participants alike may find the process equally as problematic and power laden as previous research paradigms. Due to the subtle and pervasive influence of power, participation in both traditions can become less than authentic.

Authentic participation

A danger with participatory approaches exists in the “authenticity” of participation (Lopez and Kreider 2003). Rodwell (1998) discusses five types of authenticity in participatory research: (a) a fair representation of all viewpoints, (b) an increase in participants’ awareness of other’s viewpoints, (c) an increase in participants’ awareness of the values and motivations behind other’s viewpoints, (d) research operating as a catalyst for change, and (e) a redistribution of power among stakeholders so that participants are better off for having participated. Inauthentic participation is characterized by the intentional or unintentional exploitation of the people that PAR is purporting to serve (Arnstein 1969; Cooke and Kothari 2001; White 2002). Despite good intentions, PAR projects are often begun by traditional academic researchers with a research orientation at odds with participants’ interests and thus threaten authenticity.

Koroloff and Friesen (1997) present an excellent picture of the challenges that arise between traditional research and family centered research. To a traditionally trained researcher, “beneficial” children’s mental health services research means the research contributes to an academic body of generalizable knowledge about effective services, as determined by sampling methods, measures, and clinical outcomes. For families, “beneficial” research contributes to local knowledge, the best possible functioning as determined by the child and family, and treatment that is specific to the local context. Well-intentioned researchers may honestly tell a community that their participation will be beneficial, while satisfying few or none of the expectations of the community and delivering results of little to no direct, local use. Some researchers or funding agencies may manufacture consent through a bait-and-switch of promising participation and then using the participatory process to engineer support for certain policies or to placate the community (Arnstein 1969; Conchelos and Kassam 1981; Cooke and Kothari 2001; Hall 1981). Inauthentic participation also results when communities do not want to be involved in research (Cornwall and Jewkes 1995), which has been a challenge in some CSP grantee communities (Federation of Families 2007).

As the idea of participatory research gains momentum, it is increasingly being scaled-up to larger projects at national levels, including the CSP. However, when participation is adopted as a goal by large institutions such as federal agencies, its authenticity is likely to be questionable (Gaventa and Cornwall 2001). PAR is ideally a grassroots effort in which communities partner with researchers to find local answers to problems. The question that arises is, can participation be authentic when it is mandated as a top-down requirement over a widespread area rather than emerging from a community? When federal grants require a participatory approach, traditionally trained researchers, who are those most likely to receive federally funded work, may promise something more than their values, training, and experience can facilitate.

Additionally, authenticity is impacted by an interaction between the context of the system and the choice of utilitarian and liberatory approach. If a utilitarian approach, which assumes a relatively flat system of rational decision makers, is applied to a system with large differences in political power, then participation may be inauthentic or even manipulative, and the benefits purported to spring from participatory research may be lost. Families served in the CSP are often marginalized by poverty, low education, and lack of employment. They are enlisted to participate in a traditional research paradigm of data collection through quantitative surveys, analysis, and dissemination. It is dangerous to “add families and stir” without addressing the concerns of the liberatory tradition, which focuses on the systemic power dynamics and traditional epistemologies that shape research. And, as I describe below, authentic participation is more likely to lead to community mobilization and collaborative, transformative system change, which is the primary goal of the CSP.

We might question whether much of the research as currently required by SAMHSA and conducted in many of the CSP grantee communities is authentically participatory, because (1) it was largely planned by experts on the national evaluation team, (2) it generally focuses on developing a traditional body of knowledge, (3) it assumes cooperation and collaboration among the various parties, and it deemphasizes the often extreme hierarchical discrepancies in power, values, and goals, (4) it assumes that decision making will be data based and rational, and (5) most grantee communities focus on meeting federal requirements for data collection rather than utilizing evaluation data or developing a comprehensive local evaluation for assessing and improving system performance (R. Paulson et al., unpublished manuscript). This is most likely due to the evolutionary nature of the CSP evaluation, which began to incorporate family participation after pressure from family advocacy groups rather than taking a participatory approach from its inception. Below, I argue that the evaluation of the CSP now has the opportunity to authentically embrace participation in order to stir system change.

As Slaton (2004) pointed out, power relationships are at the heart of authentic participation, “One of the biggest challenges to successful partnership between families and evaluators—to real family engagement in evaluation—are issues of power” (p. 5). The CSP is just the type of situation—a utilitarian participatory research approach applied to a hierarchical system in which families have less power than all other stakeholders—that is likely to lead to inauthentic participation and a lost opportunity to take advantage of a useful tool for transformative, empowering systematic change. Therefore, an examination of power is in order.

Power/Knowledge

The concepts of knowledge and power are tightly intertwined (Foucault 1980; Gaventa and Cornwall 2001; Williams 1999). A multidimensional view of power/knowledge and its relationship to PAR was outlined in detail by Williams (1999) and Gaventa and Cornwall (2001), based on work by Park (1993), Lukes (1974), Foucault (1980), Freire (1970), and others. In this paper, I define oppression and liberation in specific ways. Oppression is using power to maintain or increase an inequitable distribution of resources and knowledge at the expense of the less powerful. Liberation is using power to reduce inequalities, restore social justice, and build on community capacity. Oppression and liberation can be intentional or unintentional. However, given that the dominant cultural paradigm favors the status quo power hierarchy, my assumption is liberation is often intentional, and oppression often unintentional. Traditional research can be unintentionally oppressive because it is a product of and by the status quo, and the status quo generally favors existing hierarchies. Below, I briefly describe the multidimensional view of power/knowledge as espoused by Williams (1999), and Gaventa and Cornwall (2001), and examine oppressive and liberatory expressions of power/knowledge in children’s mental health research.

Representative knowledge

Table 2 depicts three types of power/knowledge. First, and most familiar to westerners as “knowledge,” is representative knowledge, which explains the world through the development of predictive theories, focuses on objective data, and aims to measure and/or produce something tangible. Traditionally trained academics highly value representative knowledge, and many believe research should begin and end with its production. The primary focus of the research and evaluation activities in the CSP is on the collection and use of representative knowledge. This type of knowledge is best utilized if systems are open, democratic, and rational, so that the uncovered facts can be weighed freely during decision making. Both the utilitarian and liberatory traditions utilize this type of knowledge, but it is emphasized more often in the utilitarian tradition. The power corollary to this form of knowledge is advocacy, a decision making type of power. Power in this context is expressed by who wins and who loses rational debates in open decision making forums. While representative knowledge and advocacy can be effective in stimulating change in power relations, to do this requires a collective pursuit based on the needs of the community (Williams 1999) and a place at the decision making table.

Table 2 Types of power/knowledge and examples in children’s mental health
Full size table

Relational knowledge

Relational knowledge derives from a sense of common experience, shared understanding, collective action, history, tradition, and culture (Williams 1999). While representative knowledge is about the what, relational knowledge is about the who: who is involved, whose perspectives are considered, and who is making decisions. While both traditions of PAR include relational knowledge, the liberatory tradition focuses on it more. In the CSP, relational knowledge is the heart of family organizations such as the Federation of Families for Children’s Mental Health (Osher and Osher 2002). However, the research component of the CSP may fail to formally acknowledge and utilize this relational knowledge.

In this dimension, power is expressed as organizing or mobilizing. This can be used oppressively through “nondecision making,” where conflicts do not arise due to the intentional exclusion of certain people (Stone 1980), often because many decision-making systems are not open and are based instead on hierarchical power relationships. In research, this oppression occurs when academics do not involve the people whom they study in formulating research questions or interpreting findings. The academic environment often systematically excludes people without academic credentials. Even when academic researchers want community participation, involvement may be limited by university hiring policies and pay scales that devalue life experience in favor of formal education (Koroloff et al. in press; Slaton 2004).

Reflective knowledge

The third dimension of knowledge/power focuses on reflective knowledge, which is based upon people’s awareness of a problem, their ability to reflect on it, and a consciousness of how their struggles are related to broader societal structures. It relates the personal to the political. Whereas the other two dimensions of knowledge/power were about the what and the who, this dimension is about the why and how. The power corollary is control over consciousness, or altering how people think about problems, and the “whys” people attribute to problems. This type of knowledge is the paradigm through which we interpret the world. This power is exercised through education, awareness building, socialization, and the media (Gaventa and Cornwall 2001). As with relational knowledge, it is the liberatory tradition of PAR that focuses the most on reflective knowledge. In reflective knowledge, too, oppressive power is expressed through nondecision making (Stone 1980) but in a more insidious way. Instead of preventing certain actors from reaching the table, the powerful create a consciousness that prevents people from realizing there is a problem in the first place. Through controlling the framing of an issue, the powerful may set up the terms of debate in oppressive ways.

Example of a Liberatory Approach in Children’s Mental Health

The liberatory and utilitarian approaches are theoretical ideals, and as such are unlikely to be perfectly embodied in the real world. However, an example may help illustrate how a liberatory approach to participatory research can utilize all three types of power/knowledge. Friesen and colleagues offer insights into the use of representative, relational, and reflective knowledge as a liberatory force when describing their study of the “custody problem” (Friesen et al. 2003, p. 39). The custody problem refers to the practice in many states of requiring parents to relinquish custody of their child in order to access and pay for out-of-home mental health treatment. These caregivers are not abusive or neglectful; they simply do not have enough money to pay for their child’s treatment. Families who relinquish custody are frequently excluded from decision making in their child’s treatment. In some tragic cases, families have been unable to reclaim custody of their child.

The fact that the custody problem persisted for so long may have some basis in the traditional approach to research on families with children that have mental health problems. Historically, children’s mental health research has produced knowledge about “ineffective parenting,” “mother’s affective dysfunction,” “coercive parenting,” and “inconsistent discipline” (Friesen et al. 2005). Framing problems in this way may be an oppressive use of reflective knowledge, serving to reinforce negative stereotypes about the weaknesses of the people being studied, often at the expense of investigating ecological factors that may be of more interest to families with children that have emotional and behavioral problems, such as poverty, systemic barriers to care, and insurance parity.

Beginning in the early 1990’s, family advocates at the Oregon Family Support Network (OFSN) approached researchers at the Research and Training Center on Family Support and Children’s Mental Health at Portland State University (RTC) seeking help in addressing the custody problem. These organizations completed a collaborative, participatory research project uncovering facts on the custody problem’s prevalence, consequences, cost to the state, and potential solutions. Interviews with caregivers who had faced the custody problem captured detailed accounts of their experiences, connected caregivers who had common experiences, and identified families willing to advocate for change.

Organizing and advocacy by the OFSN built enough social power to gain access to the state legislature. In a series of legislative hearings, advocates—including some who were new to advocacy and had been identified through the research process—successfully lobbied the state to change its policy so it no longer required custody relinquishment in order to obtain mental health services. The OFSN and RTC combined the data they had collected (representative knowledge) with the personal stories, experiences, interpersonal connections, and perspectives of caregivers (relational knowledge), raising the consciousness of themselves and the state legislature on how the custody problem was perceived and should be addressed (reflective knowledge).

As Friesen et al. (2003) point out, data alone are rarely enough to create change. Change requires advocacy and organizing, the integration of the three types of power and knowledge. Through the creation of relational knowledge, caregivers came to understand that their tragic experience of custody relinquishment was shared by other families. They exercised this relational knowledge and power through the organization and mobilization of caregivers to enter the political arena.

Several frames of consciousness may have prevented the state from addressing the custody problem prior to this episode. These could include a historical blaming of caregivers who have a child with emotional and behavioral problems, a belief that parents should be solely responsible for their children, and a deep suspicion of caregivers in contact with child welfare. The OFSN and RTC raised consciousness through several mechanisms. Family organizations—through relational knowledge, shared connections, and common experience—reflected on, framed, and communicated the issue to researchers, politicians, and the general public. When family advocates presented their own, often heartbreaking, stories about custody relinquishment to politicians, these stories were framed as a tragedy rather than due punishment for bad parenting. They framed mental health problems as analogous to physical health problems, equally deserving of treatment. The child welfare system was framed as a rigid bureaucracy stuck in old policies that no longer made sense.

In liberatory approaches such as this one, those who have been traditionally excluded from knowledge production are authentically participatory. Family members determined the research questions, participated in data collection, helped to analyze the data, and took the lead in disseminating the findings in a way that would create change. Organizing was necessary in order to surmount marginalization and gain access to the legislature. The research was directly useful and participants were better off from having participated.

Children’s System of Care: Current Participatory Research Practice

As depicted by the example above, a liberatory approach to PAR includes all three forms of power/knowledge in a continual, reflexive process of investigation, education, and action. This shows that participation can move beyond the “added value” model of the utilitarian tradition to a liberatory, transformative approach that builds and engages social power (Speer and Hughey 1995). By requiring family participation in research, the CSP has an opportunity to harness power and knowledge for maximum positive impact. Whether the CSP takes advantage of this opportunity is unclear, although a few studies have examined family participation in evaluation in the CSP.

Koroloff et al. (in press) present a case study of one CSP grantee site. This study could be best described as a utilitarian approach to participatory research with some liberatory aspects. On a continuum of collaboration, with professional control of research on one end and family member control on the other, the system of care evaluation of this site is described as largely professionally-led. The national evaluation team developed the study’s principal research questions, design, and instruments with a relative absence of caregiver involvement. In this utilitarian approach, the professional evaluators were seen as the experts. Family member involvement, while deemed important, was generally focused on data collection. Using the research process as an organizing tool or a tactic to raise the consciousness of system partners, as would be appropriate in a liberatory approach, took a much smaller role in this example. Much of the case study is spent describing the benefits of family participation with an emphasis on the development of representative knowledge (the principle concern of a utilitarian approach), including a belief that the data collected was more accurate and that participation legitimized the research to the community of family members. However, they do describe authentic, liberatory involvement in some aspects of the research; for instance, family members were described as taking the lead in developing a new questionnaire to collect data on a topic in which they were interested. This report makes it clear that this CSP site took large steps to be authentically participatory, but due to the constraints of being part of a national evaluation this was not always possible.

Other research supports the notion that the utilitarian approach to participation is common throughout the CSP. Osher et al. (2001) explored the involvement of families in research at several CSP grantee communities. They asked participants, including professional evaluators and family evaluators, nine dichotomous questions about the involvement of families in tasks such as data collection, questionnaire modification, and the review and utilization of data. There was general agreement between family members and evaluators on the roles family members had taken in the research. However, only half of family evaluators reported they participated in the review and utilization of data, whereas 82% of professional evaluators reported that family members had taken part. PAR, particularly in the liberatory tradition, is highly focused on utilizing data to advocate for system restructuring. The fact that evaluators at system of care communities believe family evaluators are more involved in the actual use of the research data than family evaluators believe indicates that participation at CSP grantee communities may be less than fully authentic.

In another study, Jivanjee and Robinson (2007) interviewed both professionals and family member evaluators from several CSP grantee communities who had been nominated by experts as effective collaborators. They found that family evaluators served in a variety of evaluation roles, including data collection, questionnaire design, training, data analysis and interpretation, and presentation. The most common role for family members was data collection. They also found a wide range of family member influence. In some grantee communities families had very little influence and worked principally as interviewers. At other grantee communities, family members were described as co-directing research. Participants noted that researchers would often promise more than the evaluation budget would allow, resulting in the tokenization of family evaluators and inauthentic participation. Others described a tension between advocacy and objective research; family members were frustrated by not being able or allowed to utilize findings for system improvement. This research suggests that while some CSP sites are taking steps to authentically involve families to the point of co-directing research, which is a move toward a liberatory research approach, many sites “add families and stir,” by incorporating caregivers’ participation into a traditional research paradigm.

Discussion and Implications

The findings described above indicate that professional evaluators tend to retain much of the power and control over the evaluation of the CSP—the national evaluation team decided the principal research questions and instruments for all sites in relative absence of family involvement (Koroloff et al. in press), the most common role for family members has been data collection rather than directing the research (Jivanjee and Robinson 2007), and caregivers have more often involved in roles with less decision-making authority (Osher et al. 2001). This may be a result of the evolutionary nature of the CSP evaluation. Although the evaluation has made great strides in incorporating family participation in evaluation, it may now be time to take the next step towards authentic participation.

In this paper, I argue that an authentically participatory, liberatory approach would add value to the CSP evaluation in that it would be more likely to ensure community buy-in, create a critical consciousness in the community, and mobilize the community for truly collaborative, transformative system change which is, after all, the primary goal of the CSP. The question then becomes, is it even possible to implement authentically participatory, liberatory PAR in the current context of the CSP? Research in the CSP utilizes a utilitarian participatory approach in the context of a hierarchical system, with researchers who are often unfamiliar with participatory methods and burdened by a centralized, predetermined, lengthy research protocol. Viewing this through the framework of power/knowledge, this context emphasizes developing representative knowledge using traditional forms of knowledge generation. Improvements in the system of care resulting from this approach to evaluation are unlikely due to the lack of emphasis on relational and reflective knowledge. In other words, the utilitarian approach as used in the CSP is out of balance, too focused on representative knowledge, and needs the incorporation of relational and reflective knowledge as embraced by the liberatory tradition. Three general tactics are suggested by this framework.

It may appear that a first step would be training families in evaluation techniques. Training families in, and utilizing their talents and experiences for, the production of representative knowledge is an admirable goal, but it may miss a valuable opportunity to fully utilize the strengths of the family movement. The strengths of family advocate groups reside in developing relational knowledge and the sharing of lived experiences, in other words, community organizing. Unfortunately, evaluation training for families may merely be a repeat of the same traditional, oppressive assumption that families have a deficit (i.e. lack of knowledge about research) that must be filled by those with more power (i.e. knowledgeable researchers). Rather than focusing on making family members more like researchers, perhaps we should consider the strengths of the family movement and, in a sense, make researchers more like family advocates.

Hence, the first of the three changes I recommend is to train researchers in the power corollaries of relational and reflective knowledge, which are community organizing and adult education. Ideally, successful and effective family advocacy groups should conduct the trainings. This represents the co-learning process inherent in participatory methods, and aims to introduce researchers to the goals of participatory research, give researchers the skills of family advocacy organizations, and place power in the hands of advocacy groups in the form of a trainer-trainee relationship.

While evaluator training is an important step, I expect that it would have little impact without systemic changes in the CSP because researchers are still limited by a centralized and predetermined evaluation protocol. Therefore, the second recommendation is to simultaneously reduce the burden of the national evaluation through a careful elimination of survey instruments, and increase the grant requirements for development of a local evaluation in conjunction with and approved by local stakeholders, especially family advocacy groups. I expect that this change would push grantee communities into a more liberatory tradition, as it compels communities to include families on the front end of determining the research question, localizes the research process, frees up resources for conducting the evaluation, and reinforces the importance of families in decision making.

The third and probably most influential step is to place some or all of the funding authority for the evaluation into the hands of families. At most CSP communities, federal funding is granted to a mental health administrative body, which then contracts with a university or research agency to conduct the evaluation. Researchers are obligated through these funds to collaborate with the administrative body and the federal government, while their motivation to collaborate with family groups is less compelling. Rather, if funding for the evaluation went through family groups, then their power and authority in the evaluation would increase dramatically. This could help to place families in the highest levels of decision making as described by Turnbull et al. (1998), where families act as co-researchers or research leaders, and professional evaluators are contracted as consultants. In the liberatory tradition, these resources and power would enable families to take an active and influential role in the system of care evaluation.

In summary, the CSP is an attempt to radically transform the systems of care that serve youth with serious emotional and behavioral health problems. Part of that transformation is the inclusion of caregivers at all levels of planning, delivery, and evaluation. Unfortunately, many researchers are unfamiliar with approaches for doing this well. Evidence suggests that the research component of most CSP grantee communities is moderately participatory, with caregivers often taking on less influential roles such as data collecting, and professional researchers taking the lead in formulating the research questions, designing the protocol, and interpreting the data. This results in an emphasis on the creation of representative knowledge in the utilitarian tradition of participatory research. This is a missed opportunity to make use of some of the most valuable aspects of family advocacy groups, which is their firm grasp of relational knowledge and organizing, and their ability to create reflective knowledge in the community by educating and training others. As outlined in the framework above, these types of knowledge contribute to radical system transformation and could be harnessed for effective and positive systems change in the CSP through a liberatory approach.