Introduction

There is a clarion call for greater coordination and collaboration among human service organizations and agencies—particularly for those who share a common investment in addressing a particular issue of community concern. Public and private funders, scholars, and practitioners alike have joined in the appeal for increased inter-organizational collaboration as a means to address complex issues and improve uncoordinated and fragmented health and human service delivery systems fraught with expensive redundancies and ineffective programming (Hoge and Howenstine 1997; McLaughlin and Covert 1984). Interorganizational collaboratives (referred heretofore as collaboratives) have become a prominent response to this call in communities throughout the United States.

Commonly referred to by many names (e.g., coalitions, coordinating councils, partnerships), collaboratives are community-based groups comprised of leaders and staff representing both nonprofit and for profit organizations as well as public agencies who share a common issue domain (e.g., mental health services, domestic violence, HIV/AIDS). These groups meet regularly for the purpose of identifying and implementing strategies for improving the community’s collective response to their targeted issue(s) (Foster-Fishman et al. 2001). Because of their community-system level focus, collaboratives tend to work towards change on multiple fronts using a range of strategies, most of which can be categorized into two broad outcome areas: improving interorganizational coordination and fostering broader systems change (e.g., Himmelman 2001; Pence and Shepard 1999; Stevenson and Mitchell 2003). Interorganizational coordination, as it is used here, refers to a collaborative’s efforts at helping members to see themselves as part of a larger community system for addressing a particular issue, increasing members understanding of that system, and facilitating the development of new protocols and practices that will lead to organizations and agencies working together more effectively and efficiently. Such coordination has been suggested as critical for creating a coherent and efficient community response to a targeted issue (Pence and Shepard 1999). Broader systems change, as it is used here, refers to a collaborative’s efforts at facilitating more externally focused activities aimed at addressing gaps and inadequacies in the community system and improving community level outcomes through developing or expanding programs/services and changing policies, as well as engaging in more prevention-oriented activities designed to increase public knowledge and awareness of targeted issues (Pence and Shepard 1999; Stevenson and Mitchell 2003).

Through a combination of the above strategies, collaboratives seek to create a more comprehensive and coordinated community system for addressing a given issue or problem area. In fact, collaboratives are formed based on the premises that adequately addressing the issues for which they are convened is beyond the scope of any one organization or agency and an effective community response will therefore require coordination and collaboration among community stakeholders (Allen et al. 2003; Fowler et al. 2000; Gamache and Asmus 1999; Glisson and James 1992; Sullivan and Allen 2001). The term stakeholder as it is used here refers generally to organizations, agencies, or groups whose actions impact the overall capacity of the community to address a given issue of concern.

Because stakeholders frequently have at least some degree of autonomy in determining the extent to which—and manner in which—they will interact with other organizations or agencies, such collaborative approaches are often described as dependent upon participating stakeholders developing cooperative relationships (Foster-Fishman et al. 2001). The notion that cooperative relationships among stakeholders are important to the process of collaboration and its resulting outcomes is common almost to the point of being axiomatic in the literature (Bond and Keys 2000; Campbell et al. 1999; Gray 1996; Mulroy 1997; Tapper et al. 1997). However, despite this, there has been very limited work focused on systematically operationalizing and empirically examining the role stakeholder relationships play in the effectiveness of collaboratives (see Feinberg et al. 2005; Wells et al. 2007 for exceptions). As a result, we know very little about the types of relationships that matter, the extent to which they matter relative to other factors, and the outcomes for which they matter most.

This lack of knowledge comes at a substantial cost as there is growing evidence that the work of a collaborative is challenging and many struggle in their efforts to accomplish their goals (Halfors et al. 2002; Roussos and Fawcett 2000). To the extent that relationships are important to effectiveness, developing a more sophisticated understanding of the role stakeholder relationships play in multi-stakeholder contexts can lead to additional insights in diagnosing current collaborative capacity and in identifying levers for improving the effectiveness of a collaborative. This study contributes to this end by presenting a framework for conceptualizing and exploring the relative value of cooperative stakeholder relationships for promoting two critical areas of a collaborative’s work: improving interorganizational coordination and fostering broader systems change.

The Importance of Cooperative Stakeholder Relationships

There are two distinct but related literatures that provide the theoretical basis for the importance of cooperative stakeholder relationships to the work of a collaborative. In particular, key supporting theoretical frameworks are found in the literatures on collaboration and social capital. The contributions of each will be discussed below.

Collaboration

Collaboration scholars have long argued for the importance of social relationships among stakeholders for promoting the effectiveness of actions requiring collaboration. Gray (1989) defines collaboration as a “process through which parties who see different aspects of a problem can constructively explore their differences and search for solutions that go beyond their own limited vision of what is possible” (p. 5). Within collaboratives, collaboration can be thought about as the means through which groups are able to identify and enact strategies for improving their collective response to their targeted issue. Within this literature, the theoretical necessity of social relationships for supporting collaboration is often based on transactional costs theory (e.g., Williamson 1979), which premises that cooperation with others has costs (e.g., loss of autonomy, commitment of time, energy, and/or resources, investment of political capital), and therefore risk, associated with it (Alter and Hage 1993; Bailey and Koney 2000; Gray 1989). These costs are a key consideration for stakeholders in deciding whether to cooperate with others (Butterfoss et al. 1993; Huxham 1996). If each stakeholder within a cooperative effort effectively does his or her part, then the collective effort is more likely to be successful and the resource investment worthwhile. However, if some follow through but others fail or, worse yet, take advantage of the situation for their own gain, then those who committed their resources ultimately stand to lose (Alter and Hage 1993).

Therefore, cooperation entails that parties make themselves vulnerable to a certain level of risk stemming, in part, from uncertainty regarding whether the other parties are capable and willing to both follow through on their commitments and protect the interests of their fellow collaborators (Alter and Hage 1993; Neilsen 1988; Nielson 2004). Even if the cost is merely the possibility of wasting time and energy on an unproductive effort (Butterfoss et al. 1993), it is this very vulnerability that can make cooperation so tenuous. In order to minimize the risk to themselves, stakeholders may hold back on what they are willing to commit to a collective effort (Gray 1989). Unfortunately, outcomes requiring collective action frequently cannot be achieved in a context of such restraint (Mizrahi and Rosenthal 1992).

Given this bleak prescription—how, then, does collaboration occur? One common premise is that over time and through interaction, stakeholders can develop relationships with one another that have value (Gray 1989). They learn what they can expect from one another and begin to trust in one another’s demonstrated capabilities (Nielson 2004). Each party to the relationship can be counted upon to act in ways that take into consideration the interests of the other party because, by doing so, the relationship is strengthened (Vangen and Huxham 2003). Among multiple stakeholders, norms begin to develop that further reinforce the value of the relationships and sanction behaviors that threaten to undermine them (Deutsch 2000). Thus, consistent with Deutsch’s (2000) Crude Law of Social Relations, the relationships themselves become a commodity of value because the stronger the relationships, the greater chance that parties will respond cooperatively in future collective endeavors.

Social Capital

A complementary literature concerned with the value of social relationships for accomplishing productive outcomes is the literature on social capital. Social capital refers to the value that exists within social-structural relationships that an “actor” such as an individual, an organization, or a network of organizations can mobilize to make possible the achievement of certain ends that would not be attainable in its absence (Coleman 1988). This literature posits that the manner in which relationships are structured within a defined social network, such as a collaborative, will have significant implications for what can be accomplished by that network due to the resources that will be made available through those relationships (Coleman 1988; Lin 2001; Oh et al. 2004). For example, research in this area has found that social relationships are important because of their ability to serve as conduits for the flow of information through the system (Bailey and Koney 2000; Frank and Zhao 2004; Tsai and Ghoshal 1998); support the development and transferal of norms, values, beliefs and attitudes (Coleman 1988; Ibarra 1993; Putnam 1995); and provide a mechanism for system members to access opportunities (Burt 2000; Powell et al. 1996). In sum, social capital is understood as the value that is embedded within social networks that can facilitate collaboration.

Conceptualizing Cooperative Stakeholder Relationships in Collaboratives

Collectively, the above literatures make a compelling case for the importance of social relationships among stakeholders in promoting or impeding the effectiveness of a collaborative. But what do stakeholder relationships need to look like in order to constitute a form of capacity within the context of interorganizational collaboratives? The collaboration literature contributes important perspectives concerning the relevant qualities of relationships that facilitate collaboration. For example, these scholars have discussed the degree to which stakeholders communicate, trust one another, view one another as contributing a unique perspective or form of expertise valuable to the group, or share a similar philosophy or set of beliefs concerning the nature of the problem and means for addressing it as important to effective collaboration (e.g., Alter and Hage 1993; Deutsch 2000; Foster-Fishman et al. 2001; Gray 1989; Jenkins and Davidson 1999; Mattessich et al. 2001; Zakocs and Edwards 2006). However, the limited number of efforts seeking to quantitatively explore the implications of relational qualities within collaboratives have tended to operationalize relationships as perceptions of the general relational climate within the collaborative (e.g., Butterfoss et al. 1996; Gottlieb et al. 1993; Kegler 1998). As such, much of this literature has been unable to account for heterogeneity in relationships among different stakeholders that commonly characterize the social dynamic within collaborative group settings (Kozlowski and Klien 2000; McGrath et al. 2000).

In comparison, organizational scholars interested in social capital have focused primarily on the structure of social relations within a network through the use of social network analysis (SNA) techniques that capture patterns of relationships such as who knows, communicates with, and/or trusts whom within a defined network (e.g., Ahuja 2000; Balkundi and Harrison 2006; Kilduff and Tsai 2003; Reagans and Zuckerman 2001). Within this literature, a structural characteristic that has emerged as particularly advantageous in team contexts such as a collaborative is network density (Balkundi and Harrison 2006). A dense network is one characterized by a pattern of closely knit ties, signifying more and stronger relationships among stakeholders.

Research on organizational work teams has found strong support for the importance of network density to team functioning. For example, in a recent meta-analysis of 31 studies examining the impact of social networks on teams, network density was found to be positively associated with task performance (Balkundi and Harrison 2006). Similar findings have recently been found in the context of collaboratives as well. Feinberg et al.’s (2005) study of 23 prevention coalitions found that network density was positively correlated with “readiness” indicators identified as preconditions to coalition effectiveness, while centralization was negatively associated. Consistent with this, a comparative case study of two coalitions by Wells et al. (2007) found that the more active coalition had a less centralized network structure than its less active counterpart.

The integration of these two literatures provides a framework for conceptualizing and operationalizing cooperative stakeholder relationships within collaboratives in a manner that attends to both qualities and structure. First, because relationships are multi-dimensional (Deutsch 2000) and the importance of different qualities of relationships varies by context (Adler and Kwon 2002; Krackhardt 1992), this study considers five different relational qualities selected based on their identified importance to collaboration by both current literature as well by key informants.Footnote 1 These are: (1) communication frequency; i.e., how frequently a stakeholder communicates with another stakeholder outside of collaborative meetings (e.g., Austin 2000; Reagans and Zuckerman 2001); (2) responsiveness to concerns; i.e., the extent to which a given stakeholder is perceived by another to be responsive to them and their concerns (e.g., Mizrahi and Rosenthal 2001; Rivard and Morrissey 2003); (3) trust in follow-through; i.e., the extent to which a given stakeholder is perceived by another to be trustworthy to follow through on commitments (e.g., Campbell et al. 1999; Mizrahi and Rosenthal 2001; Nielson 2004; Vangen and Huxham 2003); (4) legitimacy; i.e., the extent to which a given stakeholder is perceived by another to contribute a unique perspective or area of expertise valuable to the work of the collaborative (e.g., Gray 1985); and (5) shared philosophy; i.e., the extent to which a given stakeholder is perceived by another to share a similar philosophy about the collaborative’s targeted issue and how it should be addressed (e.g., Gamache and Asmus 1999; Mizrahi and Rosenthal 1992).

Because there is strong reason to believe these qualities are interdependent and mutually reinforcing of one another (Alter and Hage 1993; Gray 1989), one stakeholder’s relationship to another could be characterized as a particularly strong cooperative relationship to the extent it exemplifies all five qualities. As a group, a collaborative could then be characterized as having strong cooperative relationships among its stakeholders to the extent its network structure is integrated, as suggested by its network density.

Present Study

Accordingly, this study explores the role of stakeholder relationships in promoting the effectiveness of a collaborative guided by the following hypotheses:

H1

Cooperative stakeholder relationships will be positively related to coordination effectiveness.

H2

Cooperative stakeholder relationships will be positively related to systems change effectiveness.

In addition, a highly applied as well as theoretical concern in understanding the role of stakeholder relationships is the relative importance of stakeholder relationships in comparison to other factors identified in the literature as important to a collaborative’s functioning and effectiveness. For example, to what extent do relationships matter for effectiveness relative to the collaborative’s leadership and decision-making capacity, size, age, or degree of organizational structure (e.g., Foster-Fishman et al. 2001; Granner and Sharpe 2004; Mattessich et al. 2001)? For scholars, addressing this question builds greater understanding of what types of processes and factors are critical drivers for what types of outcomes. For practitioners, this question speaks directly to how capacity-building efforts should be prioritized. Therefore, this study adopted a comparative approach to examining the importance of stakeholder relationships by considering them in the context of other factors related to collaborative capacity (Foster-Fishman et al. 2001).

Lastly, this study provides the opportunity to explore whether relationships have greater or lesser importance for the different outcomes of coordination and systems change effectiveness. The theoretical basis for this question comes from current collaboration theory, which argues that collaborative groups engage in different types of work that exist on a continuum requiring progressively greater levels and additional types of capacity. Specifically, scholars such as Himmelman (2001), Hogue (1993), and Gajda (2003) have suggested that coordination activities, such as developing protocols for sharing information or service referrals within a system, are likely to allow for considerable autonomy among organizations and thus require less relational capacity. In contrast, systems change efforts, such as organizing stakeholders to change the practices or policies of the local court system, are likely to require more complex forms of collective action sustained over time (Foster-Fishman et al. 2007) and a higher degree of political accountability for the actions of the other stakeholders done under the auspices of the collaborative. Such collective action is theorized to require greater interdependence and thus greater relational capacity (Himmelman 2001). This would suggest that stronger relationships are more important for systems change outcomes relative to coordination outcomes.

H3

Strong stakeholder relationships will be more important to systems change outcomes relative to coordination outcomes.

Methods

Study Context

The above hypotheses were examined based on survey and social network data collected from a sample of 48 different Midwestern collaboratives formed to improve their communities’ responses to domestic violence. These collaboratives are groups comprised of representatives and leaders from an array of criminal justice and human service agencies/organizations (e.g., police, prosecutors, judges, advocates, health providers, shelter providers), as well as community groups invested in addressing domestic violence within their community (Shepard et al. 2002; Sullivan and Allen 2001). Their intended function is to reduce the prevalence of—as well as improve the community’s response to—incidences of domestic violence by increasing the level of coordination and collaboration among community stakeholders (Hart 1995; Sullivan and Allen 2001).

Sample

The sample for this study was recruited from a population of 57 domestic violence collaboratives within a single Midwestern state. Due to the multi-level nature of this study, sample characteristics from two levels must be considered. At the highest level, Level II, sample considerations focus on the number of collaboratives represented in the study. Nested within each of the collaboratives participating in the study are members; that is, organizational representatives who attend the collaborative’s meetings on behalf of a given organization, agency, or group. Therefore, at Level I, sample considerations are focused on the number of organizational members who participated in the study. In the event there were multiple individuals representing the same organization or agency on a collaborative, the one most active in the collaborative was selected as the sole representative of their organization in the study.Footnote 2

In order for a collaborative to be included in analysis, a member response of 70% or greater was required. This meant that surveys had to be received from at least 70% of members identified as current by the collaborative’s leader. Current membership in the collaborative was defined as organizations/agencies that were represented at at least one collaborative meeting in the past year. The use of 70% as the cut off was selected based on the distribution of response rates across collaboratives with the goal of balancing the competing needs for both high within-group representation in calculating network properties (Scott 2001) as well as maximizing the number of collaboratives represented in the study. Forty-eight collaboratives (84%) met criterion for inclusion. According to leaders, these collaboratives on average had existed for approximately 10 years but ranged in age from 2 to 21-years-old. Collaboratives ranged in size from 6 to 61 current member organizations with an average size of 16 organizations per collaborative. Within these 48 collaboratives, data were collected from 641 organizations with an average within-collaborative response rate of 88%. However, in order to further increase the validity of member report measures of effectiveness and executive functioning capacity, participants were asked at the conclusion of the survey the extent to which they felt they were sufficiently knowledgeable about their collaborative to respond to the survey questions. This was done in recognition of the fact that collaboratives often have “peripheral” or new members who may have had very limited involvement with and subsequent knowledge of their collaborative (Allen 2005), thereby making them questionable informants of collaborative functioning and effectiveness. Thirty-seven participants indicated they had limited or no knowledge of their collaboratives with regard to the survey questions and were subsequently omitted from hierarchical linear modeling analyses resulting in a final modeling sample size of 604. However, because these non-knowledgeable participants had attended at least one meeting in the past year and therefore were part of the collaborative network, network data from these participants was included in calculations of social network characteristics.Footnote 3

Procedures

The principal mode of data collection for this study was a social network survey distributed to all current members of the collaboratives that also included measures of members’ perceptions of the executive functioning capacity of their collaborative and its effectiveness at improving coordination and fostering broader systems changes. Descriptive information about the collaborative was also collected from membership rosters and phone interviews with the collaboratives’ leaders.

Measures

Network Measures

For the purposes of the social network analysis, each participant was categorized by the collaborative’s leader based on their stakeholder group (e.g., law enforcement, prosecuting attorney, domestic violence survivor service provider). Social network measures were then collected using a valued-tie roster questionnaire (Wasserman and Faust 1994) which asked participants to rate on a scale from 1 to 6 their relationship with each stakeholder group currently represented on their collaborative on a given quality (e.g., communication frequency). Participants filled out a separate roster questionnaire for each of the five relational qualities discussed: communication frequency, responsiveness to concerns, trust in follow through, legitimacy, and shared philosophy. Based on these responses, network density was calculated in SAS for each quality using a standardized valued tie density measure as the sum of all ties, normed by the total number of stakeholders in the network (Wasserman and Faust 1994). The resulting score provides an overall valued average of the strength of relationships among stakeholders with a possible range of 1–6.

In the event that there were multiple organizations representing the same stakeholder group within a single collaborative (e.g., both state and local law enforcement representatives), their ratings for every other stakeholder group on a given quality were averaged to create a stakeholder-by-stakeholder matrix prior to calculating density scores. On average across collaboratives, a given stakeholder group was represented by 1.66 members. The network analysis focused on stakeholder level, as opposed to organizational level relationships, for several reasons. First, stakeholder group affiliation is a highly salient categorization in collaboratives, given that they are focused on improving coordination and fostering collaboration across diverse stakeholders (Foster-Fishman et al. 2001; Himmelman 2001). Second, several theories of social capital (e.g., structural holes theory, weak tie theory) suggest that the strength of the linkages across different stakeholder groups represents the strongest form of social capital because such links are more likely to provide access to the broadest range of knowledge and resources (Burt 2000; Granovetter 1973). This again supports focusing on cross-stakeholder relationships within the context of collaboratives. Lastly, on a pragmatic level, given the large numbers of members in some collaboratives, the time-consuming nature of social network measures, and a required member response rate of 70% or higher, it was believed that organizational level rosters would hinder participation, thereby significantly diminishing the Level II sample size and biasing the sample towards smaller collaboratives. Thus, the less burdensome stakeholder level roster was deemed both conceptually and methodologically to be the most appropriate strategy.

In preparation for a subset of the analyses, the density scores for each relationship quality were averaged to create a general construct referred heretofore as “stakeholder relationships”, which served as a measure of the overall strength of cooperative relationships among stakeholders across the five qualities for a given collaborative. The creation of this construct was supported by a principal components analysis with Varimax rotation of the five density scores resulting in a single factor solution, accounting for 64% of the total variance. The combined stakeholder relationship scale had an alpha of .8, indicating good internal consistency.

Outcome Measures

Two outcomes of effectiveness were examined in this study: interorganizational coordination and broader systems change. In conceptualizing and operationalizing indicators of effectiveness across a large number of collaboratives, there are several methodological challenges that arise. Statistically and conceptually, the cleanest approach is to identify a specific externally verifiable indicator, such as implementation of evidence-based prosecution policies, upon which to judge the performance of all collaboratives. However, for such an indicator to have internal validity, it would need to be equally relevant for each collaborative in the sample. In other words, each collaborative would had to have given roughly equal priority to implementing evidence-based prosecution policies in the focus of their work. For the sample in this study, collaboratives have discretion in defining their priorities. Therefore, the strategies used for improving coordination and promoting systems change vary greatly in their form across collaboratives, making the identification of an externally verifiable indicator more complex. While sophisticated systems for documenting community change for addressing these issues have been developed (e.g., Fawcett et al. 1995), such systems often involve in-depth, case-study type methodologies that can be prohibitively onerous when seeking to quantitatively model relationships across large numbers of collaboratives.

Another alternative is to identify an informant who is familiar enough with a given collaborative and the focus and intentions of its activities, such as the group chairperson, to provide an independent rating of effectiveness. However, this approach is conceptually problematic in that it denies the pluralistic and interpretivistic nature of the concept of effectiveness in bottom-up collaborative processes that engage diverse types of stakeholders. It is reasonable to assume that the different roles organizations play in the community system are likely to result in some degree of heterogeneity of perspective concerning the performance of the collaborative. Because collaboratives focus primarily on institutional-level change, all members are presumably also legitimate actors in the system that is the target for change and their evaluations are thus valid. Indeed, one of the very premises underlying the formation of collaboratives is the importance of input from diverse stakeholders.

One way to address this limitation has been to utilize aggregated measures of effectiveness in which evaluations are taken across group members and then averaged to create a group level score. This approach has advantages in that it allows for incorporation of different views across stakeholders; however, it too has limitations, including the fact that aggregation eliminates meaningful within-group variation relative to between-group differences. Even with evidence of significant group clustering (e.g., ICC1), this concern remains unless within-group agreement across all collaboratives is quite high.

The approach for operationalizing group effectiveness at improving coordination and promoting systems change that was adopted by the present study was selected in an effort to balance these competing issues and limitations by capturing evaluations of effectiveness in two broad areas across members. Analysis of the data resulting from these measures then utilized a hierarchical linear modeling approach which, while described in more detail in the analysis section, is relevant in considering the operationalization of the outcomes. Specifically, in comparison to aggregated outcome measures, an HLM approach models intercepts (i.e., mean group differences) between collaboratives on the outcome variable while still taking into account heterogeneity in perceptions among members within the same collaborative (Hofmann et al. 2000).

Members’ evaluations of their collaborative’s effectiveness were measured by items adapted from previous research with this population of collaboratives (Allen 2005). Interorganizational coordination (alpha = .89) was measured with a four-item scale pertaining to the effectiveness of the collaborative at increasing the level of coordination among community organizations and agencies (e.g., to what extent has the efforts of your council resulted in organizations and agencies working together more efficiently?) and increasing members’ awareness of the interorganizational system within which they were embedded (e.g., to what extent has the efforts of your council resulted in members seeing their organization/agency as part of a broader system for responding to domestic violence?). Broader systems change (alpha = .92) was measured on a scale consisting of seven items measuring the extent to which the collaborative is perceived to have been effective at fostering community change in areas such as influencing policy, changing public attitudes, improving existing services, and developing needed services as well as overall effectiveness in improving the community’s response to domestic violence. Both measures asked participants to respond to items using a scale of 1–6 with identical anchors to ease comparability. Principal component analysis indicated that the two scales represented separate, but related, factors.

Capacity Factors

A total of four additional capacity characteristics prominent in current literature on interorganizational collaboratives were included in the comparative models for this study. These were (1) executive functioning capacity, (2) size, (3) age, and (4) degree of organizational structure. Executive functioning capacity represents the aggregate of two measures—leadership and decision-making—that were combined due to their high degree of multi-colinearity (Pearson correlation = .78, p = .000). The resulting executive functioning measure concerns the extent to which the collaborative was perceived to have effective leadership and decision-making practices and was measured as part of the member survey using items adapted from previous research (alpha = .92). All items were adapted from previous research. Specifically, it asked members to rate the extent to which they felt the collaborative’s leadership was effective at managing interpersonal dynamics (3 items; Allen 2005; Weiss et al. 2002), accomplishing instrumental tasks (3 items; Allen 2005) and inspiring vision and transformation (9 items; Bass 1985; Bass et al. 2003; Tepper and Percy 1994). It also measured the extent to which participants felt decision-making within the collaborative was: shared among members (5 items; Allen 2005), reflexive (6 items; Carter and West 1998; De Dreu 2002), synergistic (2 items; Weiss et al. 2002) and innovative (2 item; Anderson and West 1998). Mean scores for leadership and decision-making were calculated separately for each participant and then aggregated to the collaborative level, where they were averaged as a measure of the overall executive functioning capacity of the collaborative. Variable aggregation was supported by one-way ANOVAs which found significant clustering of member perceptions concerning decision-making and leadership within collaboratives (ICC{1} = .11 and .09, respectively; Bliese 2001; Bryk and Raudenbush 1992).

The size of the collaborative was operationalized as the number of current members based on membership rosters and confirmed by the collaborative’s leaders. A collaborative’s age was operationalized as the number of years since the inception of the collaborative as reported by the collaboratives’ leaders. Degree of organizational structure was measured on an 18-item dichotomous (yes/no) scale adapted from Butterfoss (1994) based on leader reports. Items in this scale collected descriptive data about the presence or absence of various aspects of formalization in a collaborative (e.g., whether the collaborative had written meeting agendas, a written mission statement, formal decision-making procedures, sub-committees or work groups, 501{c}3 status). Structure was conceptualized as additive, operationalized based on the scale sum, and represents the overall level of formalization that has been adopted by the collaborative.

Controls

The social network measurement approach utilized in this study necessitated a control variable of familiarity density. Because of the valued nature of the social network data, lower numbers generally indicate the presence of a negative relationship (e.g., lack of trust, absence of a shared philosophy) rather than the absence of any relationship. In cases where a member may be relatively new or an infrequent attendee to meetings, the assumption that a relationship exists at all may not be appropriate. As such, in addition to the scale choices ranging one through six, participants were also given the option of responding “I’m not familiar enough with them to know” on all network measures. Thus, this covariate essentially represents the extent to which members in the collaborative knew each other well enough to have developed an opinion about one another. This added a familiarity dimension to the social network data that was controlled for in order to maximize the validity of the stakeholder relationships network data. Specifically, a separate density score using a dichotomized matrix was created for each relationship quality based on the network of familiarity (1 if familiar enough to respond, 0 if not). These familiarity density scores were then averaged across the five indicators in the same manner as the stakeholder relationships scale. This familiarity density score was entered as a covariate in all analyses.

Analysis

Hypotheses were tested using a hierarchical linear modeling (HLM) technique in which the intercepts (within-group means) of members’ perceptions of their collaborative’s effectiveness at fostering interorganizational coordination or systems change (Level I) were predicted by characteristics of the collaborative (Level II; Hox 2002). In order to examine whether cooperative relationships significantly improved models of effectiveness over and above other factors, hypotheses one and two were tested using a hierarchical approach adapted from Cohen and Cohen (1975) in which each outcome was modeled first as predicted by executive functioning capacity, age, size, and structure, along with the control of familiarity density. The network integration score for cooperative relationships was then entered into the model. A comparison of the variance explained (ICC{1}) between the first and second models was examined as an indicator of improvement in model fit. A chi square test on the differences of log likelihood ratios was used to determine the significance of this improvement. Lastly, in order to explore the relative importance of stakeholder relationships compared to other factors (Hypothesis 3), effect sizes were calculated using a Pearson correlation coefficient (r; Rosenthal 1994) through the equation:

$$ r = \sqrt {{\frac{{t^{2} }}{{t^{2} + df}}}} , $$

where t is the t-statistic for each coefficient and degrees of freedom (df) are based on the formula for level II fixed effects proposed by Snijders and Bosker (1999): df = N − q − 1 where N is the number of level II units and q is number of explanatory variables. Effect sizes were interpreted using the following benchmarks as proposed by Cohen (1988); r = .10 represents a small effect, .30 a moderate effect, and .50 a large effect.

Results

Univariate Analysis

Descriptives and zero-order correlations among dependent and independent variables are presented in Tables 1 and 2. As shown, stakeholder relationships were significantly and positively correlated with both outcomes. Stakeholder relationships are relatively independent from the other capacity factors. However, as would be suggested by network theory (Scott 2001), the size of the collaborative shows a negative correlation that trends toward significant with stakeholder relationships. This likely reflects the fact that higher levels of overall network density are more easily achieved in smaller networks where there are fewer actors with whom any given actor can interact (Wasserman and Faust 1994) and highlights the importance of including network size in models that include density measures. With regards to the outcome variables, on average collaboratives were perceived to be significantly less effective at promoting broader systems changes in the community (X = 3.76) relative to their perceived effectiveness at improving coordination (X = 4.38; t = 9.646, df = 47, p = .000).

Table 1 Collaborative-level descriptives
Full size table
Table 2 Zero order correlations among predictors
Full size table

Finally, in order to justify the use of multi-level models, null models were run for both outcome variables to test whether significant between-group differences existed relative to within-group variance. The results of this analysis indicated significant variation both within members of a single collaborative as well as between collaboratives. The interclass correlations (ICC1) for coordination and systems change outcomes were .10 and .17, respectively. These are consistent with previous multi-level research on this population (Allen 2005) and suggest that collaborative level predictors can be useful in helping to explain differences between collaboratives.

Relative Importance of Stakeholder Relationships to Coordination Effectiveness

As summarized in Table 3, analyses demonstrated support for hypothesis 1, finding a significance relationship between coordination effectiveness and stakeholder relationships after taking into account other collaborative factors. The addition of stakeholder relationships accounted for 7% additional variance, which was a significant improvement of model fit based on one degree difference in the degrees of freedom. The full model explained 79% of the differences between collaboratives in coordination effectiveness. Concerning the size of the effect, stakeholder relationships were shown to have a moderate impact (r = .30) on the outcome variable of coordination effectiveness.

Table 3 Hierarchical analysis for coordination effectiveness
Full size table

In exploring the relative importance of stakeholder relationships to coordination effectiveness in comparison to other factors, executive functioning capacity was clearly distinguished among the factors as having the strongest unique effect (r = .56). The impact of stakeholder relationships on coordination outcomes was comparable in its effect size to the collaborative being larger and more tenured. After accounting for the other factors, the degree of organizational structure had no significant effect. Collectively, these findings suggest that while stakeholder relationships appear to be positively associated with coordination effectiveness, there is not evidence to suggest they are the key driver of effectiveness in this area of a collaborative’s work. Relative to all other examined factors, it is the executive functioning capacity that appears to have the strongest influence on this outcome.

Relative Importance of Stakeholder Relationships to Systems Change Effectiveness

To test hypothesis 2, a second set of analyses identical to those described above were run with systems change as the outcome variable. Results supported hypothesis 2, finding that stakeholder relationships were significantly and positively associated with systems change effectiveness (Table 4), accounting for 23% additional variance in comparison to the initial model. The significance of this improvement in model fit was supported by a significant chi square on the difference in the log likelihood ratio. In examining the strength of the effect, stakeholder relationships were shown to have a strong effect (.49) on systems change outcomes. In fact, results indicated that stakeholder relationships were the strongest factor associated with systems change effectiveness relative to the other examined collaborative factors. Executive functioning capacity showed a positive effect roughly equivalent to the collaborative being larger and more tenured.

Table 4 Hierarchical analysis for systems change effectiveness
Full size table

Taken as a whole, these analyses provide tentative support for hypothesis three; namely, that stakeholder relationships are more important for systems change outcomes relative to coordination outcomes. While there are methodological limitations to the direct testing of this comparison, the pattern of findings is consistent. First, findings show a clear improvement in model fit in predicting systems change effectiveness by accounting for stakeholder relationships, whereas the model improvement was much smaller for coordination effectiveness. However, this comparison must be approached with caution as this might also be explained by the fact that there was simply more between-collaborative level variance for the systems change outcome relative to coordination effectiveness. Second, the pattern of effect sizes among capacity factors indicates a moderate relationship between stakeholder relationships and coordination effectiveness. However, this effect was over-shadowed by the relative importance of executive functioning capacity. In contrast, relative to other factors, stakeholder relationships were distinguished as the strongest predictor for systems change outcomes.

Unique Effects of Relationship Qualities

Given evidence of a moderate to strong positive relationship between the quality of stakeholder relationships and effectiveness, a final exploratory analysis examined whether certain qualities of relationships were more or less important for different outcomes. In order to examine this, another set of HLM analyses were run in which the five relational qualities that comprised the overall measure of stakeholder relationships were unpacked and entered as separate predictors. Given the degree of multi-colinearity among these qualities (Table 5), the research question explored was whether certain relational qualities distinguished themselves as significant predictors of the effectiveness of a collaborative over and above their contribution to the shared variance of generally strong stakeholder relationships. Due to the aforementioned confound between the size of a network and its density, size was entered into the models as a control along with familiarity density.

Table 5 Correlations of relationship qualities to outcome variables
Full size table

Results for coordination effectiveness showed a trend toward significance for communication frequency with a moderate effect size (Table 6). For systems change, both communication frequency and shared philosophy were significant and both with moderate effect sizes. It is also important to note that shared philosophy showed very little unique effect on coordination effectiveness. Given that all five relationship qualities showed positively directed correlations with both outcomes (Table 5), the negative non-significant estimates for responsiveness to concerns and recognized expertise are attributed to the high multi-colinearity of these variables.

Table 6 Relative effect of different relationship qualities to coordination and system change effectiveness
Full size table

Discussion

In order to deal with complex public issues in a more systemic and comprehensive way, collaboratives often work toward change on multiple fronts using a range of intervention strategies (Stevenson and Mitchell 2003). The present findings contribute empirical support for the premise that stakeholder relationships are an important aspect of capacity for supporting the effectiveness of a collaborative. As hypothesized, the findings indicated collaboratives characterized by stronger relationships among stakeholders were more likely to be perceived as more effective at both improving coordination and promoting broader systems changes. However, a key finding of this study is that this aspect of capacity appears to be more important for systems change outcomes relative to coordination.

As such, this study provides partial support for the ideas underlying the capacity continua proposed by Himmelman (2001) and others (Gajda 2003; Hogue 1993). As predicted by these scholars, findings suggest that stakeholder relationships have greater importance for outcomes emphasizing greater collective action (i.e., systems change) relative to those emphasizing system awareness and protocols for referral and information sharing among stakeholders. In some ways, this is also consistent with the conclusions reached by Krackhardt (1992) who, in a case study investigation of the influence of social networks during an organizational unionization attempt, concluded that major changes that may threaten the status quo concerning power and decision-making require strong affective relationships.

Developing empirical support for these capacity “contingency theories” has important implications for both research and practice as it highlights the outcome-specific nature of the concept of capacity and suggests that the level of capacity that is adequate for achieving certain collaborative outcomes may not translate into a general capacity for all outcomes. The finding that collaboratives may need more and stronger relationships in order to effectively engage in systems change types of efforts relative to coordination efforts is particularly significant in light of the fact that collaboratives are increasingly being called upon by outside funders to adopt more systems change-oriented goals (e.g., Foster-Fishman et al. 2007; http://www.cdc.gov/ncipc/DELTA/default.htm; retrieved April 17, 2007). Findings from this study suggest that collaboratives which may have been characterized as highly effective within a coordination context may still struggle in adopting this new goal orientation if they lack adequately strong relationships among their collaborating stakeholders. This further suggests that strengthening the quality and connectedness of relationships among stakeholders may be an important lever for helping a collaborative to effectively make such a transition.

A finding of additional interest concerns the different profiles of what emerged as most important between coordination and systems change outcomes. Continuum approaches to thinking about capacity suggest that certain outcomes require more capacity. However, examining the effect of stakeholder relationships relative to other collaborative factors allows us to expand upon the continuum framework to examine whether different outcomes actually have different profiles of requisite capacity. Toward this end, findings suggest that if the targeted goal is improved coordination, then fostering strong leadership as well as collaborative and reflexive decision-making practices may be the most powerful levers for improving effectiveness. Conversely, if the goal is systems change, then collaboratives may be well advised to pay particular attention to the quality of relationships among their stakeholders.

Consistent with previous research, findings further suggest that in addition to strong stakeholder relationships and executive functioning capacity, collaboratives that have been around longer and have more members are generally perceived to be more effective (Allen 2005; Kreuter et al. 2000). Unexpectedly, the degree of organizational structure was not significantly related to either coordination or systems change effectiveness. This is contradictory to much of the current literature which has found that the degree of formalization is a significant predictor of effectiveness (e.g., Garland et al. 2004; Gottlieb et al. 1993; Jasuja et al. 2005). However, these studies have tended to operationalize formalization more narrowly. Present findings may be explained by the fact that most collaboratives in the study (i.e., 90% or more) were found to have a moderate amount of organizational structure in terms of meeting regularly, having written agendas to guide meeting activities, recording and distributing meeting minutes, and possessing some means for communicating with members outside of the meetings. Non-significant findings may therefore suggest that while a certain degree of structure is necessary to collaborative functioning, a high degree of organizational structure (e.g., 501{c}3 status) may not be particularly value-added in promoting coordination or systems change effectiveness, and may potentially be detrimental. This is a somewhat different interpretation than has been offered in the recent literature (e.g., Zakocs and Edwards 2006) and should be examined further in future research.

In addition to examining the relative importance of stakeholder relationships across different outcomes, this study also represents one of the first attempts to examine whether certain qualities of relationships are uniquely important for different outcomes of effectiveness. While exploratory in nature, results provide a foundation for the further development of testable theories concerning the differences between coordination and systems change efforts.

First, findings suggest that frequent interaction amongst members outside of meetings has a unique effect on both coordination and systems change outcomes. This makes sense as direct communication and interaction provide the vehicle and opportunity for many network benefits to occur (Alter and Hage 1993). For example, open channels of communication can provide the opportunity for earlier identification of problems and greater information sharing (Ahuja 2000).

Another result of interest in this analysis is the finding that shared philosophy had one of the strongest unique effects on systems change outcomes. While this relationship must be interpreted with caution, it does provide a basis for examining linkages with other research and theory. Shared philosophy, as it was used in this study, directly relates to how stakeholders think about the targeted issue of domestic violence and what beliefs and assumptions they hold about the most effective means for addressing it. The ability to bring together diverse stakeholders with different perspectives is what makes collaboratives so promising as a vehicle for systems change. However, findings indicate that perceptions of fundamental differences in philosophies concerning the targeted issue and how it should be addressed may significantly hinder a collaborative’s ability to promote systems change. This finding is consistent with other recent investigations into factors that facilitate or impede problem solving in multi-stakeholder contexts. For example, in a recent case study of a protracted environmental conflict, Gray (2004) concluded that the most significant factor preventing a collaborative solution was the existence of differences among stakeholders in their frameworks of understanding related to how they conceptualized the problem that linked them and how they felt the problem should be resolved.

It is particularly interesting that shared philosophy appears uniquely important only for systems change and shows relatively little importance for coordination outcomes. It may prove fruitful to consider this finding in the context of frameworks provided in the organizational change literature. Change theorists have argued that change processes meaningfully differ and therefore can be distinguished by the extent to which they involve a shift in the strategic orientation of a system and challenge existing schemata (Bartunek and Moch 1987; Greenwood and Hinings 1996). First-order changes are characterized by incremental shifts that are consistent with an established framework or set of operating assumptions. Coordination effectiveness, as it was operationalized in this study, can be thought of as how effective the collaborative has been at making first-order improvements in how things work within the existing community response system. As might be suggested by change theories, it does not appear particularly critical for these stakeholders to hold similar beliefs or understandings about the nature of the issue in order to carry out this coordination function.

On the other hand, systems change, as it was operationalized in this study, involves making changes to the infrastructure of the system itself. Scholars of systems change have argued that community systems emerge out of—and are therefore reflective of—the attitudes, values, beliefs, and understandings of the institutions of which they are formed (Foster-Fishman et al. 2007). As such, systems change efforts often require second order change—actions that modify existing operating frameworks and challenge underlying beliefs and assumptions embedded within the status quo (Bartunek and Moch 1987). Given these conditions, it holds that an alignment of philosophies among collaborating stakeholders would be particularly important. However, in light of the exploratory nature of these findings, future research is needed to further investigate this proposition.

Limitations

Interpretations of findings from this study need to take into account several limitations. First, as discussed, the relationships hypothesized were tested using cross-sectional data. As with all cross-sectional studies, the directionality of the relationships is statistically ambiguous and statements of directionality are based solely on theory.

Second, collaborative coordination and systems change effectiveness were both operationalized based on members’ perceptions of effectiveness. This operationalization was deemed to be of theoretical interest as collaborative members are also key stakeholders in the system the collaborative is attempting to change. However, an associated limitation is the possible inflation of correlations resulting from key independent (e.g., stakeholder relationships) and dependent variables (e.g., effectiveness) operationalized using data from the same source. However, in this study, independent and dependent variables were operationalized in a way that minimizes the likelihood of Type I errors resulting from a common method bias. Specifically, stakeholder relationships were operationalized using a social network approach as the average network density. This creates a collaborative level variable comprised of information about how each stakeholder is perceived by every other stakeholder, which was then modeled in its relationship to differences in the average perceived effectiveness between collaboratives. Given these operationalizations, a collaborative’s density score was relatively independent from any one participant’s perceptions of his relationships with the other members. Confidence in the findings is further suggested by the fact that effect size measures for non-member report measures in some analyses (e.g., age, size) showed an equal or greater effect than member report measures (e.g., density, executive functioning capacity).

Third, the network for this study was conceptualized and operationalized at the stakeholder group level of analysis. This creates a less fine-grained picture of the network relative to if it had been measured at the organizational member level. The coarseness results from the degree of cognitive abstraction required on the part of the respondents in rating their relationship to a stakeholder group, even if stakeholder group affiliation was a salient category by which they viewed one another. Specifically, participants may have differed from each other in how they cognitively understood and reported on the overall relationship with a stakeholder group consisting of, on average, 1.66 organizational representatives.

Fourth, contingency theories hypothesize that it is the greater interdependencies associated with collective action which results in the need for stronger relationships among collaborative members. While this is consistent in explaining the contrasting profiles of coordination and systems change, further development of this theory in future research should attempt to examine this relationship more directly. For example, comparative case studies could examine more directly what types of interdependencies members of a collaborative experience across aspects of their work together and seek to validate differences in patterns between the degree of interdependency and the focus of the efforts.

Fifth, it is important to recognize that collaboratives are impacted by a diverse range of factors associated with member characteristics, group process characteristics, relationship characteristics, structural characteristics, and environmental factors (e.g., Foster-Fishman et al. 2001). While an attempt was made to examine a reasonable cross-section of characteristics prominent in the literature, the factors considered were not exhaustive. Specific conclusions concerning the relative importance of stakeholder relationships are therefore bounded to those factors considered in the present study that served as the basis for comparison.

Lastly, it is important to note that findings from this study were based on data collected from one prominent type of collaborative focused on domestic violence. While collaboratives across different substantive areas appear similar in terms of their structure, internal processes, and function, they may also differ in important ways such as the broader policy contexts within which they operate. For example, the relevance of shared philosophy to systems change efforts may be particularly important in issue domains characterized by long-standing differences in philosophy. The field of domestic violence has been marked by a history of struggle among key stakeholders such as women’s advocates, law enforcement agencies, court systems, and mental health service providers directly related to philosophical understandings about the nature of domestic violence (Pence and McDonnel 1999). For example, stakeholders may differ in whether they understand domestic violence as a crime committed in an effort to gain and/or retain power and control over another person or whether it is viewed as an escalated interpersonal conflict resulting from dysfunctional relationships (Pence 1983). Different philosophies carry with them significant implications for policy and practice in terms of the extent to which and the ways in which the community intervenes to protect victims of domestic violence and hold batterers accountable.

However, it is important to note that philosophical struggles are not unique to the field of domestic violence. Differences in philosophy have been documented in many areas such as community development, community health, and criminal justice. For example, practitioners, scholars, and policy makers in these fields have long differed in the extent to which they characterize and understand populations based on their deficits or assets (Kretzmann 1995; Trickett et al. 2001), whether they view problems from an individual or structural/institutional perspective (Riessman 1990), and whether they approach problem intervention from a prevention or remediation paradigm (Felner et al. 2001). This suggests that the perceptions of alignment of philosophies among stakeholders may have significant implications for collaborative systems change efforts in many arenas beyond the field of domestic violence. As such, the extent to which findings from this study generalize to collaboratives in other substantive areas remains an interesting empirical question and an important area for future research.

Conclusion

Collaboratives are charged with the challenging assignment of both improving the level of coordination among organizations within the existing community system as well as making needed changes to the infrastructure of the system itself. The relationships among stakeholders can represent an important form of capacity that can have significant implications for what can be accomplished by a collaborative. However, findings from this study suggest that the relevance of stakeholder relationships in these contexts is not the same for achieving coordination as it is for promoting system change. When the work of a collaborative calls for not just improving how information and resources flow through the existing system, but actually making changes to the infrastructure of the system itself, the collaborative will likely need stronger relationships among participating stakeholders than may have been required for promoting coordination. Relationship-building efforts may want to focus particular attention to identifying what differences in philosophies about the targeted issue may exist among members and work toward building more shared frameworks of understanding.