What happens at the convenings (meetings, events, and activities) held by community-based civil society organizations (CSOs)? CSOs, also referred to as voluntary associations, civic associations, or membership-based organizations, are formal organizations whose participants voluntarily assemble to pursue common purposes—a population that ranges from religious congregations to business associations to hobby groups (Edwards, 2014). What characteristics shape the internal dynamics of CSO convenings? What outputs emerge? How do convenings impact organizational outcomes? Scholars have asked these questions since Tocqueville (2000 [1835]) published his insights on American CSOs. Ethnographies—the primary method for studying convenings—have provided answers to the first question (e.g., Blee, 2012; Eliasoph, 1998), but offer limited insights into the other questions because of the small number of organizations studied. Surveys of organizations or individuals (e.g., Chaves, 2004; Verba et al., 1995; Wood & Fulton, 2015) offer a broad look at CSO participants, structures, and outcomes, but their reliance on participant self-report and recall limits their ability to reveal internal convening dynamics.

To address these issues, the field needs a middle-range approach to data collection that generates detailed, comparable, observational data at scales larger than traditional ethnography. Recently, Baggetta and Bredenkamp (2019) developed a new approach by adapting systematic social observation (SSO) to study college student organizations. SSO enables researchers to gather detailed observational data at relatively larger scales by using multiple observers and standardized protocols to collect comparable data from observable settings. To address long-standing questions about CSO convenings, SSO needs to be applicable beyond student groups. We addressed this need by redesigning and expanding the SSO tool, so it applies to a broad range of community-based CSOs.

In this article, we demonstrate the tool’s viability through a 15-month pilot study of three community-based CSOs in Indianapolis, observing 99 convenings. We describe how we adapted the SSO approach to community-based CSOs and discuss the human and technological resources needed to effectively collect this type of data. We illustrate the value of SSO for studying CSOs by presenting distributions of interaction styles and cross-demographic interaction across organizations. We note how such data could be combined with survey and administrative data to analyze how CSOs internal dynamics affect organizational outcomes. Finally, we discuss the broader utility and limitations of this middle-range data collection method, suggesting where it can advance research on CSOs and the third sector.

Background: Systematic Social Observation

Systematic social observation is a technique for collecting quantitative, observational social science data. It originated in developmental psychology, where researchers studied childhood behaviors and interactions, often in lab settings (Weick, 1968). Sociologists began using the approach in field settings as early as the 1920s (McCall, 1984), with notable applications from the 1970s (Reiss, 1971) to the present (Lee & McCabe, 2021).

At the core of any SSO application is a standardized coding form that lists variables of interest likely to be encountered in the field with either discrete response categories or quantitative entry fields (e.g., counts, times). The items are informed by theory and prior research and may be distilled from initial qualitative observations (Nippert-Eng, 2015). Research assistants are then trained on where and what to observe, and how to code their observations.

Data collected in SSO studies are often used to describe and compare three units of analysis: environments, actors, and interactions. Environments are physical spaces whose dimensions, arrangements, or contents may matter for social outcomes. Sampson (2012), for example, had research assistants code Chicago’s streets for indicators of physical disorder like broken glass and graffiti. Actors are the people in the environment. Whyte (1980), for example, coded the demographic characteristics and behaviors of people in New York City’s public parks. Interactions are verbal and nonverbal communication between people. Reiss (1971), for example, had observers ride-along with police and record elements of officer-civilian encounters.

Scholars have used SSO data in several ways. Some correlate characteristics of environments, actors, and interactions. For example, in Underhill’s (1999) study of retails stores, he observed that customers in narrow aisles often had their backsides brushed by other shoppers passing by. Ultimately, these customers were more likely to leave without making a purchase. Other scholars examine relationships between observed variables and particular outcomes. Sampson (2012), for example, integrated his physical disorder data with survey data from Chicago residents to reveal that what had been considered “broken windows” effects were actually racialized perceptions of disorder.

A Methodological Need: Quantitative Data on Convenings

While findings from some of the SSO-based studies cited above have implications for third-sector theory and research, they deal primarily with actors and interactions in public (streets, parks) or semi-public (stores, classrooms) environments. However, while some third-sector activity occurs publicly (Sampson et al., 2005), such events are a small subset of CSO convenings–– “the intentional assembly of two or more people for some public purpose under the auspices of an at least quasi-formal organization” (Baggetta & Bredenkamp, 2019:10). They include not only relatively rare public events like protests or celebrations, but also the myriad other meetings and events that usually occur out of public view.

Research on CSOs, social capital, and civic engagement are replete with questions about convenings (Edwards, 2014). Who participates? How common are certain participant experiences? How do certain experiences impact civic participation or organizational outcomes? Answering such questions requires detailed, valid, reliable, generalizable, and theoretically relevant data about the characteristics and dynamics of large numbers of convenings. Methods used in third-sector research have struggled to capture such data.

  • Ethnography produces detailed, valid, reliable, theoretically relevant data on a wide range of participant experiences because expert observers, independent of the setting being studied, can witness, record, and analyze an array of convening dynamics (e.g., Eliasoph, 2009; Grubb & Henriksen, 2019). Ethnographies are limited, however, in scale and thus, generalizability.

  • Administrative records and other databases often include many cases, but details about convenings must often be inferred from organization names, mission statements, and activity descriptions (e.g., Kaufman & Tepper, 1999). The validity of inferences drawn about convenings from such data are questionable because convenings are not observed.

  • Histories of organizations provide detailed and valid pictures of what convenings were like, as scholars use archival materials and interviews to build rich descriptions of organizations and their activities (e.g., Barakso, 2004). However, to reach many cases, histories must be coded into categories for reliable comparison. This requires narrowing the theoretical focus to a small set of broadly defined organizational characteristics (e.g., Skocpol, 2003), losing much of the detail available in each source.

  • Organizational documents like newsletters, websites, and reports can be assembled in large numbers, but they typically do not offer detailed data on convenings. When they do, they likely highlight unusual events. Such sources, therefore, are better for providing insights on organizations as a whole (e.g., Schnable, 2021). Meeting minutes and press releases are exceptions. However, minutes are often sparse, stylized documents (Stone, 1996). Similarly, press releases also tend to refer to particular events hosted or attended by CSOs (Bail, 2012), offer stylized accounts aimed at particular audiences, apply only to a small, public subset of CSOs convenings. The validity and generalizability of minutes and press releases is therefore limited.

  • News reports and social media posts about CSO events can offer substantial detail on convenings (e.g., Gaby & Caren, 2012) and often have the advantage of being written by direct observers, rather than participants caught up in the activity, which may improve data validity. These sources, however, typically focus on large, dramatic public events, which comprise a fraction of CSO activity (Earl et al., 2004).

  • Key informant interviews or surveys (e.g., Albareda, 2018; Chaves, 2004) and surveys of CSO participants (e.g., Verba et al., 1995) can provide reliable data across many cases. However, data collected through these methods are often theoretically limited and questionably valid because respondents are asked to recall details about a particular prior convening or homogenize their reports to refer to convenings in general. Field surveys, administered with participants during a convening, offer more valid reports because respondents can include details about the convening in progress. However, such studies typically focus on rare, large, multi-organization events, like mass protests (Fisher et al., 2005).

While these data sources have strengths for answering particular research questions, they have important shortcomings when studying CSO convenings. Systematic social observation offers comparative advantages to each. First, because SSO data are collected by trained observers, the data collected can be more detailed than information recorded in organizational documents and more valid than data generated from recall-based surveys or interviews where respondents report on convenings in general. Second, because SSO data are collected in the field, findings are more easily generalizable to other CSOs. Third, because SSO tools are designed by researchers, they capture more theoretically relevant data than organizational documents or histories, and those data are more reliable across observations because they are recorded similarly from the start. Fourth, because SSO can be applied to the full range of CSO convenings, findings are more generalizable than those generated from examinations of news, social media, organizational documents, or field surveys. Finally, standardizing data collection allows a large team of observers to reliably collect data on substantially more cases than ethnographic studies.

Developing an SSO Tool for Community-Based CSOS

SSO has recently made inroads into the study of CSOs. In particular, Baggetta and Bredenkamp (2019) adapted SSO to collect data on the internal dynamics of student organizations’ convenings. Their study demonstrated SSO’s feasibility to collect observational data on CSO convenings at scale and the added-value for theory testing when analyzing such data. Their analysis contradicted Coffe and Geys’ (2007) findings about bridging social capital among individuals in CSOs, while confirming Paxton’s (2002) findings about bridging ties across organizations.

Although a novel application of SSO, Baggetta and Bredenkamp’s study had a narrow focus (student organizations) and an inefficient data collection format (paper forms). To become more broadly applicable for third-sector research, the SSO tool needed to be: 1) adapted for use in a wide range of CSOs and their convenings and 2) converted to a digitized and customizable data entry format. We made these developments through a 15-month pilot study.

Based on a review of the CSO literature, conversations with CSO leaders and scholars, and materials from the Baggetta and Bredenkamp (2019) study, we created an initial version of the SSO-for-CSOs tool. We recruited three large CSOs in Indianapolis—a community organizing coalition, a neighborhood council, and a business association—whose convenings served as observation sites to test the tool. While these organizations share a broad commitment to improve the conditions of the local community, they approach that goal from vastly different philosophical and tactical backgrounds and include very different participants. Because analogous organizations exist in most cities nationwide, these organizations allowed us to observe convenings across an array of characteristics likely to appear in other CSOs and communities.

We trained a team of nine research assistants—diverse in gender, race/ethnicity, and national origin—on the tool and procedures for observing and coding. To ensure coding consistency, a project manager assigned observers to convenings, reviewed data files, and debriefed observers after observations. At times, pairs of research assistants observed the same convening for training purposes or to assess inter-observer reliability. The project manager solicited feedback from observers about problems and suggestions for items, categories, or other aspects of the tool. We met weekly with the project manager to discuss modifications. At the conclusion of the pilot study, the tool included 97 items grouped into 10 thematic modules. Although we cannot describe all items here due to space constraints, we briefly introduce the module themes.

Characteristics and Arrangements of the Physical Space

While spatial dynamics have been an area of rising interest in social science research (Small & Adler, 2019), spatial perspectives have made limited inroads into CSO research (Fine, 2012). Because capturing characteristics of physical space is a common focus of SSO (e.g., Sampson, 2012; Whyte, 1980), the approach provides an opportunity to expand CSO research on spatial dynamics. Our SSO tool includes items about the location, size, boundaries, and entry points of convening spaces, the presence of relevant objects in the space (furniture, technology), and other space characteristics (lighting, temperature).

Counts, Characteristics, and Co-Presence of Participants

CSO research has long been concerned with the creation and sustenance of social ties among participants (Woolcock, 2011). Such dynamics depend heavily on who participates. To track these dynamics, the tool allows researchers to enter counts of participants at six time points and to demographically categorize participants based on their self-presentations.Footnote 1 This feature provides real-time tracking of diversity among convening participants—an area where research that uses other methods is likely flawed (Firat & Glanville, 2017).

Cross-Demographic Interaction

Arguments about CSOs creating “bridging” ties (Putnam 2000) rely on interactions across lines of demographic difference. However, diversity at a convening does not necessarily mean that cross-demographic interaction occurs (Fulton, 2021a). Indeed, CSOs often struggle to facilitate such interactions (Braunstein et al., 2014; Weare et al., 2009). As such, the SSO tool records whether any cross-demographic interaction occurred.

Counts and Characteristics of Participants Exercising Leadership

CSO leadership is an area of renewed scholarly interest (Mathews, 2020). The SSO tool includes a section to record characteristics of convening leaders (defined broadly as anyone who takes control of the convening at any point). This component enables researchers to track the composition of people exercising convening-level leadership, not merely people with leadership titles.

Boundaries Between In- and Out-Groups Drawn by Participants

Drawing boundaries between social groups are common in many social settings (Lamont & Molnár, 2002). CSOs are no exception––participants regularly make statements that define in-groups and out-groups. The SSO tool contains items to document which gender, racial/ethnic, age, and political groups the participants treat as in- or out-groups.

Procedures and Norms That Structure Convening Activity

What happens at a convening is governed by a combination of formal rules, established practices, and tacit norms (Fine, 2012; Rogelberg, 2019). The tool includes items to capture various procedures and norms including those governing the start/end of convenings, languages used, discussion protocols, breaks in formal activity, and participants’ movement.

Convening Activities

Participants engage in a wide array of activities during CSO convenings (Baggetta, 2009; Doussard & Fulton, 2020; Eliasoph, 1998; Fulton & Wood, 2018). The SSO tool contains items for tracking 26 activities, identifying which participants produce and consume the activity’s content and whether the activity occurs with all the participants or in sub-groups.

Deliberation, Strategizing, and Decision-Making

Common activities are discussions and decisions about organizational actions (Larrick, 2016). The tool can document the breadth of deliberations and decisions related to recruiting/training new members; planning activities; forming partnerships; and influencing individuals, institutions, or policies. The tool can also capture the rate, scope, contentiousness, resolution form, acceptance, and implementation of collective decisions.

Frequency and Content of Discussions About Public-Sphere Topics

While most convening discussions focus on core content or organizational business, there is long-standing theoretical interest in discussions that include topics beyond an organization’s boundaries (Eliasoph, 1998; Skocpol, 2003). The tool can document the discussion of public-sphere topics including whether discussions are explicitly political, how many participants engage, and how long they last.

Styles of Interaction

Ethnographies provide vivid descriptions of particular styles of interaction common within CSOs (e.g., whether participants hug) (Eliasoph & Lichterman, 2003). The tool contains items for 14 interaction styles (e.g., interrupting, clapping, or laughing), allowing researchers to document how often they occurred and whether any occurrences appeared awkward (e.g., a hug in a no-hugging group).

Data Entry Format

We worked with IT specialists to construct a digital version that could be administered via tablet computer and whose data could be uploaded to secure servers. Digitizing increases the method’s efficiency, streamlines potential modifications, reduces data transcription errors, and makes it more user-friendly.

Overall, the pilot study enabled us to develop and test our SSO-for-CSOs tool and make it suitable for collecting data across a broad range of CSOs.

Application and Illustrative Results

The dataset from the 15-month pilot study contains data from 99 convenings.Footnote 2 We use these data to demonstrate the viability and value of applying SSO to study the internal dynamics of community-based CSOs by presenting example distributions of convening characteristics and outputs, and by connecting convening-level data to organizational outcomes. Convening characteristics are the raw materials from which social dynamics arise. Characteristics can be physical, like the size of a room or the objects within it, or social, like the demographics of participants or established norms for interaction. Characteristics set the stage for action and interaction to occur. Convening outputs are features of convenings that emerge from the actions and interactions of participants amid the convening’s physical and social characteristics. Some outputs may be intended, like decision-making, while others may be spontaneous, like a handshake. Organizational outcomes are the products of organizational efforts (e.g., recruitment and retention, donations generated, participants at public events) and the impacts of those efforts beyond the organization (e.g., recognition by public officials, changes in public opinion). Convening outputs are the causal bridge between characteristics (e.g., demographic composition) and organizational outcomes (e.g., effective public influence campaigns). For illustrative purposes, this section describes variation in one set of convening characteristics—interaction styles—and one set of convening outputs—cross-demographic interaction. We then provide examples for how researchers can combine such data with organization-level data to assess their influence on organizational outcomes.

Interaction Styles

Ethnographers have noted substantial variation across CSOs in how participants interact with each other and the behaviors they consider acceptable within convenings (Eliasoph & Lichterman, 2003). Some CSOs have boisterous gatherings while others feature only quiet, restrained speech. The extent to which such styles vary across CSOs and across convenings within CSOs remains an open question.

Our observers used the SSO tool’s styles of interaction module to document the number of times participants engaged in particular types of interaction during the convening (see Fig. 1 for a simulated screenshot of the data entry interface). We present data on three of those types: hugging, interrupting, and shouting. Figure 2 shows the percentage of convenings by organization during which participants hugged, interrupted each other, and raised their voices. At 87 percent of the community organizing coalition’s convenings participants hugged each other, whereas hugging occurred at only 16 percent of the neighborhood council meetings. Meanwhile, participants in the business association were the least likely to interrupt each other with two-thirds of its convenings having no interruptions. In contrast, participants in the community organizing coalition and neighborhood council were twice as likely to have participants interrupt each other during their convenings. Similarly, at least one participant had raised their voice at 60 percent of community organizing coalition’s convenings, whereas shouting occurred at only 3 percent of the business association meetings.

Fig. 1
figure 1

Simulated screenshots of data entry interface for illustrative items

Full size image
Fig. 2
figure 2

Percentage of convenings during which participants hugged, interrupted each other, and raised their voices

Full size image

With such data, researchers can analyze whether and how interaction styles relate to organizations’ ability to navigate emotionally heated deliberations (Lei & Lehmann-Willenbrock, 2015) or foster and sustain member engagement (Yukich et al., 2020), especially when members are asked to take difficult or risky actions (Fulton et al., 2019; Oyakawa et al., 2020). Researchers can also combine these data with organization-level data to examine relationships between a CSO’s interaction styles and outcomes including recruiting volunteers, retaining staff, fundraising, and event turnout.

Cross-Demographic Interaction

Research on organizational diversity indicates that an organization’s ability to realize the benefits of being diverse depends on how much cross-demographic interaction occurs among the organizational members (Fulton, 2021b). However, most large-scale research on CSO convenings does not collect data on the occurrence of cross-demographic interaction (Weare et al., 2009).

Our observers used the SSO tool’s cross-demographic interaction module includes items to document the occurrence of interactions between socially dissimilar people for each social group at the convening. We define interaction as any meaningful communicative act that suggests a social tie. For example, any purposive physical contact (e.g., a handshake) or speech (ranging from “hello” to a full conversation) counts as an interaction. While cross-gender interaction was observed at almost every mixed-gender convening, cross-racial/ethnic interaction varied across convenings and organizations. We present data on the dimension with the greatest variation: interaction between white participants and participants of color. Figure 3 shows the percentage of convenings by organization during which various cross-racial/ethnic interaction occurred. Interaction between white and Black participants occurred at 94 percent and 90 percent of the convenings hosted by the community organizing coalition and the business association, whereas white-black interaction occurred at 71 percent of the neighborhood council convenings. Interaction between white and Latinx participants was twice as likely to occur at a community organizing coalition convening than at a neighborhood council or a business association convening. However, interaction between white and Asian participants was more than four times more likely to occur at a business association convening than a coalition or a council convening.

Fig. 3
figure 3

Percentage of convenings during which cross-racial/ethnic interaction occurred

Full size image

The likelihood of cross-demographic interaction occurring depends partly on the demographic composition of the participants; however, demographic diversity does not guarantee that cross-demographic interaction will occur. Documenting cross-demographic interaction allows researchers to measure the actual occurrence of social bridging rather than assuming that social bridging occurs when demographically different people collocate. Further, such data allows researchers to assess the regularity with which cross-demographic interaction occurs and determine whether such rates change over time in response to organizational efforts. Finally, these data enable researchers to examine the independent effects of demographic composition and cross-demographic interaction on outcomes like the CSO’s effectiveness in recruiting and retaining participants from underrepresented demographic groups.

Discussion and Conclusion

Our results demonstrate the viability of applying SSO to CSO settings. Using the SSO-for-CSOs tool, researchers can reveal variation across convenings among many variables of theoretical interest. Such quantitative description is essential for understanding how frequently the dynamics hypothesized by theorists and identified by ethnographers occur in CSOs and how those distributions vary across organization types (e.g., recreational groups versus advocacy organizations), participant demographics (e.g., gender exclusive convenings versus gender mixed), and geographic contexts. Scholars could also use SSO data to analyze how convening characteristics and outputs influence organizational outcomes such as organizational survival and community impact by connecting convening-level data to organization-level outcome data.

The method has limitations. Although observing convenings in situ provides more details about what happens in CSOs, some dynamics will remain unobservable (e.g., “backstage” conversations) or too complex to code (e.g., topic content). Similarly, many of the interpretive, symbolic, meaning-making components of convenings that ethnographers uncover are beyond the scope of SSO. SSO also shares some of the challenges that ethnographers face in gaining access to field sites. However, this challenge can be eased somewhat by providing a copy of the data collection tool to organization leaders when asking them to participate in a study (seeing that the data collected will not include specific details of internal discussions can ease concerns). Also, organizations may be interested in seeing the patterns of action and interaction across their convenings once data are collected.

SSO tools also share some of the limitations of larger-N studies. As with surveys, SSO topics and categories must be settled in advance so that data are comparable. New insights that arise in the field typically cannot be added until the next round of field work. SSO is also labor-intensive, making it difficult to obtain observations at numbers as large as online or telephone surveys or administrative datasets. In effect, SSO provides middle-range data that can complement insights derived from other techniques.

Balancing its limitations, SSO offers several important advantages relative to other attempts to scale up observational research, like multi-person, multi-site ethnography. To thickly describe a field setting and identify emergent themes, ethnographers on collaborative teams need PhD-level training in both the method and the relevant theoretical and empirical literature. Team members produce lengthy field notes and hold extensive discussions to ensure consistency and completeness (see Baiocchi et al., 2013; Blee, 2012). In contrast, SSO predefines coding categories, reducing demands on observers and making it easier to train and deploy them. Once trained, an observer can complete many observations independently, as new concepts and categories are rarely (if ever) added once a study has begun.Footnote 3 The process is closer to face-to-face survey interviews, like those conducted for the US General Social Survey, than multi-person ethnography projects. With good planning and logistical support, 10 observers could attend 200–300 convenings across ten or more organizations in four months. With modest staff increases, a year-long study could complete 1,000 observations across dozens of CSOs. While not reaching sample sizes in the tens-of-thousands like some survey studies, this increase in scale still markedly outpaces ethnography.

Obtaining extensive, observational data on convening characteristics and outputs at a relatively large scale is critical for advancing scholars’ understanding of CSOs’ internal dynamics and corresponding outcomes. The tool we developed and used to collect convening data demonstrates its utility in studying the internal dynamics of community-based CSOs and indicates its viability and value for applying this method to a substantially larger sample of organizations. We hope SSO—using our tool which we developed to apply to any convening held by any CSO or new tools developed by others—will become a standard methodological tool for CSO scholars. Our tool is available for other researchers to use, with access information posted on our project website (https://oce.indiana.edu/). We look forward to hearing from interested scholars. We expect that more extensive use of SSO will produce better understandings of CSO convenings, their effect on organizational outcomes, and how CSOs impact society.