Introduction

Back in the days when the field of philanthropic and voluntary sector studies just emerged, David Horton Smith (1973) raised the provocative question of what the impact of the third sector is. Flynn and Hodgkinson (2001) treated measuring the impact of the nonprofit sector as one of the major tasks facing third sector scholars and practitioners. In the field of public administration, Lohmann (2007) listed how public administration can be changed by nonprofit organizations as one of the big questions facing nonprofit management. Despite the universal recognition of the importance of these questions, existing literature tends to focus on how public administration (e.g., government funding) and societal conditions (e.g., race and income) influence the size, density, and composition of the third sector (Grønbjerg & Paar.lberg, 2001; Lecy & Van Slyke, 2013). Theory building and testing also predominantly take place in the domain of understanding the determinants of the third sector activities. This unidirectional understanding of the third sector prevents the field from gaining more attention from other fields of studies and limits theory building in third sector scholarship (Cheng, 2019). Even as third sector scholars are paying increasing attention to the performance of the sector, the focus tends to be on how different management strategies impact performance for individual nonprofit organizations instead of examining the sectoral level impact (Berrett & Holiday, 2018; Kim, 2017).

So why don’t we have more studies examining the impact of the third sector on society? Existing literature has listed several key challenges in studying the societal impact of the third sector, including quantitatively measuring social impact (Weisbrod, 2001), building linkages among outputs, outcomes, and impact (Ebrahim & Rangan, 2014; Kendall & Knapp, 2000), establishing the counterfactual to estimate the causal relationship between the third sector and its societal impact (Clotfelter, 1992), and aggregating data at the correct level of analysis (DiMaggio, 2002). While these challenges are legit concerns about establishing the relationship between social impact and the third sector, recent development in the availability of administrative data (e.g., the U.S. Open Government Initiatives) and methodological advancement in establishing a causal relationship using observational data make it possible to reexamine this big question in third sector research. In fact, in recent years, several empirical studies have taken on the task to examine the societal impact of the third sector across different disciplines. For example, in sociology, Sharkey et al. (2017) has examined the impact of local nonprofits on violent crime rates in society. In environmental management, Rousseau et al. (2019) examined how the density of local environmental nonprofit organizations may impact a city’s environmental performance. Compared to the advancement in other fields in addressing this important question for third sector research and the availability of data on societal outcomes (often tracked longitudinally by the government), the field of third sector research is relatively behind.

In this research note, we want to particularly focus on the empirical and methodological challenges in addressing endogeneity in studying the societal impact of the third sector. We use impacts instead of outcomes because outcomes often refer to the changes in individual lives while impacts emphasize the lasting results at the community or societal level (Ebrahim & Rangan, 2014). As previous studies have sufficiently theorized and tested, third sector organizations are formed at least partially in response to government support or social conditions like poverty or lack of education (Steinberg, 2003). Therefore, it is very challenging to convince the reviewers and readers that the third sector makes a significant impact on society without addressing endogeneity in the research design. From a research design perspective, it is also impossible to randomly assign the location of third sector organizations to use experimental approaches to answer such questions.

So, why is it critical to address endogeneity and how can third sector scholars use observational data to deal with endogeneity to assess the impact of the third sector? To answer these questions, we first use a few examples in third sector research to demonstrate what endogeneity is and how it may significantly bias our findings. We then conduct a literature review of more than a decade of quantitative research on the impact of the third sector, both within and outside the field of nonprofit and third sector studies. We also systematically track the type of methods they use to address endogeneity. Based on these findings, we provide a few recommendations for third sector scholars to further advance our understanding of the societal impact of the third sector.

What is Endogeneity and Why is it Important for the Study of Third Sector Impact?

To understand what endogeneity is, we have to go back to the key assumption of the classical ordinary least square (OLS) estimation—the errors must be uncorrelated with the explanatory variables. We call explanatory variables meeting this condition exogenous explanatory variables. However, if an explanatory variable is correlated with the error term, the explanatory variable is endogenous or the statistical model has the endogeneity problem (Wooldridge, 2007). More intuitively, endogeneity means that there are omitted variables or uncaptured causes that both drive the changes of the explanatory variable and the response variable. To more vividly illustrate why we need to take endogeneity seriously in studying the societal impact of the third sector and how it may create a spurious correlation between the explanatory and response variables, let’s consider two classic examples in the nonprofit and voluntary studies.

First, think about the question of how the number of community-based nonprofits may impact the crime rate of communities—a classic and important question of the third sector’s impact on community conditions. It will be very challenging to tease out the causal directions of this relationship. On the one hand, nonprofits may help create social capital and cohesion, therefore reducing the crime rate. On the other hand, the formulation of these nonprofits may be driven by the crime rate of the community (the classical “filling the gap” argument). In other words, nonprofits may self-select them to be located in communities with a higher crime rate. If we run a regular OLS regression between the size of the nonprofit sector and crime rate, we may get the spurious correlation that the more nonprofits a community has, the more crimes there are in that community. However, does that mean that more nonprofits cause more crimes to take place? Without addressing the endogeneity problem due to the self-selection of where nonprofits are located, we are likely to get inconsistent, or even contradictory causal relationships. Also, think about the relationship between medical treatments and mortality. Only sick people go to the hospital to get medical treatments and sick people have higher mortality rates. Therefore, if we only run the correlation between those two without controlling for the self-selection of patients getting medical treatment, we may conclude that medical treatments result in more deaths. For either case, the policy and management consequences of such conclusions can be disastrous.

Second, think about another classic question in the nonprofit and voluntary studies – whether government funding crowds out charitable donations. One key empirical challenge in estimating this relationship is endogeneity or some omitted variables that may drive both variables (Pearl & Mackenzie, 2018). This problem is particularly difficult to address when the omitted variable is hard to measure and collect information on. For example, citizens’ passion for a particular social issue may drive both government funding to nonprofits and charitable donations in that policy subsector, therefore generating an upward bias in the correlation between government funding and charitable donations. In fact, according to a recent meta-analysis of empirical literature of the crowding-out hypothesis, De Wit and Bekkers (2017) found that the findings are strongly shaped by the research methods used in those studies. Studies using the experimental design, which effectively deals with the endogeneity problem by randomly assign the treatment (in this case, information about different levels of government funding to those nonprofits), find that government support decreases the level of charitable donations. However, nonexperimental studies find the opposite pattern – government support increases the level of charitable contributions by donors (De Wite & Bekkers, 2017, p.301). In other words, the choices of whether to control for endogeneity or not may significantly impact the findings and conclusions we get from our studies.

From these two examples and linking them to the question of the third sector’s impact on society, almost all questions we ask may suffer from the endogeneity problem, either through the self-selection of the creation of third sector organizations or some omitted variables that may drive both the creation of third sector organizations and community conditions. Overlooking endogeneity in our study design may generate opposite findings of whether the third sector improves our society. Given the importance of this question to our foundational understanding of the third sector and some of the most critical public policies toward the sector (e.g., the tax-exempt status), third sector scholars must take endogeneity seriously. Given the difficulty in randomly assigning nonprofits to communities, fully addressing endogeneity in third sector studies is not only a hard but maybe impossible task. Even with panel data, this problem of endogeneity cannot be easily solved (Leszczensky & Wolbring, 2019). So, is there a way forward to deal with endogeneity to understand the societal impact of the third sector? Next, let’s turn to the existing literature on the quantitative analysis of the third sector’s impact on society to see the methodological trend of addressing endogeneity and possible solutions.

Existing Quantitative Studies of the Third Sector’s Impact on Society

To provide a benchmark of how existing quantitative studies have assessed the third sector’s impact on society, we conduct a systematic literature review across different disciplines to understand the methods used in these studies. Here below, we provide a detailed illustration of our literature search strategies and inclusion criteria. We then group these articles according to the methods they use to deal with endogeneity and present our main findings.

To accomplish a relatively comprehensive grasp of how scholars across fields have studied the impact of the third sector using panel data, we implemented the following process for our literature search and gathering. First, as it is challenging to nail down those specific keywords in a broad search of empirical studies on the third sector’s impact on society (think about what impacts may mean in different policy subfields), we used our professional knowledge in the field to pick four latest articles that make a causal claim of the third sector’s impact on society as our baseline references: Shareky et al. (2017) from Sociology, Rousseau et al. (2019) from general management studies, Alonso and Andrews (2020) from public administration, and Crubaugh (2020) from third sector studies. By starting our preliminary analysis from the four articles, we were able to build our knowledge on how each discipline has discussed the impact of the third sector and the issue of endogeneity. This knowledge enabled us to plan and conduct consistent literature searches in the next phase of the analysis. In addition to these four articles, we also reviewed all the references of each and included additional articles that cover the impact of the third sector into this preliminary group. Accordingly, we have a total of 12 articles on our list and these articles help us generate a list of keywords for the subsequent literature search.

Based on this preliminary group of literature, we initiated an extensive literature search using Google Scholar. To grasp all relevant literature as much as possible, we devised a unique strategy for our search protocol. As the first step, we focused only on articles that are published in nonprofit journals. We targeted four journals in the discipline: Nonprofit Management and Leadership (NML), VOLUNTAS: International Journal of Voluntary and Nonprofit Organizations, Nonprofit and Voluntary Sector Quarterly (NVSQ), and Nonprofit Policy Forum (NPF). NML, VOLUNTAS, and NVSQ are considered as the three most influential peer-reviewed journals that publish quality scholarship on nonprofit and voluntary studies. They are also the only three Social Sciences Citation Index journals in the field of third sector studies. Along with these three journals, we also included NPF, a journal from the Emerging Sources Citation Index, as NPF focuses on the public policy impacts of the third sector. Within these four targeted journals, we searched keywords using the Google Scholar search engine.

The keywords used for the search were constructed as follows. We listed conceptual names that refer to or are closely related to the third sector: Third Sector, Nonprofit or NPO, Nongovernmental organization or NGO, Community organization, and Civil society. Then we made phrases by linking these names with words implying causal relationships such as effect, impact, and influence. By combining each, we had a total of 15 phrases for keyword search. And we used the Exact Phrase option in the Advanced Keyword Search function of Google Scholars with these phrases as we wanted to include only impacts made by the third sector at this first step. For instance, we can get a list of articles that contains the exact phrase "effect of nonprofit" with this option. As a result of reviewing searched articles to page 30, a total of 3 articles were added to the list as of September 6, 2020 (duplicates with the former step were excluded).

Next, as the second step, we attempted to search the keywords in a broader setting. That is, we did not limit our search to the four targeted journals. Instead, the same phrases used in the first step were searched with the Exact Phrase option of Google Scholar’s Advanced Search function. As a result of reviewing searched articles to page 30, we could include a total of 28 articles into the list as of September 6, 2020 (duplicates with the former step were excluded).

As the last step, we broadened our search setting once again. In addition to targeting all the journals, we replaced the Exact Phrase option with the “All Fields” option so that the search engine can display articles with any part of the keywords in the phrases. We obtained a total of 14 additional articles at the last step as of September 6 (duplicates with the former step were excluded). Figure 1 summarizes each step of our search procedure.

Fig. 1
figure 1

Literature search procedure

Full size image

It should be noted that there may be some limitations in our final list. First, there may be a certain risk of publication bias. In the search process, we only included articles from peer-reviewed journals while leaving out books and professional reports. Because of our focus on methodologies, we expect that peer-reviewed journal articles may better capture methodological advancements in addressing endogeneity than books or other professional reports. Second, there might be a minor bias in choosing eligible academic disciplines. We attempted to consider social science disciplines as many as possible. However, we might indeed miss a few disciplines that cover the impact of the third sector due to the limitation of the author’s knowledge.

Following the above search protocol, we were able to find 57 articles that discuss the impact of the third sector. The list covers a variety of research methods from qualitative case study methods to theoretical arguments. However, we had to exclude the articles that do not utilize quantitative methods from the final list for the purpose of this study. After excluding these articles, 29 articles remain in our database. We also added two more eligible articles that were published after the last day of our search (September 6, 2020) during the review process, which brings the total number of articles to 31. Table 1 shows detailed information about the articles included in our final database.

Table 1 List of articles included in the final database following the order of publication dates
Full size table

Findings and Analysis

With the search protocol, we were able to find 29 quantitative articles that cover the impact of the third sector. In terms of academic disciplines, more than half of the articles in the final list were published in sociology and political science journals. The third sector studies journals have a relatively small portion among the listed articles. In terms of quantitative methods, the final list clearly shows advances in methodological rigor. Until the 2010s, the articles mainly used ordinary least squares regression to test the impact of the third sector. For instance, Smith et al. (1997) tried to explore the "ability to address crime problems (p. 71)" of community-based organizations. Nevertheless, the authors did not consider the issue of endogeneity in their research design so that they were able to show only the linear relationship between the number of community-based organizations and the number of total crimes within the community. After 20 years, however, Sharkey et al. (2017) also attempted to test almost a similar research question on the impact of changes in the prevalence of community nonprofits on the community's crime rate. Unlike the past attempt, the authors point out the issue of endogeneity in identifying the causal effect of nonprofit formation on crime rates. That is, they accept the possibility that the larger formation of community organizations may partly be the consequence of the high crime rate. Accordingly, the authors used the fixed effects estimation and the instrumental variable approach to deal with the endogeneity in testing the causal relationship.

Based on the authors’ careful reading of these 31 eligible articles, we want to highlight three main findings. First, we examine in which academic disciplines those articles were published. As with our expectation, journals in third sector studies (VOLUNTAS, NML, NVSQ, and NPF) lag behind other disciplines in terms of the volume of research on the third sector’s impact on society. According to Fig. 2, half of the articles were published in disciplinary journals such as political science and sociology. However, third sector study journals only have 4 articles published, about 13% of total articles. In other words, most of the articles on the societal impact of the third sector tend to be published in more disciplinary journals, rather than third sector journals. Given the central importance of this question in third sector studies, this should be of concern for third sector scholars. As the study of the third sector flourishes across different disciplines, this finding shows great potential for third sector scholars to catch up with research on this central question of the third sector’s impact on society.

Fig. 2
figure 2

Pie chart of the journals publishing quantitative studies on the third sector’s impact. Note For others, public administration and urban affairs cover most of the eligible publications

Full size image

Second, we turn to the trend of methodologies used in these articles. We group the methodology into three main categories based on its ability to solve the endogeneity problem. The first stream of articles examines the societal impact of the third sector with cross-sectional data by using Ordinary Least Squares or Maximum Likelihood Estimation. Unfortunately, as explained above, these methods are not sufficient to address the problem of endogeneity, thus producing biased and inconsistent estimates for those models (Angrist & Pischke, 2009). Indeed, most authors of the articles in the first stream acknowledged that their results could be compromised by the possibility of endogeneity.

The second stream of articles takes advantage of longitudinal data and uses the fixed effects model to account for time-invariant unobservable omitted variables or the lagged explanatory variables to take care of time dynamics. The use of the fixed effects is very powerful in addressing the omitted variables bias, particularly those time-invariant unobservable variables. Going back to the example of the impact of the third sector crime rate, the population's general attitudes toward violence may be an important variable in explaining the crime rate but it is very hard to measures. If applied researchers can argue that this attitude is not likely to change during a short period of time, using the fixed-effects model helps address this omitted variable problem. Given researchers are hard to get all the information or variables related to the phenomenon they study, the fixed-effects model using longitudinal data offers a more robust estimate compared to the OLS estimators. In addition to the fixed effects modeling, some of the authors in this group lag all explanatory variables and the dependent variables in their model. Advocates of the approach claim that the replacement of \({x}_{it}\) with \({x}_{it-1}\) can solve the problem of endogeneity as \({y}_{it}\) cannot cause \({x}_{it-1}\) and the causality runs only from \({x}_{it-1}\) to \({y}_{it}\). However, the approach of lagged explanatory variables is not sufficient to completely address the endogeneity problem as the assumption for the absence of temporal dynamics among the unobservables is untestable (Bellemare et al., 2017).Footnote 1 We group these two strategies (often being used together in modeling) as the second stream of methodology as they represent a significant improvement over the OLS or maximum likelihood estimation based on cross-sectional data. However, they are particularly competent in addressing the endogeneity problem.

The third stream of articles takes a more developed approach to address endogeneity in studying the societal impact of the third sector. Three modeling approaches emerge from the literature and represent promising ways for third sector scholars to use to further address endogeneity. The first modeling strategy uses an instrumental variable combined fixed effects modeling approach to both address endogeneity and controls time-invariant unobservable omitted variables. To examine the causal relationship between an endogenous variable and a dependent variable, the authors attempt to search for an instrumental variable that only affects the endogenous variable, not the dependent variable, and include it into analysis by using techniques such as two-stage least squares. If the instrumental variable is truly relevant, then it will satisfy both the independence assumption and the exclusion limitation, which allows researchers to obtain the local average treatment effect (LATE), or causal effect of x on y (Angrist & Pischke, 2009). Particularly, Rousseau et al. (2019) used damage from natural disasters, the level of religious pluralism, and city carrying capacity as instruments for the density of environmental NGOs as these factors are closely related to the creations of environmental NGOs but do not have a direct linkage to city’s environmental performance. Sharkey et al., (2017) used the density of arts, medical research, and environmental protection NPOs as instruments for the density of community development NPOs. There are standard statistical procedures scholars could use to test whether the instrument variables are valid and robust (Sharkey et al., 2017, p. 1225). These statistical procedures are important to follow as weak instruments may cause even more harm than the traditional OLS estimators.

In addition to the instrumental variable with fixed effects approach, recent studies also used the Generalized Method of Moments (GMM) method (Cheng, 2019) and the synthetic control method (SCN) (Alonso & Andrews, 2020) to better address the problem of endogeneity in studying the third sector’s impact on society. A dynamic panel GMM estimator takes advantage of the structure of the longitudinal data and uses lags of the endogenous variables to serve as the instrumental variables. It is regarded as a significant improvement over the OLS estimators in terms of its ability to address the endogeneity bias. Ullah et al. (2018) provided a nice guide (with generic STATA codes) for applied researchers to implement the GMM model. However, there are some cautions about using the GMM estimator as the researchers need to predetermine which explanatory variables are exogenous and how many instrumental variables need to be included (Zhu, 2012). In terms of SCN, it alleviates the endogeneity bias by constructing a counterfactual by a weighted average of units in the non-treatment group. Abadie et al. (2011) offered detailed illustrations of how to use the STATA module “synth” to implement SCN.Footnote 2

Taking together the three streams of articles that use different types of modeling approaches to quantitatively study the societal impact of the third sector, we find that most of the studies are still dominated by OLS or MLE estimators using cross-sectional data (18 out 31 articles). Only four articles in our dataset implemented more robust estimation strategies to control for endogeneity beyond the use of the fixed-effects model for panel data and they all appeared after 2017. Besides, all these four articles are outside the main third sector journals we have identified.

Recommendations and Implications

What lessons can the third sector scholars learn from our literature search and the main findings discussed? Here we propose four recommendations for third sector scholars to address endogeneity in their research design and advance the understanding of the third sector’s impact on society.

Refocus on the Big Question of the Third Sector’s Impact on Society

It should be concerning to third sector scholars that few studies in the main third sector journals use quantitative data to directly address the societal impact of the third sector. Although the value of qualitative studies should not be discounted, quantitative approaches have a unique value in addressing the societal impact of third sector activities. Why the third sector exists and what values they bring to society are of core concern to the field of voluntary and nonprofit studies. However, most empirical investigations published in third sector journals focus on the determinants and conditions of third sector activities, while neglecting the ultimate question of impact. With the increasing availability of administrative data to track societal outcomes over time and various initiatives of opening nonprofit tax filing data around the world, third sector scholars need to seize this opportunity and refocus on the big question of the third sector’s impact on society.

Catch Up with the Methodological Advancement in Addressing Endogeneity

Recent methodological advancements in addressing endogeneity in studying the third sector’s impact seem to take place mainly in more disciplinary fields such as sociology, general management, and political science. While this is somewhat concerning, it also creates great opportunities for third sector scholars to learn from other fields and disciplines. Besides, as our data shows, basically all articles which used a more robust estimation than fixed effects of panel modeling to study the impact of the third sector appear after 2017. So the gap is not huge. Beyond the methods we reviewed in this research note, methods such as matching (Stuart, 2010), regression discontinuity (Cattaneo et al., 2019), structural equation modeling (Paxton, Hipp & Marquart-Pyatt, 2011), and difference-in-differences (Angrist & Pischke, 2009) can also inform third sector scholars’ choices in selecting the best model to alleviate the endogeneity problem.

Be Creative and Transparent About Addressing Endogeneity

Despite the importance of addressing endogeneity, it is not always possible to have the perfect instrument or the right data to run those more complex models. Third sector scholars need to be creative and transparent about how endogeneity may influence their research findings and the limitations of their empirical strategies. Besides, even correlational or descriptive information can be very valuable when exploring an important question when little research has been conducted in that area. Cheng (2019) explored the impact of nonprofit spending on government spending on parks and recreation services for the 100 largest U.S. cities in the last two decades. This research suffers from endogeneity concerns as government funding could also flow to nonprofits via contracts or grants (this is less common in parks and recreation services though). Substantively, this potential source of endogeneity suggests that it is more likely to find a positive correlation between nonprofit spending and government spending due to the double-counting created by this cross-sector funding flow. However, as the empirical findings suggest a negative correlation between nonprofit spending and government spending, the discussion of this endogeneity concern further proves the robustness of the findings that as nonprofits spend more on parks services, the government spends less. Therefore, the substantive knowledge of where endogeneity may come and the transparency with these potential limitations are critical as third sector scholars figure out strategies to address endogeneity in their empirical research.

Build Better Theories to Link the Third Sector to Broad Societal Impact

The disproportionate burden of addressing endogeneity borne by studies on the societal impact of the third sector suggests that theories about how the third sector may influence society are underdeveloped, especially compared to theories related to the determinants of third sector activities. For example, theories about how heterogeneous demands of citizens may shape nonprofit density are well established and continuously refined (Paarlberg & Zuhlke, 2019). However, theories about how nonprofits may shape public opinion and citizens’ demand for certain services are not yet well-articulated (Cheng, 2019; Steinberg, 2003). Methodological advancement needs to go hand in hand with theory development if third sector scholars want to better understand the third sector’s impact on society. In this regard, in-depth qualitative and case studies are critical as theory development in this direction of the relationship is still in its infancy. Third sector scholars should also better integrate theories from other disciplines to shed light on the theoretical mechanisms through which the third sector may influence society.

Conclusion

From a methodological perspective of bettering addressing endogeneity in the research design, our article contributes to the growing yet still incomplete understanding of the third sector’s impact on society. It also helps us better understand the paradox of why scholars call for more studies on this important topic yet most of the third sector studies focus on how social conditions impact the third sector. Our literature review points to the urgency and opportunities for third sector scholars to catch up with the methodological advancements in other disciplines. It also highlights the importance of theory building in better connecting the third sector to its desired or unexpected societal impact. As the third sector around the world plays important roles in society yet faces increasing scrutiny from citizens and the government, this is a call that third sector scholars must answer.

As Box (1976) has famously pointed out: “all models are wrong, but some are useful.” While our main goal in this research note is to introduce third sector scholars to endogeneity and how to better address it in research designs, we are fully aware that it is often impossible to eliminate the endogeneity problem in observational studies. Equally dangerous as we ignore endogeneity in third sector research, the excessive interest in addressing endogeneity may make third sector research disconnect from the real world and ignore those big questions of our field, in the name of methodological rigor. Taking endogeneity seriously should not be at the expense of the diversity and provocativeness of the questions scholars ask. Instead, it should help third sector scholars articulate their contributions and clarify the scope conditions of their findings. A critical reflection on endogeneity can also push scholars to pay more attention to underdeveloped theoretical mechanisms. Dealing with endogeneity is not just a methodological or statistical exercise. Third sector scholars must take advantage of their substantive and theoretical expertise to account for endogeneity both theoretically and empirically. We hope our presentation of endogeneity and its remedies in the existing literature help advance the field of research in understanding the societal impact of the third sector.