Keywords

1 Introduction

Contemporary civil societies face challenges that need new approaches, approaches that mainly come from third-sector organizations. These new approaches for solving social problems are called social innovation. In order to generate and develop social innovation, these organizations should enhance their potential and improve their management to address the changes that are occurring in society and strongly affect the way third-sector organizations operate and their way of interacting with other stakeholders. In recent years, the public spotlight on voluntary-sector organizations, and their ability to demonstrate effectiveness, has become intense. Monitoring (collecting information and tracking progress routinely and systematically) and evaluation (gathering research and other data to make judgments and determine the value of worth of something) has become a critical part of organizational life. The approach taken in this study is to identify “third sector” with “non-profit sector.” Inside the category of third sector, the Third Sector of Social Action (TSAS) is distinguished by the intended purpose of its organizations. Through the Second Strategic Plan of the Spanish TSAS, we can confirm the definition of TSAS used in this research: “The Third Sector of Social Action is formed by private voluntary and non-profit entities, arising from free citizens’ initiative, that work independently to promote the recognition and exercise of social rights, to achieve social cohesion and inclusion in all its dimensions and to avoid certain social groups exclusion from adequate levels of well-being” (Plataforma de ONG de Acción Social 2013). According to a perspective that takes into account the finalist sense, entities are considered non-profit (not-for-profit organizations) when oriented to meet social needs or, in other words, when the goal of providing services to members or the community prevail over profit. This chapter reports on the range and extent of monitoring and evaluation practices in the TSAS. The research proposed is needed to build an evidence base about monitoring and evaluation practice and its benefits, if any, and to place that practice within the context of funders’ requirements and evaluation activity. We want to identify the resources available to Third Sector Organizations (TSOs) as well as the extent to which organizations had learnt the skills and techniques needed to carry out evaluation.

  • Had they also acquired the skills necessary not only to manage and interpret data but also to make it useful?

  • How had the emphasis on value and on outcomes and impact that were pervading the policy and funding environment affected learning from evaluation and created benefits for users?

This knowledge, that we will have gained from the first two phases of the research, will allow us to develop the third-stage methodology that serves the TSOs to monitor and evaluate their activities systematically over time as well as to serve stakeholders’ evaluation requirements.

2 Background and Current Status

During the last three decades, Spanish Third Sector (TS) has gradually gained presence in the social, political and economic Spanish scene. The economic boom lived until 2008 and the emphasis on the welfare state resulted in a dizzying growth of Spanish TSAS organizations during the years before the economic crisis. They received a large amount of public funding, and the diversification and positive evolution of intervention programmes resulted in the creation of a great number of TS entities. Additionally, regulatory framework changes occurred that caused private financing participation in the sector.

The economic relevance of the Spanish TS, according to the TSAS annual report (2015), is revealed by its contribution to the Spanish Gross National Product (1.51%) and its social implication (29,737 organizations that employed 644,979 workers and had 1,272,338 volunteers). Out of all these organizations, the three that are considered “singular entities” (ONCE, the Red Cross and Caritas) employed 77,000 paid staff and managed a total income of €14,470 million, 1.5% of Spanish GDP (Ruiz 2015).

The financial structure of social service TSOs is highly dependent on public funding. The percentage of public funding was already high before the crisis, 61.3% (Ruiz 2015: 94). However, the degree of dependence of the different TSO clusters is very uneven, so the impact of the budgetary austerity policies applied during the crisis years has also been uneven. Public funds accounted for two thirds of the income of medium-sized TSOs. These were the TSOs that had grown and expanded most during the period of economic prosperity that preceded the crisis. They were also those that suffered the greatest reduction in public funding. Another group of very large TSOs and “singular entities” is less dependent on the public purse (around 33%). Lastly, small TSOs also depend less on public funding (33%), which they mainly receive from the regional and municipal levels of government . One major challenge for the TSOs is diversifying their funding and correcting its concentration on a single source. While this dependence has lessened in part, in the boom years it also increased, placing a number of TSOs in a vulnerable position of excessive dependence and consequent financial risk (Fundación Lealtad 2013). External audits continue to be on the to-do list for many TSOs, as 58.45% do not audit their accounts (Ruiz 2015: 92). The major form of funding is subsidies and grants, rather than agreements and contracts with government bodies. The budgetary austerity policy was applied two years after the crisis began, mainly from the end of 2010 onwards. The reduction in public funding had a broad, widespread but uneven impact on the TSOs. The total public funding of Spanish social sector TSOs fell from €10,480.5 million in 2010 to €8002.34 million in 2013, in other words, it fell by an average 23.6% (Ruiz 2015: 95). By level of government , approximately 50% of the funding is from the regional governments , around 30% from the provincial and municipal councils and 10–15% from the central government .

The social service TSOs budget volume has decreased significantly during the last past years. In the boom years, TSOs budgets figures reflected high expenses, placing a number of TSOs in a vulnerable position of excessive dependence and consequent financial risk (Fundación Lealtad 2013) (Fig. 8.1).

Fig. 8.1
figure 1

Total Spanish TSAS’s income and expenses. Years 2008, 2010, 2013. Source: Authors’ own work

Full size image

The financial structure of social service TSOs has evolved in order to confront the effects of the economic crisis. Although the sector is highly dependent on public funding, a diversification of the funding sources is observed, fundamentally among the singular and largest TSOs. The rest of the organizations are making an attempt at this, but diversifying their funding and correcting its concentration on a single source is still a big challenge to deal with (Fig. 8.2).

Fig. 8.2
figure 2

Percentage of each type of funding in relation to the total funding of TSAS entities. 2008, 2010, 2013. Source: Authors’ own work

Full size image

The degree of dependence of the different TSO clusters is very uneven, so the impact of the budgetary austerity policies applied during the crisis years has also been uneven. External audits continue to be on the to-do list for many TSOs, as 58.45% do not audit their accounts (Ruiz 2015: 92).

The budgetary austerity policy was applied two years after the crisis began, mainly from the end of 2010 onwards. The reduction in public funding had a broad, widespread but uneven impact on the TSOs. The total public funding of Spanish social sector TSOs fell from €10,480.5 million in 2010 to 8002.34 million in 2013, an average 23.6% (Ruiz 2015: 95). Private funding suffered an overall decrease from €3179.09 to €2807.33 but the percentage that private funding represent relative to the total funding of the sector experienced a relative growth (Table 8.1). Largest TSOs balance their accounts better than the smaller ones, even getting to make a positive difference in their outcome . Smaller companies, on the other hand, normally exhibit an average deficit. The more efficient management of largest entities could be due to the provision of monitoring and control mechanisms of the activity, which help managers to take good decisions that result in end-of-year budgets without deficit. Small entities are in a more difficult position, probably motivated by the multipurpose character of the personnel structure, not being used to adjusting expenses during a period of austerity and the absence of monitoring and control mechanisms of the activity. A better-trained and prepared management staff and implementation of activity monitoring systems could be the key to more efficient management of small TSOs.

Table 8.1 TSAS’s total income per funding source
Full size table

Future assessments predict that financing situation for TSOs will aggravate, with funding cuts of 20% or even 30%. This forecast has implications that go further than the sector funding, and make its transformation and overcoming some crucial challenges a necessity. In other respects, funders’ new demands and the need to strengthen the sector image will imply a change in the TSOs’ accountability that should be based on the impact that these organizations generate in society.

Therefore, one of the biggest challenges the sector faces is how to develop an accurate measurement of the impact it generates and how to monitor, evaluate and control the activities performed to generate that impact. It needs not only to rationalize its management in order to increase its effectiveness but to create adequate information systems leading to the achievement of information transparency about the management of the organization’s resources.

To this end, it is essential to develop a methodology that could make impact measurement a tool of efficient management, communication and transparency. An innovative mechanism that permits to show the stakeholders the economic, social and environmental value that the organization is generating should be developed. And to do so, it won’t be enough to show the results obtained but how they have been achieved. This will enable the stakeholders of the organization to evaluate the entity’s enforcement and value generation capacity, and adopt the appropriate decisions.

3 Improving the Methodology for Monitoring and Evaluating

The analysis and measurement of the social impact is an aspect that directly affects the provision of public services and in many cases the use of public resources. In addition, all the stakeholders committed to social impact seek to be able to know which initiatives provide the best results, in terms of efficiency. Pressure on non-profit organizations for more rigorous impact measurement and reporting can come from a range of involved actors. These include the clients, other social purpose organizations and so-called patrons (Ebrahim 2003). The challenges posed by the measurement of social impact in TS entities arise from their ultimate objective: to know the influence generated by the project, policy, action or activity being analyzed, in aspects related to the development, values and human needs. With the growth of the “social investment state” and the gradual phasing out of grant-giving to the third sector, we anticipate that measures of social impact will become increasingly central for the assessment of policies and social action plans and to resource acquisition in particular. Used properly, it also provides an engaging narrative of the organization’s impact.

Social impact assessment serves three principal functions: (1) performance measurement (both for internal and external purposes); (2) to attract funding (and other resources); and (3) to reinforce the organizational mission (Pathak and Dattani 2014). Out of more than 40 approaches that have been developed for measuring social impact (Stevenson et al. 2010), some of the most recognized are: Logic models, Social Accounting and Audit, methods based in Cost-Benefit and Cost-Effectiveness and Social Return on Investment (SROI). Among them, SROI has undoubtedly grown into the pre-eminent means of evaluating net social returns. Although these are the most widely used approaches, there are many others, so it is unlikely to establish a single reference and, in fact, no standard has been agreed upon to measure social impact . Recognizing there are no “one-size-fits-all” solutions, it is important for organizations to choose the approach that best suits their specific environment and requirements. This, far from being an inconvenience, is a positive factor in the objective of pursuing a correct measurement of social impact , since the objective of any organization that takes into account the social impact of its initiatives is to optimize impacts in several dimensions rather than to maximize impacts against any dimension (Maas and Liket 2011); the measurement process can not pose in a unidirectional way. Combined measurements would allow a common framework of comparison, homogenized and benchmarked, that would allow a comparative evaluation of the goodness and virtues of the future projects that are being analyzed. Additionally, using a single approach could ignore the different stages that organizations are at, not only in terms of their size and resources, but in terms of their cultural openness to and capacity for evaluation. Given the diversity of the organizations within the non-profit sector, it is virtually impossible that one single measure of impact or one approach will be suitable for all of them.

The analysis of the different methods of measurement of social impact will allow us to draw some conclusions and set future goals.

The Logic models, such as the Logical Framework Approach (LFA), are, rather than a method, a global project-planning framework that often precedes specific methodologies for measuring social impact . The advantage and attraction of Logic models is that they provide a framework that enables organizations to embed evaluation and performance assessment into the program design and life-cycle process of the program (Zappalà and Lyons 2009). Logic models or the Logic approach to program design and evaluation emerged in the 1970s as a response to the shortcomings of many program evaluations that were being conducted. Since the 1990s, different agencies linked to development cooperation have promoted the application of more advanced approaches than LFA but have taken many elements of this early approach. This included “Results Based Management” (RBM), which focuses the analysis on the achievement of results in the interventions and not only, as before, in the management of invested resources.

In relation to Accounting and Social Audit models (Gibbon and Dey 2011), the internal data collection and analysis procedures (social accounting) are followed by an independent audit of the results (social auditing) before finally disseminating the outcome more widely (reporting). Therefore, it will be important to establish information standards and methodologies and generally accepted accreditation processes, since, as González et al. (2010: 23) indicate “it is true that there are reports commonly and popularly called ‘Social Balances’, but on the other hand they are very far from what could be expected from a document with that name. None of these tools is really useful to help the knowledge of the generation of social value, communication and transparency of the value generated and, consequently, to win it back.” This method has a longer history of innovation and use in the United Kingdom; it has been to some extent eclipsed by the importing and development of the SROI approach, which has been supported by recent UK and Scottish government initiatives.

On the other hand, it would be very useful to design indicators and/or tools that allow us to gain a broader understanding of the reality of projects and their execution, with a focus on social value and impact. In this sense, the Social Balance model could benefit much of the work done in the other methodologies and, in this sense, could serve to offer new analysis to solve different needs.

The methods based on the comparison between costs and benefits have very limited methodological development. The possibilities of use are directed to the adaptation of the concept to the specific casuistry of the projects under analysis and, in particular, for the determination and quantification of those effects of a non-economic nature but that have an evident social impact .

The non-excessive complexity of these models and the intuitive interpretation of the results allow them to be a good initial option for those entities that are not applying more complex procedures in determining the social value of their projects, or for those whose small size limits the implementation of broader and more demanding methodologies.

Based on a more generalized use of Cost-Benefit and Cost-Effectiveness analyses, these methodologies can be taken as a reference for the development of research and new valuation systems (Retolaza and Ramos 2005), allowing to advance in this field towards models of more complex and global management that pursue the maximization of social impact and integrate the entire entity. Cost-benefit analysis has traditionally been used for evaluating the costs and benefits of policies and programmes by government , with more recent work, such as that by Fujiwara and Campbell (2011), looking to incorporate social and well-being considerations into the policy tool. This is the most demanding approach for the analysis of costs and outcomes , as it requires a comprehensive measurement of costs and program impacts. Unfortunately, classical cost-effectiveness analysis and cost-benefit analysis (and therefore any approach that incorporates these methodologies) do not currently incorporate a consistent approach in order to deal with value judgments. Each study reflects the researcher’s assessment about how the costs and benefits are distributed among stakeholders and how the various outcomes are valued.

SROI is probably the methodology most widely applied and the most widely analyzed by researchers (Solórzano et al. 2015). SROI’s application is a relatively young discipline. As a result, there is great variability in how SROI is applied across projects. This makes robust and consistent comparisons across social ventures difficult while rendering the validity of SROI measures vulnerable to contestation. On the other hand, probably the most widely acknowledged quality of SROI is its effectiveness as a communication tool.

SROI has detractors, as seen in Fujiwara (2015), which synthesizes the ideas of those who consider it a vague concept; that its approach to the stakeholders may be limited, which is based in a methodology that it considers to be obsolete and incomplete; that it does not have a clear normative approach; and that it approaches the principles of equality and equity unduly, giving greater importance to groups that are not the most needy. It also finds that the calculation of the ratio is susceptible to bias and that, since its design has not followed the advances in statistics and econometrics, it does not meet the criteria necessary to raise inference tests, which causes the application of statistical methods to infer causality to be limited or even problematic. It is possible to identify three groups of limitations in connection with SROI analysis (Maier et al. 2015). Firstly, there are certain fundamental and irresolvable issues that have the potential to call the method as a whole into question. Secondly, there are some issues that are also irresolvable but that do not preclude using the method, as long as they are understood and knowingly taken into account. Thirdly, there are a number of technical issues that might be remedied as the method matures.

Methodologically, there are many advances that have been proposed in relation to the SROI, given its quantitative nature, although with important qualitative foundations in its construction. Already in Tuan and Emerson (2000) two challenges were posed: on the one hand, the determination of an adequate discount rate—the social discount rate; and on the other hand, the incorporation of the degree of difficulty to reach the proposed social objectives, implying the calculation in social terms of a traditionally financial magnitude—the beta. Subsequently, other authors have explored the methodological keys and conceptual difficulties of SROI, such as Moody et al. (2013) and Pathak and Dattani (2014), among others. In addition to the importance of the aforementioned, it will be important to make progress in setting standards for the allocation of general costs and methodologies related to weightings, which allow us to know the actual counter-value of the initiatives under assessment.

4 Conclusions

Based on the above, we propose, as main conclusions of this research, the main fronts where we consider that work in the field of social impact measurement must progress:

  • The first challenge in measuring and/or estimating social value beyond the lack of common measures is the lack of quality data for these measures overall. Measuring data (social outputs, outcomes , impacts and costs) requires a lot of resources and time. Collecting and analyzing data can be very expensive and this expense is often borne by the grantees of foundations that require such data. Typically, non-profits have limited time and money to pursue activities outside of their mission. Additionally, most non-profits do not have the administrative depth or expertise to track social outcome and cost data.

  • Another significant issue is to improve the transparency of the organizations and the structuring of information based on their usefulness and relevance. Under the premise that resources are scarce and social intervention needs are very high, the entities must be transparent. Likewise, it should be possible to demonstrate a certain degree of usefulness in all projects undertaken that may be observable and identifiable by stakeholders.

  • It is very relevant to advance in the definition and selection of indicators, given the complexity of this point. On the one hand, because it is difficult to collect the value of qualitative aspects such as well-being, social relations or mental health status and to reflect them in monetary values can be sometimes inappropriate. It is important not to force monetization when this is not possible and in this case, it may be useful to use “financial proxies,” ensuring their quality and integrity, in order to avoid the consumption of significant resources for organization and generation of unnecessary or inefficient information.

  • The institutional commitment to measuring the impact of its initiatives, on all stakeholders, both the positive and negative effects, direct and indirect, and on the entire temporal scope of the results of these initiatives, constitutes the third overall challenge. It should be noted that sometimes the social impact of the initiatives is not immediate, so the measurements should be made the moment they materialize, and it is necessary to keep track of those results. To date, it is common to stop analyzing the results of the initiatives once they have been finalized, the funding line has been replaced or the associated policy has disappeared, causing loss of information about its real impact in the long term. Social impact should be measured taking into account the greater good (solution of the social problem) for the greater number of stakeholders (who should be paid attention to and whose problem is solved through the social action implemented) because their interests have intrinsic value. Additionally, in the Practical Guide for Measurement and Impact Management, it is recommended that entities focus their calculation on results, as noted by the European Venture Philanthropy Association (2013: 94), “techniques for doing so (i.e. randomized control groups) are very costly in terms of time and can also raise ethical issues in the case in which potential beneficiaries of the Social Project are excluded.”

  • Another challenge is related to the establishment of adequate communication mechanisms with stakeholders, focused on the information of their interest and in the form and time necessary for decision making, facilitating comparison with other initiatives. And for this, the measurement should be oriented not to the triggers of payment by the promoter, but to the results and impacts actually sought, since its objective is to improve knowledge in the organization and boost its effectiveness and efficiency.

In any case, regardless of the method selected for the measurement of social impact , what is truly relevant for organizations is always to follow a structured methodology, properly reasoned and focused on continuous improvement. And for all those involved—organizations, users, patrons, researchers and public authorities—the important thing is to continue to make progress, so that small progresses continue to be made with the ultimate goal of promoting equitable development, the defence of values and satisfaction of the needs of the disadvantaged.