Introduction

The contribution of grassroots or community-based initiatives (CBIs) towards sustainability transitions is the subject of growing interest (Seyfang and Smith 2007; Middlemiss and Parrish 2010; Feola and Nunes 2014; Frantzeskaki et al. 2016). The aim of this article is to widen the scope of assessing the impacts of CBIs beyond the focus on environmental dimensions presented in the previous article of this special issue. It presents and tests a methodological framework for the multi-dimensional assessment of 37 CBIs across six European regions (supplementary material 1), in which the impact of these initiatives in terms of carbon emissions is considered, but is presented alongside their capacity to promote social capital, social inclusion, human capital externalities, economic revitalization, financial sustainability, and innovation.

To this end, we combine a comparative and multi-dimensional assessment exercise to a multi-criteria analysis (MCA) technique. The aim is to identify which initiatives and typologies of initiative perform best with respect to a plurality of dimensions of societal sustainability, in order to highlight those initiatives that show the best potential and, consequently, demand/deserve closer attention, primarily in terms of policy-making; we identify success factors, commonalities and differences between CBIs, and relations between dimensions of impact.

The assumption is that socio-technical or sustainability transitions require changes along with many different dimensions: political, technological, organizational, economic, social, cultural, etc. (Markard et al. 2012). These dimensions, we believe, provide mutual reinforcement, and thus make a strong contribution to the capacity of initiatives to persist, grow, upscale, spread (Forrest and Wiek 2014; Seyfang and Longhurst 2016; Ehnert et al. 2018a; Gorrisen et al. 2018), and ultimately produce an impact in terms of societal sustainability.

CBIs have indeed been investigated from a number of different points of view; they have been regarded as venues to address environmental, social, and institutional resilience (Buijis et al. 2016); knowledge and learning (Hjerpe et al. 2017; Luederitz et al. 2017; Wolfram 2018); empowerment (Hopkins 2008; Seyfang and Haxeltine 2012; Wolfram 2018); behavioral changes (Heiskanen et al. 2010; Middlemiss 2011; Walker 2011); ecosystem services (Colding et al. 2013; Krasny et al. 2014); social or “grassroots” innovation (Seyfang and Smith 2007); and market innovation (Arentsen and Bellekom 2014); to mention just a few (see also Seyfang et al. 2013; Forrest and Wiek 2014; Frantzeskaki et al. 2016; Wolfram 2018).

Assessments of the performance or impact of CBIs in these regards have already been attempted, but those attempts differ from the one proposed hereafter in several crucial respects. Often, transition initiatives are the object of assessment, including community-led—but also governmental and sometimes private—projects. These analyses are usually aimed at a taxonomy of the characteristics of initiatives (Castán Broto and Bulkeley 2013; Ehnert et al. 2018a; Mattijssen et al. 2018) and more rarely at an evaluation of their outcome. In this last regard, several assessment frameworks have either simply been proposed (Luederitz et al. 2017; Raymond et al. 2017), have been applied to a small number of case studies (Graugaard 2012, Schapke et al. 2017), or have proven more qualitative in nature (Bai et al. 2010). Forrest and Wiek (2014) proposed an assessment framework focusing specifically on CBIs, which has been applied to four initiatives (2015). Krasny et al. (2014) proposed various methods for assessing ecosystem services produced by CBIs, but applicable only to ecology initiatives. Of particular relevance is the framework proposed by Hobson et al. (2016), which the authors co-created with 20 community groups, but did not apply directly. Besides this, cross-case, cross-domain, and cross-country applications of multi-dimensional assessment methods are almost completely missing from the literature, as the same authors mentioned above have often lamented (Forrest and Wiek 2014; Luederitz et al. 2017; Hobson et al. 2016). Evaluations of specific domains are more frequent, and that of community energy in particular. Seyfang et al. (2013), for example, presented the results of the first UK-wide survey about the “success factors” of community energy projects. Similarly, Hicks and Ison (2018) compared 25 community energy projects (see also Lantz and Tegen 2009; Seyfang et al. 2014). Other assessments are limited to a single issue or dimension of impact, e.g., behavioral change (Middlemiss 2011) or diffusion pathways (Seyfang and Longhurst 2016; Boyer 2015), to give but two examples.

Besides being limited in terms of sectoral and geographical focus, previous research often identifies the strengths and weaknesses of CBIs by assessing the (self-)perceptions of their founders or participants (Axon 2016; Byrne et al. 2017; Hicks and Ison 2018). In this, the actual impacts of initiatives are not always distinguishable from the motivations behind them, while in reality the two may diverge substantially. Moreover, there are dozens of interesting and valuable case studies, but to generalize from this research is not straightforward, particularly because the selected cases are often success stories (for a review see Sengers et al. 2016). This is not to say that these studies do not provide useful findings: on the contrary, this research has been crucial for establishing the scope of the assessment we present hereafter. Our goal, in this framework, is not to disprove previous research, but instead, to consolidate existing findings based on quantifiable and comparative evidence.

The innovativeness of our methodology concerns not only its multi-dimensional character, but also the possibility of it being applied to very different CBIs that operate across a broad spectrum of activities, economies, political contexts, and geographical locations. In the next few pages, we first present our methodological toolkit: the assessment criteria we chose, and how they have been identified, measured, and weighted based on the preferences expressed by CBIs on the one hand and a panel of stakeholders on the other. Based on this indicator set, we provide a comparative assessment of the average performance of six categories of CBI: community gardens, solidarity purchasing groups, food cooperatives, recycling, sustainable mobility, and renewable energy initiatives. Finally, we present a multi-criteria analysis of individual initiatives, based on an outranking exercise, and a sensitivity analysis of the influence upon the results of both the chosen criteria and their weights. The result is a ranking of both individual CBIs and their typologies, based upon their multi-dimensional performance.

We are aware that a definitive assessment of community organizations according to a uniform metric is challenging. This is because of the radical variety of CBIs and of their organizational practices, even within each domain of active citizenship. This article does not neglect these issues, and indeed presents some solutions to address these methodological challenges, but should still be considered experimental and inevitably partial. The results we present are obviously influenced by the choices we made about what to measure and how. However, we believe these results provide useful insights both to quantify the overall potential of CBIs in promoting sustainability transitions and to qualify how different initiatives perform differently in this regard. Some further reflections on those challenges and the limits of this study are presented in the concluding sections.

Data and methods

Sample and data source

The analysis presented in this paper was conducted within the European research project TESS (Towards European Societal Sustainability) (www.tess-transition.eu). For the aims of this research, a CBI was defined as a collective action that is initiated and managed by a group of individuals that feel they share a connection—whether of interest, place, lifestyle, culture, or practice—and have self-organized in order to implement projects to serve the community. The sample has been obtained from an initial mapping of 269 eligible initiatives conducted in 2014 based on local knowledge and snowball sampling. Subsequently, a selection process was undertaken (see Tikkanen et al. 2019) that resulted in a list of 63 initiatives. Primary data collection was conducted in 2015 via face-to-face interviews. Due to a lack of reliable and complete information, the initial sample was reduced to 37 initiatives (Supplementary material 1).

Assessment indicators of community-based sustainability initiatives’ performance

In order to identify the assessment criteria presented hereafter, we conducted an extensive review of existing literature which permitted the extraction of the most recurrent meanings and impacts ascribed to CBIs (see Celata and Sanna 2014). The assessment criteria were therefore identified by the authors in collaboration with the project’s research team and further refined via a survey of what CBIs themselves perceive as being their most important aims (Table 1). This permitted the selection of those dimensions that were judged important by at least 65% of initiatives, and whose average rating in terms of importance (see the next section) exceeded 7 out of 10. Furthermore, we excluded those dimensions of performance that cannot be translated into assessment indicators, either because they do not produce a measurable outcome or because of the unavailability of complete and reliable information. We then translated the selected dimensions into quantifiable assessment indicators, described hereafter, which were also based on relatively inductive testing of alternative indicators (for further information see Celata et al. 2016). A discussion and sensitivity analysis of how the choices we made influence the results of the assessment is presented in the “Results” section.

Table 1 Criteria, indicators, and dimensions applied for the assessment of the performance of community-based sustainability initiatives
Full size table

The assessment framework is based on eight indicators, which in turn are grouped into four dimensions (Table 1). The first two indicators concern the environmental impact of CBIs. As discussed in Landholm et al. (2018), a cross-sectoral framework for greenhouse gas (GHG) accounting of CBIs was developed, which permitted the extraction—among other things—of two indicators. The first is an estimate of the total mitigation by each CBI of climate change through the reduction of CO2 emissions per year, intended here as a measure of carbon reduction. The second is a measure of GHG avoided, thanks to the CBIs, expressed as a percentage of the individual carbon footprint of beneficiaries per year. This second indicator is intended here as a measure of “carbon efficiency,” a proxy of the ability of individual CBIs to provide efficient solutions for reducing GHG emissions. The baseline scenario against which these measurements are obtained is a standard counterfactual, defined by how the beneficiaries of the initiative would obtain the same service, on average, in the absence of the CBI’s activity (see Landholm et al. 2018).

Another dimension considered in this study concerns the social impact of CBIs, i.e. their ability to provide opportunities and some sort of infrastructure for socializing, community building, and the strengthening of ties between CBIs’ participants and within their communities. We opted first for a measure of “social capital” to be applied at the scale of individual organizations. This required avoiding any indirect or contextual measurement of outcomes such as the ability to enhance trust. Instead, we opted for a “grounded” conceptualization of social capital—close to that originally proposed by Pierre Bourdieu (1980)—intended here to show the extent to which CBIs provide opportunities for building or strengthening social ties. Moreover, the indicator is meant as a measure of “bridging” social capital, i.e. the extent to which CBIs are able to build ties between individuals who are not already part of the same social circle (Gittell and Videl 1998). The components on which the indicator is constructed—and the underlining assumption—are that the more opportunities or events CBIs provide for socializing and face-to-face interaction, the more those events are attended by the largest possible number of individuals; and the higher the number of people who did not know each other previously, the higher the potential of the initiative to create new social ties and strengthen social capital.

The fourth indicator also concerns the “social” dimension of CBIs, and a frequently discussed topic in relation to grassroots organizations: social inclusion, intended here as the ability of CBIs to involve a diversity of individuals. The existing evidence in this regard is mixed and often some forms of community activism—for example alternative food networks—are accused of being more or less exclusionary, in that they involve disproportionally fewer minorities or low-income individuals (see Argüelles et al. 2017). In order to measure the social inclusion potential of CBIs, we opted for an unweighted sum of the normalized score of the following five sub-indices: the standard deviation from a balanced composition of CBI participants in terms of gender; the proportion of beneficiaries who have low income; the proportion of disabled people among the CBI’s beneficiaries; a Shannon index of diversity in terms of beneficiaries’ country of origin; a Shannon index of diversity in terms of beneficiaries’ age groups. We assumed that the more CBIs achieve in terms of gender balance, involvement of non-nationals, diversity in terms of beneficiary age, and the capacity to reach poor and/or disabled people, the more inclusive they are.

Regarding the economic functioning and outcomes of CBIs, this dimension was translated, initially, into an internal measure of financial sustainability, i.e. the ability of CBIs to sustain themselves financially. A specific problem, frequently discussed in relation to non-profit organizations or associations, is the degree of diversification of revenue sources (Tuckman and Chang 1991). To measure this, we used the Herfindahl index and calculated, for each initiative, the square of the percentage share that each of six revenue sources represents, divided by its total revenue, in order to account for both the number of revenue sources and the extent of revenue dispersion. The rationale here is that the more initiatives rely upon a diversified set of funding sources, the more resilient they are likely to be with respect to internal and external changes or shocks (Dinnie and Holstead 2017; Ehnert et al. 2018b). Additionally, in order to identify the external economic impact of CBIs, we opted for a simple measure of productivity by dividing the yearly expenditures of CBIs (expressed in purchasing power parity to eliminate the differences in price levels between countriesFootnote 1), by the total labor input from employees and volunteers in terms of full-time employment equivalent (FTE). This sixth indicator is therefore considered to be a proxy of the local economic impact of CBIs.

A further dimension for assessment is in regard to the capacity of CBIs to experiment, create, nurture, and diffuse innovation and knowledge. The first of the two indicators in this regard, and the seventh indicator, is a proxy of the potential of CBIs to produce human capital externalities. In line with existing research, we assume that knowledge is diffused both through formal training, learning, and informational activities, and through informal occasions for face-to-face interaction. We therefore measured the intensity to which CBIs provide both formal training and informal venues for knowledge diffusion. We assumed, moreover, that the value of these opportunities in terms of enhancing human capital, depends on the level of the skills available within the CBI, which varies between initiatives due to their different domains of activity, the technical complexity of these activities, and other contingent and contextual circumstances. The level of skill is measured as the mean between the number of years of education and work experience needed to acquire the most common skill employed by the CBI, and those required to achieve the highest skill.

The eighth indicator measures CBIs’ innovative capacity; i.e. their ability to provide or improve new products, technologies, production processes, or organizational models. Providing an objective measure for the outcomes of innovative efforts of CBIs (for example in terms of new products) is extremely challenging and runs the risk of being reductive, given the breadth of CBIs’ efforts towards innovation. We therefore opted to assess their innovative potential by adapting a scheme proposed by Bergek et al. (2008), which is rather broad and identifies seven sub-functions of an innovation system: we asked the initiatives whether or not they have perceived innovation as relevant to their activities (pressure for change); we measured the intensity of training activities that the CBIs organized (which is intended as a proxy for the propensity of CBIs to promote knowledge diffusion); if they tested innovations developed by others (experimentation); created any new goods/services/markets (market formation); collaborate with others in the production or testing of innovations (networking); have registered any patents (patenting); and/or if other organizations use the innovation/s they produced (replication). Given that the survey is based on CBIs’ self-perception, the information provided by initiatives was verified and filtered by the authors based on the description that the initiatives provided regarding the abovementioned innovative activities, also in the light of the difficulty of discerning what should be considered “innovation.” These seven components were added and, consequently, all equally weighted, except for the most important and/or diversified sub-functions, namely knowledge diffusion, market formation, and patenting—which were assigned double weighting.

Each of the selected criteria is a proxy for its underlying performance dimension, which are not measures of outcomes but rather of the “process,” i.e., the potential capacity for the CBI to produce an impact. Where required, the indicators have been divided by the number of CBI beneficiaries, in order for the indicator not to be influenced by the size of the initiative; this is the case for the social capital and human capital indicators. In the end, none of the eight indicators shows a significant correlation with the scale of the initiatives. In order to test the interdependence of the criteria, a Pearson correlation coefficient was calculated for each pair of indicators (supplementary material 2). Six significant correlations were highlighted, which confirms that some dimensions of performance are to an extent related—an issue to which we will return in the next sections. Insofar as the fact that significant correlations never exceed 0.54 (which is the case for innovativeness and carbon efficiency), we excluded collinearity, and all variables were then considered for further analysis. A summary of the ten criteria and a short description of the corresponding indicator are provided in Table 1 (for complete formulae, see supplementary material 3). Normalized scores for each CBI included in the sample are reported in supplementary material 4.

Weighting the importance of the assessment dimensions

In order to assess the performance of CBIs across several dimensions, and to run the MCA, each of the criteria/indicators already described has been weighted based on assessing the opinions of a panel of 25 stakeholders. A “stakeholder” was defined as someone with an interest in, and sound knowledge and/or experience of the functioning of CBIs, but who is not directly connected to any specific CBI. The panel was selected from a pool of 50 potential participants in order to guarantee balanced participation of stakeholders in terms of profile (policy-makers, experts/researchers, practitioners, representatives of networks of CBIs, activists, representative of citizens) and area of expertise.

Insofar as the MCA criteria were measured according to an ordinal scale, stakeholders were asked to assign a rating to each criterion that expressed how important they perceived each dimension of CBI performance to be, in order for that CBI to contribute to sustainability. This was aimed at eliciting the personal view of respondents based on their individual experience rather than the official stance of any organization.

In addition to the ratings expressed by stakeholders, we estimated the second set of values or weights: those expressed by the 63 CBIs included in the original sample (see Tikkanen et al. 2019), who were asked to rate the importance of each performance dimension for their specific initiative. A summary of the questions asked to both stakeholders and CBIs is shown in the right-hand column of Table 1. The main question is in supplementary material 5. A summary of the normalized ratings expressed by the different categories of those interviewed is presented in Fig. 1.

Fig. 1
figure 1

Importance of eight dimensions of the performance of community-based sustainability initiatives as perceived by stakeholders and by initiatives

Full size image

Some interesting divergences which merit brief discussion have emerged. Stakeholders acknowledge the contribution CBIs can make in strengthening social ties, the skills of their beneficiaries, and, to a lesser extent, their potential in terms of economic impact, social inclusion, and innovation. However, they regard financial sustainability to be a problematic issue and are skeptical about the environmental impact. On the other hand, environmental goals seem to be high on CBIs’ agendas, followed by economic revitalization, while innovativeness and, to a lesser extent, social capital are judged less important (see also Fischer et al. 2017).

Method for multi-criteria analysis

An MCA facilitates the comparative assessment of diverse entities (in our case CBIs); when these have different impacts or performances, the measurement of which cannot be aggregated otherwise. A vast number of MCA approaches are available (see Belton and Stewart 2002). For the purposes of this analysis, and as explained in Celata et al. 2016, the ELECTRE I method was chosen. ELECTRE (ELimination Et Choix Traduisant la REalité) is an outranking method based on pairwise comparisons, where alternatives are compared with respect to different criteria, and which results in a degree of dominance of one alternative over another (Ibidem; Mendoza and Martins 2006; Huang et al. 2011; Cinelli et al. 2014). These methods are particularly flexible and useful to combine radically different performance dimensions or measurement metrics, and are frequently applied in multi-stakeholder contexts (Saarikoski et al. 2015). Moreover in our case, all the conditions for the applicability of these methods outlined by Figueira et al. (2005 pp. 136-137) are satisfied. In terms of the normalization technique, we applied vector normalization, as suggested by different authors (Triantaphyllou 2000; Zamani-Sabzi et al. 2016; Vafaei et al. 2017). A longer description of the method and the parameters we adopted is presented in the supplementary material 6.

Sensitivity analysis

The main limit of the MCA is that the results may be substantially influenced by how criteria are weighted, as well as by the indicators that are used to express each criterion. In order to test the robustness of the results, a sensitivity analysis was therefore conducted. In an MCA, sensitivity analysis can be used to test the influence on the results of the chosen method, of the criteria, or of the weights (Geneletti 2013). The most widely used sensitivity analysis is that of weighting, which is performed by changing the set of weights assigned to the criteria, in order to ascertain their degree of influence on the final decision. Establishing criteria weights is in fact one of the most challenging components of an MCA. This process implies some degree of arbitrariness and uncertainty, as it may reflect the subjectivity of decision-makers. In our case the final ranking is the one obtained by applying the preferences expressed by stakeholders, because those are the ones to whom the results of our analysis are addressed. At the same time, we cannot assume that the values we obtain from our panel of stakeholders are indicative or representative. By using instead the values expressed by the larger sample of the CBIs (originally 63), as reported in the third section, and checking the extent to which the results vary, we can confirm the validity of our results. To the same end, we also perform a sensitivity analysis by attributing an equal value to each criterion. The final aim is to check whether the minimum increase and/or decrease of the value of the weight creates substantial rank reversals of one alternative with another. The results of this sensitivity analysis of weights are reported in Table 3 in terms of the maximum difference between the ranking obtained by initiatives in the three cases, i.e. when using stakeholders’ preferences, CBIs’ ratings, or equal weights.

The sensitivity analysis of weights returned very positive results. The six top-ranked CBIs consistently rank equally when the set of weights changes (Table 3). The results for those initiatives ranking lower are more varied, but in only a few cases are the differences in the ranks higher than three. The tail of the distribution is relatively stable too, with differences in the rankings that never exceed two. The sensitivity analysis, in short, confirmed the robustness of the results, especially when considering that the MCA presented in this article includes a high number of alternatives and criteria with respect to most of the existing applications of the MCA.

In order to test the influence of each assessment indicator on the results, a second sensitivity analysis was conducted by running eight MCAs where, in turn, the weight of each of the eight criteria was set to zero. The results of this sensitivity analysis are reported in Table 3 as well, in terms of the maximum difference between the resulting eight rankings. These results are more problematic. The head and the tail of the ranking are again more stable, but the differences in the rankings are notable. On one hand this should be obvious, given that these differences are the result of a comparison between a vast number of MCAs obtained by removing one of the eight criteria. On the other hand, this confirms that our results should be interpreted with caution, since as previously mentioned, they are influenced by the choices we made about which specific assessment indicators to include.

Results

Multi-dimensional assessment of community-based initiative typologies

The first result extracted from the set of criteria introduced in the previous sections is the average performance of different typologies of CBI. Given that these averages were extracted from sub-samples with only 5 to 10 observations, the results are expressed in the form of a ranking. In order to test the robustness of the averages and the influence of potential outliers, we compared the ranking of CBI typologies obtained from mean average values with that obtained from mean average values calculated excluding the maximum and minimum values, i.e. those CBIs with the highest and the lowest scores within each of the sub-sample distribution, and for each criterion. This comparison led to the identification of 12 discrepancies. However, none of these differences were greater than one and did not affect the highest and lowest rankings. In Fig. 2, we summarize the results for six categories of CBIs according to the eight assessment criteria; the discrepancies mentioned are solved by attributing an equal ranking to those categories whose position in the overall ranking change when accounting for maximum and minimum values or not. The same data is reported in a table format in the supplementary material 7 and further discussed in the next section for each typology of CBI.

Fig. 2
figure 2

Six typologies of community-based sustainability initiatives ranked from the first to the sixth position according to their average performance in the eight assessment criteria

Full size image

In order to assess the overall performance of each typology of CBI, we ran an MCA on those typologies using the same method described in the previous section. The results are reported in the supplementary material 8 and show that sustainable/community energy is the best-performing typology—mostly thanks to its environmental effects and economic impact—although it is weak in terms of social capital and inclusion. The opposite is true for community gardens, whose overall performance is equal to that of community energy initiatives when criteria are weighted according to the preferences of stakeholders, and rank second only when different sets of weights are used. This is because of the extraordinary capacity of community gardens, relatively to the other typologies, to promote both social capital and social inclusion which, according to stakeholders, are among the most important contributions of CBIs to societal sustainability. Mobility initiatives are third in terms of average overall performance, due to their fairly good performance in environmental and economic terms, and to their very weak capacity to strengthen social capital and to engage with a diversity of beneficiaries. The next and fourth typology is that of recycling and upcycling initiatives. This typology turned out to be the best performing in terms of financial sustainability but the worst performing in terms of human capital externalities and economic impact, while its social and environmental impacts are moderate. Food cooperatives and solidarity purchasing groups have the weakest performance overall, at least according to our metrics and to most of the assessment criteria. The implications of those results are further discussed in the next sections.

Table 2 shows the results according to the same criteria and typologies and from the perspective of how CBIs in our sample assess the degree of achievement of the initiatives’ aims, which is intended as a self-assessment of their impacts. These data can provide an indication of how satisfied CBIs are with what they do and to what they aim for but is not directly comparable with the results of the performance assessment. In terms of dimension of impact, and echoing with what the existing literature suggests, the more positive evaluation is about the capacity of CBIs to provide occasions for learning (human capital) and social interactions (social capital), whereas the economic dimensions seem to be more problematic, as well as the inclusiveness of community initiatives, something that is frequently discussed, as previously mentioned. The environmental impact of CBIs is judged as their weakest dimension, if intended as the total carbon-reduction potential of initiatives, but is seen as much more positive when it comes to their carbon efficiency. In terms of typology, the least satisfied typology is that of community gardens, followed by sustainable mobility initiatives; in our view, this final piece of evidence has more to do with expectations than with actual outcomes.

Table 2 Self-assessment of the degree of achievement (from 1 to 10) of eight dimensions of impact, by six typologies of community-based sustainability initiatives
Full size table

Multi-criteria analysis and ranking of community-based sustainability initiatives

It should be noted that the results reported in the previous section are based on the average performance of initiative typology and, when interpreting these results, any form of ecological fallacy should be avoided. In other words, the individual initiatives may obtain substantial impacts even in those dimensions of assessment where, on average, they perform weakly. In order to assess this, and present evidence about the capacity of each single CBI across the same dimensions of performance, in Table 3 we report the results of the MCA for the 37 CBIs that are part of our sample, together with the typology of CBIs, a brief description of their main activities, and the results of the sensitivity analysis.

Table 3 Results of the multi-criteria analysis and sensitivity analysis. Ranking of community-based sustainability initiatives belonging to the following typologies: community gardens (CG), food cooperatives (FC), re/upcycling (RU), sustainable/community energy (SCE), sustainable mobility (SM), solidarity purchasing group (SPG)
Full size table

For the most part, these results confirm the assessment of CBIs’ typologies, but can also provide additional findings. In particular, all of the six top-ranked CBIs are active in the energy domain which, as already mentioned, is also the best-performing typology. On the other hand, the MCA shows that results between initiatives are to some extents variable, even within the same typology. In other words, some CBIs can have substantial impacts even if their overall typology is weak. More importantly, five of the six best-performing initiatives are active in more than one typology, and all multi-activity initiatives consistently rank in the top 14. Finally, top-ranked initiatives very rarely appear at the top of any separate assessment criterion. More precisely, the three initiatives with the highest overall rank show the best performance in only one of the eight assessment criteria. The implications of those findings will be further discussed in the next sections.

In terms of the relationship between dimensions of impact, another finding which the research highlights is that those dimensions are to a good extent interrelated (supplementary material 2). Indeed the capacity to obtain substantial effects in terms of overall carbon reduction, in particular, is not correlated with the initiatives’ size, but with carbon efficiency which, in turn, is associated with the degree of innovativeness and their potential in terms of human capital. Human capital effects, even more surprisingly, are strongly correlated with the degree of social inclusiveness, an issue to which we will return in the concluding section.

Discussion

Our analysis permits the extraction of several findings that, on the one hand, are not directly comparable with previous research. This is because, as mentioned in the introduction, this research rarely provides quantifiable and comparable evidence across several domains of active citizenship or dimensions of impact, with only a few and partial exceptions (Castán Broto and Bulkeley 2013; Mattijssen et al. 2018; Forrest and Wiek 2015). On the other hand, most of our findings should not be surprising to those who deal with CBIs through more qualitative or case study based research. Our research was meant indeed to confirm the existing evidence by providing more comprehensive, robust, and generalizable findings.

These results clearly show the capacity of community energy initiatives to achieve substantial impacts in terms of both average performance per typology and results for individual initiatives. This is for several reasons: the substantial environmental effects that a transition to sustainable energy resources can bring, the economic impact of energy initiatives, and their propensity to host a variety of activities in addition to energy production (see also Seyfang et al. 2013; Hobson et al. 2016). Community energy initiatives are often North-European and aimed at providing alternative and community-based solutions to electricity or heat production and/or distribution using mainly biomass or solar panels (see also Seyfang and Haxeltine 2012). This typology is by far the highest performing in environmental terms, with a carbon-reduction potential that is more than double that of any other typology (see also Landholm et al. 2018). It is also the best performing in terms of innovativeness, which correlates with its ability to provide efficient solutions for carbon reduction. It comprises large initiatives which are also very often active in other domains, in particular food production. This translates into an important economic impact (see also Sanna 2018) which is twice that of other categories, while the degree of diversification of revenue sources is moderate. Initiatives in this typology, on the other hand, are rarely keen nor able to build social ties in their communities, and also perform weakly in terms of social inclusion; however, they are particularly active in providing occasions for knowledge diffusion, mostly in the form of formal training activities.

The second best-performing typology is radically different. Community gardens are tracts of urban land which are collectively managed by groups or associations of active citizens whose aim is not solely to produce food but also, and sometimes primarily, to provide a recreational or public space (Guitart et al. 2012). These are small initiatives which can have a substantial social impact by (re)connecting their communities and involving the highest diversity of participants (Yotti Kingsley and Townsend 2006; Mattijssen et al. 2018). They also perform well in several of the other dimensions, given that many initiatives experiment with novel food production methods, improve the skills of participants in this regard, have low running costs, and are mostly based on volunteer/free labor, which explains their financial sustainability. In terms of economic impact, results are varied, and some community gardens rank indeed poorly in this regard. Their performance in terms of carbon reduction and carbon efficiency is extremely weak, compared with other categories (see also Landholm et al. 2018). Due to those weaknesses, they can obtain substantial overall performances only to the extent that they host a plurality of activities, with few exceptions.

The third typology is that of sustainable mobility, i.e., initiatives that provide alternative transport solutions for goods or persons, in the form of bike courier services or by facilitating the use of bikes for people’s mobility and, more rarely, electronic vehicles (Ross et al. 2012). Their capacity to achieve substantial environmental impacts varies, as it is correlated with the intensity of their social and political engagement in their communities, which translates into high performance in terms of both social and human capital. The beneficiaries of these initiatives seem to be the most homogeneous in terms of social status, geographical origin, and even gender. The financial sustainability of mobility initiatives is also questionable, given that more than 50% of their budget comes from external sources of funding such as public grants and private sponsorships, significantly exposing them to adverse conditions or changes in funding priorities (see also Dinnie and Holstead 2017).

The next typology is that of re/upcycling, which includes CBIs aimed either at facilitating the sorting or recycling of (mainly) electronic and plastic waste, the collection and processing of food waste, the upcycling of broken bicycles or electronics into usable equipment, etc. (Hobson 2016). Similar to mobility initiatives, the carbon-reduction potential of these initiatives is significant, particularly when they exceed a certain size. They have low running costs and are particularly able to diversify their sources of revenue, but their overall economic impact is weak. Their degree of engagement with communities is lower than most of the other categories, as well as the degree of the technical complexity of their activities, which explains their relative weakness in terms of social capital, human capital, innovativeness, and carbon efficiency.

However, results for individual initiatives in both the mobility and recycling domains are substantially varied, as CBIs in these categories are observed in both the head and the tail of the distribution in terms of individual ranking. They can obtain notable impacts, but only to the extent that the degree of engagement within their communities is intense.

The last two categories are food cooperatives and solidarity purchasing groups (SPGs). Food cooperatives are aimed at the distribution of food or meals and, in some cases, the transformation of primary products based on the principles of cooperativism, social responsibility, and ethical/fair-trade practices (Seyfang 2007); SPGs are networks of consumers that buy directly from local producers or retailers who respect various principles in terms of ethics and sustainability (Grasseni 2013). An SPG does not normally imply the establishment of any formal organization nor any sort of retail space, and they tend to be part of clearly identifiable networks such as the Italian Gruppi di Acquisto Solidale (GAS). The performance of both categories in most of the assessment criteria is weak when compared with other typologies. SPGs may have a significant capacity to enhance social ties between affiliates, but they are unable to attract a diversity of participants. Cooperatives seem to be more inclusive and can also be creative in finding innovative and carbon-efficient ways for processing and distributing food, but their environmental impact in absolute terms is negligible. The financial sustainability of both categories is low; this is because they both rely almost exclusively on a few funding sources - subscriptions and sales of services in the case of SPGs; public grants and private sponsorships in the case of food cooperatives - which make them vulnerable to external changes (Sanna 2018). It should be noted that the abovementioned findings derive from a comparative analysis of only tangible outcomes. Even the least performing CBIs, in other words, may in fact have important intangible, indirect, or long-term effects that are not captured by our methodology (Brunori et al. 2012).

More importantly, and as already mentioned, some of the “top” initiatives are indeed active in domains which were not the best performing when considered in terms of average performance per typology of CBI, and vice versa. What this means is that what specific activities CBIs conduct, as well as their general characteristics, is less important to overall performance than how this specific activity is conducted. Our results show, moreover, that diversification is an asset, as previous research has already stressed (Forrest and Wiek 2014, 2015; Ehnert et al. 2018b). Those multi-activity and cross-typology initiatives that are internally more diverse than single-activity initiatives have a substantial advantage in terms of performance across the entire range of assessment criteria. Finally, the fact that top-ranked CBIs are not necessarily the best performing according to any of the assessment criteria confirms that the performance of the “top” CBIs is in fact explained by their capacity across several dimensions combined.

Limitations of the study

As mentioned in the “Introduction” section, the exercise reported in the previous pages must be considered experimental and limited both by the availability of reliable information and by objective measures of performance, and due to the specific choices we made to assess this performance. In particular, the assessment framework was not entirely co-designed with CBIs. The choice of assessment criteria is indeed crucial, as well as challenging. In this research, such a choice was based on a comprehensive literature review and refined through a survey of what were considered to be the most important criteria according to the CBIs in the sample. Future research may reuse our methodological approach and extend the co-design to the identification of the assessment criteria.

The second challenge was to identify a set of performance indicators for the eight selected assessment criteria. The result was the identification of a set of proxies, which for most capture the potential impact of CBIs and the intensity of their activities in each regard, rather than the observable outcome. Our aim is not, however, to obtain an exact measurement or rating of the performance of CBIs, but rather to create a uniform metric that highlights the best or worst performer within each impact dimension, and then to compare the results by assigning a weight to each of those dimensions.

The choices we made were also influenced by the availability of measurable, reliable, accurate, and complete information. Some of those choices are, of course, debatable and have an influence on the analysis that should be taken into account when considering the results. In particular, as mentioned earlier, our tests confirmed that the results are sufficiently robust with regard to the set of weights we used. On the other hand, when one of the eight assessment criteria is eliminated, divergences in the results of the MCA increase, albeit not dramatically, and not to an extent to compromise the validity of our findings.

Conclusions

In the article, we present one of the first attempts to systematically and comparatively assess the performance of a large sample of CBIs across several areas of impact and domains of active citizenship. What makes CBIs unique is their ability to impact each of the main driving forces behind sustainability transitions. To render visible the societal and environmental importance of these initiatives requires therefore to go beyond a monodimensional and “sectoral” perspective.

We show that different categories of CBIs tend to produce different and even complementary impacts. This is particularly true for the two categories that show the strongest average performance—community energy initiatives and community gardens—and when comparing the impacts in terms of environmental versus social outcomes. Yet, we also show that CBIs which produce the most tangible effects overall are those which are able to overcome such complementarity and obtain significant performance across all the dimensions of community-led or sustainability transitions.

More generally, in order to fully realize their potential, CBIs need to provide efficient, effective, and innovative low-carbon solutions, and also they should engage closely within their communities by providing an open, attractive, and inclusive social infrastructure. The “social” capacity of CBIs is usually regarded as their most important contribution, although the degree of inclusiveness of CBIs is often considered limited. Our research confirms this and shows that this social dimension is not just important per se: the more initiatives are able to attract a diversity of participants, the more they provide occasions for knowledge diffusion and the more they turn out to be creative in providing carbon-efficient solutions, something which correlates with their overall carbon-reduction potential, as well as to their economic impact.

Finally, our research provides generalizable evidence, but in order to obtain a proper explanation of the capacity of certain CBIs to perform better in one dimension or another, a closer look at each of the “best” performing initiatives, and at the variety of contexts in which they operate, is needed. Another challenge for future research is to design an assessment framework in which not only the selection and weighting, but also the definition of the evaluation criteria and of the underlying performance indicators, is conducted jointly with CBIs themselves, and with all relevant stakeholders.