Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

9.1 Introduction

A recurrent feature in the literature on the evaluation of public participation is the distinction between process and outcome evaluation (Abelson and Gauvin 2006, p. iii; Rowe and Frewer 2004). By and large, a focus on process asks how a participation exercise has been implemented and conducted whereas an outcome evaluation is interested in its results, that is, what it has accomplished, what change it has brought, and whether targets have been achieved. While most studies focus on either process or outcome evaluation, it seems clear that the quality of the process most likely will influence the quality of the outcome. This calls for evaluation approaches which link the view on the implementation of the process to the assessment of its outcomes. In the environmental governance literature there are various examples of such conceptions; special emphasis is put on developing appropriate process and outcome criteria (e.g., Blackstock et al. 2012). Also the e2democracy (e2d) project applied an integrative framework for a longitudinal evaluation of a specific form of public participation, that is, citizen panels collaborating with local governments on climate protection; it integrates the assessment of process, outcome, and impact (see Chap. 7).

The e2d project studied (e-)participation processes based on citizen panels collaborating with local governments in seven locations in three countries: Bregenz and Mariazell in Austria; Bremen, Bremerhaven, and Wennigsen in Germany; and Pamplona and Saragossa in Spain. Seven very similarly organized (e-)participation processes, each targeting a reduction of CO2e Footnote 1 emissions at local level by at least 2 % p. a., provided a quasi-experimental field study of the impacts of this special participatory design combining individual and collective activities. Panelists documented their consumption behavior bimonthly over up to 2 years (online using a CO2e calculator or offline using paper, telephone, and mail) and received individual CO2e balances with historical and comparative information in return. Meetings and various events offered opportunities for exchange and additional information was also provided (see Chap. 7). A basic hypothesis was that participation in a collective local climate initiative in combination with individual information feedback would promote pro-climate sensitization and behavior change to curtail greenhouse gas emissions, and that e-participation would facilitate this. To measure and analyze the effects of these long-term participation exercises, the components that determine their implementation and setup need to be identified, since these also represent a major basis for their impacts.

This chapter is structured as follows: Sect. 9.2 explains the basic building blocks of the evaluation framework. Section 9.3 focuses on the first two components, input and activities, and Sect. 9.4 continues with describing the various categories of output of the (e-)participation processes under study. Section 9.5 presents the outcome analyzing core characteristics and representativeness of the participants, usage of participation process output, and the overall evaluation of the citizen—government collaboration by the participants. Section 9.6 sums up the results and draws some conclusions on the relation between output and outcome.

9.2 Output and Outcome as Evaluation Framework Components

The Input-Activities-Output-Outcome-Impact evaluation framework introduced in Chap. 2 and described in Chap. 7 as part of the e2d project’s study design integrates basic building blocks linking the (e-)participation process to its outcome and impact. Referring to these building blocks, the focus in this chapter is on the relation between output and outcome of the participation processes studied. As important preconditions for these two components, two further ones—input and activities—will be pointed out first. The contents and interplay between the single elements displayed in Fig. 9.1 affect the degree of success of citizen participation.

Fig. 9.1
figure 1

Basic building blocks of the evaluation framework

Full size image

  • Input refers to financial and human resources including the immaterial preconditions on which the specific activities can and have to build upon: management, organizational and administrative procedures as well as political commitments of relevant political and administrative bodies.

  • Activities which are necessary to organize an effective participation process not only include those for the provision of appropriate technical tools and contents but also the required communication and information measures to raise awareness among and provide feedback to participants.

  • Output of a participation process includes different devices and products providing for information and communication and refers to all kinds of media, in particular online and offline channels (e.g., polls, meetings, focus groups, newsletters, online forums, monitoring tools/services). Basically, output stands for the supply side of the process in terms of quantity and quality, including the appropriateness of the products for different target groups, their usability and accessibility.

  • Outcome, refers to immediate effects of the output, that is, seen from a demand-side perspective how what is offered is made use of. It covers number and activities of participants, the participants’ composition as compared to the target population, and characteristics of the participants’ contributions.

  • Impact represents the final and most important part in the evaluation of (e-)participation processes. In the case of the e2d project this includes above all the extent of pro-climate changes of awareness, attitudes, and behaviors and measurable changes of CO2e emissions (see Chaps. 10, 11, 12).

While most (e-)participation processes have a relatively short duration of just a few weeks or months, the collaboration of citizen panels with local governments in the e2d project lasted over a period of up to 2 years. This longer time span was necessary to accommodate the bimonthly monitoring of the panelists’ CO2e emissions across different seasons and allowed to gather extended insights into the relation between output and outcome. Before focusing on the output, we will first provide a comparative account of the nature of the input and activities spent to implement and carry out the participation processes at the different locations of the seven citizen panels.

9.3 Input and Activities

The seven collaborative (e-)participation processes on climate protection studied in the e2d project were conducted between March 2010 and August 2012. The concrete local configuration of the citizen panels in Bregenz and the Mariazell region (Austria); Bremen, Bremerhaven, and Wennigsen (Germany); Saragossa and Pamplona (Spain) differed depending on regional peculiarities. Table 9.1 gives an overview of major features characterizing the establishment and management of the participation processes.

Table 9.1 Overview of major input and activities components of the participation processes
Full size table

The institutional background of local organizers of the participation processes differed in the seven locations and included: (a) municipal departments for citizen participation and environmental affairs, (b) a local grassroots organization in the field of sustainable development, and (c) a nongovernmental climate agency. The projects that were directly organized and conducted by municipal departments (Bregenz, Pamplona, Saragossa) enjoyed a significantly higher commitment by local administration and politicians than the other cases. The involvement of companies in the local setting was relatively low in most regions with the exception of the two German cities, Bremen and Bremerhaven.

Following a common approach in principle, the implementation of the seven participation projects varied according to financial and organizational inputs and activities. The level of personnel and monetary resources can be seen as a correlate to the amount of time and care devoted to a participation process.

As Table 9.1 shows, financial and personnel inputs were above average especially in Bregenz and Bremen and below average in the Mariazell case; the five municipalities constituting this region could hardly provide any money for the project; however, members of a local grassroots organization with support from a regional development agency conducted the process with strong personal commitment. The amount of financial and personnel input in the regions is not directly related to the number of participants but is reflected in the output of the process (see Sect. 9.4). The technical implementation mainly differed between the processes in Austria and Germany on the one side and Spain on the other, that is, the licensed KlimAktiv CO2e calculator as the main tool versus an emulated Spanish CO2e calculator with slightly reduced interactive functionality (see Chap. 8). As far as the participant help desk and offliner support is concerned, the differences were less pronounced.

Measures to raise awareness tended to be similar whereas recruitment strategies for citizen panels varied. A core of multimedia-based communication measures and standardized telephone surveys among the local populations were practiced everywhere. Recruitment measures by personal invitation letters were most extensively conducted in the Mariazell region, Bregenz and Bremen. In Saragossa, the city council made use of direct contacts to citizens who had volunteered for participation in regional matters in previous projects whereas Pamplona mainly relied on interested citizens identified during the telephone survey.

Furthermore, differing levels of experience with (e-)participation and participation culture influenced the execution of the participation exercises. The range spans from Bremen as highly experienced, followed by Bregenz, to Mariazell as least acquainted with participation on the other end.

9.4 Output

Figure 9.2 shows the main contents of the output of the collaborative (e-)participation processes in the e2d project (see Chap. 7 for more details). As participants were free to choose the mode of participation, that is, between online and offline communication channels, process output was provided in both modes.

Fig. 9.2
figure 2

Overview of the main categories of output of the participation processes

Full size image

9.4.1 Three Basic Categories of Output

Three categories constituted the main products of the participation process: (1) information offering guidance on CO2e reduction and climate-friendly behavior (via the project website, CO2e calculator, regular newsletters, and various events), (2) bimonthly individual CO2e balances based on consumption monitoring (via an online carbon calculator or a “CO2e household accounts book” on paper), (3) various forms of issue-specific meetings and exchange (e.g., group meetings with expert talks, group excursions, chats with experts, discussion platforms).

In accordance with the project’s focus on environmental democracy, individual bimonthly CO 2e balances played a key role. For this purpose, a carbon calculator was employed (for details see Chap. 8). Although this tool was provided online, panelists had free choice between an online and an offline mode of usage in all regions. Onliners (i.e., participants preferring to communicate online) used the tool directly via a web interface and were supported via e-mail. Offliners (i.e., panelists who preferred traditional media) had an identical paper version, a so-called CO2e household accounts book for periodically reporting their consumption and were regularly contacted by support staff via telephone who entered the data, produced the CO2e balances, and returned them via mail. The CO2e calculator had been adapted to meet the requirements of bimonthly monitoring and feedback which allowed to enter baseline measurement and up to 12 periodic measurements saved in the joint database. Features of the calculator included visualizing the progress of one’s CO2e balance over time and comparing it to those of other participants. The Spanish calculator provided the identical content but did not include line-charts and emoticons and had somewhat limited interactive capabilities: Panelists received the carbon balance as a feedback only immediately after each bimonthly data entry and could not access it until the next time they entered their data while in Austria and Germany this was possible at any time. Apart from these differences, the common tool delivered individual, comparative feedback on CO2e effects (over time and compared to the group average) plus supporting information and learning opportunities to all citizen panels in the same way.

As a basic output component, issue-specific information sources were offered in various forms in all regions. The online calculator as well as its paper version included condensed advice on energy saving and CO2e reduction on each of its sections: heating, electricity, mobility (private car, public traffic, and flights), nutrition, and consumer goods. Additional information offering tips and guidance on climate-friendly behavior was available on local project websites. As these were linked to on the central e2d project website,Footnote 2 contents were mutually accessible for organizers and panelists from all regions. In Bregenz, Pamplona, and Saragossa local organizers also offered information via links to local websites. Other kinds of information provided in the regions involved were regular newsletters, media articles, movie clips, brochures, and training material.

Opportunities for exchange among panelists and with local partners were an important output with potentials for mutual learning, community building, organizing support measures and the like. These included theme-oriented meetings, excursions to local sites relevant for sustainability issues (e.g., energy-saving buildings), discussion talks and online chats with experts, and other events. Each region held public kick-off, midterm and final events where participants received advice on CO2e balancing and saving, information on the local project’s progress and (interim) results as well as on results in the other regions and could discuss these among each other. However, range and number of such occasions varied to some extent from region to region.

Table 9.2 gives an overview of process outputs in the regions and shows that some differed in variety, quantity, and execution. As mentioned in the previous section, factors such as the institutional backing of the local participation process, the extent of company involvement as well as participation experience and other available resources had some influence on the information and support provided to the participants. Depending on local circumstances, periods of CO2e monitoring varied in starting point and length: Bremerhaven and Wennigsen provided measurements for 18 and 12 months, respectively whereas all other locations provided data for 24 months.

Table 9.2 Overview of different categories of process output
Full size table

Major differences between the regions are to be noted regarding the output categories newsletters and events. The number of newsletters provided to participants ranges between four in Mariazell and 15 in Bregenz. The newsletters contained different forms of supporting information related to the core topics of the CO2e calculator (heating, power, mobility, nutrition, consumption) such as sustainable living, energy efficiency, hints for energy saving in the household, and e-mobility. As it concerns events, in Spain both the number and range was low, as these were practically limited to a kick-off, a midterm, and a final event. In Bregenz, the number of events held in addition to this minimum number of three core events was especially high, which also holds true for Wennigsen and Bremerhaven when considering the shorter period of operation. In two regions, additional information channels for citizens were set up by the local institutions: In Bregenz, the department for environmental issues created their own website as a hub for local information such as event invitations and announcements, local news on sustainable development etc., and also stands out with the variety of additional information offered, and in Wennigsen an online chat with a local environmental expert was offered. The climate agency in Bremen prepared a catalog of environmental measures for companies and managed, in contrast to most other regions, to achieve at least some joint activities of local companies.

9.4.2 Usability of CO2e Calculator and Household Accounts Book

For a comprehensive assessment of the usability of the CO2e calculator and the practicality of the household accounts book citizens were asked to evaluate both tools in detail. To assess aspects specific to the usability of the online calculator an index was built summing up the ease of use with regard to login, comparable applications, data entry, suitability to the users’ type of household, conversion of meter data, and the intelligibility of the resulting CO2e balance.

As shown in Table 9.3, the overall usability assessment for the online tool was quite positive in all regions. The results for the Spanish CO2e calculator were slightly better than the results for the Austro—German tool which might reflect a trade-off between reduced interactivity and increased simplicity of operation.

Table 9.3 Usability index of the CO2e calculator—regional level. (Source: panel survey 2; Scale: 1 = very difficult to handle, 4 = very easy to handle; Kruskal-Wallis equality-of-populations rank test; chi-squared with ties = 29.010**; df = 6)
Full size table

Two of the aspects used in the usability index were also relevant for the users of the CO2e household accounts book: the first is household fit (considering different types of households such as rented apartments or detached houses) and the intelligibility of the CO2e balance. In both cases, differences in the assessments of tools were minor since they basically met the same requirements. A closer comparison of the online and the offline tools shows a slightly better overall assessment of the household accounts book (see also Sect. 9.5.4). While 87 % of the users of the CO2e calculator said they completed all tasks successfully, among the users of the CO2e household accounts book 92 % felt successful to a very great or great extent in task completion. Eighty-five percent found the menu items of the calculator well organized and the functions easy to find; among the offliners a slightly higher percentage (89 %) said the book was easy to handle. Online help texts and explanations in the book were assessed equally positively—among each group 90 % agreed to a very great and great extent. For 87 % of the online users entering data via the input mask was easy, while even more, namely 91 %, answered questions in the household book without difficulties, and 95 % found transmitting consumption data via the telephone uncomplicated. This is particularly interesting as the necessary steps for collecting data is the same in both cases. As concerns difficulties in data preparation, both groups gave a similar response: 10 % of the onliners and 8 % of the offliners encountered some troubles in this regard. Being asked about particular difficulties, practical problems with meter-reading, calculating and recording the consumption data were mentioned most often, however, more frequently among the offliners (62 %) than among onliners (44 %). This indicates the high value of the support offered: 93 % of the onliners found the optional e-mail support helpful (49 % to a great, 44 % to a very great extent) and offliners could discuss problems immediately when reporting their data via the telephone.

9.5 Outcome

The relation between output and outcome is not causal in a sense that the former determines the latter. The outcome is affected by a variety of external factors such as levels of awareness and salience of climate change, the role of climate policies among political key issues, and, last but not least, governance and participation cultures at national and regional levels (see Chap. 7). The response to the widespread invitation by the various local organizers and governments to take part in citizen panels to collaborate in the reduction of CO2e emissions is reflected in the amount of participation displayed in Table 9.4.

Table 9.4 Participation in citizen panels and CO2e monitoring over time
Full size table

The registration numbers reached a total of 1159 participants but were far below what could be expected from the level of interest expressed in local population surveys at the start of the initiatives (see Chap. 7). Compared to the targeted population potential, the smallest municipalities (Mariazell region, Bregenz, Wennigsen) yielded the highest participation rates.

9.5.1 Representativeness and Inclusiveness of Participation in the Citizen Panels

Output and outcome are characterized by a certain dynamic over the long duration of the participation process. This implies changes of the contents offered as well as of panel composition and usage patterns over time. Dropout rates were highest in the initial process stages (during first and second measurement) whereas the panels remained more stable in the later measurement periods.

The basic characteristics of the citizen panels participating in the local climate initiatives in e2d show some commonalities but differ in certain respects from country to country. As Table 9.5 shows for the two Austrian panels, a fairly representative composition is given with regard to gender, but not for the distribution by age and still less by education. The age group below 30 is significantly underrepresented whereas older citizens (aged 50 years and above) are significantly overrepresented. The most significant difference, however, is observable with regard to the composition by education: in both panels citizens with compulsory education level only are underrepresented, those with university education are overrepresented. This mismatch is especially pronounced in the Bregenz panel where 40.7 % of the panelists have a University degree compared to 9.6 % in the population.

Table 9.5 Representativeness of panel composition in Austria compared to target population. (Source: Panel survey 1; national census data)
Full size table

As concerns Germany, the overall pattern is similar. The citizen panels in Bremen, Bremerhaven, and Wennigsen also show a rather equal participation of men and women corresponding to their shares in the population; therefore Table 9.6 only presents the composition by age and education since in these cases the deviation is significant. In all three panels the youngest age-group is underrepresented. With nearly three quarters of the participants being aged 50 years and older, the age structure is especially skewed in the Wennigsen panel. Also the educational mix is highly imbalanced in all German panels, more than in all other panels, and most extremely in Bremen: 71.6 % of the participants have a University degree in contrast to 18.7 % in the local population.

Table 9.6 Representativeness of panel composition in Germany compared to target population. (Source: panel survey 1; national census data)
Full size table

In the citizen panels in Spain we find the same overall pattern again, including a representative share of men and women, however, with some noticeable variation as regards the panelists’ age and education. The deviation from the age structure in the population is somewhat less pronounced than in Austria and Germany, though the youngest as well as the oldest age group is underrepresented. Also, the composition by education is less imbalanced than in the panels on the two other countries.

To sum up, the overall pattern reflects a configuration which is well-known from research on political participation and invites explanations by the so-called “SES model,” i.e. socio-economic status (education, income, occupation) and resource factors (e.g., time, money, civic skills; cf. Brady et al. 1995): people with higher education, corresponding civic skills and time resources are more likely to make use of participation offers. Despite purposefully inclusive recruitment strategies and invitations that were distributed very broadly at all seven locations to participate in local citizen panels on climate targets, it was not possible to break this traditional pattern. The obvious failure in adequately attracting young participants has to do with the required ‘long breath’ for a quite demanding participation over the quite long time of 2 years and with important fields of intervention addressed concerning decisions at household level which are to a large extent beyond the decision space of the youth. Apart from that, local differences within the general pattern outlined are largely owed to special local conditions and network effects in the recruitment of panelists (Table 9.7).

Table 9.7 Representativeness of panel composition in Spain compared to target population. (Source: panel survey 1; national census data. Figures for education taken from representative population surveys in Pamplona and Saragossa.)
Full size table

9.5.2 Issue-Specific Core Characteristics of the Panels

Further crucial aspects in characterizing the basic profile of the panelists concern core issues of climate change and citizen participation: the participants’ interest in, attitudes towards and knowledge of climate change. In order to determine this profile, we carried out a cluster analysis as an exploratory approach to detect certain groups according to their homogeneity with respect to a number of relevant variables. The following eight variables were used for the analysisFootnote 3: information about climate change; satisfaction with measures against climate change; interest in politics; interest in climate policies; satisfaction with local participation opportunities; information on local actions against climate change; motivation by environmental concerns, motivation by energy cost savings. The analysis revealed three clusters showing a good fit to the dataFootnote 4 and can be interpreted as follows (see Fig. 9.3):

Fig. 9.3
figure 3

Three cluster model of issue-specific characteristics of the citizen panels. (Source: panel survey 1)

Full size image

  • Cluster one shows above average values in all but two variables which represents the vanguard to be labeled ‘environmentalists’.

  • Cluster two largely shows values oscillating around the mean and can be labeled ‘mainstream’.

  • Cluster three, finally, shows below average values on all variables, most pronounced on information about climate change, which can be addressed as ‘less informed’ compared to the rest.

The distribution across the three clusters in quantitative terms is displayed in Table 9.8. As the row total shows, the cluster labeled ‘mainstream’ is the largest one comprising half of the total; 27.8 % fall into the category of ‘environmentalists’, and 21.4 % belong to a cluster of comparatively ‘less informed’ participants. However, when looking at the local level, there are big differences in this general pattern.

Table 9.8 Distribution of panelists across three basic clusters
Full size table

The share of ‘environmentalists’ predominates by far in Bregenz (73.3 %), followed by Wennigsen and Mariazell. The panels in the two Spanish cities have the highest percentage of participants characterized as ‘mainstream’ (60 %) and the lowest percentage of ‘environmentalists’. In all other locations the ‘mainstream’ clusters are the minority. Among the three German cities, Bremen sets itself apart from Wennigsen and Bremerhaven with just around one third of the panelists being ‘environmentalists’ and showing a higher percentage in the category of ‘underinformed’ related to climate change.

With this composition of the citizen panels we have to accept the fact that the participants in these climate initiatives demanding long-term commitments and continuous input show a special profile: they are characterised by significantly higher levels of interest in fighting climate change, of sensitisation, issue knowledge, civic engagement and beliefs in efficacy of targeted action. In other words, they show attributes of ‘sustainable citizenship’ (Micheletti and Stolle 2012) to a much higher extent than those that are to be found in the population at large. The skewness towards people with such a profile is most pronounced in the panels in Austria and Wennigsen in Germany, whereas the composition of the Spanish panels more closely resembles that of the general population.

9.5.3 Usage of Participation Offerings

A further important element of the demand-side effects is the usage of process outputs such as the CO2e balance and the information offered during participation as well as the attendance of events that have taken place as part of the citizens’ participation or other electronically mediated forms of exchange between participants.

9.5.3.1 CO2e Monitoring Tools

As described above, CO2e balances were available to the participants either online immediately after they entered their households’ consumption data in the CO2e calculator or, for those participants who used the household accounts books, a paper print was sent via postal services. Asking the citizens how often they actually had a look at their CO2e balances, a big difference between panelists who used online and offline communication as well as big regional differences became apparent: while, in total, 42 % of all online panelists took a look at their CO2e balances after every data entry, considerably more, namely 80 % of the offline panelists looked at their CO2e balances every time they got a print of their results. This result was especially influenced by three regions in which the number of offline panelists who checked their paper balances regularly was considerably higher than the number of participants who looked at them online: In the Spanish regions hardly one fourth of the online panelists (Pamplona: 22 %; Saragossa 25 %) checked their CO2e balances on a regular basis. This might be due to the limitation of the Spanish calculator regarding the presentation of how the CO2e results progressed over time as was mentioned above. Also in the Austrian city of Bregenz, the majority stated that they seldomly (42 %) or never (25 %) took a look at the online results. In this case the information seems to have been of less importance because the panel largely consisted of environmentalists who already knew about their consumption related CO2e emissions very well.

While some forms of citizen participation such as consultations rely on the relevance of knowledge and opinions shared by citizens, a collaborative activity such as the e2d project depends much more on the quality of data provided by the participants. Important measures in this respect are the accuracy levels of reported data (see Chap. 15) and their validity in terms of plausibility and consistency (see Chap. 8). Comparing accuracy levels shows that data collection via telephone interviews provided support for valid measurement from the beginning, whereas online reporting in general delivered less valid results initially but improved during the project.

In spite of the overall lower numbers of online participants who took notice of their CO2e balances on a regular basis, in total, with nearly 90 %, an equal percentage of online and offline participants felt that observing their results over time has shown them the relevance of their personal consumption behavior. Likewise more than 80 % of all participants said that the CO2e results in the various areas of consumption such as heating, mobility or nutrition gave them helpful hints for improvement in those areas.

9.5.3.2 Information Offerings

The two major ways of offering information on climate issues to the participants were the bimonthly newsletters and the local project websites (with excerpts as printed versions in the CO2e household accounts book). The newsletters were sent via email or postal services. The paper version received more attention in all regions: 91 % of the offline participants and 69 % of the online participants read the newsletter often or every time. In general, the information provided on saving energy was found equally valuable by both groups throughout the regions. When being asked if the information was useful for reducing their personal energy consumption, participants were also positive in general, but participants in the German and Austrian regions were more critical than their Spanish counterparts. Some participants also shared the information on climate saving with friends and family. Interestingly, more offline participants (31 %) than online participants (24 %) handed the newsletters on to others.

Both groups could access the local project websites. Online participants visited the project website on a regular basis; less surprisingly the majority did this in accordance with the measurement intervals on a bimonthly basis and another 28 % once or several times a month. The website did have some relevance for those who preferred the offline version for their bimonthly CO2e measurements; 25 % of them visited the website regularly. In addition, three quarters of all participants used further information on climate protection on the Internet or in other media.

9.5.3.3 Events and Online Exchange

In most regions a number of events were organized to provide opportunities for advice, discussion and exchange on climate issues and sustainable living among participants. In total, more than 50 % of the panelists took part in the events offered, especially in the start-up, midterm and final meetings and workshops. At some locations various additional events were organized, such as excursions (visited by 6 % of the panelists), lectures (2 %), festivities (2 %) and individual advice-giving (1 %). In Austria and GermanyFootnote 5, the majority of those who participated in events stated that they had an active exchange with others at the events and that this showed the importance of climate friendly behavior. More than three quarters of the attendees discussed open questions with other participants, not always in mere uniformity of views, since 20 % also noted that their statements had not been very openly received.

In addition to the events described, opportunities for exchange via online fora have been offered in all regions. However, 95 % of the panelists never or hardly ever used the local forum. The majority felt no need to use the fora and a quarter of the participants did not use them due to time restrictions. It seems that the regularly required contributions to CO2e measurements over a long time absorbed practically the entire time budget participants were willing to devote to the climate collaboration. Of those who used a forum, most participants only read posts of others and only a quarter of them actively posted a statement. In effect, the local fora were primarily used for procedural and tool-related topics, however, they were in general assessed very positively with a large majority saying that organizers and participants reacted in a supportive way in fora discussions. A large number of users in Bregenz, Wennigsen, Pamplona and Saragossa also felt that they have learnt from fora contributions on CO2e saving, whereas this was much less the case among users in Mariazell, Bremen and Bremerhaven.

9.5.4 Evaluation of the (e-)Participation Processes

After an active engagement in the project over a period of 2 years at most locations, participants were asked to evaluate the output and different aspects related to the setup of the (e-)participation process and to give an overall assessment of the collaborative exercise.

As Fig. 9.4 shows, between roughly 60 and 75 % of the participants at the seven locations assessed the project as ‘very good’ and ‘good’.

Fig. 9.4
figure 4

Overall assessment of the (e-)participation project

Full size image

Overall, participants highly appreciated the regular calculation of personal CO2e balances and possibility to compare their results over time and with others in the region, the information, newsletters and advice on energy saving, the personal supervision and friendly support as well as the collective effort. The most critical overall assessments were given in Wennigsen and Pamplona. In particular, participants in Wennigsen suggested improving and simplifying the CO2e calculator and in Pamplona citizens wished for more events for community building and exchange. In addition, when asked what should be done differently, the most prominent suggestions across the regions were to recruit more participants by providing additional incentives and by raising the media coverage of the initiative, and furthermore, to offer even more personal advice and information events.

For evaluating the offers at each location in detail panelists assessed the quality of the information material, the website, the CO2e tools and the support in using the tools. Moreover, panelists gave their opinion on how trustful their data have been handled as well as on the events and opportunities for exchange offered.

As Table 9.9 shows, all items offered during the climate initiatives received positive assessments. The data protection standards of the exercise were especially highly appreciated throughout the seven panels, with particularly high values in Bregenz and Mariazell. In Austria and Germany also the information material and events were assessed as being of rather high quality by the panelists, whereas the results in Spain differ: there the panelists’ ratings on information material were especially high, but on events especially low. Overall, the websites got comparatively modest ratings and the exchange opportunities got the lowest ratings compared to the other items, which was especially pronounced in Spain.

Table 9.9 Evaluation by the panelists of various items offered during the local climate initiatives. (Source: panel survey 3; means based on a 4-point scale: 1 = not good, 4 = very good)
Full size table

Comparing the assessment of the CO2e calculator and the CO2e household accounts book shows that the household accounts book tended to be evaluated better than the online calculator, with the exception of Bremerhaven and Pamplona, where there was no significant difference. Likewise, the support for using the CO2e household accounts book adequately was mostly assessed as better than the support for the online calculator—in this case except for Bremen and Pamplona where the ratings for both were essentially equal.

For the panelists’ evaluation of the citizen—government collaboration some further aspects are crucial, in particular their perception of the commitment of other actors involved such as local administrations and businesses, and the transparency of the process. As shown earlier, the general setup of the participation processes at the seven locations differed somewhat, e.g. regarding the organizational and executive support, the commitment of local administrations and the involvement of local companies.

As Table 9.10 shows, throughout all locations panelists assessed the organization and execution of the (e-)participation process with slight variations as good to very good. In comparison to this positive result, the participantsFootnote 6 found the commitment of local administrations rather meager.

Table 9.10 Evaluation of regional processes by panelists. (Source: panel survey 3; Means based on a 4-point scale: 1 = not good, 4 = very good)
Full size table

The only exception to this can be found in the Austrian city of Bregenz where the corresponding ratings were relatively high. When it comes to the involvement of local businesses in the collaborative effort we find similar results: apart from a slightly better result in Bregenz, the panelists at the other locations tended to assess company involvement as rather not so good. Eventually, we see more positive ratings regarding the transparency of the exercise: Austrian and German panelists were rather satisfied to very satisfied with the communication on interim results of the collective CO2e saving effort. By contrast, they felt less well informed on the activities and results of the partner projects going on in parallel in the other local communities. Apart from the few deviations noted, the results of the process evaluation appear rather consistent throughout all regions and with the patterns of results presented above on input and activities.

9.6 Summary and Conclusions

According to the building blocks of the evaluation framework depicted in Fig. 9.1, inputs in terms of resources as well as activities to organize an effective participation process provide the basic components to produce adequate process outputs and outcomes. Comparing the inputs and activities in seven citizen—government collaboration processes on local climate targets showed differing organizational settings in terms of institutional backgrounds, financial and personnel input and levels of experience in citizen participation. Information measures on the local projects started out in a similar fashion in all seven regions, whereas recruitment strategies eventually differed according to pilot experiences and local circumstances. For example, the city of Saragossa could partly count on a registered volunteer group of prospective participants; in the city of Bremen a representative selection of inhabitants was invited to participate; and in the rural region of Mariazell local organizers recruited participants with a mix of invitations via postal delivery to local households, public announcements and personal networks.

Interestingly, these different recruitment strategies resulted in partly similar characteristics of the citizen panels though, showing an overrepresentation of citizens with higher education and of the age group of 50–64 years. In the Spanish panels the distribution was a bit less skewed in this respect and represented a wider reach into mainstream strata of the local population. This holds also true with regard to the core characteristics of the panelists regarding their initial information status and interest in environmental issues; both were rather high to start within the other regions, especially in the Bregenz panel in Austria.

In addition to the commitment of local administrations to the objectives of the (e-)participation processes and the availability of adequate technical means for information, communication and data monitoring, also the financial and personnel input as well as experience of the local organizers with (e-)participation were important foundations for providing adequate project outputs. In Bregenz, financial input, personnel input as well as the organizers’ participation experience were rather high, whereas in Mariazell all three factors were assessed comparatively low. In Bremen, the financial input was medium while personnel input as well as experience were roughly on a level with Bregenz. In all other cases the three input factors were more or less provided on a medium level.

Within the varying organizational setups some categories of output were offered in a rather equal fashion in all regions. This included local project websites, an online CO2e calculator together with an equivalent CO2e household accounts book for consumption monitoring at individual level, online helpdesks as well as support by telephone and online fora. The Spanish CO2e calculator had identical content but less interactive functions than the Austro-German device. It was perceived as easier to handle which is reflected in a better overall evaluation of the tool. However, due to limited interactivity the calculator’s results, i.e. the CO2e balances, received less attention by the panelists in Spain. Panelists in all regions particularly appreciated the trustful handling of the data they provided as part of their participation in the local projects.

Other output categories noticeably differed between the seven locations in quantity and variety, especially the number of newsletters and the events offered to the citizens. The number of newsletters was highest in the Austrian city of Bregenz and in the two Spanish cities. Bregenz together with Wennigsen also had the highest number of events, whereas the two Spanish locations stand out from the rest with the lowest number. These differences are clearly reflected in the outcome patterns. In the final evaluation, the information material received the best ratings in the Spanish locations and in Bregenz, whereas the events and exchange opportunities were assessed least favorably in Spain, but again best in Bregenz and the German region of Wennigsen. The online fora played a rather marginal role and were not so well appreciated; it seems that the panelists’ time budgets were too much absorbed by regular reporting of their consumption data so that there was not much time and motivation left for other kinds of input.

The overall evaluation of the collaborative climate initiatives at the seven locations yielded a very positive result: between 60 and 70 % of the participants rated them as good or rather good. In view of the long duration and amount of regular active contributions demanded this is an encouraging result. It underlines that the participation format studied in the e2democracy project proofed worthwhile in principle, also from the participants’ perspective. The local level and community-oriented approach, combined with the learning effects of individual CO2e monitoring and a strong role of e-participation, provide a promising route to engage citizens with climate protection. Appropriately adapted, lowering the demand from participants by a shorter duration of consumption monitoring and feedback, and adding flexible advice packages according to individual needs, the approach could also be tried on a broader scale.