1 Introduction

It is agreed in the literature on stated preference methods that some respondents do not state their actual value for the good in question. Therefore, in many studies it is checked whether respondents state a protest answer, i.e. that respondents reject (protest against) some aspect of the constructed market scenario. If protesting occurs, stated preference methods might fail to determine the correct economic value of the good in question. Boyle (2003) suggests that there are three main reasons why respondents might not express their true value. The first is that some people may not understand what they are asked to do in the survey, but they answer the valuation question anyway. The second reason is that respondents might act strategically aiming for the change in the provision of the good to be paid for by other people. Thirdly, respondents may protest against some component of the valuation scenario or, as Mitchell and Carson (1989, p. 166) put it, these respondents “refuse to play the game” economists want them to play.

The usual way of differentiating between a true zero WTP and a protest response is to present respondents who are unwilling to pay with a set of debriefing questions. Based on the answers to these questions, researchers decide whether each zero WTP corresponds to the economic concept of value or whether respondents are protesting against some aspect of the valuation scenario. Meyerhoff and Liebe (2006) find that some of those who are willing to pay also hold protest beliefs. This finding is in contrast to the common assumption that only respondents who are not willing to pay hold protest beliefs, and it is in support of Jorgensen and Syme (2000) who argue that censoring of protesters is unjustified. Nevertheless, the typical way of dealing with protest zero responses, i.e., protesters among those who are not willing to pay, is to delete them from the sample (Morrison et al. 2000). In the course of the debate about protest responses, a variety of reasons why respondents might protest have been suggested. A number of possible reasons have been mentioned in the literature: Dissension with specific aspects of the study such as for instance the payment vehicle, the policy context, ethical beliefs indicated by for example lexicographic preferences, misunderstandings or lack of information, fairness aspects, the type of good, institutional settings of the survey, and demographic characteristics of the respondent (Boyle 2003; Jorgensen et al. 1999, 2001; Meyerhoff and Liebe 2006; Mogas et al. 2005; Morrison et al. 2000; Strazzera et al. 2003; Söderquist 1998).

While many issues of different respondent and survey specific aspects potentially affecting protest behaviour have thus been mentioned in the literature, to the authors’ knowledge many of these aspects have yet to be empirically investigated. For instance, Meyerhoff and Liebe (2008) stress the need for future studies to investigate whether protest beliefs and protest responses differ when the constructed market settings vary in terms of, e.g., using taxes as a payment vehicle and/or the dichotomous choice (DC) as a payment question format in CV. In a recent meta-study Meyerhoff and Liebe (2010) found that some survey characteristics such as using an open-ended question format to elicit willingness to pay values, using entrance fees as a payment vehicle or online surveys as a survey mode significantly influences the number of protest responses. However, a limitation of their study is that only survey and not respondent characteristics are taken into account.

The present paper brings a novel contribution to the literature by exploring and explaining potential causes of protest behaviour in Stated Preference (SP) surveys through a meta-study based on full datasets collected from 38 different SP surveys.Footnote 1 More specifically, the objective of the study is to identify both respondent specific variables, which vary for each single respondent, as well as survey specific variables, which vary only across surveys. This distinction between respondent specific variables and survey specific variables is quite important for one major reason: While we would typically aim to minimize protest behaviour in any SP survey, the respondent specific variables offer only very limited options to the researcher for actively pursuing this objective. However, adjusting the survey according to the survey specific variables is to a much larger extent within the grasp of the researcher. In other terms, if the survey specific variables significantly affect the probability of observing protest behaviour, the researcher can actively adjust the overall design of the survey in order to reduce this probability.

The results of the meta-study show that a number of both respondent specific variables as well as survey specific variables have a significant influence on the propensity to protest in SP surveys. Age and parental status affected the probability of observing a protest response in a positive way, while household income and use of the good had a negative effect. With respect to the survey specific variables, characteristics like the good being a market good, employing a CE, longer scenarios and Internet surveys seem to reduce the likelihood of a protest response. The reader should though bear in mind that the results to some extent are limited as this study does not cover the entire range of possible design choices. For example, none of the sampled studies use an entrance fee as a payment vehicle.

The paper is organized as follows. Section 2 briefly describes the data used in this study and presents the modelling framework applied in the analysis. In Sect. 3 the results are presented. Finally, we discuss and conclude in Sect. 4.

2 Data and Econometric Approach

The dataset used in the analysis is based on data from several stated preference surveys comprising 38 samples. The samples cover a wide range of different topics and methodological design decisions, e.g. market as well as non-market goods, CEs as well as CVMs, open ended (OE) as well as DC and payment card (PC) approaches, and different modes of survey and different ways of formulating the valuation questions are represented. Furthermore, within many of the surveys there are also several different experimental split samples used, resulting in even further variance at the survey level. For an overview, the 38 samples are summarized in Tables 2 and 4 “Appendix A” with respect to applied valuation method, number of respondents, identified share of protesters, and a short description of the topic of the survey.

From each of these datasets, we have extracted the information corresponding to the variables described in Table 1. Subsequently, the datasets have been merged into a single large dataset forming the basis for the empirical analysis.

2.1 Econometric Approach

A common feature of our data is that it is of a hierarchical or multilevel nature. At the top level we have the 38 different surveys displayed in “Appendix A”. Nested within each survey we then have a number of different respondents. Hence, we have to take account of both variance at the survey level (level 1) and variance at the respondent level (level 2). Consequently, we use a hierarchical logistic regression to model the probability of observing a protest response (Goldstein 1995). We specify a model that seeks to explain the outcome of the dummy variable \(y_{ij}\) which takes the value 1 if respondent \(i\) nested in survey \(j\) is classified as a protester, and 0 otherwise. In a simple form this can be expressed in the following way:

$$\begin{aligned} y_{ij}^{*} =\alpha _j +\beta ^{\prime }x_{ij} +e_{ij}, \quad y_{ij} =1\left[ {y_{ij}^{*} >0} \right] \end{aligned}$$
(1)

where \(\alpha _{j}\) is the average rate of protest in survey \(j\), \(\beta ^\prime \) is a fixed effects coefficient vector associated with the respondent-level vector of predictor variables, \(x_{ij}\), and \(e_{ij}\) is the respondent-level error which in the logit framework is assumed Extreme Value distributed, \(e_{ij} \sim \text{ EV }(0, \sigma ^{2}\)). Under this distributional assumption, the latent variable \(y_{ij}^{*}\) maps the impact of the \(x_{ij}\) on the probability of observing a protest response, i.e. \(P(y_{ij}=1\left| {x_{ij}}\right. )\), through the index, \({\beta }^{\prime }x_{ij} \), and the indicator function, 1[\(\cdot \)]. Since there might also be a correlation across respondents within a given survey, we add a survey level specific error component by decomposing the average rate of protest in a survey into an overall grand mean, \(\alpha \), and random deviations from that, \(u_{j}\):

$$\begin{aligned} y_{ij} =\alpha +u_j +\beta ^{\prime }x_{ij} +e_{ij} \end{aligned}$$
(2)

where the survey level random error \(u_{j}\) is assumed normally distributed, \(u_{j}\sim iid\,\text{ N }(0, \sigma _u^2)\) and independent of the respondent level random error, \(e_{ij}\). This essentially adds a random effects component to the model, making it a mixed effect model. This entails that the specification in Eq. (2) further assumes that \(u_{j}\) is uncorrelated with \(x_{ij}\). Since we find it quite likely that the respondent-level covariates do in fact contribute to the survey-level residual \(u_{j}\) in our case—i.e. the random effects assumption is potentially violated—we further decompose the survey level error by adding a ‘control function’ incorporating a respondent-level random effects covariate vector, \(\eta ^\prime \), to explicitly take such correlations into account:

$$\begin{aligned} y_{ij} =\alpha +\eta ^{\prime }x_j +u_j +\beta ^{\prime }x_{ij} +e_{ij} \end{aligned}$$
(3)

Assuming that the control function captures all potential correlations between \(x_{j}\) and \(u_{j}\), the random effects assumption is satisfied and the survey-level residual in \(u_{j}\) will contain only survey-level omitted factors that might influence the propensity to protest. As we are interested in testing for the potential impact of survey level characteristics on protest behaviour, we decompose the survey-level average rate of protest even further by incorporating a vector of predictor variables at the survey level, \(z_{j}\):

$$\begin{aligned} y_{ij} =\alpha +\gamma ^{\prime }z_j +\eta ^{\prime }x_j +u_j +\beta ^{\prime }x_{ij} +e_{ij} \end{aligned}$$
(4)

with \(\gamma ^\prime \) being a vector of survey level fixed effects coefficients. Hence, the difference in propensity to protest across surveys is now explained by the observed characteristics of surveys, \(z_{j}\), in addition to the random term, \(u_{j}\), and the random effects in \(\eta ^\prime \). The percentage of total variance attributed to the survey level characteristics which is also known as the intraclass correlation coefficient, \(\rho \), is found by adding the survey-level variances in the random effects component \(\eta ^\prime \) with the residual survey level variance in \(u_{j}\) and then dividing with the total variance:

$$\begin{aligned} \rho =\frac{\sigma _u^2 +\sum _{t=1}^T {\sigma _{\eta _t }^2}}{\sigma ^{2}+\sigma _u^2 +\sum _{t=1}^T {\sigma _{\eta _t }^2}} \end{aligned}$$
(5)

where \(t=1,\ldots ,T\) denotes the different estimated random effects in \(\eta ^\prime \). Similarly, the percentage of total variance attributed to each of the random effects can be calculated by dividing the variance estimate, \(\sigma _{\eta _t }^2 \), with the total variance. Within other meta-studies, similar types of hierarchical model have been applied (see e.g. Bateman and Jones 2003).

The explanatory variables in \(x\) and \(z\) have been chosen on account of a priori expectations of their potential impact on protest behaviour—expectations which are based on economic theory and common sense as well as previous research on protest behaviour. The model is estimated once for the whole sample of studies reported in Table 2 (Model 1) and “Appendix A”. To illustrate the stability of our findings, we additionally exclude the two studies with the highest percentage of protest responses and the two studies with the lowest percentage of protest responses and rerun the model on the remaining 34 studies (Model 3). While the decision to exclude four studies in the sensitivity analysis is somewhat arbitrary, it corresponds to a 10 % alpha trimming approach and should as such shed some light on the sensitivity of the regression results to outliers and extremes. Moreover, as an additional form of sensitivity analysis we run the model also among those respondents who were not willing to pay (Model 2).

2.2 Variables Used in the Analysis

As the aim of the analysis is to establish determinants of protest bidding, the dependent variable is a dummy variable (PROT_RESP) taking the value one if the respondent is classified as a protest bidder and zero otherwise. In all surveys used in this analysis debriefing questions concerning potential protest bidding have been asked in order to determine protest respondents. This reflects the practice employed in the majority of valuation studies that report on protest responses and how they were determined (Meyerhoff and Liebe 2010). However, the approaches differ with respect to both the statements used and the question format employed. This is because still no established theoretical criteria or protocols for excluding protest responses exist (Boyle and Bergstrom 1999). The main difference among the surveys used in this analysis is whether respondents have been asked to choose the statement that best reflects the reason why they are not willing to pay,Footnote 2 or whether they have been presented a set of attitudinal statements and were asked for each statement to indicate to what extent they agree or disagree. In the former, responses are defined as protest responses when the respondent has chosen a statement that is defined as a protest motive (e.g. “The government should use existing revenue to pay for forest conversion”). In the latter case a response pattern was predefined that would identify protesting. For example, if a respondent had answered ‘completely agree’ to two specific attitudinal protest-related statements this was defined as a protest response. The different approaches might influence the number of protest responses identified in each study. However, their influence is beyond the scope of this paper; in the present analysis we take the protest responses as they were defined in each of the primary studies. Table 1 gives an overview of the descriptive statistics of the dependent variable as well as the explanatory variables that are used in the hierarchical model.

Table 1 Description of variables
Full size table

The explanatory variables can be assigned to two structurally different groups: Respondent specific variables varying for each single respondent (the \(x\) vector), and sample specific variables varying only across samples (the \(z\) vector).This distinction is quite important: While we would typically aim to minimize protest behaviour in any SP survey, the respondent specific variables offer only very limited options for the researcher to actively pursue this objective. However, adjusting the survey according to the survey specific variables is to a much larger extent within the grasp of the researcher. In other terms, if the survey specific variables significantly affect the probability of observing protest behaviour, researchers can actively adjust the overall design of their surveys in order to reduce this probability. In Sect. 3, the explanatory variables and the a priori expectations for these are explained in more detail in conjunction with the presentation of the obtained results. The reason for not presenting the a priori expectations here is twofold—first, we want to avoid repetitious listing of variables, and, secondly, the paper would be extended gratuitously.

Examining the samples more thoroughly, Table 2 shows the difference in the number of potential protesters (defined as all those who stated a zero WTP) and the number of identified protesters across surveys. The table reveals that the share of identified protesters varies quite a lot, with a spike in the lower range below 10 % and with another spike between 40 and 50 %. Examining the share of potential protesters, Table 2 again shows a rather wide range across surveys, but more interesting is the share of identified protesters out of the potential protesters (last column of Table 2). Here it is shown that it is generally a rather large proportion of those stating a zero WTP who also end up being a classified as a protester, often above 60 %. This is examined further in the following.

Table 2 Shares of protesters
Full size table

3 Expectations and Results

The results obtained using the hierarchical logistic regression model are displayed in Table 3. Model 1 uses the whole sample and is based on 52,935 observations from the 38 samples. Besides reporting parameter estimates and significance levels, Table 3 reports the odds ratios as a measure of effect size. Generally, the odds ratio measures the ratio of the odds that a result will occur to the odds of the event not happening. An odds ratio of 1 indicates that the probability that a result happens, i.e., in our case to be a protester, does not change when the independent variable increases by one unit. If the odds ratio is above one, the probability that the result occurs is positive when the variable increases by one unit and the other way round when the odds ratio is below one.

3.1 Respondent Specific Variables

3.1.1 Age

The respondent’s age typically exhibits a negative relationship with WTP in SP surveys (Carson et al. 2001). Thus, it seems reasonable to expect a positive relationship between the age and the probability of protest behaviour, as the probability of protest behaviour increases when more respondents report a zero bid in the valuation question(s). Answering a valuation question typically constitutes a cognitively demanding task. If mental capacity reduces, as you get older, a heuristic to cope with such difficult tasks could be to state a protest bid. Furthermore, younger people are often thought of as being more open-minded. This might be a prerequisite for respondents to accept the hypothetical scenario. Meyerhoff and Liebe (2009) establish a link between protesting and the likelihood of choosing a status quo alternative (i.e. a zero bid) in CE. Moreover, it has been found that increasing age leads to increasing probability of choosing the status quo alternative (von Haefen et al. 2005). This is also in support of the expectation of a positive relationship between age and the probability of protest behaviour. The results in Table 3 are in support of this expectation. Increasing age has a highly significant and positive influence on the likelihood of observing a protest response.

Table 3 Hierarchical logistic mixed effects regression models describing the probability of protesting
Full size table

3.1.2 Gender

Several studies have found gender differences in WTP (e.g. Dupont 2004; Brown and Taylor 2000; Berrens et al. 1997). Considering this, it does not seem farfetched to conjecture that protest behaviour might differ across gender. Mitani and Flores (2007) and Ladenburg and Olsen (2008) find evidence that biases in SP surveys may impact differently across gender. One explanation is that females are comprehensive information processors whereas males are much more selective when processing information (Meyers-Levy 1989). Thus, if male respondents tend not to read the information, they might lack some information, which could help them answer the valuation question. If a respondent has not actually read all the information that is provided, then answering the valuation question may become even more difficult than it inherently already is. Stating a protest bid could be an easy and quick way out of such a (to some extent self-inflicted) difficult valuation question. Hence we might expect that males will exhibit a higher tendency to protest. Turning to the results in Table 3, this expectation is not met. The MALE parameter estimate is not significantly different from zero. Hence, our data does not exhibit any gender differences with regard to protest behaviour when all the other variables in the model are accounted for.

3.1.3 Household Income

Income typically exhibits a positive relationship with stated values in SP surveys (Carson et al. 2001). This is in accordance with economic theory which would also prescribe that low-income groups should have a higher propensity to state a zero WTP bid than high-income groups. We might expect that some of the zero bidders feel a need to justify their zero bids with other reasons than inability to pay, simply because they think that they ought to pay (especially for the non-market goods). Another reason might be that they are affected by some normative social influence in terms of others expecting them to pay (Kelman 1958). If such a compliance conformity effect is indeed present, then zero bidders who would like to follow the social norm even though they have decided to bid zero might feel inclined to state reasons other than simple budget restraints.Footnote 3 Consequently, we might hypothesize that increasing income would reduce the propensity to protest bid. The negative H_INCOME parameter estimate in Table 3 would support this hypothesis, though the estimate is not significantly different from zero at conventional levels of statistical significance. Nevertheless, this could suggests that there is indeed a general tendency for lower income groups to be more likely to exhibit protest behaviour than higher income groups.

3.1.4 Children in the Household

There is some evidence in the literature that the number of children in the household affects the stated values in SP surveys at least when the surveys consider environmental goods (Teal and Loomis 2000). Specifically, it has been found that parents tend to state higher WTP than their childless counterparts (Dupont 2004). Thus, based on arguments similar to those above, we might expect that parents are less likely to state a protest response. However, Table 2 shows the opposite to be the case. Respondents with children in their household exhibit a significantly higher likelihood of protest behaviour than those without children in the household. The odds of protesting are 1.15 times greater when children are in the household. One explanation could be that, on average, in households with children the available income is lower and, thus, it may be viewed as unfair that people are asked to pay for improving environmental quality.

3.1.5 Use of the Good

A positive relationship is typically expected between the use of a good and the stated values for the good (Carson et al. 2001; Bateman et al. 2002). Of course, if considering a good that contains mainly non-use values, e.g. protection of animal species in remote or non-accessible areas, this relationship becomes less relevant. However, for all the surveys in the present meta-study there are clear elements of use values associated with the goods being surveyed. Thus, it has been possible to construct a user-variable (USE) based on survey questions concerning the respondents’ use of the good. According to the typically observed positive relationship between use and WTP, we expect users of a good to exhibit a lower probability of protesting than non-users. The estimates in Table 3 clearly support this.Footnote 4 As is evident from the odds ratio estimate of 0.76, being a user clearly has a negative effect on protesting.

3.2 Survey Specific Variables

3.2.1 Non-market Versus Market Goods

In the valuation literature on market versus non-market goods it is found that the problem of hypothetical bias is less in the market good case (Hanemann 1991). One may hypothesise that the extent to which such bias (as well as other biases) is prevalent depends on context, and may be less of a problem if preferences are better formed. On a rather general level, Hanley et al. (1997) note that respondents often find it relatively difficult to answer WTP questions when they have no prior experience of trading with the good in question. As previously argued, respondents might resort to the heuristic of stating a protest answer in such a situation. Brookshire et al. (1982) have looked at the issue in the context of embedding, and they suggest that in cases where private purchase is conceivable, embedding is less likely to occur because the respondents have experience in determining their WTP through their daily shopping. The variable NON-MARKET captures the effect of whether or not the good being valued is a market or a non-market good. As all the market good surveys in our dataset are using CE, and we have incorporated a number of other variables to account for differences between CE and CVM, the NONMARKET variable essentially measures the difference between market and nonmarket good CE surveys. To that end, the argumentation for our expectation of the sign of this variable follows e.g. List (2003), Cherry et al. (2003) and Carlsson et al. (2012) who find that stated WTPs become more consistent with true preferences as market experience increases through a process of repetition and learning. Day et al. (2012) further underline the importance of knowledge and experience, specifically in relation to learning effects in SP surveys. The fact that learning effects are important for respondents to be able to state their true WTP implies that the initial level of knowledge of, and experience with, the good in question is potentially insufficient for respondents to make fully informed and rational choices in a nonmarket good context. Thus, we might conjecture that the likelihood of observing protest behaviour is affected by the respondent’s initial level of knowledge about the good which can be expected to be lower for nonmarket than market goods. On the basis of this we expect the NONMARKET variable to have a positive effect on the probability of a protest answer. This is confirmed by the model estimates. Non-market goods compared to market goods strongly increase the probability of a protest response. The odds of a protest response are more than five times higher for non-market goods when doing CE surveys.

3.2.2 Choice Experiment Versus Contingent Valuation Method

Only one study to date has explicitly analysed whether the number of respondents who hold protest beliefs or who give protest responses significantly differs between CV and CE (Meyerhoff and Liebe 2008). They find no clear pattern of differences between CE and CV with respect to the rate of protest answers. On this basis, we expect to find no effect on protest behaviour between CE and CV. Other more general statements have been made claiming that CE generates a lower number of protest responses. For example, Mogas et al. (2005) argue that the lower response rate they observed in their CV compared to their CE was due to a ‘protest motive’ in CV. It may be argued that CE reduces strategic behaviour due to the CE method being less ‘transparent’ than the CV method (Hanley et al. 1998a, b). One could hypothesise that the same pattern would exist with respect to protest answers, thus a reduction in protest answers when using CE instead of the CV method would be expected. Additionally, the repeated choice nature of CE might add to this expectation in the sense that the more times respondents are asked to choose, the higher the likelihood that the respondent at some point decides to choose a non-status-quo alternative. This is generally supported by the parameter estimates for the three CV variables. When using the CV method instead of the CE method, the probability of observing a protest answer increases. When interpreting this result and the results below regarding the CV method, one has to be aware that these results only relate to nonmarket goods—since no CV studies for market goods were included.

More interesting is the relative effect between the CV variables, i.e. depending on which CV format is used. In the analysis we distinguish between OE (D_CVM_OPEN), dichotomous choice (D_CVM_DICH) and payment card (D_CVM_PC). It has been argued that using an OE rather than a dichotomous choice WTP question format places a more difficult cognitive burden on respondents (Hanley et al. 1997). In their recommendations regarding the use of CVM, the NOAA panel presents two arguments for using the dichotomous choice approach rather than the open-ended approach (Arrow et al. 1993). First of all, they argue that open-ended question scenario descriptions are less realistic than those of dichotomous choice questions, as people in their everyday life are more used to deciding whether or not to accept a given price for a good than coming up with a figure themselves. As previously argued, a heuristic providing an easy way out of a difficult choice might be to state a protest answer. Based on this, we might expect to find increased probability of protesting in studies using open-ended WTP questions compared to using dichotomous choice formats. Secondly, the NOAA panel argues that open-ended questions are more prone to strategic behaviour than dichotomous choice questions. It is less clear what effect we might expect from the choice of question format on protest behaviour based on this. In their seminal book on the CVM methodology, Mitchell and Carson (1989) argue that open-ended question formats generally obtain higher non-response rates and higher percentage of protest-zeros than closed ended formats. Based on this, it is not surprising that our model returns a significantly positive parameter estimate for D_CVM_OPEN variable which also is significantly larger than the parameter estimate for D_CVM_DICH. This implies that using an OE WTP question format rather than a dichotomous choice format will, ceteris paribus, increase the probability of obtaining protest responses. This finding is also in line with the results presented by Meyerhoff and Liebe (2010) in their meta-analysis.

Finally, with respect to the payment card format, it is often argued that PC and OE formats comprise the most resemblance. Cameron et al. (2002) made a comparison of seven preference elicitation methods, and found that the OE and the PC methods were the least consistent. With respect to biases due to behavioural reasons, Holms and Kramer (1995) found that the PC method was less prone to behaviour such as yea-saying and starting point bias. These findings give rise to a priori expectations with regard to protest behaviour in relation to payment card formats, which goes in both directions. One could argue, that because the PC method has the most resemblance with the OE method, this should also go for the issue of protest behaviour, implying that the PC method should increase the probability of protesting compared to the DC method. However, another interpretation following Holms and Kramer (1995) would point at the PC method decreasing the probability of protesting compared to the DC method. Our results however show that the PC method does not have any significant effect on the probability of protesting in our dataset.

3.2.3 Length of the Scenario Description

As respondents might not have sufficient information about the good in question, particularly when it is a non-market good, a decent amount of information provided in the survey is likely to be welcomed by respondents. Assuming that the length of the scenario description serves as a suitable proxy for the amount of information provided to the respondents, longer descriptions equal more information. Concerning information provision and WTP, studies have shown that an increased amount of information leads to increased WTP (see e.g. Mørkbak and Nordström 2009). Thus, under the assumption of the length of the scenario description being positively correlated with the amount of information provided, our ex ante expectation of the variable NO_WORDS is that it will contribute with a negative effect on the likelihood of a protest response. On the other hand, the length of the scenario description could be argued to affect the cognitive burden for the respondents. Turning to the literature on questionnaire research, meta-analyses suggest that longer mail questionnaires are associated with lower response rates (Heberlein and Baumgartner 1979; Yammarino et al. 1991), whereas no significant effect is found on the response rate of the length of web-based questionnaires (Cook et al. 2000; Sheehan 2001). Based on these observations, one could argue that due to an increased cognitive burden, respondents are more likely to become protesters when they feel too much bothered by the number of questions they are asked. Therefore, as a second measure we also test for an effect of the number of question itemsFootnote 5 requiring an answer before the valuation scenario (NO_ITEMS) is reached. It should thus better capture the burden for the respondent. The estimates in Table 3 show that a longer scenario description indeed decreases the likelihood of a protest response. In contrast, however, the number of requested responses as measured by the number of response items prior to the valuation scenario has no statistically significant influence on the propensity to protest.

3.2.4 Using Tax as a Payment Vehicle

Mitchell and Carson (1989) argue that the choice of payment vehicle should be expected to influence WTP amounts. In the CV literature Kontoleon et al. (2005) found no differences in WTP for genetically modified foods when using tax and fee as payment vehicles respectively, while Hayes et al. (1992) found that the WTP for improved water quality was larger when using tax as payment vehicle as opposed to a fee payment. However, with respect to the response rate, Hayes et al. (1992) found that it decreased with 33 %, when using a fee payment instead of a tax payment. Finally, Daubert and Young (1981) found that using taxes as payment vehicle opposed to using entrance fee increased the number of protest bidders. As this shows, there is no clear evidence of either taxes or fees resulting in increasing or decreasing protest responses, but the only study examining the effect of choice of payment vehicle on the rate of protest responses finds a reduced protest response rate associated with an entrance fee payment vehicle opposed to a tax payment vehicle (Daubert and Young 1981). With this in mind our expectations to the tax payment vehicle variable PV_TAX is that it should have a positive influence on the likelihood of a protest response. Similar to the NONMARKET variable only measuring the difference between market and nonmarket good CE surveys, the PV_TAX variable only measures the impact of using tax as payment vehicle versus using other payment vehicles in nonmarket good surveys—since none of our market good surveys use tax as payment vehicle. The results suggest that our hypothesis regarding the tax payment vehicle leading to increased probability of protesting is not supported. On the contrary, the variable for using tax as a payment vehicle has a negative sign, though it is not statistically significant.

3.2.5 Defining the Opt-out Alternative as the Current Situation

Typically, the choice tasks respondents are faced with in Choice Experiment surveys include a zero-priced opt-out alternative. In CV surveys such alternatives are usually referred to as status quo alternatives or no-purchase alternatives. The inclusion of an opt-out alternative provides realism and ensures that the respondents are not forced to choose. The opt-out alternative can be, and has in empirical surveys been, defined in several different ways, e.g. as an ’actual status quo’, a ’none-of-these’, or another pre-specified alternative. Furthermore, some opt-out alternatives can be considered real while others are defined as hypothetical opt-outs (Whittington and Adamowicz 2011). In a recent study by Campbell et al. (2012) the effect of using three different definitions of such opt-out alternatives in a CE context has been investigated—a current situation status quo (perceived), a hypothetical status quo (provided) and a none-of-these alternative. With respect to the effect of these definitions on protesters’ behaviour Campbell et al. (2012) find that when the opt-out alternative is defined as the current self-perceived situation, the magnitude of protesters in the sample is tripled, compared to a none-of-these definition. Based on this, our a priori expectation with regard to the variable SQ_CURR (the status quo defined as the current situation) is that this will contribute with an increased probability of a protest answer as compared to using other types of status quo definitions. From Table 3 it is evident that our a priori expectation is not confirmed. When the status quo is defined as the current situation, the probability of obtaining a protest answer apparently decreases according to the negative sign of the parameter estimate.

3.2.6 Internet Survey Mode Effects

In a study focusing on survey mode effects, Marta-Pedroso et al. (2007) found no impact of survey mode on protest responses when comparing an internet survey to an in-person survey. However, in a similar comparison, Nielsen (2011) found significantly more protest votes in the Internet sample.Footnote 6 In a similar survey mode comparison, though focusing on Internet and mail surveys, Olsen (2009) found more than a doubling in the share of protesters when moving from an Internet survey to a mail survey. Olsen (2009) suggests that his finding implies that Internet surveys have an advantage over mail surveys in terms of receiving more valid replies, which translates into higher effective response rates, ceteris paribus. However, this might simply reflect a larger degree of self-selection in the Internet sample that could potentially bias results. In that case, it might be argued that the additional self-selection processes in the Internet sample result in respondents being more willing to accept the hypothetical scenario and play along in the valuation exercise. Indeed, the significantly negative impact of the INTERNET variable on the propensity to protest in our model suggests that Internet respondents are more willing to accept the hypothetical scenario than respondents being interviewed personally or by mail. This finding also confirms the results presented by Meyerhoff and Liebe (2010). Whether this is due to self-selection processes that could bias WTP estimates or it merely reflects that Internet panellists are generally more trained in answering questionnaires and they, thus, have better knowledge of their own preferences as well as the structure of questions, is an open question. As noted in Olsen (2009), there is no a priori expectation as to whether this will affect stated preferences or not.

3.3 Random Effects

Looking at the random coefficient standard deviation estimates in Table 3, it is evident that the effects of all the respondent specific characteristics on the propensity to protest vary a great deal across surveys. In particular, it is interesting to note that even though the fixed effects estimates suggest that the mean effect of gender is insignificant, the random coefficient standard deviation estimate for gender reveals that about 1 % of the total variance in our data is actually explained by different impacts of MALE across the 38 samples. Similarly, different impacts of the USE variable across surveys account around 6 % of the total variance. All in all, about 58 % of the total variance in our data is attributable to survey level characteristics and heterogeneous impacts of the respondent characteristics across surveys. The fact that the errors within each sampled survey are obviously correlated underlines the importance of using random effects in the hierarchical model in order to make more accurate inferences about the fixed effects that are our main interest here. Of course, the other side of the coin implies that the remaining 42 % of the observed variance in our data is attributable to respondent-level traits.

3.4 Sensitivity Analysis

In order to investigate how sensitive the findings of model 1 are, the model was re-estimated using a modified sample. We use a 10 % alpha trimming procedure, where we remove the two studies with the highest share of protest respondents and the two studies with the lowest share of protest respondents (model 3). Interestingly, among the respondent specific variables the effects on protesting are rather constant while on the other hand survey characteristics are to a limited extent sensitive to the changes. To begin with the former, the effect of the covariates AGE, CHILD, and USE are statistically significant at the 5 % levels and the sign remains the same across both models. The MALE and H_INCOME variables remain insignificant in both models. Among the survey characteristics the number of words, open-ended CVM, and whether Internet was used are highly significant and show the same influence on protesting in both models. The same goes for the survey characteristics NONMARKET and SQ_CURR. The remaining survey characteristics do not change. So in conclusion the sensitivity analysis shows that neither the surveys with the lowest share of protester nor the surveys with the highest share of protesters are driving the results.

Finally, we investigate whether the parameters affecting the likelihood of a protest response in the entire sample (model 1) are the same when looking only at the potential protesters (model 2). We estimate a model identical to model 1 but this time only for the potential protesters. As can be seen from model 2 results in Table 3, the overall picture remains to a large extent the same as in model 1. Generally, levels of statistical significance drop and a few variables become insignificant. This is not surprising considering the heavy reduction in sample size compared to model 1. Nevertheless, the overall findings in model 2 are quite similar to model 1.

4 Discussion and Conclusion

If the distribution of protest responses is not independent of respondent demographics, the elicitation method, the question format, etc., then simply expelling protesters from surveys will lead to sample selection issues, but also WTP results will not be comparable across surveys (Jorgensen et al. 1999). The present meta-study is the first that tries to explain the determinants of protest responses at both the individual and the survey level simultaneously. So far there is only one other meta-study regarding protest responses (Meyerhoff and Liebe 2010), but this one has solely focused on the influence of survey characteristics on protesting. Our results suggest that some of the differences in WTP typically observed between different demographic groups, different elicitation formats and different question formats might actually be attributed to inherent differences in the propensity to protest. Specifically, our results show that the propensity for respondents to exhibit protest behaviour when asked a stated preference type valuation question depends on a number of factors, individual-specific as well as survey-specific.

Individual-specific factors influencing the propensity to protest are generally beyond the influence of the researcher. However, there are possible countermeasures against such external factors leading to for instance sample selectivity problems when omitting protest responses from analysis. One such is the use of protest reduction entreaties as suggested by, for instance, Bonnichsen and Ladenburg (2009) among others. They find that entreaties clearly instructing respondents that the payments in the SP survey are purely hypothetical are able to reduce the number of protesters significantly. In relation to the individual-specific variables affecting the propensity to protest identified in the present analysis, such entreaties might be targeted directly at respondents with a relatively higher risk of protesting, i.e. older people, low income groups, people with children, and people who cannot be characterised as users of the good. Internet surveys would offer the opportunity to identify these groups of respondents prior to the preference eliciting questions and maybe subject them to an entreaty.

For the survey-specific factors, which are much more controllable to the researcher, we find a number of interesting results which might be used in order to reduce the impact of protest behaviour. First and foremost, our results suggest that if the researcher chooses to employ a CVM using the open-ended question format, the number of protesters is likely to increase compared to using CE. The odds ratio reveals that this decision has a large impact on the probability of obtaining protest responses. Also, to opt for a CVM with a dichotomous choice format increases the probability of a protest response relative to CE. Furthermore, if the good under consideration is a non-marketed good, the probability of protesting increases drastically compared to a market good case. Of course, whether a survey considers a market or a non-market good may be somewhat out of the hands of the researcher. Typically, some external demand for valuation of a specific good is the driver of a survey, and in that case the character of the good is given. The implications are that especially in the non-market good cases researchers should be aware that using CE rather than CVM seems to significantly reduce the number of protest responses.

When it comes to the actual construction and setting up of questionnaires to be used for SP surveys, our results indicate that researchers can reduce the risk of obtaining protest responses in a number of ways. In the description of the hypothetical scenario, it seems that the length of the scenario as measured by the number of words can be considered. Even though the impact is not large, there is a significant tendency that the longer the scenario, the lower the amount of protesters. The obvious interpretation here would be that information is important to respondents. It is a well-known fact that one should generally aim to keep questionnaires as short as possible (Dillman 2007). In case of information overload, some respondents may not read all the information available (Meyers-Levy 1989). Hence, researchers might want to keep their scenario descriptions short. Our results underline that this entails a risk of increasing the number of protesters slightly. However, considering the relatively small impact, this may be an acceptable price to pay in order to ensure that respondents have actually read and assimilated all the available information in the scenario description which might be necessary in order to answer the valuation questions. Finally, the choice of payment vehicle needs to be considered. As the brief literature review shows, there is no clear evidence of either taxes or fees resulting in generally increasing or decreasing protest response rates. Our results supports that this is the case since we find no statistically significant effect of the payment vehicle on protesting. However, it should be noted that the studies used in this analysis do not cover the whole range of available payment vehicles. The meta-study by Meyerhoff and Liebe (2010) found that, compared to taxes, surcharges to a bill, for example a water or energy bill, lead to less protesting while using entrance fees seemed to evoke protest responses.

The results of the present meta-study should be interpreted with some degree of caution. The studies used in the analysis do not represent all stated preference studies. As this kind of meta-analysis requires having the whole sample available, it is difficult to draw a random sample out of all stated preference studies. Among the samples we used more than 70 % have used internet sampling, clearly not representative for the studies conducted so far. Therefore, the results are likely to be influenced by an ‘availability bias’ (see Rosenberger and Johnston 2009) as we have mainly used studies conducted by ourselves or by willing colleagues who had well documented data that they were willing to share. Many of the samples are therefore not completely independent from each other although they have been conducted with respect to different goods and at different points in time. It would thus be beneficial to include further samples done by other researchers but availability and documentation limits this severely. Finally, an important question left unanswered here relates to the definition of protest responses. There is, as already stated, no generally agreed upon protocol describing what constitutes a protest response. The present meta-study exemplifies this as several different approaches to protest response classification have been used in the surveys from which the data used in the analysis originate. It would be beneficial in future studies to examine to what extent the current findings depend on the definition of protesters. However, investigating this question also depends on very well documented data as it would be crucial to know exactly how protest responses have been determined in each of the individual surveys.