Abstract
Recognised as an integral part of the political process, the topic of institutional failure has recently received increased attention in the literature, particularly with respect to policy failure. Nevertheless, the difference between various types and aspects of failure is unclear conceptually, hampering the development of cumulative theory building into its causes and consequences. Furthermore, while ample attention has been paid to negative consequences, insights into the possibly ‘productive functions’ of failure are scattered and largely remain on the fringes of existing research. The present paper offers a systematic review of the failure literature, particularly its definitions, causes and consequences, setting existing research in the different scholarly fields in relation to each other. Special emphasis is placed on the ways failure may serve to advance the effectiveness and efficacy of public policy and the wider political system, opening ‘windows of opportunity’ as leverage points for institutional change. In doing so, we identify a number of factors which may facilitate or hinder the activation of this productive potential on an individual, institutional, and societal level.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Introduction
Failure by political institutions to adequately respond to societal problems is ubiquitous (e.g. McConnell 2015; Richardson 2007; Kirkpatrick 2012; Sparrow 2008). What we term ‘institutional failure’ as an umbrella concept embraces various, partly overlapping realms, e.g. ‘policy failure’, ‘government failure’, or ‘regulatory failure’, and becomes manifest in different ways, e.g. through ‘crises’ (Stern 1997; Alink et al. 2001), ‘breakdowns’ (e.g. Murray and Dollery 2005) ‘fiascos’ (Bovens and Hart 1995), and ‘blunders’ (King and Crewe 2013). Policy failure, in particular, has recently gained increasing attention in the literature, witnessed by no less than three special issues in the past 3 years. Public Policy and Administration investigated the persistence of policy failures (Howlett et al. 2015), the Journal of European Public Policy brought together the separate literatures on public policy and foreign policy fiascos (Oppermann and Spencer 2016a), and most recently, Policy and Politics highlighted how rarely failure is averted or followed by learning (Dunlop 2017b).
Why study failure at all? Given its widespread occurrence, failure is an integral part of the political process. An understanding of public institutions, their genesis and performance would remain incomplete disregarding the multiple aspects of failure. From a problem-solving perspective, proper understanding of processes and causes of failure—i.e. the negative and undesirable aspects of failure—may be necessary in order to avoid it in the future. Therefore, scholars have studied how failure erodes trust and confidence in major social actors and institutions, including governments and intergovernmental organisations (Prakash and Potoski 2016), how established political systems may degenerate (e.g. Farazmand 2012) and with them, their problem-solving and governance capacities (Alink et al. 2001).
However, while ample attention has been paid to such negative consequences, the potentially desirable aspects of failure and decline have received less systematic attention in the failure literature to date. And yet, it is almost a truism that negative episodes, such as crisis and failure, may also provide positive aspects and open ‘windows of opportunity’ for reform and learning (e.g. Stern 1997; Wallis and Dollery 2002) and may thus serve as a ‘leverage point’ for systemic change (Abson et al. 2017). Yet, much of the literature fails to go beyond such platitudes. Not only does this inhibit policy-makers’ ability to learn from past experiences or mistakes, it also critically hampers efforts to systematically assess and better understand in which ways, and under which conditions, failure may lead to subsequent success. Crises, for example, are often followed by a period of reform as old structures make way for something new. Deeper insights into what have been termed the ‘productive functions’ of failure (Newig 2013) may thus prove instrumental in coping with the consequences of failure, for deriving adequate and intentional responses, and for supporting the resilience of the wider institutional system.
It is against this backdrop that this article reviews the available literature on institutional failure with a view to its potential productive functions. We put forward no original argument. We rather attempt a first organised stock-taking of claims and insights about productive consequences of failure by reviewing 111 recent articles on institutional failure. We seek to highlight the ways in which productive functions of failure can unfold and intentionally be approached to advance the effectiveness of public policy and the wider political system. A closer look at the factors enabling and preventing the activation of failure’s productive potential will provide further insights into how and when institutional failure may serve as a leverage point towards achieving collective purposes. As a first systematic attempt in this direction, the paper thus aims to strengthen the conceptual basis for future studies on the productive functions of failure.
The paper is organised as follows. The "Methods" section briefly explains the research methodology. The "Concepts of failure" and "Causes of failure" sections lay the foundation to study the productive (and unproductive) consequences of failure. We begin by identifying the different topics and aspects commonly associated with the concept of institutional failure and map out the conceptual ambiguities surrounding the terms. This exercise aims to avoid the “conceptual confusion” over the different aspects and types of failure, which so far has prevented the development of “cumulative theory building into the causes and consequences of policy success and failure” (Howlett 2012, 545). As governance systems are complex and multiple interactions exist between structural institutional arrangements and particular (sets of) policies (see e.g. May and Jochim 2013), we consider the full spectrum of institutional failure. This approach acknowledges that public problem solving is multi-dimensional, covering the political dimensions of policy, polity and politics, as well as multiple systemic levels, from the individual (micro) to the societal (macro) level. Both of these analytical categories—political dimensions and societal levels—will serve to guide the review and analysis in the subsequent sections and ultimately helps to understand and structure our ‘findings’ on the productive potentials of institutional failure and decline. We explore the different definitions and causes found in each of the different political dimensions. Building on this, "Consequences of failure—exploring its productive potential" section focuses on the consequences of failure. While not sparing the ‘negative’ consequences, the main part reviews the potentially productive functions of failure, as well as the factors conditioning their activation. The final section draws out the most important findings and implications of our review.
Methods
This review was conducted following the PRISMA guidelines (Moher et al. 2009), a set of items used for reporting in systematic reviews. We have closely followed the PRISMA checklist and documented our steps (see also Online Appendix) throughout all stages of this research. In the following section, we provide an in-depth description of our approach.
In order to identify the relevant literature for our review, we searched the Scopus database using the search strings specified in Online Appendix 1. Despite the limitations inherent to a focus on published academic literature, we believe this approach is justified for two main reasons: Firstly, as we aim for the consolidation of the present academic body of research, we targeted peer-reviewed publications. Second, this strict approach strengthens the transparency and replicability of our review.
Search strings included and combined terms related to various forms of institutional failure (e.g. government failure, policy failure, bureaucratic failure) and decline, collapse and deinstitutionalisation. By formulating a search string with a relatively high sensitivity and low specificity, we deliberately ‘cast a wide net’ to capture as many relevant papers as possible. The search was limited to publications in English, including all results up to April 2017.
Our search yielded a total of 8388 results. To filter only the most relevant records, we established a number of exclusion criteria developed alongside the research question. The criteria listed in Fig. 1 were used to assess the publications’ relevance based on their title, abstract and keywords in a first-stage screening, excluding those that failed to meet any one of these criteria. The remaining articles were then assessed for eligibility in a second-stage screening of the full text using the same exclusion criteria. This resulted in a final selection of 111 publications to be included in our systematic review (Online Appendix 2). Figure 2 gives a breakdown of the number of publications per year and an overall trendline.
Both the first-stage and second-stage screenings were undertaken by the first author. To minimise errors and personal bias, extensive and rigorous trial-screening was undertaken by all of the authors at every stage until agreement on the correct interpretation of the exclusion criteria was high and all reviewers arrived at a common understanding for screening.
The assessment of the included publications was undertaken through a catalogue of questions and variables operationalising our research questions. Following a comprehensive pretest for specification and calibration of the variables, following the same procedures of trial and calibration as during screening, coding of individual papers was undertaken by single reviewers. The coding scheme (Online Appendix 3) applied to the selected publications largely consists of open-ended questions. Relevant text fragments were identified through a qualitative content analysis of publications and coded according to the relevant variables, enabling the subsequent analysis presented in this paper.
Concepts of failure
Failure and crisis are not singular, clear-cut concepts, but instead comprise of a wide variety of different and at times closely connected types. An overview of definitions can be found in Online Appendix 4. Despite this review’s focus on conceptual contributions, we found that the majority of included publications do not offer an explicit definition of the concept discussed. Nevertheless, there appears to be some consensus on which aspects the different concepts of failure entail. Overall, definitions can be distinguished regarding the domain of the political system they address. First, there are concepts of failure with policy as the centre of focus (e.g. policy failure, policy fiasco), discussing the failure of specific policies, or their policy-making process. Second, there are those which focus on the structures, particularly on the government apparatus, or the polity-level (e.g. institutional failure, governance failure), discussing the inability of the institutional system to deliver. Finally, several definitions address the failures of the political process, focusing on the politics dimension and the qualities and legitimacy of processes. While these domains may be analytically separated, several definitions address multiple domains of the political system and, hence, stress the interlinkages between these, highlighting, e.g. how political struggles influence particular policy-making and failure in one dimension may not lead to failure in another (see e.g. McConnell 2015).
Failure in the policy realm is often defined as the inability to achieve goals or targets formulated in a particular policy or series of policies (e.g. McConnell 2010, 2015), or the failure to do so in a cost-effective (Murray and Dollery 2005) or coherent manner. A number of definitions of policy failure (Hall 2011; Hajnal 2010) and government failure (e.g. Keech and Munger 2015; Harris 2007) focus exclusively on substantial or content-related criteria or benchmarks for success. Commonly used indicators to measure performance in this context include the physical measurement of effectiveness, e.g. in terms of water quality (Juhasz 1989), commonly comparing outcomes to the initial goals. Discussions of government failure, in particular, strongly focus on content-related benchmarks, commonly assessing the ability to achieve intended outcomes in relation to indicators such as the efficient allocation of resources or optimal social welfare through traditional cost–benefit analyses and economic modelling.
Going beyond individual (policy) failures, several definitions consider the structural dimension upon which failures build up, thus focusing on the polity-, or structural level. Fike and Gwartney (2015), for example, refer to the importance of the ‘structure of incentives’ leading to the counter-productive use of resources. Some of the definitions of governance failure (e.g. van der Steen et al. 2015) similarly focus on the inability on a system level to deliver the goals of a policy programme. Definitions of crises, in particular, concentrate exclusively on shocks to the institutional structure (Alink et al. 2001) and wider sociopolitical order (Boin et al. 2008).
Lastly, focusing on the aspects of politics, success or failure is often considered in relation to the legitimacy of policies and the wider political system. This legitimacy can be assessed in different ways; indicators here relate to the representation of different interests (e.g. Farazmand 2012), the use of correct and legal procedures or accountability (Benitez et al. 2012; Dolfsma 2011), distributional equity (Acheson 2006; Cullis et al. 1993), and the existence of political and/or public support (Howlett 2012; McConnell 2010). A strong focus on accountability and legitimacy is found in discussions of governance failure, in which goals are commonly modified through processes of negotiation and reflection. Without predetermined reference points or criteria by which to judge, the inability to self-organise or formulate shared objectives is therefore judged a failure (Peters 2015; Walker 2014).
In these attempts to define failure, two ‘lenses’ (Howlett 2012) or ‘counter-tendencies’ (McConnell 2015) can be distinguished within the literature. The first is a constructivist lens, in which failure is construed through argumentative processes (e.g. Kay and Boxall 2015; Zittoun 2015) or rhetorical and political ‘framing contests’ (Bovens and Hart 2016), rather than considered to exist in its own right (Dunlop 2017b). The production of a dominant interpretation of failure reflects underlying power relations in the political arena or society (Marsh and McConnell 2010; Oppermann and Spencer 2016b), as a “socially and politically significant group of people” (McConnell et al. 2008) needs to perceive something to be a failure for it to considered one.
In contrast, the rationalist, scientific lens (McConnell 2015) aims to draw a clear line between evidence and evaluation and has sought to develop specific and objective indicators to measure clearly defined goals, thus allowing an assessment to be made of whether something is functional or failing (Kotrusová and Výborná 2015). This lens includes a ‘technical’ approach to failure that sees (policy) success and failure as purely technical issues that can easily be solved (Howlett et al. 2015), and a ‘politico-administrative’ approach focusing on highly complex procedural and political aspects that are difficult to analyse (Howlett et al. 2015; Dunlop 2017b). While some definitions may be firmly grounded in one of these lenses, many contain elements of both.
Causes of failure
In order to learn from failure or positively use it as a leverage point for systemic change, it is important to first understand what causes it. We find a wide array of causes throughout the selected publications, nearly all of which focus on the underlying roots of failure, rather than mere triggers. Triggers, in this respect, are considered as short-term events that start something which is already primed to happen, while causes are understood to be the deeper reasons for an action or event to occur in the first place. We distinguish between three levels of causes found in the literature (see also McConnell et al. 2008; Dunlop 2017b): individual (micro), institutional (meso) and societal (macro-) level causes. On an epistemological level, the publications analysing causes of failure tend to adhere to a rational-scientific as opposed to a constructivist view on failure.
Levels of causality
Micro-level causes of failure occur on the level of individual actors, mostly on the part of a policy-maker or administrator. Despite being only little discussed in the reviewed publications, a number of sources of failure can be found on the individual level, particularly in relation to government failure and policy failure. An important distinction is made here between ‘intentional’ failure, as the result of intentional behaviour on the one hand, and ‘unintentional’ or accidental failure on the other hand. Commonly mentioned causes of intentional individual-level failure are self-interested behaviour of actors in the political system (e.g. Kortt and Dollery 2012; Harris 2007) and corruption (Backhouse and Medema 2012), with policy-makers pursuing private goals over societal interest. Unintentional sources of failure are largely the result of either cognitive and behavioural limitations (Hajnal 2010) and include a lack of competence or expertise of individual bureaucrats (Short 2013; Wibe 1992), human errors by individual decision-makers (Bovens and Hart 1995; McConnell 2016), individual personality traits such as over-confidence (Brummer 2016) and a lack of prudence or foresight (Howlett 2014).
Meso-level causes of failure are found on an institutional level and include intra-organisational factors. The literature on government failure, in particular, strongly addresses structural sources of failure: One of the most frequently cited causes of government failure is the lack of incentive for the efficient use of organisational resources (e.g. Vining and Weimer 1999), resulting in low morale and poor management/performance (Meier and Bohte 2003; Andrew 2008), aggravated further where promotion is based on longevity rather than effectiveness (Young 1977; Backhouse and Medema 2012). Another important source of failure is the existence of information asymmetries between the public and private sector (Acheson 2006; Helm 2010). Here, governments are perceived to inherently have less information and knowledge at hand than the private parties they are tasked with regulating. Other important operational sources of failure include a lack of (financial) resources, skills or capacity to design and enforce policies (Short 2013; Vining and Weimer 1999), weak or absent checks and balances (Benitez et al. 2012), and deficient feedback mechanisms (Mitchell and Simmons 1995), with political processes failing to provide the information required to reach socially optimal policies (Cullis et al. 1993). While the policy failure literature similarly traces meso-level causes back to these institutional design and capacity issues, it often focuses specifically on sources of failure in the implementation stage (Vince 2015), such as the use of incomplete or out-of-date policy information (Wibe 1992; Schuck 2014), inadequate planning (Kotrusová and Výborná 2015), poor choice of regulatory instruments (Hansen 1983), and the lack of diversity and dissent in policy-making processes (Bovens and Hart 2016). Discussions of governance failure, finally, focus strongly on procedural sources of failure, considering the failure of parties to coordinate and cooperate on an personal, organisational, and systemic level as an important source of meso-level failure (Jessop 1998; Peters 2015). Where institutions are insufficiently flexible to deal with new challenges or changing circumstances, they will be susceptible to rejection or replacement (Prakash and Potoski 2016; Mol 2009).
Finally, macro-level causes of failure are found in the wider societal, economic, and natural environment. While these causal factors are found outside the political system, there is an important interaction between system actors and their environment. One important external source of failure is the power of private sector, interest groups, and the public, whose opposition can result in a policy or other measure being unsuccessful (e.g. Acheson 2006; Gibb 2015). In those situations where there is a mismatch between the underlying principles of a policy and those held by the community, this can be an important source of failure (Leong and House 2012). The crisis-related literature most strongly considers macro-level causes, focusing on society-wide crises and external shocks. Such crises are often economic or political in nature, but can similarly be caused by natural disasters, infrastructure breakdown or (industrial) accidents (Boin et al. 2008), often involving a large degree of social unrest or threats to national security (Grossman 2015).
Establishing causality
Locating the actual causes of failure is subject to debate. Recent research (e.g. Howlett et al. 2015) argues that the persistence of policy failure suggests its causes lie beyond mere technical causes which would be relatively easy to correct. Particularly in the case of so-called wicked problems—complex problems with multiple causes lacking clear solutions—policy-makers may tend to focus on the symptoms of a problem, thereby remaining on a technical or operational level, rather than address its underlying social causes (e.g. McConnell 2010), which would require also a broader time perspective (Brändström and Kuipers 2003). While technical, short-term solutions may allow an issue to move down the policy agenda and demonstrate policy-makers’ ability and commitment to deal with a problem, it does little to address its root causes (McConnell 2015).
Beyond the focus on single causes, several contributors stress that there may be multiple, complex, and interacting factors which together result in failure (McConnell et al. 2008; Bressers et al. 2013; Mol 2009). In this line, McConnell (2016) emphasises that to argue that “one factor alone is the cause of a failure would be to neglect the range of individual, institutional, and societal factors that interacted to produce that failure—as well as their complex interdependencies” (675).
Finally, it is argued that, like measuring failure, establishing causality is not a purely rational-scientific or value-neutral exercise, with the nature and significance of causal factors considered to be contestable and dependent on one’s wider perspective on society (McConnell et al. 2008). Without a scientifically objective standard by which to judge, under constructivist approaches, success or failure is constructed socially through “informed debates” (Ugyel and O’Flynn 2017) among competing interests. Under such constructivist approaches, technocratic evidence-based expert enquiries into the causes of failure are often set against public enquiries with a stronger focus on public perceptions of performance (Kay and Boxall 2015).
Consequences of failure—exploring its productive potential
Based on this informed understanding of how different concepts of failure fit together, and what causes failure, we now look into to its potential consequences. Most obviously, and following conventional wisdom, failure is clearly negatively connoted. It is therefore not surprising that the literature predominantly describes socially undesirable consequences. Some works, however, point to potentially productive functions of failure. Below, we first briefly discuss the negative consequences of failure, before turning our attention a discussion of the productive potential of failure and the factors enabling or constraining its activation.
Negative consequences
Negative consequences of failure are well established in the literature. Arguably, the hardest consequences will be felt by the addressees of failing institutions and policies, which may lead to damage to their property or their social or material well-being (van der Steen et al. 2015; Ber 2013), particularly for those most dependent on the state (Gibb 2015; Schuck 2014). Failure can also have serious career implications for policy-makers and individuals within the political or administrative system (Griffin 1987; McConnell 2016; Howlett et al. 2015), whether as the result of fair and balanced evaluation or of blame games and the search for individual culprits (e.g. Bovens and Hart 2016).
These individual effects are often interrelated to negative developments that threaten well-being on the wider societal and economic level (Gibb 2015; Schuck 2014). Where individual actors pursue self-interested behaviour, such as (political) rent-seeking and outright corruption, this is often in direct contrast to wider societal interests, causing detrimental effects on a system-wide level. System-wide effects may become manifest in, e.g. the inefficient allocation and use of resources (e.g. Juhasz 1989; Wallis and Dollery 2002), the distortion of market prices, and increased levels of social cost to the economy (Venkatachalam 2004), potentially also resulting in the depletion of scarce resources (Acheson 2006) and environmental degradation (Juhasz 1989).
In the political system itself, failure and crisis often generate a sense of urgency, threat and uncertainty, thus revealing vulnerabilities in the system (Boin et al. 2008). If sufficiently strong, such events can severely undermine trust in the system (e.g. Saurugger and Terpan 2016; Farazmand 2012) and reduce electoral support (Howlett 2012; Schuck 2014), particularly as challenges become more frequent. Ultimately, fundamental threats to its core values can result in the destabilisation of the system as a whole (Saurugger and Terpan 2016; Hansen 1983), undermining the position of regime actors and encouraging the growth/success of challengers (May 2015; McConnell 2010).
Productive potential
Around half of publications in our sample discuss the consequences of failure, with only around a quarter addressing the ways in which these are associated with desirable implications of failure. Most publications remain relatively generic in their findings, simply arguing that failure can provide a window of opportunity for reform (e.g. Farazmand 2012) or learning (e.g. Gibb 2015), or may spur positive change (Marsh and McConnell 2010). While largely implicit, many of these publications hint at deeper-lying assumptions and understandings of these productive functions of failure. The "Productive potential" and "Factors conditioning the activation of productive potential" sections tease out this implicit understanding of productive potential on a policy, polity and political level in greater detail, as well as the conditions enabling or hindering its activation.
While the destabilisation of the political system was previously perceived as a major detrimental effect of failure, some contributors also stress its productive potential to open up windows of opportunity (e.g. Saurugger and Terpan 2016; Schwartz and McConnell 2009). As explicated in theories of the policy process, e.g. the Multiple Streams Framework (Kingdon 1984), events such as failure and crisis punctuate the normal rhythm of policy-making (Schwartz and McConnell 2009). Failure may stimulate political momentum, helping to unfreeze institutional rigidity and inertia (Alink et al. 2001). In some cases, the complete dismantling or removal of institutions may even be required to unlock existing pathways and allow the creation of new ones (Giest 2017), an argument closely linked to Schumpeter’s (1994) ‘creative destruction’. Hence, these windows of opportunity present potential turning points for reform, institutional change (e.g. Kay and Boxall 2015; Grossman 2015), and learning (Hall 2011; Gibb 2015). Synthesising these diverse aspects, a deeper analysis of the collected literature reveals that the ‘window of opportunity’ takes on a distinct character in the different dimensions of the political system—policy, polity, and politics (see Table 1 for an overview), and with it alternative strategies for reform, learning, and lesson drawing are emphasised.
In the policy domain, given its focus on the policy tools and targets, failure may result in the fine-tuning of individual policies or instruments but they may also enable extensive reform of (individual) policies and lead to changes in the policy-making process. However, as the wider system remains essentially stable, reform and learning will most likely not take place beyond the relatively ‘shallow’ level. In this sense, the literature extensively refers to May’s (1992) ‘instrumental learning’.Footnote 1 Instrumental learning largely focuses on modifying policy tools or instruments to achieve programmatic roles. Thus, while it may lead to the adoption of new policy instruments and techniques, overarching policy goals remain unchanged.
In the polity domain, the consequences of failure may reach beyond the policy field and also trigger structural, institutional change in the political system, altering the systemic incentives and structures that led to institutional failure. Individual or low-level policy failures alone are unlikely to trigger change at this level. Rather, repeated failure and the erosion or even disappearance of institutions are required for system-wide changes to be able to take place (Kay 2017). While this inevitably comes with negative consequences for some, it also offers the opportunity to break with older patterns of inertia and path-dependency (Ball 2005; Boin et al. 2008), or for “rebirth and rethinking” (Dunlop 2017b, 9). Pressures that may potentially be able to stimulate a break with established patterns include: political pressures (e.g. a change in the way power is distributed), functional pressures (i.e. changing views about the instrumental value of institutionalised practices and procedures), and social pressures, or the loss of cultural consensus (Ball 2005). Hence, failure in this respect is likely to be accompanied by deeper ‘social learning’ (May 1992) and reforms, involving changes in underlying policy beliefs and paradigms. There is little indication in the literature, however, that processes of destabilisation and reinstitutionalisation are intentionally used within a public policy framework, nor does it become clear whether or how these processes can be steered consciously with a particular alternative in mind.
In the realm of politics, windows of opportunity may involve the break-up of incumbent power structures and discourses. This may allow actors to form alternative coalitions or policy entrepreneurs to advance major, previously condemned changes (Grossman 2015; Saurugger and Terpan 2016; Walsh 2006). By putting the blame on one set of political actors—e.g. the government currently in power, or the proponents of a policy—opposing actors can seek a break with the status quo and push forward their preferred solutions. This strive for reformism may, however, face strong opposition from incumbent elites, who could use the failure as an opportunity to demonstrate the relevance and fundamental stability of the system, thus legitimising its very existence (Van Assche et al. 2012). Learning on this level, referred to as ‘political learning’ (May 1992), is thus concerned with advocating for preferred policy ideas and to enhance their political feasibility (Hall 2011).
As these potentials for change in all three domains highlight, learning may play a pivotal role for harnessing this change. Learning is defined in the literature as “improved understanding” (Newman and Bird 2017) or “the updating of beliefs” through own or witnessed experience, analysis or interaction (Dunlop 2017b). But sources of learning are not limited to own experiences: failure may also stimulate learning and reform with those who have not been direct subject to it. Referred to as ‘policy transfer’ in the literature (e.g. Giest 2017) experiences in other political settings in the past or present enable learning without the costs of negative experiences or an immediate need for action (Stern 1997; Marsh and Sharman 2009). In doing so, it is important to keep in mind the high context-specificity of policy environments, which can in some cases cause perceived lessons to be almost entirely opposed to another (Peters et al. 2011). Learning, especially in contexts of failure, also includes the ‘unlearning’ of certain routines to ensure openness to change, and ‘negative lesson drawing’, or learning what not to do (Stone 2017; c.f. Rose 1991).
Factors conditioning the activation of productive potential
Given the potentially productive functions of failure, the question remains how and under which conditions these can be effectively realised. Overall, the discussion of how to overcome negative events in the selected publications is relatively light-touch and little specific. While the literature frequently refers to the different ways in which failure can lead to productive consequences, it risks becoming a hollow statement if not sufficiently elaborated. The pertinent question is therefore which factors facilitate or hinder turning a negative consequence into a positive one, or how, and under which conditions the positive potentials of failure can be activated and reaped. While some of these factors are internal to the political system (e.g. institutional structures and actors, policy precedents, and information), others are external (e.g. economic markets, international politics) and thus more difficult or impossible to control. A comprehensive list of the enabling/hindering factors discussed in the literature is compiled in Table 2.
On the individual (micro-) level, ideological constraints and mental traits such as defensiveness and risk-averseness, as well as the tendency to reject negative lessons play an important role in actors’ ability to turn experiences of failure into something more conducive. Ideological and psychological openness may enable individuals to activate the positive potential of failure. Dunlop (2017a), for example, refers to the ‘absorptive capacity’ of policy-makers’, the ability to distil key information when gathering information, whilst at the same time staying open to significant criticisms and assessing the appropriate speed and intensity with which to react.
Ample consideration is given to factors on the institutional (meso-) level. Here, factors such as the deep institutionalisation or lock-in of rules and practices, the lack of institutional capacity, communicative, and analytical capacity are considered to be important factors hindering or preventing the activation of productive potential. On a political level, ‘political realities’ such as power imbalances between actors and blame-avoidance strategies also appear to prevent negative consequences from becoming productive, with short-term action standing in the way of viable long-term solutions.
Consequences on the micro-level, such as the resignation or demotion of responsible officeholders (or scapegoats) may “serve as pressure valves that stymie attacks from opponents” (Hinterleitner 2018), preventing further action on a system level. On the other hand, enabling factors such as the presence of high levels of interpersonal trust, channels of effective communication and coordination, and institutional memory are considered to be conducive to activating this potential. Regardless of all these factors, good timing appears to be crucial if one is to exploit an available window of opportunity (Alink et al. 2001).
Finally, on a societal (macro-) level, uncertainty surrounding available policy options and their associated outcomes may be a significant barrier preventing factors, while a political climate conducive to policy change is considered to be an important enabling factor. How these factors “work” and how they might interact in practice remains unclear from the literature, offering interesting potential for future research.
Discussion and conclusions
Reviewing the literature in a systematic fashion revealed that while the negative consequences of failure and decline are discussed more widely in the literature, to date relatively little attention has been paid to the potentially productive consequences of failure. While failure plays an implicit role in many public policy frameworks (e.g. in punctuated equilibrium-theory), only scant attention is paid to a more explicit treatment of its productive consequences. Moreover, those publications that do discuss pathways of how failure may bring about positive change have largely remained generic and descriptive in their explanations, merely claiming that they may open a window of opportunity for reform and/or learning. It is our belief that such a coarse and simplistic view of failure inhibits meaningful change on the three dimensions of the political process. A more conscious appraisal of failure and a better understanding of its productive functions are thus required to further enrich studies of public policy and contribute to the advancement of some of the present frameworks.
The work presented here should be seen as a first step in this direction. We highlighted how failure can open up ‘windows of opportunity’ for reform, institutional change, and learning on different levels of the political system. By looking beyond the individual types in the literature, hitherto mostly treated separately, we have addressed different domains of the political system, notably the policy-level, the polity-level, and the politics-level. While failure of individual policies will mostly result in instrumental learning and the fine-tuning of policy instruments and techniques, on the polity and politics-level, the potentially productive effects of failure are much larger. Here, the destabilisation or removal of existing institutions, as well as deeper forms of (social and political) learning may provide important opportunities to overcome institutional lock-ins and break up dominant power structures. While this literature thus clearly recognises the potential ways in which failure may lead to subsequent success, it presently fails to consider how such processes can consciously be initiated and directed so as to improve existing weaknesses or flaws in the system.
As a first step in this direction, we identified a wide range of factors that may enable or hinder the activation of the productive functions of failure, particularly deep-level structural changes. These include individual traits and constraints of decision-makers, institutional capacities, power relations, as well as the overall complexity of the wider system and tie in closely with the causes of failure, which can similarly be found on the micro, meso, and macro level. Particularly on the meso level, for which the most information is available, the identification of these enabling and hindering factors offers a great potential to help prevent, or better utilise, structural-level failures should they occur.
So how to move forward from here? More attention, through empirical and conceptual studies, will be needed to improve our understanding on how and through what mechanisms failure is likely to unfold its positive aspects or may even be activated intentionally and subsequently steered or guided in the desired direction. On a conceptual level, to further map and explore the positive potentials of failure, research should identify archetypical mechanisms of productive functions going beyond the windows of opportunity and reform, identified in our review. Special attention may be paid to the role of agency and ways to harness productive potentials. On a more empirical level, looking at cases of failure through the lens of enabling and hindering factors may enable us to better determine the relevance of individual factors and provide policy-makers with valuable clues as to where their intervention in the system may be required. A promising strategy forwards also lies in exploring the links between the concepts and productive functions of failure and established political theories and concepts of institutional change. Theories, such as the Social Construction Framework (Schneider et al. 2014), or the Narrative Policy Framework (Crow et al. 2018) may offer insights into the causes and construction of failure and the roles and strategies of actors to reap the productive potentials. Especially the tension between blame avoidance and learning and reform (Howlett 2012), as a major hurdle for the activation of productive potentials of failure on an individual and institutional level, may benefit from more attention and cross-fertilisation. Such an effort would be conducive to better anchoring the subject in broader policy and failure research and contribute towards fixing the “broken link” (Dunlop 2017b) between failure and learning.
Empirically, a next step may be to set the conceptual literature into dialogue with empirical findings; attention should be devoted to identifying relevant cases in which failure has been productive in practice, and where mechanisms identified here, such as negative lesson drawing, could be observed. Such an endeavour would help answer the question how and when failure can be put to use, and which actors play an important role, a question that cannot be answered from the conceptual literature alone. However, the empirical study of failure is facing particular challenges: issues of desirability bias and limited access to detailed information may prevail, especially in situations where actors report on their own experiences. Empirical research on failure and its productive functions may benefit further from innovative approaches for research design and methodology.
Finally, we strongly encourage more cross-disciplinary fertilisation to better understand how different fields deal with failure. The organisational literature, in particular, with obvious interfaces to the institutional literature, offers a strong potential to contribute to this debate (e.g. Mone et al. 1998; Frese and Keith 2015). Combined, these different research efforts may ultimately prove instrumental in coping with the real-life consequences of failure and developing adequate responses in practice.
Notes
Similar typologies of learning appear throughout the literature, all based on the depth of learning (see Dunlop 2017b).
References
Abson, D. J., Fischer, J., Leventon, J., Newig, J., Schomerus, T., Vilsmaier, U., et al. (2017). Leverage points for sustainability transformation. Ambio, 46(1), 30–39.
Acheson, J. M. (2006). Institutional failure in resource management. Annual Review of Anthropology, 35, 117–134.
Alink, F., Boin, A., & Hart, P. T. (2001). Institutional crises and reforms in policy sectors: The case of asylum policy in Europe. Journal of European Public Policy, 8(2), 286–306.
Andrew, B. (2008). Market failure, government failure and externalities in climate change mitigation: The case for a carbon tax. Public Administration and Development, 28(5), 393–401.
Backhouse, R. E., & Medema, S. G. (2012). Economists and the analysis of government failure: Fallacies in the Chicago and Virginia interpretations of Cambridge welfare economics. Cambridge Journal of Economics, 36(4), 981–994.
Ball, A. (2005). Environmental accounting and change in UK local government. Accounting, Auditing and Accountability Journal, 18(3), 346–373.
Benitez, D., Estache, A., & Søreide, T. (2012). Infrastructure policy and governance failures. CMI Working Paper (Vol. 5). Bergen: Chr. Michelsen Institute (CMI).
Ber, I. (2013). Governmental strategies towards poorly-performing municipalities: From narrow perceptions to ineffective policies. Lex Localis—Journal of Local Self-Government, 11(1), 33–52.
Boin, A., McConnell, A., & Hart, P. T. (2008). Governing after crisis. The politics of investigation, accountability and learning. Cambridge: Cambridge University Press.
Bovens, M., & Hart, P. T. (1995). Frame multiplicity and policy fiascoes: Limits to explanation. Knowledge and Policy, 8(4), 61–82.
Bovens, M., & Hart, P. T. (2016). Revisiting the study of policy failures. Journal of European Public Policy, 23(5), 653–666.
Brändström, A., & Kuipers, S. (2003). From ‘normal incidents’ to political crises: Understanding the selective politicization of policy failures. Government and Opposition, 38(3), 279–305.
Bressers, N., van Twist, M., & ten Heuvelhof, E. (2013). Exploring the temporal dimension in policy evaluation studies. Policy Sciences, 46(1), 23–37.
Brummer, K. (2016). ‘Fiasco prime ministers’: Leaders’ beliefs and personality traits as possible causes for policy fiascos. Journal of European Public Policy, 23(5), 702–717.
Crow, D. A., Albright, E. A., Ely, T., Koebele, E., & Lawhon, L. (2018). Do disasters lead to learning? Financial policy change in local government. Review of Policy Research, 35, 1–26.
Cullis, J. G., Jones, P. R., & Morrissey, O. (1993). The charge of the tax brigade. A case study of government failure and tax reforms. European Journal of Political Economy, 9(3), 407–425.
Dolfsma, W. (2011). Government failure—Four types. Journal of Economic Issues, 45(3), 593–604.
Dunlop, C. A. (2017a). Pathologies of policy learning: What are they and how do they contribute to policy failure? Policy and Politics, 45(1), 19–37.
Dunlop, C. A. (2017b). Policy learning and policy failure: Definitions, dimensions and intersections. Policy and Politics, 45(1), 3–18.
Farazmand, A. (2012). The future of public administration: Challenges and opportunities—A critical perspective. Administration and Society, 44(4), 487–517.
Fike, R., & Gwartney, J. (2015). Public choice, market failure, and government failure in principles textbooks. Journal of Economic Education, 46(2), 207–218.
Frese, M., & Keith, N. (2015). Action errors, error management, and learning in organizations. Annual Review of Psychology, 66, 661–687.
Gibb, K. (2015). The multiple policy failures of the UK bedroom tax. International Journal of Housing Policy, 15(2), 148–166.
Giest, S. (2017). Overcoming the failure of ‘silicon somewheres’: Learning in policy transfer processes. Policy and Politics, 45(1), 39–54.
Griffin, D. M. (1987). Implementation failure caused by institutional problems. Mountain Research and Development, 7(3), 250–253.
Grossman, P. Z. (2015). Energy shocks, crises and the policy process: A review of theory and application. Energy Policy, 77(1), 56–69.
Hajnal, G. (2010). Failing policies or failing politicians? Policy failures in Hungary. World Political Science Review. https://doi.org/10.2202/1935-6226.1089
Hall, C. M. (2011). Policy learning and policy failure in sustainable tourism governance: From first- and second-order to third-order change? Journal of Sustainable Tourism, 19(4–5), 649–671.
Hansen, S. B. (1983). Public policy analysis: Some recent developments and current problems. Policy Studies Journal, 12(1), 14–42.
Harris, E. (2007). Historical regulation of Victoria’s water sector: A case of government failure? Australian Journal of Agricultural and Resource Economics, 51(3), 343–352.
Helm, D. (2010). Government failure, rent-seeking, and capture: The design of climate change policy. Oxford Review of Economic Policy, 26(2), 182–196.
Hinterleitner, M. (2018). Policy failures, blame games and changes to policy practice. Journal of Public Policy, 38(2), 221–242.
Howlett, M. (2012). The lessons of failure: Learning and blame avoidance in public policy-making. International Political Science Review, 33(5), 539–555.
Howlett, M. (2014). Why are policy innovations rare and so often negative? Blame avoidance and problem denial in climate change policy-making. Global Environmental Change, 29, 395–403.
Howlett, M., & Ramesh, M. (2014). The two orders of governance failure: Design mismatches and policy capacity issues in modern governance. Policy and Society, 33(4), 317–327.
Howlett, M., Ramesh, M., & Wu, X. (2015). Understanding the persistence of policy failures: The role of politics, governance and uncertainty. Public Policy and Administration, 30(3–4), 209–220.
Hulme, R., & Hulme, M. (2012). Policy learning? Crisis, evidence and reinvention in the making of public policy. Policy and Politics, 40(4), 473–489.
Jessop, B. (1998). The rise of governance and the risks of failure: The case of economic development. International Social Science Journal, 50(155), 29–45.
Juhasz, F. (1989). Water: Is there a crisis? OECD Observer, 160, 4–9.
Kay, A. (2017). Policy failures, policy learning and institutional change: The case of Australian health insurance policy change. Policy and Politics, 45(1), 87–101.
Kay, A., & Boxall, A.-M. (2015). Success and failure in public policy: Twin imposters or avenues for reform? Selected evidence from 40 years of health-care reform in Australia. Australian Journal of Public Administration, 74(1), 33–41.
Keech, W. R., & Munger, M. C. (2015). The anatomy of government failure. Public Choice, 164(1–2), 1–42.
King, A., & Crewe, I. (2013). The lunders of our governments. London: Oneworld.
Kingdon, J. W. (1984). Agendas, alternatives, and public policies. Harlow: Pearson.
Kirkpatrick, D. D. (2012). As Clashes Continue, Egypt Soccer Riot Becomes Metaphor for Government Failure. (2012, 03.02.2012). The New York Times.
Kortt, M. A., & Dollery, B. E. (2012). Australian government failure and the Green Loans Program. International Journal of Public Administration, 35(2), 150–158.
Kotrusová, M., & Výborná, K. (2015). A policy fiasco: The institutional (non-)reform of Czech public employment services in 2011. Central European Journal of Public Policy, 9(1), 148–168.
Leong, C., & House, S. R. (2012). Rethinking policy success: The normative basis for public action. Journal of Asian Public Policy, 5(1), 3–22.
Marsh, D., & McConnell, A. (2010). Towards a framework for establishing policy success. Public Administration, 88(2), 564–583.
Marsh, D., & Sharman, J. C. (2009). Policy diffusion and policy transfer. Policy Studies, 30(3), 269–288.
May, P. J. (1992). Policy learning and failure. Journal of Public Policy, 12(4), 331–354.
May, P. J. (2015). Implementation failures revisited: Policy regime perspectives. Public Policy and Administration, 30(3–4), 277–299.
May, P. J., & Jochim, A. E. (2013). Policy regime perspectives: Policies, politics, and governing. Policy Studies Journal, 41(3), 426–452.
McConnell, A. (2010). Policy success, policy failure and grey areas in-between. Journal of Public Policy, 30(3), 345–362.
McConnell, A. (2015). What is policy failure? A primer to help navigate the maze. Public Policy and Administration, 30(3–4), 221–242.
McConnell, A. (2016). A public policy approach to understanding the nature and causes of foreign policy failure. Journal of European Public Policy, 23(5), 667–684.
McConnell, A., Gauja, A., & Botterill, L. C. (2008). Policy fiascos, blame management and AWB limited: The Howard government’s escape from the Iraq wheat scandal. Australian Journal of Political Science, 43(4), 599–616.
Meier, K. J., & Bohte, J. (2003). Not with a bang, but a whimper. Explaining organizational failures. Administration and Society, 35(1), 104–121.
Mitchell, W. C., & Simmons, R. T. (1995). Pathological politics: The anatomy of government failure. Society, 32(6), 30–38.
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. J., & The PRISMA Group. (2009). Preferred reporting items for systematic reveiws and meta-analyses: The PRISMA Statement. PLOS Medicine, 6(7), e1000097.
Mol, A. P. J. (2009). Environmental deinstitutionalization in Russia. Journal of Environmental Policy & Planning, 11(3), 223–241.
Mone, M. A., McKinley, W., & Barker, V. L. (1998). Organizational decline and innovation: A contingency framework. Academy of Management Review, 23(1), 115–132.
Murray, D., & Dollery, B. (2005). Institutional breakdown? An exploratory taxonomy of Australian University failure. Prometheus, 23(4), 385–398.
Nair, S., & Howlett, M. (2017). Policy myopia as a source of policy failure: Adaptation and policy learning under deep uncertainty. Policy and Politics, 45(1), 103–118.
Newig, J. (2013). Produktive Funktionen von Kollaps und Zerstörung für gesellschaftliche Transformationsprozesse in Richtung Nachhaltigkeit. In J. Rückert-John (Ed.), Soziale Innovation und Nachhaltigkeit. Perspektiven sozialen Wandels (pp. 133–149). Wiesbaden: Springer.
Newman, J., & Bird, M. G. (2017). British Columbia’s fast ferries and Sydney’s Airport Link: Partisan barriers to learning from policy failure. Policy and Politics, 45(1), 71–85.
Oppermann, K., & Spencer, A. (2016a). Studying fiascos: Bringing public and foreign policy together. Journal of European Public Policy, 23(5), 643–652.
Oppermann, K., & Spencer, A. (2016b). Telling stories of failure: Narrative constructions of foreign policy fiascos. Journal of European Public Policy, 23(5), 685–701.
Peters, B. G. (2015). State failure, governance failure and policy failure: Exploring the linkages. Public Policy and Administration, 30(3–4), 261–276.
Peters, B. G., Pierre, J., & Randma-Liiv, T. (2011). Global financial crisis, public administration and governance: Do new problems require new solutions? Public Organization Review, 11(1), 13–27.
Prakash, A., & Potoski, M. (2016). Dysfunctional institutions? Toward a new agenda in governance studies. Regulation and Governance, 10(2), 115–125.
Richardson, W. (2007). Public policy failure and fiasco in education: Perspectives on the British examinations crises of 2000–2002 and other episodes since 1975. Oxford Review of Education, 33(2), 143–160.
Rose, R. (1991). What is lesson-drawing? Journal of Public Policy, 11(1), 3–30.
Saurugger, S., & Terpan, F. (2016). Do crises lead to policy change? The multiple streams framework and the European Union’s economic governance instruments. Policy Sciences, 49(1), 35–53.
Schneider, A., Ingram, H., & Deleon, P. (2014). Democratic policy design: Social construction of target populations. Theories of the Policy Process, 3, 105–149.
Schuck, P. H. (2014). Why government fails so often: And how it can do better. Princeton: Princeton University Press.
Schumpeter, J. A. (1994). Capitalism, socialism and democracy. London: Routledge.
Schwartz, R., & McConnell, A. (2009). Do crises help remedy regulatory failure? A comparative study of the Walkerton water and Jerusalem banquet hall disasters. Canadian Public Administration, 52(1), 91–112.
Schwartz, R., & Sulitzeanu-Kenan, R. (2004). Managerial values and accountability pressures. Challenges of crisis and disaster. Journal of Public Administration Research and Theory, 14(1), 79–102.
Short, J. L. (2013). Self-regulation in the regulatory void: “Blue Moon”or “Bad Moon”? Annals of the American Academy of Political and Social Science, 649(1), 22–34.
Sparrow, A. (2008). Government condemned for failure to tackle bribery (2008, 17.10.2008). The Guardian.
Stern, E. (1997). Crisis and learning: A conceptual balance sheet. Journal of Contingencies and Crisis Management, 5(2), 69–86.
Stone, D. (2017). Understanding the transfer of policy failure: Bricolage, experimentalism and translation. Policy and Politics, 45(1), 55–70.
Ugyel, L., & O’Flynn, J. (2017). Measuring policy success: Evaluating public sector reform in Bhutan. International Journal of Public Administration, 40(2), 115–125.
Van Assche, K., Beunen, R., & Duineveld, M. (2012). Performing success and failure in governance: Dutch planning experiences. Public Administration, 90(3), 567–581.
van der Steen, M., Scherpenisse, J., van Twist, M., & Hart, P. T. (2015). How to see failure: Attempts by the Inspectorate of Education to detect and disarm failure in Dutch education policy. Public Policy and Administration, 30(3–4), 320–341.
Venkatachalam, L. (2004). Sources of government failure and the environmental externality: Analysis of groundwater pollution in Tamil Nadu, India. Water Policy, 6(5), 413–426.
Vince, J. (2015). Integrated policy approaches and policy failure: The case of Australia’s Oceans Policy. Policy Sciences, 48(2), 159–180.
Vining, A. R., & Weimer, D. L. (1999). Inefficiency in public organizations. International Public Management Journal, 2(1), 1–24.
Walker, G. (2014). Water scarcity in England and Wales as a failure of (meta)governance. Water Alternatives, 7(2), 388–413.
Wallis, J., & Dollery, B. (2002). Wolf’s model: Government failure and public sector reform in advanced industrial democracies. Review of Policy Research, 19(1), 177–203.
Walsh, J. I. (2006). Policy failure and policy change. British security policy after the cold war. Comparative Political Studies, 39(4), 490–518.
Wibe, S. (1992). Policy failures in managing forests. In R. K. Turner, S. Wibe (Eds.), Market and government failures in environmental management (pp. 45–82). Paris: OECD.
Young, D. R. (1977). Consumer problems in the public sector: A framework for research. Journal of Consumer Policy, 1(3), 205–226.
Zittoun, P. (2015). Analysing policy failure as an argumentative strategy in the policymaking process: A pragmatist perspective. Public Policy and Administration, 30(3–4), 243–260.
Acknowledgements
This research was supported by the Volkswagenstiftung and the ‘Niedersächsisches Ministerium für Wissenschaft und Kultur’-funded project ‘Leverage Points for Sustainable Transformations: Institutions, People and Knowledge’ (Grant Number A112269). We thank the anonimous reviewers, whose constructive comments have helped to substantially improve the quality of the paper.
Author information
Authors and Affiliations
Corresponding author
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Derwort, P., Jager, N. & Newig, J. Towards productive functions? A systematic review of institutional failure, its causes and consequences. Policy Sci 52, 281–298 (2019). https://doi.org/10.1007/s11077-018-9339-z
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11077-018-9339-z