Introduction

Failure by political institutions to adequately respond to societal problems is ubiquitous (e.g. McConnell 2015; Richardson 2007; Kirkpatrick 2012; Sparrow 2008). What we term ‘institutional failure’ as an umbrella concept embraces various, partly overlapping realms, e.g. ‘policy failure’, ‘government failure’, or ‘regulatory failure’, and becomes manifest in different ways, e.g. through ‘crises’ (Stern 1997; Alink et al. 2001), ‘breakdowns’ (e.g. Murray and Dollery 2005) ‘fiascos’ (Bovens and Hart 1995), and ‘blunders’ (King and Crewe 2013). Policy failure, in particular, has recently gained increasing attention in the literature, witnessed by no less than three special issues in the past 3 years. Public Policy and Administration investigated the persistence of policy failures (Howlett et al. 2015), the Journal of European Public Policy brought together the separate literatures on public policy and foreign policy fiascos (Oppermann and Spencer 2016a), and most recently, Policy and Politics highlighted how rarely failure is averted or followed by learning (Dunlop 2017b).

Why study failure at all? Given its widespread occurrence, failure is an integral part of the political process. An understanding of public institutions, their genesis and performance would remain incomplete disregarding the multiple aspects of failure. From a problem-solving perspective, proper understanding of processes and causes of failure—i.e. the negative and undesirable aspects of failure—may be necessary in order to avoid it in the future. Therefore, scholars have studied how failure erodes trust and confidence in major social actors and institutions, including governments and intergovernmental organisations (Prakash and Potoski 2016), how established political systems may degenerate (e.g. Farazmand 2012) and with them, their problem-solving and governance capacities (Alink et al. 2001).

However, while ample attention has been paid to such negative consequences, the potentially desirable aspects of failure and decline have received less systematic attention in the failure literature to date. And yet, it is almost a truism that negative episodes, such as crisis and failure, may also provide positive aspects and open ‘windows of opportunity’ for reform and learning (e.g. Stern 1997; Wallis and Dollery 2002) and may thus serve as a ‘leverage point’ for systemic change (Abson et al. 2017). Yet, much of the literature fails to go beyond such platitudes. Not only does this inhibit policy-makers’ ability to learn from past experiences or mistakes, it also critically hampers efforts to systematically assess and better understand in which ways, and under which conditions, failure may lead to subsequent success. Crises, for example, are often followed by a period of reform as old structures make way for something new. Deeper insights into what have been termed the ‘productive functions’ of failure (Newig 2013) may thus prove instrumental in coping with the consequences of failure, for deriving adequate and intentional responses, and for supporting the resilience of the wider institutional system.

It is against this backdrop that this article reviews the available literature on institutional failure with a view to its potential productive functions. We put forward no original argument. We rather attempt a first organised stock-taking of claims and insights about productive consequences of failure by reviewing 111 recent articles on institutional failure. We seek to highlight the ways in which productive functions of failure can unfold and intentionally be approached to advance the effectiveness of public policy and the wider political system. A closer look at the factors enabling and preventing the activation of failure’s productive potential will provide further insights into how and when institutional failure may serve as a leverage point towards achieving collective purposes. As a first systematic attempt in this direction, the paper thus aims to strengthen the conceptual basis for future studies on the productive functions of failure.

The paper is organised as follows. The "Methods" section briefly explains the research methodology. The "Concepts of failure" and "Causes of failure" sections lay the foundation to study the productive (and unproductive) consequences of failure. We begin by identifying the different topics and aspects commonly associated with the concept of institutional failure and map out the conceptual ambiguities surrounding the terms. This exercise aims to avoid the “conceptual confusion” over the different aspects and types of failure, which so far has prevented the development of “cumulative theory building into the causes and consequences of policy success and failure” (Howlett 2012, 545). As governance systems are complex and multiple interactions exist between structural institutional arrangements and particular (sets of) policies (see e.g. May and Jochim 2013), we consider the full spectrum of institutional failure. This approach acknowledges that public problem solving is multi-dimensional, covering the political dimensions of policy, polity and politics, as well as multiple systemic levels, from the individual (micro) to the societal (macro) level. Both of these analytical categories—political dimensions and societal levels—will serve to guide the review and analysis in the subsequent sections and ultimately helps to understand and structure our ‘findings’ on the productive potentials of institutional failure and decline. We explore the different definitions and causes found in each of the different political dimensions. Building on this, "Consequences of failure—exploring its productive potential" section focuses on the consequences of failure. While not sparing the ‘negative’ consequences, the main part reviews the potentially productive functions of failure, as well as the factors conditioning their activation. The final section draws out the most important findings and implications of our review.

Methods

This review was conducted following the PRISMA guidelines (Moher et al. 2009), a set of items used for reporting in systematic reviews. We have closely followed the PRISMA checklist and documented our steps (see also Online Appendix) throughout all stages of this research. In the following section, we provide an in-depth description of our approach.

In order to identify the relevant literature for our review, we searched the Scopus database using the search strings specified in Online Appendix 1. Despite the limitations inherent to a focus on published academic literature, we believe this approach is justified for two main reasons: Firstly, as we aim for the consolidation of the present academic body of research, we targeted peer-reviewed publications. Second, this strict approach strengthens the transparency and replicability of our review.

Search strings included and combined terms related to various forms of institutional failure (e.g. government failure, policy failure, bureaucratic failure) and decline, collapse and deinstitutionalisation. By formulating a search string with a relatively high sensitivity and low specificity, we deliberately ‘cast a wide net’ to capture as many relevant papers as possible. The search was limited to publications in English, including all results up to April 2017.

Our search yielded a total of 8388 results. To filter only the most relevant records, we established a number of exclusion criteria developed alongside the research question. The criteria listed in Fig. 1 were used to assess the publications’ relevance based on their title, abstract and keywords in a first-stage screening, excluding those that failed to meet any one of these criteria. The remaining articles were then assessed for eligibility in a second-stage screening of the full text using the same exclusion criteria. This resulted in a final selection of 111 publications to be included in our systematic review (Online Appendix 2). Figure 2 gives a breakdown of the number of publications per year and an overall trendline.

Fig. 1
figure 1

Systematic case selection process as PRISMA flow diagram (Moher et al. 2009)

Full size image
Fig. 2
figure 2

Number of publications per year. Note: Publications for 2017 include January to April only. The dashed line indicating the broader trend was calculated through Friedman’s ‘Supersmoother’ algorithm

Full size image

Both the first-stage and second-stage screenings were undertaken by the first author. To minimise errors and personal bias, extensive and rigorous trial-screening was undertaken by all of the authors at every stage until agreement on the correct interpretation of the exclusion criteria was high and all reviewers arrived at a common understanding for screening.

The assessment of the included publications was undertaken through a catalogue of questions and variables operationalising our research questions. Following a comprehensive pretest for specification and calibration of the variables, following the same procedures of trial and calibration as during screening, coding of individual papers was undertaken by single reviewers. The coding scheme (Online Appendix 3) applied to the selected publications largely consists of open-ended questions. Relevant text fragments were identified through a qualitative content analysis of publications and coded according to the relevant variables, enabling the subsequent analysis presented in this paper.

Concepts of failure

Failure and crisis are not singular, clear-cut concepts, but instead comprise of a wide variety of different and at times closely connected types. An overview of definitions can be found in Online Appendix 4. Despite this review’s focus on conceptual contributions, we found that the majority of included publications do not offer an explicit definition of the concept discussed. Nevertheless, there appears to be some consensus on which aspects the different concepts of failure entail. Overall, definitions can be distinguished regarding the domain of the political system they address. First, there are concepts of failure with policy as the centre of focus (e.g. policy failure, policy fiasco), discussing the failure of specific policies, or their policy-making process. Second, there are those which focus on the structures, particularly on the government apparatus, or the polity-level (e.g. institutional failure, governance failure), discussing the inability of the institutional system to deliver. Finally, several definitions address the failures of the political process, focusing on the politics dimension and the qualities and legitimacy of processes. While these domains may be analytically separated, several definitions address multiple domains of the political system and, hence, stress the interlinkages between these, highlighting, e.g. how political struggles influence particular policy-making and failure in one dimension may not lead to failure in another (see e.g. McConnell 2015).

Failure in the policy realm is often defined as the inability to achieve goals or targets formulated in a particular policy or series of policies (e.g. McConnell 2010, 2015), or the failure to do so in a cost-effective (Murray and Dollery 2005) or coherent manner. A number of definitions of policy failure (Hall 2011; Hajnal 2010) and government failure (e.g. Keech and Munger 2015; Harris 2007) focus exclusively on substantial or content-related criteria or benchmarks for success. Commonly used indicators to measure performance in this context include the physical measurement of effectiveness, e.g. in terms of water quality (Juhasz 1989), commonly comparing outcomes to the initial goals. Discussions of government failure, in particular, strongly focus on content-related benchmarks, commonly assessing the ability to achieve intended outcomes in relation to indicators such as the efficient allocation of resources or optimal social welfare through traditional cost–benefit analyses and economic modelling.

Going beyond individual (policy) failures, several definitions consider the structural dimension upon which failures build up, thus focusing on the polity-, or structural level. Fike and Gwartney (2015), for example, refer to the importance of the ‘structure of incentives’ leading to the counter-productive use of resources. Some of the definitions of governance failure (e.g. van der Steen et al. 2015) similarly focus on the inability on a system level to deliver the goals of a policy programme. Definitions of crises, in particular, concentrate exclusively on shocks to the institutional structure (Alink et al. 2001) and wider sociopolitical order (Boin et al. 2008).

Lastly, focusing on the aspects of politics, success or failure is often considered in relation to the legitimacy of policies and the wider political system. This legitimacy can be assessed in different ways; indicators here relate to the representation of different interests (e.g. Farazmand 2012), the use of correct and legal procedures or accountability (Benitez et al. 2012; Dolfsma 2011), distributional equity (Acheson 2006; Cullis et al. 1993), and the existence of political and/or public support (Howlett 2012; McConnell 2010). A strong focus on accountability and legitimacy is found in discussions of governance failure, in which goals are commonly modified through processes of negotiation and reflection. Without predetermined reference points or criteria by which to judge, the inability to self-organise or formulate shared objectives is therefore judged a failure (Peters 2015; Walker 2014).

In these attempts to define failure, two ‘lenses’ (Howlett 2012) or ‘counter-tendencies’ (McConnell 2015) can be distinguished within the literature. The first is a constructivist lens, in which failure is construed through argumentative processes (e.g. Kay and Boxall 2015; Zittoun 2015) or rhetorical and political ‘framing contests’ (Bovens and Hart 2016), rather than considered to exist in its own right (Dunlop 2017b). The production of a dominant interpretation of failure reflects underlying power relations in the political arena or society (Marsh and McConnell 2010; Oppermann and Spencer 2016b), as a “socially and politically significant group of people” (McConnell et al. 2008) needs to perceive something to be a failure for it to considered one.

In contrast, the rationalist, scientific lens (McConnell 2015) aims to draw a clear line between evidence and evaluation and has sought to develop specific and objective indicators to measure clearly defined goals, thus allowing an assessment to be made of whether something is functional or failing (Kotrusová and Výborná 2015). This lens includes a ‘technical’ approach to failure that sees (policy) success and failure as purely technical issues that can easily be solved (Howlett et al. 2015), and a ‘politico-administrative’ approach focusing on highly complex procedural and political aspects that are difficult to analyse (Howlett et al. 2015; Dunlop 2017b). While some definitions may be firmly grounded in one of these lenses, many contain elements of both.

Causes of failure

In order to learn from failure or positively use it as a leverage point for systemic change, it is important to first understand what causes it. We find a wide array of causes throughout the selected publications, nearly all of which focus on the underlying roots of failure, rather than mere triggers. Triggers, in this respect, are considered as short-term events that start something which is already primed to happen, while causes are understood to be the deeper reasons for an action or event to occur in the first place. We distinguish between three levels of causes found in the literature (see also McConnell et al. 2008; Dunlop 2017b): individual (micro), institutional (meso) and societal (macro-) level causes. On an epistemological level, the publications analysing causes of failure tend to adhere to a rational-scientific as opposed to a constructivist view on failure.

Levels of causality

Micro-level causes of failure occur on the level of individual actors, mostly on the part of a policy-maker or administrator. Despite being only little discussed in the reviewed publications, a number of sources of failure can be found on the individual level, particularly in relation to government failure and policy failure. An important distinction is made here between ‘intentional’ failure, as the result of intentional behaviour on the one hand, and ‘unintentional’ or accidental failure on the other hand. Commonly mentioned causes of intentional individual-level failure are self-interested behaviour of actors in the political system (e.g. Kortt and Dollery 2012; Harris 2007) and corruption (Backhouse and Medema 2012), with policy-makers pursuing private goals over societal interest. Unintentional sources of failure are largely the result of either cognitive and behavioural limitations (Hajnal 2010) and include a lack of competence or expertise of individual bureaucrats (Short 2013; Wibe 1992), human errors by individual decision-makers (Bovens and Hart 1995; McConnell 2016), individual personality traits such as over-confidence (Brummer 2016) and a lack of prudence or foresight (Howlett 2014).

Meso-level causes of failure are found on an institutional level and include intra-organisational factors. The literature on government failure, in particular, strongly addresses structural sources of failure: One of the most frequently cited causes of government failure is the lack of incentive for the efficient use of organisational resources (e.g. Vining and Weimer 1999), resulting in low morale and poor management/performance (Meier and Bohte 2003; Andrew 2008), aggravated further where promotion is based on longevity rather than effectiveness (Young 1977; Backhouse and Medema 2012). Another important source of failure is the existence of information asymmetries between the public and private sector (Acheson 2006; Helm 2010). Here, governments are perceived to inherently have less information and knowledge at hand than the private parties they are tasked with regulating. Other important operational sources of failure include a lack of (financial) resources, skills or capacity to design and enforce policies (Short 2013; Vining and Weimer 1999), weak or absent checks and balances (Benitez et al. 2012), and deficient feedback mechanisms (Mitchell and Simmons 1995), with political processes failing to provide the information required to reach socially optimal policies (Cullis et al. 1993). While the policy failure literature similarly traces meso-level causes back to these institutional design and capacity issues, it often focuses specifically on sources of failure in the implementation stage (Vince 2015), such as the use of incomplete or out-of-date policy information (Wibe 1992; Schuck 2014), inadequate planning (Kotrusová and Výborná 2015), poor choice of regulatory instruments (Hansen 1983), and the lack of diversity and dissent in policy-making processes (Bovens and Hart 2016). Discussions of governance failure, finally, focus strongly on procedural sources of failure, considering the failure of parties to coordinate and cooperate on an personal, organisational, and systemic level as an important source of meso-level failure (Jessop 1998; Peters 2015). Where institutions are insufficiently flexible to deal with new challenges or changing circumstances, they will be susceptible to rejection or replacement (Prakash and Potoski 2016; Mol 2009).

Finally, macro-level causes of failure are found in the wider societal, economic, and natural environment. While these causal factors are found outside the political system, there is an important interaction between system actors and their environment. One important external source of failure is the power of private sector, interest groups, and the public, whose opposition can result in a policy or other measure being unsuccessful (e.g. Acheson 2006; Gibb 2015). In those situations where there is a mismatch between the underlying principles of a policy and those held by the community, this can be an important source of failure (Leong and House 2012). The crisis-related literature most strongly considers macro-level causes, focusing on society-wide crises and external shocks. Such crises are often economic or political in nature, but can similarly be caused by natural disasters, infrastructure breakdown or (industrial) accidents (Boin et al. 2008), often involving a large degree of social unrest or threats to national security (Grossman 2015).

Establishing causality

Locating the actual causes of failure is subject to debate. Recent research (e.g. Howlett et al. 2015) argues that the persistence of policy failure suggests its causes lie beyond mere technical causes which would be relatively easy to correct. Particularly in the case of so-called wicked problems—complex problems with multiple causes lacking clear solutions—policy-makers may tend to focus on the symptoms of a problem, thereby remaining on a technical or operational level, rather than address its underlying social causes (e.g. McConnell 2010), which would require also a broader time perspective (Brändström and Kuipers 2003). While technical, short-term solutions may allow an issue to move down the policy agenda and demonstrate policy-makers’ ability and commitment to deal with a problem, it does little to address its root causes (McConnell 2015).

Beyond the focus on single causes, several contributors stress that there may be multiple, complex, and interacting factors which together result in failure (McConnell et al. 2008; Bressers et al. 2013; Mol 2009). In this line, McConnell (2016) emphasises that to argue that “one factor alone is the cause of a failure would be to neglect the range of individual, institutional, and societal factors that interacted to produce that failure—as well as their complex interdependencies” (675).

Finally, it is argued that, like measuring failure, establishing causality is not a purely rational-scientific or value-neutral exercise, with the nature and significance of causal factors considered to be contestable and dependent on one’s wider perspective on society (McConnell et al. 2008). Without a scientifically objective standard by which to judge, under constructivist approaches, success or failure is constructed socially through “informed debates” (Ugyel and O’Flynn 2017) among competing interests. Under such constructivist approaches, technocratic evidence-based expert enquiries into the causes of failure are often set against public enquiries with a stronger focus on public perceptions of performance (Kay and Boxall 2015).

Consequences of failure—exploring its productive potential

Based on this informed understanding of how different concepts of failure fit together, and what causes failure, we now look into to its potential consequences. Most obviously, and following conventional wisdom, failure is clearly negatively connoted. It is therefore not surprising that the literature predominantly describes socially undesirable consequences. Some works, however, point to potentially productive functions of failure. Below, we first briefly discuss the negative consequences of failure, before turning our attention a discussion of the productive potential of failure and the factors enabling or constraining its activation.

Negative consequences

Negative consequences of failure are well established in the literature. Arguably, the hardest consequences will be felt by the addressees of failing institutions and policies, which may lead to damage to their property or their social or material well-being (van der Steen et al. 2015; Ber 2013), particularly for those most dependent on the state (Gibb 2015; Schuck 2014). Failure can also have serious career implications for policy-makers and individuals within the political or administrative system (Griffin 1987; McConnell 2016; Howlett et al. 2015), whether as the result of fair and balanced evaluation or of blame games and the search for individual culprits (e.g. Bovens and Hart 2016).

These individual effects are often interrelated to negative developments that threaten well-being on the wider societal and economic level (Gibb 2015; Schuck 2014). Where individual actors pursue self-interested behaviour, such as (political) rent-seeking and outright corruption, this is often in direct contrast to wider societal interests, causing detrimental effects on a system-wide level. System-wide effects may become manifest in, e.g. the inefficient allocation and use of resources (e.g. Juhasz 1989; Wallis and Dollery 2002), the distortion of market prices, and increased levels of social cost to the economy (Venkatachalam 2004), potentially also resulting in the depletion of scarce resources (Acheson 2006) and environmental degradation (Juhasz 1989).

In the political system itself, failure and crisis often generate a sense of urgency, threat and uncertainty, thus revealing vulnerabilities in the system (Boin et al. 2008). If sufficiently strong, such events can severely undermine trust in the system (e.g. Saurugger and Terpan 2016; Farazmand 2012) and reduce electoral support (Howlett 2012; Schuck 2014), particularly as challenges become more frequent. Ultimately, fundamental threats to its core values can result in the destabilisation of the system as a whole (Saurugger and Terpan 2016; Hansen 1983), undermining the position of regime actors and encouraging the growth/success of challengers (May 2015; McConnell 2010).

Productive potential

Around half of publications in our sample discuss the consequences of failure, with only around a quarter addressing the ways in which these are associated with desirable implications of failure. Most publications remain relatively generic in their findings, simply arguing that failure can provide a window of opportunity for reform (e.g. Farazmand 2012) or learning (e.g. Gibb 2015), or may spur positive change (Marsh and McConnell 2010). While largely implicit, many of these publications hint at deeper-lying assumptions and understandings of these productive functions of failure. The "Productive potential" and "Factors conditioning the activation of productive potential" sections tease out this implicit understanding of productive potential on a policy, polity and political level in greater detail, as well as the conditions enabling or hindering its activation.

While the destabilisation of the political system was previously perceived as a major detrimental effect of failure, some contributors also stress its productive potential to open up windows of opportunity (e.g. Saurugger and Terpan 2016; Schwartz and McConnell 2009). As explicated in theories of the policy process, e.g. the Multiple Streams Framework (Kingdon 1984), events such as failure and crisis punctuate the normal rhythm of policy-making (Schwartz and McConnell 2009). Failure may stimulate political momentum, helping to unfreeze institutional rigidity and inertia (Alink et al. 2001). In some cases, the complete dismantling or removal of institutions may even be required to unlock existing pathways and allow the creation of new ones (Giest 2017), an argument closely linked to Schumpeter’s (1994) ‘creative destruction’. Hence, these windows of opportunity present potential turning points for reform, institutional change (e.g. Kay and Boxall 2015; Grossman 2015), and learning (Hall 2011; Gibb 2015). Synthesising these diverse aspects, a deeper analysis of the collected literature reveals that the ‘window of opportunity’ takes on a distinct character in the different dimensions of the political system—policy, polity, and politics (see Table 1 for an overview), and with it alternative strategies for reform, learning, and lesson drawing are emphasised.

Table 1 Productive functions of failure: Intervention types and instruments in the dimensions of the political system
Full size table

In the policy domain, given its focus on the policy tools and targets, failure may result in the fine-tuning of individual policies or instruments but they may also enable extensive reform of (individual) policies and lead to changes in the policy-making process. However, as the wider system remains essentially stable, reform and learning will most likely not take place beyond the relatively ‘shallow’ level. In this sense, the literature extensively refers to May’s (1992) ‘instrumental learning’.Footnote 1 Instrumental learning largely focuses on modifying policy tools or instruments to achieve programmatic roles. Thus, while it may lead to the adoption of new policy instruments and techniques, overarching policy goals remain unchanged.

In the polity domain, the consequences of failure may reach beyond the policy field and also trigger structural, institutional change in the political system, altering the systemic incentives and structures that led to institutional failure. Individual or low-level policy failures alone are unlikely to trigger change at this level. Rather, repeated failure and the erosion or even disappearance of institutions are required for system-wide changes to be able to take place (Kay 2017). While this inevitably comes with negative consequences for some, it also offers the opportunity to break with older patterns of inertia and path-dependency (Ball 2005; Boin et al. 2008), or for “rebirth and rethinking” (Dunlop 2017b, 9). Pressures that may potentially be able to stimulate a break with established patterns include: political pressures (e.g. a change in the way power is distributed), functional pressures (i.e. changing views about the instrumental value of institutionalised practices and procedures), and social pressures, or the loss of cultural consensus (Ball 2005). Hence, failure in this respect is likely to be accompanied by deeper ‘social learning’ (May 1992) and reforms, involving changes in underlying policy beliefs and paradigms. There is little indication in the literature, however, that processes of destabilisation and reinstitutionalisation are intentionally used within a public policy framework, nor does it become clear whether or how these processes can be steered consciously with a particular alternative in mind.

In the realm of politics, windows of opportunity may involve the break-up of incumbent power structures and discourses. This may allow actors to form alternative coalitions or policy entrepreneurs to advance major, previously condemned changes (Grossman 2015; Saurugger and Terpan 2016; Walsh 2006). By putting the blame on one set of political actors—e.g. the government currently in power, or the proponents of a policy—opposing actors can seek a break with the status quo and push forward their preferred solutions. This strive for reformism may, however, face strong opposition from incumbent elites, who could use the failure as an opportunity to demonstrate the relevance and fundamental stability of the system, thus legitimising its very existence (Van Assche et al. 2012). Learning on this level, referred to as ‘political learning’ (May 1992), is thus concerned with advocating for preferred policy ideas and to enhance their political feasibility (Hall 2011).

As these potentials for change in all three domains highlight, learning may play a pivotal role for harnessing this change. Learning is defined in the literature as “improved understanding” (Newman and Bird 2017) or “the updating of beliefs” through own or witnessed experience, analysis or interaction (Dunlop 2017b). But sources of learning are not limited to own experiences: failure may also stimulate learning and reform with those who have not been direct subject to it. Referred to as ‘policy transfer’ in the literature (e.g. Giest 2017) experiences in other political settings in the past or present enable learning without the costs of negative experiences or an immediate need for action (Stern 1997; Marsh and Sharman 2009). In doing so, it is important to keep in mind the high context-specificity of policy environments, which can in some cases cause perceived lessons to be almost entirely opposed to another (Peters et al. 2011). Learning, especially in contexts of failure, also includes the ‘unlearning’ of certain routines to ensure openness to change, and ‘negative lesson drawing’, or learning what not to do (Stone 2017; c.f. Rose 1991).

Factors conditioning the activation of productive potential

Given the potentially productive functions of failure, the question remains how and under which conditions these can be effectively realised. Overall, the discussion of how to overcome negative events in the selected publications is relatively light-touch and little specific. While the literature frequently refers to the different ways in which failure can lead to productive consequences, it risks becoming a hollow statement if not sufficiently elaborated. The pertinent question is therefore which factors facilitate or hinder turning a negative consequence into a positive one, or how, and under which conditions the positive potentials of failure can be activated and reaped. While some of these factors are internal to the political system (e.g. institutional structures and actors, policy precedents, and information), others are external (e.g. economic markets, international politics) and thus more difficult or impossible to control. A comprehensive list of the enabling/hindering factors discussed in the literature is compiled in Table 2.

Table 2 Factors enabling and/or hindering productive functions
Full size table

On the individual (micro-) level, ideological constraints and mental traits such as defensiveness and risk-averseness, as well as the tendency to reject negative lessons play an important role in actors’ ability to turn experiences of failure into something more conducive. Ideological and psychological openness may enable individuals to activate the positive potential of failure. Dunlop (2017a), for example, refers to the ‘absorptive capacity’ of policy-makers’, the ability to distil key information when gathering information, whilst at the same time staying open to significant criticisms and assessing the appropriate speed and intensity with which to react.

Ample consideration is given to factors on the institutional (meso-) level. Here, factors such as the deep institutionalisation or lock-in of rules and practices, the lack of institutional capacity, communicative, and analytical capacity are considered to be important factors hindering or preventing the activation of productive potential. On a political level, ‘political realities’ such as power imbalances between actors and blame-avoidance strategies also appear to prevent negative consequences from becoming productive, with short-term action standing in the way of viable long-term solutions.

Consequences on the micro-level, such as the resignation or demotion of responsible officeholders (or scapegoats) may “serve as pressure valves that stymie attacks from opponents” (Hinterleitner 2018), preventing further action on a system level. On the other hand, enabling factors such as the presence of high levels of interpersonal trust, channels of effective communication and coordination, and institutional memory are considered to be conducive to activating this potential. Regardless of all these factors, good timing appears to be crucial if one is to exploit an available window of opportunity (Alink et al. 2001).

Finally, on a societal (macro-) level, uncertainty surrounding available policy options and their associated outcomes may be a significant barrier preventing factors, while a political climate conducive to policy change is considered to be an important enabling factor. How these factors “work” and how they might interact in practice remains unclear from the literature, offering interesting potential for future research.

Discussion and conclusions

Reviewing the literature in a systematic fashion revealed that while the negative consequences of failure and decline are discussed more widely in the literature, to date relatively little attention has been paid to the potentially productive consequences of failure. While failure plays an implicit role in many public policy frameworks (e.g. in punctuated equilibrium-theory), only scant attention is paid to a more explicit treatment of its productive consequences. Moreover, those publications that do discuss pathways of how failure may bring about positive change have largely remained generic and descriptive in their explanations, merely claiming that they may open a window of opportunity for reform and/or learning. It is our belief that such a coarse and simplistic view of failure inhibits meaningful change on the three dimensions of the political process. A more conscious appraisal of failure and a better understanding of its productive functions are thus required to further enrich studies of public policy and contribute to the advancement of some of the present frameworks.

The work presented here should be seen as a first step in this direction. We highlighted how failure can open up ‘windows of opportunity’ for reform, institutional change, and learning on different levels of the political system. By looking beyond the individual types in the literature, hitherto mostly treated separately, we have addressed different domains of the political system, notably the policy-level, the polity-level, and the politics-level. While failure of individual policies will mostly result in instrumental learning and the fine-tuning of policy instruments and techniques, on the polity and politics-level, the potentially productive effects of failure are much larger. Here, the destabilisation or removal of existing institutions, as well as deeper forms of (social and political) learning may provide important opportunities to overcome institutional lock-ins and break up dominant power structures. While this literature thus clearly recognises the potential ways in which failure may lead to subsequent success, it presently fails to consider how such processes can consciously be initiated and directed so as to improve existing weaknesses or flaws in the system.

As a first step in this direction, we identified a wide range of factors that may enable or hinder the activation of the productive functions of failure, particularly deep-level structural changes. These include individual traits and constraints of decision-makers, institutional capacities, power relations, as well as the overall complexity of the wider system and tie in closely with the causes of failure, which can similarly be found on the micro, meso, and macro level. Particularly on the meso level, for which the most information is available, the identification of these enabling and hindering factors offers a great potential to help prevent, or better utilise, structural-level failures should they occur.

So how to move forward from here? More attention, through empirical and conceptual studies, will be needed to improve our understanding on how and through what mechanisms failure is likely to unfold its positive aspects or may even be activated intentionally and subsequently steered or guided in the desired direction. On a conceptual level, to further map and explore the positive potentials of failure, research should identify archetypical mechanisms of productive functions going beyond the windows of opportunity and reform, identified in our review. Special attention may be paid to the role of agency and ways to harness productive potentials. On a more empirical level, looking at cases of failure through the lens of enabling and hindering factors may enable us to better determine the relevance of individual factors and provide policy-makers with valuable clues as to where their intervention in the system may be required. A promising strategy forwards also lies in exploring the links between the concepts and productive functions of failure and established political theories and concepts of institutional change. Theories, such as the Social Construction Framework (Schneider et al. 2014), or the Narrative Policy Framework (Crow et al. 2018) may offer insights into the causes and construction of failure and the roles and strategies of actors to reap the productive potentials. Especially the tension between blame avoidance and learning and reform (Howlett 2012), as a major hurdle for the activation of productive potentials of failure on an individual and institutional level, may benefit from more attention and cross-fertilisation. Such an effort would be conducive to better anchoring the subject in broader policy and failure research and contribute towards fixing the “broken link” (Dunlop 2017b) between failure and learning.

Empirically, a next step may be to set the conceptual literature into dialogue with empirical findings; attention should be devoted to identifying relevant cases in which failure has been productive in practice, and where mechanisms identified here, such as negative lesson drawing, could be observed. Such an endeavour would help answer the question how and when failure can be put to use, and which actors play an important role, a question that cannot be answered from the conceptual literature alone. However, the empirical study of failure is facing particular challenges: issues of desirability bias and limited access to detailed information may prevail, especially in situations where actors report on their own experiences. Empirical research on failure and its productive functions may benefit further from innovative approaches for research design and methodology.

Finally, we strongly encourage more cross-disciplinary fertilisation to better understand how different fields deal with failure. The organisational literature, in particular, with obvious interfaces to the institutional literature, offers a strong potential to contribute to this debate (e.g. Mone et al. 1998; Frese and Keith 2015). Combined, these different research efforts may ultimately prove instrumental in coping with the real-life consequences of failure and developing adequate responses in practice.