1 Introduction

The “policy mix” concept has become central to the innovation policy debate over the past two decades. The term refers, on the one hand, to the composition of a policy, the features of its constitutive elements (or components) and their relative balance, and on the other hand, to the potential interactions between these components, the idea of interaction being central to the concept itself. Nonetheless, despite growing policy interest, the concept has been more often used in a normative sense, with few evidence of how innovation policy mixes are composed and even thiner evidence of the interactions at play.

The phrase emerged in the economic policy domain when Mundell (1962) examined the combination and trade-offs between monetary and fiscal policies. The term became increasingly popular in the 1980s and 1990s when it was echoed in several studies, notably in the environment policy domain. As concerns about environmental degradation called for better climate policy action, further attention was given to policy effectiveness. Evidence since then pointed to misalignments across policy domains, such as trade, innovation and skills policies, misalignments between finance, taxation and investments, and the coexistence of regulatory and policy frameworks outside the climate policy portfolio that were not aligned with climate objectives (OECD 2015a). In the same vein, Rogge and Reichardt (2013) recalled that the environment policy mix needs to incorporate environmental policy instruments together with eco-innovation policy instruments. More recently, the articulation and balance between framework policies and size-contingent policies, as well as well between national and subnational policy action, come at the forefront of a debate on rethinking entrepreneurship and small-and-medium-sized (SMEs) policies (Amorós et al. 2019; OECD 2019).

In the innovation policy domain, international organisations, such as the European Commission (EC), the Organisation for Economic Cooperation and Development (OECD) and the United Nations Educational, Scientific and Cultural Organisation (UNESCO), were instrumental in spreading the concept within the policy discourse. Further elaboration on its historical diffusion in the innovation policy literature is available in Guy et al. (2009), Flanagan et al. (2010), OECD (2010a) and Edler et al. (2013).

Whereas the term was increasingly used among innovation policy communities, and the need for better evidence of good innovation policies became increasingly pressing, discussions around innovation policy mix remained to a large extent theoretical and rhetoric (Ghazinoory et al. 2019; Neicu et al. 2016). In fact, the concept is most often used in a normative sense, research work questionning the features of the mix in terms of “coherence”, “consistency”, “coordination”, “efficiency”, “appropriateness”, “balance”, “stability”, “predictability”, “comprehensiveness”, “legitimacy”, “credibility” etc. (Ringeling 2005; Howlett and Rayner 2007; Guy et al. 2009; Nauwelaers et al. 2009; Flanagan et al. 2011; Bórras and Edquist 2013; Edler et al. 2013; OECD 2010a, 2016a; Rogge and Reichardt 2016; Rogge et al. 2017).

In addition, evidence of the impact and interplay of policy mixes in the innovation policy domain is remarkably thin (Edler et al. 2012). Evaluations are largely done in isolation, focusing on two or three instruments but rarely considering the interaction of one instrument with broader mixes (Edler et al. 2013). More recently though, a particular effort was paid to examine how policy instruments, and their design features, may interact with policy strategies and policy processes for supporting energy transitions (Rogge et al. 2017).

Edler et al. (2012) also noted that the role of implementation structures and processes in policy mixes are often neglected in evaluations, despite being major determinants of success and impact. Policy making is actually no linear process that would directly translate scholarly ideas into policy rationales, and policy rationales into policy instruments (Flanagan et al. 2010). Therefore, policies have to be analysed together with the policy making process that shaped them (Kay 2006) and histories of policy mixes are needed, although such assessments remain extremely rare (Edler et al. 2012).

The article therefore highlights the following research questions:

  • Which principles for monitoring innovation policy mixes apply?

  • How to map and measure the composition of innovation policies and identify interactions within policy mixes?

The remaining of the article is structured as follows. The first section presents the reasons why governments’ demand for a ‘policy mix’ understanding of innovation policies has not been fully meet. This section identifies in particular three types of gaps that have prevented policy makers from implementing a more holistic approach in innovation policy evaluation: gaps in concepts, gaps in data and gaps in data management systems. The second section shed the foundations of a mapping, drawing on a literature review and a comparative analysis of large-scale policy mapping exercises conducted by the EC, OECD and UNESCO between 2009 and 2018. The third section provides indications for future research: it proposes different and complementary methodologies for analysing the composition and transformation of an innovation policy mix over time; and it translates the mapping exercise into measurement with a range of new policy metrics. The concluding section provides lessons for advancing research on innovation policy mix and improving measurement and monitoring.

1.1 Gaps in effectively assessing innovation policy mixes

1.1.1 Increasing complexity of innovation and related policies

The innovation “policy mix” concept gained popularity in a context of growing policy complexity and uncertainty. The scope of innovation policy has broadened beyond its traditional scientific and economic missions. More innovation, more inclusive innovation, greener innovation are needed to improve social well-being, or cope with environmental and societal challenges (OECD 2010a, b, 2014, 2015b). Climate change has been reshaping innovation policies for over a decade (OECD 2010c, 2012a). Global value chains (GVCS) are changing the modalities of public intervention which is increasingly geared for impact to be achieved beyond national boundaries. Likewise, grand challenges require greater international coordination, cross-border governance and a partial transfer of authority to intergovernmental organisations to benefit from transnational externalities (OECD 2016a).

Another major driver of complexity was the replacement of the linear model of innovation with a more holistic and system-based paradigm, that has opened up the spectrum of rationales and instruments for public intervention (David and Foray 1995; Metcalfe 1995; Nelson 1993; OECD 1996, 1998a). Innovation was understood as a linear process, from the R&D laboratory where basic research is undertaken, to the firm’s workshop where applications are developed, to the market where new products or services are diffused. Science was at the core of innovation dynamics and policy action focused on ensuring an optimal level of R&D expenditure (Kotsemir and Meissner 2013).

More recent understanding of the innovation process stressed the importance of open innovation, human resources, innovation culture and interface management, together with incremental and non-technological innovation for socio-economic development (OECD 2001; Sheehan et al. 2005). Evidence show that most innovative firms combine different modes of innovations (OECD 2010d, 2015c) and complementarities between non-R&D and R&D innovation even suggest that non-R&D innovation can be a stepstone to more, more systematic and more valuable R&D.

The changing nature of innovation prompted governments to go beyond the market failures that lead to a sub-optimal investment in R&D (Meissner et al. 2017). A range of system failures related to knowledge transfers and system capacities was identified (Arnold 2004). Policy attention shifted away from science, as a driving force of the innovation process, and placed corporate activities at the centre of innovation systems. Research policy became a sub-policy domain of innovation policy.

Complexity also arose from the very way policies are made. Policy dynamics are path-dependent and sometimes irreversible. Policies are adopted in a context of pre-existing policy mixes and institutional frameworks, and they are often rationalised retrospectively, theory being used to justify action while original choices were influenced by norms, beliefs or lobbying. Specific ideas come to prominence at certain times and, as ideas are institutionalised, they shape future policy choices (Flanagan et al. 2010; Švarc and Dabić 2019). National policy styles can also influence the selection of policy instruments (Borrás and Edquist 2013). For instance some instruments, particularly the financial ones, dominate others for no other reason than they have been important in the past and have attracted around them vested interests that protect their position.

In fact, policy arrangements reflect bargaining processes that take place in multi-actor arenas where policies are formulated and evaluated. Resistance to change is particularly strong at earlier stages in the policy cycle, because processes, there, are highly political and subject to bargaining (Rogge and Reichardt 2016). That said, resistance can also appear during policy implementation, especially at subordinated governance levels.

Such instrument ‘lock-in’ contributes to create a complex landscape of policy arrangements, as new instruments add a further ‘layer’ on top of older ones. Recent analysis shows that policy sets for encouraging knowledge transfer by universities and public research institutes tend to be denser in more advanced STI systems (Kergroach et al. 2017). Based on policy information for 14 countries at different stages of economic development, the authors point out that the density of national innovation policy mixes, as measured by the number of major policy initiatives in place, tend to increase as STI systems become more R&D intensive.

Policy instruments are therefore not neutral devices but rather bear a history and some social and technical values (Flanagan et al. 2010). Legitimacy arises to a certain extent from their popularity and their degree of political endorsement. Indeed, credibility depend on the commitment of the political leadership, the stability and consistency of the policy mix over time and the level of support earmarked, i.e. the capacity of existing governance structures and agencies to administrate and eventually enforce their instruments (Rogge and Reichardt 2016). Public acceptance, sustainability and effectiveness arise from this legitimacy (Borrás and Edquist 2013).

1.1.2 Lack of standards and shared concepts for innovation policy measurement

A major barrier to better understanding how innovation policy mixes are shaped and perform is a lack of definitions and measurement standards. There is no common approach about how the concept could be adapted to the innovation policy domain (Flanagan et al. 2010; Edler and Fagerberg 2017). Rogge and Reichardt (2016) propose an overview of the various definitions found in the R&D and innovation literature. Rogge et al. (2017) compare examples of policy mix definitions taken from innovation studies with definitions used in environmental economics and policy sciences.

Some attempts to conceptualise the ‘policy mix’ have driven major operational developments, by international organisations seeking new tools and evidence for their policy recommendations. Various online databases (accessible at different points in time) present—or presented—STI policy information in a cross-country, structured, and more or less comprehensive way.

Guy et al. (2009) identified a generic approach to the design of R&D policy mixes. The authors proposed a conceptual framework including models of governance and innovation systems, policy domains and policy mix characterisation. This model served for the EC platform on European research systems.

The EU Joint Inventory of Policy Measures was created to facilitate access to information on research and innovation policies within Europe and beyond. The database presented national research and innovation policy measures (i.e. policies and programmes) by bringing together two pre-existing infrastructures, ERAWATCH and INNO-Policy (ProInno) TrendChart. Information was reported by a network of national correspondents that monitored policy developments in their country along a common framework (Annex 1). ProInno TrendChart and ERAWATCH were operational as from 1999 and 2007 respectively, and merged in 2010. The Joint Inventory was updated until 2013 and archived in 2015.Footnote 1

The EU Joint Inventory was the most comprehensive source of cross-country STI policy information at this time. Merging ERAWATCH (public research) and ProInno TrendChart (business innovation) expanded the policy domain under review and was consistent with a broader view of innovation. Information was presented in the format of descriptive sheet which structure is given in Annex 1. A large range of policy instruments and strategic objectives were referenced but the Joint Inventory, was the only repository, among those listed below, that described policy measures according to the stage of the research and innovation process they were targeted to. The inventory included policy measures back to the early 2000, provided they were still active at the time of reporting (i.e. not repealed in the meantime), permitting tracking the time dimension and the sequencing of instruments partially.

However, the EU Joint Inventory presented several drawbacks. The conceptual framework reflected the STI policy thinking of the early 2000s. Some issues that have become prominent in the debate in-between were missing, e.g. open science, demand-side instruments etc. The typologies used did not capture the variety of actors involved in the innovation process, e.g. by focusing more on institutions than individuals (such as venture capitalists, women, students, civil society, consumers etc.) and overlooking the role of public administration (such as public agencies). The distinction between strategic objectives and policy instruments was not clear-cut (e.g. priority “2.3 State aid measures in support of business R&D”), probably as a consequence of the close interdependency between a strategic objective and the set of policy instruments available to meet it. Finally, reporting was a resource-intensive process, requiring a large amount of detailed policy information which was not always available in the country.

Emiliozzi et al. (2009) elaborated around the policy mix concept from a series of studies carried out across Latin American and Caribbean (LAC) countries, and examining legal frameworks, governance structures and policy instruments. The authors adopted a “portfolio” approach by linking five policy objectives to five families of operational instruments (Annexes 2 and 3). This stream of work provided the conceptual foundations of two pioneer platforms on STI policies in LAC countries and a prototype of the UNESCO Global Observatory of Science, Technology and Innovation Policy Instruments (GO-Spin) (UNESCO 2017).

  • The Economic Commission for Latin America and the Caribbean (ECLAC) platform presented a selection of instruments in support of S&T development in 41 countries (CEPAL 2007), focusing on a sub-system of innovation policy. Ten types of instruments were identified (Annex 2). Information was presented in the format of descriptive sheets, that are similar to the ProInno TrendChart and became a standard for LAC initiatives. Like the Joint EU Inventory, distinction between policy instruments, strategic objectives and target population was not clear-cut. Moreover, the coverage was uneven. Out of the 143 instruments referenced, almost half were technological funds and two-thirds were implemented in non-LAC countries. The ECLAC platform was a one-off (launched in 2007 and never updated since then).

  • The “Politicas CTI” portal gives access to information on innovation policy instruments and institutional settings in 23 LAC countries (CAEU 2017). The portal complements the ECLAC platform with a regional inventory of legal frameworks and STI institutional settings (Annex 3). Instruments are presented in descriptive sheets which content and format came from the ECLAC prototype. However, the “Politicas CTI” portal inherits from the gaps of its conceptual work, i.e. a dichotomy between policy instruments and governance arrangements and an unclear distinction between policy instruments, strategic objectives and target populations. Although the “Politicas CTI” platform extends the coverage of policy instruments (from 10 to 20 instruments up to 2016, and down to 16 instruments in 2017), some major instruments are still missing (e.g. procurement, innovation vouchers, patent policies etc.). On the governance side, major guiding documents, i.e. national STI strategies, plans and roadmaps, seem missing as well. The portal provides a snapshot of active policies at a time (which is not specified) and the frequency of update is unclear. The “Politicas CTI” platform was partially redesigned in 2017. In particular information is now classified along a new typology of instruments (Annex 3) which definitions are provided in Osorio and Sánchez Macchioli (2016). Nevertheless, remarks made above are still valid.

  • A prototype Go-SPIN was operational in 2012 for 33 LAC countries (UNESCO 2013, 2014, 2017). It was built on the conceptual framework developed for prior LAC initiatives and presented policy instruments along the same ‘fiche’ model as the ECLAC and “Politicas CTI” repositories. However, some of the gaps identified above were addressed: (1) the distinction between policy instruments, strategic objectives, and target population (referred as beneficiaries) was made clearer (Annex 4); (2) national STI strategies and plans were included (with links to official websites); and (3) intellectual property laws were mentioned (with links to the World Intellectual Property Organisation country webpages). However, the SPIN prototype treated governance arrangements and legal frameworks separately from policy instruments, as in the other LAC platforms. In addition, some policy instruments were not included (e.g. procurement, vouchers) and the mapping was incomplete since about a third of LAC countries were not effectively covered. The time dimension was not available. Online visualisation was similar to the EU Joint Inventory (Annex 4).

The UNESCO expanded its pilot platform to Africa, Arab States and Asia-Pacific regions in 2018. GO-Spin provides open access to information on STI policies and instruments, and it maps national STI landscapes across more than 50 developing countries in Africa, Latin America and Asia (UNESCO 2018). Information is reported by country representatives at the UNESCO. The new platform proposed enhanced analytical options, by linking instruments and institutions to the United Nations’ Sustainable Development Goals (SDGs), and by linking instruments to the classification of the Frascati Manuel, i.e. the socio-economic objectives of public budget appropriations, and the fields of science where R&D is undertaken (OECD 2015d). In addition, policy typologies were revised and simplified. The platform remains however heavily oriented towards S&T and most of the policy instruments mapped are financial instruments.

The OECD examined national innovation policy mixes in Iceland, Poland and Spain in 2006 and 2007 (OECD 2006, 2007a; OECD/FEYCT 2007) and the Organisation published in 2006 the Swiss Country Reviews of Innovation Policy, the first in a series of comprehensive country reviews that builds upon foundational work on National Innovation Systems (OECD 1999, 2015e). The OECD Science Technology and Industry Outlook 2010 presents empirical work carried out through the OECD’s Country Reviews of Innovation Policy and the conceptual work undertaken for the construction of a web-based infrastructure in support of innovation policy making, the Innovation Policy Platform (OECD 2010a; OECD/World Bank 2019).

The OECD also initiated a desk research on innovation policy monitoring with a view to developing a new generation of policy indicators via a policy mix approach. Drawing upon the EU Joint Inventory, the LAC mapping exercises and national documentation, this exploratory work helped develop a conceptual framework with preliminary typologies of policy goals, strategic objectives, policy instruments and target populations (Kergroach 2010). The framework was tested with two case studies, France and the United Kingdom, that helped sketch the features of a relational database on innovation policy measures. The work led to a restructuring of the OECD STI Outlook and its biennial policy survey (OECD 2012a). Country responses to the survey served for consolidating the preliminary typologies and fed into a first generation STI Policy Database (STIP) that includes information on STI policy initiatives in 54 OECD and non-OECD countries (OECD 2012b; EC/OECD 2016). Since 2016, the EC and the OECD have been working together on developing a joint monitoring system. A second generation STIP database, the STIP Compass,Footnote 2 was subsequently launched in early-2018 (EC/OECD 2018). It uses semantic technologies and a revised set of policy typologies, drawing upon—but distinct from—those presented in this paper. STIP Compass incorporates more than 500 interactive dashboards and provides a sophisticated search tool with smart filtering that facilitates policy discovery. These interfaces allow users to seamlessly query the database to identify country policies on a wide range of STI policy issues.

The lack of uniformity in concepts, terminology and monitoring approaches is a clear barrier to better understanding how policy mixes are shaped within and across countries, and ultimately how to better aligning policy action across policy domains. Standardising approaches for a policy mapping is a necessary step towards building time series and enabling cross-country analysis of STI policy mix compositions and performance. In addition, the lack of standards prevents integrating information drawn from different mapping exercises, limiting data exchange between platforms and repositories.

1.1.3 Inadequacy of the measurement toolkit and data management systems

As the innovation policy mix became increasingly complex in design and implementation, its measurement and data management systems turned somehow inadequate.

First, input/output metrics for assessing the performance of innovation systems, including their policy mixes, are slow to evolve. National and international statistical systems have been originally designed along the linear vision of innovation, giving a strong weight to R&D and patent data (OECD 2002, 2015d, 2018). Although the changing nature of innovation is not new phenomenon and has been widely documented, there are few alternative methods and data available for capturing and analysing the very diverse forms of innovation. For instance, patent applications are still frequently used in empirical work as a proxy of innovation output or performance. Therefore, the current measurement framework overlooks non-technological forms of business innovation, that may be critical for technology adoption or for more radical forms of innovation to occur.

Second, countries often lack capacity for monitoring their own policy mix. National and subnational statistical systems are not always developed enough for documenting policy interventions, and even less for assessing policy interactions. The lack of national infrastructure and arrangements for collecting policy information may be striking and information on policy programmes and initiatives, when it exists, is dispersed, irregular and of uneven quality. This has a major impact on time, cost and feasibility of data collection across countries and greatly limits international comparability.

Third, existing mapping exercises may lack the flexibility and comprehensiveness needed to effectively monitor a innovation policy mix. Indeed, “the design of innovation policy mix cannot be fixed once and for all and cannot be a one-fit-all solution” (OECD 2010a). Concepts, frameworks and, subsequently, data management systems should evolve as their object evolves. This means collecting information in a dynamic way, i.e. by capturing connections, hierarchies and sequencing between the components of the mix, and their changes over time. The issue of transformation in the policy mix is to be considered at an early stage during the design phase of the data management system, and re-consider any time theoretical principles or policy practices change. Currently, mapping exercises are built along the relational database management system (RDBMS). Along this model, a database describes a collection of predicates over a finite set of predicate variables, that are organised in the format of typologies and linked through tables of correspondence. The RDBMS model is rigid as little adaptable to adjustments in concepts and therefore changing policies.

Fourth, monitoring a policy mix cannot be limited to a statistical exercise as most of the mix components and interactions are not directly quantifiable, e.g. governance arrangements, regulation etc. Alternatives (e.g. qualitative indices, network maps etc.) are to be explored, that will require innovative approaches for combining sources, different types of data, including unstructured data, and different analytical methods.

1.2 Mapping innovation policy mix

The policy mix term refers to the set of policy rationales, arrangements and instruments implemented to deliver public action in specific policy domains, as well as the interactions that can possibly take place between these elements. Therefore, the concept covers both the composition of the policy mix composition and the issue of interactions between its components. This is the operational definition used in this paper and applied to the innovation policy domain.

In that vein, a mapping would require the following steps to be taken:

  1. 1.

    Defining the components of the policy mix and their features;

  2. 2.

    Defining the areas of interactions in the policy mix (i.e. where the different mix components could interplay, for instance in terms of objectives, target population, geographical space etc., hence going beyond interactions between instruments only);

  3. 3.

    Setting structuring principles for policy information collection, management and analysis.

1.2.1 Defining the policy mix components

Earlier mapping exercises commonly used a major policy initiative as the unit of observation, albeit implicitly in many cases.

Based on the EU definition, an innovation policy measure is “any activity that mobilises resources (financial, human, organisational) through innovation orientated programmes and projects; information (road-mapping, technology diffusion activities, coordination) which is geared towards innovation activities and institutional processes (legal acts, regulatory rules) designed to explicitly influence environment for innovation. At the same time, [an innovation measure] must achieve public policy objectives in the area of innovation with a percentage of (national) public funding; on a continuing basis (usually not a one-off ‘event’) and where the target group or eligible participants include enterprises.” (EC 2009a, b).

In practice, the terms policy initiative and policy measure are often used as synonyms, while the “plan” entails a strategic dimension and the ‘programme’ a more operational feature.

Consequently, a innovation policy initiative is understood as a public action that :

  • aims to achieve one or several public policy goals in the STI policy area,

  • is expected to modify the behaviours of actors and stakeholders who are part of, or influential on, the national innovation systems, and

  • is implemented with a minimum time horizon or on a continuous basis (i.e. not as a one-off ‘event’).

The STI policy area is defined in a broad sense, including a large range of different policy fields, from science and research policy (STI policy in a narrow sense), to technology policy (which is aimed at applying research solutions for different purposes and to different contexts), to various policy fields, such as education, labour, migration, intellectual property, fiscal, or regional development etc.

A major challenge arises there on how many policy areas to include in a mapping and what optimal number of observations to map, especially since documenting policies is a resource-intensive exercise. Indeed, there is a trade-off to find between the completeness and the feasibility of an evaluation (Magro and Wilson 2013), also in order to limit the cost of compiling information. In that respect, there is a common understanding across the innovation policy community that evaluation should focus on the “key”, most “relevant”, “meaningful” or “important” policy initiatives in the mix (Magro and Wilson 2013; Veugelers 2015; Kivimaa and Kern 2016; Rogge and Reichardt 2016).

1.2.2 Defining the areas of interaction in the policy mix

The idea of interaction between polices is central to the policy mix concept. Interaction may be intended or unintended. They may take the form of complementarities, reinforcing the effectiveness of other policies in the mix, or trade-offs attenuating the impact of each policy. Interactions may also be neutral and may occur within and across the different dimensions described below. Based on Rogge and Reichardt (2016), Edler et al. (2013), Borrás and Edquist (2013), Flanagan et al. (2011), OECD (2010a), Howlett (2005), Smits and Kuhlmann (2004) and Vedung (1998):

  • Policy space/domains/areas refer to the variety of policy sub-systems, in this case associated with innovation performance. Such sub-systems are characterised by different sets of norms, actors and institutions. They focus on distinct but related areas, such as public research, higher education, support for SMEs and entrepreneurs, regional innovation etc. Policy subsystems that evolve with external events and internal dynamics shape policies in a particular policy area, in a particular jurisdiction and at a particular time.

  • Policy goals/objectives/rationales provide the justification for policy intervention and relate to the causes for sub-optimal performance of actors and systems in particular domains. They typically include market, governance and system failures as well as deficiencies in framework conditions.

  • Strategic objectives/targeted actors or processes refer to the broad direction(s) innovation policy action can take. There are policy intentions specific to the STI policy area. They are derived from the rationales for policy intervention, some diagnostics of the state of the innovation system and a vision of its future. They usually target specific STI actors or groups of STI actors (e.g. firms, universities) or specific STI processes (e.g. technology transfer).

  • Instruments are identifiable techniques for public action and the means for achieving the goals they are designed for. The UNESCO has adopted the following definition : “A policy instrument constitutes the set of ways and means used when putting a given policy into practice. It can be considered as the vehicle through which those in charge of formulating and implementing policies actualise their capability to influence decisions taken by others” (UNESCO 2011).

Although there is large leeway in the choice of instruments, the selection of instruments is ultimately determined by the strategic objective(s) they are aimed to. The type (and optimal number) of instruments to be used simultaneously is context—and time-specific.

By combining policy instruments, policy makers aim to cumulate—or multiply—positive externalities of each. Although it cannot be assumed that any combination of instruments will be better than a single instrument approach (Gunningham and Sinclair 2002), it has been increasingly recognised that a multiplicity of instruments are needed to face multiple barriers and market and system failures. However, while mixing instruments, there is also a danger of redundancy, “counterproductivity” and of increasing administrative costs (ibid; OECD 2007b, 2010a). Therefore the notion of comprehensiveness of a policy mix does not refer to the completeness of the instrument portfolio, but to the policy mix capacity to address in a comprehensive way the failures and barriers it has been designed for, i.e. by mobilising the appropriate mix of instruments (Rogge and Reichardt 2016). Accordingly the following types of instruments are identified:

  • Economic and financial instruments, like grants, subsidies or tax concessions, are the most traditional instruments in STI policy (“carrots”). They are pecuniary incentives (or disincentives), in kind or in cash.

  • Regulatory instruments are legal tools that provide ‘the rules of the game’ for knowledge and innovation processes. They include laws and binding regulations (“sticks”).

  • Non-financial and “soft” instruments are voluntary and non-coercive tools that encourage the uptake of innovation (“sermons”). Information campaigns, public-private partnerships or technical norms are soft instruments.

  • “Systemic” or system-enabling instruments are interfaces, platforms, infrastructures or networking facilities that enable and strengthen interactions and knowledge flows between STI actors. In a broader definition, system-enabling instruments also support policy learning, experimentation and debate.

  • Meta instruments, i.e. benchmarking, STI indicators, technology foresight and assessment or peer reviews etc., provide strategic intelligence to innovation policy makers. They are distinct from the other types of instruments due to their reflexive role.

  • Geographic spaces refer to different innovation sub-systems (e.g. regions, cities) and different layers of governance.

  • Time refers to the timing when policy action (or inaction) takes place and the path previously followed. Policy rationales, goals, values and instruments are shaped by a changing context, itself shaped by external factors and internal path-dependent dynamics.

In sum, instruments interact along any of these dimensions and at the boundary of two or more dimensions. Public intervention in different policy domains may pursue the same policy goals, whereas strategic objectives, instruments, actors and institutions remain specific to the policy domain. Reversely, a strategic objective could be shared among different policy domains (e.g. developing skills, strengthening framework conditions), as instruments (e.g. tax incentives) and targets (e.g. SMEs, minorities etc.) might be. Magro and Wilson (2013) have described such multi-dimensional interactions between policy domains, strategic objectives, levels of governance, instruments and actors applied to the Basque innovation system.

1.2.3 By setting structuring principles for data collection and management

The literature review allows to define a policy initiative along the properties it takes within the different areas of interaction of the policy mix (Fig. 1).

Fig. 1
figure 1

Source: A prior version was presented at the Eu-SPRI Forum 2017 (Kergroach 2017)

Properties of a policy initiative.

Full size image

A policy initiative takes place in a particular policy domain (i.e. innovation policy, education policy, climate policy etc.) A policy domain is defined by answering a series of questions with regards to where public action for innovation starts and ends, whether policy intervention that is relevant for innovation but implemented by non-STI ministries—e.g. energy, environment etc.—should be taken into account and how?

A policy initiative takes place in a particular geographical space The geographical space could be defined according to the level at which the policy initiative is defined or initiated, rather than administrated, as to account for the growing decentralisation of STI policy implementation and the greater autonomy of actors (e.g. universities).

A policy initiative takes place at a particular time The time dimension refers to the temporal space in which the policy initiative is active.

A policy initiative addresses a one (or multiple) policy goal(s) reflecting the broader agenda of innovation policy. Multiple goals affect policy directionality, agenda-setting, resource prioritisation, actors’ roles and policy instrumentalisation, in ways that often differ across countries (Fig. 2). The United Nations’ Sustainable Development Goals (SDGs) provides a comprehensive framework for identifying the broad range of policy challenges governments are facing.

Fig. 2
figure 2

Source: Kergroach (2018)

From the overarching policy goals to the characteristics of a policy initiative.

Full size image

A policy initiative makes use of one (or multiple) policy instrument(s) Financial support is provided through an array of direct (e.g. grants, loans etc.), indirect (e.g. tax concessions) or mixed instruments (e.g. mezzanine funding). Non-financial (“soft”) support includes the provision of services (training, information etc.), access to facilities (ICT, labs, etc.), expertise (e.g. mentorship, consulting etc.), or visibility (e.g. awards, certification etc.). Platforms and STI infrastructures are system-enabling instruments (e.g. large-scale interfaces, one-stop shops, networking facilities). Regulatory instruments include a broad range of Acts, Laws and binding regulations in wide-ranging areas (e.g. intellectual property rights, University Act etc.). Institutions and governance arrangements are also relevant to the national innovation system (e.g. mission—and contract-based relationships between governments, agencies and actors, performance agreements of universities).

A policy initiative can be generic or targeted to one (or multiple) target population(s) Target populations include organisations (such as firms, universities and public labs, STI intermediaries like technology transfer agents, research consortia, public administration, civil society) and individuals (such as researchers, entrepreneurs, students, investors etc.). In addition, target populations, individuals like institutions, may have a domestic or an international dimension, e.g. in terms of citizenship (i.e. origins), or in terms of country of residence or tax base (i.e. location).

A policy initiative can target one (or multiple) specific sector(s) or technology(s) This is the case of policy initiatives that are designed for encouraging innovation in priority areas, or that are instrumental to new industrial policies.

In addition a policy initiative presents several characteristics, for instance in terms of directionality (demand—or supply-side, top-down or bottom-up), selectivity (competitive or universal, selective or discretionary), market orientation (aiming to blue-sky research or prototyping) and governance (e.g. centralised or decentralised). The list of such characteristics is not exhaustive and these characteristics are intrinsically related to the properties sub-mentioned.

1.3 Translating a policy mapping into measurement and evaluation

Once the structuring principles of a policy mix mapping are set, i.e. the relevant components in the mix (“major policy initiatives”), their features (or properties) and their balance are identified and quantified, the mapping can be translated into measurement. In a statistical perspective, the mapping is a scatter plot of “major policy initiatives” described along a series of individual properties. The scatter plot (policy mix) could therefore be analysed along its various dimensions (the policy mix properties).

Ffurther analysis is based on the first generation EC/OECD STI Policy database. The STIP database is one of the most comprehensive and most regularly updated source of national innovation policy information since 2012. It covers governments views on their policy mix across 50 countries worldwide. Based on the information collected in an internationally comparable and consistent way over time, by means of the OECD STI Outlook 2012–2016 surveys, the following section proposes four axes of analysis:

  • Demographics of policies, or monitoring how the scatter plot (i.e. policy mix) may change over time and across dimensions (properties);

  • Network and cluster analysis, or identifying the types and depth of connections within the scatter plot (mapping) between units (initiatives) and dimensions (properties);

  • Developing indicators for capturing the normative features of a policy mix, through descriptive statistics of the scatter plot (mapping); and

  • Empirical analysis, or better understanding the nature of the interactions within the scatter plot (mapping).

Examples presented below are provided under the assumption that policy initiatives are quantified following a simple non-weighted count. Nevertheless, should policy initiatives be accounted in budgetary, monetary or other quantitative terms, the same approaches could be followed.

1.3.1 Demographics of policies

The demographics of policies aim to monitor policy changes over time. The different periods of time are likely to be the successive periods of mapping. The approach requires tracking the sequencing of policies and is similar to that applied to business or population demography (Fig. 3).

Fig. 3
figure 3

Demographics of policies

Full size image

The churning rate represents the number of major policy initiatives that have been implemented, repealed or substantially revised during the period under review, expressed as a percentage of the total policy initiatives that are active in the portfolio at the end of the period. The churning rate is the sum of the renewal rate (new initiatives), the revision rate (revised initiatives, that have gone through substantial changes over the period), and the repealing rate (initiatives that have been repealed over the period). The churning rate can reflect changes in STI policy focus and activity across policy areas.

The stability rate is the number of policy initiatives that have remained unchanged during the period under review, expressed as a percentage of the total number of initiatives that were active at the end of the period. The stability rate could serve as a proxy of predictability in the policy mix.

In addition, a time dimension in a mapping could provide information on the lifetime of policies: average or median age of the policy mix, share of open-ended programmes in all initiatives, frequency at which programmes are repealed (average or median duration before programmes are repealed).

All demographics indicators could be broken down along the standardised features of a policy initiative, i.e. by policy goal, strategic objective, policy instruments, target population etc.

This approach was followed in the OECD STI Outlook 2014 and 2016 where experimental indicators capture the main changes in national STI policy mix by comparing the number of major policy initiatives implemented, repealed or substantially revised during the period under review to the total number of policy initiatives active at the beginning of the period.

In 2016 for instance, demographics of policies showed that government support to business innovation and entrepreneurship had slightly changed focus, forms and targets, with governments adopting “no spending” approaches and favouring policy tools that did not require additional public expenditure in the short term, particularly public procurement and tax incentives (OECD 2016a). Same method showed that some countries had undertaken a deeper overhaul in their policy mix than others. This was the case in Australia, the Netherlands, New Zealand, Spain or Turkey where new agendas and programmes were implemented between 2014 and 2016 and some existing initiatives in 2014 were extensively repealed. This experimental approach has also helped assess to which extent a revision of the questionnaire could affect the balance of responses and in particular to which extent the introduction of a new question could affect the churning rate in the policy mix.

1.3.2 Network analysis

Network mapping is a technique used to present graphical representations of connections, for instance physical connectivity of networks such as the Internet. A number of software tools exist for that purpose. This method used to serve for producing computer network diagrams. The approach has however become popular for social networks analysis as well.

1.3.3 Capturing the normative features of a policy mix

Table 1 provides examples of how to develop policy indicators for capturing the normative features of a policy mix. Some of these indicators would require further conceptualisation. The use of composite indices could also be explored.

Table 1 Tracking the features of a policy mix
Full size table

In 2017 Kergroach et al. (2017) tested this approach with a benchmarking concept that compares policies across 12 countries, taking account of their structural features, the way policies are embedded in the national STI policy context and the country’s development stage. Based on an exploratory analysis of the density of national innovation policy mixes, they found different policy approaches across countries towards supporting technology transfer.

Following a same exploratory text-as-data approach, Kergroach (2018) looked at national policy mixes for integrating global value chains (GVCs) and for technology upgrading. Results illustrate that policy mixes are polymorphs in so far as they combine different instruments across different policy domains with different functions according to national structural features and prior comparative advantages in GVCs. Industrial and cluster policies emerge as essential channels of policy intervention for technology upgrading (Rothgang et al. 2017).

1.3.4 Conducting empirical analysis

Once a policy mix mapping collates qualitative and quantitative information in a structured or semi-structured format, it can support empirical analytical work through the use of several data analytics methods.

Maghe and Cincera (2016) used the first generation STIP Database, complemented with other sources, in order to develop a cross-country typology of institutional policy settings. Policy initiatives are mapped across four dimensions: objectives, instruments, sectors and beneficiary populations. Their budgets account for between 60 and 120% of GBAORD or publicly-funded GERD and are weighted equally among different members of a same dimension. A multiple factorial analysis allows differentiating countries on their most diverging features and a hierarchical ascendant classification allows clustering countries on their most similar characteristics. This work was an early attempt to develop more quantitative benchmarking and performance assessment techniques.

2 Discussion

The proposed approach is a step forward towards achieving a more comprehensive mapping of innovation policy mixes and a better understanding of STI policies. It presents concrete specifications for operationalising an internationally comparable policy information collection, building a relational data management system, including typologies, and developing novel metrics on the innovation policy mix.

The article addresses the issues of lack of standards and shared concepts for innovation policy measurement, as well as the inadequacy of the measurement toolkit and data management systems. By setting structuring principles for data collection and management, and by extrapolating a policy mapping into policy indicators, the article provides policy makers and researchers with new policy indicators for capturing the normative features of a policy mix and new tools for monitoring and evaluating national innovation policies. In particular, the methodology proposed helps track overlaps in STI policy design and possible crowding out effects between policy initiatives.

However, under the assumption that an efficient mix mapping should capture major policy measures but not necessarily include all of them, the following core questions remain unsolved:

  • How can the major policy initiatives be selected? What are the possible criteria of selection?

  • In a budgetary approach, what is the optimal level—or ratio—of public support that a mapping should reach to be representative?

  • How should budgets be consolidated?

  • How should intended and actual spending be treated?

  • How should countries with decentralised governance and funding systems be treated?

  • How should public-private partnerships and measures that match public investment with private funding be accounted?

The notion of relevance, or importance, of policy initiatives remains difficult to appreciate. Veugelers (2015) matched the composition of the innovation policy mix in EU countries with their ranking along the EU summary innovation index. She selected the national policy measures that took up to at least 5% of total public budget and came up with six major instruments that accounted on average for about 70% of budget outlays in the EU TrendChart database. Kivimaa and Kern (2016) based their instrument mix mapping on expert views.

Nonetheless, a budgetary approach on its own cannot be sufficient for a policy mix evaluation. Focusing on public budgets increases the risk of overestimating the weight of large scale R&D and S&T-oriented policy measures, so at the detriment of public support to non-technological forms of innovation. In addition, a purely budgetary approach would exclude policy action in the form of regulation, soft instruments and governance from any assessment. As a matter of fact, the empirical analysis done so far do not explain causality and the interdependencies of policy instruments.

Another challenge for policy mapping and impact assessment arises from the question of quantifying policy initiatives. Previous reflections on how to derivate indicators from a policy mapping have mainly focused on two axes: (1) qualifying the relevant features of a policy mix, that should be reflected in measurement, (2) quantifying these normative features (Kergroach 2009). For instance, the “prolixity” of the policy mix, also referred by Guy et al. (2009) as the “density” of the policy mix, points to the number of instruments deployed. A simple counting of policy initiatives presents the advantage to be easy to understand –albeit not necessarily easy to implement or to interpret—and the counting could be discriminated by policy area, instrument, target population, sectors etc. Policy initiatives could also be accounted in terms of input (e.g. public budget allocated, see above), output (e.g. new academic spin-off, new patents filed) and outcome (e.g. additional funding leveraged, net job creation). The lack of data at disaggregated level, i.e. at the level of the policy initiative, is a clear limitation in this statistical approach. It is difficult to envision any aggregation on this basis.

3 Conclusions

The “policy mix” concept has gained popularity among innovation policy communities over the past two decades, reflecting two major STI policy trends: the increasing complexity of innovation policy that requires a more holistic approach in governance, and the increasing need for policy evaluation to support the design of evidence-based policies. Despite growing interest, the concept has often been used in a normative sense and evidence of the composition and interplay of policy mixes has remained remarkably thin.

International organisations, in search for new evaluation tools, have been instrumental in its diffusion in the innovation policy domain. The EC, the OECD and the UNESCO have developed operational definitions and typologies and conducted large-scale policy mapping exercises between 2009 and 2018. However, concepts used were not always clear-cut. In some cases, mappings failed adjusting to changing innovation policy practices. Policy processes and sequencing were often overlooked. Policy information remains of uneven quality across countries. The structuring principles of data management systems also reflected shortcomings in conceptualisation.

The article introduces the structuring principles of a mapping, as they have been applied in the first generation EC/OECD STIP database. The first generation STIP database was built on the principles of a relational database and contains countries’ responses to three surveys (2012, 2014 and 2016) (EC/OECD 2016). A second generation database, STIP Compass, has since been developed, building on the first generation database and migrating to a NoSQL technology environment (EC/OECD 2018). However, this second generation database still largely remains an exercise in policy mapping and little attempt has been made so far to measure or evaluate policy mixes.

The article raises a series of practical questions for operationalising a mapping and considers how these issues have been addressed, explicitly or implicitly, in the literature. It also provides the operational specifications of a relational data management system, including typologies. The last section explains how a policy mix mapping, with the structuring principles described above, could be translated into measurement.

This article also opens several research avenues regarding the strategic orientations of the mix, its instrumentalisation, its governance, the mainstreaming of innovation policies (as compared to more targeted approaches) and shifts over time . In fact, the analytical framework elaborated herein could be applied to different policy spaces and an interesting exercise would be to experiment this approach in other policy domains, or at other levels of governance, and identify gaps and areas for improvement.

If governments are to deliver on growth, well-being and sustainability, re-examining government policy making is the only way forward (OECD 2016b). Governments have shown a growing interest in the transformative potential of system innovations, an horizontal policy approach to systemic problems. System innovation provides a new rationale for policy intervention and would require combining new policy tools, changing the architecture and mechanisms of governance, engaging more actors into policy making, building policy intelligence and new capabilities and sequencing policies along with the different stages of the transition.

System innovation calls for going further beyond the traditional policy mix approach and, overall, for a shift in data management paradigm and measurement. Changing policy conditions and practices require moving towards more flexible and adaptable data management systems that could evolve as their object evolve and that could support data exchange and interoperability across different applications, data sources and data formats. For instance, the institutionalisation of policy evaluation and the growing number of evaluations that are conducted in countries, regions or at other levels of governance generate large and under-used amounts of information.

The semantic technology offers unprecedented opportunities for monitoring and analysing innovation policy mixes. The adoption of international standards, i.e. common concepts, definitions and lexicon, are prerequisite. In this regard, the EC-OECD STIP Compass has published its policy taxonomies on its website, including in machine-readable formats, for inspection and reuse by other knowledge management systems. International cooperation will be key for ensuring the relevance and sustainability of standards like these.

A shift towards a new data management paradigm making use of native language recognition, semantic and machine learning technologies may open new perspectives on innovation policy mixes and bring novel evidence on how they take shape and how well they serve their purposes. It is however most likely that no single method could be comprehensive enough for capturing the complexity at play and that evaluating policy mixes will require adopting a measurement mix approach.