Keywords

FormalPara The key points of the chapter are the following ones:
  1. 1.

    Introduce the Europe 2020 strategy

  2. 2.

    Discuss the importance of resources allocation for innovation

  3. 3.

    Provide an introduction to the impact assessment of research & innovation programmes and investments

  4. 4.

    Compare the official economic assessment of EU FP7 with alternative exercises

  5. 5.

    Provide a contribution to the design of a comprehensive evaluation framework of research, development and innovation (R&D&I) programmes

1 Introduction

The past decade saw a disruptive economic and financial crises that obliged to re-discuss the working rules of the worldwide economy . The innovation programmes that European Union supports since 1980s needed to be redesigned according to the new economic and societal changes also addressing the gradual European ‘lag’ in comparison with the US, Asian tigers and several emerging economies in terms of research, development and innovation (R&D&I).

The Europe 2020 strategy promotes seven flagship initiatives to catalyse progress (European Commission 2010):

  • “Innovation Union” to improve framework conditions and access to finance for research and innovation with the aim to ensure that innovative ideas can be turned into products and services that create growth and jobs.

  • “Youth on the move” to enhance the performance of education systems and to facilitate the entry of young people to the labour market.

  • “A digital agenda for Europe” to speed up the roll-out of high-speed internet and reap the benefits of a digital single market for households and firms.

  • “Resource efficient Europe” to help decouple economic growth from the use of resources, support the shift towards a low carbon economy, increase the use of renewable energy sources, modernise our transport sector and promote energy efficiency.

  • “An industrial policy for the globalisation era” to improve the business environment, notably for SMEs , and to support the development of a strong and sustainable industrial base able to compete globally.

  • “An agenda for new skills and jobs” to modernise labour markets and empower people by developing their of skills throughout the lifecycle with a view to increase labour participation and better match labour supply and demand, including through labour mobility.

  • “European platform against poverty” to ensure social and territorial cohesion such that the benefits of growth and jobs are widely shared and people experiencing poverty and social exclusion are enabled to live in dignity and take an active part in society.

Each of these initiatives is “innovation intensive” but if the 2016 European Innovation Scoreboard (European Commission, 2016) has shown really positive signs in some regions, the overall Innovation Index that measures the innovative capability of the “enlarged Europe”,Footnote 1 shows a flat trend since from 2008 (coinciding with the beginning of the financial crises).

Although Horizon 2020 is Europe’s largest single research and innovation programme ever, it accounts for only a very small proportion of the public research and innovation effort in Europe. The headline indicator of 3% fixed in the Europe 2020 strategy is made up of:

  • 1% public expenditure (of which Horizon 2020 is a minority);

  • 2% private expenditure.

To have any chance of progressing towards this goal, efforts need to go well beyond the effective implementation of a Framework Programme and understand how this can produce a leverage effect by attracting new public and private investments.

In this framework, also the way how resource are allocated for innovation becomes crucial. The role of resource allocation, in particular financial resources, was central to Schumpeter’s (1934) analysis while the relation between finance and innovation was neglected by some contemporary economists (O’Sullivan 2005). The latter approach influenced a set of policies that were often oriented towards dysfunctional incentives and opportunities across a range of sectors with the result of undermining the productive investment. This dysfunction produces effects beyond the short-term and unproductive value extraction is encouraged at the expense of value creation (FINNOV Project 2012).

It was then realised that is now the time to reconsider the Schumpeter’s foundations by identifying two levels of analysis:

  • A micro level where the entrepreneur decisions drive the resource allocation.

  • A macro level where the resource allocation interacts with structural economic changes .

For example, ICTs (information communication technologies) sectors are highly dynamic sectors in Europe and account for almost one half of the EU productivity growth; on the other hand, the EU still lacks behind US in these sectors and this is the main reason of EU-US gap. The boost of Europe’s ICT sector becomes even more crucial with the Digital Agenda for Europe, which promises to contribute an estimated 500 billion euros, or 4% of GDP to the EU economy. Which is impact of EC sponsored investments in ICT research ? How to efficiently and effectively sustain ICTs innovation strategies?

In this debate evaluations become an essential part of the EU’s Framework Programmes. They create a crucial evidence base to steer the implementation of research and innovation programmes, as well as guide future Framework Programmes. At the same time, especially after the financial crises, an accurate evaluation and the communication about the impacts generated by EU funded research represent an important instrument of “accountability” with reference to the usage of tax payers resources.

The European Commission (EC) is now carrying out an interim evaluation of Horizon 2020 that is both a legal requirement (European Commission 2017) and an opportunity to steer the debate on future R&I activities in Europe and to deliver on the Commission’s Better Regulation agenda (European Commission 2015).

The ex-post evaluation of the 7th Framework Programme (FP7) is completed and was also used to inform the intervention logic of Horizon 2020 (European Commission 2013). The evaluation of a Framework Programme is a complex exercise that covers many aspects and areas of intervention and, according to this, a structured and uniform approach is strongly needed despite of many proposal and theories that were deployed in different assessment exercises. The aim of this paper is exactly to give a, even small, contribution to the debate with reference to the macro economic impacts of EU research Programmes. In order to run this exercise, we started from the official Ex-Post-Evaluation of the 7th EU Framework Programme (2007–2013) (European Commission 2015) and we used the results obtained in three support actions projects, namely SEQUOIA,Footnote 2 MAXICULTUREFootnote 3 and IA4SI,Footnote 4 that evaluated the socio-economic impacts of the research projects funded in three ICT domains under the FP7.

2 Impact Evaluation of the R&D Programmes

2.1 Evaluating Economic Effects, Growth and Jobs

This section briefly explores the existing studies regarding the impact assessment , the evaluation of investment in technological research programmes and reviews the key findings and the methodologies used in such studies.

The value chain, that starts with research and ends with innovation activities, is characterized by a number of complex interactions, feed-back loops and variable time spans between research, development and innovation (R&D&I) phases until market penetration. The beginning of the innovation processes is marked by R&D activities in basic research , applied research or design and development. The following step is the integration of innovative products and processes, their introduction and subsequently the diffusion into the market. The assessment of economic effects of R&D requires sophisticated methods to connect the increases in R&D intensity/investments to the increases in productivity, domestic and external demand and, finally, growth and employment . While the outputs of innovation processes, such as e.g. publications, patents or prototypes, appear immediately, impacts in terms of increased sectorial competitiveness, GDP increases, increased employment or improved living conditions, need more time to become evident.

Evaluation techniques to perform projects’ impact assessment are numerous. For example, in Berghout and Renkema (2001), 65 methods were identified. Each differs in its level of detail, the range of stakeholders considered, and the characteristics of the data required. The selection of an appropriate method is critical, since success and evaluation accuracy depend on the technique’s suitability and the rigor with which it is applied (Berghout and Renkema 2001; Khalifa et al. 2001; Pouloudi and Serafeimidis 1999). To help in identifying a suitable method, Farbey (1999) proposed a set of matrices that enable to match project characteristics and evaluation techniques.

The method chosen is influenced by many factors and these include: social and organisational contexts, the organisational domain, the level of analysis, evaluation purpose and perspective, investment purpose, measurability of system impacts, and ICT application. It is now widely believed that several metrics are required to evaluate the different aspects of an ICT project (Passani et al. 2014).

The number of existing evaluation techniques are classified in various ways in the literature. For example, (De Jong et al. 1999) categorised techniques as “fundamental measures”, “composite approaches” or “meta approaches”. Lech (2005) distinguished among “financial techniques” and “qualitative methods” such as multi-criteria methods, “strategic analysis methods” and “probabilistic methods”. Berghout and Renkema (2001) categorised four predominant approaches, which they termed the “financial approach”, “multi-criteria approach”, “ratio approach” and “portfolio approach”.

Many more existing classifications are not cited here. Some overlaps between the various classifications are evident, however there are also distinct differences between them. This highlights the difficulty associated with establishing an agreed, coherent framework for evaluating ICT investments. A review of all available techniques cannot be exhaustive; new methods continue to be introduced while other techniques combine several existing tools (Bellini et al. 2014).

The majority of the approaches to the impact measurement focus on the input, output, outcomes and impact model (Fig. 1) where:

  • Inputs are the investments made in, or the resources required to, produce a product or develop/undertake an activity.

  • Outputs are the products or services provided (e.g. number of grids/networks created, papers published, events held, etc.)

  • Outcomes are the immediate changes resulting from an activity—these can be intentional or unintentional, positive or negative (e.g. employment, increased connectivity, etc.)

  • Impacts are the net difference made by an activity after the outputs interact with society and the economy (e.g. transformational research enabled by the project which would otherwise would not have occurred or occurred as fast enabling EC funded researchers to be world-leading).

Fig. 1
figure 1

Approach to impact measurement (Source: United Nations Development Programme 2002)

Full size image

Monitoring and ex-post evaluation exercise is a combination of qualitative, statistical and econometric techniques aimed at analysing the effects of the policy intervention.

The methodologies employed in ex-post evaluation include (Fahrenkroget et al., 2002):

  1. a.

    Statistical data analysis

    • Innovation Surveys: provide basic data to describe the innovation process, summarised using descriptive statistics.

    • Benchmarking allows to perform comparisons based on a relevant set of indicators across entities providing a reasoned explanation of their values.

  2. b.

    Modelling methodologies

    • Macro-econometric modelling and simulation approaches: allows to estimate the broader socio-economic impact of policy interventions.

    • Micro-econometric modelling: permits to study the effect of policy intervention at the level of individuals or firms. There are mechanisms for the counterfactual control by specifying a model which allows to estimate the effects on the outcome of the participant if the programme would have not taken place.

    • Productivity analysis: permits to assess the impact of R&D on productivity growth at different levels of data aggregation. This is particularly relevant to analyse the broader effects of R&D on the economy.

    • Control group approaches: allows to capture the effect of the programme on participants using statistically sophisticated techniques.

  3. c.

    Qualitative and semi-quantitative methodologies

    • Interviews and case studies: use direct observation of naturally occurring events to investigate behaviours in their indigenous social setting.

    • Cost-benefit analysis: allows to establish whether a programme or project is economically efficient by appraising all its economic and social effects. The approaches to quantify the socio-economic gains of a policy instrument include contingent valuation studies, simulating the existence of a market for a non-marketed good such as for example the capacity to produce a genome mapping in less time. These studies generally adopt questionnaires incorporating willingness to pay schemes to try to infer the price a certain public good is worth to the respondent. Other approaches include the use conjoint analysis in surveys to determine the price users place on the attributes or features of goods and quality adjusted hedonic pricing for new or improved goods

    • Expert Panels/Peer Review: measures scientific output relying on the perception scientists have of the scientific contributions made by other peers. Peer review is the most widely used method for the evaluation of the output of scientific research .

    • Network Analysis: allows to analyse the structure of co-operation relationships and the consequences for individual decisions’ on actions providing explanations for the observed behaviours by analysing their social connections into networks.

    • Foresight/Technology Assessment: used to identify potential mismatches in the strategic efficiency of projects and programmes.

Ex-post economic evaluation methodologies have proved a successful mechanism to:

  • Determine the efficiency and efficacy of the intervention (e.g. productivity studies).

  • Provide a quantitative estimation of the impact of the intervention (e.g. microeconomic evaluation studies).

  • Quantify the various dimensions in which returns should be considered within a defined framework.

  • Assess environmental sustainability and wealth issues (e.g. cost-benefit analysis), organisational impact (e.g. case studies, network analysis, innovation studies), strategic impact (e.g. foresight).

2.2 Evaluation of FP7

Before Horizon 2020, FP7 was longer and larger than previous Framework Programmes. It was funded with a budget of 55 billion euros , which accounts for an estimated 3% of total RTD expenditure in Europe or 25% of competitive funding. Over the 7 years duration of FP7, more than 139,000 research proposals were submitted, out of which 25,000 projects of highest quality were selected and received funding (about 17.9% of success rate). The budget breakdown among the 29,000 organizations participating in FP7 shows that 44% of the funding went to universities, 27% to research and technology organizations, 11% to large private companies and 13% to SMEs , while the public sector (3%) and civil society organizations (2%) played a minor role. The European Commission estimated quite substantial economic impacts despite the fact that FP7 only accounts for a small proportion of total RTD expenditure in Europe. According to European Commission (European Commission 2015), each euro of EC funding in FP7 generated more than 11 euros of estimated direct and indirect economic effects through innovations, new technologies and products. This happened through a short-term leverage effects and long-term multiplier effects. In total, the indirect economic effects of FP7 are estimated at approximately 500 billion euros over a period of 25 years, contributing with 20 billion euros per year to the growth of European GDP . These economic impacts generated also effects on employment by creating 1.3 million person-years within the projects funded (over a period of 10 years) and indirectly 4 million person-years over a period of 25 years. There is also evidence of positive impacts in terms of microeconomic effects with participating enterprises reporting innovative product developments , increased turnover, improved productivity and competitiveness. Modelling involves three main mechanisms: (1) the leverage effect that enables determination of total R&D expenditure; (2) the spill over of knowledge describes knowledge transfers to other sectors and other countries; and (3) the economic performance of knowledge.

It is really interesting understand how the above mentioned figures are built. The assumptions and estimates of increases in GDP and employment that are directly connected to FP7 spending are provided by Fougeyrollas et al. (2012) and Zagamé et al. (2012) in their assessments of the programme years 2012 and 2013. Their key indicators were used for a rough estimate of the economic impacts of the whole 7th Framework Programme. More in detail:

  • 50 billion euros (out of the total budget of 55 billion euros) of EC contribution to FP7 were taken as starting point for the estimation.

  • Both the studies mentioned above estimated the leverage effect at 0.74, indicating that for each euro the EC contributed to FP7 funded research , the organizations involved (such as universities, industries, SME, research organisations) contributed in average 0.74 euro. Consequently, based on the 50 billion euros mentioned above, the own contributions of other organizations to the funded projects can be estimated at 37 billion euros.

  • In addition, the total staff costs for developing and submitting more than 139,000 proposals (about 21,500 euros each) at an estimate of 3 billion euros were taken into account (6% of the EC funding).

  • The contribution of FP7 beneficiaries (matching funding) can be then estimated at 40 billion euros .

  • Consequently, the total investment into RTD caused by FP7 can therefore be estimated at approximately 90 billion euros.

  • For estimating the time scale of these investments, the duration of FP7 (2007–2013) plus the average project duration (3 years) was taken into account. Therefore, a total running time of 10 years and an annual RTD expenditure of 9 billion euros covered by EC funding and matching funding of participating organizations were calculated.

  • Both the studies mentioned above estimated a cumulative GDP multiplier of 6.5 for a period of 25 years. This consists of the total investment into RTD (90 billion euros) and the indirect economic effects (caused by new technologies, products and markets). Applying this estimation, the indirect economic effects can be estimated at approximately 500 billion euros giving an additional annual GDP of approximately 20 billion euros for the next 25 years.

  • Considering both—the leverage effect and the multiplier effect—each euro contributed by the EC to FP7 caused 11.7 euros of direct and indirect economic effects.

  • In order to translate these economic impacts into job effects, it was necessary to estimate the average annual staff costs of researchers (for the direct effects) and of employees in the industries effected by RTD (for the indirect effects). Based on estimated average annual staff costs for researchers of 70,000 euros , the report concludes that FP7 directly created 130,000 jobs in RTD over a period of 10 years (i.e. 1.3 million persons-years).

  • When estimating the indirect job effects, it has to be considered that new technologies in some cases create new jobs while in other cases they might lead to job losses as well. This has already been taken into account in the two studies mentioned above. By applying their results to FP7 approximately 160,000 additional jobs are indirectly caused by FP7 over a period of 25 years (i.e. 4 million person-years).

  • In order to make these figures comparable for our exercise we introduce here two new indicators that become the expected benchmark for the assessment of FP7 sub-domains:

    • Direct jobs in 10 years per million of EC investment: 25.71

    • Indirect jobs in 25 years per million of EC investment: 80

Table 1 summarises these results.

Table 1 Framework Programme 7 economic impact evaluation
Full size table

3 Impact Evaluation of the R&D Sub-programmes

3.1 Evaluating Micro-economic Effects, Growth and Jobs

In the last years we run several support actions devoted to the assessment of the socio-economic impact of FP7 project in different ICT domains such as Internet of services and software as a services, technologies for cultural heritage (DIGICult) and digital social innovation (CAPS). The applied assessment methodology was not intended to directly assess the macro-economic impact of research projects (e.g. on growth—GDP ) and it is conceived for evaluating, at a micro level, the potential benefits deriving to Consortium’s partners, to final users, and to the whole society from the implementation of the research and the exploitation of the resulting products (Passani et al. 2014). Nevertheless, through the approach briefly described in this paragraph we were able to gather for each domain the same type of information used for determining the macro economic impact of FP7.

According to Evalsed Guide 2012 (European Commission 2012), four main methodologies are currently used for socio-economic impact assessments :

  • Contingent evaluation : this is also called priority evaluation method. Its aim is to involve the general public in decisions. The method combines economic theories with social surveys to simulate market choices and to identify priorities of choices and preferences. This approach is useful for decision-making, especially with techniques using value judgements. The aspects of the current scenario are compared to an ideal scenario to assess public preferences. This method is usually applied in an environmental impact assessment , especially to evaluate non-marketable environmental goods.

  • Cost-Benefit Analysis (CBA): it is aimed at evaluating the net economic impact of a project involving public investments. A CBA is used to determine if project results are desirable and produce an impact on the society and on the economy by evaluating quantitatively monetary values. Compared to other accounting evaluation methods, a CBA considers externalities and shadow prices, allowing also the consideration of market distortions. Usually, a CBA is used in ex-ante evaluations for the selection of an investment of a project or in the ex-post evaluation in order to assess the economic impact of project activities (Bellini and Dulskaia 2017).

  • Cost-Effectiveness Analysis (CEA): it is a method for selecting the most effective alternative in terms of costs between projects with the same objective. A CEA is used for evaluating benefits that are not expressed in monetary values. It is not based on subjective judgements and it is not useful in case of projects with many different objectives (in this case a weighted CEA is used). The main objective of a CEA is to evaluate the effectiveness of a project, but it does not consider the efficiency. A CEA is mainly applied to projects in the health sector with a strict definition of the programme objectives. A CEA should be applied only to compare simple programmes providing the same kind of impact.

  • Multi-Criteria Analysis (MCA): it is used to evaluate non-monetary values of a project and to compare heterogeneous values. A MCA combines different decision-making techniques for assessing different impacts of the same project. It is aimed at identifying the opinion expressed by all stakeholders and end-users of a project in order to formulate recommendations and to identify best practices Bellini and Dulskaia, ibid.).

Considering these different methods and related perspectives, we decided then to ground our assessment methodologies on the Cost-Benefit Analysis (CBA) and on the Multi-Criteria Analysis (MCA). The results presented in the following paragraphs only refer to the information useful for comparing the economic impacts with the data provided in Sect. 2.2. Nevertheless, the analysis tried to cover the various interactions, feed-back loops and variable time spans between research , development , innovation activities and market penetration (see Sect. 2.1), as well as the increases in productivity, outputs of innovation processes, such as publications, patents and prototypes.

3.2 Evaluation of FP7 Software and Services Domain

The SEQUOIA project carried the assessment of 30 research projects (9 from Call 1 and 21 from Call 5) of the ICT FP7 co-financed by the European Commission in the domain of Internet of Services (IoS) and Software as a Service (SaaS) (Passani et al. 2014).

We apply now the same methodology and assumptions made by EC for the FP7 assessment to the figures gathered for the SEQUOIA projects’ sample.

  • The 120,144,852.00 euros of EC contribution to FP7 were taken as starting point for the estimation.

  • We applied the leverage effect at 0.74 and the own contributions of other organizations to the funded projects can be estimated at 88,907,190.48 euros.

  • The total staff costs for developing and submitting the proposal can be estimated in 7,208,691.12 euros (6% of the EC funding).

  • In total, the contribution of grantees can be estimated at 96,115,881.60 euros.

  • The total investment into ICT FP7 IoS and SaaS research can therefore be estimated at 216,260,733.60 euros.

  • For estimating the time span of these investments, we considered the actual duration of the projects assessed equal to 5 years. Therefore, an annual R&D expenditure of 43,252,146.72 euros covered by EC funding and own contributions of other organizations were calculated.

  • We applied the EC estimated multiplier effect of R&D of 6.5 for a period of 25 years to the total investment into R&D. The indirect economic effects can be then estimated at approximately 1.4 billion euros giving an additional annual GDP of approximately 56 million euros for the next 25 years.

  • When translating these economic impacts into job effects, we were able to compare the expected jobs created according to the EC estimation and the actual jobs (direct and indirect) created according to the projects feedback. Software & Services projects actually created 162 direct jobs per year while these jobs should have been more than 600 according to the EC metrics. The number of yearly indirect jobs created is about 37 (921 in 25 years).

  • We compare the Software & Services domain results with FP7 aggregated evaluation though the two new indicators:

    • Direct jobs in 10 years per million of EC investment: 13.48 (expected 25.71).

    • Indirect jobs in 25 years per million of EC investment: 7.67 (expected 80).

The economic impact of FP7 IoS and S&S projects is summarised in Table 2.

Table 2 FP7 ICT Internet of Services and Software as a Services domain economic impact evaluation
Full size table

3.3 Evaluation of FP7 DIGICult Domain

The MAXICULTURE support action performed the analysis of 19 projects in the domain of digital technologies applied to the cultural and creative domain (DIGICult) (Bellini et al. 2014).

We apply now the same methodology and assumptions made by the EC for the FP7 assessment to the figures gathered for the MAXICULTURE projects’ sample.

  • The 52,475,448 euros of EC contribution to FP7 were taken as starting point for the estimation.

  • We applied the leverage effect at 0.74 and the own contributions of other organizations to the funded projects can be estimated at 38,831,831.52 euros .

  • The total staff costs for developing and submitting the proposals can be estimated (6% of the EC funding) in 3,148,526.88 euros.

  • In total, the contribution of grantees can be estimated at 41,980,358.40 euros.

  • The total investment into ICT FP7 digital cultural heritage technologies research can therefore be estimated at 94,455,806.40 euros .

  • For estimating the time span of these investments, we considered the actual duration of the projects assessed equal to 4.5 years. Therefore, an annual R&D expenditure of 20,990,179.20 euros covered by EC funding and own contributions of other organizations were calculated.

  • We applied the EC estimated multiplier effect of R&D of 6.5 for a period of 25 years to the total investment into R&D. The indirect economic effects can be then estimated at approximately 614 million euros giving an additional annual GDP of approximately 24.5 million euros for the next 25 years.

  • When translating these economic impacts into job effects, we were able to compare the expected jobs created according to the EC estimation and the actual jobs (direct and indirect) created according to the projects feedback. DIGICult projects actually created 78 direct jobs while these jobs should have been about 300 according to the EC metrics. The indirect jobs created were only eight.

  • We compare the DIGICult domain results with FP7 aggregated evaluation though the two new indicators:

    • Direct jobs in 10 years per million of EC investment: 14.86 (expected 25.71).

    • Indirect jobs in 25 years per million of EC investment: 3.81 (expected 80).

The economic impact of FP7 DIGICult projects is summarised in Table 3.

Table 3 FP7 ICT DIGICULT domain economic impact evaluation
Full size table

3.4 Evaluation of FP7 CAPS Domain

The IA4SI support action performed the analysis of 11 project s in the domain of digital social innovation, namely Collective Awareness Platforms for Social innovation (CAPS) (Bellini et al. 2016).

We apply now the same methodology and assumptions made for the FP7 assessment to the figures gathered for the IA4SI projects’ sample.

  • The 17,204,988.00 euros of EC contribution to FP7 were taken as starting point for the estimation.

  • We applied the leverage effect at 0.74 and the own contributions of other organizations to the funded projects can be estimated at 12,731,691.12 euros.

  • The total staff costs for developing and submitting the proposals can be estimated (6% of the EC funding) in 1,032,299.28 euros .

  • In total, the contribution of grantees can be estimated at 13,763,990.40 euros.

  • The total investment into ICT FP7 digital social innovation research can therefore be estimated at 30,968,978.40 euros .

  • For estimating the time span of these investments, we considered the actual duration of the projects assessed equal to 3 years. Therefore, an annual R&D expenditure of 10,322,992.80 euros covered by EC funding and own contributions of other organizations were calculated.

  • We applied the EC estimated multiplier effect of R&D of 6.5 for a period of 25 years to the total investment into R&D. The indirect economic effects can be then estimated at approximately 201 million euros giving an additional annual GDP of approximately 8 million euros for the next 25 years.

  • When translating these economic impacts into job effects, we were able to compare the expected jobs created according to the EC estimation and the actual jobs (direct and indirect) created according to the projects feedback. CAPS projects created 28.6 direct jobs while these jobs should have been about 147 according to the EC metrics. The indirect jobs created were 31.

  • We compare the CAPS domain results with FP7 aggregated evaluation though the two new indicators:

    • Direct jobs in 10 years per million of EC investment: 14.86 (expected 25.71).

    • Indirect jobs in 25 years per million of EC investment: 3.81 (expected 80).

The economic impact of FP7 CAPS projects is summarised in Table 4.

Table 4 FP7 ICT CAPS domain economic impact evaluation
Full size table

4 From Micro to Macro Evaluation of Impact: Discussion of Results

From the analysis proposed in the previous paragraphs clearly emerges a substantial misalignment of derivated indicators with the official reports of European Commission. The 60 projects evaluated in the three ICT domains (about 190 million of EC investments) represent in our opinions a good sample also considering that ICT research accounted for about 8.3 billion of FP7 budget.

The indicator Direct jobs per million invested shows a substantial alignment among the three ICT domains with a mean of 14.99 that might correctly reflect the actual need for FP7 participants of hiring people for the projects’ development . Anyway, this result is quite far from the EC data derived estimation of 25.71.

The indicator Indirect jobs per million invested shows a certain volatility across the different ICT domains. It is known that this kind of data is strongly influenced by the assumptions made for its estimation and that might be influenced by sector specificities. In our case the research results are producing impact on employment in very different labour sectors—software, cultural heritage and preservation, social innovation—that are characterised by different labour and salary dynamics (Bellini et al. 2011, 2016). Nevertheless, the mean (18.44) and the maximum value obtained for this indicator (45.05) are really far from the EC data derived estimation of 80 jobs.

Table 5 summarises these results.

Table 5 FP7—cross domain economic impact evaluation
Full size table

5 Conclusions

We are perfectly aware that the evaluation of a research and investment programme is a very difficult exercise that implies strong assumption and it is still very difficult to identify the best evaluation approach especially when the objective is to determine the macro economic impact on GDP and employment .

EC experts are aware that the main limitations of their assessments include the time frame of only one FP7 funding year per study, the assumption that each year’s budget is allocated in form of a “one-off shock”, as well as model limitations.

With this paper, we want to contribute to the debate on the micro foundations of these models gaining from the experience accumulated during our projects developed to evaluation of some FPT ICT activities. First of all, the leverage effect of 0.74 proposed by Fougeyrollas and Zagamé (ibid.) and adopted by the European Commission experts looks quite optimistic also considering that for many beneficiaries such as universities, research centres and SMEs is already very difficult to cover the requested matching funding when the instrument does not allow the full coverage of R&D&I expensed. Another assumption used by the EC evaluation is that entire investment (EU plus participants’ contribution) is allocated on the personnel cost; this item surely represents a relevant part of the project budget but also other cost items and indirect costs are included in the eligible research investment.

Another assumption refers to the duration of indirect benefits estimated in 25 years: we are now observing very short technology life cycles to which are connected specialised jobs with short life cycles too.

All these parameters sound rather optimistic and influence positively the 11.7 direct/indirect effect the FP7 investment estimated by the European Commission.

Finally, the average cost of researchers used of 70,000 euros per year looks rather high while available statistics (Eurostat 2017) suggest a value around 40,000 euros per year. In this case, the reduction of the researchers’ cost should enable a positive effect on employment  accounts.

In conclusion, we believe that a structured approach to evaluation and data gathering is strongly needed in order to have suitable standards metrics for the assessing the EU R&D investment programmes. A clear and standard evaluation framework need to be identified and policy makers should take the lead in this process also in order to define clear roles and responsibilities in data collection and data provision (i.e. Eurostat, project partners etc.).

Questions and Activities

  1. 1.

    Which are the seven flagship initiatives of the Europe 2020 strategy?

  2. 2.

    Which is the target indicator in terms of R&D&I expenditure?

  3. 3.

    What are the main theoretical approaches with reference to the resources allocation for innovation?

  4. 4.

    Which are the main steps followed by impact measurement techniques?

  5. 5.

    Which are the methodologies adopted for ex-post evaluations?

  6. 6.

    Summarise the main characteristics of Cost-Benefit Analysis (CBA), Cost-Effectiveness Analysis (CEA) and Multi-Criteria Analysis (MCA).

  7. 7.

    Discuss the figures shown in tables and try to provide alternative indicators, metrics and measurements.

  8. 8.

    Which are EU institutions involved in the evaluation of R&D&I programmes?