Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

10.1 Introduction

The overall purpose of ForSTI is to provide input into policy and strategy planning and to mobilise collective strategic actions. In the Intervention phase we move on from the issue of formulating recommendations, to experience in following these through in the form of concrete action to implement structural and behavioural transformations. Actions suggested at this phase aim to give messages on the first and most immediate interventions to the existing systems. Operational level questions are asked for actions such as: ‘what and how’, ‘where and how’ and ‘who and how’. The actions for change are determined by considering the following capabilities of the system under investigation: (1) Adapting; (2) Influencing and shaping its context; (3) Finding a new milieu or modelling itself virtuously in its context; and (4) Adding value to the viability and development of wider wholes in which it is embedded. Action plans, Operational plans, Priority lists can be among the outputs produced at this phase, in addition to the outcomes achieved through ForSTI, such as networking, mutual learning and collective visioning, which are key enablers for follow up actions upon the completion of the exercise.

During this chapter, we will also consider the evaluation activities. ForSTI process requires substantive investments, often through public funding, and implies considerable costs in terms of time and expertise invested. If impacts of ForSTI cannot be made clear, the commitment for investing resources will decrease, and as a result the activity will be discontinued. Therefore, an Impact phase is added to the process, which is concerned with the review, evaluation and renewal of the ForSTI exercise. This phase will examine the impacts during the process (e.g. production of baseline reports, articulation of appraisals of future prospects and alternatives (“visions”), and building new linkages), immediately after (e.g. new integrated projects and programmes) and sometime later (e.g. innovation impacts and new working communities).

10.2 Outputs and Outcomes of ForSTI

In the first place, it is useful to differentiate between the “outputs” and the “outcomes” of ForSTI. The first refers to what the exercise produces at the end of the process, and the second to what happens as a result of carrying out the exercise and especially of carrying out actions or implementing policies that are adopted.

In general, the outputs of ForSTI may be addressed to one or more specific target groups, which typically involves the sponsor of ForSTI, whereas the outcomes are likely to address different audiences. Therefore, when starting ForSTI, the target groups should be defined (i.e. the “users”, who should receive and benefited from outputs). This is a key reason for ensuring that these prospective users have an appropriate level of involvement in the ForSTI process from the outset. Moreover, they can help to define the targeted outcomes that should be achieved for the relevant target group as well as for broader society. Table 10.1 outlines some of the types of outputs and outcomes that can be expected through ForSTI activities.

Table 10.1 Some types of outputs and outcomes from ForSTI
Full size table

Different outcomes are produced at the different phases of the exercise, both immediately after and sometime later (see Table 10.2).

Table 10.2 Various outcomes from ForSTI at different times
Full size table

ForSTI exercises produce ‘action plans’ and ‘demonstrators’. The action plans indicate what needs to be done, by whom and when. They should also indicate verifiable measures and indicators for monitoring progress to assess the degree of success attained. Demonstrators incorporate a series of action points and identify sets of actions that can be pressed simultaneously on the same agency. The main danger regarding demonstrators is that they may give an impression that the ForSTI activity emphasises too much focus on a certain project.

It is important to have a clear notion of the types of outputs and outcomes to be expected from ForSTI. This should be done right from the beginning at the Initiation stage, and should be communicated with the sponsors, clients and stakeholders of ForSTI. Some changes may happen in the type and quantity of outputs and outcomes, however, ForSTI should strive for delivering actionable recommendations and creating impact beyond formal reports to bring different actors of the system to work together as the agents of change.

10.3 Reporting on and Disseminating the ForSTI Process and Findings

Typically, a ForSTI exercise will be obliged to present a report to its sponsor, and most often there will be a number of publications, of different forms, aimed at wider audiences, and quite possibly different outputs for specific types of stakeholder. In this, ForSTI is not so different from many other types of project that involve working with considerable volumes of information and drawing on high levels of expertise, to inform debate and decisions. Much experience has been developed around science communicationFootnote 1—and especially in biomedical fields, where there has been much attention paid to provide resources that can enable patients and carers to make informed consent to medical treatment or clinical trials.Footnote 2 Plans for dissemination and interaction around the ForSTI outcomes should be made early on in the exercise, rather than be made only after results have been fully articulated.

While the project report may well need to include much detail of the management and the methodological decisions (and difficulties) of the exercise, the important messages from the exercise can be spelled out in more succinct ways, and illustrated by means of such communication devices as cartoons and other images. It is important to allow users access to information that will enable them to understand just how the results may be influenced by the methods and other choices made in the course of the exercise,Footnote 3 but this needs to be balanced against the need to present critical results in a clear and informative way. The Executive Summary of a report can play a significant role here.

Various types of conventional and online publication can be employed, and tools such as press releases and online tweets used to alert media and other users to their existence. As well as a broad overview document, various summaries of critical issues, and of implications for specific stakeholder groups, can be prepared. It is often very helpful to enlist professional journalists or other skilled communicators into these tasks. Material can be dramatised with storytelling, graphics, and even short videos—again drawing on skilled professionals for assistance in ensuring quality products. Video and audio interviews with ForSTI participants and practitioners can also be helpful ways of communicating key messages. Material may be disseminated through both public and more specialised news media, for example magazines and journals aimed at specific stakeholder groups. (Efforts at building bridges to relevant journalists are liable to be well-rewarded). This can mean creating informed intermediaries who will be able to grasp the significance of developments in an ongoing exercise and to separate key messages from those that may be superficially more newsworthy. It may also help offset the danger that media interest will only be parked by controversy and criticism (for example, running stories only when someone feels slighted for not having their specific views or lines of work endorsed by the ForSTI priorities).

In addition to traditional publication, there are various ways in which further debate can be facilitated. Comments can be allowed on webpages (these will typically need moderation, to exclude spam and trolls). Blogs can be setup to post updates and news items of relevance. Consultation processes can be used, with stakeholders encouraged to generate extensive reactions to the exercise. In some cases, modelling tools may be made available for users to generate their own scenarios or assessments, guidelines may be presented for conducting scenario and other workshops, material may be produced for schools and other bodies to use to seed their discussions or even to use as a basis for generating plays and poems. Online communities can be fostered to take the exercise further. We anticipate increasing accumulation of practice in these fields.

It is often the case that the biggest impacts of an exercise are achieved through direct contact between ForSTI and individual stakeholders, in the course of dissemination and consultation as well as when active engagement in the project is involved. In addition to straightforward presentations of key issues, it is possible to use the ForSTI process and its results as a starting point for workshops in which, for example, residents of a town or members of an innovative cluster can explore the implications of various scenarios for their own practices and future options. Such an approach had been employed in various EU ForSTI activities over recent decades.Footnote 4 Both conventional seminars/lectures and more interactive workshops can be employed at various regional and sectoral levels, and aimed at various stakeholder groups, not least young people, effecting much deeper understanding and engagement than might otherwise be the case.

It is important not to let such “formal” outputs displace more informal means of communication. The capturing of results in publications is sensible but more informal outputs in the form of improved networks and the embodiment of new knowledge in people’s practices and organisations’ approaches to issues are critical. These may be harder to identify and quantify than documentation, but represent achievements that may have more long-lasting and substantial effects.

10.4 Influencing Outcomes: Strategies, Actions and Stakeholders

ForSTI is not an ivory tower exercise. It is intended to achieve impacts upon the behaviour of sponsoring and/or other stakeholders—for example leading to changes in public STI policy or that in other fields. In effecting this, ForSTI may work in various ways, which may take four basic forms:

  1. 1.

    ForSTI typically produces texts and other symbols bearing narratives that can frame how people approach issues. This production of symbols is an important part of any exercise: it can result directly in policy advice

  2. 2.

    ForSTI can highlight the need to develop and coordinate policies and strategies in fields connected with its focal object, not least by assessments of priorities for resource allocation

  3. 3.

    ForSTI can play a role in coalition building with its narratives and insights facilitating actor alignment and mobilisation around a new or transformed agenda

  4. 4.

    ForSTI as a social ‘technology’ can broaden the participation of stakeholders in discussing the long-term issues, and helping to establish new networks and communities around these (Keenan 2005).

A more extensive list of possible impacts of ForSTI is provided by the EFMN project—see Table 10.3.

Table 10.3 Reported impacts from the practice of Foresight
Full size table

We discuss the evaluation of ForSTI efforts below, but as this discussion of different functions and modes of influence suggests, the complexities of the process can make simple evaluation difficult. It is widely recognised that it is difficult to attribute decisions and directions uniquely to ForSTI. Often other policy interventions are being tried out, and the thinking and experience of decision makers are evolving under various influences, so a shift in behaviour may or may not be largely attributed to a ForSTI effort. The impact of ForSTI is liable to involve interaction of ForSTI processes and their outputs with the broader strategic behaviour of policy and economic actors (and with the other influences on this behaviour). One practical implication of this is the need for ForSTI exercises to be designed with interfaces with existing strategic processes in mind. This may mean ensuring that ForSTI results are available before, rather than after, the formulation and publication of major STI policy measures, for example (In any case, aligning the ForSTI phases with the policy cycle is rather important).

Keenan (2005) made this point about the policy cycle, also stressing that information needs to be presented in such a way that policy/strategy mechanisms can receive and absorb it. In addition to the issue of timing, it may be vital to consider how far recommendations should take into account available funding, and even the capacity for reform of existing procedures. There may well be a need to introduce highly disruptive thinking, but this may require some accommodation with the pragmatics of policy (in the event that sufficiently powerful coalitions cannot be established to achieve transformative change).Footnote 5

One of the main long-term outcomes of ForSTI exercises goes beyond informing action around the focal object of the exercises—it involves influencing attitudes to ForSTI itself, so that hopefully each exercise can contribute to the fostering of a “ForSTI culture”. Given the importance of STI issues, it is important that a wide range of economic organisations and social actors should recognise the relevance of longer-term perspectives, and can themselves initiate relevant ForSTI processes when needed to guide action.

Panel members can play significant roles in embedding Foresight in their own organisations. They can also contribute to the development of ForSTI capabilities by liaising with other organisations to see how far they are adopting the messages of the ForSTI exercise. For example, panel members could share out responsibilities for monitoring the implementation of action plans, etc., by relevant parts of national or local government; they can provide briefings and inputs of other kinds. It can be very demanding of Panel members, especially unpaid ones, to maintain such a level of activity, of course, and they may benefit from the support of more “centralised” activities of one sort or another. This is one reason why a Foresight Unit of some sort can be a valuable instrument—this may be situated within national government, an executive agency, or even be based in Universities or public laboratories.

The following section will address the issue of ForSTI evaluation, which involves assessing the impact of ForSTI and determining what lessons are to be drawn from the conduct and management of the exercise.

10.5 From Intervention to Impact: Evaluation of Impacts

The impacts of ForSTI should be kept in mind from the beginning of the process, and the methodology should be designed to achieve those impacts. An effective communication strategy is essential during and after the ForSTI process for assisting the participants and target audience in making sense of the results. Impacts of ForSTI are measured through an evaluation exercise, which is commonly conducted based on three criteria including (1) appropriateness of objectives and methodology; (2) efficiency of implementation with a focus on management and organisational processes, and appropriate use of funds; and (3) impact and effectiveness through the recognition of the results, creation of a ForSTI culture and new combinations of stakeholders and networks.

Evaluation helps to discover whether, or to what extent, the exercise has achieved its desired outcomes—where it may have fallen behind expectations, and where it may have exceeded them. Information on the achievements of the activity can be used for other purposes (such as dissemination of results and decisions about renewal of the activity). This information is useful for those participating in the activity as well as for those managing the exercise. Evaluation provides a good opportunity for participants to express their views about what worked and what did not, and to learn about:

  • the appropriateness of the original objectives, and the degree to which these were adequately formulated and communicated to those involved

  • the management of the exercises, such as whether the activities might have been performed more efficiently and effectively with a different organisational structure and methodology

  • whether or not the ForSTI methods selected were appropriate to the objectives and tasks

  • what were the barriers to ForSTI and in which ways should these be tackled

Although the Impact phase can be considered as the final phase of the process, there is a strong learning element involved in this process, which determines how to design and implement and better ForSTI exercise next time round. Thus, it can also be considered as a beginning of the next cycle of ForSTI.

10.5.1 Tools for ForSTI Evaluation

ForSTI is a very resource-intensive activity itself and produce recommendations for the allocation of even bigger resources, for instance by identifying key STI priority areas to be invested. Therefore, it is reasonable to expect that the process itself and its outputs should be subject to evaluation in a systematic way. Georghiou and Keenan (2006) propose three basic tests to consider when evaluating ForSTI:

  • Accountability—with questions such as whether the activity was efficiently conducted and proper use made of the funds

  • Justification—with questions such as whether the effects of ForSTI justify its continuation and extension

  • Learning—asking how can ForSTI be performed better, and tailored to particular circumstances

The evaluation activity may aim to consider either one or a combination of these aspects. However, it must be noted that the evaluation of ForSTI is not a straightforward process. It has its own challenges. Some challenges are associated to the nature of the evaluation activity in general, and some others are more specific to the evaluation of the Foresight activity. It is certainly a good idea to be aware of them before embarking upon a ForSTI evaluation.

Regarding the evaluation activities, OECD (2006) lists four generic challenges:

  1. 1.

    Timing: Have expected effects come about already?

  2. 2.

    Attribution: Can we confidently assign outcomes to the intervention being evaluated? Related challenge of so-called ‘project fallacy’ effect.

  3. 3.

    Appropriability: Where should we look for effects?

  4. 4.

    Inequality: Where effects tend to be skewed to a few ‘blockbuster’ projects.

In addition, there are the following challenges associated more specifically with the ForSTI activities (Barre and Keenan 2008):

  • Non-standard approaches to ForSTI in most places

  • Wide-ranging and often vague objectives

  • Difficulty of assessing intangible objectives associated with ForSTI

  • The complexity of cause-effect relationships

  • The systemic and distributed nature of ForSTI

  • ForSTI’s embeddedness

  • ForSTI’s dynamism and flexibility

  • Experimental nature of ForSTI

Following the understanding of challenges, we now look at the criteria used for the evaluation of ForSTI. Typically, a set of criteria is identified for the purpose of evaluation. Georghiou and Keenan (2004) list three criteria:

  1. 1.

    Appropriateness

    1. (a)

      Appropriateness of objectives

    2. (b)

      Appropriateness of approach in view of the objectives

    3. (c)

      Role of benchmarking

  2. 2.

    Efficiency and implementation

    1. (a)

      Focus on management and organisational processes

  3. 3.

    Impact and effectiveness

    1. (a)

      To ‘measure’ impacts, indicators will be need to be developed, based upon the Programme’s ‘theories of action’ and their representation in ‘logic charts’

Logic charts have been used in order to assess all these criteria in an integrated way. In ForSTI, they are used as diagrammatic representations that display the hierarchical relationships between higher and lower levels of objectives, activities and expected outputs and effects with logical links and assumptions. Figure 10.1 illustrates the logic chart, which was used as a design framework for the evaluation of the third cycle of the UK ForSTI programme.

Fig. 10.1
figure 1

Logic chart for the evaluation of the third cycle of the UK ForSTI programme. Source: Georghiou et al. (2006)

Full size image

Three different layers are seen in the example logic chart above. The first two layers are concerned with the objectives. The overall objectives are the ones that sponsors hope to contribute to by funding research in this area. Information on overall objectives can be found in policy statements. At the national level, this can be the improvement of the performance and use of science and engineering as in the case of the UK. Programme objectives are more specific and are related with the ForSTI activity itself. These can be classified into different groups, for instance, strategic (e.g. wealth creation), structural (e.g. changes in the relations between industry and academic actors), and technological (e.g. future technological opportunities to address societal challenges).

The next layer in the logic chart is concerned with the programme activities. These represent actions that are implemented to achieve the aforementioned objectives, e.g. projects/themes to be focused upon, application of the ForSTI methods and engagement with policy makers and stakeholders.

The third group of layers is concerned with the effects of ForSTI. These may be observed in different time frames. Some of them are immediate effects, which are observed during the course of the ForSTI activity. These may be in the form of collaborative programmes, which may bring together previously separate groups like experts and stakeholder groups and enable mutual learning and collective visioning. Intermediate effects are observed during or shortly after the completion of the programme. The aforementioned ‘outputs’ from the ForSTI programmes usually fall into this category. Examples may include future visions, recommendations for actions, and on the ‘outcome’ side, it can be the establishment of action networks. Finally, the ultimate effects refer to the effects that may be expected sometime after the completion of the programme. The effects on research agendas, funders, government policies, and generation of a ForSTI culture can be considered as examples of ultimate effects. The timescale for these will vary considerably according to the type of programme.

Thus, the logic charts portray relationships between expected outputs and effects of ForSTI as well as the links between ForSTI and broader policy objectives. They allow for the consistency of a programme with higher goals to be assessed and enable their contribution to these to be evaluated. Logic charts also help to identify expected outputs and effects, thereby allowing (1) the configuration of appropriate indicators to measure success-relevant for data collection during the evaluation and (2) the analysis of success of the programme vis-à-vis expectations.

Production of a logic chart itself is a process. Necessary information such as policy statements and programme documentation related to ForSTI should be at hand. Discussions should be undertaken with the programme managers. Production of an agreed logic chart may require a few iterations. Below we will take a closer look at the evaluation process.

10.5.2 The Evaluation Process

An evaluation process begins with the definition of the scope and purpose of the activity. The scope of the evaluation usually covers the begging of the ForSTI activity from the initiation, i.e. with a decision to commit resources, establish a Project Team, and set up a Steering Committee. The programme typically ends with the publication of a final report, or list of priorities, which may mark an end for the programme—though further dissemination and implementation activities may follow. Three types of evaluation activities may be undertaken before, during and after this process, which are:

  1. 1.

    Ex-ante evaluation

  2. 2.

    Real-time evaluation

  3. 3.

    Ex-post evaluation

Ex-ante evaluation takes place before the launch of a ForSTI activity. This type of evaluation helps to make decisions whether ForSTI should be undertaken or not, and if yes, to make decisions about the programme by simulating it. Real-time evaluation is undertaken during the course of the ForSTI activity. Progress and problems related to the execution of the activity are considered at this stage, and necessary revisions are made to improve the process and outcomes. Finally, Ex-post evaluation is conducted following the completion of the programme. Ex-post evaluation can also be undertaken in a few years upon the completion of ForSTI to consider the impacts of the implementation of the outputs and outcomes. The timing issue is also linked to the type of question being asked. If a linear or sequential view of ForSTI is taken, issues related to process are best investigated while the activity is still under way. However, many outputs and outcomes will not be clearly visible at this time and will need to be considered during the ex-post evaluation stage.

Evaluation of ForSTI may be concerned with the process and/or outputs and outcomes (Georghiou and Keenan 2008). Process evaluation covers the organisation and management of ForSTI. The following are examples of questions, which may be asked at this stage:

  • Were the ‘right’ people involved?

  • Did expert panels receive adequate support?

  • Was the exercise adequately linked to decision-making centres?

Questions related to the appropriateness and efficiency of methods used can also be asked at this stage, such as:

  • Should a Delphi have been used?

  • Were scenario workshops properly facilitated?

These questions may be asked in the real-time evaluation or immediately after the completion of the activity.

The evaluation of outputs may involve the numbers of participants, reports disseminated, meetings held, website hits etc., and thus measures the activity itself, not its effects. Effects should not be shadowed by the numbers. For instance, the new networks established may vary, for instance, in their novelty, size and significance.

However, in order to prove real impact, ForSTI needs to demonstrate behavioural additionality, meaning that the activity and its outputs and outcomes should be embedded in the behavioural routines of the organisations concerned (OECD 2006). Evaluating the impact of ForSTI output on the strategic behaviour of policy and economic actors is one of the challenging, but a crucial task. The interaction between ForSTI outputs and the strategic behaviour of policy and economic actors should be investigated as a part of ForSTI evaluation. The aim is to understand whether ForSTI activity is accountable for the resources allocated to support it. Following sorts of questions can be asked in order to evaluate the additionality:

  • Would ForSTI have happened without the policy intervention?

  • Is ForSTI done differently/better because of the policy intervention?

  • Are the resulting actions better because of ForSTI?

  • Have persistent changes been achieved (e.g. establishing a ForSTI culture)?

As the implementation of ForSTI outputs is stretched over time, the answer lies in a longer timescale and is affected by the creativity or commitment, which may be coming from other sources. When making assessments about what effects can be attributed to the ForSTI activity, there is a need for an understanding that it is only one of several influences upon public policy, or the strategy of firms. Figure 10.2 illustrates this non-linear relationship between ForSTI and its implementation context.

Fig. 10.2
figure 2

ForSTI in a non-linear relationship with its implementation context. Source: Georghiou (1996)

Full size image

It may take a longer term to observe the accuracy of the future appraisals in ForSTI. This may not be a problem with short time horizon, say in a 5-year critical technology exercise. However, if ForSTI is looking into a longer term futures, say into next 15–20 years, it may be more difficult to see the expected developments come into place. In long running ForSTI programmes, like the Japanese one with a continuation from the early 1970s, it is possible to make assessments of earlier rounds. In order to test the accuracy of the appraisals in more recent exercises, various indicators can be formulated, which may indicate the progress towards the foreseen developments. For instance, scientific evidence can be used by analysing publications and patents, which may lead to an eventual technology, product or market foreseen through the ForSTI activity. Other evidence from policy/strategy documents, media or other material can be considered as a source of evidence to track transformations and what sorts of signals they may give about future developments.

Some normative issues should also be mentioned regarding evaluation. ForSTI information should be presented in a way that it can be received, interpreted and absorbed by policy mechanisms. Therefore, factors associated to the successful adoption of ForSTI, including timing, availability of funding, and capacity and commitment to change and reform should be considered when evaluating the success of ForSTI.

10.5.3 Some Evaluation Experiences

Table 10.4 shows some experiences of ForSTI evaluation, and serves mainly to emphasise that diverse range of approaches exist for evaluation and there is currently not a consistent and comparable approach emerged. Two of the countries mentioned are discussed further as case studies.

Table 10.4 Recent ForSTI evaluation activities
Full size table

Case l: UK ForSTI Evaluation Experiences

The UK ForSTI programme has been accompanied by a fairly large number of evaluation activities, many of which were examining just one or other feature of the programme. (Table 16.2 in Georghiou et al. 2008, lists many of thee.) While most of these were not confidential (indeed several were very widely circulated), most were never published in any formal sense. One which is publicly available, though not all that easily accessible, is a PhD thesis (Keenan 2000), which examined the whole of the first cycle of the programme, but focused in detail on the network-building activities of just a couple of panels. Georghiou et al. (2006) undertook the first formal evaluation of UK ForSTI, examining the various projects that had been undertaken in the first few years of the third cycle of the UK TFP; this evaluation was until recent cuts downloadable from the UK Foresight website. (One-year progress and impact reports on past projects are available in the online archive, however.Footnote 6) The Parliamentary Office of Science and Technology (POST 1997) had earlier published a less systematic but nevertheless insightful and impactful review of the first cycle of UK ForSTI. There had been numerous studies conducted for internal use by the ForSTI team, including, in the first cycle a survey of panel members, and later an effort to establish performance indicators on the basis on panel opinions. Several other efforts to develop indicator and evaluation frameworks had been undertaken, but more lasting impact was achieved by an urgent review of the second cycle of the programme, based mainly on interviews with stakeholders and experts in 2002. This confirmed (and perhaps was designed to confirm) views from senior policymakers that ForSTI had failed to build on the success of the first cycle; it led to the complete restructuring of the programme, though it is not clear just how the rather successful new model put in place for the third cycle was determined.

The UK case is one in which there was little thought about evaluation of ForSTI at the outset, and the result was a number of small evaluation efforts—many of which were, according to Georghiou et al. (2008, p. 389) “at a sub-critical level, or else relied very heavily on anecdotal and potentially prejudiced evidence”. Around the turn of the century, there was growing government enthusiasm for applying performance indicators to monitor and evaluate public sector activity (part of the “New Public Management” drive.). This is particularly problematic for ForSTI activities, which are aimed at long-term impacts and where short-term output indicators may fail to capture important process outcomes. Several efforts were made to establish ways of assessing ForSTI in terms of key indicators, but these confront a further difficulty. This is that ForSTI, as we have seen repeatedly throughout the present book, involves much unpaid involvement by engaged stakeholders (especially, but not only, panel members), rather than public servants; asking these “volunteers” to do further work that effectively represents a monitoring of their own performance is, as they say, “a big ask”. Georghiou et al. (2008) report an effort to develop a “softer” evaluation approach was adopted that separated process from impact. Process information could be compiled by civil servants and the ForSTI team; much impact information could be sourced from participants who would have a clear interest in how effective their efforts were proving. Five main stakeholder groups were identified (the research community; those involved in education, training and public understanding of science; industry and commerce; the voluntary sector; and government itself), and A set of key indicators developed, as displayed in Fig. 10.3.

Fig. 10.3
figure 3

UK second cycle evaluation framework. Source: Georghiou (2003)

Full size image

Georghiou et al. (2006), as noted, undertook a fairly wide-ranging evaluation of the third cycle of the UK ForSTI Programme, launched in 2002. The second cycle had been terminated abruptly, following discontent with its achievements; there was much interest in seeing how far the new model was faring. The new model mainly involved a succession of projects investigating specific areas of STI in some detail, as compared to an effort to overview developments across all sectors and technologies. Since the projects were typically designed to take 2 years to complete, by 2006 a number of them had already been completed. The evaluation study considered the impact of the programme and of its constituent projects (several of them had been completed by that time), and to examine its cost-effectiveness (was it value for money? Did it provide additionality in the context of STI policy initiatives?) and its management (was the organisations of ForSTI adequate and running well?).

In this evaluation, semi-structured interviews were the key instrument for collecting information during on the ForSTI exercise, allowing for in-depth exploration of the issues arising. (It was not believed that this could be effectively achieved via survey methods, not least because of the diversity of the projects being examined.Footnote 7). 8 members of the ForSTI team and 28 stakeholders (e.g. the policymakers that the project was meant to influence) were interviewed.

The evaluation study was broadly favourable as to the value of the TFP. The Stakeholders who were interviewed were overwhelmingly positive about the achievements of and need for the Programme. Outputs like reports and meetings were of high quality, and new valuable combinations of knowledge were presented in accessible ways. Further, the evaluation concluded that the objectives of identifying both ways in which STI could address future challenges for society, and potential opportunities for application of STI. National policies and programmes had been influenced, in some cases very substantially, and in the one case where influence was limited the reasons were largely beyond the control of the ForSTI team, and more to do with changes in policymaking personnel and reluctance to grasp some difficult nettles. (In retrospect, it could be argued that the project, on drugs, was extremely accurate in terms of pinpointing emerging problems, but that this is a focal object that policymakers are loath to confront.) There was evidence for behavioural additionality, in that beyond the impact of the new knowledge that the project had injected into the policymaking process, some interviewees reported that the policymaking process itself had been changed: more attention was given to longer-term issues and ForSTI-type inputs A further achievement was that the projects constituted interdisciplinary platforms bringing together and establishing dialogue among a range of stakeholders who were not being mobilised through conventional departmental or basic research funding efforts.

Case 2: Evaluation of German FUTUR Initiative

The second case concerns a recent evaluation, that of the German FUTUR initiative. This was commissioned by the responsible ministry, BMBF, and was largely a process evaluation, focusing upon:

  • The objectives of FUTUR, which were assumed to summarize the central assumptions upon which the exercise is based

  • The different instruments and methods with regard to their effectiveness, efficiency and interplay

  • The process in general

The evaluation approach was developed by ISI-Fraunhofer and involved formulating the underlying assumptions and hypotheses that underpinned the ideals and conduct of FUTUR. These hypotheses were then “tested” through their operationalisation into questions that could be detailed in surveys and interview protocols.

Following a survey of participants a document was constructed to support an International Panel of Foresight Evaluation Experts. This panel held a one-day hearing with interviews and the Chair consulted with the Ministry as a user at the most senior level before producing the evaluation report. The limitations of this exercise were: too little time and resources available, and the fact that the exercise was conducted too early to pick up outcomes. However, several process-related recommendations were made and an impetus was gained for the continuation and improvement of the activity. A key finding was that the participants felt disconnected from the implementation process and, to a lesser extent, the programme managers responsible for implementation lacked a sense of ownership of FUTUR.

10.6 Conclusions

This chapter has discussed the output and outcomes of the ForSTI activity. It has considered what these might be, and how the desired outcomes are liable to be very much influenced by the outputs that are designed into the process. One major outcome is the extent to which the exercise has contributed to building a ForSTI culture that persists beyond the original activity.

We have stressed that outputs as understood in terms of formal publications and presentations are not, however, the only influence upon outcomes. In addition to changes in the context within which ForSTI is operating, the outcomes may be very dependent on the processes of the exercise, the ways in which it has been implemented. In particular, the range of stakeholders involved, and the depth of their engagement in various activities, can have a huge bearing on the reception of results, and the extent to which the proposed actions are adequately understood and put into practice.

Evaluation is important for assessing the actual and envisaged outcomes of ForSTI, and for learning lessons concerning the success and limitations of an exercise. Because ForSTI is context-dependent, and an exercise will ideally be closely tailored to its context, it is important that evaluation takes into account the period in which ForSTI emerged and its interaction with other elements of the system.