Keywords

1 Overview of the EVA Survtool

The tool has been organized into three main sections to capture all the elements critical to an evaluation process and highlighted by the experts during the iterative development process of the tool (Fig. 4.1) [1, 2]:

  • Section (1): a general introduction to the tool and essential information on evaluation concepts, including evaluation attributes and economic methods to promote the understanding of the evaluation process and economic evaluation

  • Section (2): guidance on how to define an evaluation plan based on steps 1 and 2 with data entry on the evaluation context and the evaluation question and steps 3 and 4 where the tool facilitates the selection of relevant evaluation attributes and assessment methods (including economic analysis)

  • Section (3): guidance on how to perform the evaluation and how to report the outputs of the evaluation to decision makers

Fig. 4.1
figure 1

General organization of the EVA tool. Section (1): general introduction to evaluation concepts and economic methods; Section (2): guidance on how to define an evaluation plan; and Section (3): guidance on how to perform the evaluation and how to report the outputs of the evaluation to decision makers. (From [3])

Full size image

In the introduction section, the tool provides essential information on its organization and on how it was developed. A manual is also available for download to facilitate the use of the tool. This section further provides general information on evaluation concepts, evaluation attributes, and economic evaluation methods to promote the understanding of the evaluation process and economic evaluation. An evaluation process should encompass three main aspects: planning, implementation, and reporting (Fig. 4.1). This should promote the implementation of “better” evaluation and therefore the quality of the data generated by those evaluations, along with the use of economic evaluation in the decision-making process.

1.1 Step 1: Describing the Surveillance System

Survtool takes the user step by step into describing the surveillance system and component under evaluation (https://survtools.org/wiki/surveillance-design-framework/doku.php?id=1-surveillance-system).

The first step in this process is the characterization of the surveillance system that you can do in the SURVEILLANCE SYSTEM tab. Here you can describe a NEW SURVEILLANCE SYSTEM or LIST EXISTING SYSTEMS that you have previously described. Within the tool, you need to either select a surveillance system from the drop-down list in the top right-hand corner of the screen or create a new surveillance system before you can use the design or evaluation tools. The name of the system that is currently active will be displayed in the top right-hand corner of the screen.

A surveillance system is defined here as a collection of various surveillance components that are all aimed at “describing health-hazard occurrence and contributing to the planning, implementation, and evaluation of risk-mitigation actions” [4] for a specific health hazard and in a defined region. A surveillance system is therefore characterized by (1) one defined hazard that is targeted by surveillance (a disease or another health threat); (2) the objective of the surveillance system (the following have been identified within the RISKSUR project: case detection, prevalence estimation, demonstrate disease freedom and early detection; see details below); and (3) the geographical area covered by the surveillance system.

These surveillance systems are designed within a context that includes (1) the specific animal population susceptible to the hazard in the region of interest; (2) characteristics of the distribution of the hazard (or hazard risk) at the population level, herd level, or animal level, which can impact the design of surveillance; and (4) political and economic context link to the surveillance system priorities.

1.2 Step 2: Describing the Surveillance Evaluation Context

The tool allows for a sound and standardized identification of the most relevant question adapted to the user and/or decision maker needs according to its specific surveillance context. The EVA decision tool provides this flexibility to adapt the evaluation protocol to the specific case study context including, for example, surveillance objective and decision maker needs.

The evaluation context provides the background information that will help to make choices about how the evaluation needs to be carried out (Why? What? How?). The tool then asks the user to enter information on the elements of the context (surveillance system and evaluation needs) that are essential to frame the evaluation and define the evaluation question and also to analyze and discuss the outputs of the evaluation by setting them back in their context (Table 4.1). Some of these context elements are being retrieved from the surveillance system description section.

Table 4.1 List of evaluation context elements included in the EVA tool and their relevance in the framing of the evaluation process (from [3])
Full size table

1.3 Step 3: Selecting the Evaluation Question

The evaluation question is the most important aspect of the evaluation process. As evaluation is intrinsically linked to action, it makes little sense, and is of limited interest, to perform an evaluation without a specific objective for action. Indeed, an evaluation process with the aim to gather knowledge is qualified as a “quasi-evaluation” as it provides elements on how the system is working but no elements about why the system is performing this way and is therefore limited in terms of recommendation for corrective actions and improvement. This is especially important to consider in resource-scarce situation as evaluation has a cost, both in terms of human and financial resources, and should therefore provide meaningful and practical recommendations for improvement.

A list of 11 evaluation questions were defined within the EVA tool aiming to account for diverse evaluation needs including economics (Table 4.1). However, this list might not be exhaustive and could be reviewed based on feedback from users of the tool and/or comments made on the EVA wiki. A decision tree was also developed to assist the user with the choice of the evaluation question. In brief, the users are guided through a series of questions (11 in the longest pathway) to define their evaluation priorities (e.g., level of evaluation, system, or component; previous knowledge of effectiveness; need for economic analysis); and to identify the most relevant evaluation question. At the end of the pathway, the user is redirected to the evaluation question list and the tool will preselect the appropriate question.

In order to promote best practices in economic evaluation of surveillance, guidance and practical information on economic evaluation is provided both in the tool itself and the Wikispace. A series of relevant questions that allow defining an economic evaluation question has been developed to help frame the evaluation context and the evaluation questions according to this context. Out of the 11 evaluation questions defined in the tool, 5 are economic evaluation questions covering three common types of economic evaluation methods: least-cost assessment, cost-effectiveness, and cost–benefit analysis (Table 4.2).

Table 4.2 List of evaluation questions developed within the EVA tool and evaluation criteria and methods linked to each question (from [3])
Full size table

1.4 Step 4: Selecting the Relevant Evaluation Attributes

The tool provides the full list of attributes organized by level of relevance according to the surveillance system under evaluation and the evaluation context and question. The user can then select the attribute(s) he wants to include in his evaluation. This selection could be done a priori, according to the data currently available on each attribute, or in a second step, by first selecting all the highly relevant ones and then reviewing the type of methods and data needed to assess each relevant attributes (see next step).

A total of 19 evaluation attributes were included in the final list consolidated within the RISKSUR project team (Table 4.3). The differences in relevance of evaluation attributes mainly depended on the surveillance objective (e.g., early detection, freedom from disease, case finding), the evaluation question (e.g., value attributes, organizational attributes), and in some situations on the surveillance design (e.g., risk-based surveillance).

Table 4.3 Final list of evaluation attributes consolidated within RISKSUR project and number of related assessment methods
Full size table

1.5 Step 5: Selecting the Evaluation Attribute Assessment Methods Including Economic Analysis

A list of 70 different methods and/or specific applications of a method was retrieved from the scientific literature. Their characteristics including advantage, limits, and competences required to apply the methods were validated by the relevant experts and included in the EVA tool and the Wikispace (https://survtools.org/wiki/surveillance-evaluation/doku.php). The number of methods validated for each evaluation attribute is indicated in Table 4.3. The tool allows the user to select the most relevant method according to the list of attributes (selected in step 4) and the data availability.

Novel methods developed as part of the RISKSUR project were also included in the EVA tool including

  • EVARisk to assess the risk-based definition criteria; EVARisk is a specific questionnaire developed to collect data on how risk criteria are defined when designing a risk-based surveillance approach. This questionnaire was developed based on a systematic literature review on risk assessment and risk-based sampling methods. A fact sheet document on the standard methods to define risk of risk-based surveillance was also produced based on the literature review to allow the user to qualitatively assess his/her risk selection approach compared to standard methodology available in the literature.

  • AccePT to assess the acceptability, engagement, and benefits of a surveillance system [5] (see Part IV, Chap. 8); AccePT is based on participatory approaches and allows gathering semiquantitative information on the level of acceptability and/or benefits of the surveillance system by the actors [6]. The use of participatory approaches allows engaging the stakeholder in the evaluation process and in the definition of practical corrective actions to improve the system [6].

  • A new approach to assess the effectiveness of the system [7] (see Part V); this new rationale to assess effectiveness consists in a generic rationale in which effectiveness of a surveillance system is expressed in terms of discrepancy between the modalities and intensity of ideal prevention and/or control measures (given a perfect knowledge of the true epidemiological status of a population and the modalities) and/or control measures that are likely to be actually implemented (often based on the analysis and interpretation of the data produced by a surveillance system).

  • A cost calculator tool, which was developed by RVC, is provided to estimate the cost of health surveillance; the cost is essential to perform economic analysis. Economic analysis techniques encompassing least cost, cost-effectiveness, and cost–benefit assessments are listed and described in the tool and linked to the economic evaluation methods described in detail in the evaluation Wikispace. Further information on economic evaluation of surveillance system is presented in Part III of this book.

1.6 Steps 6 and 7: Implementing the Evaluation and Reporting on the Evaluation Outputs

The tool allows to produce a comprehensive evaluation plan that will support the evaluators in the implementation of the evaluation (see examples in case studies section below).

2 Practical Application of the EVA Survtool

2.1 Case Study 1. Evaluation of the Swine Disease Surveillance System in Vietnam

Information on swine disease srveillance in Vietnam was inputted in the EVA web tool in combination with information on the context of evaluation (e.g., decision maker, legal requirements). The tool generated an optimum selection of evaluation attributes and measurement methods to assess the efficiency, effectiveness (e.g., sensitivity), and also functional aspects influencing the overall performance of the surveillance system under evaluation. Then information on the evaluation protocol was exported from the EVA web tool into PDF file.

2.1.1 Evaluation Plan—Outputs of EVA Tool Steps 1–5

Evaluation name

Swine disease surveillance system in Vietnam

Evaluator name(s)

Thi Thanh Pham Hoa, Marisa Peyre

2.1.1.1 Selected Evaluation Question(S)

Evaluation question

EVA tool question number

Assess the costs and effectiveness of different surveillance scenarios to determine which achieves a defined effectiveness target at least cost, the effectiveness needs to be determined

Question 4

Assess whether a surveillance component generates a net benefit, the biggest net benefit or the biggest under a budget constraint for society, industry, or animal holder(s): Benefit to be measured in monetary terms

Question 5

2.1.1.2 Selected Evaluation Attributes

Evaluation attribute

Attribute assessed

Yes/no

Justification on the choice/removal

Surveillance system organization

Yes

An assessment of the strengths and weaknesses of the surveillance system including the existence of clear, relevant objectives is an essential aspect of its evaluation to (1) identify the needs for improvement and aspects to be evaluated and (2) ensure meaningful recommendations.

Acceptability

Yes

Pig producers’ acceptance to pig disease surveillance and control measures and qualitatively assessed based on DCE study results (qualified as either high, medium, or low).

Timeliness

Yes

Considered as highly relevant for early detection of cases (objective of swine disease surveillance in Vietnam) and has great influence on the effectiveness of interventions. Timeliness was defined here as the time between case detection and reporting by the farmer.

Sensitivity

Yes

Defined as the ability of the system to detect cases (the percentage of reported cases over the total number of cases occurring).

Economic efficiency (cost-effectiveness)

Yes

Among feasible (acceptable) options that comply with minimum legal requirements, the least-cost option should be chosen to use resources rationally.

Economic efficiency (cost–benefit)

Yes

Among feasible (acceptable) options that comply with minimum legal requirements, the option with the higher benefit/cost ratio should be chosen.

2.1.1.3 Assessment Methods

Attribute

Assessment method

Surveillance system organization

The system was mapped and qualitatively assessed using data on official veterinarian actions to animal disease reporting and management of disease outbreaks generated in the different studies performed under the framework of the same project and using social network analysis method [8] (see Part V).

Sensitivity

Quantitatively assessed using data generated in the discrete choice experiment (DCE) study previously performed [9] (see Part III).

Timeliness

Assessed qualitatively as high, medium, or low according to the data generated in DCE study [9].

Acceptability and engagement

Quantitatively assessed using data generated in the DCE study [9].

Monetary benefits

Defined as the avoided losses (i.e., the monetary value of the number of pigs saved from infectious and/or culling). Assessed using an epidemiological simulation model (see Part V).

Costs

As the study compared different scenarios, only the variable costs were considered. The fixed costs such as veterinarian salaries were not included in the calculation as they would be the same for all scenarios. The costs considered were payment for veterinarians for movement control and daily disease reporting, costs of destroying pigs (disinfection, labors), and compensation payment for infected households (see Part III).

2.1.2 Implementation of the Evaluation: Step 6

2.1.2.1 Descriptive Analysis (Qualitative Assessment)

The objective of this task was to describe the surveillance system under evaluation and to build an action diagram (flow chart of the links and actins between actors of the system considered).

Animal disease information needs to be reported officially from pig farmers to commune para-veterinarians, then para-veterinarians report to commune people council and district veterinarians (Fig. 4.2). District veterinarians will report animal disease to district people’s committee and province veterinarians. In case of emergency, animal disease can be reported by phone so that investigation of animal disease is performed at the same day or the day after disease reporting. Sampling for animal disease confirmation is often carried out by province veterinarians. Laboratory testing takes around 2 days for giving the results so the disease can be confirmed in 3–4 days. However, disease outbreak notification needs to be official and issued by provincial people committee before disease information is being sent to the national veterinary center (DAH). Process of official reporting of animal disease was considered by the farmers as complicated with several levels of veterinary services (commune, district, province, national center) (Fig. 4.2). The action of official veterinarians upon animal disease reporting was considered between medium to sufficient quality by pig farmers as veterinary officers often visit farm at the same day or the day after disease reporting and give advice on disease prevention and control. Control measures such as disinfection or destroying dead animals are applied to infected holdings even before disease confirmation by the laboratory. Pig farmers can report their swine disease to district/province/national level by the hotline. In each administrative veterinary service (i.e., district, province, or DAH), there are office staffs receiving animal health information through hotline and send to lower level (e.g., from DAH (national level) to sub DAH (province level), then to veterinary station (district level) for disease investigation. Disease progress is then reported daily to upper level. Control measures will be applied if necessary. Informal swine disease surveillance system also exists in parallel with formal surveillance system and is well developed with the involvement of private actors (Fig. 4.2). Feed/drug companies often provide free technical support for large pig farms or drug/feed shops as a market strategy. Drug/feed delegations from companies often visit farms and feed/drug shops to collect data on feed/drug sales as well as the swine disease information. When there are swine disease outbreaks at farm, they might give advice on diseases treatment or send technicians from the company to do disease investigation, take samples, and send to their laboratory for disease confirmation. Small pig holders often contact drug/feed shops to buy the drug and ask for advice of disease treatment when there is any disease problem at their holdings. They also get animal health information in surrounding areas from drug/feed sellers to consider the disease prevention measures at their holdings. According to pig farmers’ perception, informal swine disease surveillance is effective in terms of timeliness and scale of animal health information transmission.

Fig. 4.2
figure 2

Organization of the swine disease surveillance and control system in Vietnam. (From [8])

Full size image
2.1.2.2 Acceptability of Surveillance and Control Policy by Pig Farmers

The willingness of pig farmers to report swine diseases at their farms depends largely on how veterinary officers interact and control the disease after reporting. Three scenarios of disease control measures were used to elicit farmers’ preference to report swine diseases (Table 4.4) [9]. Scenario 2 (destroying only dead or unrecovered pigs in infected households with compensation of 70% market value of pig) was well accepted by most pig producers. Around 90% infected households would report pig diseases in their holding if they were certain of being compensated. Pig farmers mentioned that they can even accept the lower compensation level (50% market value of pig) for destroying only dead/unrecovered pig at their farm (scenario 3). However, scenario 1 (stamping out all pigs in infected households with compensation of 70% market value for culled pigs) was the least accepted by most pig farmers due to the unacceptability of mass culling of clinically healthy pigs and especially culling of healthy breeders. This low acceptability had a great impact on the reduction in the proportion of pig farmers willing to report swine diseases and therefore had significant influence on the performance of surveillance system.

Table 4.4 Summary of the evaluation of swine disease surveillance in Vietnam
Full size table
2.1.2.3 Timelines of Swine Disease Reporting

Information on swine disease situation in the study area, swine disease ranking, and swine disease reporting was obtained from focus group discussions and key informant interviews [9]. Most pig holders mentioned that they will try to treat sick animals at least during 1 week before considering reporting; reporting will be based on the disease progress upon treatment. They will first contact the veterinary drug sellers or commune para-veterinarians who work as drug sellers or private veterinarians to buy drugs or ask for advice on disease treatment (Fig. 4.2). The commune para-veterinarian also waits one more week to see the result of the treatment and risk of disease spread. The quickest timing for reporting disease would therefore be at least 2 weeks even in case of high transmission rate between household. If the disease does not spread quickly, then the reporting would take even longer time as the veterinarians often give the advice for disease treatment first and assess the disease progress (i.e., the severity and/or spread rate) before reporting the disease to upper veterinary levels.

Compensation for culled pigs was considered as a good incentive for pig farmers to report swine disease early. However, if the farmers are not certain of being compensated for culled pigs, almost all of them will not report the disease. Some pig farmers would report the disease at the end of the outbreak when many infected pigs have died and management of dead pigs by the farmer is difficult (Table 4.4). Culling all pigs in the pig holding negatively affects early reporting even if the compensation is paid for culled pigs. Under this culling regulation, pig farmers consider to report only when infected pigs do not response to treatment and the disease spreads among most pigs in the pig holdings. Culling only unrecovered pigs and being compensated for culled pigs was perceived by farmers as a strong incentive to report early. Anyhow, pig farmers or commune veterinarians will still wait for the response of infected pigs to treatment, so the earliest reporting time will remain at 2 weeks.

2.1.2.4 Sensitivity of Swine Disease Surveillance System

As seen in the DCE study, under the current scenario (scenario 0) only 4% of the farmers actually report swine disease outbreak to the official surveillance system. Around half of pig farmers (50%) would report PRRS at their farms under scenario 1 (i.e., if they are certain to get compensated and if the compensation level is of 70% market value for culling all pigs in infected farms) [9]. However, nearly all pig farmers (90%) would report the disease if only dead/unrecovered pigs at their farms are destroyed and compensated with 70% market value (scenario 2). Most pig farmers (80%) even accepted a lower compensation level of 50% market value for destroying dead/unrecovered pig (scenario 3) [9].

2.1.3 Cost Analysis

The total surveillance and control costs for scenario 1 were nearly twice as high as scenario 2 costs and three times higher than scenario 3 costs [8]. This is due to the highest number of culled pigs in scenario 1 (culling of all pigs in infected holding). This led to the highest compensation payment and costs of pig destroying. Scenario 2 had higher costs than scenario 3 due to the higher compensation level and the higher number of culled pigs. The total cost of each scenario was mainly influenced by the amount of compensation payment for pig farmers and therefore by the number of culled and dead pigs compensated.

2.1.4 Cost-Effectiveness Analysis

Overall result: Scenario 2 was found to be the most effective in terms of acceptability, timelines, and sensitivity (Table 4.4). However, the total cost of scenario 2 was 1.5 times higher than scenario 3. Scenario 3 had a medium acceptability level, timeliness, and a 7% lower sensitivity level than scenario 2. In order to assess the added value of scenario 2 versus scenario 3 in terms of disease control, a cost–benefit analysis was performed.

2.1.5 Cost–Benefit Analysis

Benefit–cost ratio of scenario 0 could not be estimated as under this current scenario almost all pig farmers did not report the disease (Table 4.5). The disease spreads easily to surrounding areas without any control. A few farmers (4%) might report the disease when there are many dead pigs, but it is usually at the end of the outbreak. Thus, the benefit of control measures could not be estimated at the commune level using the simulation model developed in this study. Scenario 3 had the highest benefit–cost ratio (B/C = 5.2) compared to scenario 2 (B/C = 3.5) and scenario 1 (B/C = 1.1). The benefit–cost ratio of scenario 3 was higher than that of scenario 2 due to saved cost of compensation (i.e., lower compensation level and lower number of culled pigs).

Table 4.5 Cost–benefit analysis
Full size table

2.1.6 Recommendations: Step 6 (Addressing the Evaluation Questions) and Step 7 (Reporting on the Evaluation Outputs)

The objectives of this task were to review the meaning of the results considering the surveillance system in its global aspects and provide recommendations for decision makers based on the economic evaluation results.

2.1.6.1 Quality of the Evaluation Performed

A comprehensive approach was used to perform economic evaluation of swine disease surveillance and control in Vietnam. Multiple attributes covering different aspects of surveillance system effectiveness and efficiency such as organization, function, and value of the system were considered. The effectiveness of surveillance system was assessed based on the cost-effectiveness analysis of different control options, which had a significant influence on the performance of surveillance system (i.e., proportion of farmer reporting). This economic analysis technique did not allow differentiating between the two most effective scenarios (2 and 3). A cost–benefit analysis was required to assess the added value of detecting 7% more cases (sensitivity) with a 50% increase in costs in terms of disease control. The results of the cost–benefit analysis highlighted the limited importance of an increase of 7% of surveillance sensitivity in terms of efficient disease control. This comprehensive evaluation provided meaningful recommendations; however, such evaluation could be time consuming and laborious and required simulation modeling and socioeconomic field studies (i.e., DCE) to provide information on farmer’s decision-making useful for both quantitative evaluation (sensitivity) and qualitative evaluation (acceptability) of surveillance system. Other biases linked to the hypothesis used in the modeling study and the cost estimation have been identified and addressed to ensure validity of the recommendations and are described elsewhere [8].

2.1.6.2 Recommendations to Improve Surveillance Strategies

Control scenario involving destroying of only dead and unrecovered pigs and compensation of 70% of pig market value got the highest acceptability, but the scenario with compensation of 50% market value for unrecovered pigs gave the highest B/C ratio. Culling all pigs in infected households and compensation of 70% market value of pig were considered as limited effectiveness due to the low famers’ acceptability even though the B/C ratio was 1.1. In case of endemic disease, this scenario was thought as wasted money as most farmers considered that they can treat sick animals and morbidity and mortality of the diseases depends much on the prevention of other diseases (e.g., classical swine fever, epizootic pneumonia, etc.). Keeping infected pigs at pig holdings in infection period showed the potential risk of disease spread to surrounding areas. However, improving biosecurity at pig holdings and ensuring proper outbreak management (i.e., disinfection, movement restriction, and vaccination) can help to prevent the disease spread. Indeed, previous study highlighted that emergency vaccination was a more effective strategy to control PRRS outbreaks rather than stamping out [10].

2.2 Case Study 2: Evaluation of Classical Swine Fever Surveillance System in Germany

The Federal State Rhineland-Palatinate in Germany comprises 24 districts and 12 municipalities, covering a total area of about 20,000 square kilometers. In this state, classical swine fever (CSF) infection in wild boars was detected in 1995 and between 1998 and 2009 with the two most recent outbreaks occurring in two separate parts in the beginning of 2009. Since 2002, infection in the wild boar population was controlled in some part by means of oral immunization with vaccination baits. In May 2012, the state was officially declared free from CSF. Since then, regular surveillance measures have to be applied to demonstrate freedom from the disease. In this case study, the current and alternative surveillance strategies to demonstrate were evaluated to demonstrate freedom from disease.

2.2.1 Evaluation Plan—Outputs of EVA Tool Steps 1–5

Evaluation name

Classical swine fever in Germany

Evaluator name(s)

Katja Schulz, Christoph Staubach, Marisa Peyre

2.2.1.1 Selected Evaluation Question(S)

Evaluation question

EVA tool question number

Assess the effectiveness of one or more surveillance component(s) or system(s) in relation to a surveillance objective and rank the options accordingly

Question 2

Assess the timeliness of different surveillance strategies and assess the acceptability in hunters of different surveillance strategies—These questions relate to the EVA tool question: “Assess the technical effectiveness of one or more surveillance components and the functional aspects of surveillance that may influence effectiveness”

Question 9

Assess the costs of surveillance component(s) or system(s) that achieve(s) a defined objective and rank them according to costs to identify the least-cost option

Question 3

2.2.1.2 Selected Evaluation Attributes

Evaluation attribute

Attribute assessed

Yes/no

Justification on the choice/removal

Surveillance system organization

Yes

An assessment of the strengths and weaknesses of the surveillance system including the existence of clear, relevant objectives is an essential aspect of its evaluation to (1) identify the needs for improvement and aspects to be evaluated and (2) ensure meaningful recommendations.

Sensitivity

Yes

Sensitivity is a critical requirement to fulfill EU regulations for CSF. An improvement in timeliness is expected to lead to fewer outbreaks.

Timeliness

Yes

Acceptability

Yes

Any change in a surveillance strategy for CSF is likely to be influenced by stakeholders.

Economic efficiency (cost-effectiveness)

Yes

Among feasible (acceptable) options that comply with minimum legal requirements, the least-cost option should be chosen to use resources rationally.

2.2.1.3 Assessment Methods

Attribute

Assessment method

Surveillance system organization

Descriptive analysis (qualitative assessment)and SWOT analysis using OASIS trop tool (see Part IV)

 

Component 1

Component 2

Sensitivity and timeliness

Field data: Sensitivity of the whole surveillance system by using capture–recapture method (see Part V)

Simulation data (see Part V)

Acceptability and engagement

AccePT method (see Part IV)

2.2.2 Implementation of the Evaluation: Step 6

2.2.2.1 Descriptive Analysis (Qualitative Assessment)

The objective of this task was to describe the surveillance system under evaluation and build an action diagram (flow chart of the links and actions between actors of the system considered).

Method: Application of the OASIS Trop protocol and collation of expert opinion.

System mapping examples (Fig. 4.3):

Fig. 4.3
figure 3

Information flow (a: bottom-up; b: top-down) within the surveillance system of the currently implemented, active surveillance for classical swine fever in wild boar in times of disease freedom on the basis of the Federal State of Rhineland-Palatinate

Full size image
2.2.2.1.1 SWOT Analysis Report

The objective of this task was to gather information on the system performance process (strengths and weaknesses of the system). This was conducted using the OASIS Trop tool (accessible here: https://survtools.org/wiki/surveillance-evaluation/doku.php?id=surveillance-system-organisation).

The OASIS Trop tool results are presented in the following manner:

  • Output 1 gives the satisfaction level of each criterion, which provides an indication of the functioning and the global situation of the surveillance system.

  • Output 2 indicates the critical points of the surveillance system.

  • Output 3 gives the scores of the quality indicators.

Output 1: Strengths and Weaknesses of the System: Fig. 4.4 demonstrates the strengths and weaknesses of the surveillance system process based on the application of the OASIS tool (described in Part IV) [11].

Feedback on the results of the strengths and weaknesses of the system process (output 1)

Process section

Feedback/recommendations

Objectives (100%)

The objectives are clearly stated and documented and take into consideration all stakeholders.

Central unit (74%)

There is no specific steering committee but they are operational and they meet regularly (every day communication); their material and financial resources are not considered sufficient. However, the human resources are sufficient.

Field institutional organization (92%)

Only human resources in the field are considered as low sufficient (material and financial resources are very sufficient).

Some limits in the representativeness as only some age groups are being hunted.

Laboratory (82%)

Some tests are not included in interlaboratory trials (pathology and sequencing); however, this is not considered as a limitation of the system.

Laboratory has only a minor role in the surveillance system.

Investigation team not assigned 100% on the task but could be mobilized upon request: Not considered as a limitation.

Low specificity of the suspicion and confirmation tests.

Delivery from laboratory to CU of results sometimes delayed but minor problem.

Surveillance tools (86%)

Limits in acceptability of the consequences of a suspicion or case for the source or collector data: Important constraints linked to control measures implemented.

Specificity of the case definition is low, but this is not an issue as high sensitivity is preferred under freedom from disease objective.

Some issue in the delay of sending samples to the laboratory.

Not all but the majority >95% of the collection form and sample are correct.

Surveillance procedures (86%)

The surveillance is not exhaustive: No = caused by spatial aggregate + deficiencies minor (due to hunter engagement) + level of underreporting = between 15% and 30% (field agents); intermediary units <5% (assumed).

No indemnities for hunters (data source); no awareness building.

Limited representativeness: This bias justifies the need/will to change surveillance protocols in place.

The samples obtained are based on the hunting bag and the hunting bag itself is already biased by several factors, e.g., hunting intensity, hunting ground, environmental conditions, season, hunting ethics, etc.

Furthermore, randomly distributed over the hunting bag, e.g., no classification by age classes.

Randomly distributed over the year following the hunting bag distribution, e.g., no classification by season.

Data management (90%)

A relational database exists and is considered adequate but does not hold all the data (only a majority).

There is a delay in data input time, but this time lag is considered minor and does not affect the system efficacy.

Training (89%)

Some actors are not concerned by the initial training.

There is no refresher training in place.

Communication (70%)

The frequency of report/publications is not planned by the network.

There is a feedback of results to field actors but no means to control it.

There is regular dissemination of reports, easy to control but no guarantee that it reaches all field actors.

Communication system: Not all the actors use the communication means.

Evaluation (67%)

Performance indicators measured: Depending on the federal states sometimes major improvements are required.

There is no external evaluation.

Fig. 4.4
figure 4

Strengths and weaknesses of the surveillance system process: level of compliance of each section of the system process according to an ideal system (OASIS Output 1)

Full size image

Output 2: Critical Points Analysis: The OASIS tool identifies seven critical points for the surveillance system using the hazard analysis critical control point (HACCP) method [12]:

  • Objectives: This point assesses the capacity of the objectives to be coherent and consider the expectation of the different stakeholders and of the surveillance procedures to be appropriate with the surveillance objectives.

  • Sampling: This point assesses the quality of the sampling.

  • Animation: This point evaluates the quality of the trainings and the awareness of field agents and the organization of the surveillance system.

  • Tools: This point assesses the quality of the collection, testing, analysis of the samples (i.e., the quality of the laboratory processes).

  • Collection and circulation of data: This point assesses the quality of the samples and results collection and the system communication.

  • Processing and interpretation: This point assesses the quality of the data management.

  • Dissemination and information: This point assesses the quality of the communication of the results.

Figure 4.5 shows that the most critical point in the system in need of improvement is “dissemination and information.” This has an impact on the level of acceptability and compliance with the system. Further, improvement could be made regarding sampling to improve representativeness. The points “tools and processing” and interpretation of data show smaller margins for improvement.

Fig. 4.5
figure 5

Critical control point assessment of the classical swine fever system in Germany (OASIS Trop Output 2)

Full size image

Output 3: Qualitative analysis of the evaluation attributes (how the system process affects the effectiveness and functional attributes of the system). The score of the performance and functional attributes results from a combination of scores of the different criteria from the OASIS Trop score grid. The size of the blue portion of each bar on the radar represents the level of the satisfaction for the performance attribute considered.

The freedom from disease surveillance system requires a higher sensitivity over specificity. However, the sensitivity of the system is not optimal (66%) as it is affected by the acceptability of the stakeholders (70%) and the representativeness (69%), which are two elements of the system to be improved (Fig. 4.6).

Fig. 4.6
figure 6

Qualitative assessment of classical swine fever surveillance system performance and functional attributes (OASIS Trop Output 3)

Full size image
2.2.2.1.2 Analysis of Changes in the System Process Incurred by the Change in Surveillance Design

A wide range of different active, passive, and combined surveillance scenarios were simulated as listed below varying the number of samples, targeted districts, frequency, types of samples, and age classes of animals. The strategies based on different age classes will only change the shooting behavior of the hunters, which means that they need to shoot more animals of a defined age group. The strategies based on defining districts will change the workload and reduce the cost for some veterinary authorities and some hunters (transport costs, sampling costs). The strategies based on season will change the workload and the cost within 1 year as the samples have to be delivered and picked up only in certain month, which impacts on transport and sampling costs. The strategies based on sample size will reduce or increase transport and sampling costs due to a lower or higher volume. Moreover, the costs are dependent on the examination of samples, that is, whether they were only serologically, virologically examined, or in both ways.

2.2.3 Cost Analysis

The aim was to assess the cost of the different options considered for the case study (current and novel design).

Method: A cost-effectiveness analysis was used to identify which strategies could achieve a defined target at the least cost [13]. First, all strategies that achieved a detection probability of at least 95% (i.e., an effectiveness of 100%) were identified. Second, the costs for each strategy were calculated using the cost calculation spreadsheet taking into account variable additional costs for labor, operations, and expenses associated with each strategy (https://survtools.org/wiki/surveillance-evaluation/doku.php?id=cost-analysis) (Table 4.6). The costs for each of the 100% effective scenarios were expressed as cost units (instead of euros) due to data confidentiality issues. To estimate these cost units, the costs of the reference strategy were estimated taking into account the variable costs for transport and analysis. The costs for the alternative surveillance strategies were then estimated and represented as a proportion relative to the estimated costs of the reference strategy.

Table 4.6 Overall evaluation of all strategies, in which all three evaluation attributes and costs were investigated. S represents the score, whereby 1 constitutes the best result; sero = simulation of serological sample examination, vise = serological and virological sample examination (from [13])
Full size table

Discussion: The cost assessment showed that—among all the scenarios that would comply with legislative requirements (i.e., 95% detection probability)—there would be 19 scenarios leading to lower surveillance costs when using serological testing and 11 scenarios when using serological and virological testing. Consequently, there are cost savings to be made in CSF surveillance in Germany.

2.2.4 Effectiveness Assessment

Sensitivity was assessed by measuring the detection probability. This was done using a simulation model [13]. The model was based on real data including population estimates, population structure, hunting data, and course of infection and used to determine the detection probability of infection within 1 year.

There were many scenarios that achieved the target effectiveness. The difference between random or real distributed sampling over the year was found to be very small. The sensitivity of serological examination was nearly as high as if performing both examinations. However, virological examination only was found to result in a lower sensitivity. When samples were examined serologically as well as virologically almost in all strategies, the detection probability was above 95% (in 21 from 24). When the samples were only examined virologically, the detection probability did not exceed 85%. The reference strategy was shown to have almost 100% detection probability. Most of the risk-based strategies showed a detection probability above 95% provided that the samples were not only examined virologically.

Timeliness was defined as the time between introduction and detection of a CSF virus infection. Using the model described above, for every repeated simulation with detection of the infection, timeliness was estimated as the number of months between introduction and detection of the infection. The results for the timeliness assessment were consistent with those found for sensitivity. The best timeliness could be seen in risk-based strategies that were taking into account the age of the animals. It was found that smaller sample sizes resulted in lower timeliness.

Acceptability, which provides evidence on the functional aspect of the surveillance strategies, was investigated using participatory methods as described previously [14]. Within the surveillance system for CSF in wild boar, hunters play a key role in sample collection. The acceptability by hunters and veterinarians of the current system as well as of some of the alternative strategies was examined [15] (Table 4.6) (see Part IV).

The trust in the system by the hunters was quite high (0.75). However, their acceptability of the objective of the system was moderate (0.4) and the acceptability of the operation even lower (0.1).

Their acceptability of the strategy in which only samples resulting from passive surveillance should be examined was very low (−1.3). This was mainly due to the fact that sampling dead, decomposing animals can be rather unpleasant. Interestingly, the hunters’ acceptability of the strategy of taking 59 samples only quarterly was only moderate (0.4). The most accepted strategy was taking the 59 samples only within the age group of subadult animals (1). Remarkably, also the currently implemented active surveillance strategy was well accepted (0.9). It was not possible to present all scenarios to the hunters; therefore, only the results for a few scenarios are available. Because it was not straightforward to recruit hunters and they were mainly from one age group, there was a risk of bias and subjectivity.

2.2.5 Cost-Effectiveness Assessment

Overall Result: For each attribute, a rank was given to each attribute and the combined rank was determined weighing each attribute equally (Table 4.6). Doing this, it was possible to present an overall priority list of the potential scenarios.

2.2.6 Recommendations: Step 6 (Addressing the Evaluation Questions) and Step 7 (Reporting on the Evaluation Outputs)

The objectives of this task were to review the meaning of the results considering the surveillance system in its global aspects and provide recommendations for decision makers based on the economic evaluation results.

2.2.6.1 Quality of the Evaluation Performed

Because fewer data are commonly available on passive surveillance compared to active surveillance, the outcomes of the simulation model may not have full reliability. The differences in detection probabilities were found to be rather small (e.g., 99.9 vs. 99.5), which may be due to the simulation differences (1000 simulation runs). The combination of sensitivity analysis and timeliness analysis did produce important additional information. Even when strategies result in an equal detection probability, the timeliness can be the crucial factor for the final choice of a strategy.

The focus group discussion added value to the evaluation. It became obvious that some of the strategies, although they would be cheaper and more effective, would not be very well accepted by the hunters.

The final results of the analysis showed the benefit of combining different evaluation attributes and also performing an economic evaluation. Thereby, it was possible to identify the strategy resulting in the best overall performance, but it needs to be kept in mind that the attributes may need to be weighted differently, which would change the ranking of the combined performance assessment.

2.2.6.2 Recommendations for Surveillance Design Changes

While the current surveillance strategy has a good performance, it is worth considering to sample only in defined age groups. If a decision maker is interested in saving costs, it is advisable to discontinue the double serological and virological examination of the samples. The results suggest that the sample size could be decreased to 50 or even 40 samples per district as the 59 sample size calculation is based on the assumption of an infinite population and a homogenous distribution of infection with the population. Finally, the combination of different strategies (lower sample size within a defined age class) was found to be a feasible and cost-effective alternative.

3 Conclusion

The EVA tool provides a comprehensive framework for integrated evaluation of health surveillance, including a step-by-step guide on how to best perform the evaluation and providing a reference online platform for health surveillance evaluation methods. Its applications in the field highlighted the importance of such a comprehensive evaluation approach to improve the quality of the evaluation outputs (economic evaluation; multiple attributes assessment) and demonstrated the usefulness of the guidance provided by the tool itself. Comprehensive evaluations might be constrained by practical issues (e.g., confidentiality concerns, data availability) and resource scarcity. In the long term, the EVA tool is expected to increase professional evaluation capacity and help optimizing health surveillance system efficiency and resource allocation for both public and private actors of the surveillance systems.