Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Before we can start to synth esise evidence we need to have some understanding of what evidence is and where the new evidence being explored comes from. Fundamentally as Lomas et al. (2005, p. 1) suggest ‘evidence concerns facts (actual or asserted) intended for use in support of a conclusion.’ Decision makers tend to view evidence colloquially, that is evidence is anything that can give a reason for believing something relevant is considered evidence. Researchers will tend to view evidence scientifically, it must be produced by robust, systematic and replicable methods that are clearly defined. So evidence is something that can be used to support a conclusion, but it is not the same as a conclusion (Lomas et al. 2005). Evidence can, and should, support decision making but the collection of evidence alone is not going to make the decisions.

Evidence for Public Health impacts and interventions is generated through the process of research . Research is about generating new information, doing something new, collecting information to answer specific research questions and testing ideas or hypotheses.

There are several characteristics of good research . It should be:

  • Systematic: there is an agreed system for performing observations and measurement

  • Rigorous: the agreed system is followed exactly.

  • Reproducible: all the techniques, apparatus and materials used in making observations and measurements are written down in enough detail to allow other to reproduce the same process.

  • Repeatable: researchers often repeat their observations and measurements several times in order to increase the reliability of the data.

    (Bruce et al. 2008)

Types of Research

Research is typically divided into quantitative or qualitative methods but these are not mutually exclusive and should be seen as complementary approaches to obtaining evidence. Creswell (2014) has described these two approaches as being on a continuum, with quantitative methods tending to explore issues through numbers, and qualitative methods tending to examine issues through words. Using this research typology (or classification), this would on ly distinguish questions which seek to measure something (quantitative) from those that don’t (qualitative). This seems to be only a small element of the range of research questions that social scientists address. You might have already noticed that qualitative research doesn’t seek to measure anything … but may do much more, whilst quantitative research does seek to measure … but may also do much more. A third type of research is termed mixed methods research , where researchers would combine both qualitative and qu antitative methods (Creswell 2014). We can see this range of approaches in evidence syntheses as well as in the primary research.

So we can summarise research into three main approaches:

  • Quantitative

  • Qualitative

  • Mixed-methods (Using both quantitative and qualitative methods)

Although these can be viewed as the main divisions they can be subdivided into many more specific categories. For example, a working paper for the National Centre for Research Methods categorised research designs into 20 types (Beissel-Durrant 2004).

Types of Research Questions

We will now focus mo re on the types of questions that one needs to think about. Research questions should always relate to the nature of the problem being explored. If you look at any research proposal or research article, they will tend to have clearly stated aim(s), objectives and/or specific research questions.

Discussion Task

Search for research papers on a public health topic, such as alcohol consumption, levels of inactivity etc. How do researchers frame their aims or research questions?

Comments: You will notice that there are 4–5 main types of questions.

Some research questions (outcome) relate to describing things within contextual paradigms (descriptions of type of research), some relate to comparing one aspect to another (comparisons), others are related to measurements, giving/emphasising values or determining some relationships between or amongst different attributes.

All of these approaches can form the framework for evidence synthesis and individual studies using these approaches can contribute to the evide nce being synthesised.

Research Synthesis

Within this section we will try to define the meaning of research synthesis , its importance, and how to synthesise diverse sources of evidence. It is important that policy-makers and managers always use a wide range of sources of evidence in making decisions about policy and the organisation of services. However, they are under increasing pressure to adopt a more systematic approach to the utilisation of the complex evidence base in healthcare. Sometimes, decision-makers must address some complicated questions about the nature and significance of the problem to be addressed; the nature of proposed interventions; their varying impact; cost-effectiveness; acceptability and so on (Mays et al. 2005).

As Coast (2006) notes, the meaning and purpose of synthesising research is identifying

  • What is known from what has been done?

NOT

  • What has been done?

Why Do We Need Evidence Synthesis ?

A quick perusal of public health research journals will rapidly show the huge variety of research evidence generated to help us understand public health problems. Firstly the subject is very multidisciplinary, evidence is generated by a wide range of disciplines including:

  • Biostatistics

  • Epidemiology

  • Environmental health

  • Global health

  • Health education

  • Health improvement /promotion

  • Health protection

  • Occupational health

Furthermore health is a broad concept and many different factors influence our health. All these are studied and explored in increasing volume and depth e.g:

  • Alcohol

  • Communicable diseases

  • Drugs

  • Mental health

  • Obesity

  • Sexual health

  • Exercise

  • Tobacco

To focus on just one example consider a case study for evidence around heart disease, our largest cause of death.

Case Study: Heart Disease Research

Heart disease is our largest cause of death and numerous research has been conducted to try and understand how heart disease progresses, what factors increase the risk of developing heart disease and how effective interventions are at preventing its development? Conduct a simple web search (e.g. using Google Scholar) for academic research articles on these three topics:

  1. 1.

    Aetiology of heart disease (how it progresses)

  2. 2.

    Risk factors for heart disease

  3. 3.

    Interventions for heart disease

Comments: you will find your searches yield huge numbers, millions of papers. Also within these topics there is great variety e.g. for risk factors , research might have explored factors such as:

  • Age

  • Hypertension (high blood pressure)

  • Cholesterol

  • Diabetes

  • Smoking

  • Alcohol

  • Diet

  • Cocaine

  • Obesity

  • Exercise

  • Preeclampsia

  • Family history

and there are many others (NHLBI 2015), so there is a need to synthesise all this evidence.

Keeping track of all this evidence is clearly a challenge. Public health decision makers can be overwhelmed by the volume of data and evidence, i.e. by too much information. Increasingly public health organisations and researchers have been developing tools and techniques to help synthesise the evidence, transforming the mass of information in to useful intelligence, so that we can make decisions based on clear and accurate summaries of the range of evidence available (Fig. 1).

Fig. 1
figure 1

Transforming information into informed decisions

Full size image

Research synthesis is important in public health because it allows researchers to view problems from multiple perspectives, contextualise information, develop a more complete understanding of a problem, triangulate results, quantify hard-to-measure constructs, provide illustrations of the context for trends, examine processes/experiences along with outcomes, and capture a macro picture of a health system (Creswell and Clark 2011). We can argue that balanced inference on best available evidence is always important, not a detailed description of everything on the subject. It is equally important to note that chosen approaches or methods should be rigorous, using ‘scientific’ methodology in terms of being replicable/accountable/having updateable findings, minimising biases and errors, as well as being appropriate to answer the focussed question(s).

Discussion Task

Read the definition of research synthesis first: Research synthesis can be considered as an approach of integrating/synthesising several studies’ attributes, i.e. study design, findings and quality; not only to be able to identify research gaps or ‘silences’ that require new primary studies, but also to provide a unique presentation of multiple realities or truths (see Mosteller and Colditz 1996, p. 4; Pope and Mays 2006).

Identify four or five important purposes and consider how this concept relates to making appropriate design in public health or health policy.

Types, Methods and Approaches to Research Synthesis

As Popay and Roberts (2006, p. 1) note ‘since the early 1990s the science of evidence review and synthesis has had a growth spurt and has developed rapidly.’ There are now well established tools and methodologies to support effective research synthesis , an example being the international Cochrane collaboration, which publishes evidence synthesis as well as providing best practice guidance on methodologies and online training materials (Cochrane 2015).

Research synthesis is a comprehensive review that looks for, and evaluates, existing research evidence, rather than a traditional literature review. It is interesting to note that methods of synthesis that can accommodate diversity both of questions and of evidence are needed. For example, policy-makers seeking to understand barriers to access to healthcare will need to draw on qualitative evidence (for example, generated through ethnographies and interview studies of help-seeking behaviour) as well as quantitative evidence (perhaps generated through studies of rates of referral). Excluding any type of evidence on grounds of its methodology could have potentially detrimental consequences (Mays et al. 2005, p. 45). There is no single, agreed framework or method for synthesising such diverse forms of evidence, and many of the approaches potentially applicable to such an endeavour were devised for either qualitative or quantitative synthesis and/or for analysing primary data.

Traditional literature reviews can be limited by bias in the selection of information, driven by researcher bias, limited search strategies, and lack of access to resources. Digitisation of resources and the ubiquitous access to computer based search engines have eliminated some of these problems and the systematisation of reviews means we can summarise evidence in a less biased manner.

Mays et al. (2005) identify four basic approaches:

  • Narrative (including traditional ‘literature reviews’ and more methodologically explicit approaches such as ‘thematic analysis’ , ‘narrative synthesis’, ‘realist synthesis’ and ‘meta-narrative mapping’)

  • Quantitative (which either use data from quantitative sources only or convert all evidence into quantitative form using techniques such as ‘quantitative case survey’ or ‘content analysis’)

  • Qualitative (which convert all available evidence into qualitative form using techniques such as ‘meta-ethnography’ and ‘qualitative cross-case analysis’),

  • Bayesian meta-analysis and decisio n analysis (which can convert qualitative evidence such as preferences about different outcomes into quantitative form or ‘weights’ to use in quantitative synthesis ).

The choice of approach or method will be determined by the purpose of the review and the nature of the available evidence. Often more than one approach or method will be required to make the evidence credible and dependable.

As Sheldon (2005) warns, we need to be aware of the following points while assessing and synthesising any research evidence:

  • Low statistical power—studies may be small and in themselves insufficient to be able to provide reliable evidence of benefit or harm. This increases the risk of false negatives.

  • Researcher/expert bias—different researchers will favour certain conclusions and may give different weights to the same evidence and write up their results with a different spin

  • Contextual variability—there may be reasons to think that contextual factors are likely to affect the impact of an intervention, so policy which is effective in one context may not be so in another

  • Methodological and theoretical incompleteness—the reliability of the evidence is likely to be greater if studies of different designs are examined and are mutually supportive. Studies approaching the same problem from different theoretical perspectives can also be illuminating

  • Policy relevance—Policy decisions often need information not only on whether an intervention works but also on the factors that influence how well it works, whether these can be modified, and the distribution of the benefits and costs. It is most unlikely that any one study will cover all these issues.

So how do we go about conducting a research synthesis ? We will explore these separately for quantitative and qualitative as these different research approaches also result in some differences in synthesis methods.

Quantitative Synthesis : Systematic Reviews and Meta-analysis

A casual analysis of daily newspapers will identify health stories based on single quantitative research studies e.g. extolling the benefits of moder ate consumption of red wine. A few months’ later readers might view another story, based on a different study, which has identified risks associated with moderate alcohol consumption. It’s not surprising that the public and policy makers alike are confused. Systematic reviews attempt to avoid the difficulties of relying on single studies, which can often be misrepresented or ‘cherry picked’ in this way. They combine information from multiple sources in a clear and systematic process that tries to minimise any bias in selection of studies. Systematic reviews, assess, select and combine the findings of the selected studies to identify common conclusions. There are commonly seven steps in a systematic evidence synthesis or systematic review (Cooper 2010) (Fig. 2):

Fig. 2
figure 2

The seven steps in a systematic review

Full size image

  • Step 1: Formulating the problem—in any research synthesis the first step is always to identify what are the concepts that are to be studied? What is the hypothesis to be tested, or what research questions will be explored?

  • Step 2: Searching the literature—this should be systematic in nature. So clear search criteria are established that address the aim(s) of the study. There is a defined strategy for identifying literature from a range of sources that includes published journal articles, reports and grey literature and unpublished material. A great deal of time and effort in systematic research synthesis goes into ensuring that the literature search is comprehensive.

  • Step 3: Gathering information from studies—this involves extracting the information from each study that is relevant to the synthesis aim(s). Not all information in each identified study might be relevant to the aims of the synthesis. Typically, due to the limitations of electronic search engines, the majority of studies identified in the literature review are rejected at this point as not being directly relevant to the synthesis. Again clear criteria need to be established to identify what information is to be extracted in order to ensure this is done systematically and without bias.

  • Step 4: Evaluating the quality of studies—once data are extracted the researcher(s) make critical judgements about the data quality for each study. This will be informed by factors such as the clarity of the study methodology reporting, or the type of study (e.g. case contro l or randomised controlled trial). Clearly if this step is not conduced in a fair and systematic way it can be a major source of bias and lead to criticism of the synthesis .

  • Step 5: Analysing and integrating the outcomes of studies—in this step the data from each individual study is integrated with the other studies. In sy stematic reviews this might be a process of identifying common findings between different studies and exploring those collectively. Within quantitative synthesis this can be taken a step further by using a meta-analysis (or metanalysis). A meta-analysis will take the statistical data from each selected study and combine these to provide pooled data that can then be statistically analysed. The advantage of meta-analysis is to provide a much larger effective sample size and to reduce statistical uncertainty .

  • Step 6: Interpreting the evidence—as with any study the researcher needs to interpret the evidence provided by the study and identity clear conclusions from the synthesis .

  • Step 7: Presenting the results—for the synthesis to be useful the results must be presented effectively. Of particular importance within research synthesis is to clearly communicate how the synthesis was conducted i.e. how the first six steps were performed.

An example of the results from a quantitative meta-analysis is shown in Fig. 3. This is a typical type of plot used in metanalysis studies called a Forest Plot.

Fig. 3
figure 3

An example Forest plot showing odds ratios for ten studies and the pooled overall estimate when the individual studies are combined. (Source: Neyeloff et al. 2012)

Full size image

Discussion Task

Examine Fig. 3. Imagine this is showing results from ten studies on tobacco smoke exposure of householders and the odds of their developing lung cancer. What does this tell you about the power of applying meta-analysis and pooling data from multiple studies?

Comments: The plot shows odds ratios and the 95% confidence intervals for each study. There is considerable variation in the estimates and in the levels of confidence. Some studies have odds ratios, and confidence intervals that are >1.0 (i.e. no difference between those exposed and those not exposed to tobacco smoke) e.g. Study 1. These suggest that lung cancer is more likely in those exposed at home. But other studies have odds ratios that are < 1.0 (e.g. Study 2) or confidence intervals that span 1.0 e.g. Study 4 and for these we cannot be sure there is any difference in lung cancer risk between the exposed and not-exposed. However, when all ten studies are combined we see an increase in our confidence of the estimate, so the method has allowed us to move from some uncertainty due to conflicting estimates from different studies to an overall prediction.

Qualitative Synthesis

Qualitative evidence synthesis is an umbrella term increasingly used to describe a group of review types that attempt to synthesise and analyse findings from primary qualitative research studies (Booth et al. 2011). Some researchers have criticised qualitative synthesis suggesting that it is not valid to synthesise non-quantifiable data. Booth (2001) on the other hand has suggested that this can be viewed as ‘insidious discrimination’ via ‘institutionalised quantitativism’. So are qualitative research (data) appropriate for research synthesis?

At one level this is difficult to say, qualitative synthesis are unlikely to use the sort of randomised controlled study designs that that been the mainstay of quantitative synthesis such as those seen in the Cochrane collection. It is also less likely to have been used as the methodology for intervention research so it will tend not to be useful for evaluating the success of interventions. On the other hand qualitative synthesis is becoming increasingly accepted as having a role in the syste matic evaluation of qualitative evidence. Clearly people are complex and not always rational organisms and so qualitative studies can provide us with information that is not amenable to quantitative investigation. If we are going to make use of this information then methods to systematically synthesise this evidence are needed. Increasingly well-established organisations such as the Cochrane collaboration are incorporating qualitative synthesis. Since 2012 the Cochrane collaboration has had a qualitative and implementation methods group that provides resources and training on the use of qualitative synthesis methods.

As with quantitative synthesis we can utilise the same seven steps of synthesising evidence which were mentioned earlier and shown in Fig. 2:

  1. 1.

    Formulating the problem

  2. 2.

    Searching the literature

  3. 3.

    Gathering information from studies

  4. 4.

    Evaluating the quality of studies

  5. 5.

    Analysing and integrating the outcomes of studies

  6. 6.

    Interpreting the evidence

  7. 7.

    Presenting the results

Most of these steps require similar processes for quantitative and qualitative synthesis . However in qualitative synthesis searching the literature can be more problematical. Qualitative research tends to be less well indexed than quantitative research, although this is improving. It is also often found in smaller and less well circulated journals, making obtaining individual articles more difficult and time consuming. Evaluating the quality of the studies will also require different criteria, e.g. the size of the study (in terms of numbers of participants) is less important in qualitative research.

One of the complexities when trying to understand qualitative synthesis is the plethora of methods that have been developed in comparison to quantitative synthesis. Figure 4 shows the variety of methods that have been proposed and a selection of these are discussed below.

Fig. 4
figure 4

Qualitative synthesis methodologies

Full size image

Meta-ethnography

Meta-ethnography was proposed as an alternative to meta-analysis by Noblit and Hare (1988), whereby synthesis is about bringing separate parts together into a whole. It is viewed as a synergistic process so the result of the synthesis is greater than sum of the parts. There are several tools used within meta-ethnography to conduct the synthesis. (1) Reciprocal translational analysis (RTA) requires the ‘translation’ of ideas and concepts between individual studies to identify underlying concepts that are common to the individual studies. (2) Refutational synthesis examines contradictions between individual studies. The synthesis must then take into account the competing explanations. (3) Lines-of-argument (LOA) synthesis involves building up an overall picture of the whole (e.g. organisation, community, culture, etc.) from studies of its separate parts.

Grounded Theory

Grounded theory is a common methodological approach for individual qualitative research studies, but it has also been applied as a technique for qualitative synthesis . The general processes within a grounded theory approach include: simultaneous data collection and analysis; an inductive approach to analysis i.e. allowing theory to emerge from the data; use of the constant comparison method; use of theoretical sampling to reach theoretical saturation; and importantly, the generation of new theory (Barnet-Page and Thomas. 2009). The majority of applications of grounded theory synthesis (Eaves 2001; Kearney 1988 have used the approach because it matches ‘like with like’, i.e. they have used individual papers based on grounded theory approaches to generate a grounded theory synthesis. This is in contrast to meta-ethnography synthesis which seeks to integrate qualitative studies that might have used different theoretical approaches into a single synthesis.

Thematic Synthesis

The aim of thematic synthesis is to combine the approaches used in meta-ethnography and grounded theory (Thomas and Harden 2008). It was developed in order to produce reviews that are able to explore intervention need , appropriateness and acceptability, as well as effectiveness. People’s views and experiences are taken into account and hypotheses are then developed that can be tested against the findings of qualitative studies. There are three main steps to a thematic synthesis: (1) Free line-by-line coding of textual findings from the primary studies; (2) Organisation of the free codes into ‘descriptive’ themes; and (3) Generation of ‘analytical’ themes, using the descriptive themes the analysis develops a new interpretation which goes beyond the original studies.

Textual Narrative Synthesis

This is an approach which takes individual studies and organises them into more homogenous groups. Typically the characteristics of the studies, their context, quality and types of findings are reported using a standard format and then the scope, similarities and differences are compared across studies in order to generate overall conclusions from the data i.e. the synthesis . The technique has been particularly successful in synthesising different types of research evidence e.g. qualitative, quantitative and economic studies. In their comparison of textual narrative synthesis with thematic synthesis (Lucas et al. 2007) found that textual narrative synthesis is valuable for generating future research hypotheses as it is particularly good at identifying gaps in the evidence.

Meta-study

This is a multi-faceted approach to synthesis with three components of analysis that are conducted before the synthesis: (1) Meta-data-analysis (the analysis of findings). This is similar to meta-ethnography as it is interpretive, looking to identify the similarities and differences between study accounts; (2) Meta-method (the analysis of methods) examines the methodologies used by individual stu dies e.g. issues of sampling, data collection, research design etc. This is similar to the procedure of critical appraisal frequently used in quantitative synthesis; and, (3) Meta-theory (the analysis of theory). This involves exploration theoretical assumptions of the different studies included in the synthesis. Meta-synthesis is then required to ‘bring back together ideas that have been taken apart’ and create a new interpretation of the phenomenon under investigation (Ring et al. 2011).

Meta-narrative

Greenhalgh et al. (2005) developed the meta-narrative approach in order to synthesise evidence that would then inform policy-making. Their work was around the diffusion of innovations in health service delivery and organisation, where there was a need to synthesise findings from studies based on different theories and that utilised different study designs. They identified different research ‘traditions’ and sampled studies from each of these. Key features of each tradition were then mapped (e.g. historical roots, scope, theoretical basis; research questions; instruments used; main findings; historical development of the body of knowledge; and strengths and limitations). This exercise generated maps of 13 ‘meta-narratives’ from which seven key dimensions (themes) were synthesised. The approach is relatively new and not yet fully established but offers the potential to examine and synthesise policy-relevant research, such as exploring the success or failure of complex health interventions.

Critical Interpretive Synthesis

Critical interpretive synthesis (CIS) is an adaptation of meta-ethnography and grounded theory techniques. It was developed by Dixon-woods et al. (2006) as they needed to adapt traditional meta-ethnographic methods for synthesis of both quantitative and qualitative data. Dixon-woods et al. (2006) suggest two key features of CIS distinguishes it from conventional systematic review methods (1) it rejects the stage approach to review. Processes of question formulation, searching, selection, data extraction, critique and synthesis are characterised as iterative, interactive, dynamic and recursive rather than as fixed procedures to be accomplished in a pre-defined sequence; and (2) there is an explicit orientation towards theory generation in CIS.

Framework Synthesis

Framework synthesis is based on framework analysis. It is based on the observation that qualitative research produces very large amounts of textual data in the form of transcripts, fieldnotes etc. and that this volume of information presents a challenge for rigorous analysis. To overcome this framework synthesis takes a highly structured approach to organising and analysing the data that has a quantitative ‘feel’ and a deductive approach. It uses a pre-defined ‘framework’ for the analysis rather than developing themes etc. directly from the data. In addition it typically involves numerical index codes and rearranging data into charts etc.

Discussion Task

Look at Fig. 4 and choose 2–3 specific methods for qualitative synthesis . Explore the literature to identify individual studies that have used these methods in a topic that interests you and consider the following:

  • What was the approach and why was it used for this topic?

  • What information was gained from this approach that would not have been obtainable from a quantitative synthesis ?

  • What is the potential relevance for public health/public health intelligence?

Selection of Qualitative Synthesis Method

One of the difficulties with qualitative synthesis , as with qualitative research is the large and increasing number of alternative approaches, which is why Fig. 4 portrays the methods as a Pandora’s Box (there are wonderful things inside, but some risk in letting them all out!). As we have shown in the previous summaries the various synthesis methods all approach collection, review, analysis and synthesis differently. Some of the differences are modest e.g. Critical Interpretive Synthesis shares many attributes of meta-ethnography, and some are more substantial, e.g. Framework synthesis is deductive (testing theory using analysis of the data) whereas all the other approaches are inductive (inducing the conclusions and new theory from the data). There is no ‘correct’ approach to take and many more experienced researchers often tend to stick to the familiar, ignoring the Pandora’s Box of alternatives. Ideally though, we should choose a suitable method based on the purpose of the research, the type of studies that will be examined and the nature of the data that they have produced. But other factors such as the timeframe for the synthesis, resources available, researcher expertise and the audience for the synthesis will also inevitably to be taken into account.

Discussion Task

Though we highlighted some strengths, some questions (adopted from Mays et al. 2005) are still unanswered. You should consider these, for example:

  • Is it always acceptable to synthesise studies?

  • Is it feasible to synthesize disparate evidence?

  • Should reviews start with a well-defined question and how many papers are required?

Conclusion

This chapter has sought to explore the nature of research evidence and how this is synthesised using different synthesis approaches. We have seen that both quantitative and qualitative approaches to synthesis are both possible and valuable. These should be seen as complementary rather than competing, and all feeding into the process of transforming raw data into information and then intelligence that can be used for the development of public health policy and practice. The different synthesis methods are very varied and the choice of method will depend upon whether concepts and theories are clear in advance, the purpose of the work, whether they are generating, exploring or testing theory or advancing understanding; to inform the choice of interventions, or to inform the development and implementation of interventions; and whether data is qualitative, quantitative or mixed.