Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

Qualitative evidence synthesis, also known as qualitative systematic review, offers a vehicle for presenting patients’ attitudes, beliefs and feelings as originally captured by individual qualitative research studies. By aggregating or integrating views from multiple studies, rather than a single study, the science of systematic reviews takes steps to protect against allowing findings from an isolated study to overly influence our understanding or even to lead us to omit important perspectives. This chapter examines the wide range of uses to which qualitative evidence synthesis can be applied within HTA (Ring et al. 2011a, b) and introduces methods to identify, synthesise and analyse patient narratives from the research literature. The chapter concludes by briefly reviewing methods by which qualitative data might be integrated with quantitative data from an effectiveness review.

The power of a single patient’s voice is not to be underestimated. However, as each individual patient represents a composite of unique experiences, attitudes, opinions and values, there is even greater power to be harnessed from numerous patient accounts collected and interpreted through accepted methods of qualitative data collection and analysis. In short, decision-makers are interested not simply in an isolated perspective nor in a smoothed-out mythical statistical average (as in the mythical family with 2.2 children) but in a wide and diverse range of experiences of a shared phenomenon (Pluye and Hong 2014). This phenomenon of interest could relate to perceptions and experiences of a particular health condition or, equally, attitudes towards a specific intervention. Patients’ perspectives can be accessed via qualitative research. Context-sensitive primary qualitative research can be conducted to address a specific research question. However, the time and resources required to plan and conduct primary research can prove prohibitive. One alternative, used by many health technology agencies, is to harness the collective richness of multiple qualitative research studies within an evidence synthesis. Such an approach starts from an implicit assumption, contested by others, that qualitative research findings may be considered ‘transferable’ (Finfgeld-Connett 2010). In recent years qualitative evidence synthesis (QES) has therefore become a flexible vehicle for collecting and analysing the collective accounts of patients or health service users.

Why is ‘qualitative evidence synthesis’ the preferred term for what has been otherwise labelled as qualitative meta-syntheses or qualitative systematic reviews? In 2011 the co-conveners of the then Cochrane Qualitative Methods Group settled on this term to distance the emerging methodology from the dominant methods of systematic reviews of effects and to signal the potential of this group of methods for a wider range of types of evidence. So, future ‘evidence’ might use these same methods of synthesis to incorporate patients’ perspectives from online bulletin boards or narrated patient real-life experiences collected by interview (Healthtalk 2016).

While the incorporation of more diverse types of evidence remains aspirational, the methodology of QES has enjoyed accelerated wide-scale development. In 1998 a landmark meta-synthesis sought to incorporate perspectives from 43 interpretive research reports of the lived experience of patients with diabetes (Paterson et al. 1998). The review team explicitly sought to extend ‘the analysis of individual research studies beyond individual experience to incorporate dominant system beliefs and health system ideologies’ (Paterson et al. 1998). This intent is shared by many current QES in seeking to produce a more nuanced understanding of how patients interact within the context of health services and the professionals and support staff who deliver those services.

2 Eliciting Patients’ Perspectives in HTA

From their earliest years, QES in health care have offered a vehicle for otherwise disenfranchised patient groups (Warr 2004; Booth 2016). As Toye and colleagues observe: ‘Affirming a person’s experience and allowing an empathetic interpretation of their story is not an adjunct [i.e. optional extra], but integral to care’ (Toye et al. 2013, p. e835). Recent examples of QES within an integrated HTA include those on male obesity (Robertson et al. 2014), teenage repeat pregnancy (Whitaker et al. 2016) and prevention of postnatal depression (Morrell et al. 2016). The last two of these HTA reports not only include qualitative synthesis components but extend to ask ‘what works for whom under what circumstances’ using a specific methodology, realist synthesis.

Health systems place increasing emphasis on the design and delivery of services that are ‘patient focused’ (Hansen et al. 2011). In response to this imperative, commissioners of HTA, and of health services research more generally, may commission a review team to undertake robust secondary research to understand the diverse experiences and perspectives of patients that can be assessed alongside quantitative evidence of clinical and cost-effectiveness to inform health policy and clinical decision-making. In addition, QES addresses contemporary concerns about possible research waste.

Patient participation can also constitute a topic for QES in its own right. QES of patient participation has examined patients’ feelings about ward nursing regimes (Alexander 2006), participation in nursing care on medical wards (Tobiano et al. 2015) and shared decision-making in palliative care (Bélanger et al. 2011).

Mixed methods approaches to synthesis remain in their infancy; mixed methods reviews may summarise quantitative (i.e. from a review of randomised controlled trials) and qualitative (i.e. within a QES) data separately and then seek to integrate the two types of evidence, or, alternatively, they may seek to review only mixed methods primary studies (Heyvaert et al. 2016). In an example of the former, Gagnon and colleagues have demonstrated that patient or public perspectives could add important dimensions to the evaluation of health technologies, while cautioning of a need for more systematic approaches to considering patient and public perspectives in HTA (Gagnon et al. 2009).

3 Choosing an ~Appropriate Method of QES

Seven factors are important when selecting an appropriate method of QES (Booth et al. 2016). These factors, identified from the literature, can be organised under the mnemonic RETREAT (Research Question, Epistemology, Time, Resources, Expertise, Audience and Purpose, Types of Data) (Box 15.1). These are considered in turn.

Box 15.1 Considerations When Selecting a Method of Qualitative Evidence Synthesis (RETREAT Mnemonic)

Review Question

Epistemology

Time

Resources

Expertise

Audience and Purpose

Type of Data

A key consideration when selecting a method of synthesis relates to the nature of the Research question [R]. Will the research question share the same scope as an associated effectiveness question, is it complementary or does it have a wider ambition? Observers comment on two particular characteristics of question formulation for qualitative reviews; first, the review question is more a ‘compass’ rather than the ‘anchor’ associated with effectiveness reviews (Dixon-Woods et al. 2006b). As the HTA review team follows up initial leads, they may unearth further lines of inquiry—in a similar way to inquiry in primary qualitative research. Secondly, an HTA review team may be interested in qualitative data that extends beyond the experience of an intervention, particularly as a health technology may be novel and qualitative research scarce. The QES may have to examine patient experiences of a condition, both with and without any intervention, and may consequently be broader in scope than the effectiveness question (Lorenc et al. 2012). The case study on antimicrobial wound dressings in Chap. 27 is one such HTA example where the literature search had to be broadened beyond the original review question in recognition of a shortage of evidence. Where the QES shares broadly the same scope as an effectiveness question, the HTA review team can use an aggregative method of synthesis (e.g. meta-aggregation or thematic synthesis without theory generation). However, if an HTA review team seeks to explore qualitative aspects of an intervention at a more theoretical, conceptual level, then they may prefer the greater freedom offered by an interpretive method such as meta-ethnography.

Epistemological concerns [E] may hold comparatively lesser weight within a pragmatic health technology assessment than they do in a thesis or similar academic work. Is the type of knowledge being generated by the QES a generalisable theory, or is it to be confined to specific points for implementation? The HTA review team must stay sensitive to the epistemology that underpins each methodology when making a selection (Barnett-Page and Thomas 2009).

Of more immediate concern for a review team when selecting a method for performing a QES within the context of HTA is the triad of Time, Resources and Expertise [T,R,E]. HTAs are frequently conducted within severe time and resource constraints. HTA teams are commonly assembled from an existing pool of staff within an institution. Less ambitious, and more easily acquired, methods of synthesis, such as meta-aggregation, thematic synthesis and framework synthesis are more amenable to a rapid approach. Realist synthesis accommodates its own specific rapid variant—labelled rapid realist synthesis (Saul et al. 2013). As HTA agencies trade timeliness against rigour, a greater range of rapid QES variants is likely to be developed.

Key within an HTA context are considerations of Audience and purpose [A]. Decision-makers favour methods that yield a clear link between findings and subsequent recommendations. Barnett-Page and Thomas observe that ‘the output of some methods of synthesis (Thematic Synthesis, textual Narrative Synthesis, Framework Synthesis, and ecological triangulation) is more directly relevant to policymakers and designers of interventions than the outputs of methods with a more constructivist orientation…which are generally more complex and conceptual’ (Barnett-Page and Thomas 2009, p. 9).

A final logistical consideration relates to the Types of data [T] to be synthesised. Interpretive methods such as meta-ethnography require data that is conceptually rich and contextually thick. Where qualitative data sources offer minimal data, as in thin case study reports published in professional journals, the potential to undertake a more interpretive exploration is compromised. Thin data is unlikely to be able to sustain meta-ethnography. More superficial approaches, such as thematic synthesis, may be indicated as appropriate alternatives.

Given the potentially bewildering variety of choices, the most practical advice is for an HTA review team to settle for thematic synthesis where the topic is poorly theorised or where there is little consensus on prevailing theory. Thematic synthesis offers additional merit given that thematic synthesis is a precursor to meta-ethnography, and so this remains an open option should data prove rich and thick enough to sustain this more interpretive process. Alternatively, where a field is well theorised and one or more frameworks receive widespread recognition, this becomes an indication for choosing a framework synthesis (Dixon-Woods 2011). Further details on selection of an appropriate QES methodology within HTA are available from the free online INTEGRATE-HTA guidance on this topic (Booth et al. 2016).

4 Undertaking a Qualitative Evidence Synthesis

While great variety exists in the overall methods available for qualitative synthesis, Garside (2008) demonstrates that nine phases are common to most types of synthesis (Table 15.1). Individual methods vary in the precise sequencing of these phases and the degree of iteration required by each method.

Table 15.1 Comparison of the phases of a qualitative evidence synthesis and a systematic review
Full size table

The first phase of undertaking a QES parallels that for a quantitative systematic review in requiring (1) development of a clearly formulated review question. Whereas those conducting effectiveness reviews favour the PICO (Population, Intervention, Comparison, Outcomes) format, those conducting qualitative synthesis find it helpful to adopt a more relativist ‘lens’ (Stern et al. 2014). One question format that is gaining in popularity in QES is Setting, Perspective, phenomenon of Interest, Comparison, Evaluation (SPICE) (Riesenberg and Justice 2014).

For the above reasons, (2) scoping becomes a prerequisite second phase before undertaking the actual review itself. Such scoping may involve identification of ‘clusters’ of related studies that can be forensically pursued in order to add thicker contextual detail and a richer conceptual understanding (Booth et al. 2013b). Data sources may include pilot studies, feasibility studies and process evaluations as well as ‘sibling’ qualitative studies that run alongside a higher-profile trial. Health Services Research PubMed Queries (https://www.nlm.nih.gov/nichsr/hedges/search.html) offers a rapid search facility for scoping qualitative research topics or related topics of appropriateness, process assessment or quality improvement, using a choice of either broad sensitive or narrow specific search filters.

Once the HTA review team has articulated the review question and set its conceptual, logistical and terminological limits, the team proceeds to (3) formal identification of relevant literature (Finfgeld-Connett and Johnson 2012). While determining the actual population of studies for inclusion is no less important than for an effectiveness systematic review, the underlying rationale may be markedly different. An effectiveness review seeks to minimise bias by assembling as comprehensive sample of the existing studies as resources allow. However, for a QES, an HTA review team wishes to gain a holistic understanding of the phenomenon of interest. The intent is configurative, rather than aggregative. To illustrate, an effectiveness review often seeks to demonstrate that an intervention is effective on average for a general population. For a qualitative synthesis, an HTA review team may be equally interested in those who find an intervention unacceptable or those who receive less than the expected benefit from the intervention. This interest in the ‘disconfirming case’ alongside other sources of variation opens up a full array of methods of sampling from qualitative research (Benoot et al. 2016; Suri 2011).

The unrivalled coverage of MEDLINE makes it a first port of call for most qualitative synthesis questions (Booth 2016). Admittedly retrieval of qualitative research often proves more challenging given such factors as limited indexing, non-indicative titles and abstracts (Dixon-Woods et al. 2006a) and the sheer predominance of quantitative studies. CINAHL, with its focus on literatures where qualitative research is more accepted together with its inclusion of theses and dissertations, is also considered a primary source (Subirana et al. 2005). EMBASE, PsycINFO, Sociological Abstracts and Social Sciences Citation Index (Web of Knowledge) also feature prominently in QES search methods. When searching the UK literature, these may be augmented by country-specific databases such as ASSIA and the British Nursing Index and the Index to Theses (Stansfield et al. 2012). This may be equally true for other geographic regions. Predesigned filters exist for retrieving qualitative research studies from the four main international databases: MEDLINE (Wong et al. 2004), EMBASE (Walters et al. 2006), CINAHL (Wilczynski et al. 2007) and PsycINFO (McKibbon et al. 2006). However, it may be equally useful to use hedges of key terms associated with a particular perspective or phenomenon such as patient involvement (Resource 2016b) or quality of life (Resource 2016a). In several cases, a short list of qualitative terms has been found to perform comparably to a more expansive list, possibly because multiple retrieval terms often occur in the same abstract (Flemming and Briggs 2007; Gorecki et al. 2010). However, this requires testing across a greater range of review topics and literatures.

Certainly, it is important not to rely too much on conventional subject searching of bibliographic databases but to use numerous supplementary techniques such as backward and forward citation searching, handsearching of relevant journals such as The Patient; Health Expectations; Value in Health; Social Science and Medicine; Culture, Medicine, and Psychiatry; Research Involvement and Engagement; Anthropology and Medicine; and Sociology of Health and Illness and contact with authors and experts (Papaioannou et al. 2010; Greenhalgh and Peacock 2005). Websites of national patient organisations may also yield useful information. The case study in Chap. 27 on antimicrobial wound dressing offers a good example where reliance on subject searches on bibliographic databases alone would have seriously degraded the HTA response.

The fourth phase involves (4) an initial assessment of study reports. After preliminary reading and re-reading, the QES team forms a picture of the literature and how it is structured. Theories, either explicitly stated or implicitly referenced, start to become apparent (Booth and Carroll 2015b). Such conceptual frameworks may become a useful vehicle for data extraction through framework synthesis (Booth and Carroll 2015a).

Next, (5) analysis and synthesis takes place. Constant comparison is used to identify patterns and similarities across reports. Refutational findings must be reconciled (Booth et al. 2013a). At this point quality assessment may be undertaken, either using a single generic assessment tool or checklist or a battery of checklists designed for individual types of study (Carroll and Booth 2015). The review team considers the extent to which the synthesis and its findings are based on robust qualitative studies (Carroll et al. 2012).

Preliminary synthesis (6) involves organisational procedures such as categorising, tabulation and the creation of mind maps. The review team explores relationships both within and between studies. Full synthesis (7) may, in its simplest form, be achieved through a process of thematic synthesis or, with greater interpretive complexity, through translation of concepts and metaphors as undertaken for meta-ethnography. Meta-ethnography seeks to interpret studies rather than simply aggregating them, with the intent being to generate a new theory or ‘line of argument’ to explain all the studies (France et al. 2014).

Considerations of the intended audience subsequently inform the methods chosen for (8) dissemination. Exploratory methods of presentation include idea maps and concept maps (Popay et al. 2006). The review team assesses the strengths and limitations of the review itself and of the body of included studies. Optimally, all stakeholders are consulted so that emerging findings become an organic product of knowledge co-creation. However, it is not unexpected to find that stakeholders are not able to recognise the synthetic findings from the interpretive process in their entirety as they often possess only a fragmented, yet valid, perspective. Essentially, therefore, a review team is substituting the authenticity of a single participant’s view of the phenomenon with a more overarching interpretive account that attempts to identify and reconcile multiple perspectives.

Throughout the process (9), the multidisciplinary team brings together their different perspectives not for consensus, as is the case for multiple reviewers in an effectiveness review, but more for divergence and interpretive richness (Booth et al. 2013a). Reflexivity, the facility of qualitative researchers to consider the impact of their own role as researchers on the synthetic process and resultant product, is surfaced and discussed (Newton et al. 2012). Notwithstanding the iterative and recursive nature of the qualitative synthesis, it shares the requirement of systematic reviews more generally to document methods and decisions to increase confidence in the findings (Benoot et al. 2016).

Recent years have seen attention focused on an additional stage in the QES process, to make such reviews even more comparable to effectiveness reviews, namely, the production of assessments of qualitative findings. The GRADE-CERQual subgroup has developed a four-component approach that assesses individual review findings for adequacy, coherence, methodological limitations and relevance (Lewin et al. 2015). Assessments of the findings from a QES are designed to parallel the strength of findings tables produced for GRADE assessments, whereby quantitative findings have previously been assessed against four corresponding components. Limited examples exist of the use of this CERQual approach within current HTA processes (Morrell et al. 2016; Whitaker et al. 2016), but proof of concept has been demonstrated for Cochrane and WHO systematic reviews.

5 Integration of Quantitative and Qualitative Data

Finally, integration of quantitative and qualitative evidence allows a team to produce evidence products to inform complex HTA problems. Approaches to integrating patients’ perspectives with effectiveness data can utilise one or more of seven potential mechanisms:

  1. 1.

    Use a review methodology designed to handle both quantitative and qualitative data (i.e. integration at a methods level). Realist synthesis seeks to identify and then explore configurations of context, mechanism and outcomes for those circumstances under which an intervention or programme is likely to work well and those under which it may perform suboptimally (Rycroft-Malone et al. 2012). The HTA review team may also extract such configurations from the introductory or discussion sections of randomised controlled trials or from qualitative or process evaluation data. Critical interpretive synthesis reviews a purposively sampled selection of literature to examine how the literature has problematised a particular phenomenon. More broadly meta-narrative review examines how a particular concept has been characterised within different paradigms and disciplines. Essentially all three methodologies seek to reconcile the quantitative and qualitative literatures within an overarching narrative.

  2. 2.

    Use an external conceptual framework, typically identified from a parallel search process specifically for theory, as a structure by which to bring together qualitative and quantitative data. This framework may be specific to the topic of the review, may be a ‘best-fit’ framework that matches against several critical characteristics of the topic or may be a meta-framework that fuses together multiple models or frameworks (Booth and Carroll 2015a).

  3. 3.

    Use an internally generated framework derived from consultation with stakeholders (Oliver et al. 2008) or a simple matrix that places themes from the qualitative literature alongside outcome domains from the quantitative studies (Candy et al. 2011; Millar et al. 2012).

  4. 4.

    Use a programme theory, also an essential feature of the realist synthesis methodology in (1) above, against which the review team maps various features of the quantitative and qualitative literature to ‘chains’ of causation.

  5. 5.

    Related to (4) above, construct a logic model as a framework against which data is mapped and then analysed. This is an atheoretical variant of the framework method in (2) above (Baxter et al. 2014).

  6. 6.

    Perform subgroup analyses to bring quantitative and qualitative data together for particular subgroups.

  7. 7.

    Use quantitative and qualitative techniques sequentially, rather than in parallel. For example, Bayesian synthesis uses qualitative evidence to identify important factors associated with an intervention and then the quantitative evidence to explore their relative effects (Roberts et al. 2002). Alternatively, qualitative comparative analysis involves using truth tables to explore the internal logic by which factors identified qualitatively may exert an influence, as presented in the quantitative data (Thomas et al. 2014; Brunton et al. 2014).

Many methods for integrating quantitative and qualitative data remain tentative with few worked examples, and a considerable agenda persists for empirical testing. Currently, when QES has been undertaken in an HTA, the HTA report generally includes the QES as a separate stand-alone chapter thus sidestepping methodological difficulties. Nevertheless, it is clear that considerable potential for enhanced integration of quantitative and qualitative data exists, and this remains a major methodological challenge over the next few years.

6 Discussion

As is apparent from the above consideration, qualitative evidence synthesis is one of the fastest growing areas of research synthesis methodology. Particular drivers for this growth include increased recognition of the complexity of decision problems and increasing acknowledgement of the complexity of many human-mediated technologies. Both of these drivers are particularly relevant in the context of patient involvement in HTA. Valuing the patient experience requires incorporation of patient values and perspectives in the decision-making process. Recognising that the effectiveness of many health technologies is mediated by multiple factors related to the patient-clinician interaction makes it critical to explore such relationships more thoroughly.

Not to be overlooked is a vital role that patient and public involvement can play in improving the design and analysis of qualitative evidence syntheses, as for systematic reviews more generally (Harris et al. 2015; Boote et al. 2012; Oliver et al. 2015). Many considerations regarding the timing, extent and nature of patient involvement in HTA are shared by quantitative and qualitative systematic reviews alike.

As with other methods of synthesis, QES is limited by the quality of reporting of primary studies. Further limitations relate to whether the primary research questions of the included studies map exactly to the review question or whether the primary studies only yield incidental insights. For many commentators, particularly those who are more familiar with the quantitative paradigm, the degree of interpretation is a source of discomfort – the HTA review team is in effect offering interpretations (by the team) of interpretations (by the primary authors) of the experiences and perspectives of research participants.

Innovations in grading of recommendations using the GRADE-CERQual system for qualitative evidence syntheses (Lewin et al. 2015), envisaged as comparable to, and potentially integrated with, the GRADE system for effectiveness studies, offer further opportunities for incorporation of synthesised patients’ perspectives within HTAs, health-care policy and decision-making.