Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 The Competitive Advantage of Qualitative Methods

Qualitative research methods have been developed several decades ago and have been applied since then to further knowledge in many social scientific disciplines. Numerous qualitative research communities have been actively sharing their expertise and insights in fields that are closely related to HTA such as sociology of health and illness, healthcare management, health policy, and knowledge synthesis. As a result, there exists today a rich body of scholarship that deepens our understanding of the strengths, limitations, and comparative relevance of specific established qualitative data collection techniques (interviews, focus groups, observation) and explores how innovative qualitative approaches could tap on online environments and tools, including social media (Khodyakov et al. 2016; Marques 2009). Building on this diversified and mature scholarship, four chapters in Part II of this book provide readers with clear guidance on the ways in which particular qualitative methods can help HTA practitioners to elicit patients’ perspectives, experiences, and preferences. These chapters also contribute to the science of HTA by making more explicit the epistemological underpinnings of the “patient’s view.” Along these lines, the current chapter critically discusses the kinds of patient-based evidence one may generate through qualitative methods (1), summarizes key lessons from the four chapters (2), identifies methodological challenges that lie ahead (3), and formulates take-home epistemological messages for the consolidation of patient-based HTA (4).

2 What Kinds of Patient-Based Evidence Qualitative Methods Generate?

Almost two decades ago, Murphy and colleagues produced an important and exhaustive monograph on the role of qualitative methods in the HTA. For these authors, there are some problems in HTA (as introduced in Part I of this book), “which cannot be fully resolved using quantitative methods alone, and there are some circumstances in which qualitative methods represent the technically superior option” (Murphy et al. 1998, p. 87). In our view, what provides qualitative research a “competitive advantage” has to do with its key general features, which include a focus on the perspective of the study participants (be they managers, providers, or patients), an in-depth description of the research setting, a holistic view on the phenomena of interest, an emphasis on processes, and a flexible study design that is responsive to emerging findings.

Typically, qualitative research is suited to address “why” and “how” research questions, and it does so by recognizing the centrality of subjectivity in human action: “Health technologies are applied by people (be they doctors, nurses, technicians or patients) to other people (usually patients). One of the distinctive features of a human action is that it is meaningful. People act on the basis of what they believe to be true rather than what may be objectively true” (Murphy et al. 1998, p. 87).

Because their object of inquiry is permeated by subjectivity, qualitative researchers have developed theoretical frameworks and methodological tools in order to handle rigorously their informants’ subjectivity as well as their own subjectivity (better than their quantitative colleagues, we would be tempted to say). Such scholarly advancements have relied not exclusively but often on constructivist epistemologies, which problematize the relationship between the “knower” and the knowledge being produced (i.e., reflexivity), and on social scientific theories, which are necessary to make sense of meanings and social interactions.

Together, epistemological and theoretical frameworks define what count as knowledge, how such knowledge can be produced, what is knowable, and why it should be known. Because “the choice of theory, although often unacknowledged, shapes the way practitioners and researchers collect and interpret evidence” (Alderson 1998, p. 1007), qualitative research that is not firmly grounded in a social scientific framework suffers from severe limitations (perhaps like a quantitative study of poor quality that has insufficient power to detect a statistically significant change).

For Murphy et al. (1998, p. 87), qualitative research brings an important contribution to HTA “whenever the context in which a health technology is to be implemented can be expected to have an impact upon the outcome of that technology.” Qualitative studies can clarify the organizational, political, and sociocultural dimensions that affect the diffusion of technology in different settings as well as its real-world effectiveness. According to the Canadian Health Services Research Foundation, three categories of evidence bring a distinct contribution to health policy: (1) context-free scientific evidence (such as the knowledge generated through a randomized controlled trial on the efficacy and safety of a new technology), (2) context-sensitive scientific evidence (such as the knowledge generated by an implementation study that examines regional variations using a sociology of innovation framework), or (3) colloquial evidence, which refers to the expertise, views, and realities of stakeholders (CHSRF 2006, p. 5). The term “colloquial evidence” was hotly debated in the Canadian health services and policy research community; the idea behind the Foundation’s initiative was to recognize a form of knowledge that often remains informal, but which is very valuable in health policymaking since it sheds light on “resources, expert and professional opinion, political judgment, values, habits and traditions, lobbyists and pressure groups, and the particular pragmatics and contingencies of the situation” (CHSRF 2006, p. 1). By drawing on these three categories of evidence, it becomes possible to develop a more patient-centered HTA.

3 How to Generate Qualitative Evidence for HTA: Key Lessons from the Four Chapters

Each of the four chapters (12, 13, 14, and 15) focuses on a particular qualitative method or approach to elicit and synthesize patients’ perspectives, experiences, and preferences. While Street and Farrell address the emerging and still indeterminate potential of social media, the other chapters bring to the readers’ attention methodologies for which there is now much more scholarship available. Below, we address successively each chapter’s key contributions and critical lessons for the international HTA community to ponder.

3.1 Ethnography Is More than the Sum of “n” Individual Interviews, and Focus Groups are not a Cheaper and Quicker Means to Increase One’s “n”

For Tjørnhøj-Thomsen and Hansen (Chap. 12), ethnography is first and foremost concerned about the patient’s everyday life and context, not just about their discourse regarding a particular service, technology, or policy. Because ethnography pays attention to the social situation in which the use of a technology unfolds (defined by the place, actors, and activities involved), it leads to a detailed understanding of how particular settings influence its real-world effectiveness. Hence, ethnography goes well beyond the knowledge one may obtain from conducting a series of “x” interviews since it is designed to capture holistically the interactions between the technology, patients, and their social surroundings, including providers, insurers, employers, neighbors, family, etc.

These authors also aptly underscore that the nature of the technology—that is, whether its use is lifesaving or not, whether it supports chronic or acute care, or whether it involves small or large trade-offs in one’s personal life—will influence the relevance of using a particular method to gather patients’ perspectives. The tacit desire to develop a “one-size-fits-all” approach to generate patient-centered evidence appears unsound and would divert the attention from a key object of inquiry in HTA: technology’s impact on patients.

While it is true that ethnography is necessarily time-consuming, one of its particular strengths lies in its long duration. Considering that some patients may be “of a few words” but certainly not without intent, expectations, or emotions, the time ethnographers patiently invest in the field is directly proportional to the completeness of the analyses they will be able to achieve. Just like focus groups are not a cheaper and quicker means to increase one’s “n,” we concur with Tjørnhøj-Thomsen and Ploug Hansen when they stress that ethnography is not a simple exercise of collecting or gathering patients’ views, but a research endeavor per se. When critically examining focus group research that we had conducted with patients, we argued that such a method does not “derive epistemological authority simply because of the identity of its participants” (Lehoux et al. 2006, p. 2103). Researchers need to recognize that patients do not arrive “with a logically coherent system of pre-formed ideas that just need to be skillfully elicited or discovered. Nor do they share in any straightforward way their knowledge or naively endorse all knowledge claims put forward by others” (Lehoux et al. 2006, p. 2103).

This is one of the reasons why ethnography must be understood and practiced as a comprehensive research endeavor. Tjørnhøj-Thomsen and Hansen rightly tear to pieces the assumption according to which qualitative research would not require special training and expertise. A rigorous, in-depth understanding of what works and does not work in particular settings is very precious when policymakers and practitioners are looking for ways to adequately implement new technologies that raise similar patient-related challenges. This is why, despite the time it takes, ethnographic research is likely to provide results with a profound and lasting usefulness, thereby supporting the broader mission of HTA.

3.2 Deliberation Aims to Produce More than a Collection of Opinions, and, as Such, It Constitutes a Demanding Process for Patients and Researchers

Street and Lopes (Chap. 13) provide readers with a comprehensive introduction to the use of deliberative methods in patient-centered HTA, from their democratic theoretical aspirations to their ethical and methodological specificities. Deliberation brings the elicitation of patient’s perspectives, preferences, and experiences to another analytical level: what matters is not to collect a range of views, but to bring these views in a collective dialogue (Bombard et al. 2011; Degeling et al. 2015). The goal of deliberation is to reflect on and ponder what may seem like a reasonable collective course of action. This is why Street and Lopes carefully define the principles underlying a “good deliberation” as well as the trade-offs that are associated to various deliberative democracy models.

While a deliberative intervention in HTA may be organized around either a policy or research question, it has to explicitly support the expression of challenging viewpoints and learning opportunities for all participants (Abelson et al. 2010; de Vries et al. 2011). As Giacomini and Cook underscore (2000, p. 480), dialogue “affects the meanings of social experiences, and the results of a dialogue translate these experiences for persons who might not otherwise understand each other’s perspectives well.” Beyond the necessary translation between different perspectives, a dialogue may also prove transformative. A puzzling issue when one seeks to design (and later assess) a deliberative intervention is to clarify what kind of transformation one expects exactly (Carman et al. 2014). For some scholars, participants have to be selected by ascertaining whether they can prove flexible in their thinking or not to hold “too strong views.” This would entail applying certain exclusion criteria, which is obviously an intricate issue that may undermine the legitimacy of the whole deliberative endeavor. Moreover, it might be entirely legitimate that some opinions remain unchanged if, at the same time, mutual learning between participants takes place (Black et al. 2011; Lehoux et al. 2009).

More specifically, Street and Lopes describe five dimensions that may be seen as key ingredients to a deliberation of quality (Table 14.1). They also aptly underscore the need to reach out to “unsuccessful” patients, that is, patients for whom new promising treatments may have failed. Yet, these authors bring to the readers’ attention how deliberative methods may prove physically and/or emotionally burdensome, and this, in itself, creates a formidable tension when the aim is to gather the views of individuals who are already afflicted by health problems.

While deliberative methods are increasingly being applied in the health field, their evaluation “continues to be carried out in the absence of any guiding frameworks that define the theoretical basis for the public engagement process or the relationships among the public engagement mechanism and process or outcome variables of interest” (Abelson et al. 2010, p. 10). For Popay (2014) and the Public Involvement Impact Assessment Framework (PiiAF) Study Group, the “intervention theory” should be made explicit by providing a description of the ways in which a particular approach to involving patients will lead to the expected effects. Such recommendations are likely to further the informed development of deliberative methods in HTA.

3.3 Social Media May Be at Risk of Remaining an Unrealized Opportunity If One Does Not Recognize the Need to Apply Mixed Methods

Street and Farrell (Chap. 14) bring a much-needed contribution by addressing the fast, moving domain of social media-based research, which opens up an array of “(as yet largely) unrealized opportunities.” Among the key arguments for exploring how social media may be used to elicit patients’ perspectives, experiences, and preferences in HTA, one finds the barriers that limit the participation of specific individuals and groups to traditional, face-to-face qualitative data collection methods. Such barriers may be physical, geographical, sociocultural, or a mixture of these. For instance, there are many topics in HTA that raise social desirability or sensitive issues (e.g., sexual health, substance abuse) that require recruiting over a very large area (e.g., rare diseases, discriminatory practices based on gender, handicap, or ethnicity) or that call for the ability to secure collaboration with hard-to-reach groups (e.g., migrants, stigmatized lifestyles). For many observers, and not unlike the hype that surrounded telemedicine in the mid 1990s, social media would easily reduce if not eliminate all of these barriers.

While we clearly share the enthusiasm of Street and Farrell toward the use of online environments in research, a number of contentious methodological issues need to be tackled before social media-based research may deliver its promises. Right at the outset, one has to define more precisely what social media-based research is and what it is not. For instance, social media differ from online surveys and online interviews, which respectively enable the gathering of quantitative and qualitative data. The “quasi” qualitative nature of the data social media may provide access to needs to be acknowledged. For instance, if one may create a “snapshot of views” by aggregating a large number of online posts, such research may fall short of fulfilling the essential qualitative research features we introduced earlier. It would amount to a quantification of qualitative data that may not provide context-sensitive in-depth interpretations.

As Street and Farrell aptly stress, online environments are fluid, they change rapidly, they lack social cues and nuances, and knowing who is talking exactly (for instance, health technology industry-sponsored and/or physician-led patient groups) remains at times an act of faith. One puzzling issue researchers face is to define what a purposeful (or reasoned) “sample” may be when geographical boundaries are made more or less irrelevant and when online identities are made explicitly plural by those using social media to share their views and experiences.

Hence, we would be careful before recommending that “disinvestment policy scenarios” rely on such methods and have doubts that using “recognized” patient associations may increase legitimacy in the eyes of HTA stakeholders. In our view, HTA scholars and practitioners may even have to slow down policymakers’ demand for online tools, especially “in situations where gathering people in an in-person venue is difficult or impractical” (Carman et al. 2014, p. 109). One cannot underestimate current digital divides around the globe and within individual countries as well as the inequalities varying levels of e-health literacy may reinforce. In our view, the risk is that online tools be “used as a standalone, second-best method, which may increase civic inequalities in countries with a geographically dispersed population” (Lehoux et al. 2016, p. 13, Marques 2009).

Because of the volume of data available and their discursive and (self-) representational nature, social media-based research may, to a certain extent, be compared to media coverage analysis. While one may analyze what is said on social media, it remains difficult to clarify why it is said and with what impact on social media users. This is why we believe that it may prove more realistic and productive to apply to social media-based research an integrated mixed method approach that would cohesively combine detailed qualitative interpretations and meaningful quantitative measures. Overall, we disagree with the idea that social media could be used “to conduct relatively fast, inexpensive and feasible” qualitative research. Yet, the methodological challenges raised by social media-based research are likely to be reduced as more scholars study how patients mobilize and make sense of social media throughout their illness trajectories.

3.4 Doing Without Qualitative Evidence Synthesis Is Not Anymore an Option

In Chap. 15 by Booth, one finds an enlightening and state-of-the-art description of the reasons why qualitative evidence synthesis has become an inclusive term and an important tool in HTA. Key distinctions between such syntheses and traditional systematic reviews include their “configurative rather than aggregative” nature, which implies drawing meta-theoretical links between different types of empirical findings, and their bibliographic search strategies, which require a strong command of the specificities of biomedical and social scientific publication databases alike.

While there are “few worked examples” of qualitative evidence synthesis, an increasing number of meta-ethnographies are published every year. This can be understood as a result of the sheer number of publications that are already “out there”—and hard to ignore—and of the natural connection between qualitative research and the “patient’s view.” In other words, HTA practitioners can no longer begin a new study without taking stock of the available published qualitative evidence on the topic. The good news is that synthesis methodologies have made an enormous leap forward in the past decade, exemplified by the international efforts Booth describes and which have produced and shared tools and methodological resources. Such efforts are particularly important since rigor in qualitative evidence synthesis cannot depend upon a hierarchy in study designs.

Interestingly, while each individual published qualitative study may never have been “intended to be generalizable,” a solid synthesis methodology may contribute to increase their scope and policy impact. When reviewing a set of qualitative studies that address a similar topic, it becomes possible to identify divergent and convergent findings across and within different populations and settings. A qualitative evidence synthesis, thanks to its “interpretative richness,” can help to piece together why and how such variations occur. In the quest to support patient-focused HTA, one piece of the methodological puzzle that Booth clarifies very well is the distinction between a synthesis of the patients’ experience of a condition and a synthesis of their appreciation of the outcomes of a particular technology or service. Whereas the former will necessarily be broader in scope and require some theory building (i.e., configurative), the latter may prove much more focused and potentially less time-consuming (i.e., aggregative). Henceforth, for an HTA body not to engage in the production of qualitative evidence syntheses would reveal an ideological rather a methodological decision.

4 What Do Future Methodological Challenges Lie Ahead?

Considering the time and resources constraints that typically plague HTA bodies, one may wonder whether these HTA bodies are able to conduct the kind of qualitative primary research described in Part II of this book. Engaging in the production of qualitative evidence syntheses seems more realistic. Yet, for an HTA body to be able to tap on the competitive advantage of qualitative methods, it has to hire or subcontract researchers who possess such research skills and experience. HTA producers who are generally trained to perform quantitative systematic reviews would have to learn how to read critically qualitative studies and be able to meaningfully extract from social scientific studies evidence that is relevant to patients’ perspectives, preferences, and experiences. They may also have to develop methodological skills in the design of syntheses that integrate both kinds of evidence as suggested by Booth.

When it is not feasible to conduct a synthesis of qualitative evidence, for instance, in the case of emerging technologies, performing qualitative primary research may prove necessary. Preliminary ethnographic fieldwork or other qualitative methods could be indicated in those circumstances. It would provide at the same time a great opportunity to engage patients in the research process itself.

Among the future methodological challenges that the HTA qualitative research community will have to handle is the place face-to-face data collection methods should occupy within a digital world. For instance, deliberative interventions that rely on audiovisual material-based tools may succeed in supporting informed online deliberations among nonexperts (Lehoux et al. 2016; Lehoux et al. 2014). Nonetheless, such online approaches may have to be combined with face-to-face methods in order to foster inclusiveness and meaningful involvement of all participants, thereby maximizing the opportunity to democratically learn from each other and co-produce rigorous patient-centered knowledge (Khodyakov et al. 2016).

4.1 Take-Home Epistemological Messages

Beyond knowing how to choose and apply the right methods, one needs to unpack the epistemological underpinnings of patient-based evidence. As Tjørnhøj-Thomsen and Hansen point out, while patient preferences and patient experiences both reflect patients’ perspectives, they are two different objects of inquiry. Moreover, patients’ perspectives on these objects are “emerging, relational, and shifting.” This observation has tremendous implications since it brings us back to the task of defining what counts as patient-related knowledge, how such knowledge can be produced, what is knowable, and why it should be known. While Booth underscores that a constructivist orientation in qualitative methods may prove less directly relevant to decision-makers, we believe that it prevents from “positing an apparent consensus as the patient’s view” since it explicitly recognizes how such views remain “the result of context-dependent social interactions wherein perplexing dynamics contribute to the creation of dominant narratives” (Lehoux et al. 2006, p. 2103).

As underscored by Street and Lopes, methods to elicit patient’s perspectives, experiences, and preferences are likely to deliver the diversity a deliberative democracy requires. Yet, one then needs to clarify the policy implications of such diversity. Here it is not so much the type of evidence being gathered that is at play, but the relationship HTA entertains with the policy sphere. The credibility and legitimacy of an HTA body that ventures into the generation of patient-based evidence will be increased if it can consistently show a strong command of qualitative methods and, as underscored by Tjørnhøj-Thomsen and Hansen, this is not a mere technical challenge. Rigor in qualitative research requires a strong ability to think qualitatively and therefore to reflexively address the participants’ and one’s own subjectivity.

4.2 Concluding Remarks

In this chapter, we argued that qualitative research methods cannot reasonably be considered an “emerging” topic anymore by HTA producers. The qualitative research scholarship is vast, mature, and dynamically responsive to the digital world in which our societies evolve. Furthermore, when it comes to making sense of patients’ perspectives, experiences, and preferences, it possesses a clear competitive advantage over quantitative methods since it can clarify why, how, and in what context patient-based evidence-based healthcare may flourish.

Yet, like any other specialized scientific endeavor, qualitative methods call for a specific body of knowledge, know-how, and skills. Such methodological expertise has to be rigorously acquired and applied. It often also requires a social scientific “lens”; otherwise, one may be trying to piece together different data fragments without using an explicit, consistent theoretical framework.

We do not believe that there is neither a “quick and fast” way to become a solid qualitative researcher nor a “simple and cheap” way to produce rigorous and non-complacent patient-based evidence. HTA scholars and practitioners should therefore resist ill-informed policy demands for such kind of diluted evidence since they entail a waste of precious human and financial resources. Providing policymakers with instant coffee may temporarily relieve them from a lack of caffeine, but, in the long run, they are likely to miss the very substance that makes a coffee a coffee, including the long and taxing process by which the coffee beans are grown and harvested, packaged and shipped, and roasted and ultimately brewed.

In other words, HTA as a field must stay true to the rich, complex, and at times conflicting realities of patients. Fulfilling this aspiration requires, beyond the necessary allocation of time and resources, a reflexive, theory-informed, and rigorous distillation of large amounts of qualitative data, which will improve our ability to account for technology’s role in patients’ everyday life and context.