BACKGROUND

To work well, research needs to address questions that are relevant to patients, physicians, and other health decision makers. While the US research enterprise produces new evidence in great volume,1 much of this evidence has been difficult to implement in practice.2 Clinical and health services research has been found wanting because of differences between settings where research is conducted and settings where medicine is practiced;3 for failure to report how treatment effects vary in individual patients and subgroups;4–6 and for the under-representation of women, children, racial and ethnic minorities, and patients with co-morbidities.7–9 Although researchers may prefer to see their work being used in practice, the presumed link between publication and application of research has not been especially strong and is in need of reinforcements.

There is widespread agreement that better stakeholder engagement can help to address this need. Advocates for comparative effectiveness research (CER) and patient-centered outcomes research (PCOR) have been especially strong proponents of this recommendation, on the basis that stakeholder engagement could improve the relevance of research, increase its transparency, and accelerate its adoption into practice.10–14 These entreaties could result in a new era of stakeholder-engaged research, and an important benchmark for patient-centered research in future years may be that it “is useful to clinicians and patients—and is used.”15

A NEW TAXONOMY

To date, however, no common taxonomy exists to guide researchers and stakeholders into a new era of stakeholder engaged research. We set out to develop such a taxonomy by offering a definition of “stakeholder” and “engagement,” and by addressing three key questions: (1) Who are the stakeholders in PCOR and CER? (2) What roles and responsibilities can stakeholders have in PCOR and CER? (3) How can researchers start engaging stakeholders?

We developed this taxonomy by following an iterative process of drafting and vetting definitions, key questions, and content. The first three drafts and reviews were conducted internally by co-authors to address the key questions, until a complete fourth draft was prepared for an external review panel (Table 1) composed of representatives from a comprehensive array of stakeholder groups. A fifth draft was reviewed by publication committees in the Centers for Disease Control & Prevention, the Clinical and Translational Science Awards program of the National Institutes of Health, and the Centers for Medicare & Medicaid Services (see “Other Disclosures”).

Table 1 Stakeholder Review Panel
Full size table

We developed the following definitions to guide our work:

Stakeholder:

An individual or group who is responsible for or affected by health- and healthcare-related decisions that can be informed by research evidence.

Engagement:

A bi-directional relationship between the stakeholder and researcher that results in informed decision-making about the selection, conduct, and use of research.

Who Are the Stakeholders in PCOR and CER?

Researchers may find it challenging to envision the engagement of all groups with a stake in clinical, health services, or health policy research. The “7Ps Framework to Identify Stakeholders in PCOR and CER” (see Table 2) was developed to assist with this challenge. The 7Ps framework identifies key groups to consider for engagement. The first, patients and the public, represents the current and potential consumers of patient-centered health care and population-focused public health. The second is providers, including individuals and organizations that provide care to patients and populations. Purchasers, the individuals and entities responsible for underwriting the costs of health care, such as employers, make up the third group. The fourth group consists of payers who are responsible for reimbursement of medical care, such as insurers. The fifth is composed of public policy makers and policy advocates working in the non-governmental sector. Product makers, representing drug and device manufacturers, comprise the sixth group, and principal investigators, or other researchers, make up the seventh.

Table 2 The 7Ps Framework to Identify Stakeholders in PCOR and CER
Full size table

We recommend a flexible application of this framework, using it as a guide but not a strict formula for decisions about whom to engage; all groups may not have a stake in every research question. Any stakeholder may be responsible for or have an interest in several types of health decisions, and therefore the categories are not meant to be strictly exclusive of each other. For example, some purchasers are also payers and some payers provide care. Stakeholders may have an interest individually or as representatives of an organization. Patients and their advocates may also be providers or employers with policy-making responsibilities. Overlap may be inevitable, but care should be taken to ensure that stakeholders’ multiple roles do not create unacceptable conflicts of interest. Conflict of interest16 and conflict between those with competing interests17,18 are topics that have been explored elsewhere in great detail. Research teams and their stakeholders may need formal processes to deal with conflicts even after full disclosure, and resources are available to help with this.16,19

To determine which groups have a stake in a particular research project, several questions might be considered: (1) What topic(s) does the research address? (2) What health care decision(s) is the research meant to inform? (3) Who are the decision makers responsible for these decision(s)? (4) Who are the individuals and groups that are affected by these decisions? It may be obvious that health decision makers are key stakeholders in health care research. Stakeholders who are not primary decision makers but have a direct interest in the selection, conduct, or use of research also warrant consideration. For example, purchasers and payers are decision makers when it comes to insurance coverage. However, providers, patients, and product makers have a direct stake in these decisions and are therefore important participants in the research that informs them. Research that addresses methods questions may, incidentally, also benefit from stakeholder input, and this is a primary reason to include other researchers (“principal investigators”) as stakeholders in PCOR and CER.

When assembling a team of stakeholders, it is important to consider balance between groups with different or competing interests. Stakeholders with a commercial interest in the outcomes of research—including product makers, specialty providers and some payers—should represent at most a minority in evidence prioritization activities, and they might best be excused where their commercial interests are a factor. Forethought should be given to the appropriate balance between stakeholder groups that are involved in providing or using health care services and those that are involved in paying for them. Since the informational needs of primary care physicians may be distinct from those of specialists, thought might also be given to balance between physician groups.

What Roles and Responsibilities Can Stakeholders Have in PCOR and CER?

Stakeholder engagement in PCOR and CER is a multi-dimensional challenge. First, there is substantial diversity in the types of research that are included in PCOR and CER, and the meanings of these terms may evolve as new public and private research programs are developed. To begin exploring how to define the roles and responsibilities of stakeholders, researchers might consider framing PCOR and CER projects according to a six-stage model developed by the Tufts Clinical and Translational Science Institute (CTSI) (see Fig. 1).20 The model illustrates both a sequential flow from evidence prioritization to feedback and assessment as well as a cyclical, iterative process. Each stage is an activity that may be carried out by researchers, as illustrated by the light-shaded oval. Researchers are surrounded by stakeholders, as illustrated by the dark-shaded oval. Although some research projects, such as meta-analyses, may be focused primarily on one stage, most research projects involve multiple stages.

Figure 1.
figure 1

The Six-Stage Model for PCOR and CER. This model illustrates six stages in the translational spectrum of comparative effectiveness research (CER). Each stage is an activity that may be carried out by researchers and research organizations, as illustrated by the light-shaded oval. Researchers and research organizations are surrounded by stakeholders, as illustrated by the dark-shaded oval. The model illustrates both a sequential flow from evidence prioritization to feedback and assessment as well as a cyclical, iterative process. *©2012 Tufts Clinical and Translational Science Institute

Full size image

Second, stakeholder roles will vary by expertise; stakeholders may be invited to serve as co-investigators on a community-based participatory research project,21 as consultants, members of an advisory board, expert panel, or other committee. Clinical and methodological expertise can help with some research activities while experience as a patient, patient advocate, community member, provider, or other group can help with others.

Third, engagement strategies must be responsive to the values and interests of each stakeholder, no easy task for research projects that depend on relationships with many. A central consideration involves the strategies that will be used to support the participation of patients, patient advocates, and the public; their full participation is not guaranteed by issuing an invitation to sit on a board.22 General principles can be drawn from community-based participatory research, which underscores that engagement is a relationship-building process.23 Researchers and stakeholders should be committed to the process at the outset; neutral and expert facilitators should be used to guide research discussions; connections among stakeholders should be encouraged; and an environment of mutual respect should be fostered. Mechanisms for engaging stakeholders may include a modified Delphi or similar process, concept mapping, asynchronous web-based input, focus groups, surveys, computer games, and others, but the choice of mechanism must be aligned as best possible with stakeholder preferences.

The multi-dimensional nature of this challenge suggests that findings about engagement in one stage of research, for one research program, or with one type of stakeholder will not likely generalize to every other case. Although notable exceptions exist,24,25 stakeholder-engaged research is still nascent, and experimentation is needed to develop approaches that can be customized to individual research programs. The 7Ps/Six Stage taxonomy is a helpful tool for outlining the breadth and extent of experimentation that is needed.

Below, we illustrate some of the topics that need further exploration within each stage of research from Figure 1.

Evidence prioritization (stage 1)

This stage of research includes establishing a vision and mission for research, identifying topics, setting priorities, and refining questions. Federal research programs have reported on substantial work in this area, including the Agency for Healthcare Research and Quality (AHRQ) Effective Healthcare Program,24,26 the Institute of Medicine (IOM) report Initial National Priorities for Comparative Effectiveness Resaerch,10 and the Patient-Centered Outcomes Research Institute Draft of National Priorities.27 Individual research organizations have also reported on evidence prioritization activities.25 Continued study can improve methods to synthesize viewpoints from a diverse range of stakeholders in the nomination, combination, ranking, and selection of research priorities.

Evidence generation (stage 2)

Evidence generation involves a variety of methods including clinical trials,28 practical trials,29,30 and observational studies.31,32 Some but not all of the engagement activities in this stage depends on complex conceptual or methodological expertise. Some stakeholders may not be able to contribute to the technical aspects of study design, but any stakeholder can contribute to the relevance and transparency of new evidence generation. Insight is needed on how to engage stakeholders in developing hypotheses, selecting methods, recruiting research participants, testing hypotheses, and interpreting findings.

Evidence synthesis (stage 3)

Evidence synthesis refers to the systematic review of research to provide an assessment of what is known, what is not known, and what methods have been used. Syntheses may identify which clinical or health care delivery intervention is best, for whom, and under what conditions. Many organizations and individuals are working on quality standards in this area,33–35 and AHRQ recently released a detailed report examining engagement in developing priorities for evidence synthesis.24 Continued exploration of engagement in the conduct and assessment of reviews is needed.36

Evidence integration (stage 4)

This stage integrates clinical, behavioral, economic, and systems evidence in decision analysis, simulation modeling, cost-effectiveness analysis, and other methods. Some but not all engagement activities in this stage may depend on expertise. Stakeholders may be engaged in framing research questions, identifying appropriate data inputs in a simulation, developing preference and contingent valuation surveys, interpreting findings, and communicating results.

Dissemination and application (stage 5)

Dissemination and application include the active distribution of research findings to decision makers, as well as the adoption and implementation of research findings in real-world settings. It has always been a goal of evidence-based medicine to account for patient preferences, and some researchers have begun to include patient preferences formally in guideline development processes.19,37 Further experimentation is needed on how to engage stakeholders in developing guidelines and decision aids, comparing interventions and strategies at the front lines of care, assessing findings, developing communication strategies, and serving as ambassadors for high-integrity evidence.

Feedback and assessment (stage 6)

Research will benefit from an active review and revision of its processes over time.38 Although there is considerable experience with community engagement in public health and community-based research, the inclusion of stakeholders in PCOR and CER is in its infancy. Stakeholders can offer feedback regarding their participation, including on mechanisms for engagement, intensity of engagement, and support throughout the process. Similarly, researchers may have feedback for stakeholders and funders.

Recommendations—How Can Researchers Start Engaging Stakeholders?

We developed this taxonomy to help researchers identify stakeholders and to outline the breadth and depth of experimentation that is needed to support engagement across all stages of research. Which categories of stakeholder are engaged and the exact path toward meaningful engagement will not be uniform across every research institution and project. Our recommendations for the path forward, therefore, follow a plan-do-study-act approach to quality improvement:

1. Prioritize engagement and adopt a common taxonomy (plan)

An evaluation of recent federal spending on comparative effectiveness research noted a shortfall in spending on dissemination, translation, and stakeholder engagement investment.39 Research funding needs to account for the costs of implementing meaningful engagement activities, and funders could incorporate the 7Ps framework and six-stage model taxonomies into funding opportunity announcements. Investigators might consider how the roles and responsibilities of stakeholders change throughout the six-stage model. Stakeholders themselves might be encouraged to develop familiarity with these taxonomies.

2. Experiment with alternative strategies (do)

After adopting a common taxonomy, the next step is to begin experimenting. The challenge associated with knitting investigators and stakeholders together in bi-directional relationships across the full spectrum of research activities is enormous, akin to building from scratch “the infrastructures of integrated healthcare systems.”40 To get started, research institutions, funders, and individual investigators need a general roadmap, but the exact path forward must be customized to individual research programs.

Research institutions can begin by starting with an internal scan of stakeholder engagement activities in current or recent programs. One way to conduct such a scan is to identify funded grants from public and private research programs that have some form of stakeholder component. Grantees of AHRQ’s Effective Healthcare Program, for instance, have experience gathering stakeholder input in the nomination and development of research questions,24 and the national program supports dissemination of findings through its John M. Eisenberg Center for Clinical Decisions and Communications Science.38 Grantees of the NIH Clinical and Translational Science Award (CTSA) program41 have invested in community engagement that is focused on all types of translational research. The Centers for Disease Control and Prevention fund community-based prevention research through the Prevention Research Centers (PRC) Program, including four ARRA-funded awards with a specific focus on CER. The Veteran’s Administration (VA) funds stakeholder-engaged research through its Quality Enhancement Research Initiative (QUERI) program, whose mission is to “enhance the quality and outcomes of VA health care by systematically implementing clinical research findings and evidence-based recommendations into routine clinical practice.”42 Institutions and investigators with Community-Based Participatory Research (CBPR) grants have experience enabling community residents to participate actively in research, and these projects are funded by a range of agencies within the Department of Health and Human Services.43–45 These are a handful of examples, and there are many more.11

Investigators and research groups may consider forming a project-specific or program-wide stakeholder board using the 7Ps framework. Project-specific boards can be tailored to the needs of a specific research question. A program-wide board is unlikely to be focused on the details of individual research projects, but it could become a helpful building block for new research projects. A typical first task for a board of this kind would be an evidence prioritization process (stage 1). New funding for PCOR and CER may introduce new requirements for stakeholder engagement,46,47 but investigators do not have to wait for these requirements before including plans for engagement in proposals, establishing advisory panels, initiating other engagement strategies, and describing stakeholder contributions in publications.

3. Evaluate alternative strategies (study)

New efforts are meaningless without a follow-up effort to study them. Funders and investigators can begin immediately to identify appropriate intermediate and long-term benchmarks for evaluating the effectiveness of engagement, keeping in mind that the optimal organization and roles of stakeholders will vary by institution and project. Future research might consider whether and what kind of stakeholder engagement leads to informed decision-making and improved uptake of research evidence into practice.

4. Report on outcomes, implement changes as needed, and iterate (act)

Investigators can report stakeholder activities in manuscripts and contract reports. Mainstream journals can publish quantitative and qualitative research on the topic, to begin establishing an evidence base across various settings. As the evidence base grows, funders, research institutions, and investigators need to implement changes in their research programs. As changes are adopted, an iterative assessment process should follow.

CONCLUSION

Perhaps the most important benefit of an active stakeholder engagement program is its potential to move research evidence off of bookshelves and into practice. If bi-directional relationships are sustained over time, stakeholders can serve as ambassadors for high-integrity evidence even where the findings are contrary to generally accepted beliefs. The purpose of PCOR and CER is to assist patients, providers, and others to make informed decisions. To accomplish this, researchers must begin to engage the full range of stakeholders in all stages of research.