FormalPara Key Points for Decision Makers

Including core outcome sets (COS) in clinical development programs and health economics and outcomes research will ensure that consistent outcomes are available for healthcare decisions and should result in faster access to high-value and novel therapies for patients.

Patient-centered COS development is essential to ensure that research captures patient-relevant outcomes and that regulatory, market access, and clinical decisions are made using patient-relevant outcomes.

COS development requires careful study design to overcome barriers to patient engagement. Engagement approaches might need to be adapted based on the therapeutic area of interest and patient population.

1 Introduction

A core outcome set (COS) is a minimum agreed-upon set of outcomes that should be measured and reported in all clinical trials for a specific condition [1]. Consistent use and reporting of core outcomes can improve the quality of clinical studies and health economics and outcomes research (HEOR) by providing transparency around which outcomes are expected for regulatory and post-regulatory decision making [2]. In addition, if patients are involved in the development process, the use of COS can ensure that outcomes are relevant to those who receive or prescribe treatments. Use of a COS helps to prevent bias in reporting study results [3], provides consistent data for systematic reviews [4], and allows for comparison of therapies to answer questions about the (cost) effectiveness of interventions.

The utility of COS is increasingly recognized, with calls for COS development across multiple disease areas [5]. Methods for COS development are being refined to include all relevant stakeholder groups. Where previously outcome recommendations were frequently made by a panel of experts, today patients are included as the experts on living with their disease [6]. The inclusion of patients in this work aligns with the goals of the 21st Century Cures Act [7] and recommendations for increasing patient involvement in drug development [8]. This method of incorporating patient values “upstream”, i.e., before pivotal trials begin, increases the patient centeredness of evidence available for further HEOR and across decision-making contexts (regulatory, coverage and reimbursement, patient/clinician). When including patients in various roles throughout the COS development process, established principles of patient engagement apply [9]. However, the approach to meaningful engagement is evolving, and the COS process brings additional challenges in engaging patients.

2 Current State of Patient Engagement in Core Outcome Set (COS) Development

Historically, patient engagement in COS development has been limited. In a review of the COMET (Core Outcome Measures in Effectiveness Trials) database, only 16% of COS projects reported including the public (defined as “patients, carers, health and social care service users and people from organizations who represent these groups”), and in 39% of those reports, it was not clear how the public was involved [10]. Instead, early COS development used clinical experts as the spokespersons representing the patient perspective. This practice has tended to favor outcomes important to clinical experts whose outcome priorities might differ from those of patients [11, 12]. However, there has been an increasing trend in patient engagement for COS development [13]. Across five annual updates of systematic reviews of published COS, the increase in efforts involving patients or other public representatives is evident [6]. COMET now recommends including patients in all COS development [14, 15].

Although there is agreement that patient engagement is beneficial and can sharpen the focus on patient priorities as treatment technologies advance, COS developers have indicated that patient participation can be challenging and that guidance for patient inclusion is needed [16]. They have commented that patients had difficulty understanding COS concepts and prioritizing outcomes, rated everything as important, and “lacked realism” about what is important to measure. It was noted that the onus is on the COS development community to address these issues and enable patients to participate meaningfully. In a series of workshops focused on patient involvement in COS development, advice for COS developers emphasized the importance of providing clear explanations of COS and associated concepts and suggested that providing instruction about clinical trials and systematic reviews would help patients understand why COS are needed [17].

Building on our organizations’ history of patient engagement work [18,19,20], we have implemented patient-centered COS development. Using a method that has demonstrated success, in terms of arriving at consensus on a COS by the different stakeholders involved in the development process, in three disease areas—hemophilia, nonalcoholic steatohepatitis (NASH), and sickle cell disease (SCD)—we discuss the challenges and adaptations involved in improving patient engagement in COS development. This method utilizes a spectrum of engagement with patients in various roles and recognizes patients as the “experts on living with the disease.” Translating these challenges and lessons learned into key steps, we provide recommendations for involving patients in COS development as part of HEOR.

3 COS Development Process

The most frequently used method for COS development is the modified Delphi [6, 13]. A Delphi is a structured consensus process that utilizes anonymous surveys and provides personalized results between survey “rounds” for voters to see how their responses align with those of other stakeholders (in our approach, provided in deidentified data summaries stratified by stakeholder group). Because the process is completed over a series of rounds, each voter can review and revise their votes in subsequent rounds.

We engage a multi-stakeholder Delphi panel, including clinicians, patients, patient advocates, researchers, industry representatives, representatives from health technology assessment (HTA) organizations, payers, and regulators. This panel rates the importance of candidate outcomes in each round, with outcomes of lesser importance eliminated before moving to the successive round for re-rating. Each iteration of the list thus contains fewer outcomes until a core set is reached. We typically reach consensus on a COS in three online voting rounds, with a consensus meeting between the second and third round. The described COS development process has adapted the COMET methodology [21,22,23] to include components that promote patient engagement and amplify the patient voice. Notably, the process allows for outcomes to be labeled as “patient important,” which ensures they are carried through to the consensus meeting and third round of voting. Thus far, we have used this process in three COS development projects in hemophilia (coreHEM) [24], NASH (coreNASH) [25], and SCD (coreSCD) (manuscript in preparation).

4 The Spectrum of Engagement

Rather than sporadic touchpoints or token inclusion, it is important to include patients, carers, family members, and advocates in all aspects of the COS development process. Patient involvement in COS development can be viewed on a spectrum, with varied time commitment and effort depending on the prescribed role in the initiative (Fig. 1).

Fig. 1
figure 1

The spectrum of patient engagement in core outcome set development initiatives

Full size image

We supplement outcomes identified in a targeted literature review by conducting informal background interviews with patients. The interviews are particularly important when novel treatment technologies may affect which outcomes are deemed important by changing the standard of care and expectations for therapy. For example, in hemophilia, gene therapy has the potential to prevent bleeds, rendering the usual primary outcome of annualized bleeding rate less important in gene therapy trials [26]. In coreHEM, we asked patients to consider what outcomes may rise in importance instead.

Patients, family members, and advocates comprise a significant proportion (15–20%) of the Delphi panel. Over the course of COS development exercises, these participants are invited to attend several webinars, participate in online surveys, and attend a full-day consensus meeting. Special expertise is not required to serve on the Delphi panel, but participants must make the time commitment to prepare for and participate in the consensus process.

An advisory board role may be fitting for a patient who has a more technical knowledge of the disease area or a comprehensive understanding of clinical trials. The advisory board meets virtually at periodic project milestones to provide input on the composition of the Delphi panel, the outcomes and definitions, the consensus meeting agenda, and materials for communicating with stakeholders. They help to interpret and provide context for the final COS and may be authors on manuscripts and participate in other dissemination efforts.

Finally, patients or advocates may take a leadership position as a co-primary investigator. This role has the greatest time commitment and is appropriate for a patient with a high level of knowledge about the disease area and the treatment pipeline. These patients may be representatives of the partnering patient organization, and often have previously participated in other research. A patient as a co-primary investigator has a role in all aspects of the COS development process, from developing the protocol to leading sessions at the consensus meeting to taking an active role in dissemination of the final COS.

The patients involved with coreHEM, coreNASH, and coreSCD have taken on each of these roles. Some examples include: (1) patients on the coreNASH Delphi panel candidly shared their experience and played an important role in the decision to create two different COS based on disease stages; (2) an individual living with SCD played a critical role on the advisory board in helping us understand patient perspectives, developing and reviewing project materials, and serving as a facilitator of the patient voice at the consensus meeting; and (3) a representative of a hemophilia patient organization served as a co-primary investigator for coreHEM and helped to develop our current patient-focused COS development process. In addition, patients from both coreHEM and coreNASH have participated in panel discussions about the work at various conferences.

5 Amplifying the Patient Voice in the COS Process

The patient-centered COS development process incorporates three elements of patient engagement.

5.1 Partnering with Patient Organizations

A key pillar of this patient-centered approach is that, for each COS project, we work together with at least one patient organization as an equal partner. Patient organization representatives served on the advisory board and Delphi panels for coreHEM, coreNASH, and coreSCD. They provided introductions to patients who might join the initiative, had significant input into the COS processes, and played an important role in COS dissemination. Their involvement ensured that the COS development process considered the needs of the disease communities and helped facilitate acceptance of the COS.

Partnerships between the research team and patient organizations are mutually beneficial. Engagement affirms the commitment of the patient organization to improving clinical research, provides direct involvement in shaping the project and assuring the patient voice is heard, and highlights the group as an influential member of the disease community. Meanwhile, the patient organization helps ensure the project maintains a patient-centered focus, encourages other stakeholders to participate, and provides additional avenues for dissemination of the COS.

There are several considerations when partnering with patient organizations for a COS project. Disease areas may have more than one patient organization or foundation, with differing missions or views of the condition; it is important to understand their roles in the community and how they do, or do not, work together. Furthermore, organization representatives who are co-investigators or serve on the advisory board are not able to vote on the outcomes that should be included in the COS during the modified Delphi. Therefore, only including patient organization representatives on the advisory board and not including patients in the Delphi panel is not enough to ensure a patient-centered COS development process.

5.2 Orientation and Training

The importance of adequately preparing patients to participate in COS development cannot be overemphasized. Patients should receive training to help them see the impact that outcome selection might have on the success or failure of trials, on access to new therapies as determined by HTA and payer decisions, and on clinical decision making. Patient participation in COS development has the advantage of incorporating patient values “upstream” in the development of new treatments. This requires some understanding of the complex process by which novel therapies become clinically available treatment options.

Training patients to meaningfully participate in a COS project can be challenging. The use of COS for clinical trials or to inform further HEOR can seem far removed from patients’ direct experience in clinical settings. It is therefore necessary to clearly link decisions that are made in clinical trial development and HEOR to decisions that patients might face in the clinic. With each subsequent COS project, we seek to extend and refine the training provided to patient participants. Training topics include (1) epidemiology and natural history of the disease; (2) clinical trial phases; (3) types of post-regulatory studies; (4) clinical trial design and the population, intervention, comparison, outcome, time (PICOT) framework [27]; (5) the importance of outcome selection and different types of outcomes; (6) the role of clinical trials in regulatory, market access, and patient care decisions; and (7) COS rationale and methods.

Patients who are familiar with the drug development process and clinical trials may find it easier to make decisions regarding the importance of outcomes to include in a COS. Less education may be required for patients who have been living with their condition for longer and have knowledge of the natural history of the disease and treatment options. However, it is important to consider the pros and cons of including patients who require less training as these patients might also be less representative of the general patient population.

The amount of orientation and training required must be assessed for each COS development project as the developers become familiar with the engaged patients. For example, in coreNASH, the participating patients were mostly unfamiliar with conceptualizing their disease in terms of outcomes and experienced difficulty rating outcomes for importance. During a patient webinar, we learned that the patients (perhaps especially those in the asymptomatic early stages of NASH) were deferring to the ratings cast by the clinician voters, believing that the medical professionals had a better understanding of what should be measured. With this insight, we were able to empower them to advocate for their priorities during the in-person consensus meeting. At the meeting, patients spoke about the impact and importance of fatigue on their daily lives even though this outcome is rarely measured or considered in the clinic. Though this outcome fell just below the threshold to be included in the final set, we believe that other stakeholders in the room sincerely took note of the patients’ struggle with fatigue.

5.3 The Consensus Process

Several aspects of the standard modified Delphi technique can help ensure patient centeredness. A key advantage of the modified Delphi is that it creates a level playing field where every individual’s vote is equally important and one dominant individual cannot assert undue influence. Each participant also receives an individualized report after each voting round that compares their vote with those of the other participants. Participants may decide to reconsider their responses based on this information without any one individual imposing their viewpoints on others.

The modified Delphi process can be adapted to further amplify the patient voice. Our COS development process includes a special “patient-important” criterion. In the first two rounds of voting, outcomes voted as “critical” by the patients are retained for the next round, even if the rest of the panel did not rate the outcome highly enough to retain. This ensures that outcomes highly important to patients are not eliminated without discussion at the consensus meeting. For example, coreHEM patients explained what a novel treatment such as gene therapy would mean for them. They believed it was important to measure the impact of gene therapy on mental and emotional health and spoke passionately about how gene therapy would be transformational on the mental health outlook for people who thus far have been living with a burdensome chronic disease. Additionally, gene therapy will give more consistent protection from bleeds compared with the experience of conventional therapy and the associated peaks and troughs of factor level. Hence, a typical hemophilia primary endpoint, the annualized bleeding rate, would no longer be relevant, and patients would instead look to a tangible measure, their achieved clotting factor activity level. Patients noted they would scrutinize the sustained factor activity level achieved in gene therapy trials to help guide the decision to receive gene therapy. Both mental health outlook and factor activity level were included in the final core set.

Ensuring that the number of patients is approximately equal to the number of voters from each of the other stakeholder groups (e.g., clinicians, payers and HTA, industry representatives) helps patients feel empowered to share their opinions and not feel overshadowed by “experts” at the consensus meeting. Additionally, we assign seating groups for the meeting to ensure a mix of stakeholders, including at least one patient, at each table. Small group discussions are interspersed with plenary sessions, allowing additional opportunities for patients to voice their opinions even if they are uncomfortable speaking in the larger group setting. A trained facilitator can help to ensure that patient input is solicited and considered during large group discussions.

6 Discussion

The importance of patient engagement is increasingly recognized across research contexts from drug development [28] to patient-reported outcomes measures (PROM) development [29] to systematic reviews [30] to guideline development [31]. However, there are unique challenges in patient engagement for COS, especially in making the distinction between outcomes important for clinical trials and outcomes important in personal discussions with a clinician. Additionally, there is a distinction between engaging patients as research subjects, as is often the case in PROM development [32] or patient preference studies [33], and engaging patients as research partners, as in the described COS approach. Use of COS in drug development and HEOR helps to ensure that a consistent and comparable body of evidence for treatment of a condition is available for healthcare decision making. Patient involvement in COS development efforts is essential so that selected outcomes reflect the treatment priorities of patients and family members.

Patient engagement in research entails challenges [34, 35]. As with other engagement processes [36], the modified Delphi process used for COS development requires repeated engagement over an extended time period. Sustained engagement may suffer in certain diseases if, for example, exacerbations or hospitalizations are likely over the course of the project. Patients with diseases that affect physical functioning may need accommodations to attend the consensus meeting in person or to travel. Patients with diseases that affect cognitive or neurological functioning may need assistance filling out a lengthy survey, perhaps from caregivers, whose values can differ from those of patients [37]. A discussion about adverse disease endpoints such as mortality or impaired functioning and quality of life could be upsetting to some patients [38]. In such circumstances, a liaison from a patient organization can provide insight on required accommodations, methodologic adjustments, and on potential patient experiences of project tasks.

Although partnering with patient organizations can be mutually beneficial [39], challenges may emerge if the priorities of a patient organization do not match those of the COS developers. Like other stakeholders, patient groups have their own institutional goals and objectives and may also harbor fears of exploitation because of previous negative experiences. COS developers and patient organizations must discuss in advance what the partnership should look like, the roles of each organization, and the planned engagement methods.

To the best of our knowledge, the COS initiatives described here are the first to include “patient-important” analysis criteria. Highlighting outcomes that patients rate as most important may reveal discrepancies between patients and other voters, thus potentially highlighting outcomes that have been overlooked by researchers. Caution is needed to assess whether the discrepancy is due to true differences of opinion on an outcome’s importance or rather due to inconsistencies or misconceived ideas on the outcome’s definition or use.

COS development requires trade-offs, as only the most important outcomes can be included in the COS. It might be difficult for patients to select the “most critical” outcomes to include in the COS as many outcomes can have significant impact on their lives. However, outcomes that may be important to patients based on personal experiences may not be the most relevant for evaluating the benefits of an intervention in a clinical trial. Therefore, it is important to explain that outcomes need to be prioritized and that eliminating an outcome does not mean it is not important per se; it is simply less important to include in the COS than other outcomes.

Patient-engaged COS can facilitate future research and decision making. The COS outcomes might serve as input for economic evaluations or burden-of-illness studies, and any important novel outcomes identified may then be further developed. For example, no measures currently exist to effectively evaluate a novel mental health outcome identified during coreHEM. As a result, work is in progress to develop a patient-reported outcome measure of the impact of gene therapy on mental health outlook. However, the impact of patient engagement on COS development might not always be directly apparent. For example, patient engagement may not always clearly lead to the inclusion of novel outcomes in the COS. By engaging patients, we ensure that the people ultimately impacted by clinical decisions are involved in research design.

The process we describe has some limitations. When working with a patient organization to recruit Delphi participants from their general membership, the potential participant pool is likely to be more educated, more knowledgeable about their condition, and more motivated toward self-care and engagement. While this may facilitate COS participation, these patients are likely not representative of the average patient. More educated and engaged patients may benefit from higher-than-average quality of care, potentially with better control of symptoms. Therefore, their experience of the condition and priorities for outcomes may differ from those of other patients. Engaging more representative patients is likely to increase cost and timelines throughout the project, as less engaged patients are more difficult to recruit and may be more challenging to orient and retain in long-term participation. Hence, there are costs and benefits to patient representativeness, and each individual COS development project must carefully consider the required level of knowledge for patients participating in the modified Delphi process. COS developers should make efforts to understand each participant’s baseline knowledge as this ultimately provides context for their votes. In some cases, other approaches to incorporating input from underrepresented groups may need to be considered as a supplement to the structured consensus process.

While engaged patients have expressed satisfaction anecdotally with their participation in the COS development projects and the resulting COS, patients’ satisfaction was not evaluated formally. Future COS development projects will benefit from a systematic evaluation process by using, for example, the Patient Engagement In Research Scale [40]. However, in doing so, it is important to separate satisfaction with the engagement process from satisfaction with the final outcomes selected in the COS.

7 Conclusion

Prioritizing patient opinions early in drug development and HEOR through COS development empowers patients and patient advocates to speak up as representatives with a unique perspective of living with or caring for someone with the disease. Patient engagement in COS development ensures that the patient perspective is reflected across many different studies that use the COS rather than in one clinical trial or HEOR study. As such, patient engagement in COS has a unique impact on evidence generation, with the potential to ultimately increase the patient centeredness of different healthcare decision-making processes, from regulatory, to market access, and clinical decisions.