Introduction to Comparative Effectiveness Research

Abstract

Health expenditures in the United States have increased without the equal improvement in quality. Comparative effectiveness research (CER), the evaluation of health-care outcomes that are achieved relative to the cost incurred, is seen as an approach that would help stem the ever-increasing health-care costs. The CER initiative has been denoted as a novel way to improve health-care decisions made by patients, physicians, and other stakeholders. Large efforts have been and continue to be made to establish this initiative. These efforts were formalized by financial support confirmed through the American Recovery and Reinvestment Act (2009) and the establishment of a Patient-Centered Outcomes Research Institute (PCORI) to support CER through evidence generation. This chapter will serve to introduce CER definitions, terminology, role, characteristics, and the level of evidence specific to CER, to individuals

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Past Health-Care Costs

Increasing costs and system inefficiencies in the United States have led to the pursuit of improved quality of care and the evaluation of the relative value of medical treatments [1]. In the past decade, the total health expenditure has doubled, reaching $2.7 trillion ($8,680 per person), which represented 17.9 % of the nation’s gross domestic product (GDP) in 2011 [2]. Despite the increase of funds, the United States underperforms other advanced nations on important health measures. This has led to a national focus on improving the quality of care while decreasing the cost.

Several seminal events have detailed a national research initiative to support better decision making about interventions by physicians and patients in health care [3]. The American Recovery and Reinvestment Act (ARRA) of 2009 formalized this initiative by allotting $1.1 billion toward this research initiative, which became known as comparative effectiveness research (CER). “The law states that the funding will be used for the conduct, support, or synthesis of research that compares the clinical outcomes, effectiveness, and appropriateness of items, services, and procedures used to prevent, diagnose, or treat diseases, disorders, and other health conditions and for encouraging the development and use of clinical registries, clinical data networks, and other forms of electronic health data that can be used to generate or obtain outcomes data” [4]. CER initiative is significant because it is a high-profile national research commitment to promote and improve decision making by patients and their physicians [5]. The CER initiative has now expanded to organizations beyond the government and includes medical associations, the health industry, health plan providers, and purchasers, to name a few. As a broad-based initiative, the CER hopes to mobilize system change by improving the effectiveness of care and suspending the growth in health-care costs by providing applicable real-world information for improved decision making [5]. Solidifying this initiative was the establishment of an independent institute to conduct comparative effectiveness research (CER), the Patient-Centered Outcomes Research Institute (PCORI), in March of 2010. This institute is a nonprofit corporation that is administered through a 19-member board of governors that includes the directors of the National Institutes of Health (NIH) and the Agency for Healthcare Research and Quality (AHRQ), patient representatives, physicians, private payers, medical industry representatives, health researchers, federal government representatives, and a 17-member “Methodology Committee” that includes scientific experts in clinical research, biostatistics, genomics, and research methodologies [6].

Efficacy Versus Effectiveness

The evaluation of medical interventions is critical to provide evidence of the benefits and harms of the interventions. The conditions under which the medical intervention is evaluated determine how generalizable that study can be when applied to current practice. They range from studies that have extremely controlled and restrictive conditions (efficacy studies) to those that evaluate an intervention in the real-world circumstances under which people actually use it (effectiveness studies). Historically, the effects of medical interventions on health have been evaluated based on efficacy. Efficacy studies determine whether an intervention produces the expected result under restrictive and controlled conditions in order to maximize the likelihood that the true effect will be evident if it indeed exists [7]. Effectiveness studies are at the other end of the continuum and are the evaluation of treatments in practice. The benefit of effectiveness studies is that the results can be applied to a broader population than efficacy studies, and thus these studies provide evidence that has great utility in health-care decisions for multiple stakeholders (providers, patients, and health care insurance providers).

Defining Comparative Effectiveness Research

The simple definition of CER is the comparison of two or more different health-care interventions’ effectiveness within a defined set of individuals in real-world clinical settings. Organization-specific definitions of CER shown in Table 1.1 provide comprehensive definitions to outline the characteristics of the research that included the organization’s CER. These definitions inform the public of the focus of their research and its importance in their lives. The definitions for various organizations share similarities and differences.

Table 1.1 Organization specific definitions of CER

Role of Comparative Effectiveness Research

The role of CER is to inform decision making using a range of research tools and methods. As defined by the Institute of Medicine (IOM), it is to “assist consumers, clinicians, purchasers, and policymakers to make informed decisions that will improve healthcare at both the individual and population levels” [5]. These include systematic reviews of existing studies and evidence, modeling to simulate effects of interventions on different populations, head-to-head clinical trials comparing one treatment to another, and studies using data available from registries, electronic health records, administrative records, and other databases.

Although a broad scope of research defines comparative effective research, the central tenet of CER is to determine which treatment works, for whom, and under what conditions it works best. The following six characteristics, developed by the IOM, describe the elements that define a CER study. At least one characteristic should be present for a study to be considered CER [8].

The six characteristics of CER as defined by the IOM [8]:

• 1. CER directly informs a specific clinical decision (patient perspective) or a health policy decision (population perspective).

• 2. CER results are described at the population and subgroup levels.

• 3. CER compares at least two alternative interventions, each with the potential to be “best practice.”

• 4. CER employs methods and data sources appropriate for the decision of interest.

• 5. CER is conducted in settings that are similar to those in which the intervention will be used in practice.

• 6. CER measures outcomes—both benefits and harms—that are important to patients.

CER directly informs a specific clinical decision (patient perspective) or a health policy decision (population perspective)

Questions that evaluate the health and health care of an individual patient or a population are important components that fulfill the goal of CER to identify what works best and for whom. The scopes of the studies include preventive, screening, diagnostic, therapeutic, monitoring, rehabilitative intervention, or policy-focused studies that synthesize knowledge or evaluate public health programs or initiatives involving the organization, delivery, or payment for health services.

CER results are described at the population and subgroup levels.

The evaluation of subgroups and the use of clinical prediction rules to identify patients who are likely to benefit from an intervention are important for the application of CER results to individual patients. These evaluations are intended to aid health decisions being made by providers and patients in the determination of whether the treatment can be individualized to the intended patient. In areas of health care that are rapidly developing, such as genomics and other biomedical sciences, the opportunity to develop individual targeted therapies and expand the application of personalized medicine is the greatest. Focusing on interventions that target individual patient decisions is a process that diverges greatly from decisions based on the results of clinical trials, where the results are expressed as the average group effect and are difficult to translate to individuals.

CER compares at least two alternative interventions, each with the potential to be “best practice.”

Translating evidence to individual patients requires intervention comparisons that patients and providers can apply to their cases. Prior to CER, the accepted standard for clinical decisions was randomized controlled trial (RCT) studies that compared an intervention to a placebo and used a very restricted and homogeneous population. This type of study demonstrates whether the intervention is safe and efficacious and establishes “does it work?” CER studies seek to expand the comparison to establish whether one intervention is better than the other and for whom it works better. To make these conclusions, CER studies compare a test intervention with a viable alternative in subgroups of people that may not have been included in the RCT. Accepted viable CER comparators could be different intervention modes (surgery versus drug) or the evaluation of an intervention in the current standards of clinical practice.

CER comparisons can also extend to the evaluation of the clinical and resource effects of health-care delivery. Comparators that may be used in these situations may be medical benefit designs, integrated organizational models, population health models, and cost-sharing techniques.

CER employs methods and data sources appropriate for the decision of interest.

Methods

The three primary research categories applicable to CER are experimental studies (randomized controlled trials), nonexperimental studies (nonrandomized and observational), and synthesis studies (systematic reviews and meta-analysis, technology assessments, and decision analysis). These categories will briefly be described here with further discussion.

Experimental research involves a treatment, procedure, or program that is intentionally introduced, and a result or outcome is observed. The randomized clinical trial (RCT) is a classic experimental study design to be used to determine whether a difference exists between two or more groups. This design is conducted within a setting that is controlled, to a certain extent, by enrolling patients who meet prespecified criteria and provide informed consent. A defining characteristic of an RCT is the random allocation of participants to the experimental group or the comparison group. An example of an experimental study that constitutes a method of CER is a head-to-head trial. This RCT study compares two groups of people with the same disease; one group receives an active intervention, while the other group does not. Other CER study designs in this category are cluster randomized, adaptive designs, and pragmatic trials.

Nonexperimental research consists of studies that collect data by observation, without making changes or introducing treatments. Observational research includes prospective and retrospective cohort studies, case-control studies, and case series analyses. The advantages of observational studies are that the population being studied can be diverse, treatments are more likely to be delivered in a manner consistent with clinical practice, and treatments that may be unethical to withhold in an RCT study design can be investigated. This study type is more in line with the goals of CER; research conducted in a real-life situation is easier to extrapolate to a real-life patient problem. The disadvantage of observational studies occurs as a result of the lack of randomization: group differences are expected, introducing a risk that the difference in outcomes could be due to the initial differences between patient groups. Methodologies to minimize this disadvantage are progressing and have been propelled forward with the advancement of electronic data collection in registry and database studies.

Relevant evidence that informs real-world health care decisions for stakeholders is a central tenet of CER. The plethora of available evidence requires evidence synthesis in the form of systematic reviews, meta-analysis, technology assessments, and decision analysis studies. These studies summarize the information from more than one study and are required to have systematic review strategies that are transparent and consistent, which allows valid comparisons of effectiveness. In turn, this methodological rigor allows these documents to become primary sources of information for stakeholders to make confident decisions [9, 10].

Data Sources

CER data is derived from multiple different existing sources that are readily accessed as a result of computers and health information technology. This includes information from the patient interaction in the health system, administrative claims data from large national insurers, the patient’s electronic medical record that contains the standard medical and clinical data gathered in one provider’s office, and the electronic health record that is a comprehensive collection of all of a patient’s medical records. Another source includes patient registries, a collection of information about individuals in a standardized way using observational research methods. Registries can be focused on a specific diagnosis or condition and are used to identify people who may be underrepresented in research studies and for tracking of outcomes from clinical practice interventions [11].

CER is conducted in settings that are similar to those in which the intervention will be used in practice.

“Consistent with the definition of effectiveness, the settings of CER studies are a defining characteristic [8].” Studying interventions in a setting that reflects the current practice setting has the benefit of translating and disseminating CER findings in a timely and interpretable manner.

CER measures outcomes—both benefits and harms—that are important to patients.

The involvement of the patient is a priority for CER. The value of CER is in influencing real-world decision making. Therefore, determination of the net benefit (benefit-harm ratio) of the intervention is vital. The net-benefit ratio is inclusive of the harms or risks associated with the outcome of interest, and it is often measured by patient-reported outcomes (a patient’s perception of an outcome). Patient-reported outcomes are necessary in the net-benefit evaluation because these outcomes often differ from clinical outcomes.

Patient Involvement

A defining feature of CER is the patient involvement that is a critical distinction from the traditional models of health research. In the traditional model, research questions are developed by scientists and experts and based on their opinions; they determine what outcomes and interventions are important for the patient. In this model, patients are passive participants in the research process, and this has resulted in research findings that are poorly aligned with the information important to patients. The role of the patient is instrumental to CER; therefore, research is that relevant and communicable to the patient, the caregiver, and the consumer is required. From the beginning of the research process, the conceptualization of the research question, patient beliefs are incorporated into CER through the active involvement of consumers, patients, and caregivers. Patient involvement is needed throughout the research project, ending with the implementation and the dissemination of findings.

The term “stakeholder” is broadly used in CER to define “individuals, organizations or communities that have a direct interest in the process and outcome of a project, research or policy endeavor” [12]. Stakeholders may be patients, their caregivers, health-care providers and delivery systems, patient advocates, policy makers, and consumers. Stakeholders frequently have limited experience in research. Engaging stakeholders in the research process requires education, training, and support. Table 1.2 outlines several programs that have been developed to help prepare stakeholders for their role in research.

Table 1.2 Programs that have been developed to help prepare stakeholders for their role in research

Level of Evidence for CER

The nature of CER research requires study methodologies that address large and broad populations of patients in a usual-care setting while also being time sensitive so that decisions based on the results are being made with the most current available evidence. The assessment of the strength of the body of evidence is directly related to the confidence in the decision that can be made. The use of systematic grading criteria provides an accepted standard of evaluation by which to draw conclusions and transparently report the findings. Historically, levels of evidence have used a hierarchical study design model that places randomized controlled trials and meta-analysis at the top of the model. In CER studies, a more comprehensive evaluation of the evidence is recommended for decision makers.

The Evidence-based Practice Center (EPC) Program that is supported by the Agency for Healthcare Research and Quality (AHRQ) is responsible for developing and updating recommended guidelines for level of evidence assessment in CER studies. The EPC method is one system for reporting results and grading the related strength of evidence, and it presents a consistent, transparent process to document and report the most important summary information about a body of literature. The EPC Methods Guide for Effectiveness and Comparative Effectiveness Reviews (2014) provides detailed guidance on the systematic review of drugs, devices, and other preventive and therapeutic interventions [10]. The current recommendations will be briefly discussed, but due to the rapid evolution of CER methodologies, future updates to these recommendations are expected.

The EPC’s method recommends multiple-component criteria for establishing the strength of evidence (SOE) in CER studies. The EPC method is largely based on the GRADE (Grading of Recommendations Assessment, Development and Evaluation) system [12] that has been widely adopted as an approach to rating confidence in estimates of effect the quality of evidence and guidance for practice. In the EPC method, there are five main specific domains, including study limitations, directness, consistency, precision, and reporting bias as described by domain and definition from the Methods Guide for Effectiveness and Comparative Effectiveness Reviews [12] shown in Table 1.3.

Table 1.3 Domains and definitions for strength of evidence rating

Strength of Evidence Grades

EPCs assess individual domain scores and establish overall strength of evidence grades relevant to each key question. The overall grade for the strength of evidence is a global assessment that includes a judgment about the strength of the evidence for each major benefit and harm that is relevant to stakeholders in the setting in which the recommendations are being made [13]. The EPC method of strength of evidence consists of four grades: high, moderate, low, or insufficient. Each strength of evidence definition provided by the Methods Guide for Effectiveness and Comparative Effectiveness Reviews [10] is briefly described in Table 1.4. Two criteria, representing different conceptual frameworks, are required at each level. The first criterion determines the evaluator’s judgment that the evidence reflects the true effect. The continuum of choices range from high to insufficient, with a high grade representing a clear, true effect level and with an insufficient grade stating that evaluators were unable reach a conclusion. The second criterion is “a subjective assessment of the likelihood that future research might affect the level of confidence in the estimate or actually change that estimate.” It is also denoted by high, moderate, low, and insufficient grades [10, 13].

Table 1.4 Strength of evidence grades and definitions