Introduction

In 1947, the World Health Organization redefined health to include, “…not only the absence of infirmity and disease, but also a state of physical, mental, and social well being (p. 465) [1]”. Since that time, the subjective measurement of illness symptoms, functional adequacy and well-being has continued to evolve [2]. However, the effective incorporation of these patient-reported outcomes (PROs) into clinical research and clinical practice has been hampered by a number of challenges. These include floor and ceiling effects that limit sensitivity to change, lengthy questionnaires that increase patient burden, and a proliferation of measures of the same outcome. Collectively, these challenges have limited the use of patient-reported outcomes as endpoints within clinical trials and have inhibited the adoption of key trial findings by practitioners. Due to the lack of standardized instruments that have been validated in large heterogenous populations, clinicians and policymakers believe that many of these instruments may not have decision making relevance (external validity) in clinical practice [3–7].

Patient-reported outcomes in current research

Currently, patient-reported outcomes may be collected through a variety of methods in clinical research studies (Fig. 1). For example, a patient may complete a static paper questionnaire with a pre-defined set of questions during a clinical research visit. In this method, the patient functions as the information source, questionnaire administrator, and response recorder. Alternately, a patient may complete an electronic questionnaire during a clinical research visit. Here, the patient still functions as information source, questionnaire administrator, and response recorder; however, the question set may be either static, as was the case with the paper questionnaire, or dynamic, with the sequence of questions asked of each patient conditioned upon their previous responses. Alternatives to these methods include situations in which the research study patients do not administer the questionnaire and/or record their responses. Examples here include a clinician or phone interviewer who asks the questions of the patient (either static or dynamic format) and documents their responses, or a surrogate who provides responses for the patient to a clinician or telephone interviewer. However, while similar methods may be used to administer these PRO instruments, there is no uniformity in their content. This necessarily leads to questions regarding their external validity and generalizability.

Fig. 1
figure 1

Patient reported outcome data collection formats

Full size image

Roadmap initiative

The Patient Reported Outcome Management Information System (PROMIS) Network, a component of the National Institutes of Health’s Re-engineering the Clinical Research Enterprise program [8], seeks to overcome limitations in existing PRO instruments by (1) developing and testing large PRO item banks based on Item Response Theory (IRT), (2) creating a computer adaptive test (CAT) system for the assessment of PROs in clinical research, and (3) creating a publicly-available and updatable system for accessing and using the item bank via the CAT system, known as Assessment CenterSM [9, 10]. The net result of these efforts is the creation of a standardized set of patient reported outcome measures that can be adapted to the unique characteristics of different clinical populations (www.nihPROMIS.org).

Similar to translating new discoveries into patient care, the achievement of these objectives will overcome what is termed the first translational roadblock (T1) by transferring knowledge of IRT and CAT into systems for PRO measurement and assessment [11–13]. However, credible evidence of PROMIS’s effectiveness is required before researchers will adopt this new technology for their clinical studies [14]. To obtain this evidence, PROMIS researchers must overcome the second translational roadblock (T2) by developing new systems and technologies to facilitate the integration of PROMIS into clinical research. This second translation phase will require skills that focus on how these technologies are optimally implemented [15]. Our objective in the present study is to identify the technologies (systems and procedures) that are required to integrate PROMIS into the design and operations of multi-center clinical trials.

Materials and methods

Analytic approach

Previous researchers have applied the UK Medical Research Council’s (MRC) framework for the design and evaluation of complex healthcare interventions in a number of health delivery settings [16–22]. Recently, elements of this framework have been used to better understand multi-center clinical trials as complex health care interventions [23]. We sought to investigate how the PROMIS technologies might impact the design and operation of multi-center clinical trials using the methodology outlined in the second, “modeling,” phase of the framework [24].

During the modeling phase, components of the intervention, the methods by which they will interact, and ways they might influence key outcomes are identified. The objective here is to provide information for developing the optimal intervention and a research design for its evaluation in subsequent phases. As adequate quantitative data for simulations is not always available, qualitative methods are frequently employed to create this phase’s models. Previous research has demonstrated that qualitative methods can show how the proposed intervention works and identify potential barriers to its adoption [22, 25–27]. We used a similar approach in the present study.

Analysis steps

We divided the modeling phase into three steps (Fig. 2) [28–30]. In the first step, we identified likely PROMIS reporting formats and developed a process model for the design and operation of multi-center clinical trials. During the second step, we validated the process model in focus group interviews and used those interviews to identify clinical trial components that might be impacted by PROMIS in each reporting format. A significant aspect of our approach was assuring that the perspectives of key trial constituencies (the principle investigator, coordinating center, site-based researcher, and study participant) were appropriately represented. Finally, in a third step, we identified the gaps between current system capabilities and those necessary for implementing PROMIS in complex clinical trial environments. To bridge these gaps, we created a tool kit with recommendations for successful implementation.

Fig. 2
figure 2

Study organization

Full size image

Focus group sampling

Sample selection strategies in quantitative and qualitative research have different objectives and methods [29, 31]. Quantitative research seeks to test pre-defined hypotheses and produce generalizable results; whereas, qualitative research seeks to provide understanding of complex phenomena. Qualitative researchers typically use a purposeful sampling strategy that seeks to select the most productive sample for answering the research question. Subject selection criteria include presumed expertise (key informant sampling) and having had specific experiences (critical case sampling) [32]. Sampling continues until no new information emerges from the data (termed data saturation).

In selecting our expert consultants, we sought to achieve triangulation of results by choosing persons who represent different clinical trial constituencies (principle investigator, coordinating center, site-based researcher, and study participants), different coordinating center functional groups (clinical data management, clinical operations/project leadership, communications, medical center faculty, information technology, regulatory compliance, site-based research, and statistics), and who had experience in different types of trials (small vs. large, phases II–IV, industry vs. government sponsorship, and represented different clinical domains) [29]. Groups of mid- and senior-level individuals from the Duke Clinical Research Institute (DCRI) were invited to participate. The DCRI is an academic research organization that includes about 1,000 employees, the majority of whom are dedicated to the operational aspects of designing, conducting, and reporting clinical trials. All participants were either self-identified or recommended by department directors, and were selected because of their extensive experience with multi-center clinical trials and their involvement in projects implementing electronic data capture (EDC) and/or ePRO in multi-center clinical trials or trial networks. In all, 26 individuals representing 8 functional groups provided information on trial processes and their assessment of PROMIS’s impact on tasks in their areas of expertise.

Results

Step 1: Implementation formats and process map design

  1. 1.A.

    PROMIS implementation formats

We envisioned that multi-center clinical trials typically would implement PROMIS in one of three formats (Fig. 1). In the first format, a static paper form would be created using IRT theory based question selection from the PROMIS item bank. This would represent a trial-specific version of the traditional static paper form. The advantages to using this method vs. a traditional static paper format are that the number of questions could be varied (to be more or less than the traditional paper version) and the specific questions asked could be tailored to the characteristics of the clinical trial population (e.g., heart failure vs. depression). We assumed that one PROMIS static paper format would be used for all patients enrolled in the trial during each administration. In the second format, a tailored static electronic form would be created using IRT to help select the most sensitive questions from the PROMIS item bank for the population under study. This format is essentially the static PROMIS paper form in electronic format. The third format would use a dynamic electronic format. This format would make full use of the PROMIS item bank capabilities by using IRT and CAT. Here, the sequencing and number of questions asked would be determined by the patient’s previous responses. This would allow for a unique set of questions to be asked for each study subject during each administration. While there are obvious advantages in tailoring the question set in this manner, this format requires a real time computer interface with access to the PROMIS database.

After defining the various formats in which PROMIS would be implemented, we proceeded to develop a generic process model for the design and operation of multi-center clinical trials. We then determined how these clinical trial activities might be impacted by each of the three PROMIS implementation formats.

  1. 1.B.

    Process map design

A process map depicting tasks associated with design and operation of multi-center clinical trials was used as a tool of engagement and topic for the focus group interviews. We used an iterative process mapping approach to assure that the data collection methods for our exploratory, qualitative study were comprehensive and robust [29]. Three data sources were used to create our initial process map [29]: previous research conducted by the Evanston Northwestern Health PROMIS Primary Research Site, a review of the literature on clinical trial processes, and the work breakdown structure (WBS) used by the DCRI in planning and budgeting for coordinating center activities in multi-center clinical trials.

As a part of its PROMIS software development efforts, the Evanston Northwestern research team conducted a series of half- to full-day workshops with the other PROMIS network sites to obtain information on their current research processes and their expectations for the PROMIS software [33]. Byproducts of this effort were the documentation of clinical research activities for each PROMIS network site, a research process map for each site, and a combined process map summarizing research activities at all PROMIS sites. Although the site and summary process maps provide excellent detail on the activities performed by principal investigators, our research team’s perception is that they largely reflect research processes from the perspectives of single sites that both initiated and conducted research studies at their sites. Therefore, they do not contain sufficient detail to encapsulate all of the research activities required to coordinate multi-center clinical trials.

Next we conducted a review of the professional literature and clinical trial-related web sites to augment the summary process map with information from other networks. Unfortunately, we found few examples of trial maps, and none provided the level of detail our study required. Examples of sites reviewed include the UK’s Department of Health Clinical Trials Took Kit [34], the Stanford/Packard Center for Translational Research in Medicine [35], and the Center for Management Research in Healthcare’s trial process views [36].

Our third data source for the multi-center clinical trial process map was the work breakdown structure used by the DCRI. This document contains a billing-level description of the tasks performed by a coordinating center in the design and operation of a multi-center clinical trial. It reflects the DCRI’s collective experience acquired through the conduct of more than 400 clinical trials involving more than 500,000 patients enrolled at more than 3,500 sites in 64 countries [37]. The DCRI’s WBS supplemented with information from the previously described PROMIS summary process map served as inputs for our initial multi-center clinical trial process map.

  1. 1.C.

    Creating the process map

We used ‘mind-mapping’ software to create a visual activity map containing clinical trial tasks, their temporal linkages, and associated functional group responsibilities (Mind Manager 6 Pro, Mindjet Corporation, San Francisco, CA). Subsequently, we identified tasks that might be impacted by various PROMIS implementation formats. The initial multi-center clinical trial process map and the initial impact analysis provided the basis for our expert consultations.

Each expert panel was scheduled for a 1-h focus group meeting. The meetings began with an interactive presentation of the PROMIS goals, methodology, tools, and technologies before soliciting comments on the multi-center clinical trial process map and initial impact analysis. As a final step, the updated process map and impact analysis were presented in a 3-h meeting to a group of senior DCRI managers from clinical data integration, clinical operations, project management, and information technology.

Step 2: Focus group interviews; validation and trial impact analysis

In the sections below, we summarize our impact analysis results according to issues of importance to principal investigators, coordinating centers, participating sites, and study subjects that may become barriers to adoption of the PROMIS technologies.

  1. 2.A.

    Principal investigator issues

Principal investigator issues with PROMIS fell into two categories: acceptance by key regulators and the scientific community, and utility for researchers and clinicians (Table 1). In each of these areas, issues centered on how the PROMIS item bank with its variable question sets will be accepted by groups that currently are geared to the management of PRO instruments with fixed question formats.

Table 1 Issues relating to adoption of PROMIS technology
Full size table

Regulatory administrators

Regulatory issues focused on the integration of PROMIS with the existing clinical trials regulatory environment. Potential problems include data management conflicts with regulatory safeguard and harmonization initiatives, how an English-only item bank will be perceived by clinical trials that increasingly meet enrollment goals through recruitment in non-English speaking populations, and how institutional review boards will evaluate PRO instruments with static question sets that may never have been tested in the population under study, or with dynamic question sets in which the specific questions to be asked of subjects cannot be defined in advance. Perhaps the most important regulatory hurdle is the US Food and Drug Administration (FDA), which makes decisions about the appropriateness of measures such as those in PROMIS for securing specific labeling claims. The FDA’s current thoughts on these matters are contained in the Guidance for Industry: Patient-Reported Outcomes Measures: Use in Medical Product Development to Support Labeling Claims that has been distributed for comments [38].

Scientific community

As with any new measurement system, time is required for the accumulation of validity data, including responsiveness to change, in various clinical populations. Thus, early adopters of PROMIS will have limited data to cite as in support of this methodology.

Instrument development

In current practice, principal investigators use off-the-shelf PRO instruments—that is, measures defined by a fixed set of items that are ready to be inserted into a case report form. PROMIS will change this situation, and demand more from principal investigators seeking to realize the increased sensitivity and brevity of PROMIS-like measures. If principal investigators want to use a static form, they will have to determine which and how many questions from the PROMIS item bank to include in their trial’s PRO instruments, and they will need assistance in question selection to avoid creating a suboptimal instrument for their trial population.

  1. 2.B.

    Coordinating center issues

Although a number of coordinating center issues with PROMIS were raised, they largely are those faced when integrating any new technology into existing clinical trial systems and procedures (Table 1).

General system integration

General system integration issues are limited to electronic versions of the PROMIS instruments (both static and dynamic). Potential problems include accommodation of recognized data standards and conventions, integration with existing clinical trials software packages, creation of interfaces to support PRO reporting and analysis, and system validation.

Data management

Data management issues arise from system support needs and the integration of data from multiple collection systems. Potential problems include appropriate training in, documentation of, and user support for the PROMIS assessment center, the automated management of different versions of PROMIS instruments, and the need for immediate scoring when PROs are related to clinical trial inclusion/exclusion criteria or when PRO responses require immediate clinical intervention (as with suicidality). Other data management issues concern the alternate questionnaire delivery workflows and their technology implications. For example, ePRO data may be collected via hand-held devices, at computer kiosks in clinics, or from patients at home via the web; though the FDA non-repudiation requirement requires authentication at the individual user level [39]. Thus, for PROMIS applications involving ePRO, authentication by individuals is required.

Clinical operations

Clinical operations issues relate to the integration of PROMIS with existing systems and procedures at clinical investigational sites. For example, site-based clinical trial management systems and web-based EDC systems typically do not include facilities to support direct capture of patient reported outcomes, but do support patient assessment scheduling, monitoring (of data and patients), and adverse event reporting. PROMIS in its electronic formats will need to integrate with such systems. At a minimum, if integrated into existing site-based systems, PROMIS would require additional time from site personnel to train patients on using the system. Where PROMIS assessments cannot be delivered through existing systems, clinical operations will also be impacted by the distribution, training and support of additional devices at the site level (special computers or kiosks) or at the patient level (hand held devices or home computers with web access). PROMIS also will need to generate PRO forms (termed ‘worksheets’ for electronic formats and ‘case report forms’ for static formats).

Administration

Administrative issues relate to contractual assurances between PROMIS and its users. These include how copyrights will be administered for the PROMIS item bank and static forms, as well as the quality of, user fees for and data security provided when accessing the PROMIS system, and assurances regarding the availability of the PROMIS system and its technical support.

  1. 2.C.

    Participating site issues

Participating site issues are focused on the interaction of PROMIS with site coordinators and site information systems (Table 1). As with the coordinating center issues discussed above, site issues are similar to those that would be raised when any new technology is integrated into the site’s systems and procedures.

Coordinator effect

Coordinator issues are the same for all PROMIS formats; however, their magnitude will differ depending upon the specific formats used. These include the time required for conducting PROMIS studies versus the time required for other PRO systems, training and documentation support, and the production of scheduling reports for patient assessments and monitoring reports for compliance tracking.

Information systems

Site information system issues are limited to PROMIS electronic formats. These include funding for local information technology, integration with local information technologies, and the requirement for and support of PROMIS internet access.

  1. 2.D.

    Study subject issues

Several issues were raised that address the interactions of study subjects with PROMIS (Table 1). As was the case with the coordinating center and site issues, study subject issues are those that are raised when study subjects are expected to use new technologies.

Usability and accessibility

Usability issues in paper and electronic formats relate to the time required for study subjects to complete them, and subjects’ ability to navigate through PROMIS instruments. For subjects completing electronic formats, there are additional needs to easily change responses to questions and to move backward and forward through the electronic instrument. Privacy must also be guaranteed; this may require the provision of a private cubby or room for patients to complete assessments, and patient-level authentication procedures that may require patients to create and remember a personal password or challenge question known only to them. Accessibility issues relate to special subject populations, and require the inclusion of ADA compliant electronic systems for disabled subjects as well as options for subjects with different types of disabilities in paper and electronic formats. There is also the need to accommodate subjects with low literacy.

Relevance

Relevance relates to the clinical benefit a study subject receives from his or her participation in a clinical trial. In most instances, these benefits would be comparable for participation in clinical trials that use PROMIS vs. other PRO systems. However, for the rare “break glass” condition where immediate feedback is required (for example, in the case of a subject reporting they have high levels of suicidal ideation), it will be essential that PROMIS in all of its formats is able to alert trial personnel to the need for intervention.

Step 3: Bridging the gap—the PROMIS toolkit

In this section we have outlined the remaining capabilities required for PROMIS acceptance in the form of a tool kit that will bridge the gaps between currently planned PROMIS capabilities and those required for PROMIS to successfully operate in a multi-center clinical trial environment (Fig. 3).

Fig. 3
figure 3

Clinical trial PROMIS tool kit components

Full size image
  1. 3.A

    Training and documentation

Two types of training and documentation will be required, that relating to the PROMIS technologies in general and that relating to PROMIS instruments used in specific clinical trials. Coordinating center personnel, site personnel, and study subjects will require both types of documentation and training.

  1. 3.B

    Regulatory and scientific documentation

Users of the PROMIS technologies will require HIPAA assurances as well as policies, procedures, and guarantees regarding PROMIS system security, system availability, and disaster recovery. They would benefit from stock language for IRB submissions and protocols that describe PROMIS and its technologies. Principal investigators will also appreciate suggested language describing PROMIS and associated reference lists for grant submissions.

  1. 3.C

    Data integration

Clinical trial coordinating centers will require statements regarding the degree to which PROMIS systems comply with existing data management standards, dynamic data dictionaries for use in PROMIS based applications, and export formatting options for accessing PROMIS patient level and summary results.

  1. 3.D

    Item and instrument creation/selection

Principal investigators will need facilities to allow them to create new PROMIS instruments or to select from a library of existing instruments. Key capabilities will include the ability to select domains and items based upon characteristics of clinical trial populations, as well as an item selection wizard that assists the principal investigator in selecting items that are sensitive to the assumed study population and its likely range of responses to questions.

  1. 3.E

    Integration with site-based systems

Adding software systems that investigator sites must learn and use complicates the execution of clinical trials. In today’s environment, sites must commonly use an EDC system for CRF data collection, an Interactive Voice Response system (IVRS) for phone based randomization, a site-based Clinical Trial Management system, and the electronic medical record in use by the healthcare facility. If PROMIS adds an additional system, the complexity and cost of clinical trial execution will likely be increased. The extent to which PROMIS technology is able to integrate with and utilize existing site-based systems for instrument delivery may in fact determine its adoption rate.

Discussion

The National Institutes of Health (NIH) funded the PROMIS Network as a first step in bringing order to the disarray that characterized the assessment of patient-reported outcomes in clinical research. The initial funding supported the development of measures and a relatively simple data collection and scoring system. Recently, the National Institute of Arthritis and Musculoskeletal and Skin Diseases announced that the NIH will support an additional four years of the PROMIS Network. Our results help to inform this next phase of PROMIS by highlighting several opportunities and challenges for evolving PROMIS into a system that will meet the needs of the various multi-center clinical trial constituencies. Through a systematic multistage process, we identified regulatory, scientific, and user-related issues that will determine the long-term adoption of PROMIS into multi-center clinical trials. Articulating a detailed trial process map and obtaining expert input helped us to develop a “toolkit” (Fig. 3) that should serve as a guide for the PROMIS Network as it attempts to move PROMIS beyond individual NIH-funded researchers to larger clinical trial networks.

Although not an explicit PROMIS goal, the development of standardized question sets for the collection of patient reported outcomes will also benefit practitioners and policy makers. By creating a system that can adapt question sets to particular populations, PROMIS will overcome many issues regarding external validity and generalizability that have hindered the adoption of patient reported outcome in actual practice settings [4, 5, 7].

Limitations

There are several limitations to our study. First, our results and recommendations are largely shaped by personnel from a single institution, the Duke Clinical Research Institute. To the extent that the experiences of these personnel differ from those of the larger, multi-center clinical trials community, we will have introduced bias into our recommendations. Second, like any qualitative analysis, our results are necessarily exploratory and subject to confirmation in future studies. Nonetheless, we do not believe that these limitations seriously detract from our attainment of the overall study objective, the identification of technologies (systems and procedures) that are required to integrate PROMIS into the design and operations of multi-center clinical trials.

Conclusions

We have identified the issues that are likely to impede the use of PROMIS dynamic patient-reported outcomes technology in multi-center clinical trials. The thorough statement of requirements outlined in this study will serve as a framework for use by future investigators in the design, development, implementation, and operation of the technologies necessary for the successful integration of PROMIS into these trials.