Frequency and Type of Conflicts of Interest in the Peer Review of Basic Biomedical Research Funding Applications: Self-Reporting Versus Manual Detection

Abstract

Despite the presumed frequency of conflicts of interest in scientific peer review, there is a paucity of data in the literature reporting on the frequency and type of conflicts that occur, particularly with regard to the peer review of basic science applications. To address this gap, the American Institute of Biological Sciences (AIBS) conducted a retrospective analysis of conflict of interest data from the peer review of 282 biomedical research applications via several onsite review panels. The overall conflicted-ness of these panels was significantly lower than that reported for regulatory review. In addition, the majority of identified conflicts were institutional or collaborative in nature. No direct financial conflicts were identified, although this is likely due to the relatively basic science nature of the research. It was also found that 65 % of identified conflicts were manually detected by AIBS staff searching reviewer CVs and application documents, with the remaining 35 % resulting from self-reporting. The lack of self-reporting may be in part attributed to a lack of perceived risk of the conflict. This result indicates that many potential conflicts go unreported in peer review, underscoring the importance of improving detection methods and standardizing the reporting of reviewer and applicant conflict of interest information.

Introduction

A core component of the legitimacy of any peer review process is the perception that the process is fair (Wood and Wessely 2003; Merton 1973; Oleinik 2014). While most funding agencies have conflict of interest (COI) policies, there are great discrepancies in these policies across funding institutions (Langfeldt 2006). One potential cause for this discrepancy is the lack of data in the literature regarding the frequency and type of COI occurrence, despite the fact that many types of biases are known to exist in peer review (Lee et al. 2013). The COI data that are available focus primarily on clinical- or product-driven research, with very few data focusing on basic or preclinical research (Lo and Field 2009; Pham-Kanter 2014). In addition, these prior studies, and in fact most COI policy implementations, rely largely on reviewer self-report. Despite a likely high degree of reporting integrity overall, there are potential conflicts not identified by reviewers due to an inappropriate assessment of potential risk, missed detection, or even purposeful under-reporting. Therefore, it is not clear if further COIs exist, which are not reported by reviewers.

The American Institute of Biological Sciences (AIBS) has been conducting peer review for various federal and nonfederal funding bodies for over 50 years. Therefore, AIBS is strategically positioned to provide data addressing COI frequency (Gallo et al. 2013). AIBS’s COI detection process includes both an independent self-reporting phase as well as a manual detection phase, in which AIBS staff search for potential conflicts. AIBS undertook a retrospective analysis of COI data from a basic biomedical research peer review program in order to elucidate the frequency and type of COIs detected by each phase of this process.

Materials and Methods

The peer review COI dataset was generated from 282 biomedical research applications submitted in a given year to an anonymized funding program. The research funding mechanisms allowed for 2–3-year awards with a value range of $300 to $900 K. The applications were reviewed by eight onsite panels (average of 35 applications per panel) all of which occurred in the same year. Applications were grouped into panels based on their related scientific focus within the general topic area. The proposed projects focused largely on molecular and cellular biology techniques, as well as genetics and epigenetics, immunology, tissue engineering, computational biology, and basic in vivo preclinical studies; in general, the projects were very similar to the types of projects reviewed by many NIH panels. On average, onsite panels included 12 experts plus 2–3 teleconference reviewers, all of whom were recruited by AIBS to provide independent, objective assessments of applications. There was no overlapping membership between panels.

Reviewers were recruited, required to sign non-disclosure agreements and to complete COI forms, and vetted for participation. Once panel composition was finalized, assignments were disseminated, and reviewers prepared their evaluations via an online review system. At the panel meeting, assigned reviewers discussed their critiques with the panel. At the end of each application discussion, all reviewers without an identified COI scored the application. Further details on the general AIBS peer review procedures are outlined in a separate publication (Gallo et al. 2013).

In terms of seniority, the reviewers broke down into three categories: Junior (36 %), Mid-level (30 %), and Senior (34 %). While onsite reviewers were assigned 6–7 applications to provide written critiques, all onsite panel members voted and had the opportunity to provide written feedback on all applications (unless a conflict was identified). Teleconference reviewers, however, only voted and commented on the applications to which they were explicitly assigned, none of which had conflicts. Thus, teleconference reviewers were removed from this analysis, leaving 94 distinct reviewers.

The list of all key personnel, collaborators, and consultants from all applications was available at the time of reviewer recruitment. AIBS’s COI policy states that no one on this list was to be invited to serve as a reviewer; therefore, the recruitment process served as an initial step in avoiding any direct financial or scientific conflicts.

After reviewer recruitment, conflicts were identified through reviewers assessing application-specific personnel information on their panel via AIBS’s online contracts system.

The self-reporting form included seven conflict criteria:

• 1. Organizational Affiliations with Application Personnel

• 2. Current or Recent (2 years) Collaboration with Application Personnel

• 3. Financial Interests which Conflict with the Application

• 4. Sources of Research Support that Conflict with the Application

• 5. Government Service that Conflicts with the Application

• 6. Public Statements/Positions that Conflict with the Application.

• 7. Additional Information

Concomitantly, COIs were also identified through a manual detection phase in which AIBS staff searched through PDF versions of reviewer CVs and submitted applications. AIBS staff conducted a cross search of reviewer names and institutions in the submitted applications and of PI, co-PI, and collaborator names and institutions in reviewer CVs using the Adobe^® Acrobat^® Advanced Search function.

Both the self-report and manual detection COI checks occurred before applications were assigned to reviewers. If a potential conflict was found, it was assessed by AIBS staff using the seven conflict criteria and considering AIBS’s COI policy. Depending on the severity of the conflict, AIBS policy either requires the reviewer to recuse him/herself during the discussion and scoring of a particular application or the dismissal of the reviewer completely from the panel. As an example, a minor conflict would be a reviewer from the same institution as one of the applicants (but with no direct connection to the application). In this scenario, the reviewer would be asked to recuse him/herself from discussion and scoring of that application. Conversely, a reviewer that receives direct financial benefit from funding of the application would be considered majorly conflicted, and the reviewer would be dismissed from the peer review program altogether. This schema is very similar to that used by the National Institutes of Health (NIH 2014).

Using AIBS’s seven conflict criteria, COI data were analyzed for frequency of occurrence of each type of conflict. The method of detection (self-report versus manual detection) was also recorded and analyzed.

Results

Out of all potential conflicts, 66 were considered minor conflicts (Table 1). These minor conflicts prevented reviewers from participating in the voting and discussion of specific applications but not in participating in the program altogether. No major post-recruitment conflicts were found. The majority of these conflicts (65 %) were identified by manual detection. When sorted by conflict criterion, data reveal that 32 % of minor conflicts were because of organizational affiliations; 61 % were because of collaborations with application personnel; and 8 % fell under the Additional Comments criterion. With regard to the Additional Comments criterion, the majority of conflicts were either reviewer-perceived conflicts, including personal connections with application staff (sometimes with several degrees of separation), or connection through a shared resource (e.g., the reviewer uses a mouse model derived from the applicant’s laboratory, but there is no direct collaboration with the applicant).

Table 1 Listing of total number of conflict of interests (COIs) per panel, as well as broken down by COI type and whether the COI was detected through reviewer reporting or through AIBS staff detection

No conflicts were found relating to financial interests, research support, government service, or public statements/positions. This is likely partly attributed to AIBS’s recruitment policy that prohibits recruiting reviewers on the list of all key personnel, collaborators, and consultants from all applications. It is also partly attributed to the lack of financial and other information present in reviewer CVs and to the generally basic science nature of the applications.

The average number of reviewers per application with at least one COI was calculated for each panel, resulting in an overall average of 0.24 ± 0.05 conflicted reviewers per application and a median of zero, indicating that no reviewers were in conflict with most applications (Table 2). In fact, in terms of applications, there was at least one identified conflict on only 20 % (or 56) of the applications, with an average of 7.0 ± 1.4 applications with COIs per panel (out of an average of 35 total applications per panel). However, the number of reviewers conflicted with any of the applications reviewed on their panel ranged from 3 to 7 (Table 2); on average, 39 % of reviewers were conflicted on at least one application on their panel. It should be noted that on average a reviewer was in conflict with 0.71 ± 0.15 applications per panel, indicating that multiple conflicts per reviewer were a rare occurrence. Nevertheless, multiple conflicts per reviewer did occur, partly exacerbated by organizational conflicts (e.g., multiple applications on the same panel from a similar institution).

Table 2 Listing of total number of reviewers with at least one COI (N_RCOI) per panel and per application as well as mean/median levels of conflicted-ness (% C) and the mean/median number of COIs (N_COI) per reviewer broken down by panel

In order to measure the level of conflict per application relative to the size of each panel, conflicted-ness (percentage of reviewers conflicted for any one application) was assessed across all applications per panel, generating an average and a median value per panel, listed in Table 2. While the average level of conflicted-ness varied across panels (0.57–4.01 %), the median values of conflicted-ness were consistently zero across all panels, indicating the relative rarity of a conflict for any given application. The median level of conflicted-ness per panel (based on the averages in Table 2) was then calculated to be 1.4 ± 0.4 %, and the average level was 1.9 ± 0.4 %. These data are significantly lower than the 13 % median conflicted-ness reported by Pham-Kanter (2014) for regulatory reviews; although, financial conflicts were predominantly reported. No direct financial conflicts were identified in the administration of this review effort.

Discussion

Conflicted-ness was found to be much lower in this study than that reported for regulatory reviews (Pham-Kanter 2014). This may be partly attributed to the basic science nature of the applications reviewed in the program and the fundamental review criteria used to evaluate them (e.g. appropriateness of research hypothesis, methodology, etc.). As such, the vast majority of reviewers and PIs in this study held a PhD and not an MD. In evaluating efficacy and safety of potential therapeutics (all of which have an industrial sponsor), the review process naturally includes individuals with greater collaborative and financial connections to industry, products, and therapies, as these are the reviewers with the most pertinent expertise. While one could hypothesize that the greater conflicted-ness of regulatory reviews is because more senior experts (with greater connections) participate on regulatory review panels, our results indicate (at least for basic science research) that junior reviewers account for the majority of conflicts. Junior reviewers accounted for 55 % of identified conflicts; mid-level reviewers accounted for 8 % of identified conflicts; and senior reviewers accounted for 38 % of identified conflicts. These levels of conflicted-ness are disproportionate to the overall balance of junior, mid-level, and senior experts on these review panels.

The relatively low level of conflicted-ness per application is also unexpected given the fact that there were between 29 and 44 applications per panel in this review, and thus one would expect a high degree of network complexity per panel from which conflicts were derived. This conflicted-ness value is even lower, if one only considers self-reported conflicts. It may be that the connectedness amongst scientists involved in basic science is less than the widespread financial connectedness among those involved in clinical work and product generation (Bekelman et al. 2003). In any case, it is encouraging that nearly 80 % of applications had no conflicts, indicating that the initial vetting and recruitment process was a good first step in minimizing conflicted-ness.

In addition, it is noteworthy that the identified conflicts were almost solely institutional, collaborative, or personal in nature. This is likely attributed to the generally basic science nature of the research under review and to the fact that direct financial conflicts were both harder to identify and naturally less common. However, given the percentage of conflicts that were manually detected, it is clear that conflicts were under-reported by reviewers overall, indicating that financial conflicts may have in fact existed. Equity and fidelity in the review process are important to its legitimacy. To better identify conflicts and maintain the equity and fidelity of reviews, reviewers should be required to provide basic, standardized financial interest information in their CVs/Biosketches (in addition to grant funding) that includes consulting and ownership stakes, as these have been reported to be frequent sources of COIs (Pham-Kanter 2014). No such reporting standard currently exists, although efforts to standardize the reporting of COI information at funding agencies and in scientific journals are already underway (Drazen et al. 2010; Rockey and Collins 2010).

There was a significant increase in total number of COIs after AIBS staff completed the manual detection phase. On average, 65 % of conflicts were identified through manual detection. In addition to human error, this under-reporting may be attributed to a lack of awareness of what constituted a COI, although reviewers had access to AIBS’s COI policy, which was very similar to that of the NIH (2014). Also, reviewers may not have appropriately perceived the risk of a potential COI; a reviewer who believed his/her potential COI was low risk may have under-reported. However, due to the subjective nature of some types of conflicts, even a perceived conflict diminishes the equity of the review process. Therefore, the final decision on what constitutes a COI should be at the discretion of the peer review administrative staff and not the reviewers. Also, a potential source of under-reporting may be due to reviewer bias, as reviewers may want the chance to promote the applications of their collaborators (Lee et al. 2013). An important limitation to this study is the inability to discern the root causes of this under-reporting, as there is not a way to detect bias through this analysis. Bias, of course, also exists outside the scope of those investigators with whom the reviewer has collaborated. The influence of bias on the review process may result in the potential suppression of innovation (Berezin 2001). Therefore, measurements of conflicted-ness may overestimate the reliability of the peer review process. Future research efforts should focus on developing better conflict criteria and more effective implementation of double-blinded peer review to potentially detect and/or prevent biases early in the process.

These data underscore the need for vigilance by administrative staff in detecting and assessing conflicts. Although manual detection methods may add more time and cost to the process, it would strengthen the ethical foundation of review panels. In addition, there is a need for reviewers to give the conflict assessment process its due time and consideration, and administrators should work with the scientific community to make COI guidelines as straightforward as possible. However, there should be more research conducted on the potential impact of different types of conflicts on decision-making processes, the results of which should temper COI guidance to reviewers. There may be no perfect solution, but in light of increasing demands on researchers’ available time to review (for both journals and funding agencies), administrators must strike the right balance between maintaining high ethical standards and the ability to reliably recruit reviewers with the most appropriate scientific expertise.

Finally, while the overall conflicted-ness per application was low, our results still show that over a third of reviewers on average had at least one conflict on a panel. As the number of publications continues to rise in the scientific community, so will the interconnectivity of researchers, and thus, detection methods need to evolve to reflect the likely increases in frequency and severity of the conflicts reported.