Introduction

The status and continuing use of literature affected by scientific misconduct is of concern because of the potential for invalid research to misdirect subsequent research and clinical care [6, 7]. In 1990, Mark Pfeiffer and Gwendolyn Snodgrass [20] described the use of retracted, invalid scientific literature, reporting that compared with a control group, the retraction tag in the MEDLINE database reduced subsequent citation by only about one third. In 1998, John Budd and colleagues [5], reported that retracted publications were still frequently cited even through the retraction was visible in the journal and clearly noted in MEDLINE. Although biomedical science tends to be self-correcting [5, 7], a great deal of time and effort may be required to determine that some research is not valid [6], and still retracted papers continue to be cited in the scientific literature [23, 24].

Journals occasionally report on notorious research integrity violations, summarizing information from scientific misconduct investigations, and noting the affected publications [1, 2, 811, 17, 23]. Many other lesser-known cases of fraudulent publications have been identified in official reports of scientific misconduct, yet there is only a small body of research on the nature and scope of the problem, and on the continued use of published articles affected by such misconduct [3, 18, 21].

The purpose of this study was to identify published research articles that were named in official findings of scientific misconduct that involved Public Health Services (PHS)-funded research or grant applications for PHS funding, and to investigate compliance with the administrative actions contained in these reports for corrections and retractions, as represented in PubMed. This research also explored the way in which such corrections are indicated to PubMed users, and determined the number of citations to the affected articles by subsequent authors.

Background

DHHS findings of scientific misconduct

The U.S. Department of Health and Human Services (DHHS) Office of Research Integrity (ORI) Division of Investigative Oversight is responsible for the review of institutional investigations and findings of scientific misconduct leveled against individuals (named as respondents) working within the PHS, or receiving its extramural support [13, 15, 19].Footnote 1 U.S. federal policy defines scientific misconduct as fabrication, falsification, or plagiarism in proposing, performing, or reviewing research, or in reporting research results [16].Footnote 2

Figure 1 outlines the usual process of a scientific misconduct investigation, starting with an allegation from a ‘whistleblower’. When the final report of an institutional inquiry into misconduct deems that the allegation of scientific misconduct has been substantiated, the ORI issues a “Finding of Scientific Misconduct” report, which is published in its Annual Report, and also in the NIH Guide to Grants and Contracts. These reports usually specify administrative actions against the respondents. Routine administrative actions include debarment from applying for PHS funding or participating in study sections for a period of time [4], and notifying editors of any published articles determined to be fraudulent, plagiarized and/or in need of some type of correction, or directing the respondents to make such notifications [15].

Fig. 1
figure 1

Overview of how a published article comes to be named in a “Finding of Scientific Misconduct” report, and whether it is tagged as corrected or retracted in the MEDLINE database. aRequirements: Correction, errata, or retraction are labeled and published in citable form and in an issue of the journal that originally published the correction or retraction [25]

The problem of correcting the scientific record

Even when the “Findings of Scientific Misconduct” report identifies publications affected by the misconduct, a variety of factors can impede the tagging of the affected article with an erratum or retraction. The National Library of Medicine (NLM) policy for tagging articles with corrections states that notices of errata and retractions will be linked to articles indexed and available on its online PubMed database only if the journal publishes the errata or retraction in a citable form. The citable form requirement stipulates that the errata or retraction is labeled as such, and is printed on a numbered page of the journal that published the originally article. The NLM does not consider unbound or tipped error notices,Footnote 3 and for online journals, only considers errata listed in the table of contents with identifiable pagination [12, 22, 25].

Debra Parrish [23] has described the apparent reluctance of some journals to retract articles. Moreover, the format of retractions may not meet the above NLM requirements. Varying journal policies also confound the process of publishing notices of errata or retractions [3, 7, 10]. In a 2002 survey of journal retraction policies, Michel Atlas [3] noted one participant who stated that his journal did not publish retractions. Some journals allow one author to retract an article, but other journals require that every coauthor consent to the retraction [3, 22]. Fear of litigation is behind the inaction in some cases [7, 14, 22]. Thus, for such myriad reasons, some faulty articles affected by scientific misconduct remain untagged with notices of erratum or retraction.

Methods

Overview of data collection

In 2002, we conducted a content analysis of all the “Findings of Scientific Misconduct” published in two public sources (the NIH Guide for Grants and Contracts, and the ORI Annual Reports) from 1991–2001. From these reports we abstracted the information on the publications said to be affected by scientific misconduct, the administrative actions taken against the respondent, and whether the respondent was described as accepting or denying responsibility for the misconduct. We then searched PubMed for the identified articles to determine if subsequent notices of erratum or retraction were added to the citations in PubMed, and if so, the location of such notices. Finally, we used the Web of Science to determine the number of citations others made to these affected articles.

Details of data collection

As described in detail below, the study data were collected from the following four sources: (1) the NIH Guide for Grants and Contracts “Findings of Scientific Misconduct” (http://grants.nih.gov/grants/guide/index.html); (2) the ORI Annual Reports (http://ori.dhhs.gov/publications); (3) the National Library of Medicine’s PubMed online bibliographic database (http://www.pubmed.gov); and (4) Thomson’s Institute for Scientific Information Web of Science bibliographic databases (http://isi02.isiknowledge.com/portal.cgi/).

ORI annual reports and NIH Guide

There was considerable overlap in the information found in the ORI Annual Reports and the NIH Guide for Grants and Contracts, “Findings of Scientific Misconduct”. The ORI Annual Reports named only a few articles that were not listed in the NIH Guide. Information abstracted from these sources included:

  • The complete citation of the article affected by misconduct. In a few cases, the report did not include the article title or journal name, merely indicating in a non-specific statement that a published article had been affected by fabrication of data or subjects, falsification of methods or results, or plagiarism (FFP); in such cases, the specific publication citation to these articles was obtained by calling the staff at the ORI.

  • Statements about how the misconduct affected the published article (i.e. FFP).Footnote 4

  • Whether or not the author accepted or denied responsibility for the misconduct (if mentioned in the report).

  • The administrative actions pertaining to the affected article. These usually took the form of debarment from receiving PHS funding, or prohibition from service on PHS advisory or review committees or as a consultant for a specified period of time. For the purpose of this study, we were most interested in whether the administrative action indicated that a specific published article should be retracted or corrected.

PubMed

The PubMed database provided the following information:

  • The article’s unique identifier number (PMID).

  • The citation information of the affected article (i.e., authors, title, journal issue etc.).

  • Whether the citation of the affected article was linked to an additional notice of a retraction, correction or other type of corrigenda, such as in a “Comment” field.

  • The location of any such corrigenda (e.g. as part of the article citation, in a linked field, or on a subsequent linked PubMed webpage).

  • Whether the article had a link to the “Finding of Scientific Misconduct” in the open-access NIH Guide for Grants and Contracts.

Web of science

Data collection from the ISI Web of Science was repeated two times during 2003, and once during 2004 to refine the data collection methodology. Because citations increase over time, a final citation analysis was conducted during the week of May 17, 2005 and the citations as of that week are reported here. This allowed for a minimum period of 3 years between the publication of an affected article and the cut-off date for the citation analysis. The ISI Web of Science database provided the following information:

  • The number of citations for each of the 102 affected articles (“Times Cited”) was identified through the Science Citation Index Expanded databases (SCI-EXPANDED 1980—present, and Social Sciences Citation Index 1980—present). The results may have included self citations.

  • The articles that cited the 102 articles affected by scientific misconduct. The “Times Cited” link provides a list of articles that have cited the affected article. These articles were downloaded into the Endnote software to establish the specific citing articles as of the 5/17/2005 cut-off date.

Results

Implicated researchers and publications

A search of the NIH Guide for “Findings of Scientific Misconduct” of the period from 1991 to 2001 revealed that the first listing of an article affected by misconduct appeared in a 1993 NIH Guide. Between 1993 and 2001, 102 published articles were named in either an NIH Guide or an ORI Annual Report as needing retraction or erratum (see listing of these articles in the Appendix). Most of these 102 articles were listed in the NIH Guide (Table 1); only seven of 102 articles were identified exclusively in the ORI Annual Reports. Forty-one researchers were named as responsible for the scientific misconduct that affected these 102 articles. As Table 2 indicates, the scientific misconduct of 22 of 41 respondents were said to have affected two or more published articles; the remaining 19 respondents were responsible for one affected article.

Table 1 Source of the notice of misconducta
Table 2 Number of articles affected by scientific misconduct by the number of respondentsa

Nature of the misconduct

The nature of the misconduct specified in either the NIH Guide or the ORI Annual Report is presented in Table 3. Most frequently, the misconduct involved fabrication, falsification or misrepresentation of the study results (79 of the 102 articles). In 16 articles, the study methodology was falsely reported. Study subjects were fabricated in five articles, and plagiarism occurred in two.

Table 3 Nature of the misconduct as stated in the “Finding of Scientific Misconduct”a

Accepting responsibility for misconduct

The “Findings of Scientific Misconduct” final reports often included brief statements about whether the implicated researcher/author acknowledged and accepted responsibility for the misconduct. Table 4 shows that 23 of 41 individuals accepted responsibility for the misconduct which affected 53 publications. Five respondents reportedly denied responsibility, or disagreed with the findings of the scientific misconduct investigation. The remaining 13 reports of scientific misconduct findings did not include statements about whether the respondents’ accepted or denied responsibility for the misconduct (generally, the earlier reports were less likely to contain this information).

Table 4 Acknowledgement of responsibility for scientific misconducta

Prescribed corrections to affected publications

The administrative actions included in the ‘Finding of Scientific Misconduct’, usually stated if an affected article should be corrected or retracted. Table 5 shows that the reports indicated that the corrigenda was already published for 32 of the 102 articles, and “in press” for another 16.Footnote 5 The misconduct report indicated that a retraction or correction was still needed for 47 articles. For the remaining seven articles, the finding of misconduct report did not state whether a correction or retraction was needed.

Table 5 Status of the corrigenda (retraction or erratum) at the time the ‘Finding of Scientific Misconduct’ was publisheda

Nature of corrections in PubMed

We investigated compliance with administrative actions for corrections and retractions to determine which of the 102 affected articles were tagged as corrected or retracted, the location of such corrigenda on the citation’s PubMed webpage(s), and the extent to which these actions might be apparent to PubMed database users.

As of May 2005, 98 of the 102 affected articles were found in PubMed (see Table 6). Eighty-five of these 98 articles had indexed corrections: 47 had a retraction; 26 had an erratum; and 12 had pertinent information in the PubMed “Comment” field. Although there was no notice of corrigenda for the remaining 13 articles, 10 had an open access link to the NIH Guide “Findings of Scientific Misconduct” that indicated the article was affected by misconduct. This left only three articles (A5, A24 and A91 in the Appendix) without any type of indication of an erratum or a retraction, or of the misconduct investigation (i.e. no link to an NIH Guide, to a journal correction, or to a revealing “Comment”).

Table 6 Citations to 102 articles affected by scientific misconduct by type of corrigenda and by open access status

Full access to corrections

The open-access links to the NIH Guide “Findings of Scientific Misconduct” provide the researcher with an official report about the nature of the misconduct investigation, the final determination as to whether the allegations were supported, and the specific articles affected by scientific misconduct, and the administrative actions for the affected articles. PubMed had open access links to the related NIH Guide for 67 of the 98 articles found in the PubMed database. However, this varied by type of correction: 72% of the 47 retracted articles had a link to the NIH Guide “Findings of Scientific Misconduct”, compared with 81% of the 26 articles with erratum, and only two of the 12 articles with a correction indicated in the “Comment” field.

We also explored the access permitted by our university’s journal subscriptions to the information published in the journals about the retraction or corrigenda for the 102 articles. Using our library subscriptions, we had full access to the journal correction for 36% of the 47 retractions; 23% of the 26 errata, and 40% of the 12 comment corrections. As journal subscriptions vary across institutions, so will researcher access to such detailed information. Although this study was conducted at a Carnegie I Research Institution, the generalizability of this level of access is unknown, and will vary for other researchers.

Citations to affected articles

As of the week of May 17, 2005 (the cut-off date for the accumulation of citations), the Web of Science database listed 5,393 citations to the 102 articles. The distributions of citations by type of corrigenda are illustrated in Fig. 2. Table 6 shows that the 102 affected publications had an overall median of 26 citations per article (range 0–592). The 13 articles without a linked corrigendum had a median of 36 citations. The retracted articles had a median of 27 citations; the articles with erratum had a median of 33 citations; and the articles with only a correction found in the “Comment” field had a median of 18 citations.

Fig. 2
figure 2

Number of journal articles that cite 102 articles affected by scientific misconduct (as of the week of 05/17/2005) by presence and type of corrigenda.a

aLower and upper edges of each box represent the 25th and 75th centiles of observed data. The line partitioning box corresponds to median observation. Whiskers include all observations lying within 1.5 times the interquartile range. Dots indicate observations beyond the whiskers

Discussion

The goal of this study of the occurrence and nature of corrections to published articles affected by misconduct was to explore the complex dynamics of biomedical communication [6] as related to correcting the literature affected by misconduct. A content analysis methodology identified all of the 102 published articles named in the final reports of scientific misconduct investigations from 1993–2001. We examined the PubMed database to determine if these articles had links to corrections or retractions, as per the administrative actions in these reports. Close examination of the PubMed web-pages associated with each article revealed how corrections, retractions and findings of scientific misconduct are indicated to database users. Finally, we considered the pattern of citations to these affected articles by their correction status.

The study methodology allowed citations to accrue for a minimum of 3 years after publication of the article affected by scientific misconduct (most but not all of which were tagged with notices of erratum or retraction). This methodology is not directly comparable to other studies reporting on citations to retracted articles. Pfeiffer and Snodgrass [20] defined post-retraction citations as those occurring in the next calendar year, and thus had only a six-month average washout period for citing articles already in the publishing process at the time of the retraction. Budd and colleagues [5] provided for a one-year period after publication of a retraction to allow time for its indexing before a citation was considered as post-retraction. Since the time lag from initial manuscript submission to publication can frequently take up to 12 months (or more), it is reasonable to assume that authors who cite articles affected by misconduct would be unlikely to know that an article was retracted or corrected, if such notices were inserted around the time of their manuscript submission. Our three-year time lag before accumulating citations provides a larger window to assume that subsequent citing authors could be expected to know about an article affected by misconduct.

Only 41 authors, responsible for 102 published articles, were named as respondents in these final reports of scientific misconduct investigations. When searching the PubMed database, an indication of some type of corrigendum (i.e. a retraction, erratum or correction in the “Comment” field) was identified for almost all of these articles. For the majority, the affected article was tagged as retracted or with an erratum. However, for some articles, the information was found on a subsequent webpage, either through links to an unlabeled “Comment”, or to a “Finding of Scientific Misconduct” in the NIH Guide for Grants and Contracts. Either an indexed correction (including “Comments”) or a link to the open-access NIH Guide was available for 95 of 98 articles indexed in PubMed.

While indexing errata and retractions in PubMed is essential to alert users to published articles affected by misconduct, full access links to the NIH Guide, or to a journal’s statements about the corrections, can help users determine the nature of erroneous information contained in an article affected by misconduct. Oftentimes the journal’s statements contained different information about the scientific misconduct than what was found in the indexed PubMed correction. Full access to both these sources is ideal because a researcher can compare the administrative actions in the NIH Guide’s “Findings of Scientific Misconduct” with the indexed journal corrections, thereby making an informed decision about the validity of the information in question. For example, five articles (A15, A41, A50, A83, and A99 in the Appendix) were corrected instead of retracted as prescribed in the administrative actions of NIH Guide “Finding of Scientific Misconduct”. Another article (A16) had a retraction tag in PubMed, but then had links to two errata (not retractions).

Limitations

Data collection for this study started in 2002, when the 102 published articles were identified from the official reports of scientific misconduct investigations, and first searched in the PubMed database. During 2003, the study database and data collection methodology was twice reviewed and verified. A final determination of the status of the 102 articles in PubMed was made during the week of May 17, 2005. Thus, the status of articles was checked four times over the course of the study. During these repeated inspections of the PubMed database, the dynamic nature of this wonderful resource became apparent. In particular, during 2003 several articles had newly linked postings to the NIH Guide “Findings of Scientific Misconduct”, compared to the initial 2002 data collection. Recognizing that the database is periodically updated, we reviewed and updated our study database in May 2005 so that the most current data possible is presented here.

The dates that retractions or errata or links to the NIH Guide were posted in the PubMed database are not provided. Thus, it is not clear that these elements were present at the time that a particular user cited the flawed article with a corrigendum or a link to the NIH Guide. In addition, although PubMed provides links to the open-access NIH Guide, it is not known if researchers actually use these links, and thereby learn the details of the scientific misconduct and associated administrative actions related to these articles.

Conclusion

Many links to the NIH Guide for Grants and Contracts “Findings of Scientific Misconduct” were posted years after the misconduct finding, if at all: 31 of 98 affected articles indexed in PubMed had no link to the public NIH Guide. Over the course of the study, the PubMed data base added links to the NIH Guide for several articles in the study population, making it easier for current researchers and database users to be aware of the problems with articles affected by scientific misconduct. However, it appears that the thousands of researchers who cited the 102 articles affected by misconduct were unaware of the finding of misconduct, and did not notice the retraction and erratum tags that were in place for most. Similarly, others have noted the continuing use of retracted literature [5, 6, 17, 20, 23, 24].

Most journals are not open access and on-line availability of corrections in the PubMed “Comment” is determined by institutional subscriptions, making it difficult for some to learn more about the particular details related to the corrigenda. Researchers should be alert to “Comments” linked to the open-access NIH Guide for Grants and Contracts, as its “Findings of Scientific Misconduct” usually provide the most detail about the nature of the problem in the affected articles and are often more informative than the statements about the retraction or correction found in the journals (which do not always reveal that the article was affected by scientific misconduct).

How can the continued citation of research affected by scientific misconduct be reduced? More prominent labeling in the PubMed database is desirable to alert users to notices of retraction and errata. This could take the form of larger or bold fonts for these notices. In addition, a prominent placement of the word “retraction” on the first page of such articles would be useful, because once a user downloads an article, these notices are left behind. Harold Sox and Drummond Rennie [23] recently delineated the responsibilities of institutions, editors and citing authors for preventing the continued citation of fraudulent research. Included among their recommendations are two that are particularly pertinent here: (a) authors submitting manuscripts for publication are charged with the responsibility to check each reference cited in their bibliography to see if it has been retracted; and (b) authors (or readers) who discover a published article contains a reference to a retracted article are responsible for submitting a correction to the journal [23].