Rationale for evidence-based management guidelines

Evidence-based guidelines are the goal for management of nearly all medical problems: clinicians practice evidence-based medicine and strive to do the best for all patients. However, it can be very difficult to know what the evidence actually shows. The pace of change in medical knowledge can be rapid, leading to difficulty in maintaining currency with the scientific and clinical literature. In addition, there can be conflicting literature. Therefore, clinicians have come to rely on reviews published by experts on certain topics, as well as expert lectures at national and international conferences. However such reviews can be prone to bias, with a single expert perhaps giving a skewed view of the literature. Perhaps the best way to review the literature is based on systematic application of evaluative criteria that address both the quantitative and qualitative components of clinical findings. There are now established procedures for reviewing papers and for assessing the potential flaws and biases within studies, as well as for combining those studies to produce a summative guideline in a particular area [6, 7].

Guidelines for the management of oral and gastrointestinal mucositis: the process

The Mucositis Study Group (MSG) of the Multinational Association for Supportive Care in Cancer (MASCC) was created in 1998 with the express purpose of developing treatment guidelines for the management of oral and gastrointestinal mucositis (GIM). Under the leadership of Dr. Edward B. Rubenstein (chair), and with the support of Drs. Mark M. Schubert and Douglas E. Peterson (cochairs), a panel of mucositis experts from around the World was formed. The literature review was performed by Mr. Ron Hutchins at the MD Anderson Cancer Center in Houston, with the input of the three leaders and the rest of the panel. The subject was divided into 13 subgroups relating to oral mucositis (OM) and one group of GIM. The reviewers addressed the development of mucositis management guidelines, as well as terminology and assessment scales, pathobiology, epidemiology, and risk factors.

A 3-day conference was held in January 2002, in Houston, at which time each group summarized the results of their component of the literature evaluation and presented them to the group for discussion. Rigorous debate then followed to ensure that there was uniform interpretation of the data and to agree on the recommendations for treatment and at what level of evidence, based on criteria developed by the American Society of Clinical Oncology (ASCO) [6]. This process was designed to produce evidence-based guidelines, as opposed to guidelines developed by consensus per se.

A writing committee was then formed to draft the guidelines paper. with the volume of data as basis, this committee determined that two papers were needed to comprehensively address the literature. The guidelines process had resulted in an increased understanding of the pathobiology, as well as in the links between OM and GIM, to the extent that the group defined the problem as alimentary mucositis (from mouth to anus) [8]. This modeling then led to the examination of other toxicities that might be related [9, 10]. The second paper therefore examined these issues, as well as the epidemiology and economic burden of mucositis. The two were published as a supplement in the journal Cancer in May 2004 [3, 5].

Guidelines promulgation and impact

The following actions were implemented to disseminate the guidelines:

  1. 1.

    The supplement containing the guidelines was given to all delegates attending the MASCC/ISOO annual meeting in Miami.

  2. 2.

    A link to the papers was established from the MASCC web site, with permission from the journal.

  3. 3.

    Members of the MSG presented and discussed the guidelines at regional, national, and international meetings.

  4. 4.

    A survey was conducted in 2004/2005, to determine how widely the guidelines had penetrated the Hematology/Oncology community [11].

Outcomes: “halo” effects

Several strategic advances have been fostered by the guideline process, in addition to the production of the guidelines themselves. These “halo” effects have been substantive and may in turn lead to new guidelines in the future. Examples include:

  • Creation of a group of international researchers with a common interest. The guidelines process has promoted interactive dialogue among researchers and clinicians, leading to cross-fertilization of ideas. Review of the literature has aided in defining the scope of the problem and the unanswered questions. This has led to new collaborations and development of a strategic plan for the MSG, with education and research as two key priorities. This in turn led to the development of a collaborative project with the educational study group of MASCC [11], as well as development of the Triad Mucositis Burden of Illness study, in which MASCC/ISOO members are site principal investigators (Sonis, Keefe, and Elting, personal communication).

  • Mucositis has become a MeSH category effective 1 January 2006, based on a proposal developed by the reviewers and submitted to the National Library of Medicine in March 2005.

  • The “whole-tube” paradigm has been defined. It will likely be substantive in its contribution to new research and clinical modeling in the future [8].

Guidelines update

Several specific strategies were implemented to facilitate production of the revised guidelines. A smmary of these strategies is listed in Table 1.

Table 1 Strategies designed to develop revisions of the original guidelines
Full size table

Potential pitfalls

There are several major complicating factors (“pitfalls”) that contribute to the complexity of guideline development.

Funding

Funding a comprehensive guideline development program is expensive and federal and/or private funding sources may not be readily available. Therefore, industry funding is vital. This leads to potential problems with conflict of interest, because the MSG is a group of experts, with most of the members having links with industry in the form of consultancies, speaker’s bureaus, clinical studies, and/or basic research. Only companies with an interest in mucositis would be expected to sponsor such an undertaking, and, again, this could lead to bias. Unrestricted educational grants are thus viewed as the most effective strategy to eliminate the actual or perceived conflict of interests that would otherwise arise. No specific corporate sponsor would then have input into the creation of any component of the guidelines. This was the approach utilized by the MSG.

Studies with major flaws

Studies with major flaws were commonly encountered during the guideline production process. The reviewers were frequently asked to evaluate small, sometimes nonrandomized studies that had been conducted as an exploratory research investigation to test proof-of-principle concepts. The mucositis literature seems particularly affected, perhaps as a result of the historically limited understanding of pathophysiology and consequent empiric treatments. Now that more robust mucositis studies are being performed and are leading to drug registration (palifermin [12, 13]), it is to be hoped that the number of limited studies will decline over time.

Another technique employed by the sponsors of some drugs, seems to be to fund multiple small, single-centered, Phase II trials, rather than committing the necessary resources to conduct a single, well-designed, randomized controlled study of adequate power to answer the required question. This approach has resulted in at least two problems. Firstly, the results of limited scope studies are difficult to interpret and may be conflicting, possibly even obscuring a positive result. Secondly, the key research questions are not typically definitively answered, and review groups such as the MSG are thus unable to recommend or suggest the use of these agents. Likewise, regulatory authorities will not approve drugs for registration without adequate, high-quality data.

Time to publication

Once the guidelines have been formulated, the process of manuscript writing needs to start. This is labor-intensive and requires ongoing commitment from the panel members. However, coordinating a large group of authors can be difficult, and the task of actual writing falls to a smaller number. A manuscript draft with edits from ten authors can lose all sense of internal consistency! It is important to decide on publication policy at the time of the consensus panel meeting, to ensure minimal conflict over authorship, order of listing of authors, and writing tasks. It is also important to set realistic timelines that balance workload with currency of guidelines. The longer the gap between consensus conference and publication, the more chances of the guidelines being out-of-date. Immediate web publication does reduce this problem.

Education and promulgation

Guidelines that nobody knows about nor uses are a waste of time. Therefore, the group needs a policy for education and promulgation [11]. This is one of the hardest areas to address and, again, needs to be built into the design of the guidelines process.

Lessons learned

Rationale for particular guidelines

It is worth asking at the beginning of the process what exactly you are reviewing, and why. Also, is there a particular question you want the review to answer, and what is in or outside of your area? With mucositis, the answers to these questions are:

  • We are reviewing the world literature on the treatment of mucositis, which is a morbid and economically costly side effect of anticancer therapy.

  • We want to be able to advise our colleagues and our patients on what is the state-of-the-art in mucositis management, and to use this information to drive future research.

  • We are doing it because we believe that there are too many poorly performed studies that cloud the area and that the mechanisms behind mucositis, as well as its cost to society, are poorly understood.

We initially divided the problem into oral and gastrointestinal mucositis, but the first guidelines drove us toward the whole-tube paradigm of alimentary mucositis, and we are now looking beyond alimentary mucositis to any mucosal damage in the body, and related anticancer therapy toxicities.

Scientific and clinical need

Guidelines for management of a given disease or toxicity may have already been developed by one or more professional groups. If no guideline exists in the literature, then addressing this gap for important clinical conditions is important. However, guidelines that are preexistent may be limited by any one of several key parameters, including the scope of literature review, type of scoring system utilized, and degree of applicability for contemporary clinical practice. However, multiple sets of guidelines that are produced by more than one professional group may lead to confusion and/or controversy in the clinical use setting.

Does the review end up impacting on the science?

This answer is a definite “yes” for mucositis guidelines. The guidelines have defined which interventions do and do not work, and what has been so obscured by poorly done studies that one cannot tell one way or the other. The scientific and clinical communities have an enhanced understanding of pathobiology and can use this to target potential treatment.

Creating a good mucositis study

After using the Hadorn [7] and Somerfield [6] papers to evaluate multiple clinical studies, it becomes obvious that it must be possible to use these same resources to help create a study that does not have major flaws. Furthermore, once a protocol has been designed, it can be assessed for potential flaws before beginning to recruit patients. In general, studies should be multicenter, double blind, prospective, adequately powered, and randomized controlled. They should study defined populations: patient (pediatric, geriatric), defined disease status, and its therapy. Validated tools should be used for outcome assessments, and account should be made for potential confounding factors. An appropriate analysis should be performed. It would likely even be useful to plan a trial and present it to the group with invented results, allowing critical analysis in house so that one can detect flaws and eliminate them before considerable amounts of funds are expended.

Underestimating the size of the literature base:

Having performed an initial review of the literature from 1966 to 2002, the MSG anticipated an increased publication rate. However, the reviewers were impressed yet surprised at the volume of papers published from 2002 to June 2005. There were over 3,000, having been 5,400 in the initial review.

It was imperative that the reviewers have access to a professional medical librarian for undertaking this review. Users of the guidelines need to be certain that the review of the literature has been truly comprehensive and that the chances of having missed an important paper were minimal to nonexistent. Organizing the process so that each group head liaised directly with the librarian translated to minimal misunderstandings during the search process itself.

The conceptual combining of the entire alimentary canal produced a need for new modeling of the literature searches, as well as the writing, because in the initial guideline production there was only one group with primary focus on GIM. The integration in the current process allowed all reviewers to embrace the more comprehensive view of alimentary mucositis, and it will undoubtedly lead to better insights into research.

With development of web-based technology and the fact that this group is international, the move from paper- to web-based activity is imperative. Use of teleconferences has been invaluable and establishment of the online database will aid future updates.

Need for adequate scoring system

The guideline production and revision process requires a valid, evidence-based literature scoring system that can be interpreted by the collective reviewers and that allows rapid dissemination of information. In both iterations of the guidelines, we used the excellent papers by Hadorn [7] and Somerfield [6] to establish the adequacy of each study and then to combine all the studies to provide a level of evidence and grade of recommendation. While it is intuitive that levels I and II evidence are good, it is less intuitive that levels III, IV, and V evidence are not as effective, with level V amounting to expert opinion. These issues represent a fertile area for further research and discussion, to avoid spurious decisions about recommendations and suggestions.

Calibration

It is imperative to calibrate reviewers to minimize inter-rater variability. This calibration was achieved in two ways by the group. First, each paper was studied and scored by more than one reviewer, then a consensus was formed to report to the full panel. The panel then discussed each result, leading to robust discussion regarding interpretation. Final guidelines were then agreed upon by the full panel of reviewers. Selected treatments fitted into more than one group’s area, and in this case, the review was assigned to the group most centrally addressing the issue.

It is worth noting that a panel of more than 30 international experts can be a challenge to convene, so that a strong chairperson with good communication and conciliation skills, as well as the ability to take disagreement calmly, is vital. It is imperative to have key supporters within the group that can aid in this role, and we have found having two cochairs to be ideal.

Planning

Proper planning requires time and effort. For the first iteration of the guidelines, planning began in 1999, with reviewers assigned in 2001 and the guidelines produced in 2002, with publication in 2004. For the update, planning began at the strategic planning day in June 2004 (1 month after the initial guidelines were published). Regular teleconferences were conducted throughout the remainder of 2004, with the literature review conducted between October 2004 and June 2005. The guidelines were updated on the MASCC web site in September 2005, and the manuscripts were submitted in February 2006.

In parallel with this, fundraising was conducted to pay for the process; all panel members gave their time at no cost. While the MSG has no official secretarial support, we were very grateful to MASCC for allowing us to use the central facilities to coordinate teleconferences, and to both the Executive officer and the conference planner for their substantial assistance with administrative tasks.

Impact of guidelines after publication

Production of guidelines is an essential but insufficient step toward a durable impact on clinical practice. Furthermore, successfully disseminating guideline updates is an additional and significant challenge. There are several key questions to be asked, including:

  • Does anyone read them?

  • Does anyone use them?

  • Do they help?

  • How soon are they out of date?

  • Is there a plan for regular updates?

  • Is an update appropriate, or has the field changed so that a completely new set of questions needs to be addressed?

  • What is the relationship of the guidelines produced by one group to the guidelines produced by other groups?

  • Should the respective groups coordinate development of one set of guidelines, based on the most effective components of each of the existent guidelines?

From the survey conducted on the mucositis guidelines [11], it seems that people are receptive to the idea of guidelines but that they are largely unaware of their existence. Only 1/3 of US respondents were aware of the 2004 guidelines, compared with 80% of Europeans, and all felt the need for educational materials associated with them. It seems unlikely that they are in regular use outside of centers where panel members and other highly motivated practitioners are working. In the first version of the guidelines where there were so few positive recommendations, it would be difficult to determine whether they helped, but with the new guidelines recommending the use of palifermin in certain settings, there is scope for considerable impact [4]. Our first guidelines were out of date within 1 year of publication, but the turnaround time for publication of the second version has been reduced considerably. From our knowledge of current clinical trials in mucositis, it seems unlikely that the new ones will be outdated in less than 2 years. For summary of important considerations see Table 2.

Table 2 Summary of considerations for producing clinical practice guidelines
Full size table

Lessons learned from update

While the MSG was updating its mucositis guidelines, our colleagues in the MASCC antiemetic study group were also updating their guidelines [2], and we have also learned lessons from them. The antiemetic group published a suite of papers in the journal Supportive Care in Cancer rather than a single update paper. In the first mucositis guidelines, we published two papers, one with the guidelines [3], and the other looking at epidemiology, cost, terminology and assessment, and pathobiology [5]. The papers were sufficiently substantive as to require publishing as a supplement to the journal Cancer and even so, much detail had to be excluded. Similarly, only ten members of the panel of over 30 could be authors. This time, we have written the summary guideline paper that contains the overall guidelines and will be the reference paper (Keefe et al., manuscript under review). However, we have also published the suite of papers in this edition of Supportive Care in Cancer, which is able to provide much more detail in each area, for those with a specific interest [11, 14–23]. It also properly acknowledges the efforts made by all members of the panel. Another innovation of the antiemetic group was to include all other groups that produce antiemetic guidelines. As we are the first to produce comprehensive mucositis management guidelines, this is not something we can emulate at this time, but we can submit our guidelines to other cancer organizations for their information and, hopefully, their endorsement. We would be pleased to collaborate with other cancer organizations in the future.

Plans for future updates

The MSG will continue to hold annual strategic planning meetings in concert with the MASCC annual meeting, and each year, the MSG will assess the progress of clinical trials that might lead to a need for updating the guidelines. It is anticipated that this will need to be done approximately every 3 years. This will require ongoing financial backing from industry and, perhaps in the future, granting bodies. We shall increasingly move to web-based review, but this will not remove the need for face-to-face meetings for crafting of actual guidelines.

Summary

The first update of the MASCC/ISOO guidelines for management of mucositis has now been completed. The MSG has refined its technique and made use of lessons learned, not only by experience, but also from other guideline groups. The impact of the guidelines on clinical practice worldwide has been difficult to assess, but it has been significant upon research in the area. Further work is needed regarding promulgation and education. Maintaining a protocol for systematic and timely updates of the guidelines continues to be essential to the process.