Evaluation and Milestones in Continuity Clinic

Abstract

Internal medicine residency continuity clinic offers an excellent opportunity to evaluate residents on their progressive achievement of the Accreditation Council for Graduate Medical Education subcompetencies. This chapter discusses evaluation methods and evaluation tools useful for continuity clinic.

Introduction

As of 2013, the Accreditation Council for Graduate Medical Education (ACGME) requires that residency programs have clinical competency committees that assess resident performance semiannually [1]. Residency programs may take this opportunity to redesign their end-of-rotation evaluation tools, including those used to evaluate resident performance in continuity clinic. Continuity clinic offers several opportunities for assessment, including longitudinal assessment of patient care skills, direct observation of clinical encounters, 360 ° evaluations from multidisciplinary team members, and evaluation of basic procedural competency. Several sample evaluation tools are publicly available, and online evaluation programs can assist in correlating and aggregating responses from individual evaluation forms.

Learning Objectives

• 1. To describe the Accreditation Council for Graduate Medical Education (ACGME) requirement for clinical competency committees to assess resident achievement in specific subcompetencies as defined by milestones.

• 2. To discuss opportunities and approaches to evaluate residents in continuity clinic.

• 3. To design meaningful evaluation tools that can be used by clinical competency committees.

Outline

• Overview of the ACGME milestones

• Evaluation methods and sample evaluation tools

• Evaluating observed patient encounters

• 360 ° evaluations

• Evaluation of procedural competency in clinic

Clinical Competency Committees and Milestone Evaluation

Since 2013, the Accreditation Council for Graduate Medical Education (ACGME) has required all residency programs to have clinical competency committees (CCCs) consisting of a minimum of three faculty members who review each resident’s evaluations and discuss their progress at least twice per year. Although the ACGME still requires that residents be evaluated in the six main competency domains (medical knowledge, patient care and procedural skills, interpersonal and communication skills, practice-based learning and improvement, systems-based practice, and professionalism), Internal Medicine CCCs must now assess resident achievement on each of 22 subcompetencies within those domains using descriptors called milestones [1]. Milestones are intended to be specific outcomes through which trainees demonstrate progress from the beginning to the end of training. See Fig. 1 for an example of a subcompetency and its component milestones. Each specialty field has its own specific set of subcompetencies and milestones.

Fig. 1

One of the patient care subcompetencies (PC5), as it appears in The Internal Medicine Milestone Project [1]. Reproduced with permission of the ACGME and American Board of Internal Medicine (ABIM)

Residency programs must report milestone assessments for each resident to the ACGME semiannually. Milestones are also reported at the end of each academic year to the American Board of Internal Medicine (ABIM) and at the end of training to the fellowship programs to which graduates have matched.

Residents are expected to demonstrate progress by achieving successive milestones until the achievement of competence for independent practice (“4”) in each domain. Residents do not need to achieve a score of 4 in every subcompetency in order to graduate from residency. Currently, milestone information is reported to the ACGME for data collection purposes; however, in the future, the ACGME may use this information to create national standards.

Continuity clinic provides an excellent opportunity to evaluate residents in the achievement of all six ACGME competency areas. Because residents are required to follow a panel of patients, continuity clinic may be one of the best venues for evaluating practice-based learning and improvement. Clinic preceptors can assess residents’ skills in systems-based practice as residents become increasingly familiar with clinic workflows, learn to work with other members of the clinic’s multidisciplinary care team, and assist their clinic patients in transitions between in- and outpatient care. Due to the years-long nature of their supervising relationship, as opposed to typical month-long rotations, continuity clinic preceptors are uniquely situated for longitudinal evaluation of residents and can observe progressive achievement of the ACGME milestones or lack thereof.

Continuity Clinic Evaluation Tools

The new CCC and milestone reporting requirements have prompted many residency programs to redesign resident evaluation tools with this end in mind, most commonly by “mapping” each question on end-of-rotation evaluation forms to pertinent subcompetencies and using a 1–5 scale that mimics milestone levels. As evaluators, this scale requires faculty to compare resident performance to that of a fully competent provider (1 = critical deficiency, 2 = an early learner who needs direct supervision, 3 = an advancing learner who needs indirect supervision, 4 = ready for independent practice, 5 = aspirational), rather than comparing residents to other trainees at their training level. This reset scale may not be intuitive to faculty members or to residents and requires both faculty development and resident education to recalibrate expectations. For example, faculty who may be accustomed to giving “5 out of 5” to a high-performing intern will need to adjust to the new scale, on which an intern who is meeting expectations may earn a score of 2 or 3. Similarly, residents who have grown accustomed to earning the highest possible scores may find a scale comparing them to a fully competent provider, rather than to their peers, jarring.

Faculty may not have adequate information to evaluate residents on each of the 22 subcompetencies during every rotation; however, “mapping” questions from evaluations on a variety of rotations can create a full picture of performance (see Table 1).

Table 1 Sample evaluation questions on a continuity clinic evaluation and ACGME internal medicine subcompetencies to which their responses may be mapped

Many online evaluation programs (e.g., MedHub, New Innovations, E*Value, MyEvaluations, and others) have the ability to map discrete evaluation questions to a central milestone document, pulling together information needed by the CCC in an easy-to-use format. Some residency programs choose not to map their evaluation questions to milestones in this way; instead, individual evaluations may be read and discussed by the CCC, which then assigns an appropriate milestone level of achievement for each subcompetency in a more general fashion. Using the former method produces an average score and/or range for each subcompetency and can streamline CCC discussions but relies on faculty development to appropriately calibrate all raters in order for meaningful averages to be produced from end-of-rotation evaluation documents. Using the latter method may require closer CCC faculty reading of each evaluation, which can be time consuming; however, it allows for the CCC to translate a broader range of numerical scores on evaluations to the appropriate text description of each milestone outcome. In practice, residency programs may choose to use a combination of these methods to best balance the spirit of the Milestone Project with the reality of time constraints for programs with many trainees.

There is no “best” way to construct or time clinic evaluation tools. Evaluations should meet your program’s learning objectives for continuity clinic and should ask questions that preceptors can reasonably be expected to observe. The Association of Academic Internal Medicine (AAIM) recently established an online peer-reviewed, curated milestone evaluation exhibit with publicly available evaluations organized by program size, setting, and rotation or clinical area of use [2]. Additionally, some online evaluation programs, such as MedHub, allow for administrators to import evaluation forms from other residency programs within or outside of one’s own institution.

Timing of clinic evaluations may vary with the degree of the longitudinal relationship between preceptor-resident, faculty willingness to fill out multiple or frequent evaluations, and level of detail of the questions asked. Some programs may have preceptors fill out clinic evaluations of residents monthly; others may take advantage of the longitudinal nature of continuity clinic to have preceptors fill out evaluation forms quarterly or biannually.

See Fig. 2 for a sample continuity clinic evaluation form from The George Washington University. This form is filled out twice per year by each clinic preceptor using a five-point scale.

Fig. 2

Sample continuity clinic evaluation tool from The George Washington University Internal Medicine Residency Program. Abbreviations in parentheses after each question signify the ACGME subcompetencies to which each question is mapped. PC patient care and procedural skills, MK medical knowledge, ICS interpersonal and communication skills, PBLI practice-based learning and improvement, SBP systems-based practice, PROF professionalism

Evaluating Observed Patient Encounters

The ACGME Residency Review Committee for Internal Medicine (RRC-IM) requires that assessment of resident competence in patient care “must involve direct observation of resident-patient encounters” [3]. These are most commonly referred to as mini-clinical exercises or Mini-CEXs . Direct observation and timely feedback are irreplaceable learning opportunities for residents and can be eye-opening for clinical faculty in assessing resident skills in history-taking, physical examination, procedures, and patients education or counseling.

Integrating direct observation into the busy clinic setting can be challenging. Brainstorming with your clinic faculty and staff may reveal best methods for your individual practice. For example, some clinics may choose to stagger resident appointment slots to avoid a backup in the precepting line when directly observing patient education at the end of an encounter. Others may set a standard wherein each preceptor directly observes one resident’s first history of the day, which typically occurs before preceptors are pulled by other learners or competing needs. It is important to note that time spent observing in Mini-CEXs need not exceed a few minutes; brief observations often reveal enough substance to generate both reinforcing and constructive feedback for the trainee. When observing history-taking, it is helpful to tell the patient that the attending’s role is one of a “fly on the wall to observe a couple of minutes of the resident’s technique, and then quietly leave the room while their visit continues.” The most important part of the Mini-CEX is the formative feedback given to the trainee privately after observation.

Several resources are available to evaluate Mini-CEXs, including free booklets that can be ordered directly from the ABIM. Mini-CEX evaluations can be made available to the CCC as additional data points for their milestone discussions. Mini-CEX evaluation tools may be designed with milestone mapping in mind (especially to the interpersonal and communication skills or patient care subcompetencies) or may instead be an opportunity for free-text comments that may inform richer discussion by the CCC. The free ABIM evaluation tool for Mini-CEXs uses a nine-point scale that does not easily map directly to milestones; however, it is fairly straightforward and allows observers to document complexity of the encounter as well as the focus (i.e., data gathering, diagnosis, therapy, and counseling) [4].

360 Degree Evaluations

The RRC-IM requires that assessment of resident competence in interpersonal and communication skills must include “multi-source evaluation (including at least patients, peers and non-physician team members)” [5]. Continuity clinic presents a relatively straightforward opportunity to ask medical assistants, nurses, social workers, and/or front desk staff to evaluate resident performance. Additionally, clinic patients may provide real-time evaluations of resident communication skills, professionalism, and patient care approaches. Ideally and if properly informed, patients in resident clinic should expect to be a part of the educational process of young physicians and may appreciate the opportunity to provide input into the training of their doctors. Because residents are assigned a panel of patients for whom they care longitudinally, clinic patients may be particularly invested in helping their residency primary care physicians improve their skills.

Although one could choose to map 360 ° evaluations to milestones, it is important to note that correct calibration of respondents generally requires extensive faculty and staff development. It is likely more feasible to use 360° evaluations, especially those filled out by patients and peers, for richer data to inform the overall CCC discussions of milestone achievement, rather than mapping raw responses directly to subcompetencies.

Evaluation of Procedural Competency in Clinic

The ABIM requires that residents safely and competently perform five standard procedures in order to be board eligible: Pap smears, IV placement, venous blood draws, arterial blood draws, and advanced cardiac life support (ACLS). Although most of these are inpatient procedures, Pap smears occur exclusively in the outpatient setting. Current Pap smear guidelines give residents fewer opportunities to perform this procedure so programs must be mindful of documenting and evaluating resident Pap smears in the outpatient setting, especially in continuity clinic.

All continuity clinic preceptors should be prepared to supervise and, if needed, perform Pap smears. Faculty members who are not competent to do so may either be instructed and precepted by colleagues until they are themselves deemed competent, or may directly “swap” precepting responsibilities such that another preceptor in clinic supervises their trainees’ Pap smears while they precept one of that colleague’s residents on another case. If the latter method is used rather than requiring all faculty to precept their own Pap smears, caution should be exercised to create a culture of real-time swapping of precepting responsibilities in order to maintain efficient clinic flow. Programs may choose to mirror milestone language (patient care #4 subcompetency: “skill in performing procedures”) in written evaluations of Pap smear performance, thereby ensuring an easily “mappable” data point for CCCs (see Fig. 3).

Fig. 3

The fourth patient care subcompetency (PC4), which focuses on procedural skills, as it appears in The Internal Medicine Milestone Project [1]. Reproduced with permission of the ACGME and American Board of Internal Medicine (ABIM)

Conclusion

Continuity clinic presents an opportunity for thorough evaluation of resident performance in each of the ACGME competency areas. Continuity clinic directors should discuss expectations for evaluation of residents with their residency program directors so that they can design and implement evaluation forms to maximize usability for preceptors, other evaluators, and the CCC. Publicly available and peer-reviewed evaluation forms exist that may meet the needs of continuity clinic directors and may enable one to avoid reinventing the wheel [2]. If desired, several online evaluation systems, such as MedHub, New Innovations, E*Value, MyEvaluations, and others, can facilitate mapping of individual evaluation questions to subcompetencies to generate averages for milestone achievement.