Introduction

Physician competencies have increasingly been a focus of medical education at all levels. Although competency-based education is not a new concept in education, when the Accreditation Council for Graduate Medical Education (ACGME) and the American Board of Medical Specialties (ABMS) jointly agreed on six competencies for certification and maintenance of certification of physicians in 1999 (Horowitz et al. 2004), it created a major shift in how medical education was viewed. While there has been talk of the continuum from undergraduate to graduate to continuing medical education, the means of actually making it a reality were limited by differing funding mechanisms and organizational structures governing each. However, the agreement and enforcement of competencies for certification and maintenance of certification may create an alternate pathway to progress along the continuum and a truly developmental sequence of instruction through all levels of medical education.

The move to introduce competencies into undergraduate medical education (UME) actually preceded the ACGME/ABMS competencies. Brown University created undergraduate competencies in 1996 (Smith and Dollase 1999) and several other medical schools have followed suit over the intervening decade. During this period, the AAMC instituted the Medical School Objectives Project, as a major initiative (Accessed: November 29, 2005).

In addition to penetration of the ACGME/ABMS to overall undergraduate competencies, they have been used as a basis for several of the specialty society’s preparatory materials for clerkship directors. The Clerkship Directors in Internal Medicine (CDIM) recently released version 3 of the Core Medicine Clerkship Curriculum Guide: a resource for teachers and learners in which all objectives are coded to the ACGME core competencies (see http://im.org/CDIM/). The Society of Teachers of Family Medicine released the Family Medicine Clerkship Curriculum Resource in 2004 that is heavily based on these competencies (see http://www.stfm.org/curricular/index.htm). Also, objectives contained in the Association of Professors of Gynecology and Obstetrics (APGO) 8th Edition of Objectives for Obstetrics and Gynecology Clerkships have been coded to the ACGME/ABMS Competencies.

Although most of the ACGME/ABMS competencies seem readily transportable to UME by adjusting the expectations to be appropriate to the level of the learner, there are two that will be challenging and may ultimately be too advanced for UME: practice-based learning and improvement and systems-based practice. Both of these competencies may require involvement with patient care and the system that provides it that is beyond the capability of medical students. Implementing these competencies at the UME level will take care to avoid making them either an insurmountable hurdle or a trivial exercise.

One of the major unresolved issues that will determine whether this becomes a reality is how competencies of all breeds, not just the ACGME/ABMS competencies, can be translated into the education of medical students. In this article, we discuss what competency means in UME, the distinction between competency-based and other types of curricula, obstacles and potential solutions to deriving competencies for UME from the ACGME/ABMS competencies, challenges that competencies pose for student assessment, and questions that need to be answered to progress to a competency-based curriculum with as few surprises as possible.

What does competency mean at the UME level?

One of the major challenges in medical education is how to move a learner from being a newly entering medical student to being capable of meeting the ACGME/ABMS competencies. Medical school can only move them part way and they must continue their education into residency to become even a minimally competent physician. So, how does a competency-based curriculum work when your product is a work in progress and cannot be a competent physician?

If graduating medical students cannot be competent physicians, what then can they be? McGaghie et al. (1978) describe what occurs in medical education as smaller, cumulative steps that ultimately lead to a competent physician. It could be argued that they should not be called competencies so as to not dilute the concept (i.e., it would be better to call them something different like “progress competencies, pre-competencies, sub-competencies, or proto-competencies”). Hansen (2001) used the term proficiencies in reference to economics education and Brown et al. (1973) called them subcompetencies. Quinlan started with an organ system curriculum that was defined by major areas that were then broken down into topics and then topics were divided into a logical sequence of learning objectives, terminal objectives, associated conditions and learning resources (Quinlan 1975). Weinstein and Russell identified core areas in psychiatry and worked from broad goals to specific performance objectives that were grouped under knowledge, skills and attitudes (Weinstein and Russell 1976). While we were sorely tempted to adopt such terminology for the specific competencies, in the end we think it would add confusion to what is already a complex concept. Instead, we will take the position that competencies are adaptable to stage of progression toward becoming a physician by adjusting the expectations for the type of outcomes that are expected. In this context, although the graduating medical student would not be becoming a competent physician, they would, however, be competent for proceeding to residency.

The competencies that under gird a competency-based undergraduate medical curriculum can be differentiated from instructional objectives by their being building blocks of a global competency rather than being a direct product of instruction. In previous work (Albanese et al. 2008), we proposed that competencies and their UME derivatives should have the following five qualities:

  1. 1.

    Focus on the performance of the end-product or goal-state of instruction.

  2. 2.

    Reflect expectations that are an application of what is learned in the immediate instructional program.

  3. 3.

    Be expressible in terms of measurable behavior(s).

  4. 4.

    Use a standard for judging competence that is not dependent upon the performance of other learners.

  5. 5.

    Inform learners, as well as other stake-holders, about what is expected of them.

Competencies can also have a wide variety of “granularity” depending upon whether one is dealing with overarching competencies such as those of the ABMS/ACGME, or the more specific ones that would be appropriate for the first semester of medical school. These would be much more targeted to the specific material being covered and the entering skill of the learner. Whereas goals set out the instructional agenda for a course of instruction and behavioral objectives define the expected outcomes of the instruction, competencies set out the expected practice behaviors of the physician and UME competencies work backwards to establish the expected skill set needed at each level of medical education in order to ultimately achieve the terminal competencies. Whereas goals and objectives are course specific, competencies horizontally and vertically integrate across the entire set of learning experiences occurring at a given time. Horizontal integration refers to the accumulation of expertise in the various concepts that underlay medicine (normal anatomy, physiology, immune system functioning) while vertical integration requires the ability to simultaneously apply multiple concepts in a way that facilitates application to patient care. UME competencies focus on the cumulative effect of all courses taken on the developing competence of the learner. An assessment of goals and objectives would focus on specific learning experiences whereas an assessment of competencies would be integrative across courses and be cumulative across all courses taken to that point. The graduating medical student would then be termed competent for entering the supervised practice of residency.

At this point, it will be helpful to show how a more traditional curriculum builds toward creation of a competent graduate and then contrast that to what occurs in a competency-based curriculum. For ease of description, we divide the 4 years of medical school into eight semesters, recognizing that this is a gross oversimplification. In Fig. 1, we show the typical structure that is based upon courses.

Fig. 1
figure 1

Model of the typical curriculum

Full size image

This model shows courses that progress from the basic sciences in semester 1 to the (s)elective clinical experiences in semester 8 to the competent graduate. Each learning experience has its own goals with a set of objectives that are used to create the learning experiences. In each learning experience, students are assessed by methods that are idiosyncratic to the particular learning experience. By the conclusion of the curriculum, a miracle has occurred and students are ready for residency. This is an over-simplified model that does not account for the integrated USMLE exams nor does it account for problem-based learning curricula which tend to be much more integrated. However, it provides a basis for describing how competencies and the outcomes that are used as evidence of competency attainment fit into the more traditional system.

Figure 2 depicts a competency-based curriculum.

Fig. 2
figure 2

Model of a competency-based curriculum

Full size image

Snapshots at semesters 1, 5 and 8 are provided to give a clear picture of the cumulative nature of the assessment process. Note also that we have removed the individual course assessment of students from the model because in a competency-based curriculum, student assessment can be course-based, but not necessarily so. Also not shown is that in an ideal competency-based curriculum, students should be able to test out of experiences if they have already mastered the knowledge or skills being taught. They also should be able to continue to work on competencies if they have not achieved the desired level of mastery at the completion of a particular instructional program. Thus, there is flexibility in skipping instruction that is unneeded and for continuing to work on competencies that are not achieved in what would be normally considered a “course”.

A competency-based curriculum starts with the qualities of the competent graduate and then slices and dices them into digestible parts (note that in Fig. 2, top left, we have chopped the competent graduate into eight parts corresponding to the eight semesters depicted) and distributes them in a developmentally appropriate manner across the curriculum. Course goals and objectives are modified by the presence of the competencies because students must demonstrate that they have achieved each competency before they are allowed to progress in the curriculum. Course directors can ignore the competencies, but do so at their own peril. Student failure to demonstrate adequate progress on competencies related to a course is likely to lead to questions about course quality and the appropriateness of course content and whether students are being adequately prepared. Competencies are demonstrated either at the point of individual readiness (perhaps even before undergoing instruction), or in large integrated assessments. These assessments may or may not supplant those that each course administers. The assessments of competencies become progressively more complex as the number of competencies accumulates across the curriculum. The competencies accumulate because the skills assessed become increasingly complex, not because the assessments become incrementally longer as the separate competencies pile up. Rather than a miracle happening at graduation, the last building block is put into place with a final assessment and the student is ready for residency.

What is a competency-based medical curriculum?

McGaghie and colleagues (1978) distinguished a competency-based curriculum from others by three features: (1) Organization around functions (or competencies) that are required for the practice of medicine in a specified setting; (2) Grounding in the principle that students of the intellectual ability of those admitted to medical schools, when given adequate preparation, unambiguous learning goals, sufficient learning resources, and a flexible time schedule, can with rare exceptions achieve the defined competence at high levels of proficiency; and (3) It views education as an experiment where both the processes of student learning and the techniques used to produce learning are regarded as hypotheses subject to testing (p. 18). They proceed to adopt a mastery learning framework for competency-based curricula and characterize it by three elements: “(1) knowledge of what a student brings to a learning task, not merely what is to be taken from it; (2) that broadly defined competencies of medicine be dismantled into smaller, cumulative steps, through which students may work at individual rates using many learning resources (books, laboratory experience, teachers, and other things) according to their own needs and rates of progress; and (3) that student achievement be thoroughly assessed at each learning stage in order to document the growth of competence and to provide valuable feedback on the quality of instruction.” (p. 19). We are not certain that a mastery framework is an absolute necessity for a competency-based curriculum, but it is certainly compatible with how such a curriculum should operate.

Merenstein et al. (1990) defined a competency-based curriculum to have the following two attributes: “(1) Learning objectives are defined in precise, behavioral, and measurable terms, and are known to learner and teacher alike; and (2) the learner is expected to demonstrate specific competencies under the conditions agreed upon” (Howsam and Houston 1972). From these two attributes, it can be seen why there is some confusion about what separates a competency from an objective.

UME competencies and the ACGME/ABMS competencies?

In this day and age, there is increasing interest in moving the ACGME/ABMS competencies into UME. The following are the six central competencies of the ACGME/ABMS:

  1. 1.

    Patient care provide patient care that is compassionate, appropriate, and effective for the treatment of health problems and the promotion of health.

  2. 2.

    Medical knowledge demonstrate knowledge about established and evolving biomedical, clinical, and cognate (e.g., epidemiological and social-behavioral) sciences and the application of this knowledge to patient care.

  3. 3.

    Practice-based learning and improvement investigate and evaluate their patient care practices, appraise and assimilate scientific evidence, and improve their patient care practices.

  4. 4.

    Interpersonal and communication skills demonstrate interpersonal and communication skills that result in effective information exchange and learning with patients, their patients’ families and professional associates.

  5. 5.

    Professionalism demonstrate commitment to carrying out professional responsibilities, adherence to ethical principles, and sensitivity to a diverse patient population.

  6. 6.

    Systems-based practice demonstrate an awareness of and responsiveness to the larger context and system of health care and the ability to effectively call on system resources to provide care that is of optimal value.

An example of a UME competency that could be derived from the first ACGME/ABMS competency for use in an internal medicine clerkship might be: compassionate care and management of a patient with high blood pressure assessed by a passing performance on the Medicine NBME shelf exam. Evaluating this competency against the five criteria we describe earlier, we find that the content of the UME competency, while much less expansive than the overarching ACGME/ABMS competency from which it is derived, clearly relates to an important skill and ability of a practicing clinician. While the specification of a specific medical problem (high blood pressure) may seem to focus on a course-related component, hypertension would be included in a list of common medical problems that the competent physician would be expected to have the skills and ability to manage after having completed their clinical instruction in internal medicine. Because of this broader focus on the general skills of the practitioner, it meets the first criterion.

Providing a focus on the treatment of a patient takes it beyond an academic discussion of the mechanisms of action leading to high blood pressure, thereby meeting the second criterion (reflect expectations that are external to the immediate instructional program). Although the first part of this approach to a competency seems good, the use of a shelf exam as an outcome to measure the competency might be problematic relative to translation into measurable outcomes, the third criterion. NBME shelf exams represent a broad spectrum and thus using them for assessing individual competencies in a specific area like high blood pressure is unlikely to be an adequate measure of competence. The exam probably contains relatively few items that pertain to this particular content area. Further, examinations are of questionable benefit for assessing such qualities as compassionate care. Generally, a shelf exam can be a good assessment of general knowledge in an area and potentially higher order thinking skills, it may be of limited use in either determining knowledge competence in specific areas or assessing competence for clinical skills that are not primarily knowledge-based (e.g., compassionate care). Relative to the fourth criterion, “Use a standard for judging competence that is not dependent upon the performance of other learners”, use of a shelf exam may or may not satisfy this depending upon whether the school’s own norms or the national sample were used in defining competence. The national data would technically be normative, but the sample is large enough that it would meet the fifth criterion for all practical purposes. A passing score on a shelf exam has some meaning to faculty and it does tell students very specifically what they must do to achieve competence. However, it may not give them or other stakeholders a very concrete understanding of what that competence means. Thus, the fifth criterion is not really met by this competency.

To be most useful in UME, the goal-state target of competencies must at, some level, reflect the terminal point of UME. Adopting a goal-state beyond readiness for the graduated patient care responsibilities of residency may make students better physicians eventually, but will probably lead to much remedial work and/or stress for students and faculty. The faculty must be able to justify the competencies and the outcomes used to measure them as being relevant for the needs of students. Setting them to a too remote and unachievable goal-state can be self-defeating and demoralizing.

Creating a competency-based curriculum

Moving to a competency-based curriculum is not a simple change. Much depends upon answers given to a number of questions pertaining to how it will be implemented.

How do you slice and dice the overarching competencies to achieve the specificity needed for curriculum design?

Slicing and dicing curriculum competencies into chewable bites is one of the major challenges of implementing a competency based curriculum. There have been many different approaches taken. Brown et al. (1973) clustered the specific competencies that they called subcompetencies into three categories: technical, conceptual, and human. Technical subcompetencies were basic understanding of anatomy and how organs interrelate to support function (reproduction in their case). Conceptual competencies were those that looked at issues from a “higher altitude” and looked at broader interrelationships such as anatomical and biochemical. Human competencies looked at issues from an even higher altitude, addressing how concepts operated to influence maturation, biological drives and emotional reactions.

Quinlan (1975) started with an organ system curriculum. Within this larger framework, organ systems were divided into major areas that were then broken down into topics. Topics were divided into a logical sequence of learning objectives, terminal objectives, associated conditions and learning resources. Learning objectives were distinguished from terminal objectives by being process and sequential in nature whereas terminal objectives were end-point and not sequential. Associated conditions allowed inclusion of admitting diagnoses and procedures drawn from morbidity tables for ambulatory and hospital environments regarding frequency of disease occurrence to be considered. Learning resources were grouped into specific and general types, with specific being those for which it was essential for the student to read because of their conceptual nature or because they are classics. General resources were those that students could explore in breadth and depth if time were available. Together, these provided students with what he/she should learn, the diseases, and the resources which would enable him/her to become competent in a topic.

Burg et al. (1976) defined competency in Pediatrics as a comprehensive list of behaviors needed by a pediatrician to care for children properly. They organized the list of behaviors using a three dimensional array consisting of a subject matter by task by ability matrix. Subject matter was grouped by organ system or body part in which signs or symptoms originate. The task dimension represented the major clinical activities in which the abilities of the pediatrician are demonstrated. It had three levels: (1) Gathering, organizing and recording data, (2) Assessing data, and (3) Managing problems and maintaining health. The abilities dimension had five levels: (1) Attitudes as represented by maintaining work habits and reaction patterns that reflect sensitivity, empathy, and devotion to continuing comprehensive care of patients and their families, (2) Factual knowledge, (3) Interpersonal skills, (4) Technical skills, and (5) Clinical judgment (Burg et al. 1976).

Clearly, there have been many ways in which overarching competencies have been defined in more specific and developmentally appropriate ways for creating instructional programs. There probably is no single best way. It may be necessary to try out two or three different methods before deciding on the one that “feels right.”

How will students be able to demonstrate their competence?

What will be considered evidence of competent performance for each competency? The path of least resistance is to determine which currently used measures can represent outcomes for the various competencies. However, this is likely to leave some competencies with limited attention, especially those that deal with complex interactions with patients. A common complaint voiced for over 45 years is the limited number of times that faculty observe students doing a history and physical during their clinical education (Engel 1976; Seegel and Wertheim 1962). If competencies are created to alleviate this problem so that students are evaluated on their history and physical exam skills, it needs to be determined how this will happen. Will faculty have to observe students live, or can it be videotaped for later faculty review? The assessment methodology and the criteria to be used in determining minimal performance needs to be considered for each competency before implementation. A competency-based curriculum also needs to consider the possibility for “fast-tracking” students who bring substantial skills with them to medical school (e.g., students who were nurses or physician’s assistants in a previous life).

Must students demonstrate competence for every competency and all outcomes?

Depending upon the granularity of competencies and their associated outcomes, there may be hundreds or even thousands of outcomes across an entire curriculum. It would be impractical to require students to demonstrate competency in each and every outcome. A system must be developed to address this in a reasonable manner. One approach could be to classify competencies into three types: those that are critical to be assessed in a very specific manner, those that can be assessed in a more general manner and those that can be sampled. An example of a competency that is written for specific assessment might be: for a newly diagnosed diabetes patient, effectively communicate changes in life-style needed to control their disease. An example of a competency written for assessment in a more general manner might be: for a patient needing to make a lifestyle change, effectively use motivational interviewing to help them make lifestyle changes. The following lifestyle changes would qualify for achieving this competency: smoking cessation, weight loss, increased activity, and reduced alcohol consumption. For sampling, one could group competencies according to similarities and then allow students to choose some minimum number from among them. For example, one could have competencies that include X-ray reading for various parts of the body. The competency might be for the student to be able to correctly identify fractured bones in three separate X-rays from any of the extremities. In other cases, there may be distinct competencies from which students will be expected to demonstrate competence in a minimum number (say 3 of 10). Care must be exercised in this type of sampling because left to their own devices; wind, water and students will all take the path of least resistance, not necessarily the path to a balanced demonstration of competence. If it is important that competency be demonstrated in a range of contexts, cases, assessment methods, etc.; these constraints need to be specified along with the minimum number of competencies. Consideration should also be given as to how the specific competencies from a group are selected for a given student. Should the student, faculty or a random process be used to select them?

How should competencies be parsed across the curriculum?

If students wait to demonstrate their competence until their last 3 months in medical school, it will overwhelm faculty as well as the other systems built around the competencies. This is also likely to leave some students so woefully behind that they could not hope to graduate on time, if ever. Thus, it will be important to set out expectations for what competencies should be complete by the time students reach defined parts of the curriculum. If students can speed ahead, that should be encouraged. And, for students who are lagging behind, this should be noted and efforts made to determine the reasons for the lagging. If lagging students are found to be struggling with the material, they can be identified at an early point in the curriculum and interventions taken to help them succeed. Having benchmarks parsed throughout the curriculum will give students the sense of urgency and authority to push faculty to help them learn the competencies and document their achievement in a developmentally friendly way.

When will students be able to demonstrate their competence?

The two extremes for addressing this issue are: whenever students feel ready and only in large group pre-scheduled assessments (e.g., exams). The key issue here is whether timing can be flexible or not. If flexibility in timing of assessments is desired for some or all the assessments, the practicality of that goal must be determined across the competencies. If faculty are needed to determine whether the competence has been achieved, then faculty must be available to judge the level of student competence. In a clinic situation, this may be feasible with little alteration in structure if students need only have faculty observe them interact with a patient during rounds to satisfy the competency (for example, to demonstrate their ability to introduce themselves to a patient). As assessing the skill becomes more time consuming and difficult to integrate into normal interactions between faculty and students, alternatives need to be found. For example, observing students doing a complete history and physical might be difficult to manage during a busy clinic. An alternative could be to have the interaction recorded as it occurs with a standardized patient in a clinical assessment center. Faculty could then review the student’s performance as they have time available. Ultimately, all competencies will have to be given this type of consideration.

An important related issue is whether some competencies should be assessed at multiple times (i.e., in a serial fashion) or whether a “one time” assessment will suffice. This is a complex issue because both knowledge and skills tend to decay if they are not used (reinforced) and skills, especially, have been demonstrated to have a high degree of content specificity (e.g., a student may be competent to perform a physical examination for a patient presenting with symptoms for congestive heart failure, but not competent in performing a physical examination for a patient presenting with symptoms for diabetic retinopathy). Knowledge-based assessments are probably just as susceptible to content specificity, but most examinations contain enough items on them to yield generalizable estimates of knowledge. Assessments of performance of clinical skills, however, are generally time consuming and constrained by the availability of patients with the desired variety of presenting problems. Thus, obtaining an adequate sample of student performance of clinical skills to demonstrate generalizable competence is very challenging. For skills that are considered to be essential for competence, such as performing a physical examination, they should be assessed in at least two different contexts.

How will student achievement of competencies be documented and filed away?

Documenting and filing away competency achievement in a readily accessible form is a critical step. The mechanism needs to be easy to use and the result needs to go into a central data-base for tracking. Further, thought needs to be given as to whether all or some competencies need to have a faculty signature attached as verification. Some schools have adopted Personal Data Assistants (PDAs) as the mechanism for entering competency attainment into a central record bank. Faculty signatures or some other means must be linked to this data-base for verification.

How will monitoring and notification of competency attainment be accomplished?

Monitoring competency attainment is a necessary evil. Related issues are how many times should a student be allowed to attempt to demonstrate competence before “pulling the plug” and whether students must achieve competence on all competencies or only a fixed percentage of them (e.g., 80%)?

Deciding why you are monitoring student progress is an important early decision. At one level, one can provide guidelines and let students monitor their own progress. This assumes that students are responsible adults and they will take action if they are not making satisfactory progress. While this may be true for the large majority of students, there will be those who need some external prompting to keep them from getting hopelessly behind. Monitoring that can be automated is probably the best approach because it avoids human error. Automated notification of individuals in authority to contact students who are falling behind will also be beneficial. There may be some issues beyond academics that can impede student progress, so having access to counseling support as well as academic support is also important.

How will competence be measured?

While last, but not least, the final issue we address is the measurement of competencies. This is no small challenge and there have been books written about this problem (Jaeger and Kehr-Tittle 1980). We will touch upon the major issues and refer you to other resources for more in depth treatment.

If competencies designed for certification and maintenance of certification are to be used for UME, careful consideration needs to be given to which are more and less appropriate and then to what standard each shall be held. The simpler the approach used to assess competence, the easier it will be to implement and sustain. This is not to say that complex methods of assessing competencies will not be needed. One criterion for judging competencies and the measurable outcomes derived from them is that they reflect a complex set of skills. Measuring complex skills is likely to require complex measurement techniques. Fortunately, there have been advances in assessment technologies such as computerized manikins like Harvey, airway management systems, computer simulations, 3-dimensional imaging, Objective Structured Clinical Exams (OSCEs) and the like that may make complex measurement procedures relatively manageable. It should be mentioned that if it can’t be measured, it is a quality that will be hard to justify in a competency-based curriculum. The declaration of a graduate as competent for residency makes it incumbent on the medical school to provide documentation.

A key issue in competency-based education is whether competency is a binary decision, or multi-level. Is a physician either competent or incompetent; or can he/she be not only competent, but also very competent or extremely competent, etc.? One could argue that even incompetence may be multi-level with some unfortunate individuals being not just incompetent, but EXTREMELY or PROFOUNDLY incompetent.

Talbot (2004) argues that the danger in a binary graded competency system is that rather than the floor, competence becomes the goal.

Extending this argument to the UME level would require deciding if competence is a yes/no decision or whether there is something in between and things on either end. A multi-level interpretation of competence would be better able to avoid having minimum competence overtaking excellence as the goal in a competency-based curriculum. As mentioned earlier, several schools have already begun to implement the ACBME/ABMS competencies at the UME level. Kansas is adopting a portfolio approach with specific tasks delineated for each competency. Southern Illinois University (SIU) has adopted a 5-option rating scale consisting of: unsatisfactory, marginal, meets expectations, commendable, excellent. The University of Illinois (UI) has adopted a 5-option rating scale, with norm-based anchors: 1 = significant problems requiring formal remediation (≤1%), 2 = Some problems requiring attention (≤2–5%), 3 = Satisfactory (~15%), 4 = Excellent (~60% of entire class), 5 = Outstanding (approximately top 20% of entire class). Thus, SIU and UI both have used multi-level (5-level) scales. It will take time to see how these different approaches work and to determine if the ABMS/ACGME Six Competencies translate well to UME.

A second key issue is how to set standards to use in UME. If we use a 5-level continuum like that used at SIU, we need to determine where learners lie in that continuum. Plus, we need to determine expectations for each of the five pivotal points. In determining ultimate competence at graduation, the issue becomes whether a student has achieved enough that they can be considered sufficiently competent for supervised practice.

However, even this type of assessment is fraught with difficulty. Minimum competence as a concept has much in common with minimum passing in the measurement realm and establishing minimum passing standards has been a challenge in the measurement literature since at least 1954, when Nedelsky published the first generally acknowledged method of setting minimum pass levels for multiple choice items (Nedelsky 1954). Since that time, many different standard setting methods (Livingston and Zieky 1982; Buckendahl et al. 2002) have been developed and their strengths and weaknesses explored. One major problem with methods that rely on expert judgment is that the standards set tend to exceed the performance of examinees. If a large number of examinees fail an exam, it can be due to faculty having unrealistic expectations. To compensate for this tendency, testing agencies often provide normative data for the items being considered so that judges have a benchmark to calibrate their expectations. Institutions are unlikely to have that data when they initiate a competency-based curriculum, so unless careful attention is paid to this tendency, faculty may have unrealistic expectations for what students should be able to do. This can lead to a large number of failures. Because faculty will often be assessing student competence independently (and usually with very few students at any given time), they may not see (or care about) the big picture impact of their actions. If high failure rates continue unchecked, it can have important and sometimes catastrophic results. This is also a pressure point where minimum competence can overtake excellence as the goal in a competency-based curriculum.

A third key issue is how to measure competency, especially in the less observable domains such as knowledge and attitudes. As Merenstein et al. note as a danger in competency evaluation, “There can be a tendency to measure what is measurable rather than what needs to be learned” (p. 469) (Merenstein et al. 1990). Examinations have been the traditional means of assessing knowledge. There have even been theories of measurement that would seem to fit well with competency based assessment, such as domain-based assessment and mastery testing. More commonly, tests have used minimum passing levels to determine competence from incompetence. The USMLE Step exams use this approach. The problem for UME is that individual schools generally do not have enough data to draw upon to set minimum passing levels that have the psychometric quality of those set on the USMLE exams. This leaves the determination of passes or fails to be relatively error prone. While capricious application of standards in a high stakes manner might cause legal challenges, providing multiple opportunities for failing students to demonstrate competence can blunt any problems associated with the error proneness of any given outcome assessment. For example, instructor-made examinations can have reliabilities as low as .40. This means that there is a 60% likelihood of making an error in a pass-fail decision. However, if one allows a student to demonstrate competence three times before taking action, the likelihood of failure on three independent efforts is only 21.6%. For reliabilities in the range that one usually is aiming for (>.60), the likelihood of three independent failures is at most 6.4%.

The final key issue that we will deal with is how to grade in a competency-based curriculum. The 5-option scales used by SIU and UI offer alternatives that can be considered. It would also seem that a pass-fail grading system would be the most compatible with a competency-based curriculum. If a pass-fail approach were to be adopted, the additional designation of honors could be added for Alpha Omega Alpha (AOA) designation and the need for finer performance documentation for competitive residency programs. Further, every test and every form completed provides the potential for a numerical score. There is nothing to stop a school from adopting a pass-fail grading system but using an accumulation of points across all standard measures to designate AOA status. May describes accommodating an A–F grading system mandated by university policy by setting the lowest boundary of a C to be minimum competency at either 75% or 80% and then allocating higher grades according to fixed percentages. Perhaps the most audacious recommendation we encountered also came from May who stated that “The goal is to have at least 90% of the students achieve 90% of the competencies 90% of the time” (p. 28) (May 1976). Ironically, when faced with the constraint of having limited ability to provide flexible scheduling, the standards were relaxed to 75–80%. This points to the pitfalls of setting high levels of performance as a universal standard.

Faculty buy-in and faculty development

If a competency-based curriculum is to have a ghost of a chance, it will require faculty buy-in and massive amounts of faculty development. Buy-in is absolutely necessary across the spectrum because faculty have to commit to their role in teaching and assessing competencies. The more assessment of competence falls on individual faculty to document, the more commitment and faculty development will be necessary for them to fulfill their role.

Appendix A gives a description of the Indiana University School of Medicine (IUSM) experience implementing a competency-based curriculum and highlights why faculty buy-in and faculty development are critical to the success of a competency-based curriculum

Discussion/summary and educational implications

In this article, we attempted to give an overview of the development of the competency-based curriculum movement, especially since the ACGME/ABMS competencies were published and to give some guidance as to how to go about developing a competency-based curriculum of your own. We think you can probably get a good sense for the agony that comes with the process, but you may not be seeing much of the ecstasy. However, there is satisfaction in seeing growth and a competency-based curriculum tends to make that growth more explicit due to the continual need for students to demonstrate skills for faculty sign off. If student growth does not make you ecstatic, there is not much else to get excited about because a competency-based curriculum is really more of a quality control mechanism that functions by giving students greater control over their education. Quality control and relinquishing control are not things most faculty get ecstatic about. However, quality control and relinquishing control are increasingly necessary as the clinical environment becomes increasingly hostile to medical students. Faculty are overburdened with demands to see ever more patients, hospitalized patients are “sicker and discharged quicker” than in the past and outpatients are suffering from a world economy that operates 24/7, so spending time with students is something they often cannot do. The list of challenges to medical education have been well documented (Cooke et al. 2006). A competency-based curriculum is not really creating a product that is a different type of graduate from that produced by any other type of curriculum; however, what it does produce that is different is a more uniform product that does not have the random gaps in core skills that are likely to get only worse as the physician shortage begins to take hold. But the point is if we are going to maintain the quality of medical education in the face of so many problems and potentially even expand by 30% to meet the anticipated shortfall of physicians in the US, we are going to have to institute quality control measures and relinquish control to the people who care the most about medical education, the people who are receiving it, our students. That may not be ecstasy, but it will have to do.